ArcGIS Data Store, available with ArcGIS Enterprise, works with observational data by way of the spatiotemporal big data store. Observation data can be moving objects, changing attributes of stationary sensors, or both. The spatiotemporal big data store enables archiving of high-volume observation data, sustains high-velocity write throughput, and can run across multiple machines (nodes). Adding machines adds capacity, allowing you to store more data, implement longer retention policies of your data, and support higher data write throughput.
Requirements for using the spatiotemporal big data store
The spatiotemporal big data store requires ArcGIS Enterprise. For more information on ArcGIS Enterprise, see What is ArcGIS Enterprise?. For more information on base ArcGIS Enterprise deployments, see Base ArcGIS Enterprise deployment. For more information on adding functionality and capacity to your site with additional servers, specifically ArcGIS GeoEvent Server, see Additional server deployment.
When enabling the spatiotemporal big data store in ArcGIS Data Store, you must register the spatiotemporal big data store with your portal's hosting server. A data store connection to your portal must be registered with GeoEvent Server, in GeoEvent Manager, to view and write to the spatiotemporal big data store. ArcGIS Server manages multiple nodes of the spatiotemporal big data store to form a data ring.
Note:
When archiving streaming data into a spatiotemporal big data store, depending on the size of the data, disk space can fill up quickly. Administrators need to monitor the spatiotemporal big data store logs to ensure machines have sufficient disk space. If the disk space on a machine exceeds its capacity, new data will not be archived in the spatiotemporal big data store. For more information on ArcGIS Data Store and managing disk space, see Disk space requirements in the ArcGIS Data Store system requirements.
Send data to a spatiotemporal big data store
GeoEvent Server provides output connectors for writing observational data to a spatiotemporal big data store.
- Add a Feature to an Spatiotemporal Big Data Store Output Connector allows you to add new features.
- Update a Feature in an Spatiotemporal Big Data Store Output Connector allows you to update existing features.
- Send Features to a Stream Service Output Connector allows the latest observations (based on Track ID) broadcast by a stream service to be persisted (stored) if Store Latest is enabled.
Work with data in a spatiotemporal big data store
The high-volume observation data stored in a spatiotemporal big data store can be accessed through a feature service and visualized using a map service. The map service provides dynamic rendering and aggregated data representation, allowing observations to be visualized at various levels of detail specific to each user session. The map service's density aggregations are produced on the fly as you zoom in so a web map's representation of your data updates dynamically. This is important for real-time visualization use cases because observations are constantly updating; the data representation displayed by a web map must reflect the data most recently received.
The high-volume observation data stored in a spatiotemporal big data store can also be accessed through a feature service. Feature service queries to a spatiotemporal big data store are returned very quickly. You can query by any combination of ID, time, space, and attributes. The raw features in a spatiotemporal big data store can be visualized on the client side using the feature service. However, due to the volume of features, you will likely only want to view raw features at large map scales.
It is recommended that you configure scale-dependent rendering to switch between a map service layer and the feature service layer for a better visualization experience. The map service renders the aggregated data at various levels of detail at the smaller scales. The feature service turns on when you zoom in beyond a specified map scale appropriate for the visualization of raw features.
The high-volume observation data that map and feature services access is stored in a data source. Once a data source exists, it is possible to create new, edit existing, or delete any map and feature services that reference the data source. This can be done using the settings available on the Spatiotemporal Big Data Stores page in GeoEvent Manager. Data can also be deleted from a data source entirely, thereby removing the raw features returned by corresponding feature services, or the aggregated data representation returned by corresponding map services.
While the option exists to delete all of the data stored in a data source, or even delete the data source, map, and feature services entirely, it is recommended you employ a data management strategy using the data retention settings available in the data source itself.
Export data from a spatiotemporal big data store
The high-volume observational data stored in a spatiotemporal big data store can be exported as delimited text to a cloud store registered with your ArcGIS Server. Supported cloud stores for exporting data include Amazon Simple Storage Service (S3) and Microsoft Azure Blob Storage.
Exporting data from a spatiotemporal big data store is done on the Spatiotemporal big data sources page in GeoEvent Manager. You will need to provide the server connection, registered cloud data store, file name, and an optional where clause to do a selective data export.
Get started with the spatiotemporal big data store
To get started working with the spatiotemporal big data store, refer to the ArcGIS Data Store help and the Spatiotemporal Big Data Store Tutorial available from GeoEvent Server tutorials.