Feature Geo Analytics and Big Data Processing: Hybrid Approaches for Earth Science and Real-Time...
-
Upload
dawn-wright -
Category
Science
-
view
107 -
download
0
Transcript of Feature Geo Analytics and Big Data Processing: Hybrid Approaches for Earth Science and Real-Time...
Feature Geo Analytics and Big Data Processing:
Hybrid Approaches for Earth Science and Real-Time Decision Making
Mansour Raad, Erik Hoel, Michael Park, Adam Mollenkopf, Dawn J. Wright
Environmental Systems Research Institute (aka Esri)
IN12A-01 (Invited)
AGU Fall Meeting, 12 December 2016
What is Feature Geo Analytics?
A new way of processing spatiotemporal data designed for WEB-BASED big data by leveraging distributed analytics and storage
• Works with existing GIS data and tabular data
• Designed to perform both spatial and temporal analysis
• Uses familiar workflows to complete complex analyses
• “Hybridity” - integrating open-source frameworks on clusters to run analytics
Feature Geo Analytics
Geoprocessing
Distributed analytics and storage
Feature Geo Analytics
Portal
Web GIS Layers
newmore extends
Solve New Problems
Run analytics:
• against data too big for a single desktop machine
- Buffer 8.2 million points or thousands of polygons in a little over a minute
- billions of observations of ship movements ingested via GeoEvent
• designed to gain insight into both spatial and temporal patterns
• against massive collections in a scalable manner
• and meet time constraints
months weeks days hours minutes
Geo Analytics Architectural Overview
Portal
Web GIS Layers
Un-Managed Data
New Web GIS Layers
Register large data stores, then distribute spatial analysis across cluster of machines for parallel processing
Store and/or deploy to web
Web GIS layers via Pro, Portal,
Python Notebooks, or the REST API
Managed Data
Relational Data Store
SpatiotemporalData Store
FilesFiles
Delimited Files EnterpriseShapefiles Big Data Stores
Server
Cluster
Rich Collection of (Web) Analysis Tools
Summarize DataAggregate PointsSummarize NearbySummarize WithinReconstruct TracksJoin Features
Find LocationsFind Existing LocationsFind Similar Locations
Analyze PatternsCalculate DensityFind Hot SpotsCreate Space Time Cube
Use ProximityCreate Buffers
Manage DataExtract Data
* Temporally aware tools
Aggregate Points
Summarize Nearby
Summarize Within
Find Existing Locations
Find Similar Locations
Calculate Density
Find Hot Spots
Create Buffers
Extract Data
Analytical Overview: Aggregating and Summarizing
• Spatial Joins
• Space-time slices
• Spatiotemporal joins
Target Features Join Features Intermediate Result Final Result
Analytical Overview: Aggregating and Summarizing
Temporal Relationships on Intervals
• Points into Bins
Analytical Overview: Aggregating and Summarizing
Aggregation – Polygons vs Cells
Aggregation By Polygons Aggregation By Cells
• Reconstruct Tracks
- Summarize time-enabled points into tracks
Analytical Overview: Aggregating and Summarizing
Use Case: Hurricane Tracts
• Hurricane dataset
- 120,000 points, ~100 years
- Each point has:
- ID number
- Location
- Date
- Wind speed and pressure attributes
- Problems?
- Difficult to visualize that many points
- Difficult to visualize hurricane path
“Hybridity” for Distributed Computation
See also www.esri.com/software/open
“Hybridity” for Distributed Computation
See also www.esri.com/software/open
Real-Time GIS PerformanceArcGIS 10.4
10s of thousands of e/s
ArcGIS Spatiotemporal
Big Data Store
DesktopWeb Device
ArcGIS Server
4,000
e/s
Ingestion
GeoEvent
4,000
e/sVisualization
Live and Historic
Aggregates & Features
Enhanced Map and
Feature Service
• Ingest high-velocity real-
time data
• Observations in a Big Data
Store
• Visualize high-velocity,
high-volume data
- as an AGGREGATION,
- as discrete FEATURES,
- live & HISTORICALLY
• Visualizations CAN scale
Stream Service
Stream Layer
3,000
e/s
Live Features
Geo Analytics Performance
Spatiotemporal
Big Data Store
Discussion groups at geonet.esri.com
Step 1. Click orange “Join in” button to create your
account.
Step 2. Join the Big Data or Sciences groups
Step 3. Contribute to AGU conversations!
Mansour Raad, Esri Big Data Team
thunderheadxpler.blogspot.com
github.com/mraad
@mraad
For Questions/Discussion