ArcGIS API for Python: Imagery and Raster Analysis on Your ...
Image Server: Imagery and Raster Analytics · ArcGIS 10.4 Image Processing • ArcGIS 10.4 has...
Transcript of Image Server: Imagery and Raster Analytics · ArcGIS 10.4 Image Processing • ArcGIS 10.4 has...
Image Server
Imagery and Raster AnalyticsPeter Becker
Manage and process imagery into authoritative data
sources that are appropriately and efficiently disseminated
to those that need access
Enable access to imagery and analysis through a wide
range of integrated desktop, mobile, and web applications
that are interactive, informative, and engaging
Derive actionable information from imagery and
rasters by performing analytics on massive
volumes of data available from multiple sources
ArcGIS
GIS Server
ArcGIS
GeoEvent ServerArcGIS
Image Server
ArcGIS
GeoAnalytics Server
ArcGIS
Business Analyst ServerArcGIS
Image Server
Desktop
Server
Enterprise / Online
System of
Systems
A Distributed Platform built on Open Geospatial Services
ArcGIS Supports Multiple Implementation Patterns
An Interconnected Network Is EmergingServices
DataData
Web / Mobile / Cloud
Services
Information Model
Data
Identity
Apps
APIs & SDKs
ArcGIS Desktop
The ArcGIS Platform Information Model
ArcGIS EnterpriseArcGIS Online
Scene Layers
Scenes
http:
http:http:
TileLayers
Maps
Services
Data
FeatureLayers
ImageLayers
Analytics Analytics
ArcGIS Image Server 10.5
Image Server
fast on-the-fly dynamic processing
caching and serving tiled
maps
OGC services
serve and analyze scientific
data
scalable raster
analysis and image processing
weather & climate
WCS
NetCDF
HDF
GRIB
vegetation analysis
spectral processing
suitability analysis
terrain analysis
multidimensional analysis
WMS
KML
multidimensional
custom algorithms with Python
design multi-source, multi-LOD tiled services
burn geographic featuresand text into tiles
watermarking
update AOIs
scale tile creation with addition servers
persistent product generation
compliance & standards
store once, many products on-the-fly
reduce storage costs
only process what’s being looked at
100+ analytic functions
http
http
Web Maps(reports)
Web Apps
Story Maps(reports)
Powerful Desktop Apps
Mobile Apps & Devices
Developer Apps
Production Systems(automation)
Systems
Integration
new
compression control for low bandwidths
orthorectifcation and mosaicking
Imagery ArcGIS Can Process Massive Imagery in the Cloud
• . . . A key aspect of the ArcGIS Platform
Enabling
• Imagery Dissemination
• Raster and Image Analysis
• Content Access and Management
Massive
Image Storage
ArcGIS
Image ServerAny Image Source
Process imagery dynamically and serve directly into applications
Process imagery at source resolution and save for multiple uses
Image Services
ArcGIS 10.4 Image Processing
• ArcGIS 10.4 has scalable high
performance analysis of big rasters
and imagery for visual analytics
• On-the-fly processing of massive
images and massive image collections
• Desktop and server
• Visual results can be exported
Raster Functions
Desktop
Server
can be run on the server
Dynamic Raster Models
Mosaic Datasets Image Servicespublished
can be run on the desktop
ArcGIS 10.4 Geoprocessing
• ArcGIS 10.4 has high performance analysis
of standard rasters and imagery for
persistent analytics
• Processing of single images or spatial
subsets of massive images or mosaics
• Desktop and server
• Persistent results
Geoprocessing Models
Geoprocessing Tools
can be run on the desktop
Desktop
GP / SA / Data
Server
GP Servicespublished
can be run on the server
. . . Processing is distributed to Servers in a Cluster. Each process creates its own output file
What is ArcGIS Image Server 10.5
Server dedicated to the
Efficient Processing, Analysis and Disseminationof Imagery and Rasters
• Dynamic Image Services – ‘Making your imagery accessible’
- Serve large collections of imagery and rasters with dynamic mosaicking and on-the-fly
processing
• Raster Analytics – ‘Extracting information from imagery’
- Enabling massive distributed processing and analysis of imagery and rastersnew
Image Server is an additional server capability available with ArcGIS Enterprise 10.5 new
What is Raster Analytics?
A new way to create and execute
spatial analysis models and raster processing chainswhich leverages distributed storage and analytics
• Works with your existing imagery, rasters and GIS data
- Register your local data, use mosaic datasets or image services
- Optionally import data into distributed storage
- Process single massive rasters or large collections
• Enable massive distributed processing and analysis
- Persist dynamic imagery products
- Run spatial analysis as predefined models
• Integrate Analytics into your workflows
- Use ArcGIS Pro to Author models
- Outputs are rasters or features layers on your portal
Foundational Concepts
Raster Analytics adds to existing ArcGIS foundational concepts
Dynamic Raster
Models
on-the-fly processing
Geoprocessing
Models / spatial
analysis
powerful analytics Scalable distributed analytics
with persisted storage
Server-based distributed
processing and storage
Portal
Web GIS Layers
newmoremore extends
Solve New Problems with Raster Analytics
• Run models against data that is too big for single desktop
- Global rasters (big geography)
- Large Scale (high resolution)
- Large Collections (many)
• Run models and meet time constraints
months weeks days hours minutes
Raster Analytics Conceptual Overview
Enterprise
GIS
Web GIS Layers
GIS Data & Imagery
ArcGIS Pro
Developers &
System Integrators
GdbFiles WCS ServicesArcGIS Services
Web Analysis Tools
on Portal MapViewer
Desktop Web Device
Design & Run Models
Image Server
enable Raster Analytics
Raster Analytics Conceptual Overview
Enterprise
GIS
Web GIS Layers
GIS Data & Imagery
GdbFiles WCS ServicesArcGIS Services
Desktop Web Device
Design & Run Models
import and optimize
(optional)
distributed raster analytics cluster
Model Execution Distribution
distributed raster datastore
ArcGIS Pro
Web Analysis Tools
on Portal MapViewer
Developers &
System Integrators
Model Execution Distribution
Raster Analytics Conceptual Overview
Enterprise
GIS
Web GIS Layers
GIS Data & Imagery
New Web GIS Layers
GdbFiles WCS ServicesArcGIS Services
Desktop Web Device
Design & Run Models
import and optimize
(optional)
distributed raster analytics cluster
Model Execution Distribution
distributed raster datastore
analysis results as a new Web GIS Layers
ArcGIS Pro
Web Analysis Tools
on Portal MapViewer
Developers &
System Integrators
Model Execution Distribution
Using Your Own Data
• Your own registered data
- Registered data can be used as input but not output
- Models running against single rasters can be parallelized by block (*as long as the model allows it)
- Models running against a collection of rasters will be parallelized per raster in the collection
- Performance can be susceptible to underlying image format (TIFF vs. JP2)
• ArcGIS distributed storage
- Easy to use import tool gets your data into Raster Analytics optimized storage
- CRF (Cloud Raster Format)
- multi-band, block based, multiple readers, multiple writers, fast
- CRF is a format optimized for Raster Analytics computations
• All outputs of Raster Analytics are written in parallel to ArcGIS distributed storage
- Running models on new Web GIS layers is inherently optimized
Raster Analytics is Powerful
Large Collection of Raster Functions
Chain functions together into Raster Models and apply them to
answer complex questions
Math
Abs
Arithmetic
Band
Arithmetic
Calculator
Divide
Exp
Exp10
Exp2
Float
Int
Ln
Log10
Log2
Minus
Mod
Negate
Plus
Power
Round Down
Round Up
Square
Square Root
Times
Bitwise And
Bitwise Left
Shift
Bitwise Not
Bitwise Or
Bitwise Right
Shift
Bitwise Xor
Boolean And
BooleanNot
Boolean Or
Boolean Xor
Equal To
Greater Than
Greater Than
Equal
Is Null
Less Than
Less Than
Equal
Not Equal
ArgStatistics
Cell Statistics
Statistics
ACos
ACosH
ASin
ASinH
ATan
ATan2
ATanH
Cos
CosH
Sin
SinH
Tan
TanH
Data Management & Conversion
Raster to Vector
Vector to Raster
Colormap
Colormap To RGB
Complex
Grayscale
Remap / Reclass
Spectral Conversion
Unit Conversion
Vector Field
LAS to Raster
LAS Dataset to Raster
Clip
Composite
Extract Bands
Mask
Mosaic Rasters
Rasterize Features
Reproject
Interpolation
Interpolate Irregular Data
Nearest Neighbor
IDW
EBK
Swath
Correction
Apparent Reflectance
Geometric Correction
Speckle Filtering (Lee,Frost,Kuan)
Analysis: Image Segmentation & Classification
Segmentation (Mean Shift)
Training (ISO, SVM, ML)
Supervised Classification
Visualization & Appearance
Contrast and Brightness
Convolution
Pansharpening
Resample
Statistics and Histogram
Stretch
Surface Generation & Analysis
Aspect
Curvature
Elevation Void Fill
Hillshade
Shaded Relief
Slope
Viewshed
Analysis: Overlay
Weighted Sum
Weighted Overlay
Analysis: Band Math & Indices
NDVI / NDVI Colorized
SAVI / MSAVI / TSAVI
GEMI
GVI (Landsat TM)
PVI
Tasseled Cap (Kauth-Thomas)
Binary Thresholding
Analysis: Distance & Density
Euclidean Distance
Cost Distance
Least Cost Path
Kernel Density
Analysis: Zonal
Zonal Statistics
Python
Custom Algorithms
Conditionals
Con
Set Null * Does not contain all capability of Spatial Analyst
Raster Analytics is Easy
• Easy to get started, it is “out of the box analytics”
- Install Image Server on nodes -> start Raster Analytic services -> go
- Your own infrastructure, Amazon, Azure
- Scale up & down as required
• Great user experience
- ArcGIS Pro: visual modeler to design simple and complex models
- Portal Map Viewer: Run Raster Analytics Tools
- Python and REST based API for developers
• Results are immediately available in your Web GIS
- No publishing workflow required
- All outputs are written in parallel to ArcGIS distributed storage
Raster Analytics on Different Infrastructure
• Deployed as Enterprise / Web GIS On-premise | Cloud
• Infrastructure can be…
- your hardware
- your Amazon
- your Azure
• Deployment tools
- ArcGIS Enterprise Cloud Builder for Microsoft Azure
- Amazon Cloud Formation Templates
Portal
Raster Analytics Test Case: Terrain Suitability
Global SRTM 90m
0
100
200
300
400
500
600
700
800
1 2 4 8 16
790
425
252
126
80
Min
ute
s
Raster Analytics Processorsesri virtual machine (Image Server)
• 16GB RAM, 8 cores, NAS storage
13.12 hours
80 minutes
terrain suitability model• compute slope
• compute aspect
• remap
• overlay
global terrain suitability raster
Raster Analytics Test Case: Solar Power Plant Suitability
WebGIS (Image Server cluster) on Amazon
• 8 c3.2xlarge instances (8 vCPUs, 16GB RAM)
Mean Rainfall
Mean Temperature
Elevation
Landcover
30m National Solar Plant Suitability Raster
Raster Analytics
9 minutes
ArcGIS Pro
5 hours 45 minutes
suitability model
Raster Analytics Test Case: Landsat Processing
(foreach) input scene
top of atmosphere
correction
modified soil adjusted
vegetation index
remap to classes
mask
no data
output thematic raster
ArcGIS Enterprise GIS on AWS
Distributed Raster Analytics (Image Server) Cluster
• single node
• AWS c3.8xlarge
• 60GB RAM, 32 cores, 500GB SSD
• 200 Raster Analytics Processors
Infrastructure ProcessingInput Collection Output
Landsat GLS 1990
• 7422 Multispectral Scenes
• S3 storage
Thematic Rasters
• 7422 Thematic Rasters
• Distributed Raster Datastore
2 hours 48 minutes44 scenes per minute
¾ scene per second
Raster Analytics: Penn State Watershed Processing
Input data
False color composite
Segmentation
Classification
Brightness/Veg/Chromaticity
output LandCover map
ProcessingInput data Output
Penn State Watershed
• 397 GB
• Distributed datastore
LandCover Map
• Distributed Raster Datastore
100 Billion pixels!1 hour 13 minutes
10 – 20 core Azure instances
ArcGIS Enterprise on Azure
Image Server Cluster
10 Azure instances – 20 cores each
Infrastructure
Thank You to Our SponsorsE
ME
RA
LD
SA
PP
HIR
E
Questions?
Select the session
you attended
Scroll down to
find the survey
Complete Answers
and Select “Submit”
Download the Esri Events
app and find your event
Please Take Our Survey on the Esri Events App!