Analyzing Multidimensional Scientific Data in ArcGIS

Esri UC 2014 | Technical Workshop |

Analyzing Multidimensional Scientific Data in ArcGISNawajish Noman

Kevin Butler


• ArcGIS and Scientific Data ArcGIS and Scientific Data

• Ingest and aggregationIngest and aggregation

• Visualization and AnalysisVisualization and Analysis

• Service, Ready-to-Use Maps, Web ApplicationsService, Ready-to-Use Maps, Web Applications

• Extending Analytical Capabilities using PythonExtending Analytical Capabilities using Python

• OPeNDAP and Future DirectionOPeNDAP and Future Direction

Outline

Analyzing Multidimensional Scientific Data in ArcGIS


Scientific Data

• Stored in netCDF, GRIB, and HDF formats

• Multidimensional

• Ocean data

Sea temperature, salinity, ocean current

• Weather data

Temperature, humidity, wind

• Land

Soil moisture, NDVI, land cover



Scientific Data in ArcGIS - Vision



• NetCDF data is accessed as• Raster• Feature• Table

• Direct read• Exports GIS data to netCDF

Reading netCDF data in ArcGIS



• Directly reads netCDF file using o Make NetCDF Raster Layero Make NetCDF Feature Layero Make NetCDF Table View

• Directly reads HDF and GRIB data as raster

Ingesting Scientific data in ArcGIS



CClimate and limate and FForecast (orecast (CFCF) Convention) Conventionhttp://cf-pcmdi.llnl.gov/http://cf-pcmdi.llnl.gov/

Initially developed forInitially developed for•Climate and forecast dataClimate and forecast data•Atmosphere, surface and ocean model-generated dataAtmosphere, surface and ocean model-generated data•Also for observational datasetsAlso for observational datasets

•CFCF is now the most widely used conventions for geospatial is now the most widely used conventions for geospatial netCDF data. netCDF data. It has the best coordinate system handling.It has the best coordinate system handling.

•Current version 1.6Current version 1.6

•You can use Compliance checker utility to check a netCDF file. You can use Compliance checker utility to check a netCDF file. http://cf-pcmdi.llnl.gov/conformance/compliance-checker/ http://cf-pcmdi.llnl.gov/conformance/compliance-checker/

CF Convention



• Geographic Coordinate Systems (GCS)Geographic Coordinate Systems (GCS)• X dimension units: X dimension units: degrees_eastdegrees_east• Y dimension units: Y dimension units: degrees_northdegrees_north

• Projected Coordinate Systems (PCS)Projected Coordinate Systems (PCS)• X dimension standard_name: X dimension standard_name: projection_x_coordinateprojection_x_coordinate• Y dimension standard_name: Y dimension standard_name: projection_y_coordinateprojection_y_coordinate• Variable has a Variable has a grid_mappinggrid_mapping attribute. attribute. • CF 1.6 conventions currently supports thirteen predefined coordinate CF 1.6 conventions currently supports thirteen predefined coordinate

systems (Appendix F: Grid Mappings)systems (Appendix F: Grid Mappings)

• Undefined Undefined • If not GCS or PCSIf not GCS or PCS

• ArcGIS writes (and recognizes) PE String as a variable attributeArcGIS writes (and recognizes) PE String as a variable attribute.

NetCDF and Coordinate Systems



Time = 1

141 241 341

131 231 331

121 221 321

111 211 311

441

431

421

411

142 242 342

132 232 332

122 222 322

112 212 312

442

432

422

412

143 243 343

133 233 333

123 223 323

113 213 313

443

433

423

413

Y

X

Time

Changing Time Slice



What about Aggregation?

• Create a seamless multi-dimensional cube fromCreate a seamless multi-dimensional cube fromo files representing different regionsfiles representing different regionso files representing different time steps/slicesfiles representing different time steps/slices



• Supports netCDF, HDF and GRIBo Spatial Aggregationo Temporal Aggregation o On-the-fly analysis

• Accessible as Map Service

• Accessible as Image Service

• Supports direct ingest

• Eliminates data conversion

• Eliminates data processing

• Improves workflow performance

• Integrates with service oriented architecture

Scientific data support in Mosaic Dataset



Multidimensional Mosaic Datasets

• Raster Types for netCDF, HDF & GRIB

• Define variables when adding Rasters

• Each Row is a 2D Raster with variables and dimension values

• Define on-the-fly processing

• Serve as Multidimensionalo Image Serviceo Map Serviceo WMS

Aggregate (mosaic) spatial, time, and vertical dimensions


Esri UC 2014 | Technical Workshop |Esri UC 2014 | Technical Workshop |

Demo



Behaves the same as any layer or table • Display

o Same display tools for raster and feature layers will work on multi-dimensional netCDF raster and netCDF feature layers.

• Graphingo Driven by the table just like any other chart.

• Animationo Multi-dimensional data can be animated through time dimension

• Analysis Toolso Will work just like any other raster layer, feature layer, or table. (e.g.

create buffers around netCDF points, reproject rasters, query tables, etc.)

Using Scientific Data in ArcGIS



Multidimensional Mosaic Dataset - Visualization

• Visualize temporal change of a variable

• Visualize a variable at any vertical dimension

• Visualize flow direction and magnitude variables



• New Vector Field renderer for raster o Supports U-V and Magnitude-directiono Dynamic thinningo On-the-fly vector calculation

• Eliminates raster to feature conversion

• Eliminates data processing

• Improves workflow performance

Visualization of Raster as Vectors



• Several hundreds analytical tools available for raster, features, and table

• Temporal Modelingo Looping and iteration in ModelBuilder and Python

Spatial and Temporal Analysis



Modeling with Raster function template (RFT)



Demo



• Map Service (supports WMS)o Makes maps available to the web.

• Image Service (supports WMS)o Provides access to raster data through a web service.

• Geoprocessing Serviceo Exposes the analytic capability of ArcGIS to the web.

• Map Packageo To share complete map documents and the data referenced by the

layer it contains.

• Geoprocessing Packageo To share your geoprocessing workflow.

Sharing / WMS Support (for multi-dimensions)



Publishing a WMS on ArcGIS Server

• Enable WMS capabilities on Service Editor or Manager



Multi-dimensional data support in WMS

• getCapabilities

o Supports time, elevation and other dimensions (e.g. depth)

• getMap

o Returns map for any dimension value

&DIM_<dimensionName>=<value>&

o Supports CURRENT for time dimension

&TIME=CURRENT&

• getFeatureInfo

o Returns information about feature for any dimension value



Multi-dimensional WMS in ArcMap

• Supports WMS layer like any other layer

• Animates a time enabled WMS layer using time-slider

• Slices for any dimension value are accessible with ArcObjectsPublic Sub UpdateWMSServiceLayerDimensionValue() 'UID for wms service layer type Dim pUid As New uid pUid = "{27ABB9EC-7A26-4cf8-8BD4-70EC1D274E17}"

Dim pWMSMapLayer2 As IWMSMapLayer2 'calling a function to find the layer from active dataframe Set pWMSMapLayer2 = GetLayer(pUid, "myWMSLayer")

'setting values to dimensions Dim pDimNameValues As IPropertySet Set pDimNameValues = New PropertySet pDimNameValues.SetProperty "Depth", "500" 'dimension#1 pDimNameValues.SetProperty "T1", "500" 'dimension#2 Set pWMSMapLayer2.DimensionValues = pDimNameValues 'calling a function to redraw the layer RefreshActiveDataFrameEnd Sub



WMS in Dapple Earth Explorer



Multi-dimensional WMS in a Web Application

http://dtc-sci01.esri.com/MultiDimWMSViewer/

Depth

Time



ArcGIS Online

• Curated, authoritative content provided by Esrio Ready To Useo Highly scalableo Global to National

• Authoritative content provided by the communityo Hosted in your ArcGIS Online Organization accounto Hosted on your hardware and shared to ArcGIS Online

> 100 Tb of data> 150 millions maps per day



Ready-to-Use Maps

http://www.arcgis.com/features/maps/index.html



Ready-To-Use Analysis Services

• Esri hosted analysis on Esri hosted datao Simplify job of GIS Professionalso Can be used in models and scripts

just like any other toolo Extend spatial analysis to a

much broader audienceo Available in Desktop or as REST service

Best practices published to the Resource CenterAnalyzing Multidimensional Scientific Data in ArcGIS


• GLDAS Noah Land Surface Model OutputsGLDAS Noah Land Surface Model Outputs

o EvapotranspirationEvapotranspirationo Soil MoistureSoil Moistureo Snow PackSnow Packo OtherOther

Ready-to-Use Scientific Data Maps



Web Application



Tell the story of your scientific data – Create Story Maps

http://dtc-sci01.esri.com/DeadZoneStoryMap/ Analyzing Multidimensional Scientific Data in ArcGIS


Demo



Supplemental tools

•OPeNDAP to NetCDF

•Make NetCDF Regular Point Layer

•Make NetCDF Station Point Layer

•Make NetCDF Trajectory Point Layer

•Describe Multidimensional Dataset

•Get Variable Statistics

•Get Variable Statistics Over Dimension

•Multidimensional Zonal Statistics

•Multidimensional Zonal Statistics As Table

http://blogs.esri.com/esri/arcgis/2013/05/24/introducing-the-multidimension-supplemental-tools-2/

Python and Geoprocessing Tools



• Python is used to build custom tools for specific tasks or datasets

Application Specific Script Tools



• Geoprocessing Resource CenterGeoprocessing Resource Center

http://resources.arcgis.com/geoprocessing/http://resources.arcgis.com/geoprocessing/

• Marine Geospatial Marine Geospatial Ecology Tools (MGET)Ecology Tools (MGET)•Developed at Duke Univ.Developed at Duke Univ.

•Over 180 tools for importOver 180 tools for importmanagement, and management, and analysis of marine dataanalysis of marine data

http://mgel.env.duke.edu/mgethttp://mgel.env.duke.edu/mget

• Australian Navy toolsAustralian Navy tools (not publicly available)(not publicly available)

Community Developed Tools



• netCDF4-python is included in 10.3/Pro

• Read and write netCDF file

• Conversion time values to date

• Multi-file aggregasion

• Compression

https://www.unidata.ucar.edu/software/netcdf/workshops/2012/netcdf_python/netcdf4python.pdf

netCDF4-Python



Create Space-Time Cube & Emerging Hot Spot Analysis



Creating your own tool



OPeNDAP to NetCDF



• Ingest OPeNDAP ServiceIngest OPeNDAP Service

• Output dynamic multidimensional Output dynamic multidimensional rasterraster

• Support Sub-settingSupport Sub-setting

Next: Make OPeNDAP Layer



• Embrace the Common Data Model (netCDF, HDF etc.)Embrace the Common Data Model (netCDF, HDF etc.)• Use Data and metadata standards (OGC, CF etc)Use Data and metadata standards (OGC, CF etc)

• Produce and use CF complainant data Produce and use CF complainant data

• Make your data “spatial” (by specifying geographic or a projected Make your data “spatial” (by specifying geographic or a projected coordinate system)coordinate system)

• Create sample tools where possibleCreate sample tools where possible

• Clearly define workflow and requirementsClearly define workflow and requirements

Things to Consider…



Demo



• NetCDF, Kevin Sigwart, - Demo Theater – Federal Showcase, Tuesday, 15 Jul 2014, 11:30am – 12:00pm

• Atmospheric, Weather and Climate SIG- Room 24C, Tuesday, 15 Jul 2014, 12:00pm - 1:00pm

• Weather in GIS - See Weather in Esri's Maps & Apps, Sudhir Shrestha & Dan Zimble- Session, Ballroom 20D, Tuesday, 15 Jul 2014, 3:15pm - 4:30pm

• Using Rasters to Measure Impact of Weather on Military Operations, Matt Funk- Demo Theater - Imagery Island Exhibit Hall C, Wednesday, 16 Jul 2014, 11:30am - 12:00pm

• Analyzing Multidimensional Scientific Data in ArcGIS, Nawajish Noman & Kevin Butler- Technical Workshop, Room 17A, Wednesday, 16 Jul 2014, 1:30pm – 2:45pm

• ArcGIS for the Military: Analyzing Environmental Impact on Operations, John Fry & Matt Funk- Session, Omni Ballroom A/B, Wednesday, 16 Jul 2014, 3:15pm - 4:30pm

• Working with Scientific Data Using Mosaic Datasets, Hong Xu- Demo Theater - Imagery Island Exhibit Hall C, Wednesday, 16 Jul 2014, 3:30pm – 4:00pm

• Analyzing Multidimensional Scientific Data in ArcGIS, Nawajish Noman & Kevin Butler- Technical Workshop, Room 17B, Thursday, 17 Jul 2014, 8:30am – 9:45am

• Analyzing Maritime Weather, John Fry & Matt Funk- Demo Theater - Defense and Intel - National Security, Thursday, 17 Jul 2014, 11:30am - 12:00pm

Scientific Data Sessions



Thank you…

• Please fill out the session survey:

First Offering ID: 1309

Second Offering ID: 1414

Online – www.esri.com/ucsessionsurveys

Paper – pick up and put in drop box


Esri UC 2014 | Technical Workshop | Analyzing Multidimensional Scientific Data in ArcGIS

Analyzing Multidimensional Scientific Data in ArcGIS

Documents

Transcript of Analyzing Multidimensional Scientific Data in ArcGIS