Analyzing Multidimensional Scientific Data in ArcGIS

Esri UC 2014 | Technical Workshop |

Analyzing Multidimensional Scientific Data in ArcGISNawajish Noman

Kevin Butler

• ArcGIS and Scientific Data ArcGIS and Scientific Data

• Ingest and aggregationIngest and aggregation

• Visualization and AnalysisVisualization and Analysis

• Service, Ready-to-Use Maps, Web ApplicationsService, Ready-to-Use Maps, Web Applications

• Extending Analytical Capabilities using PythonExtending Analytical Capabilities using Python

• OPeNDAP and Future DirectionOPeNDAP and Future Direction

Outline

Analyzing Multidimensional Scientific Data in ArcGIS

Scientific Data

• Stored in netCDF, GRIB, and HDF formats

• Multidimensional

• Ocean data

Sea temperature, salinity, ocean current

• Weather data

Temperature, humidity, wind

• Land

Soil moisture, NDVI, land cover

Scientific Data in ArcGIS - Vision

• NetCDF data is accessed as• Raster• Feature• Table

• Direct read• Exports GIS data to netCDF

Reading netCDF data in ArcGIS

• Directly reads netCDF file using o Make NetCDF Raster Layero Make NetCDF Feature Layero Make NetCDF Table View

• Directly reads HDF and GRIB data as raster

Ingesting Scientific data in ArcGIS

CClimate and limate and FForecast (orecast (CFCF) Convention) Conventionhttp://cf-pcmdi.llnl.gov/http://cf-pcmdi.llnl.gov/

Initially developed forInitially developed for•Climate and forecast dataClimate and forecast data•Atmosphere, surface and ocean model-generated dataAtmosphere, surface and ocean model-generated data•Also for observational datasetsAlso for observational datasets

•CFCF is now the most widely used conventions for geospatial is now the most widely used conventions for geospatial netCDF data. netCDF data. It has the best coordinate system handling.It has the best coordinate system handling.

•Current version 1.6Current version 1.6

•You can use Compliance checker utility to check a netCDF file. You can use Compliance checker utility to check a netCDF file. http://cf-pcmdi.llnl.gov/conformance/compliance-checker/ http://cf-pcmdi.llnl.gov/conformance/compliance-checker/

CF Convention

• Geographic Coordinate Systems (GCS)Geographic Coordinate Systems (GCS)• X dimension units: X dimension units: degrees_eastdegrees_east• Y dimension units: Y dimension units: degrees_northdegrees_north

• Projected Coordinate Systems (PCS)Projected Coordinate Systems (PCS)• X dimension standard_name: X dimension standard_name: projection_x_coordinateprojection_x_coordinate• Y dimension standard_name: Y dimension standard_name: projection_y_coordinateprojection_y_coordinate• Variable has a Variable has a grid_mappinggrid_mapping attribute. attribute. • CF 1.6 conventions currently supports thirteen predefined coordinate CF 1.6 conventions currently supports thirteen predefined coordinate

systems (Appendix F: Grid Mappings)systems (Appendix F: Grid Mappings)

• Undefined Undefined • If not GCS or PCSIf not GCS or PCS

• ArcGIS writes (and recognizes) PE String as a variable attributeArcGIS writes (and recognizes) PE String as a variable attribute.

NetCDF and Coordinate Systems

Time = 1

141 241 341

131 231 331

121 221 321

111 211 311

142 242 342

132 232 332

122 222 322

112 212 312

143 243 343

133 233 333

123 223 323

113 213 313

Changing Time Slice

What about Aggregation?

• Create a seamless multi-dimensional cube fromCreate a seamless multi-dimensional cube fromo files representing different regionsfiles representing different regionso files representing different time steps/slicesfiles representing different time steps/slices

• Supports netCDF, HDF and GRIBo Spatial Aggregationo Temporal Aggregation o On-the-fly analysis

• Accessible as Map Service

• Accessible as Image Service

• Supports direct ingest

• Eliminates data conversion

• Eliminates data processing

• Improves workflow performance

• Integrates with service oriented architecture

Scientific data support in Mosaic Dataset

Multidimensional Mosaic Datasets

• Raster Types for netCDF, HDF & GRIB

• Define variables when adding Rasters

• Each Row is a 2D Raster with variables and dimension values

• Define on-the-fly processing

• Serve as Multidimensionalo Image Serviceo Map Serviceo WMS

Aggregate (mosaic) spatial, time, and vertical dimensions

Esri UC 2014 | Technical Workshop |Esri UC 2014 | Technical Workshop |

Behaves the same as any layer or table • Display

o Same display tools for raster and feature layers will work on multi-dimensional netCDF raster and netCDF feature layers.

• Graphingo Driven by the table just like any other chart.

• Animationo Multi-dimensional data can be animated through time dimension

• Analysis Toolso Will work just like any other raster layer, feature layer, or table. (e.g.

create buffers around netCDF points, reproject rasters, query tables, etc.)

Using Scientific Data in ArcGIS

Multidimensional Mosaic Dataset - Visualization

• Visualize temporal change of a variable

• Visualize a variable at any vertical dimension

• Visualize flow direction and magnitude variables

• New Vector Field renderer for raster o Supports U-V and Magnitude-directiono Dynamic thinningo On-the-fly vector calculation

• Eliminates raster to feature conversion

• Eliminates data processing

• Improves workflow performance

Visualization of Raster as Vectors

• Several hundreds analytical tools available for raster, features, and table

• Temporal Modelingo Looping and iteration in ModelBuilder and Python

Spatial and Temporal Analysis

Modeling with Raster function template (RFT)

• Map Service (supports WMS)o Makes maps available to the web.

• Image Service (supports WMS)o Provides access to raster data through a web service.

• Geoprocessing Serviceo Exposes the analytic capability of ArcGIS to the web.

• Map Packageo To share complete map documents and the data referenced by the

layer it contains.

• Geoprocessing Packageo To share your geoprocessing workflow.

Sharing / WMS Support (for multi-dimensions)

Publishing a WMS on ArcGIS Server

• Enable WMS capabilities on Service Editor or Manager

Multi-dimensional data support in WMS

• getCapabilities

o Supports time, elevation and other dimensions (e.g. depth)

• getMap

o Returns map for any dimension value

&DIM_<dimensionName>=<value>&

o Supports CURRENT for time dimension

&TIME=CURRENT&

• getFeatureInfo

o Returns information about feature for any dimension value

Multi-dimensional WMS in ArcMap

• Supports WMS layer like any other layer

• Animates a time enabled WMS layer using time-slider

• Slices for any dimension value are accessible with ArcObjectsPublic Sub UpdateWMSServiceLayerDimensionValue() 'UID for wms service layer type Dim pUid As New uid pUid = "{27ABB9EC-7A26-4cf8-8BD4-70EC1D274E17}"

Dim pWMSMapLayer2 As IWMSMapLayer2 'calling a function to find the layer from active dataframe Set pWMSMapLayer2 = GetLayer(pUid, "myWMSLayer")

'setting values to dimensions Dim pDimNameValues As IPropertySet Set pDimNameValues = New PropertySet pDimNameValues.SetProperty "Depth", "500" 'dimension#1 pDimNameValues.SetProperty "T1", "500" 'dimension#2 Set pWMSMapLayer2.DimensionValues = pDimNameValues 'calling a function to redraw the layer RefreshActiveDataFrameEnd Sub

WMS in Dapple Earth Explorer

Multi-dimensional WMS in a Web Application

http://dtc-sci01.esri.com/MultiDimWMSViewer/

ArcGIS Online

• Curated, authoritative content provided by Esrio Ready To Useo Highly scalableo Global to National

• Authoritative content provided by the communityo Hosted in your ArcGIS Online Organization accounto Hosted on your hardware and shared to ArcGIS Online

> 100 Tb of data> 150 millions maps per day

Ready-to-Use Maps

http://www.arcgis.com/features/maps/index.html

Ready-To-Use Analysis Services

• Esri hosted analysis on Esri hosted datao Simplify job of GIS Professionalso Can be used in models and scripts

just like any other toolo Extend spatial analysis to a

much broader audienceo Available in Desktop or as REST service

Best practices published to the Resource CenterAnalyzing Multidimensional Scientific Data in ArcGIS

• GLDAS Noah Land Surface Model OutputsGLDAS Noah Land Surface Model Outputs

o EvapotranspirationEvapotranspirationo Soil MoistureSoil Moistureo Snow PackSnow Packo OtherOther

Ready-to-Use Scientific Data Maps

Web Application

Tell the story of your scientific data – Create Story Maps

http://dtc-sci01.esri.com/DeadZoneStoryMap/ Analyzing Multidimensional Scientific Data in ArcGIS

Supplemental tools

•OPeNDAP to NetCDF

•Make NetCDF Regular Point Layer

•Make NetCDF Station Point Layer

•Make NetCDF Trajectory Point Layer

•Describe Multidimensional Dataset

•Get Variable Statistics

•Get Variable Statistics Over Dimension

•Multidimensional Zonal Statistics

•Multidimensional Zonal Statistics As Table

http://blogs.esri.com/esri/arcgis/2013/05/24/introducing-the-multidimension-supplemental-tools-2/

Python and Geoprocessing Tools

• Python is used to build custom tools for specific tasks or datasets

Application Specific Script Tools

• Geoprocessing Resource CenterGeoprocessing Resource Center

http://resources.arcgis.com/geoprocessing/http://resources.arcgis.com/geoprocessing/

• Marine Geospatial Marine Geospatial Ecology Tools (MGET)Ecology Tools (MGET)•Developed at Duke Univ.Developed at Duke Univ.

•Over 180 tools for importOver 180 tools for importmanagement, and management, and analysis of marine dataanalysis of marine data

http://mgel.env.duke.edu/mgethttp://mgel.env.duke.edu/mget

• Australian Navy toolsAustralian Navy tools (not publicly available)(not publicly available)

Community Developed Tools

• netCDF4-python is included in 10.3/Pro

• Read and write netCDF file

• Conversion time values to date

• Multi-file aggregasion

• Compression

https://www.unidata.ucar.edu/software/netcdf/workshops/2012/netcdf_python/netcdf4python.pdf

netCDF4-Python

Create Space-Time Cube & Emerging Hot Spot Analysis

Creating your own tool

OPeNDAP to NetCDF

• Ingest OPeNDAP ServiceIngest OPeNDAP Service

• Output dynamic multidimensional Output dynamic multidimensional rasterraster

• Support Sub-settingSupport Sub-setting

Next: Make OPeNDAP Layer

• Embrace the Common Data Model (netCDF, HDF etc.)Embrace the Common Data Model (netCDF, HDF etc.)• Use Data and metadata standards (OGC, CF etc)Use Data and metadata standards (OGC, CF etc)

• Produce and use CF complainant data Produce and use CF complainant data

• Make your data “spatial” (by specifying geographic or a projected Make your data “spatial” (by specifying geographic or a projected coordinate system)coordinate system)

• Create sample tools where possibleCreate sample tools where possible

• Clearly define workflow and requirementsClearly define workflow and requirements

Things to Consider…

• NetCDF, Kevin Sigwart, - Demo Theater – Federal Showcase, Tuesday, 15 Jul 2014, 11:30am – 12:00pm

• Atmospheric, Weather and Climate SIG- Room 24C, Tuesday, 15 Jul 2014, 12:00pm - 1:00pm

• Weather in GIS - See Weather in Esri's Maps & Apps, Sudhir Shrestha & Dan Zimble- Session, Ballroom 20D, Tuesday, 15 Jul 2014, 3:15pm - 4:30pm

• Using Rasters to Measure Impact of Weather on Military Operations, Matt Funk- Demo Theater - Imagery Island Exhibit Hall C, Wednesday, 16 Jul 2014, 11:30am - 12:00pm

• Analyzing Multidimensional Scientific Data in ArcGIS, Nawajish Noman & Kevin Butler- Technical Workshop, Room 17A, Wednesday, 16 Jul 2014, 1:30pm – 2:45pm

• ArcGIS for the Military: Analyzing Environmental Impact on Operations, John Fry & Matt Funk- Session, Omni Ballroom A/B, Wednesday, 16 Jul 2014, 3:15pm - 4:30pm

• Working with Scientific Data Using Mosaic Datasets, Hong Xu- Demo Theater - Imagery Island Exhibit Hall C, Wednesday, 16 Jul 2014, 3:30pm – 4:00pm

• Analyzing Multidimensional Scientific Data in ArcGIS, Nawajish Noman & Kevin Butler- Technical Workshop, Room 17B, Thursday, 17 Jul 2014, 8:30am – 9:45am

• Analyzing Maritime Weather, John Fry & Matt Funk- Demo Theater - Defense and Intel - National Security, Thursday, 17 Jul 2014, 11:30am - 12:00pm

Scientific Data Sessions

Thank you…

• Please fill out the session survey:

First Offering ID: 1309

Second Offering ID: 1414

Online – www.esri.com/ucsessionsurveys

Paper – pick up and put in drop box

Esri UC 2014 | Technical Workshop | Analyzing Multidimensional Scientific Data in ArcGIS

Analyzing Multidimensional Scientific Data in ArcGIS

Documents

Transcript of Analyzing Multidimensional Scientific Data in ArcGIS

Analyzing Multidimensional Scientific Data in ArcGISstores processing & metadata. ... Content and Services Server multivariate multidimensional mosaic dataset Disseminating professional

The multidimensional Itô Integral and the multidimensional ...

Analyzing Multidimensional Scientific Data in ArcGIS · Behaves the same as any layer or table • Display-Same display tools for raster and feature layers will work on multi-dimensional

ArcGIS 10.6.1 and ArcGIS Pro 2.2 ... - ArcGIS Desktop

Extensible Visual Business Intelligence for Analyzing XBRL Big Data … · Processing OLAP and multidimensional data analysis, data mining, statistical analysis and forecasting modules.

7 ArcGIS Enterprise - GISBALTIC.eu · Virtualizācija, SaaS, IaaS, sava infrastruktūra = + ArcGIS Enterprise Portal for ArcGIS ArcGIS Server ArcGIS Data Store + ArcGIS Web Adaptor

Collecting and Analyzing Multidimensional Data with Local ...

NetCDF and HDF Data in ArcGIS - s3.amazonaws.com€¦ · NetCDF and HDF Data in ArcGIS . ... Demo: Reading netCDF and HDF data . ... Multidimensional Zonal Statistics As Table .

Esri UC 2014 | Technical Workshop | Analyzing Multidimensional Scientific Data in ArcGIS Nawajish Noman Kevin Butler.

Analyzing GeoSpatial data with IBM Cloud Data Services & Esri ArcGIS

Q1 - GISTEC€¦ · ArcGIS Desktop Designed to help you master critical GIS skills and be productive right away with the ArcGIS tools for visualizing, analyzing and managing geographic

Analyzing Multidimensional Scientific Data in ArcGIS · Raster Mosaic Dataset for N-dimensional Data (ArcGIS 10.3) Resolves Traditional Raster Data Management Issues Processing Time

Isospectral Transformations: A New Approach to Analyzing Multidimensional Systems and Networks

ANALYZING THE PERCEPTIONS OF TURKISH UNIVERSITIES … · ANALYZING THE PERCEPTIONS OF TURKISH UNIVERSITIES USING MULTIDIMENSIONAL SCALING ... of the ten most preferred Turkish universities

Why Multidimensional Poverty? - Ministerio de Desarrollo ... · Why Multidimensional Poverty? ... UNDP’s Multidimensional Poverty Index ... Natural generalization of FGT to multidimensional

[Arcgis] Riset ArcGIS JS & Flex

SAVI COURSE CALENDAR - SPRING 2019 - Pratt Institute · Advanced GIS Intro ArcGIS Intro ArcGIS Intro ArcGIS Intro ArcGIS Intro ArcGIS Intro ArcGIS Intro ArcGIS Intro ArcGIS Intro

ArcGIS for the Military: Analyzing the Operational Environmentproceedings.esri.com/library/userconf/feduc14/papers/fed_43.pdf · ArcGIS for the Military: Analyzing the Operational

Migrating standalone ArcGIS Server to ArcGIS Enterprise · PDF fileMigrating standalone ArcGIS Server to ArcGIS Enterprise ... Migrating standalone ArcGIS Server to ArcGIS ... This

ArcGIS Enterprise: An Introduction · ArcGIS has market leading capabilities for visualizing, analyzing, and editing vector and tabular data, performing fast and efficient routing,