Latest Developments in Oceanographic Applications of GIS

including !

Dawn Wright Esri Chief Scientist

Affiliate Professor, Oregon State U.

Near-Real-Time Interactive Map Exemplars & Scientific Empowerment Through Storytelling

Schmidt Ocean Institute Research Planning Workshop, August 26, 2015

Analytics

Dashboards

Cool Maps Storytelling Apps

Analytics!.

Apps

Access

Desktop Web Device

Services

GeoEvent Extension

GeoAnalytics Extension

A GeoAnalytics

A

ArcGIS Server

Ingestion

Visualization

A Analytics

ArcGIS Spatiotemporal Big Data Store

Services

ArcGIS Spatiotemporal

Storage

High velocity and volume spatiotemporal data

Stream Layer

Stream Service

Web & Real-Time GIS

•! Integration, analysis of high-velocity & -volume data

•! Updates, alerts sent out

Types of (Geo)Analytics

Aggregate by Polygons or Cells

Thanks to Adam Mollenkopf, Esri Real-Time GIS Capability Team Lead and Ricardo Trujillo, GeoEvent Developer, amollenkopf@esri.com, rtrujillo@esri.com links.esri.com/geoevent-wiki We aim to register large data stores / data sets with ArcGIS Server, then distribute analysis across a cluster of machines for parallel processing Performance example: buffer 8.2 million points or thousands of polygons in a little over a minute, Coming: ~250,000 writes to disk per second across 5 nodes Many frameworks/technologies exist for distributing computation

E.g., Hadoop, MapReduce, Spark Spark: processes distributed data in memory; Supports MapReduce programming model

Includes additional framework level distributed algorithms ArcGIS Server integrates these technologies on a cluster to solve analytic problems

5

Parallelized  batch  analy0cs  on  tabular,  vector,  raster,  and  imagery  datasets  (big  and  standard  data)  Performance example: buffer 8.2 million points or thousands of polygons in a little over a minute, NetCDF output is supported on … Aggregate Points – Point in polygon aggregation (spatial join) Aggregate by Cell – simple aggregation Summarize Nearby – spatial join where left dataset and a distance are used to inflate the bounding box. A distance calculation is then employed (no drivetime/distance). Summarize Within – a spatial join that is a generalization of Aggregate Points Find Existing Locations – an attribute query followed by/with a spatial query (?); the attribute portion is SparkSQL, the spatial relationship is: intersections, within distance, contains, and within. Find Similar Locations – non-spatial similarity search (SparkSQL). Calculate Density – cell-based spatial aggregation by attribute with a weighted neighborhood/distance-based summarization. Find Hot Spots – very similar to Calculate Density but with the Getis-Ord Gi* statistic. This can be cell-based (point observations) or polygon-based (polygon source data). Create Buffers – simple spatial buffer generation across the dataset (the Java geometry library supports Euclidean and Geodetic buffer generation). This is done. Extract Data – a simple region query with a specified output format. Field (RASTER) Calculator – apply a Scala or SQL expression to each feature, one expression per field.

6  

http://blogs.esri.com/esri/apl/2015/03/11/dimension-explorer/

Dimension Explorer Add-In (Desktop)

Simple Integration

Out of the Box

Interoperability / Crosswalking

r-arcgis.github.io

Interoperability / Crosswalking

Apps

Access

Desktop Web Device

Services

GeoEvent Extension

GeoAnalytics Extension

A GeoAnalytics

A

ArcGIS Server

Ingestion

Visualization

A Analytics

ArcGIS Spatiotemporal Big Data Store

Services

ArcGIS Spatiotemporal

Storage

High velocity and volume spatiotemporal data

Stream Layer

Stream Service

Web & Real-Time GIS

Dashboards!

Shin Kobara, Texas A&M

Wave Glider Sensor Dashboard: bit.ly/gcooswaveglider

From Kobara et al., 2015, Ocean Solutions, Earth Solutions

Shin Kobara, Texas A&M

Esri Global Fishing Watch aimed at Illegal, Unreported, Unregulated Fishing

Active Vessels Dashboard Prototype

Cool Maps!.

coolmaps.esri.com/BigData/ShippingTracks

coolmaps.esri.com/BigData/ShippingTracks

coolmaps.esri.com/BigData/ShippingGlobe

coolmaps.esri.com/BigData/ShippingGlobe

coolmaps.esri.com/BigData/ShippingGlobe

arcg.is/1Pg7Jlg

Once upon a time!

“Stories are data with a soul, !persuasive and memorable.” – Nancy Baron, Director of Science Outreach, COMPASS

By Charles Saxon | Published March 21, 1983 Licensed personally to Dawn Wright by The New Yorker Cartoon Bank, TCB-86966, Invoice L12480

“Story Maps” Communicating with !

esriurl.com/ocnstories

esriurl.com/ocnstories esriurl.com/ocnstories

Sensor dashboard as a story map

esriurl.com/ocnstories

esriurl.com/ocnstories

storymaps.arcgis.com

esriurl.com/ocnstories

Natural Resources Defense Council, esriurl.com/ocnstories

Natural Resources Defense Council, esriurl.com/ocnstories

Natural Resources Defense Council, esriurl.com/ocnstories

esriurl.com/scicomm, esriurl.com/ocnstories, esriurl.com/ocnres

Final Chapter

Consider implications for education, training,

certification?

Consider potential of stories and scientific storytelling via maps, photos, videos, more

Consider implications for ocean science

contributions to society

Dawn Wright dwright@esri.com

@deepseadawn

Extra Slides

Story map made into a conference poster with interactive version on table for discussion &questions, e.g., esriurl.com/agustories

esriurl.com/analyticalstories

How do I get real-time data into a GIS? ArcGIS example: Use an Input Connector

You can create your own

connectors.

GeoEvent Extension

Inpu

ts

Out

puts

GeoEvent Services

Poll an ArcGIS Server for Features

Poll an external website for GeoJSON, JSON, or XML

Features, GeoJSON, JSON, or XML on a REST endpoint

Receive RSS

Receive GeoJSON or JSON on a WebSocket

Receive Text from a TCP or UDP Socket

Subscribe to external WebSocket for GeoJSON or JSON

Watch a Folder for new CSV, GeoJSON, or JSON Files

Esri

Out

of t

he B

ox

REST

Watch a Folder for new CSV, .csv

Poll an ArcGIS Server for Features

WS

WS

HTTP

Receive RSS

Receive Text from a TCP or UDP Socket Receive Text from a TCP or UDP Socket

E

sri G

alle

ry

ActiveMQ

CAP

Gal

lery

Instagram Gal

lery

G

alle

ry

Flume

Gal

lery

G

alle

ry

Cursor-on-Target

RabbitMQ

NMEA 0183

MQTT

Sierra Wireless (RAP)

KML

Kafka

Cursor-on-Target

Instagram

*

Trimble (TAIP)

Kafka

Twitter Twitter

MQTT P

artn

er G

alle

ry

CompassLDE

enviroCar

GNIP

FAA (ASDI)

exactEarth AIS

Zonar

Valarm

Networkfleet

OSIsoft

GNIP

FAA

* * *

*

*

*

Flume *

Applying Real-Time Analytics ArcGIS example: Use a GeoEvent Processor

You can create your own

processors.

GeoEvent Extension

Inpu

ts

Out

puts

GeoEvent Services

Aggregator

Buffer Creator

Convex Hull Creator

Difference Creator

Envelope Creator

Field Calculator

Field Enricher

Geotagger

Incident Detector

Intersector

Projector

Simplifier

Sliding Time Window

Symmetric Difference

Field Mapper Track Gap Detector Esri

Out

of t

he B

ox Add XYZ

Esri

Gal

lery

Bearing

Ellipse

Event Volume Control

Extent Enricher

Field Grouper

GeoNames Lookup

Range Fan

Reverse Geocoder

Service Area Creator

Symbol Lookup

Track Idle Detector

Unit Converter

Visibility

Motion Calculator Query Report Field Reducer Union Creator

*

*

How do I update and alert those who need it where they need it? Disseminate notifications, alerts, and updates using an Output Connector

You can create your own

connectors.

GeoEvent Extension

Inpu

ts

Out

puts

GeoEvent Services

Esr

i Out

of t

he B

ox

Add or Update a feature

Publish Text to a UDP Socket

Send a Text Message

Send an Email

Add or Update a feature

Push Text to an external TCP Socket

Push GeoJSON or JSON to an external WebSocket

Push GeoJSON or JSON to an external Website

Send an Email

Send a Text Message

Send an Instant Message

Send Features to a Stream Service

Push Text to an external TCP Socket Push Text to an external TCP Socket

Publish Text to a UDP Socket Publish Text to a UDP Socket

Write to a CSV, GeoJSON, or JSON File Write to a CSV, .csv

WS

im

HTTP

Write to a Spatiotemporal Data Store Write to a Spatiotemporal Data Store

ActiveMQ

Esri

Gal

lery

Cursor-on-Target

Flume

Hadoop (HDFS)

Kafka

MongoDB

*

MQTT

Cursor-on-Target

RabbitMQ

Twitter

MongoDB

Kafka

Twitter

MQTT

*

Flume

*

Two patterns Getting Real-Time Data into Web Applications

•! Feature layers pull from feature services -! Web apps poll to get periodic updates -! Must be backed by an enterprise geodatabase (EGDB)

•! Stream layers subscribe to stream services -! Web apps subscribe to immediately receive data -! Low latency and high throughput

GeoEvent Extension

ArcGIS Server

Your

Applications

Stream Layer

Map Services Feature Services

Feature Layer

feature layers

Update a Feature Add a Feature

EGDB

Polling (Pull)

Stream Services

Send Features to a Stream Service