Slide: 1

Presented at the Natural Disaster Mitigation and Earth Observations: A GEOSS Perspective Workshop in Geneva Switzerland

by Stu Frye (stuart.frye@nasa.gov)

13 January 2014

The Caribbean and Southern Africa Disaster Pilots: Lesson Learned and Challenges

Acknowledgment is to be given to Dan Mandl, Pat Cappelaere, Karen Moe, John Evans, and Fritz Policelli for contributions to this presentation

Overview• National Aeronautics and Space Administration (NASA)

satellite tasking, data processing, and distribution for disasters – Sensor Web technology and science research for full-cycle disaster management

• NASA collaboration with Civilian Space Agencies and National/Regional/International organizations for disaster applications and data provision

• Committee on Earth Observation Satellites (CEOS) Flood Pilot and Group on Earth Observations (GEO) End-To-End Disaster Systems projects

• Experiments in ground validation, crowd sourcing, and hand-held clients

2

Presenter Background• 35 years of satellite systems engineering at the NASA Goddard Space Flight Center

with focus on implementing autonomous operations and detection/classification algorithms for technology research and applied science

• 10 years of experience in providing geolocated satellite data products to users for disaster management using Sensor Web technologies that link Earth Observation satellites, aerial systems, in-situ sensors, and forecast models into an interoperable set of components and services• Working with hundreds of users from national agencies, regional centers, and

international organizations (e.g., UN, World Bank, International Red Cross) • Fulfilling requests for situational awareness coverage for thousands of disaster

events (floods, fires, volcanoes, earthquakes, landslides, oil spills, etc.)• Using radar, optical, thermal infrared, and microwave instruments (Terra,

Aqua, Aura, Landsat, EO-1, Envisat, ALOS, Radarsat, Formosat, SPOT, Worldview, SAC, FASsat, Cosmo-Skymed)

• Project lead for GEO Disaster Systems Task and Architecture Implementation Pilot• Member of the CEOS Disaster Working Group and Lead for the CEOS Flood

Thematic Team to coordinate DRM activities across all civilian satellite agencies • Contributor to the Open Geospatial Consortium (OGC) Web Services Testbeds to

demonstrate interoperability of emerging disaster technologies

3

NASA Disaster Sensor Web ConceptAnalyze Risks

Task Sensor

Acquire Data (Image)

Detect Floods

Analyze Image

Validate Model

Acquire Data

(River Gauge)

Initiate Request

4

NASA-CSA Collaboration Objectives

• To demonstrate the effectiveness of satellite imagery to strengthen regional, national and community level capacity for mitigation, management and coordinated response to natural and man-made hazards

• To identify specific satellite-based products that can be used for disaster mitigation and response on a regional level

• To identify capacity building activities that will increase the ability of the region to integrate satellite-based information into disaster management initiatives

5

Collaboration Approach• Focus on areas where Earth Observations from satellites can have most impact

(flooding, landslides, volcanoes, wildfires, etc.) and on large-scale disasters• Select a small number of regional and national partners to validate usefulness

of Earth Observations• National agencies solicited in 3 regions – Caribbean/Central America,

Southern Africa, and Southeast Asia (lower Mekong/Indonesia/Java) and partners selected based on:• Commitment to make relevant data sets available • Agreement to provide direct support (in-kind)• Assurance of close collaboration between key national, regional, and

international players• Representative cross-section of GIS capability development

• Regional Teams created in all three areas with broad participation from concerned Departments

• Global Component setup for low-spatial-resolution, high-temporal-resolution assets

6

Accomplishments in Pilot Regions

• EO-1, Worldview-2 and Radarsat-2 targeted coverage for hurricanes, earthquakes, algal blooms, wildfires, volcanoes, and landslides

• Daily coverage with MODIS (flood maps) and TRMM (Global Flood Monitoring System) • Real-time Calculations at 1 km• Streamflow and Water Storage (Routed Runoff + Bank Overflow)

• REST-ful tasking interface between the Campaign Manager (geobpms.geobliki.com) and satellites to integrate target requests (from models, other detections, and user inputs) and allocate image orders on a systematic basis

• Field validation of satellite maps using crowd sourcing• Improvement of flood and landslide models, model interfaces, and

access/discovery mechanisms through GEOSS Architecture Implementation Pilot activities

7

NASA Web Service URLs• OpenID Provider-Server https://op.geobliki.com/ controls the

security (this is where you setup your account)• Campaign Manager http://geobpms.geobliki.com/home allows

tasking requests to be submitted (i.e., target requests)• EO-1 Server http://eo1.geobliki.com/ this is where EO-1 data can

be found along with the status of future and past task requests• Radarsat Server http://radarsat.geobliki.com/radarsat where we

provide access to Radarsat-2 browse images, metadata, and processed flood products in several formats

• MODIS Flood Server http://oas.gsfc.nasa.gov/floodmap/ is where you can point your browser to manually check on daily MODIS flood maps (browser GUI-based only)

• MODIS Flood Server API http://modis.geobliki.com/modis is the server that provides an Application Programmer Interface (API) for accessing the daily MODIS maps

8

NASA URLs Concluded• Flood Dashboard Client

http://matsu.opencloudconsortium.org/namibiaflood this is an example of a client implementation that runs on a cloud computing platform provided through a collaboration with the University of Illinois/Chicago and Open Cloud Consortium

• Web Coverage Processing Service (WCPS) http://matsu.opencloudconsortium.org/wcps/session/login is for generating and executing algorithms against satellite data

• Pub/Sub Server http://opsb.geobliki.com/session/new is for setting up subscriptions to be notified when new products of interest to you are published….The notifications come via Email, SMS, or twitter and contain RSS or Atom feeds for you to follow to find the product.  Clients can be automated to monitor the feeds and pull the data they are programmed to look for

9

NASA URLs, server descriptions, and instructions available athttp://eo1.gsfc.nasa.gov/new/sensorWebExp/SensorWebReadMore.html

User Requirements Summary #1• Users are split into three categories:

a) Satellite operators and their value added providers, b) National agencies/ regional centers/ international

organizations, andc) The general public (think weather app on your smart phone)

• Users b) and c):• Want products specific to their disaster event, not data they

have to manipulate• Don’t want to program, they want to execute requests with

point and click apps• Don’t want to purchase software tools, they want open

source tools that are freely available and ubiquitous

10

User Requirements Summary #2• All users want products to be accessible using tools they usually

have installed on their desktop or hand held devices• Digital products are preferred over paper (important to disaster

users we have in our regional pilots)• PNG and JPEG are nice for pictures in reports and presentations,

BUT…• All users want map based products such as

• KML raster overlays and vectorized polygons • These can be combined with other data on a common map

background for mashups/analysis/reporting

11

User Requirements Summary #3

There are hidden services that the users will expect without necessarily knowing they need to ask for it.•Situational awareness requires a stream of new data in addition to static overlays and archive data

• New acquisitions need to be visible to all users so they can know what to request for tomorrow and the next day, available from the variety of feasibilities that are provided

•Automated mapping servers and map service organizations need to have new data availability published and the capability to subscribe for notification of URL/URI of those data sets•Product processing should be automated for every available user option – either for every data set or on-demand for specific detection and classification algorithms

12

User Requirements Summary #4More hidden services that the users will expect without necessarily knowing they need to ask for it.•Tiling and compression should be available for raster and vector-based products that can be ingested and visualized by desktop and hand-held devices without any programming knowledge•Product publication and distribution can be via RSS or Atom-type feeds, but should evolve to become “product feeds”•Product providers need to ensure that

• Native resolution is maintained• Products are terrain corrected and co-registered to map control

points• Products are delivered only within the user area of interest (i.e.,

down scaling global-regional-local)• Delivered products can be easily validated/corrected by non-

experts

13

Radarsat-2 ISAAC aftermath

• Developed processing on radarsat.geobliki.com to show water as red layer in Google Earth (next page)

• Extracted Open Street Map baseline water level as light blue layer (second page)

• Served both as tiled doc.kml layers and as Open Street Map polygons (see third page for kml combined overlay)

NASA Web Service URLs and server descriptions available athttp://eo1.gsfc.nasa.gov/new/sensorWebExp/SensorWebReadMore.html

14

Radarsat-2 Water Detection Example (red)

Open Street Map Reference Water (blue)

Combined Overlays (blue on red)Indicates Flooded Areas

Landsat 8 example in Haiti for construction of reference water maskLC80090472013117LGN01 vis_composite (5-4-3) (EPSG 4326) TIF Compressed 76.4MB

Original Data Set 8 OLI Bands TIF 108.7MB each (7461x7281 pixels 30m resolution)

Using WCPS

18

Surface Water Detection Product

5.9MB TIF LZW Compressed

Using WCPS

Note: Particular Algorithm is Not Relevant In this Example 19

Analysis

• TIF Surface Water Product (5.9MB)– Preserves coordinates/projection– RGBA (4 bands for a bit mask water/no-water)

for visualization

• PNG Product (753KB)– But no map coordinates/projection information

• After ZIP Compression– TIF: 2.2 MB– PNG: 638KB (but not useable for cartography)

20

Vectorization to reduce size and preserve map alignment for overlay

• Autotrace (potrace) and Convert to geojson– >10.6MB

• Convert to topojson– >3.5MB

• Simplify Lines 0.50 (Visvalingam Algorithm)– >2MB

• Compress– >350KB

21

Resulting Product On Mobile Browser

Mapbox.js, MapBox Terrain Layer…

Other Layers Can be Added From OpenStreetMap such as Reference Water…Or Population Density…

Final Achievement:Product Compressed Size From:2.2MB to 350KB

Files: Haiti_RWD_binary_water_0.012.tifSize is 9419, 17304Coordinate System is:GEOGCS["WGS 84", DATUM["WGS_1984", SPHEROID["WGS 84",6378137,298.257223563, AUTHORITY["EPSG","7030"]], AUTHORITY["EPSG","6326"]], PRIMEM["Greenwich",0], UNIT["degree",0.0174532925199433], AUTHORITY["EPSG","4326"]]Origin = (-72.761256,19.38983)Pixel Size = (0.000018,-0.000018)

~2m/pixel resolution

Image Structure Metadata: INTERLEAVE=BANDCorner Coordinates:Upper Left ( -72.7612560, 19.3898340) Lower Left ( -72.7612560, 19.0783620) Upper Right ( -72.5917140, 19.3898340)Lower Right ( -72.5917140, 19.0783620) Center ( -72.6764850, 19.2340980) Band 1 Block=9419x1 Type=Byte, ColorInterp=Gray

Worldview-2 Processed Data (Binary TIF File)Surface Water in White

23

Process for Vectorization of Water Detection Layer

• Generate a binary mask for water detection• Convert to PNG (Black & White) and then to PNM• AutoTrace and convert to GeoJSON• Convert to TopoJSON• Simplify Lines• Remove false detections identified in Height Above Nearest

Drainage (HAND) calculations• Compress Files for Potential Distribution• Map viewer in Plain Firefox Browser Using Open Source

Libraries

24

In addition, produce reference water layer for same footprint and deliver concurrently with water detection layer

Surface Water Detected in White

OpenStreetMap

25

natural=wetlandwetland=swamp

OSM has an area marked as wetland/swampThat could be mis-identified as flooded areain satellite imagery

26

Height Above Nearest Drainage: In Black, Areas to be masked based on Digital Elevation Model

Removed:46,297 out of 23,659,581

https://maps.google.comTerrain Map

28

Resulting Zoomable Flood Map In Browser

Red: Surface Water (after HAND masking)Blue: OSM Reference WaterBackground: Terrain Map From Mapbox

OSM Marshes / Swamp / Watersheds

29OSM Marshes/Watershed Tags Are Important

File Statistics

• Processed Binary File (tif) 163.1MB• GeoJSON

40MB• GeoJSON.zip

7.2MB• TopoJSON

6.4MB• TopoJSON.zip

689KB• Simplified TopoJSON < 1MB• Simplified Compressed 210KB

30

Issues

• HAND processing needs validation• Map Rendering (Browser)– A little slow to manage all the vectors (but does

work)

• Next steps: Vector Tiling (Vector rendering at multiple scales) to speed map rendering for high res imagery [Already implemented in OSM]

31

PostgresDatabase(Joyent)

NRT Satellit

eData

Planet OSMBaseline Water

Database

Flood Map ProcessorRadarsat.Geobliki.

Com (Joyent)

Water extent and Flood

Maps (KML tiles, MB

tiles, OSM tiles)

Baseline Water Layer

National HydrologyAgency Server W/

JOSMNational Postgres Database

AnnotatedHigh water referenceLow water reference

Flood Map

Enhanced Water Referenceand Flood Maps

Mobile PlatformTrack CollectorFor Field Data

Automated Time Stamp

GPS Stamp for shoreline(Lat, Long, Altitude)

User Name/ User Type

Edit polygons:• Drought OR

• Low Normal OR• High Normal OR

• Not flooded: misclassified as flooded• Flooded: misclassification due to deep slope reflection OR• Flooded: misclassification due to dry flat surface OR• Flooded: misclassification due to vegetation cover OR

• Flooded: misclassification due to other Enter Notes

• Input data source (field measurement/ other: describe)• Land Cover Type

• text

Regional HydrologySME Server W/

JOSM

publish corrected map

publish NRT map

OSM Field Validation Approach

32

33

African Flood PilotGround Cal/Val Exercise with Radarsat, EO-1, Ground Team, Helicopter team, OpenStreetMap, Crowd Sourcing on Kavango river in Namibia 1-30-13

EO-1 Water Edge Detection (red)

Radarsat Water Edge Detection (yellow polygon)

Team 1 walking bank to collect GPS point s (red X’s)

Team 2 walking bank to collect GPS point s (green X’s)

One of 500 GPS photos from helicopter

Integrated Water Edge Detection Display with Boat GPS Measurements, GPS located photos, Radarsat/EO-1 water edge detections

Boat Team 1 track walking bank to collect GPS point s (purple track)

Boat Team 2 track walking bank to collect GPS points (orange track with numbered waypoints)

Radarsat Water Edge Detection (yellow polygon)

EO-1 Water Edge Detection

34