SU

RA

IT M

eetin

g –

22 M

arch

200

5

SURA IT Committee MeetingMarch 22, 2005

Sara J. Graves, Ph.D.

Director, Information Technology and Systems Center

University Professor, Computer Science Department

University of Alabama in Huntsville

Director, Information Technology Research Center

National Space Science and Technology Center

256-824-6064

sgraves@itsc.uah.edu

SCOOP Status

SU

RA

IT M

eetin

g –

22 M

arch

200

5

Whereas, the Southeastern Universities Research Association has proposed the creation of an open-access network of distributed sensors, linked via an ultra-fast network to state-of-the-art computing systems that track and model the southeastern coastal zone in real time and provide components of a more comprehensive coastal security infrastructure — known as Southeastern Coastal Ocean Observing Program (SCOOP); now, therefore, be it

Resolved, That the Southern Governors’ Association supports SURA’s Southeastern Coastal Ocean Observing Program to bring more effective protection of lifeand property to the increasingly developed coastal zone, to offer a vehicle for bringingthe extensive and widely dispersed intellectual talent of the ocean sciences community toaddress program of homeland security via an integrated and spatially distributed program,and to aid in addressing the ecological and environmental concerns endangering healthand safety of inhabitants and marine resources.

SU

RA

IT M

eetin

g –

22 M

arch

200

5

SURA’s Southeastern Coastal Ocean Observing Program (SCOOP) will facilitate the assimilation of observational data into community models and provide a distributed data ingestion and support grid with broad band connectivity. This is expected to become a coastal counterpart to the Global Ocean Data Assimilation Experiment (GODAE) with emphasis on the southeast.

SU

RA

IT M

eetin

g –

22 M

arch

200

5Board of Trustees Meeting

Nov 2002

• Data fusion is critical• Modeled and observed fields must have equal

representation• Use GODAE (Global Data Assimilation Experiment)

as a guide for CODAE (Coastal Ocean Data Assimilation Experiment)

• SURA is a strong brand (we should use it)• Focused sub-regional efforts with specified

deliverables which would be new and exciting• Broad SURA effort targeted on building a culture

supporting region-wide collaboration in shared scientific goals

SU

RA

IT M

eetin

g –

22 M

arch

200

5

Integrated Ocean Observing System (IOOS)

Serves national needs for:• Detecting and forecasting oceanic components of

climate variability • Facilitating safe and efficient marine operations • Ensuring national security • Managing resources for sustainable use • Preserving and restoring healthy marine ecosystems • Mitigating natural hazards • Ensuring public health

SU

RA

IT M

eetin

g –

22 M

arch

200

5National Federation of Regional

Systems National Backbone• Satellite remote sensing• In situ sensing reference & sentinel station-network• Link to global ocean component• Data standards & exchange protocols

Regional Systems• Regional priorities• Effects of climate change & Effects of land-based sources Resolution, Variables

SU

RA

IT M

eetin

g –

22 M

arch

200

5

1. A national coastal observing program will necessarily consist of regional and sub-regional components.

2. National, regional and sub-regional observing

systems must consist of three interconnected aspects: (i) spatially distributed sensor arrays; (ii) data management and dissemination hubs; and (iii) nowcasting and forecasting models that are fused with assimilated observational data.

3. The creation and long-term viability of nested integrated and sustained coastal observing systems will depend on a high level of interagency coordination.

Overarching Principles for Coastal Observing Programs

SU

RA

IT M

eetin

g –

22 M

arch

200

5

The SURA Coastal Ocean Observing and Prediction (SCOOP) program is an initiative to create a open-access, distributed national laboratory for scientific research and coastal operations. SCOOP is designed to complement the efforts of both Ocean.US - the organization responsible for implementing the national Integrated Ocean Observing System (IOOS)- and the coastal component of NSF’s Ocean Research Interactive Observatory Networks (ORION) project. The SCOOP emphasis is on interoperability in order to create a real-time observations system for both monitoring and prediction. Through SURA Universities, SCOOP will provide the expertise and IT infrastructure to integrate observing systems that currently exist, and incorporate emerging systems. This will promote the effective and rapid fusion of observed data with numerical models, and facilitate the rapid dissemination of information to operational, scientific, and public and private users.

SCOOP Vision Statement

SU

RA

IT M

eetin

g –

22 M

arch

200

5

1. System of Systems• Ocean Observing = IOOS & OOI & ORION • Coastal Ocean Component of the Global Earth Observing

System of Systems (GEOSS)• Components: (i) Sensor arrays, (ii) Data management &

communication, (iii) Predictive models 2. Distributed National Lab for Research & Applications

• IT Glue...Bricks & Mortar• Research to Operations• Academic & Federal Agency & Industry partnership

3. IT Enabling Big Science• Environmental prediction• Standards enable innovation• Interoperable community solving the really big problems

Overarching Goals for SCOOP

SU

RA

IT M

eetin

g –

22 M

arch

200

5

Planned Capabilities

• Validate accurate and timely short and long-term predictions

• Simultaneous measurements of winds, waves, currents, water density, nutrients, water quality, biological indices, and fish stocks under all conditions

• Focus on storm surge, wind waves, and surface currents, with special attention to predicting and visualizing phenomena that cause damage and inundation of coastal regions during severe storms, hurricanes and possibly tsunamis

• Bridge the gap between scientific research and coastal operations

SU

RA

IT M

eetin

g –

22 M

arch

200

5

SCOOP Science Goals

• Assess and predict the coastal response to extreme atmospheric events – focus on storm surge, flooding & waves

• Modular modeling tools for regional issues (wave coupling, sediment suspension, etc.)

• Standardized interfaces for data and (coupled) model interoperability

• Ensemble prediction – forecasts based on many independent models runs

SU

RA

IT M

eetin

g –

22 M

arch

200

5

SCOOP Research Goals

• Measure, understand and predict environmental conditions

• Provide R&D support for operational agencies including NOAA, the U.S. Navy, and others

• Include outreach and education components that assure relevance of their observing activities

SU

RA

IT M

eetin

g –

22 M

arch

200

5

SCOOP Objectives

• Develop and deploy standards and protocols for data management, exchange, translation and transport

• Implementation of existing standards and protocols (e.g. FGDC, OGC, web services, etc.)

• Application of Grid Technologies• Deployment of the communications infrastructure to

link ocean sensors operating in extreme environmental conditions to people who need timely information

• Cultivation of industry partners

SU

RA

IT M

eetin

g –

22 M

arch

200

5

Coordination is Key

• Ocean.US - National Office for Integrated and Sustained Ocean Observations coordinates development of an operational, integrated and sustained Ocean Observing System (created by NOPP) http://www.ocean.us/

• Integrated Ocean Observation System (IOOS) a national effort to create an Integrated Ocean Observing System http://www.openioos.org/

• National Oceanographic Partnership Program (NOPP) 15 federal agencies providing leadership and coordination of national research and education programs http://www.nopp.org/

• National Federation of Regional Associations provide a framework for orchestrating regional collaborations http://www.usnfra.org/

• NSF Ocean Research Interactive Observatory Networks (ORION) an emerging network of science-driven ocean observing systems http://www.orionprogram.org/default.html

SU

RA

IT M

eetin

g –

22 M

arch

200

5

Interoperability is Key

• Ocean.US Data Management and Communications (DMAC) Plan provides the framework for interoperability http://dmac.ocean.us/dacsc/imp_plan.jsp

• Open Geographic Information Systems (GIS) Consortium (OGC) an open consortium of industry, government, and academia developing interface specifications to support interoperability http://www.opengis.org

• Marine Metadata Interoperability a community effort to make marine metadata easier to find, access and use http://www.marinemetadata.org/

SU

RA

IT M

eetin

g –

22 M

arch

200

5

Interoperability Demonstration

www.openioos.orgwww.openioos.orgwww.openioos.orgwww.openioos.org

NOAA and ONR grant recipients collaborationNOAA and ONR grant recipients collaborationNOAA and ONR grant recipients collaborationNOAA and ONR grant recipients collaboration

SU

RA

IT M

eetin

g –

22 M

arch

200

5

• Funding provided by ONR, NOAA• 2004 List of SCOOP Partners:

SCOOP System Development

• Consortium for Oceanographic Research and Education• Gulf of Maine Ocean Observation System (GoMOOS) • Louisiana State University, Center for Computation &Technology• Louisiana State University, Coastal Studies Institute• Southeast Atlantic Coastal Ocean Observing System (SEACOOS)• Southeast Coastal Ocean Observations Regional Association (SECOORA)• Texas A&M University & Gulf Coast Ocean Observing System (GCOOS)• University of Alabama in Huntsville• University of Delaware (Mid-Atlantic Regional Association (MACOORA)• University of Florida • University of Miami, Center for Southeastern Tropical Advanced Remote Sensing• University of North Carolina• Virginia Institute of Marine Science

SU

RA

IT M

eetin

g –

22 M

arch

200

5

SCOOP Program Elements

1. Data StandardsMetadata standards – compliant with existing and

emerging standards

Standard data models – to facilitate aggregation

2. Data GridOGC Web services – for distributed maps and data

Augmenting with new data, e.g., surface currents

3. Model GridStorm surge & wave prediction

Modular, standardized prediction system

SU

RA

IT M

eetin

g –

22 M

arch

200

5

SCOOP Data Architecture (high level)

NOAA

Web Browsers

GIS Clients

NDBC

Regional Association Data Center (Archive)

OGC Protocols

Other Regional

Association Data Centers

HTTP & HTML

LDM…???

NDBC MODEM

HTTP & HTML

SCOOP Modeling Partners

TBD…???Regional Data

Provider #1

Regional Data

Provider #2

Regional Data

Provider #N

SCOOP Modeling Partners

TBD…???

Transport Mediums

SU

RA

IT M

eetin

g –

22 M

arch

200

5SCOOP Prediction System

All Versions

1. Standard naming conventions – Adopt existing community standards where appropriate (e.g., CF or NCEP) and add our own conventions only when necessary.

2. Mechanisms for tracking metadata, e.g., provenance, forcing, source of OBCs, forcing used to create OBCs, etc.

3. Portals – entry point for access to models & model output. Deals with authentication & authorization.

SU

RA

IT M

eetin

g –

22 M

arch

200

5

SCOOP Prediction System, Version 1.0

• Modular wind forcing • Modular embedded regional models• Coupled models – for existing groups• Using existing computational resources• Verification – real time model-data comparisons• Model-GIS interface & OGC Web services• Web mapping with roads, etc.• Web mapping with time sequences (WMS)• Standardized time-series verification• Openioos.org for displaying results• Other…?

SU

RA

IT M

eetin

g –

22 M

arch

200

5

SCOOP Operational Prediction System Version 1.0

Regional Model Center #1

Operational Wave Predictions

(BIO/GoMOOS)

Operational Tide/Surge Predictions (SABLAM)

Coupled Wave-Surge Predictions

(Miami)

Large Scale Response

NOAA/NCEP

(ETA)

NOAA/NCEP/UNC?

(EDAS) Archive

Enhanced Winds

(Miami)

Forcing

Regional Response

Standardize model interfaces

Regional Model Center #2

Standardize Transport/Encapsulation XML, FTP, LDM,

OPeNDAP…?

SU

RA

IT M

eetin

g –

22 M

arch

200

5

SCOOP Verification & Visualization

Prediction System

Standardize model-GIS interfaces

Regional Web Server #1

Modeling Center #1 (Regional or otherwise)

Modeling Center #2 (ditto)

Regional Data Center #1

Information Providers

Regional Web Server #2

OGC, RSS…?

Regional Data Center #2

Standardize verification tools & data

Data System

SU

RA

IT M

eetin

g –

22 M

arch

200

5

Prediction System Task Elements for Version 1.0

Task: Lead Partner:Data standards TAMUData transport UAHData translation & mgmt UAHCoupled modeling MiamiNested Modeling VIMSCustomized configuration TAMUVisualization Services LSUVerification & validation MiamiComputing & storage resources LSUSecurity TAMUGrid management middleware LSUWeb Mapping Demonstration GoMOOS

SU

RA

IT M

eetin

g –

22 M

arch

200

5

Users, Modeling Partners, other Data Centers, etc.Regional Association Data CentersRegional Data Providers

Data Provider

Data Translation

Services

Data Provider

SCOOP Data Architecture:High level Services

Metadataonly

dataData andMetadata

data

data

Modeler /Data Provider

ObservationData

Model

Data Access Services

Metadata Services

SCOOPCatalog

data

Archive/Repository Broker

Data Access Services

Model orApplication

User InterfaceGeoSpatial OneStop/ FGDC

Clearinghouse

SU

RA

IT M

eetin

g –

22 M

arch

200

5

Regional Association Data CentersRegional Data Providers

Data Provider

Data Provider

SCOOP Data Architecture Specifics:Data Acquisition – example technologies to support dynamic transport and metadata cataloging

Metadataonly

dataMetadataand Data

data

data

Modeler /Data Provider

ObservationData

Model

Data Access Services

Metadata Services

data

Archive/Repository Broker

Data Access Services

Data TransportMetadata Cataloging

LDM

e.g., LDM

e.g., XML, Metadata Harvest

SCOOPCatalog

SU

RA

IT M

eetin

g –

22 M

arch

200

5

Users, Modeling Partners, other Data Centers, etc.

Data Translation

Services

data

Modeler /Data Provider

FTP OPeNDAP OGC

Model orApplication

User Interface

GeoSpatial OneStop/ FGDC

Clearinghouse

SCOOP Data Architecture Specifics:Data Discovery & Access – example technologies to support dynamic transport, analysis and

visualization

Metadata Query Services

data

Archive/Repository Broker

FTP OPeNDAP OGC

ESML

IOOS Interoperability

Demo

Z39.50

SOAP

OGC WMS

XML

HTTP

FTP

DataDiscovery

DataAccess

Regional Association Data Centers

RegionalData Providers

SCOOPCatalog

SU

RA

IT M

eetin

g –

22 M

arch

200

5

data

Modeler /Data Provider

User

GeoSpatial OneStop

SCOOP Information Architecture:Example metadata exchange technologies to support Data Discovery

IOOS Interoperability Demo

Z39.50XML

Regional Association Data/Service Centers

RegionalData Providers

Data Provider

data

LocalMetadata

SCOOPData Dictionary

SCOOP Catalog

SOAP

OGC

Get Capabilities

MetadataHarvest

MetadataHarvest

SCOOP Interactive Search U/I Model or

Application

FGDC Records

HTTPWMS data list

& metadata

Automated Data Discovery

SOAP

SOAP

SOAP

XML ?

MetadataHarvest

Metadata Services

DataDiscovery

Metadata Population

Users, Modeling Partners, other Data Centers, etc.

Ingest Svcs Query SvcsManual Updates

SCORE

SU

RA

IT M

eetin

g –

22 M

arch

200

5

SCORE: Accomplishments & Plans

• SCORE is the catalogs and services infrastructure for SCOOP data management

• Data & Model Survey provided initial snapshot of partners’ data (observations and model results)

• Developed database schema for SCORE to support– Strawman SCOOP Catalog: requesting input on improved capabilities – IOOS demo: working with GoMOOS team to integrate catalog with

demo– FGDC Clearinghouse to support Geospatial One-Stop: plan to create

FGDC metadata records from SCORE

• Issues– What data management functionality is needed within SCOOP?

• Metadata services for data collections, data files/streams, general model information, information on specific model runs,…

– How to coordinate metadata and data management across sites?– How to automate population of SCORE?

SU

RA

IT M

eetin

g –

22 M

arch

200

5

1. Environmental Prediction• Prediction systems fuse models & observations • Nonlinear dynamics limits predictability – Lorentz’s seagull• Probability and statistics – ensemble modeling

2. Hurricane Surge & Waves• Biggest uncertainty in the winds• Ensemble of winds: different models or different simulations• New paradigm & new metrics for skill assessment

3. Research to Operations• Improving upon SLOSH – a good idea 30 years ago• GIS compatibility enabling application & visualization• OpenIOOS.org is the high visibility “front end”

Science Goals for Version 2.0

SU

RA

IT M

eetin

g –

22 M

arch

200

5

RegionalArchives

Ensemble wind fields from varied and distributed

sources

ADCirc

ElCirc

WAM/SWAN

Ensemble of models run

across distributed resources

Archive

Verification

Visualization

Analysis, storage and cataloguing of output data

Select region and time range

Transform and

transport data

Wind Forcing

Wave and/or Surge Models

Result Dissemination

Synthetic Wind Ensemble

NCEP

MM5

NCARor

orOpenIOOS

Version 2.0

SU

RA

IT M

eetin

g –

22 M

arch

200

5SCOOP: Data-to-Model (D2M) Realtime

Transport and Translation (Nested Model) Scenario

UNCTAMU

VIMSTAMU, UF,

Others?

ADCIRC

MM5Translation Services• subset• subsample• re-format• re-grid

ELCIRCModel X

LDM, OPeNDAP, FTPPush/pull

POC: Rick Luettich, Brian Blanton

POC: Gerry Creager

POC: Harry Wang

LDM-push

POC: Matt Smith, Ken Keiser (UAH)

LDM-push

Water levels

NCEP (NAM)Wind Forecasts

Atmospheric Models Regional Oceanic/Coastal Models

Localized Models, Users and Archives

High-Res Wind Forecasts LDM-push

D2M Node

TranslatedWater levels

(1)

(2)

(3)

(4)

(1a)WRF(future)

(1) Atmospheric Model products are “translated” through D2M to the form requested by the client model. Currently, using ftp-pull, all NAM grids 0-84h for the 4 runs (00, 06, 12, 18 UTC) of AWIP12 and AWIP32 are sent to a D2M node and translated.

(2) Via LDM, UNC, TAMU, & UF have access to the raw and translated model data.

(3) Partners use translated ob/model data in their models. Then push their results to a D2M node. Currently, ADCIRC output files (text and netCDF) are being pushed to a D2M node (for translation) and other modeling partners via LDM.

(4) Resulting translated data products area pushed to a client model’s site and made available for other transport vehicles (FTP, OPeNDAP, OGC, etc) for use in retrospective studies and other applications. Likewise the output of other models can be processed through D2M for translation steps requested by other client models.

TranslatedWinds and fluxes

Translation Services• subset• subsample• re-format• re-grid

D2M Node

ESML

ESML

LDM-push

LDM-push (FTP-pull)

Alternate

SU

RA

IT M

eetin

g –

22 M

arch

200

5

Data Management Goals

Version 1• Provided a high-level data catalog for SCOOP data discovery, providing

descriptions of partner data holdings and pointers to partner data access points (web, ftp, OPeNDAP, etc.)

– Based initial catalog on Data & Model Survey results• Coordinated with Data Transport (Task 2) to develop initial LDM network to

exchange data in near real time among SCOOP partners.• Coordinated with Data Standards (Task 1) on development of metadata

keywords for SCOOP

Version 2• Expand SCOOP data discovery capabilities based on evolving data

management practices of SCOOP partners. – Support IOOS Demo – Field an FGDC Clearinghouse node for SCOOP

• Monitor Marine Metadata Interoperability activities and their potential interaction with SCOOP

– Assist SCOOP partners in developing standard metadata to describe their data collections

• Continue coordination with all partners on data management issues

SU

RA

IT M

eetin

g –

22 M

arch

200

5

GLOBE AMSU-A KnowledgeBase

ITSC

Coastlines

Countries

MCS Events

Cyclone EventsAMSU-A Channel 01

AMSU-A data overlaid with MCS and Cyclone events, merged with world boundaries from GLOBE.

Merged data product for on-demand visualization

Distributed Data Integration

SU

RA

IT M

eetin

g –

22 M

arch

200

5Heterogeneity Leads to Data Usability

Problems

Science Data Characteristics• Many different

formats, types and structures (18 and counting for atmospheric science alone!)

• Different states of processing (raw, calibrated, derived, modeled or interpreted)

• Enormous volumes

SU

RA

IT M

eetin

g –

22 M

arch

200

5

Interoperability: Accessing Heterogeneous Data

The Problem

DATA FORMAT 1

DATA FORMAT 1

DATA FORMAT 2

DATA FORMAT 2

DATA FORMAT 3

DATA FORMAT 3

READER 1 READER 2

FORMATCONVERTER

APPLICATION

ESML LIBRARY

APPLICATION

DATA FORMAT 1

DATA FORMAT 1

DATA FORMAT 2

DATA FORMAT 2

DATA FORMAT 3

DATA FORMAT 3

The Solution

ESMLFILEESMLFILE

ESMLFILEESMLFILE

ESMLFILEESMLFILE

One approach: Enforce a standard data format, but…

• Difficult to implement and enforce• Can’t anticipate all needs• Some data can’t be modeled or is lost in

translation• Converting legacy data is costlyA better approach: Interchange Technologies• Earth Science Markup Language

SU

RA

IT M

eetin

g –

22 M

arch

200

5

What is ESML?

• It is a specialized markup language for Earth Science metadata based on XML - NOT another data format.

• It is a machine-readable and -interpretable representation of the structure, semantics and content of any data file, regardless of data format

• ESML description files contain external metadata that can be generated by either data producer or data consumer (at collection, data set, and/or granule level)

• ESML provides the benefits of a standard, self-describing data format (like HDF, HDF-EOS, netCDF, geoTIFF, …) without the cost of data conversion

• ESML is the basis for core Interchange Technology that allows data/application interoperability

• ESML complements and extends data catalogs such as FGDC and GCMD by providing the use/access information those directories lack.

http://esml.itsc.uah.edu

SU

RA

IT M

eetin

g –

22 M

arch

200

5ESML IN ACTION:

Ingest surface skin temperature data in Numerical Models

ReanalysisGRIB filesReanalysisGRIB files

MM5MM5 GOESGOES

ESML file

ESMLfile

ESMLfile

http://vortex.nsstc.uah.edu/~sud/web/default.htm

ESML LibraryNUMERICAL WEATHER

MODELS (MM5, ETA, RAMS)

Scientists can:• Select remote files

across the network• Select different

observational data to increase the model prediction accuracy

Purpose:• Use ESML to incorporate

observational data into the numerical models for simulation

• Skin temperatures come in a variety of data formats

• GOES – McIDAS• Reanalysis Data - GRIB • MM5 Model - Binary • AVHRR – HDF• MODIS - EOS-HDF