The IOOS DMAC Metadata Expert Team:

Challenges and Status

MMI Sensor Interoperability Metadata WorkshopOctober 19, 2006

Presentation

• Brief overview of IOOS and DMAC• Overview of metadata expert team• Recommendations and next steps• Challenges

Integrated Ocean Observing System (IOOS)

U.S. Commission on Ocean Policy called for an integrated ocean observing system with 7 societal goals:

• Improve predictions of climate change and weather and their effects on coastal communities and the nation

• Improve the safety and efficiency of maritime operations • More effectively mitigate the effects of natural hazards • Improve national and homeland security • Reduce public health risks • More effectively protect and restore healthy coastal ecosystems • Enable the sustained use of ocean and coastal resources

IOOS Subsystems

3 linked subsystems:• Observing system• Data Management

and Communications (DMAC)

• Data Analysis and Modeling (DAM)

Data Management and Communications (DMAC)

Goal: Deliver real-time and delayed-mode observations to a wide variety of users

Accomplish through use of standards• Adopt• Adapt• Create only as a last resort

DMAC Functions

DMAC Standards Process

Steering TeamExpert Teams

Metadata and Data Discovery **ArchiveTransport and Access

CaucusesPrivate sectorInternationalModelingRegional

Working GroupSystems Engineering

Interagency Oversight Working Group

Metadata Expert Team

Provide recommendations on standards for use in IOOS DMAC

Starting point:“DMAC Plan” -- Data Management and

Communications Plan for Research and Operational Integrated Ocean Observing Systems, March 2005

Metadata and Data Discovery Expert Team

Current Status:• Organized in November, 2005• Teleconference calls every other week• Provided 5 recommendations to DMAC Steering

Team in May, 2006• Currently planning recommendations for next

DMAC Steering Team meeting in November, 2006

Metadata Expert Team Members

• Selected to represent various groups of interest to IOOS

• Approved by the DMAC Steering Team

• Federal agencies• Academia• Private sector• Metadata organizations (MMI, FGDC, OBIS, ISO

19115, GCMD)• Related projects (ORION, GEOSS)• IOOS Regional Associations

Metadata Expert Team Members

Name OrganizationAnne Ball (co-chair) NOAA Coastal Services Center

Julie Bosch (co-chair) NOAA National Coastal Data Development Center

Rob Cermak Alaska Ocean Observing System (IOOS Regional Association)

Helen Conover University of Alabama in Huntsville

Nan Galbraith Woods Hole Oceanographic Institute

John Graybeal Monterey Bay Aquarium Research Institute, MMI, ORION

Tom Gulbransen Battelle

Stephen Hale EPA

Lola Olsen NASA Global Change Master Directory

Karen Stocks Ocean Biological Information System (OBIS), San Diego Supercomputer Center

Linda Wayne Federal Geographic Data Committee (FGDC), ISO 19115

Metadata Definition

The information needed to identify

assess use

access exchange

transportarchive

data for IOOS

Metadata Aspects

Metadata Expert Team divided metadata issues into the following:

• Content• Format specification• Vocabularies• Discovery

Team’s Approach

Identify existing practices – What currently exists that IOOS can readily

use/adopt

Evaluate capabilities – What does the use of a particular practice buy the

IOOS community

Determine IOOS requirements– How and What DMAC and IOOS participants need

to do in order to utilize these “systems”

Recommendation - Content

Initial minimum list of elements needed to support functions:

• Consumer use• Data management• Discovery• Access• Transport• Archive

Consumer use

Example of elements for consumer use• Textual summary or abstract, purpose• Observation parameters (attribute names, units, valid values, etc.) • Data source information (instrument type, manufacturer,

calibration, etc.)• Originator • Quality assurance / quality control methods • Sample collection methods in field (when applicable)• Sample analysis methods (when applicable)• Data processing methods• Spatial extent (horizontal and vertical geographic location)• Accuracy of location • Temporal extent• Use constraints • Liability

Recommendation – Content and Format

Initial list of recommended metadata standards for use in IOOS

• FGDC Content Standard for Digital Geospatial Metadata

• Global Change Master Directory’s Directory Interchange Format (DIF)

• Ocean Biogeographical Information System (OBIS) metadata standard

• ISO 19115

Recommendation - Discovery

Recommendation: Submit metadata to the appropriate national or international catalog

• National Spatial Data Infrastructure (NSDI) Clearinghouse

• Geospatial One-Stop (GOS)• Global Change Master Directory (GCMD)• Ocean Biogeographic Information System (OBIS)

Rationale

These catalog services:

• are national and/or international in extent• are established resources with support

and ample guidance• provide for data discovery • enable initial cataloging of IOOS data• possibly applied as step in observing

system “certification”

Next Steps

Build IOOS interfaces catalog services• NSDI Clearinghouses• GCMD and OBIS• Geospatial OneStop

Recommendation - Vocabularies

Adopt a standard controlled IOOS vocabulary

• A discrete set of terms that can be referenced in metadata (keywords) and other IOOS documentation

• Drawn from existing IOOS documentation:• The First U.S. Integrated Ocean Observing System (IOOS)

Development Plan, Ocean.US Publication No. 9, January 2006.

• Contains terms identified as• IOOS Identifiers• IOOS Core Variables • IOOS National Backbone Programs

IOOS Identifiers

• Integrated Ocean Observing System• IOOS

IOOS Core Variables

• Bathymetry• Bottom Character• Contaminants• Dissolved Nutrients• Dissolved O2

• Fish Abundance• Fish Species• Heat flux• Ice Distribution• Ocean color

• Optical properties• Pathogens• Phytoplankton species• Salinity• Sea Level• Surface Currents• Surface Waves• Temperature• Zooplankton abundance• Zooplankton species

IOOS National Backbone Programs

• Altimeter Data Fusion Center • ADFC • Benthic Habitat Mapping and Monitoring • Coastal Change Assessment Monitoring • Coastal Field Data Collection Program • CFDCP • Coastal Mapping• Coastal-Marine Automated Network • C-MAN • Coastwatch • Commercial Statistics • Coral reef mapping• Coral reef monitoring• Geostationary Operational Environmental Satellites • GOES • Global Seismic Network • GSN• Habitat Assessment • Hydrographic Surveying • Integrated Buoy Program • Living Marine Resources-Ecosystem Survey • LMER-ES

• National Current Observing Program • National Data Buoy Center • NDBC • National Estuarine Research Reserve System• NERRS • National Observer Program • National Stream Quality Accounting Network• NSQAN • National Streamflow Information Program • NSIP • National Water Level Observation Network• NWLON• Physical Oceanographic Real-Time System • PORTS • Polar Operational Environmental Satellite • POES • Protected Resource Surveys • Recreational Fisheries• Shoreline Change • Stream gauging• Topographic change mapping

Next Steps

• Expand vocabulary as recommended to include:– Additional core variables as defined by IOOS– OOS characteristics as determined by Regional

Associations and individual Observing Systems• System characteristics• Sensors• Observed parameters• Etc.

• Submit Version 2 for Steering Team approval (Nov 06)

Required Vocabularies

IOOS Vocabulary Version 1• Identifiers (required)• Core variable (recommended)• National Backbone Programs (recommended)

ISO Topic Categories• farming• biota• boundaries• climatologyMeteorologyAtmosphere• economy• elevation• environment• geoscientificInformation• health• imageryBaseMapsEarthCover

• intelligenceMilitary• inlandWaters• location• oceans*• planningCadastre• sociology• structure• transportation• utilitiesCommunication

Recommended Vocabularies

GCMD's Science Keywords and Associated Directory Keywords• Apply to DIF and SERF metadata for use in GCMD• Can be applied to FGDC metadata for discovery through NSDI and GOS

IHO Codes for Oceans and Seas • Apply to OBIS discovery metadata• Can be applied to FGDC metadata for discovery through NSDI and GOS

OBIS Marine Species Groups• Apply to OBIS discovery metadata• Can be applied to FGDC metadata for discovery through NSDI and GOS

CF Standard Names• Recommended by Modeling Caucus for model output datasets• Can be applied to FGDC metadata for discovery through NSDI and GOS

Next Steps

OOS community outreach• identification of individual OOS vocabularies• conversion of vocabularies into OWL• ontology development – harmonization among

vocabularies

Work with the RA Caucus, Marine Metadata Interoperability (MMI) project and other groups to advance the above items

The Mega-Metadata Matrix

• Excel spreadsheet created to compare metadata standards

• Crosswalk of fields• Checkboxes for functions

Metadata Standards Review and Supporting Tools

Review of existing metadata standards to: • Identify those that meet IOOS requirements• Expedite the process to review future standards• Identify where further refinements are needed for

existing metadata standards

14 metadata standards + 850 metadata fields + 6 functionalities + 4 data types

Matrix

Comparing Metadata Standards

Status

Selection of metadata standards• DMAC plan, input from team• Need input from community

Crosswalk of fields• Completed for FGDC, DIF, OBIS metadata, ISO 19115• Started for others but needs lots of input and review• Not a comprehensive list of fields to date

Functionalities• Some work mapping fields to IOOS minimal metadata requirements

Data/observation types• No work to date

Matrix – Metadata Standards

14 metadata standards:• Federal Geographic Data Committee (FGDC)• DIF• OBIS metadata• OBIS schema• ISO 19115• JGOFS• netCDF+• WMO core ocean gridded data• Open Archival Information System (OAIS)• Ecological Metadata Language• Geographic Markup Language• Sensor ML• Marine XML• Earth Science Markup Language

Matrix - Functionalities

6 functionalities:• Consumer use• Data management• Discovery• Access• Transport• Archive

Results in IOOS minimal or “must have” metadata

Matrix – FunctionalitiesExample

Next Steps

Critical component of DMACRequires lots of time and attention to detail

• Complete crosswalks for other metadata standards• Analyze field content for machine-to-machine

interoperability• Research fields needed to adequately describe

observation types, forms, and parameters• Track and incorporate efforts such as MMI, QARTOD,

ORION to document observational parameters of interest to IOOS

• MMI Sensor Workshop!

Challenges

• How many metadata standards will be acceptable/workable?• How many clearinghouse standards will be acceptable/workable?• Will DMAC need to provide tools to garner participation?• Who will write the code to make data discovery work? Who will

write tools? How much funding is there for these things?• Focus on how to build data discovery from various clearinghouses

(mixture of all of above?)• How will data discovery interoperate/coordinate with data access

and transport?• How will data discovery figure into system architecture plan now

being developed?• Build a portal to which clearinghouse?

Challenges

EML

ISO

DCMI

FGDCMarineXML

GML

ADL

NetCDF

ASCII

Content

Protocol

ESML

OPeNDAP

REST

SOAPZ39.50

DFDL

THREDDS LAS

WxS

HTTP

HDF

TIF

JPEG

Metadata Data

Coards/CF

*Slide by John Graybeal

More Information

More info on IOOS and DMAChttp://www.ocean.us

More info on Metadata expert teamAnne.Ball@noaa.govJulie.Bosch@noaa.gov