Peter CornillonPeter CornillonUniversity of Rhode IslandUniversity of Rhode Island

Presented at the 10 September 2003 Presented at the 10 September 2003 NVODS WorkshopNVODS Workshop

Washington DCWashington DC

The National Virtual Ocean The National Virtual Ocean Data System (NVODS): Data System (NVODS):

A Status ReportA Status Report

Workshop ObjectivesWorkshop Objectives

Inform the community – NVODS status

Solicit input with regard to the future direction of NVODS.

Encourage participation in the NVODS effort.

Workshop ObjectivesWorkshop Objectives1. Develop a Comprehensive IOOS Data Model

2. Deliver time-critical (real-time) data to data assembly and operational modeling sites

3. Develop DMAC Middleware

4. Make data available using IOOS middleware solution

5. Data Manipulation Services

6. Develop Metrics and Implement Performance Monitoring

7. Implement Middleware Security (Cross-discipline effort with all DMAC)

8. Provide guaranteed geo-temporal-referenced browse for all IOOS data

To Inform - OutlineTo Inform - Outline

Overview of NVODS

OPeNDAP – the NVODS core infrastructure

NVODS/OPeNDAP Status

The future

What is NVODS?What is NVODS?

NVODS is operated from URI, but involves significant participation from a number of oceanographic institutions in the US and abroad.

NVODS involves for oceanographic data all data system aspects from data discovery through delivery to the user’s application.

Data transport within NVODS is based on the OPeNDAP data access protocol.

Specifically, NVODSSpecifically, NVODS Facilitates access to oceanographic data and data

products via the Internet regardless of: data type, where the data are stored, the format in which they are stored, the user's visualization package, or the user's level of expertise,

Is envisioned as a component of IOOS, the Integrated Ocean Observing System,

NVODSNVODS

Has been desgined to evolve with

the Internet, computer hardware, computer software, and the interests of its users,

Involves a major population effort, and

Makes use of the GCMD, the Global Change Master Directory, for data discovery.

NVODS does NVODS does NOTNOT deal with deal with

The acquisition of data at the sensor,

Issues related to the long term storage of these data,

A system processing capability,

The actual network (hardware) infrastructure, nor

The very low level data transport protocols, such as tcp/ip.

NVODS and the End-to-End Data SystemNVODS and the End-to-End Data System

NVODS Evolved from DODSNVODS Evolved from DODS

DODS, the Distributed Oceanographic Data System begun in the early 1990s, consisted of two fundamental parts:

a discipline independent core infrastructure for moving data on the net,

a discipline specific portion related to data –population, location, specialized clients, etc.

DODS DODS NVODS & OPeNDAP NVODS & OPeNDAP

To isolate the discipline independent part of the system from the discipline specific part, two entities have been formed:

National Virtual Ocean Data System (NVODS)

Open Source Project for a Network Data Access Protocol (OPeNDAP)

OPeNDAPOPeNDAP

OPeNDAP was formed to maintain and evolve the DODS core infrastructure,

OPeNDAP is a non-profit corporation, and

OPeNDAP is a discipline neutral data access protocol.

OPeNDAP in the End-to-End Data SystemOPeNDAP in the End-to-End Data System

NVODS/OPeNDAPNVODS/OPeNDAPStatusStatus

OPeNDAP/NVODS Server Sites

Institutions with OPeNDAP ServersInstitutions with OPeNDAP Servers

Carolinas Coastal Ocean Observing and Prediction System (Caro-COOPS) Center for Ocean Land Atmosphere Studies (COLA) Columbia University/LDEO - International Research Institute (IRI/LDEO) Florida State University - Center for Ocean-Atmospheric Prediction Studies (COAPS) George Mason University - Seasonal to Interannual Earth Science Information Partner Gulf of Maine Ocean Observing System (GoMOOS) Maine - Department of Marine Resources (Maine DMR) Monterey Bay Aquarium Research Institute (MBARI) Naval Oceanographic Office (NAVOCEANO) NASA/GSFC - Goddard Distributed Active Archive Center (GDAAC) NASA/JPL - Physical Oceanography Distributed Active Archive Center (PODAAC) NASA/JPL - Ocean Earth Science Information Partner (OceanESIP) NCAR - Vegetation/Ecosystem Modeling and Analysis Project (VEMAP) New York University - Center of Atmosphere Ocean Science NOAA - Climate Diagnostic Center (CDC) NOAA - Geophysical Fluid Dynamics Laboratory (GFDL) NOAA - National Center for Environmental Prediction (NCEP)

Universities/NGO Research Labs State Organizations

NOAA NASA

USGS Navy

Institutions with OPeNDAP ServersInstitutions with OPeNDAP Servers

NOAA - National Climatic Data Center (NCDC) NOAA - National Data Buoy Center (NDBC) NOAA - Northeast Fisheries Science Center (NMFS/NEFSC) NOAA - Pacific Marine Environmental Laboratory (PMEL) Oregon State University - Remote Sensing Ocean Optics (ORSOO) Science Applications International Corporation (SAIC) Texas A&M University (TAMU) UCAR - Unidata Internet Data Distribution (IDD) United States Geological Survey - Woods Hole Field Center (USGS) University of Alabama/Huntsville - Global Hydrology Resource Center (GHRC) University of Hawaii/SOEST - International Pacific Research Center University of Hawaii/SOEST - Sea Level Center University of Kansas - Kansas Geological Survey University of Massachusetts - School of Marine Science and Technology Universite Pierre et Marie Curie - Laboratoire de Meteorlogie Dynamique du C.N.R.S University of Rhode Island - Graduate School of Oceanography Woods Hole Oceanographic Institution - US/GLOBEC Georges Bank Program (GLOBEC) Woods Hole Oceanographic Institution - Upper Ocean Mooring Data Archive (UOP)

Universities/NGO Research Labs State Organizations

NOAA NASA

USGS Navy

OPeNDAP Client and Server StatusOPeNDAP Client and Server Status

Special ServersSpecial Servers

0

200

400

600

800

1000

1200

1400

GDS Unique Users per Month

0

200,000

400,000

600,000

800,000

1,000,000

1,200,000

1,400,000

1,600,000

1,800,000

Sep-01

Oct-01

Nov-01

Dec-01

Jan-02

Feb-02

Mar-02

Apr-02

May-02

Jun-02

Jul-02

Aug-02

Sep-02

Oct-02

Nov-02

Dec-02

Jan-03

Feb-03

Mar-03

Apr-03

May-03

Jun-03

GDS Data Access per Month

0

10

20

30

40

50

60

70

80

90

Dat

a V

olu

me

(GB

)

Sep-01

Oct-01

Nov-01

Dec-01

Jan-02

Feb-02

Mar-02

Apr-02

May-02

Jun-02

Jul-02

Aug-02

Sep-02

Oct-02

Nov-02

Dec-02

Jan-03

Feb-03

Mar-03

Apr-03

May-03

Jun-03

GDS Data Volume Delivered per Month

00,000

50,000

100,000

150,000

200,000

250,000

Sep-01

Oct-01

Nov-01

Dec-01

Jan-02

Feb-02

Mar-02

Apr-02

May-02

Jun-02

Jul-02

Aug-02

Sep-02

Oct-02

Nov-02

Dec-02

Jan-03

Feb-03

Mar-03

Apr-03

May-03

Jun-03

GDS Analysis Expressions per Month

Interesting OPeNDAP Access StatisticsInteresting OPeNDAP Access Statistics

LDEO data accesses for 1st quarter of 2002

Type Requests % Volume (gb) %

OPeNDAP

191,611 8.5 375.2 69.4

Other 2,062,681 91.5 165.2 30.6

Total 2,254,292 100.0 540.4 100.0

Type Requests % Volume (gb) %

OPeNDAP

643,892 16.5 880.2 79.6

Other 3,264,729 83.5 225.9 20.4

Total 3,908,621 100.0 1,106.1 100.0

LDEO data accesses for 4th quarter of 2002

The FutureThe Future

Near term plans – existing funding (sort of)

New funding

Challenges

Near Term PlansNear Term Plans OPeNDAP

DAP 4 Provide web service access to the DAP

XML version of the DAP

SOAP interface to the DAP

Add bit-image data type

Add cataloging capabilities to OPeNDAP-servers using THREDDS technology

Extend Ancillary Information Services (AIS) capabilites

Complete the OPeNDAP data Connector

Near Term PlansNear Term Plans

NVODSPopulate Ancillary Information Services (AIS)

site for oceanographic data

Work with groups implementing pilots

New FundingNew Funding

NASA REASoN CAN – Thematic Portals for Satellite-Derived Surface Ocean Parameters2003-2008 - Between $500 and $750/year to NVODSAbout ½ of funding for OPeNDAPWill provide OPeNDAP/LAS access to satellite-derived

SST, Surface Winds, Ocean Color, Precipitation and SSH data sets.

NOPP – GODAE2003-2005 - $70k/year to OPeNDAP Will work with the data assimilation community to

make their data accessible via OPeNDAP/LAS

ChallengesChallenges

Data discovery – Directory population

GIS access

Consistent semantic data model(s)

ChallengesChallenges

Data discovery – Directory population Finding new servers and new data sets at existing servers

OPeNDAP-enabled servers are rapidly being installed

New data sets are being added at existing server sites

Obtaining complete and consistent data set descriptions Inadequate data set descriptions

No link between the data set description and OPeNDAP access to the data.

Data sets consisting of collections of holdings pose special problems

Each of the individual items may be well defined, but not the collection as a whole, or

There may not be a single entry point for the collection

ChallengesChallenges

Data discovery – Directory population (cont.)Handling virtual data sets and server services

New data sets obtained by combining data from several sites

• The collection of mooring data obtained from several sites New data objects obtained via operations on the data at a site

• Wind stress curl derived from the wind stress field

Handling replicated data sets Some of the more popular data sets are replicated at several sites

ChallengesChallenges

GIS accessAccess from GIS-based servers

Grid access

Feature access

Access from GIS packages Grid access

Feature access

ChallengesChallenges

Consistent semantic data model(s)

NVODS:NVODS: http://nvods.org

OPeNDAP:OPeNDAP: http://opendap.org

DODS:DODS: http://unidata.ucar.edu/packages/dods

Data System URLSData System URLS