11

DAPPER: An OPENDAP DAPPER: An OPENDAP Server for In-Situ DataServer for In-Situ Data

Joe SirottJoe Sirott

Donald W. Denbo, Willa H ZhuDonald W. Denbo, Willa H Zhu

University of WashingtonUniversity of Washington

PMEL/NOAAPMEL/NOAA

22

What is Dapper?What is Dapper?

Web server that provides distributed Web server that provides distributed access to in-situ data via OPeNDAP access to in-situ data via OPeNDAP protocolprotocol

Clients include GrADS, MatlabClients include GrADS, Matlab

Available data:Available data:– PMEL EPIC databasePMEL EPIC database– NODC World Ocean Database 2001NODC World Ocean Database 2001

33

Problems with in-situ dataProblems with in-situ data

Irregular (x,y,z,t)Irregular (x,y,z,t)

Not all variables available in all profilesNot all variables available in all profiles

Data from a given profile may be revised Data from a given profile may be revised as QC is appliedas QC is applied

Each profile usually stored as individual Each profile usually stored as individual filefile

No standard APIsNo standard APIs

44

Problems with in-situ dataProblems with in-situ data

Irregular (x,y,z,t)Irregular (x,y,z,t)

Not all variables available in all profilesNot all variables available in all profiles

Data from a given profile may be revised Data from a given profile may be revised as QC is appliedas QC is applied

Each profile usually stored as individual Each profile usually stored as individual filefile

Use Climate Data Portal for aggregationUse Climate Data Portal for aggregation

55

Problems with in-situ dataProblems with in-situ data

No standard APIsNo standard APIs

Use OPeNDAP Sequence datatypeUse OPeNDAP Sequence datatype

66

Dapper architectureDapper architecture

CORBA (IIOP)

Climate Data Portal

MySQL

Java netCDFlibrary

netCDFprofile

netCDFprofile

Dapper

CDPservice

netCDFservice

OPeNDAP client

OPeNDAP Protocol(HTTP)

77

Climate Data Portal (CDP)Climate Data Portal (CDP)

CORBA basedCORBA based

Metadata in database (MySQL)Metadata in database (MySQL)

Variables (measurements) in individual Variables (measurements) in individual netCDF filesnetCDF files

Developed by Don Denbo and Willa ZhuDeveloped by Don Denbo and Willa Zhu

88

OPeNDAPOPeNDAP

Sequence example (header):Sequence example (header):Dataset {Dataset {

Sequence {Sequence {

Float32 longitude;Float32 longitude;

UInt32 time[tsize = 2];UInt32 time[tsize = 2];

Float32 latitude;Float32 latitude;

Int32 _id;Int32 _id;

Sequence {Sequence {

Float32 depth;Float32 depth;

Float32 temperature;Float32 temperature;

} profile;} profile;

} location = { { -42.01, {77, 3850926208}, 53.3402, } location = { { -42.01, {77, 3850926208}, 53.3402, 3645, { { 3, 16.698 }, { 5, 16.71 }, …3645, { { 3, 16.698 }, { 5, 16.71 }, …

99

OPeNDAP (cont.)OPeNDAP (cont.)

Sequences accessible to clients via Java Sequences accessible to clients via Java or C++ librariesor C++ libraries

_id field in “outer” sequence allows unique _id field in “outer” sequence allows unique identification of an individual profileidentification of an individual profile

1010

Dapper designDapper design

Consists of configurable Consists of configurable servicesservices– Climate Data Portal serviceClimate Data Portal service– netCDF servicenetCDF service

Services route OPeNDAP requestServices route OPeNDAP request

Implemented as Java servletImplemented as Java servlet

1111

Dapper architecture revisitedDapper architecture revisited

CORBA (IIOP)

Climate Data Portal

MySQL

Java netCDFlibrary

netCDFprofile

netCDFprofile

Dapper

CDPservice

netCDFservice

OPeNDAP client

OPeNDAP Protocol(HTTP)

1212

CDP service exampleCDP service example

1.1. HTTP request:HTTP request:

http://foo.org/.../http://foo.org/.../dset.cdp.dods?lon,latdset.cdp.dods?lon,lat, , depth,_id,temp&latdepth,_id,temp&lat>0&lat<20>0&lat<20

2.2. Dapper contacts CDP via CORBA IIOPDapper contacts CDP via CORBA IIOP

3.3. CDP accesses MySQL for metadata, appropriate profile files for CDP accesses MySQL for metadata, appropriate profile files for datadata

4.4. CDP returns data to Dapper via CDP APICDP returns data to Dapper via CDP API

5.5. Dapper returns to client as OPeNDAP sequenceDapper returns to client as OPeNDAP sequence

1313

CDP service performanceCDP service performance

< 500 us/profile for coordinate info< 500 us/profile for coordinate info5 ms/profile to read one variable with z 5 ms/profile to read one variable with z axis containing 1000 ptsaxis containing 1000 ptsnetCDF library enhancements:netCDF library enhancements:– Parsing of netCDF header/attributes is Parsing of netCDF header/attributes is

optionaloptional– Use String(char[]) constructor instead of Use String(char[]) constructor instead of

String(byte[]) constructorString(byte[]) constructor2X speed up2X speed up

1414

netCDF servicenetCDF service

Individual profile files available as Individual profile files available as OPeNDAP Grid datatypeOPeNDAP Grid datatype

netCDF library enhancements:netCDF library enhancements:– True streaming serverTrue streaming server

1515

Future workFuture work

Stream data from CDPStream data from CDP

Distribute netCDF file processing using Distribute netCDF file processing using scatter/gather patternsscatter/gather patterns

1616

AvailabilityAvailability

http://www.epic.noaa.gov/dapperhttp://www.epic.noaa.gov/dapper

Joe.Sirott@noaa.govJoe.Sirott@noaa.gov