Weathertop Consulting, LLC

Server-side OPeNDAP Analysis – Server-side OPeNDAP Analysis – Concrete steps toward a generalized Concrete steps toward a generalized

framework via a reference framework via a reference implementation using F-TDSimplementation using F-TDS

Roland SchweitzerRoland SchweitzerWeathertop Consulting, LLCWeathertop Consulting, LLC

Steve Hankin and Ansley Manke Steve Hankin and Ansley Manke NOAA/PMELNOAA/PMEL

Weathertop Consulting, LLC

Stuff I thought I’d get done this Stuff I thought I’d get done this summer, but now can only talk about summer, but now can only talk about

and ask for help.and ask for help.

Roland SchweitzerRoland SchweitzerWeathertop Consulting, LLCWeathertop Consulting, LLC

Weathertop Consulting, LLC

HighlightsHighlights Server-side analysisServer-side analysis

MotivationMotivation ImplementationImplementation

LAS as OPeNDAP Client and ServerLAS as OPeNDAP Client and Server Implementation of the server (and Implementation of the server (and

why you might care)why you might care) Community participation and “call to Community participation and “call to

action”action” SummarySummary

Weathertop Consulting, LLC

Server-side AnalysisServer-side Analysis In general server-side analysis is a In general server-side analysis is a

computation made by an OPeNDAP computation made by an OPeNDAP server at the request of a client.server at the request of a client.

The specification of the computation The specification of the computation is transmitted to the server via the is transmitted to the server via the OPeNDAP URL.OPeNDAP URL.

Weathertop Consulting, LLC

Motivation and Motivation and ImplementationImplementation

We are interested in server-side We are interested in server-side analysis for use with the Live Access analysis for use with the Live Access Server (LAS).Server (LAS). Primarily to implement comparisons Primarily to implement comparisons

between data defined on different gridsbetween data defined on different grids Our implementation borrows heavily Our implementation borrows heavily

from GDS and our experience of from GDS and our experience of running a legacy application (Ferret) running a legacy application (Ferret) from within a Java runtime from within a Java runtime environment.environment.

Weathertop Consulting, LLC

A highly configurable Web server designed to provide flexible access to geo-referenced scientific data

The Live Access Server (LAS)

Weathertop Consulting, LLC

LAS ArchitectureLAS Architecture

ProductServer

SQLBackend Service

DRDSBackend Service

Metadata(XML)

LocalRDBMS

LAS Product Server

client

product

metadataproduct request XML (REST)back endrequest (SOAP)

netCDF data

OPeNDAP server Remote

RDBMSDRDS server

Local netCDF

data

FerretBackend Service

Ferret

Weathertop Consulting, LLC

Comparing OPeNDAP Comparing OPeNDAP datasetsdatasets

metadataproduct request XML (REST)back endrequest (SOAP)

ProductServer

SQLBackend Service

DRDSBackend Service

Metadata(XML)

LAS

user

product

netCDF data 2

OPeNDAP server

netCDF data 1 OPeNDAP

server

Suppose the variables are on different grids?

FerretBackend Service

Ferret

Weathertop Consulting, LLC

LAS as an OPeNDAP ServerLAS as an OPeNDAP Server Data on grids which are available via Data on grids which are available via

LAS are guaranteed to be geo-LAS are guaranteed to be geo-referenced and at least COARDS referenced and at least COARDS compliant.compliant.

We can often “repair” (including re-We can often “repair” (including re-gridding) the data and/or metadata gridding) the data and/or metadata by writing a script of Ferret by writing a script of Ferret commands.commands.

Wouldn’t it be nice to make these Wouldn’t it be nice to make these “repaired” data available via “repaired” data available via OPeNDAP? OPeNDAP?

Weathertop Consulting, LLC

The Ferret-THREDDS The Ferret-THREDDS Data ServerData Server

F-TDS makes this possible.F-TDS makes this possible. F-TDS provides an OPeNDAP view of F-TDS provides an OPeNDAP view of

the data being served by LAS and the data being served by LAS and can make any transformations can make any transformations specified by the associated script specified by the associated script before serving the data.before serving the data.

F-TDS also implements server-side F-TDS also implements server-side analysis (including the ability to pass analysis (including the ability to pass in external data sources).in external data sources).

Weathertop Consulting, LLC

A GDS DigressionA GDS Digression The GrADS Data (DODS) Server was The GrADS Data (DODS) Server was

an early implementation of this an early implementation of this concept.concept.

Both GDS and F-TDS use the Java Both GDS and F-TDS use the Java Runtime environment to invoke the Runtime environment to invoke the associated legacy app (GrADS or associated legacy app (GrADS or Ferret) to do the heavy lifting.Ferret) to do the heavy lifting.

Weathertop Consulting, LLC

F-TDS CapabilitiesF-TDS Capabilities F-TDS takes advantage of several F-TDS takes advantage of several

characteristics of Ferret.characteristics of Ferret. New "virtual" data variables can be defined New "virtual" data variables can be defined Can build the metadata (netCDF header Can build the metadata (netCDF header

described by dimensions, coordinate variables described by dimensions, coordinate variables and the structure of data variables) without and the structure of data variables) without performing any heavy calculations for both data performing any heavy calculations for both data read from files and “virtual” data variablesread from files and “virtual” data variables

Only performs calculations when the data are Only performs calculations when the data are requestedrequested

Only calculates the minimal set needed to fulfil Only calculates the minimal set needed to fulfil the current requestthe current request

Weathertop Consulting, LLC

F-TDS EvolutionF-TDS Evolution The Java netCDF library allows new data The Java netCDF library allows new data

container formats to be plugged-in by container formats to be plugged-in by implementing the I/O Service Provider implementing the I/O Service Provider interface.interface.

Once “plugged-in” clients using nj22 have Once “plugged-in” clients using nj22 have access to the data from this container.access to the data from this container.

We implemented a Ferret I/O Service We implemented a Ferret I/O Service provider which can read Ferret command provider which can read Ferret command scripts and direct Ferret to perform the scripts and direct Ferret to perform the calculations as needed to satisfy data calculations as needed to satisfy data requests.requests.

Weathertop Consulting, LLC

The THREDDS Data The THREDDS Data ServerServer

TDS is an OPeNDAP server based on TDS is an OPeNDAP server based on netCDF Java. netCDF Java.

nj22 I/O Service Providers can be nj22 I/O Service Providers can be plugged in to TDS.plugged in to TDS.

The combination of the Ferret I/O The combination of the Ferret I/O Service Provider and TDS (aka F-Service Provider and TDS (aka F-TDS) serves via OPeNDAP data TDS) serves via OPeNDAP data which are represented by Ferret which are represented by Ferret command scripts (both data read command scripts (both data read from disk by Ferret and virtual data from disk by Ferret and virtual data computed on-the-fly by Ferret).computed on-the-fly by Ferret).

Weathertop Consulting, LLC

The Why You Should CareThe Why You Should Care If you already serve data via TDS you If you already serve data via TDS you

can add server-side analysis to your can add server-side analysis to your TDS for the cost of installing Ferret, TDS for the cost of installing Ferret, and a few Java classes and and a few Java classes and configuration files...configuration files...

and what every cycles get used by the and what every cycles get used by the analysis...analysis...

which may be cheaper than the data which may be cheaper than the data transfertransfer

Weathertop Consulting, LLC

F-TDS and Server-Side F-TDS and Server-Side AnalysisAnalysis

A DataSource handler can also be plugged A DataSource handler can also be plugged in to TDS which allows custom handling of in to TDS which allows custom handling of OPeNDAP requests based on the contents OPeNDAP requests based on the contents of the URL.of the URL.

We built such a DataSource handler which We built such a DataSource handler which recognizes URLs of the form:recognizes URLs of the form:

http://machine:port/thredds/dodsC/http://machine:port/thredds/dodsC/_expr__expr_{dataset1,dataset2,...}{dataset1,dataset2,...}{expression1;expression2;...}{expression1;expression2;...}.URLsuffix?projection.URLsuffix?projection

Weathertop Consulting, LLC

LAS ExampleLAS Example

Weathertop Consulting, LLC

F-TDS Specific ExampleF-TDS Specific Examplehttp://server.gov/thredds/dodsC/data/levitus_expr_{http://server.gov/thredds/dodsC/data/coads}{let sst_2_regrid=sst[d=2,gxy=temp[d=1]]}.dods?SST_2_REGRID[0:1:0][67:1:131][0:1:14]

/thredds/dodsC/data/levitus.dods?TEMP[0:1:0][67:1:131][0:1:14]

/coads.dods?SST[0:1:0][33:1:66][124:1:179]

Weathertop Consulting, LLC

Community ActionCommunity Action Can we define a general syntax for Can we define a general syntax for

server-side analysis requests?server-side analysis requests? Some common operations (ave, diff, Some common operations (ave, diff,

linear interp.) with standard syntaxlinear interp.) with standard syntax Still allow server-specific (native) Still allow server-specific (native)

operationsoperations Can we define a general syntax for Can we define a general syntax for

geo-referenced projections?geo-referenced projections? ERDDAP’s (180),10,(360) syntaxERDDAP’s (180),10,(360) syntax TDS’s TDS’s

north=17.3&south=12.088&west=180north=17.3&south=12.088&west=180&east=360.0 netCDF subset syntax&east=360.0 netCDF subset syntax

Weathertop Consulting, LLC

OPeNDAP “Working” OPeNDAP “Working” GroupGroup

Working is in quotes because we Working is in quotes because we have not been doing any work have not been doing any work recentlyrecently

See:See:http://docs.opendap.org/index.php/http://docs.opendap.org/index.php/Server-side_FunctionsServer-side_Functions

Weathertop Consulting, LLC

SummarySummary Server-side analysis is critical for Server-side analysis is critical for

LAS.LAS. NetCDF Java and the THREDDS Data NetCDF Java and the THREDDS Data

Server is a great platform for Server is a great platform for implementing this type of analysis implementing this type of analysis with a legacy analysis application with a legacy analysis application (like Ferret).(like Ferret).

A community-developed server-side A community-developed server-side analysis framework would make it analysis framework would make it easier to get the advantages of easier to get the advantages of server-side analysis from other server-side analysis from other servers.servers.

Weathertop Consulting, LLC

IOSP*IOSP*

*Stolen directly from John Caron with only this measly acknowledgement.

ApplicationThe low-level part of the NetCDF-Java version 2.2 architecture

NetCDF-3

HDF5

I/O service provider

GRIB

GINI

NIDS

NetcdfFile

NetCDF-4

NexradDMSP

Java Runtime

Ferret

GrADS

readData

open

isValid