Server-side OPeNDAP Analysis - A General Approach Utilizing Legacy Applications through TDS

Weathertop Consulting, LLC

Server-side OPeNDAP Server-side OPeNDAP Analysis - A General Analysis - A General

Approach Utilizing Legacy Approach Utilizing Legacy

Applications through TDSApplications through TDS Roland SchweitzerRoland Schweitzer

Weathertop Consulting, LLCWeathertop Consulting, LLCSteve Hankin and Ansley Manke Steve Hankin and Ansley Manke

NOAA/PMELNOAA/PMEL


HighlightsHighlights

Server-side analysisServer-side analysis MotivationMotivation

LAS as OPeNDAP Client and ServerLAS as OPeNDAP Client and Server Evolution of the server Evolution of the server

implementationimplementation Community “call to action”Community “call to action” SummarySummary


Server-side AnalysisServer-side Analysis

In general server-side analysis is a In general server-side analysis is a computation made by an OPeNDAP computation made by an OPeNDAP server at the request of a client.server at the request of a client.

The specification of the computation The specification of the computation is transmitted to the server via the is transmitted to the server via the OPeNDAP URL.OPeNDAP URL.


MotivationMotivation

We are interested in server-side analysis We are interested in server-side analysis for use with the Live Access Server (LAS).for use with the Live Access Server (LAS). Primarily as a way to implement comparisons Primarily as a way to implement comparisons

between data defined on different gridsbetween data defined on different grids We want our implementation to leverage We want our implementation to leverage

the analysis capabilities of legacy the analysis capabilities of legacy applications like Ferret and GrADS.applications like Ferret and GrADS.

We want to use our experience running We want to use our experience running legacy applications (like Ferret) from legacy applications (like Ferret) from within a Java runtime environment.within a Java runtime environment.


A highly configurable Web server designed to provide flexible access to geo-referenced scientific data

The Live Access Server The Live Access Server (LAS)(LAS)


LAS ArchitectureLAS Architecture

ProductServer

SQLBackend Service

DRDSBackend Service

Metadata(XML)

LocalRDBMS

LAS Product Server

client

product

metadata

product request XML (REST)

back endrequest (SOAP)

netCDF data

OPeNDAP server Remote

RDBMS

DRDS server

Local netCDF

data

FerretBackend Service

Ferret


Comparing OPeNDAP Comparing OPeNDAP datasetsdatasets

metadata

product request XML (REST)

back endrequest (SOAP)

ProductServer

SQLBackend Service

DRDSBackend Service

Metadata(XML)

LAS

user

product

netCDF data 2

OPeNDAP server

netCDF data 1

OPeNDAP server

Suppose the Suppose the variables are on variables are on different grids?different grids?

FerretBackend Service

Ferret


LAS as an OPeNDAP LAS as an OPeNDAP ServerServer

Data on grids which are available via Data on grids which are available via LAS are guaranteed to be geo-LAS are guaranteed to be geo-referenced and at least COARDS referenced and at least COARDS compliant.compliant.

We can often “repair” (including re-We can often “repair” (including re-gridding) the data and/or metadata by gridding) the data and/or metadata by associating a script of Ferret commands associating a script of Ferret commands with the data source in the LAS with the data source in the LAS configuration.configuration.

Wouldn’t it be nice to make these Wouldn’t it be nice to make these “repaired” data available via OPeNDAP? “repaired” data available via OPeNDAP?


The Ferret Data ServerThe Ferret Data Server

FDS made this possible.FDS made this possible. FDS provides an OPeNDAP view of FDS provides an OPeNDAP view of

the data being served by LAS and the data being served by LAS and makes any transformations specified makes any transformations specified by the associated script before by the associated script before serving the data.serving the data.

FDS also implements server-side FDS also implements server-side analysis (including the ability to pass analysis (including the ability to pass in external data sources).in external data sources).


A GDS DigressionA GDS Digression

The GrADS Data (DODS) Server is The GrADS Data (DODS) Server is the first implementation of this the first implementation of this concept.concept.

In fact, FDS used the Anagram In fact, FDS used the Anagram framework upon which GDS is built.framework upon which GDS is built.

Both GDS and FDS use the Java Both GDS and FDS use the Java Runtime environment to invoke the Runtime environment to invoke the associated legacy app (GrADS or associated legacy app (GrADS or Ferret) to do the heavy lifting.Ferret) to do the heavy lifting.


FDS CapabilitiesFDS Capabilities

FDS took advantage of several characteristics FDS took advantage of several characteristics of the underlying engine (Ferret).of the underlying engine (Ferret). New "virtual" data variables can be defined New "virtual" data variables can be defined Can build the metadata (netCDF header described Can build the metadata (netCDF header described

by dimensions, coordinate variables and the by dimensions, coordinate variables and the structure of data variables) without performing any structure of data variables) without performing any heavy calculations for both data read from files and heavy calculations for both data read from files and “virtual” data variables“virtual” data variables

Only performs calculations when the data are Only performs calculations when the data are requestedrequested

Only calculates the minimal set needed to fulfill the Only calculates the minimal set needed to fulfill the current requestcurrent request


FDS EvolutionFDS Evolution Keep these advantages and evolve the Keep these advantages and evolve the

implementation.implementation. The Java netCDF library allows new data The Java netCDF library allows new data

container formats to be plugged-in by container formats to be plugged-in by implementing the I/O Service Provider implementing the I/O Service Provider interface.interface.

Once “plugged-in” clients using nj22 have Once “plugged-in” clients using nj22 have access to the data from this container.access to the data from this container.

We implemented a Ferret I/O Service provider We implemented a Ferret I/O Service provider which can read Ferret command scripts and which can read Ferret command scripts and direct Ferret to perform the calculations as direct Ferret to perform the calculations as needed to satisfy data requests.needed to satisfy data requests.


ExampleExample

% more data/simple4.jnluse levitus_climatologylet/d=levitus_climatology temp_20 =temp[d=levitus_climatology,z=0:20@sum]set var/title="surface heat content"/units="deg C“ temp_20[d=levitus_climatology]


ExampleExampledncdump -c http://porter.pmel.noaa.gov:8920/thredds/dodsC/mydata/simple4.jnlfloat TEMP(ZAXLEVITR, YAXLEVITR, XAXLEVITR) ; TEMP:units = "DEG C" ; TEMP:long_name = "TEMPERATURE" ; TEMP:_FillValue = -1.e+10f ; TEMP:missing_value = -1.e+10f ; TEMP:dataset = "levitus_climatology.cdf" ;float TEMP_20(YAXLEVITR, XAXLEVITR) ; TEMP_20:dataset = "levitus_climatology" ; TEMP_20:direction = "IJ" ; TEMP_20:units = "deg C" ; TEMP_20:long_name = "surface heat content" ; TEMP_20:missing_value = "-1.00000E+34" ; TEMP_20:virtual = "true" ;


IOSP*IOSP*

*Stolen directly from John Caron with only this measly acknowledgement.

ApplicationThe low-level part of the NetCDF-Java version 2.2 architecture

NetCDF-3

HDF5

I/O service provider

GRIB

GINI

NIDS

NetcdfFile

NetCDF-4

NexradDMSP

Java Runtime

Ferret

GrADS

readData

open

isValid


The THREDDS Data The THREDDS Data ServerServer

TDS is an OPeNDAP server. TDS is an OPeNDAP server. nj22 I/O Service Providers can be nj22 I/O Service Providers can be

plugged in to TDS.plugged in to TDS. The combination of the Ferret I/O The combination of the Ferret I/O

Service Provider and TDS (aka F-Service Provider and TDS (aka F-TDS) serves via OPeNDAP data TDS) serves via OPeNDAP data which are represented by Ferret which are represented by Ferret command scripts (both data read command scripts (both data read from disk by Ferret and virtual data from disk by Ferret and virtual data computed on-the-fly by Ferret).computed on-the-fly by Ferret).


F-TDS and Server-Side F-TDS and Server-Side AnalysisAnalysis

A DataSource handler can also be plugged in to A DataSource handler can also be plugged in to TDS which allows custom handling of OPeNDAP TDS which allows custom handling of OPeNDAP requests based on the contents of the requests based on the contents of the HTTPServletRequest Object (and by implication HTTPServletRequest Object (and by implication the URL).the URL).

We built such a DataSource handler which We built such a DataSource handler which recognizes URL with embedded analysis recognizes URL with embedded analysis expressions.expressions.

The three groups of input to the server-side The three groups of input to the server-side analysis were copied from GDS.analysis were copied from GDS.

The three sets are:The three sets are: Data sources (e.g. OPeNDAP URLs).Data sources (e.g. OPeNDAP URLs). Analysis “commands” which are implementation Analysis “commands” which are implementation

specific.specific. A sub-region.A sub-region.


An Analysis URLAn Analysis URL

http://machine:port/thredds/http://machine:port/thredds/dodsC/dodsC/

_expr__expr_

{dataset1,dataset2,...}{dataset1,dataset2,...}

{expression1;expression2;...}{expression1;expression2;...}

{region}{region}

.URLsuffix?constraint.URLsuffix?constraint

Part of the original GDS specification, not necessary and not often used with FDS


FDS-Specific ExampleFDS-Specific Example

http://host.gov:9090/thredds/dodsC/data/coads_expr_{http://host.gov:9090/thredds/dodsC/data/levitus}{DIF=SST[d=1]-TEMP[d=2,g=SST[d=1]]}.asc?DIF


Community ActionCommunity Action

Can we define a implementation-Can we define a implementation-indepedent syntax for server-side indepedent syntax for server-side analysis requests?analysis requests? Some common operations (averaging, Some common operations (averaging,

differencing, linear interpolation) with differencing, linear interpolation) with standard name and syntaxstandard name and syntax

Server-specific (native) operationsServer-specific (native) operations What mechanism?What mechanism?

An encoded XML string?An encoded XML string? We use this technique in LAS with good successWe use this technique in LAS with good success

A simple command language?A simple command language?


XML Analysis ExpressionXML Analysis Expression

<datasets><datasets> <dataset <dataset id=1>"my_local_dset.nc"</dataset>id=1>"my_local_dset.nc"</dataset> <dataset <dataset id=2>"http://remote_dset.nc"</dataset>id=2>"http://remote_dset.nc"</dataset></datasets></datasets><operation><operation> <name>DIFF</name> <name>DIFF</name> <arg pos=1 dset=1>sst</arg> <arg pos=1 dset=1>sst</arg> <arg pos=2 dset=2>sst</arg> <arg pos=2 dset=2>sst</arg></operation> </operation>


With “native” OperationWith “native” Operation

<datasets><datasets> <dataset <dataset id=1>my_local_dset.nc</dataset>id=1>my_local_dset.nc</dataset> <dataset <dataset id=2>http://remote_dset.nc</dataset>id=2>http://remote_dset.nc</dataset></datasets></datasets><native_operation><native_operation>

LET diff = sst[d=2]-sst[d=1]LET diff = sst[d=2]-sst[d=1]

</native_operation> </native_operation>


SummarySummary Server-side analysis is critical for the Server-side analysis is critical for the

future LAS.future LAS. NetCDF Java 2.2 and the THREDDS NetCDF Java 2.2 and the THREDDS

Data Server is a great platform for Data Server is a great platform for implementing this type of analysis with a implementing this type of analysis with a legacy analysis application (like Ferret).legacy analysis application (like Ferret).

A community-developed server-side A community-developed server-side analysis framework would make it easier analysis framework would make it easier to get the advantages of server-side to get the advantages of server-side analysis from other servers.analysis from other servers.


AcknowledgmentsAcknowledgments

From COLAFrom COLA Jennifer AdamsJennifer Adams Brian DotyBrian Doty Joe WielgoszJoe Wielgosz

From UnidataFrom Unidata John CaronJohn Caron Ethan DavisEthan Davis

Former TMAPFormer TMAP Richard RogersRichard Rogers Yonghua WeiYonghua Wei

Server-side OPeNDAP Analysis - A General Approach Utilizing Legacy Applications through TDS

Documents

Transcript of Server-side OPeNDAP Analysis - A General Approach Utilizing Legacy Applications through TDS