NEESgrid Data Technologies

Charles Severance

January 8, 2004

NSF Site Visit

NEESgrid Data - Value Proposition An RDF like store – Referential integrity long-

term flexibility Seamless data and meta data transport Smooth integration of data with meta data Extensible tools Involved with sites through Experiment Based

Deployment

NEESgrid Data – Current Elements Local Repository Central Repository JAVA APIs – Run locally on the same system as a

repository or over OGSA Web Services– NEES File Management Services– NEES Meta Data Services– NEES Data Mapping Services

Data Viewers– Streaming (numeric, X/Y graph)– Stored (X/Y graph, 2-D structure, video)

NEESdata

NEESpop

LocalRepository

Current Elements

AP

I

CentralRepository

DataTeamlets

Data Acquisition

Workstation

AP

I

DataTeamlets

AP

I

Data/MDIngestTools

Data tools

Data viewers

Grid and Web Services

DataServlets

AP

I

Mapping

A Simple Experimental Scenario

M

DAQ System

Glue

Test S

pe

cimen

Lab

view

AP

I

Data

Developer System

Data

M

MD

Researcher System

Data

MD

Repository Browser

Sample of the Video/Data Viewer

Data Viewer

Mappings and the Data Viewer NSDS (ISO 8601 Time channel) Column data with time recorded as a column Column – generate time Column – generate time – trigger filter

Channel units: g,g,in,kipTime ATL1 ATT12002-11-13T15:48:55.26499 -0.006409 0.004272 2002-11-13T15:48:55.36499 -0.005798 -0.003662

100.0000.435 0.161 -1.016 -0.981 0.430 0.161 -1.016 -0.9770.435 0.161 -1.016 -0.977

public class NEESDataMap{ public static boolean repoMap(File mainFile,

File mappingFile, String mapping) {

// Code here }}

NFMSUploadAgent.java

Data Ingestor

NEES Metadata Representation NEES Markup Language (NEESML)

– Provides an RDF-like structure capable of representing semantic information

– XML is the syntax which is used– Logic is more “object oriented”

• Can define objects• Can create objects• Can reference objects

Meta data is many different things…. Goal if we EVER want to build reusable data tools,

we have to represent the semantics inside the meta data rather than just the information

NEESML 1 Introduction NEESML is a means for populating a NEESgrid metadata repository with objects and definitions of object types. It provides a syntax for defining object types and specifying objects, the values of their properties, and the relationships between objects. The NEESgrid metadata ingestion tool accepts NEESML files and uses them to populate a repository with objects and type definitions, communicating with the repository with the NEESgrid metadata service (NMDS) interface.

NEESML can be used to share type definitions and objects between repositories, and as an interface between other applications and repositories, because applications can either be written to read and write NEESML files, or can be augmented with translation utilities that translate the data formats they can read or write to and/or from NEESML. Finally, NEESML can be used to archive metadata to files.

1.1 About this document This document is intended as a comprehensive reference manual for the NEESML language. It does not explain how to us e NEESML to represent real-world objects, but rather completely specifies the syntax and meaning of every NEESML construct. Using this document, you will be able to write NEESML documents that conform to the proper syntax, and will be able to understand syntax errors and other problems with non -conforming NEESML documents or document fragments.

For a more general introduction to NEESML, please refer to (Futrelle, 2003).

Some NEESML-specific terminology is used in this document. The first time every such term is used, it is italicized. Definitions of all special terms are given in the glossary in section 7.

NEESML is a general-purpose metadata description language. Throughout this document, examples are used that do not directly pertain to earthquake engineering. This is done merely in the interest of simplicity. All the constructs used in the examples can be applied to metadata describing earthquake engineering experiments and simulations.

1.2 Some XML “gotchas” NEESML documents are XML documents. There are some aspects of XML that constrain the syntax of NEESML documents in important ways:

? XML element andattribute names are case- . “sensitiveID” is not equivalent to “id”, and “MyTypeName” is not equivalent to “mytypename”.

? Namespaces, if they are used as relation ID’s or the ID’s of types for which objects are created, must be declared in an enclosing element.

? Some characters are not allowed in element names. It is common practice to only use alphabetic characters and dashes. Colons and other punctuation are not allowed.

For details on XML syntax, consult the XML specification (Bray, Paoli, Sperberg-McQueen & Maler, 2000) or a good XML reference or tutorial.

Table 1: Primitive types in NEESML

Name Description Examples

string Text “Hello, world.”“BN# 493-2584x”

int Integer 3-22147483647

long Long integer. Can exceed the size of an integer.

-57823475624279223372036854775807

double Double precision floating point number.

523425.4568574636-0.0000000435234

date A moment in time, represented as a date and time stamp in UTC with 1ms resolution.

2002-10-27 15:40:32.0481969-01-12 00:03:48.774

JAVA APIs

http://neesbox.ncsa.uiuc.edu/chef/doc/repo/api/

Remaining Work

Second Generation Repository API Project Browser Electronic Notebook Data Turbine Video as data Schema/XML Ingestion RDF Model/Data Ingestion Curation Tools ***

Repository API

The NFMS and NMDS APIs are being combined into a single Repository API– Rich support for access control

Access control will use Community Authentication Service (CAS) from the Grid technology

Aligning with JSR-170 Java Content Repository

http://www.jcp.org/en/jsr/detail?id=170

Project Browser

Joint effort between the NEESgrid SI team and Oregon State technical developers

Based on a project browser prototype at Oregon State University

Provides a user friendly interface to Metadata elements - compliment to the project browser and electronic notebook

http://nees.orst.edu/prototype_4/

Electronic Notebook

Collaborative effort with the DOE SciDAC– Electronic notebook - metadata entry– Data mapping– Data provenance– Data display– Slide data/metadata jakarta.apache.org/slide/

Ultimate integration will be via JSR-170 www.scidac.org/SAM/ collaboratory.emsl.pnl.gov/docs/collab/sam/samtechoverview.html

Data Turbine

Commercial, free data streaming toolkit

Developedby NASA

Data Turbine (cont)

Existing data viewers will be adapted to access and display data from data turbine

Data acquisition software will be adapted to place information in Data Turbine Channels

Metadata elements will be developed to represent data turbine live, stored, and derived channels

New efforts (video as data) will be developed from the ground up using Data Turbine

outlet.creare.com/rbnb/

Video as Data

Follow on to initial demonstration at ORST Experiment based development: Minnesota Design phase nearly complete Joint effort, NEESGrid SI, ORST, Minnesota,

UC Davis, Texas, and others as design solidifies

DT Main System

Video as data: User ViewsStill Image / Camera Control

~

< >^

^

< >

Still Image Viewer

< >

Camera ControlGateway

Data Viewer

Thumbnail + Audio + Data

< > +

Schema/XML Ingestion

Several data efforts are expressing their data/models in Schema/XML (Cosmos, etc)

We are developing capabilities to parse XML and automatically extract relevant metadata for the repository

The entire XML file will be stored as data This allows data developers to use tools like

XMLSpy to develop their models.

RDF Integration

Some of the data and meta data task force members are using Protégé-2000 to develop their models and expressing them in RDF.

RDF and NEESML are very similar but not identical so it may be challenging to ingest any arbitrary RDF

We expect that we will be able to map a subset of RDF to NEESML for ingestion or adapt an RDF parser (Jena or Raptor) to ingest that subset directly into the repository

Curated Data Tools ***

Still evolving fine-grain requirements with community– Sites – Consortium DSAC has this as its focus

Some expected minimum requirements– Transfer between repositories– Workflow - implemented as ACLs (incoming, in-

progress, published)– Will be extensions to Repository browser as well

as a simple workflow tool

Conclusion

We are focusing on both core elements and the application of those elements

We are engaging the sites increasingly in the going forward development process

We have a lot of work - some of these efforts will continue post-transition with the sites taking an increasing role in the development