HDF5 OPeNDAP Project MuQun Yang and Hyo-Kyung Lee (THG) James Gallagher (OPeNDAP, Inc.)
-
date post
19-Dec-2015 -
Category
Documents
-
view
224 -
download
0
Transcript of HDF5 OPeNDAP Project MuQun Yang and Hyo-Kyung Lee (THG) James Gallagher (OPeNDAP, Inc.)
![Page 1: HDF5 OPeNDAP Project MuQun Yang and Hyo-Kyung Lee (THG) James Gallagher (OPeNDAP, Inc.)](https://reader030.fdocuments.net/reader030/viewer/2022033105/56649d2c5503460f94a0273c/html5/thumbnails/1.jpg)
HDF5 OPeNDAP Project
MuQun Yang and Hyo-Kyung Lee (THG)
James Gallagher (OPeNDAP, Inc.)
![Page 2: HDF5 OPeNDAP Project MuQun Yang and Hyo-Kyung Lee (THG) James Gallagher (OPeNDAP, Inc.)](https://reader030.fdocuments.net/reader030/viewer/2022033105/56649d2c5503460f94a0273c/html5/thumbnails/2.jpg)
Question of the Day
• How can I get One Sub from SUBWAY?• Chicken teriyaki with no onion
• How can I get One Subset of EOS data image from NASA?• Today’s satellite image right above the roof
![Page 3: HDF5 OPeNDAP Project MuQun Yang and Hyo-Kyung Lee (THG) James Gallagher (OPeNDAP, Inc.)](https://reader030.fdocuments.net/reader030/viewer/2022033105/56649d2c5503460f94a0273c/html5/thumbnails/3.jpg)
Subway
• Easy: anywhere / accepts Visa|MC
• Efficient: your own recipe• Cool: eat fresh and look good
How can I get One Sub from SUBWAY?
![Page 4: HDF5 OPeNDAP Project MuQun Yang and Hyo-Kyung Lee (THG) James Gallagher (OPeNDAP, Inc.)](https://reader030.fdocuments.net/reader030/viewer/2022033105/56649d2c5503460f94a0273c/html5/thumbnails/4.jpg)
Dumb-way
• Difficult: bring a USB drive and visit NASA
• Inefficient: download 10 Gbyte and search for 10 byte dataset
• Not cool: wrong or useless dataset
How can I get One Subset of EOS data image from NASA?
![Page 5: HDF5 OPeNDAP Project MuQun Yang and Hyo-Kyung Lee (THG) James Gallagher (OPeNDAP, Inc.)](https://reader030.fdocuments.net/reader030/viewer/2022033105/56649d2c5503460f94a0273c/html5/thumbnails/5.jpg)
DAP-way!
• Easy: anywhere / accepts IE|FF
• Efficient: WYSIWYG – less fat!
• Cool: Visualization clients – see fresh!
How can I get One Subset of EOS data image from NASA?
![Page 6: HDF5 OPeNDAP Project MuQun Yang and Hyo-Kyung Lee (THG) James Gallagher (OPeNDAP, Inc.)](https://reader030.fdocuments.net/reader030/viewer/2022033105/56649d2c5503460f94a0273c/html5/thumbnails/6.jpg)
DAP-way Implementation
• OPeNDAP: Free DAP-way
Software
• Server-Client Model APIs
• Widely used by Earth Science
Community
![Page 7: HDF5 OPeNDAP Project MuQun Yang and Hyo-Kyung Lee (THG) James Gallagher (OPeNDAP, Inc.)](https://reader030.fdocuments.net/reader030/viewer/2022033105/56649d2c5503460f94a0273c/html5/thumbnails/7.jpg)
OPeNDAP
• Market Dominance:
•
• http://www.opendap.org
OPeNDAP server sites
Courtesy of OPeNDAP, Inc.
![Page 8: HDF5 OPeNDAP Project MuQun Yang and Hyo-Kyung Lee (THG) James Gallagher (OPeNDAP, Inc.)](https://reader030.fdocuments.net/reader030/viewer/2022033105/56649d2c5503460f94a0273c/html5/thumbnails/8.jpg)
OPeNDAP Servers
Data Data Data Data Data Data Data
Matlab
HDF4 JDBC
FreeFromFITS
CDF CEDAR
Data
netCDF
netCDF HDF4 Matlab
Data
DSP
DSP
Data
JGOFS
Tables SQL FITS CDFFlat
Binary CEDAR
Data
CODAR
Data
ESML
GeneralCODAR
Courtesy of OPeNDAP, Inc.
![Page 9: HDF5 OPeNDAP Project MuQun Yang and Hyo-Kyung Lee (THG) James Gallagher (OPeNDAP, Inc.)](https://reader030.fdocuments.net/reader030/viewer/2022033105/56649d2c5503460f94a0273c/html5/thumbnails/9.jpg)
OPeNDAP Clients
netCDF C netCDF Java
IDVFerret GrADS VisAD ncBrowse Matlab ExcelIDL Access
MatlabClient
IDLClient
Courtesy of OPeNDAP, Inc.
![Page 10: HDF5 OPeNDAP Project MuQun Yang and Hyo-Kyung Lee (THG) James Gallagher (OPeNDAP, Inc.)](https://reader030.fdocuments.net/reader030/viewer/2022033105/56649d2c5503460f94a0273c/html5/thumbnails/10.jpg)
How OPeNDAP works
OPeNDAP Client
OPeNDAPHDF4 Server
HDF4
OPeNDAPNetCDF Server
NetCDF
![Page 11: HDF5 OPeNDAP Project MuQun Yang and Hyo-Kyung Lee (THG) James Gallagher (OPeNDAP, Inc.)](https://reader030.fdocuments.net/reader030/viewer/2022033105/56649d2c5503460f94a0273c/html5/thumbnails/11.jpg)
Architecture of OPeNDAP Server
• Server 3 (CGI)
• C/C++ based HTTP Server
• Insecure / Inflexible
• Server 4 (Hyrax)
• Java based TomCat Servlet Engine
• Secure / Flexible
![Page 12: HDF5 OPeNDAP Project MuQun Yang and Hyo-Kyung Lee (THG) James Gallagher (OPeNDAP, Inc.)](https://reader030.fdocuments.net/reader030/viewer/2022033105/56649d2c5503460f94a0273c/html5/thumbnails/12.jpg)
Server 3 (CGI) Architecture
OLFS
Java Servlet Engine
BES
Unix Daemon
BES Commands
XML- encapsulated object
DataStore
File system with data files like HDF, NetCDF and SQL Database, …
DAP
XML, GIS, KML
HTML
Optional catalogs: XML, GIS, KML
Server 4 (Hyrax) ArchitectureHTTP Server
Courtesy of OPeNDAP, Inc.
http://docs.opendap.org/index.php/Hyrax
![Page 13: HDF5 OPeNDAP Project MuQun Yang and Hyo-Kyung Lee (THG) James Gallagher (OPeNDAP, Inc.)](https://reader030.fdocuments.net/reader030/viewer/2022033105/56649d2c5503460f94a0273c/html5/thumbnails/13.jpg)
Example Usage
OPeNDAP Client
OPeNDAPHDF4 Server
HDF4
Syntactic Structure of DataSemantic Meaning of Data
Actual Content of Data
OPeNDAP Visualization Client
![Page 14: HDF5 OPeNDAP Project MuQun Yang and Hyo-Kyung Lee (THG) James Gallagher (OPeNDAP, Inc.)](https://reader030.fdocuments.net/reader030/viewer/2022033105/56649d2c5503460f94a0273c/html5/thumbnails/14.jpg)
What about HDF5?
OPeNDAP Client
OPeNDAPHDF4 Server
HDF4
OPeNDAPNetCDF Server
NetCDF
OPeNDAPHDF5 Server
HDF5
![Page 15: HDF5 OPeNDAP Project MuQun Yang and Hyo-Kyung Lee (THG) James Gallagher (OPeNDAP, Inc.)](https://reader030.fdocuments.net/reader030/viewer/2022033105/56649d2c5503460f94a0273c/html5/thumbnails/15.jpg)
Why Important?
• HDF5: NASA EOS / NPOESS
• NASA’s MISSION: ACCESS!
• Our MISSION: Build DAP-way Bridge
![Page 16: HDF5 OPeNDAP Project MuQun Yang and Hyo-Kyung Lee (THG) James Gallagher (OPeNDAP, Inc.)](https://reader030.fdocuments.net/reader030/viewer/2022033105/56649d2c5503460f94a0273c/html5/thumbnails/16.jpg)
Goals
•1st: Transform HDF5 access in DAP-way
• 2nd: Yet preserve the beauty of HDF5
![Page 17: HDF5 OPeNDAP Project MuQun Yang and Hyo-Kyung Lee (THG) James Gallagher (OPeNDAP, Inc.)](https://reader030.fdocuments.net/reader030/viewer/2022033105/56649d2c5503460f94a0273c/html5/thumbnails/17.jpg)
OPeNDAP HDF5 Server
1. Map HDF5 Datatypes in DAP-way
2. Plus, tweaks for OPeNDAP Clients
3. Plus, tweaks for HDF-EOS Files
Goal: Transform HDF5 access in DAP-way
![Page 18: HDF5 OPeNDAP Project MuQun Yang and Hyo-Kyung Lee (THG) James Gallagher (OPeNDAP, Inc.)](https://reader030.fdocuments.net/reader030/viewer/2022033105/56649d2c5503460f94a0273c/html5/thumbnails/18.jpg)
Mapping HDF5 in DAP-way• Prototype server in 2001
• NASA Grant from Nov. 2006
• Support for Compound Datatype
• Support for Group
• Support for References / Links
• First Product Release in Mar. 2008
![Page 19: HDF5 OPeNDAP Project MuQun Yang and Hyo-Kyung Lee (THG) James Gallagher (OPeNDAP, Inc.)](https://reader030.fdocuments.net/reader030/viewer/2022033105/56649d2c5503460f94a0273c/html5/thumbnails/19.jpg)
OPeNDAP HDF5 Server
1. Map HDF5 Datatypes in DAP-way
2. Plus, tweaks for OPeNDAP Clients
3. Plus, tweaks for HDF-EOS Files
Goal: Transform HDF5 access in DAP-way
![Page 20: HDF5 OPeNDAP Project MuQun Yang and Hyo-Kyung Lee (THG) James Gallagher (OPeNDAP, Inc.)](https://reader030.fdocuments.net/reader030/viewer/2022033105/56649d2c5503460f94a0273c/html5/thumbnails/20.jpg)
Subway Customers
• I want 3 inch-long sub
• I need Wasabi sauce
• I eat only wheat bread
• I eat only meat balls
![Page 21: HDF5 OPeNDAP Project MuQun Yang and Hyo-Kyung Lee (THG) James Gallagher (OPeNDAP, Inc.)](https://reader030.fdocuments.net/reader030/viewer/2022033105/56649d2c5503460f94a0273c/html5/thumbnails/21.jpg)
OPeNDAP Clients
• I hate foot-long variable names.
• I need special attributes on dataset.
• I care pre-defined Grid data type.
• I care only well-formed attributes.
Not all OPeNDAP clients are created equal!
![Page 22: HDF5 OPeNDAP Project MuQun Yang and Hyo-Kyung Lee (THG) James Gallagher (OPeNDAP, Inc.)](https://reader030.fdocuments.net/reader030/viewer/2022033105/56649d2c5503460f94a0273c/html5/thumbnails/22.jpg)
Tweaks for OPeNDAP Clients Two configuration options
• --enable-short-name = cut bread
• --enable-CF = put Wasabi sauce
![Page 23: HDF5 OPeNDAP Project MuQun Yang and Hyo-Kyung Lee (THG) James Gallagher (OPeNDAP, Inc.)](https://reader030.fdocuments.net/reader030/viewer/2022033105/56649d2c5503460f94a0273c/html5/thumbnails/23.jpg)
OPeNDAP HDF5 Server
1. Map HDF5 Datatypes in DAP-way
2. Plus, tweaks for OPeNDAP Clients
3. Plus, tweaks for HDF-EOS5 Files
Goal: Transform HDF5 access in DAP-way
![Page 24: HDF5 OPeNDAP Project MuQun Yang and Hyo-Kyung Lee (THG) James Gallagher (OPeNDAP, Inc.)](https://reader030.fdocuments.net/reader030/viewer/2022033105/56649d2c5503460f94a0273c/html5/thumbnails/24.jpg)
Subway Customers
• I want 3 inch-long
• I need Wasabi sauce
• I eat only wheat bread
• I eat only meat balls
![Page 25: HDF5 OPeNDAP Project MuQun Yang and Hyo-Kyung Lee (THG) James Gallagher (OPeNDAP, Inc.)](https://reader030.fdocuments.net/reader030/viewer/2022033105/56649d2c5503460f94a0273c/html5/thumbnails/25.jpg)
Subway Suppliers Problem
• No breads, only wheat!
• No meat balls, only meat!
• I eat only wheat bread
• I eat only meat balls
Subway Customers - OK
![Page 26: HDF5 OPeNDAP Project MuQun Yang and Hyo-Kyung Lee (THG) James Gallagher (OPeNDAP, Inc.)](https://reader030.fdocuments.net/reader030/viewer/2022033105/56649d2c5503460f94a0273c/html5/thumbnails/26.jpg)
HDF EOS
• Grid with No Dimension data
• Clients expect Grid with Dimension data
• Metadata Attribute in Two Strings
• Clients expect One Structured Format
Not all HDF5 files are created for DAP-way!
![Page 27: HDF5 OPeNDAP Project MuQun Yang and Hyo-Kyung Lee (THG) James Gallagher (OPeNDAP, Inc.)](https://reader030.fdocuments.net/reader030/viewer/2022033105/56649d2c5503460f94a0273c/html5/thumbnails/27.jpg)
Tweaks for HDF-EOS Two more configuration options
• --enable-eos-grid = bake bread
• --enable-eos-meta = make balls
![Page 28: HDF5 OPeNDAP Project MuQun Yang and Hyo-Kyung Lee (THG) James Gallagher (OPeNDAP, Inc.)](https://reader030.fdocuments.net/reader030/viewer/2022033105/56649d2c5503460f94a0273c/html5/thumbnails/28.jpg)
OPeNDAP HDF5 Server
1. Map HDF5 Datatypes in DAP-way
2. Plus, tweaks for OPeNDAP Clients
3. Plus, tweaks for HDF-EOS5 Files
Goal: Transform HDF5 access in DAP-way
![Page 29: HDF5 OPeNDAP Project MuQun Yang and Hyo-Kyung Lee (THG) James Gallagher (OPeNDAP, Inc.)](https://reader030.fdocuments.net/reader030/viewer/2022033105/56649d2c5503460f94a0273c/html5/thumbnails/29.jpg)
Day After Server Tweaks
• Finally, Happy Clients!
GrADSNCLFerretMATLAB
ncBrowse
IDVODC
![Page 30: HDF5 OPeNDAP Project MuQun Yang and Hyo-Kyung Lee (THG) James Gallagher (OPeNDAP, Inc.)](https://reader030.fdocuments.net/reader030/viewer/2022033105/56649d2c5503460f94a0273c/html5/thumbnails/30.jpg)
Live Demo
Ozone concentration level over the south pole
![Page 31: HDF5 OPeNDAP Project MuQun Yang and Hyo-Kyung Lee (THG) James Gallagher (OPeNDAP, Inc.)](https://reader030.fdocuments.net/reader030/viewer/2022033105/56649d2c5503460f94a0273c/html5/thumbnails/31.jpg)
Goals
•1st: Transform HDF5 access in DAP-way
• 2nd: Yet Preserve the beauty of HDF5
![Page 32: HDF5 OPeNDAP Project MuQun Yang and Hyo-Kyung Lee (THG) James Gallagher (OPeNDAP, Inc.)](https://reader030.fdocuments.net/reader030/viewer/2022033105/56649d2c5503460f94a0273c/html5/thumbnails/32.jpg)
HDF5 Served in DAP-way (Yet Preserve the beauty of HDF5)
• There are some things money can’t buy. For everything else, there is MasterCardTM.
• There are some things DAP can’t serve. For everything else, there is _________?
Minimize
Maximize
![Page 33: HDF5 OPeNDAP Project MuQun Yang and Hyo-Kyung Lee (THG) James Gallagher (OPeNDAP, Inc.)](https://reader030.fdocuments.net/reader030/viewer/2022033105/56649d2c5503460f94a0273c/html5/thumbnails/33.jpg)
Some Things
• Hard: Opaque, Bitmap, Enum, 64 bit Integer, Variable Length types
• Illegal: Reserved characters in DAP are used in Dataset/Group name in HDF5
(that OPeNDAP HDF5 server can’t serve)
![Page 34: HDF5 OPeNDAP Project MuQun Yang and Hyo-Kyung Lee (THG) James Gallagher (OPeNDAP, Inc.)](https://reader030.fdocuments.net/reader030/viewer/2022033105/56649d2c5503460f94a0273c/html5/thumbnails/34.jpg)
For Everything Else
• Let there be HDF5-friendly OPeNDAP client library!
• Package and Deliver in DAP-way first.
• Then, let the client library handle it.
• Concept of Trojan Horse
![Page 35: HDF5 OPeNDAP Project MuQun Yang and Hyo-Kyung Lee (THG) James Gallagher (OPeNDAP, Inc.)](https://reader030.fdocuments.net/reader030/viewer/2022033105/56649d2c5503460f94a0273c/html5/thumbnails/35.jpg)
Example: Group in HDF5Traditional OPeNDAP client library: It’s an attribute that I don’t understand. I’ll ignore it.
HDF5-friendly OPeNDAP client library: I was waiting for this key attribute to re-construct HDF5
![Page 36: HDF5 OPeNDAP Project MuQun Yang and Hyo-Kyung Lee (THG) James Gallagher (OPeNDAP, Inc.)](https://reader030.fdocuments.net/reader030/viewer/2022033105/56649d2c5503460f94a0273c/html5/thumbnails/36.jpg)
Example: Reference in HDF5
• Object / Regional Reference
• Map to DAP URL at server
• No de-referencing of URL at client library
• Important for NPOESS
![Page 37: HDF5 OPeNDAP Project MuQun Yang and Hyo-Kyung Lee (THG) James Gallagher (OPeNDAP, Inc.)](https://reader030.fdocuments.net/reader030/viewer/2022033105/56649d2c5503460f94a0273c/html5/thumbnails/37.jpg)
HDF5-Friendly OPeNDAP Client Library
• One more reason: Tame Clients
• Only ODC can display Swath properly.
• Easy but Evil OPeNDAP NC Client Library! easy: nc style API evil: fixed dimensional attributes
• Our client library must be Easy but Good
No Latitude and Longitude
Courtesy of NASA
![Page 38: HDF5 OPeNDAP Project MuQun Yang and Hyo-Kyung Lee (THG) James Gallagher (OPeNDAP, Inc.)](https://reader030.fdocuments.net/reader030/viewer/2022033105/56649d2c5503460f94a0273c/html5/thumbnails/38.jpg)
libnc-dap
???
liboc-dap
NC-Friendly OPeNDAP Client Library
libdap
dapserver
OPeNDAP NC Server
GrADS
NetCDFView NetCDFHDF5
Group/Ref/Swath
OPeNDAP HDF5 Server
View HDF5
GrADS
View G/R/Swath
HDF5-Friendly OPeNDAP Client Library
![Page 39: HDF5 OPeNDAP Project MuQun Yang and Hyo-Kyung Lee (THG) James Gallagher (OPeNDAP, Inc.)](https://reader030.fdocuments.net/reader030/viewer/2022033105/56649d2c5503460f94a0273c/html5/thumbnails/39.jpg)
Summary
• DAP-way access of HDF5 Is Easy / Efficient / Cool Loses some things Requires HDF5-friendly OPeNDAP Client Library
• HDF5-friendly OPeNDAP Client Library Serves HDF5 better Tames evil visualization clients
![Page 40: HDF5 OPeNDAP Project MuQun Yang and Hyo-Kyung Lee (THG) James Gallagher (OPeNDAP, Inc.)](https://reader030.fdocuments.net/reader030/viewer/2022033105/56649d2c5503460f94a0273c/html5/thumbnails/40.jpg)
Future Work• HDF5 DAP Mapping Document
• Finish HDF5-friendly OPeNDAP Client Library prototype
• Test it on GrADS and display Swath
![Page 41: HDF5 OPeNDAP Project MuQun Yang and Hyo-Kyung Lee (THG) James Gallagher (OPeNDAP, Inc.)](https://reader030.fdocuments.net/reader030/viewer/2022033105/56649d2c5503460f94a0273c/html5/thumbnails/41.jpg)
Project Website
http://hdfdap.hdfgroup.uiuc.edu/joomla
• Beta preview
• Feedbacks are more than welcome!
![Page 42: HDF5 OPeNDAP Project MuQun Yang and Hyo-Kyung Lee (THG) James Gallagher (OPeNDAP, Inc.)](https://reader030.fdocuments.net/reader030/viewer/2022033105/56649d2c5503460f94a0273c/html5/thumbnails/42.jpg)
Credits
Mike Folk (THG)Robert McGrath (NCSA)
Peter Leonard, Daniel Kahn, Marghi Hopkins (ADNET)Christopher Lynnes, James Johnson, Denis Nadeau (NASA)
Jennifer Adams (GrADS)Dave Brown (UCAR)