Work Group Meeting on HTAP-Relevant IT Techniques, Tools and Philosophies: DataFed Experience and...
-
Upload
herbert-tyler -
Category
Documents
-
view
223 -
download
1
Transcript of Work Group Meeting on HTAP-Relevant IT Techniques, Tools and Philosophies: DataFed Experience and...
![Page 1: Work Group Meeting on HTAP-Relevant IT Techniques, Tools and Philosophies: DataFed Experience and Perspectives Rudolf B. Husar CAPITA, Washington University,](https://reader035.fdocuments.net/reader035/viewer/2022062408/56649ef05503460f94c01252/html5/thumbnails/1.jpg)
Work Group Meeting onHTAP-Relevant IT Techniques, Tools and Philosophies:
DataFed Experience and Perspectives
Rudolf B. HusarCAPITA, Washington University, St. Louis, MO
Ispra, JRC, March 14. 2004
• Data handling approach• Software tools• Participant involvement• Data dissemination• Integration problems• Summary
![Page 2: Work Group Meeting on HTAP-Relevant IT Techniques, Tools and Philosophies: DataFed Experience and Perspectives Rudolf B. Husar CAPITA, Washington University,](https://reader035.fdocuments.net/reader035/viewer/2022062408/56649ef05503460f94c01252/html5/thumbnails/2.jpg)
Policy Guidance for HTAP IT
Terry Keating, HTAP Task Force Co-Chair:
• Transparency of the HTAP Process– Acceptance of the Tech Report will depend on openness
• Inclusiveness and Ease of Participation– Facilitate participation by smaller contributors
![Page 3: Work Group Meeting on HTAP-Relevant IT Techniques, Tools and Philosophies: DataFed Experience and Perspectives Rudolf B. Husar CAPITA, Washington University,](https://reader035.fdocuments.net/reader035/viewer/2022062408/56649ef05503460f94c01252/html5/thumbnails/3.jpg)
Current Air Quality Information ‘Ecosystem’
AQ info is distributed over many ‘dimensions’: Geography, Content, Agency, Program…AQ info content includes: emissions, ambient & satellite data and model outputsInfo is provided and consumed by different agencies, (NASA, NOAA, EPA…)Providers have different access protocols, formats, and information usage conditions
Lack of Interoperability
Poor data & model utilization
Less informed decision making
![Page 4: Work Group Meeting on HTAP-Relevant IT Techniques, Tools and Philosophies: DataFed Experience and Perspectives Rudolf B. Husar CAPITA, Washington University,](https://reader035.fdocuments.net/reader035/viewer/2022062408/56649ef05503460f94c01252/html5/thumbnails/4.jpg)
GEOSS Info Flow ArchitectureA General Framework Accepted by Members
Model
Model
Data
Data
![Page 5: Work Group Meeting on HTAP-Relevant IT Techniques, Tools and Philosophies: DataFed Experience and Perspectives Rudolf B. Husar CAPITA, Washington University,](https://reader035.fdocuments.net/reader035/viewer/2022062408/56649ef05503460f94c01252/html5/thumbnails/5.jpg)
Federated Network for Air Quality Data and Processing ServicesProject Team:
Software Architecture: R. HusarSoftware Implementation: K. Höijärvi
Data and Applications: S. Falke, R. Husar
Data Handling Approach
![Page 6: Work Group Meeting on HTAP-Relevant IT Techniques, Tools and Philosophies: DataFed Experience and Perspectives Rudolf B. Husar CAPITA, Washington University,](https://reader035.fdocuments.net/reader035/viewer/2022062408/56649ef05503460f94c01252/html5/thumbnails/6.jpg)
• The challenge is to design a general supportive infrastructure• Simply connecting the relevant provides and users for each info product is messy
Federated Data System for Air Quality
• The info system infrastructure needs to facilitate the creation of info products
AQ Compliance
Nowcast/Forecast
Status & Trends
Find Data Gaps
ID New Problems
………
Info Needs
Reports
• Providers supply the ‘raw material’ (data and models) for ‘refined’ info products
EmissionSurface Satellite
Model
Single Datasets
Providers
Wrappers
Where?
What?
When?
Federate Data
Structuring
• Structuring the heterogeneous data into where-when-what ‘cubes’ simplifies the mess
Slice & Dice
Explore Data
Viewers
• The ‘cubed’ data can be accessed and explored by slicing-dicing tools
Programs
Integrate
Understand
• More elaborate data integration and fusion can be done by web service chaining
Non-intrusive Linking & Mediation Data UsersData Providers
![Page 7: Work Group Meeting on HTAP-Relevant IT Techniques, Tools and Philosophies: DataFed Experience and Perspectives Rudolf B. Husar CAPITA, Washington University,](https://reader035.fdocuments.net/reader035/viewer/2022062408/56649ef05503460f94c01252/html5/thumbnails/7.jpg)
Data Handling Approach:
DataFed
Approach: Mediation Between Users and Data ProvidersDataFed assumes autonomous data management (a la Internet)Non-intrusive third-party data wrapping for unified web service (WS) accessEnd-user programming of applications through chaining of WS components
ApplicationsBuilding browsers and analysis tools for distributed monitoring data Serve as data and service resources for user programs (science, GIS tools)Support application projects, e.g. FASTNET, Exceptional Event Rule
![Page 8: Work Group Meeting on HTAP-Relevant IT Techniques, Tools and Philosophies: DataFed Experience and Perspectives Rudolf B. Husar CAPITA, Washington University,](https://reader035.fdocuments.net/reader035/viewer/2022062408/56649ef05503460f94c01252/html5/thumbnails/8.jpg)
Typical DataFed AQ Analysis Tools
Consoles: Data from diverse sources are displayed to create a rich context for exploration and analysis
CATT: Combined Aerosol Trajectory Tool for the browsing backtrajectories for specified chemical conditions
Viewer: General purpose spatio-temporal data browser and view editor applicable for all DataFed datasets
![Page 9: Work Group Meeting on HTAP-Relevant IT Techniques, Tools and Philosophies: DataFed Experience and Perspectives Rudolf B. Husar CAPITA, Washington University,](https://reader035.fdocuments.net/reader035/viewer/2022062408/56649ef05503460f94c01252/html5/thumbnails/9.jpg)
Quebec SmokeJuly, 7, 2002
SeaWiFS Satellite
Aerosol Chemical
Air Trajectory
Map Boarder
Web Service Composition
![Page 10: Work Group Meeting on HTAP-Relevant IT Techniques, Tools and Philosophies: DataFed Experience and Perspectives Rudolf B. Husar CAPITA, Washington University,](https://reader035.fdocuments.net/reader035/viewer/2022062408/56649ef05503460f94c01252/html5/thumbnails/10.jpg)
Software Tools:
Demo: Networked Data Access, Processing and Fusion
AeroCOM – VIEWS SO4
• http://webapps.datafed.net/dvoy_services/datafed.aspx?page=KYU_VIEWS• http://webapps.datafed.net/dvoy_services/datafed.aspx?page=KYU_V• http://webapps.datafed.net/dvoy_services/datafed.aspx?page=VIEWS_KYU
Networking
![Page 11: Work Group Meeting on HTAP-Relevant IT Techniques, Tools and Philosophies: DataFed Experience and Perspectives Rudolf B. Husar CAPITA, Washington University,](https://reader035.fdocuments.net/reader035/viewer/2022062408/56649ef05503460f94c01252/html5/thumbnails/11.jpg)
VisualizationGoal: 4D, User-selectable layers
Image below completed in 1998
Satellite Data Layers:
Land Reflectance (SeaWiFS)
Fire Pixels (ASTR Night)
High Cloud (GOES, Meteosat, GMS2)
Aerosol AOT (AVHRR)
![Page 12: Work Group Meeting on HTAP-Relevant IT Techniques, Tools and Philosophies: DataFed Experience and Perspectives Rudolf B. Husar CAPITA, Washington University,](https://reader035.fdocuments.net/reader035/viewer/2022062408/56649ef05503460f94c01252/html5/thumbnails/12.jpg)
4D Dynamic Visualization Demonstrate interactions, allow exploration
‘Google Earth’ for Earth Science is now possible
![Page 13: Work Group Meeting on HTAP-Relevant IT Techniques, Tools and Philosophies: DataFed Experience and Perspectives Rudolf B. Husar CAPITA, Washington University,](https://reader035.fdocuments.net/reader035/viewer/2022062408/56649ef05503460f94c01252/html5/thumbnails/13.jpg)
Partners
• NASA
• NOAA
• EPA
• USGS
• DOE
• NSF
• Industry…
Earth Science Information Partners
Air Quality Cluster
1. Serve as facilitator to Earth Science information community. 2. Promote efficient flow of ES data from collection to end-use. 3. Improve quality and usability of ES data and info systems 4. Expand the use of Earth science information
Participant Involvement
![Page 14: Work Group Meeting on HTAP-Relevant IT Techniques, Tools and Philosophies: DataFed Experience and Perspectives Rudolf B. Husar CAPITA, Washington University,](https://reader035.fdocuments.net/reader035/viewer/2022062408/56649ef05503460f94c01252/html5/thumbnails/14.jpg)
Data Dissemination & Use – Service Based
• Provide Catalog Services (Publish, Find data)• Allow Data Access (‘Bind’) – Use International Standards • Add tools for exploration and analysis
Near Real Time Data IntegrationDelayed Data Integration
Surface Air Quality AIRNOW O3, PM25 ASOS_STI Visibility, 300 sitesMETAR Visibility, 1200 sitesVIEWS_OL 40+ Aerosol Parameters
SatelliteMODIS_AOT AOT, Idea ProjectGASP Reflectance, AOTTOMS Absorption Indx, Refl.SEAW_US Reflectance, AOT
Model OutputNAAPS Dust, Smoke, Sulfate, AOTWRF Sulfate
Fire DataHMS_Fire Fire PixelsMODIS_Fire Fire Pixels
Surface MeteorologyRADAR NEXTRADSURF_MET Temp, Dewp, Humidity…SURF_WIND Wind vectorsATAD Trajectory, VIEWS locs.
![Page 15: Work Group Meeting on HTAP-Relevant IT Techniques, Tools and Philosophies: DataFed Experience and Perspectives Rudolf B. Husar CAPITA, Washington University,](https://reader035.fdocuments.net/reader035/viewer/2022062408/56649ef05503460f94c01252/html5/thumbnails/15.jpg)
WCS - Interoperable Data Access Service
netCDF – Machine independent encoding
ncML – XML data description
CF – Naming, structure convention
OGC Web Coverage Service - Interoperable Data Access
Query Data Syntax + Semantics
Coverage (parameter)
BBOX
Time Range
netCDF Example Profile
• SERVICE=wcs ‘service• REQUEST=GetCoverage,VER=1.0 ‘service method• COVERAGE=AIRNOW.pmfine ‘what• CRS=EPSG:4326 ‘projection• BBOX=-125,22,-61,51,0,0 ‘where• TIME=2005-06-6T15:00:00Z ‘when• FORMAT=netCDF ‘return format
![Page 16: Work Group Meeting on HTAP-Relevant IT Techniques, Tools and Philosophies: DataFed Experience and Perspectives Rudolf B. Husar CAPITA, Washington University,](https://reader035.fdocuments.net/reader035/viewer/2022062408/56649ef05503460f94c01252/html5/thumbnails/16.jpg)
GALEON Interoperability ExperimentGeo-interface for Atmosphere, Land, Earth, and Ocean netCDF
Unify Earth Science & GIS Data Flows
Strong European Participation
IT – S. Nativi, L. Bigagli
UK – Andrew Wolf
DE – Peter Bauman)
B. Domenico
B. Domenico
GALEON UNIDATA
U Florence/CNR-IMMA WCS Server
![Page 17: Work Group Meeting on HTAP-Relevant IT Techniques, Tools and Philosophies: DataFed Experience and Perspectives Rudolf B. Husar CAPITA, Washington University,](https://reader035.fdocuments.net/reader035/viewer/2022062408/56649ef05503460f94c01252/html5/thumbnails/17.jpg)
OGC WCS Demonstration: THREDDS_GFS 4Dim Dataset
Lat/Lon Box Elev Range Time RangeMap: BBOX=-180,-90,180,90, 1350,1350& TIME=2005-12-06/2005-12-06/PT3HTime: BBOX=-34,49.05,-34,49.05, 1350,1350& TIME=2005-12-05/2005-12-08/PT3HElev: BBOX=-34,49.05,-34,49.05, 0,18000 & TIME=2005-12-06/2005-12-06/PT3H
The form of the WCS query is the same for all slices through the data cube (views) The only difference in the views is the thickness of the slices in each dimension Return grid is in multiple formats (NetCDF, CSV, GML, PNG, … )
Map View Services WCS Query
Time View Services WCS Query
Elevation View Services WCS Query
![Page 18: Work Group Meeting on HTAP-Relevant IT Techniques, Tools and Philosophies: DataFed Experience and Perspectives Rudolf B. Husar CAPITA, Washington University,](https://reader035.fdocuments.net/reader035/viewer/2022062408/56649ef05503460f94c01252/html5/thumbnails/18.jpg)
Summary
• Current systems data & model analysis are heterogeneous
• Standardization is a key need for agile IT systems
• Non-intrusive mediators can achieve virtual standardization
• Technologies are currently available for dynamic NETWORKING
We eager to share our networked data, tools and methods
![Page 19: Work Group Meeting on HTAP-Relevant IT Techniques, Tools and Philosophies: DataFed Experience and Perspectives Rudolf B. Husar CAPITA, Washington University,](https://reader035.fdocuments.net/reader035/viewer/2022062408/56649ef05503460f94c01252/html5/thumbnails/19.jpg)
OGC WCS Demonstration: Grid, Image, Station Data Types
Coverage=THEEDDS.T& BBOX=-126,24,-65,52,0,0 &TIME=2002-07-07/2002-07-07&FORMAT=NetCDFCoverage=SURF.Bext& BBOX=-126,24,-65,52,0,0 &TIME=2002-07-07/2002-07-07&FORMAT=NetCDF-tableCoverage=SEAW.Refl& BBOX=-126,24,-65,52,0,0 &TIME=2002-07-07/2002-07-07&FORMAT=GeoTIFFCOVERAGE=sst& BBOX=-126,24,-65,52,0,0 &TIME=2001-01-01,2001-01-01&FORMAT=NetCDF
UNIDATA – THREDDS/GALEON WCS
DataFed GALEON WCS
U Florence, It GALEON WCS
DataFed GALEON WCS
Grid
Grid
Image
Station
Services WCS Query
Services WCS Query
Services
WCS Query
Services
WCS Query
![Page 20: Work Group Meeting on HTAP-Relevant IT Techniques, Tools and Philosophies: DataFed Experience and Perspectives Rudolf B. Husar CAPITA, Washington University,](https://reader035.fdocuments.net/reader035/viewer/2022062408/56649ef05503460f94c01252/html5/thumbnails/20.jpg)
Single Data Model for All AQ Data
Most Views are slices through a cube of data organized by lat, lon, altitude, and time (X,Y,Z,T)
Multidimensional Data Cube
![Page 21: Work Group Meeting on HTAP-Relevant IT Techniques, Tools and Philosophies: DataFed Experience and Perspectives Rudolf B. Husar CAPITA, Washington University,](https://reader035.fdocuments.net/reader035/viewer/2022062408/56649ef05503460f94c01252/html5/thumbnails/21.jpg)
OGC Web Coverage Service (WCS) Specification
• HTTP GET/POST based interfaces• Services have XML service descriptions (“Capabilities”, “Description”)• Filter parameters allow selection of subsets of source data• Output formats advertised by each service instance
OGC WCS getCoverage SchemaSuitable for wrapping with SOAP envelope, WSDL access, loose coupling
WCS is for "coverages" – information representing space-time-varying phenomena
WCS describes, requests and delivers coverages in spatio-temporal domain
WCS version 1.1 is limited to grids/"simple” coverages with homogeneous range sets
![Page 22: Work Group Meeting on HTAP-Relevant IT Techniques, Tools and Philosophies: DataFed Experience and Perspectives Rudolf B. Husar CAPITA, Washington University,](https://reader035.fdocuments.net/reader035/viewer/2022062408/56649ef05503460f94c01252/html5/thumbnails/22.jpg)
AQ Monitoring Network Data Storage and Delivery through OGC Protocols
Relational Data Model
Star Schema
WMS
WCS SOS
Sen
sorMLW
FS
Observations
Sta
tion
In
fo.
Para
m/S
en
sor/M
eth
od
Data View ServicesW
MS
Stations Par-Meth
Observations
SO
S
WCS Data Access
![Page 23: Work Group Meeting on HTAP-Relevant IT Techniques, Tools and Philosophies: DataFed Experience and Perspectives Rudolf B. Husar CAPITA, Washington University,](https://reader035.fdocuments.net/reader035/viewer/2022062408/56649ef05503460f94c01252/html5/thumbnails/23.jpg)
Technology Support for ‘Integrated Solutions’
Air Quality Information System
![Page 24: Work Group Meeting on HTAP-Relevant IT Techniques, Tools and Philosophies: DataFed Experience and Perspectives Rudolf B. Husar CAPITA, Washington University,](https://reader035.fdocuments.net/reader035/viewer/2022062408/56649ef05503460f94c01252/html5/thumbnails/24.jpg)
through
Data Access through Adapters
DataFed SOAP,HTTP Get
OGC WCS HTTP Get, Post
OGC WMS HTTP Get
Station-Point SQL Server,
Files…
Sequence Image, file
nDim Grid OpenDAP NetCDF,
…
Other Traject., Event, Pic
Sources Diverse formats
Many data models
Data Wrapper Data into geo-cubes
Queries to views
Virtual Data Cube Global geo-cube data model Makes queries data-neutral
Others? e.g.
OpenDAP
Output Protocol dependent
User specified
GeoTable CSV,XLS,GML
GeoGrid GML,NetCDF..
GeoImage GeoTIFF,
PNG..
Other MS Dataset..
Query Adapter Maps query to protocol
User selects protocols