Integration through Data Management The “SEACOOS experience” Accomplishments and Obstacles...
-
date post
19-Dec-2015 -
Category
Documents
-
view
222 -
download
0
Transcript of Integration through Data Management The “SEACOOS experience” Accomplishments and Obstacles...
Integration through Data Integration through Data ManagementManagement
The “SEACOOS experience”Accomplishments and Obstacles
SEACOOS approach to data integration Accomplishments and contributions to
IOOS interoperability Some “lessons learned” Coordination with nascent Regional
Association
SEACOOS was initiated in 2002 with ONR funding
to develop a coastal ocean information system
for FL, GA, SC and NC.
SEACOOS Over-Arching Goal: To significantly increase the quantity and quality of environmental information from the coastal ocean of the SE U.S. and facilitate its use in a wide range of societal, scientific, and educational applications.
Observational Platforms Contributed by SEACOOSPartners
Observational Platforms Contributed by SEACOOSPartners
Real-Time Observations: SE Coastal Ocean
SEACOOS Modeling Coordinating model
simulations forthe entire region
SEACOOS Information SEACOOS Information ManagementManagement
Data Management Coordinating Committee (DMCC)
CoordinationMadilyn Fletcher (USC), Dwayne Porter (USC),
P.I. representatives Ed Kearns (U Miami), Mark Luther (USF), Harvey Seim (UNC),
Nick Shay (U Miami), Jim Nelson (SkIO)People who get it done
Charlton Purvis (USC), Jeremy Cothran (USC), Vembu Subramanian (USF),
Jeff Donovan (USF), Sara Haines (UNC), Jesse Cleary (UNC), Liz Williams (U Miami), Tom Cook (U Miami)
Web site documentation /portalChris Calloway (UNC), Claire Eager (UNC)
SEACOOS achievements in SEACOOS achievements in regional Data Managementregional Data Management
“Data commons” -- Protocols for data providers to make their information available. A set of standards for use with a given file format
Methods for data aggregation and display Methods for data aggregation and display
TogetherTogether – a demonstration of how an – a demonstration of how an RCOOS can function to aggregate and RCOOS can function to aggregate and display informationdisplay information
Data “commons”Data “commons”
netCDF adopted as the common data language(“SEACOOS CDL”, currently version 2)
-- well supported (UCAR, Unidata, Boulder, CO)
-- supported under DODS/OPeNDAP-- interfaces to programming and analysis packages (e.g., Perl, Matlab)-- a flexible system for a variety of platforms/sensor types
(scalar time series, vector time series, vector profiler time series,scalar gridded maps, vector unstructured maps)
Data “commons”Data “commons”(cont.)(cont.)
“Data Dictionary”-- table that registers known standards
with each other-- provides an English language description-- “down-the-road” advantages:
options for data providers; flexibility in presentation and interpretation of information(i.e., if format is registered, the providers’ data can be represented)
Data Management:Data Management:the Nuts & Boltsthe Nuts & Bolts
Data is aggregated and storedData is aggregated and stored Data is normalizedData is normalized Data is visualizedData is visualized Data is disseminatedData is disseminated
Data Aggregation & Data Aggregation & StorageStorage
Aggregation format flavorsAggregation format flavors netCDF (in situ & RS data; model netCDF (in situ & RS data; model
output)output) PNG (RS imagery)PNG (RS imagery)
StorageStorage Relational database (in situ, model Relational database (in situ, model
output, some RS)output, some RS) Files (RS imagery)Files (RS imagery)
TechnologyTechnology PerlPerl PostgreSQLPostgreSQL
& PostGIS& PostGIS
SEACOOS Data Train A central aggregation site as opposed to distributed network (e.g.,
LAS) Link to software community developing GIS-type applications Powerful visualization tool without limits to numbers of layers
NormalizationNormalization Reporting time Reporting time
variesvaries In-situ dataIn-situ data
E.g. daily, hourly,E.g. daily, hourly,half-hourly, every 10 half-hourly, every 10 minutesminutes
Remotely-sensed dataRemotely-sensed data E.g. twice dailyE.g. twice daily
Reporting area Reporting area variesvaries Remotely-sensedRemotely-sensed
data passesdata passes
Round-the-clock Round-the-clock updating is updating is resource resource intensiveintensive Balance the Balance the
server loadserver load
Visualization : Example 1Visualization : Example 1 Production siteProduction site
http://www.seacoos.orghttp://www.seacoos.org
Visualization : Example 2Visualization : Example 2 Development siteDevelopment site
http://nautilus.baruch.sc.edu/rshttp://nautilus.baruch.sc.edu/rs
Main enginesMain engines PHPPHP
PHP-MapScriptPHP-MapScript PerlPerl MapServerMapServer
Visualization Fun : Visualization Fun : Graphs & AnimationsGraphs & Animations
Ad-hoc time-series Ad-hoc time-series graphsgraphs
Ad-hoc animationsAd-hoc animations
Hurricane/Tropical Storm Jeanne25 - 27 September, 2004
Merged wind observations and shore radar
DisseminationDissemination
OPeNDAP (DODS) data accessOPeNDAP (DODS) data access OGC-friendly: WMS, WFSOGC-friendly: WMS, WFS
pick a layer, any layer pick a layer, any layer http://nautilus.baruch.sc.edu/http://nautilus.baruch.sc.edu/seacoos_misc/show_sea_coos_obs_time_ranges.phpseacoos_misc/show_sea_coos_obs_time_ranges.php
Dissemination : example Dissemination : example 11
NC OneMapNC OneMapViewer Viewer http://gisdata.usgs.net/http://gisdata.usgs.net/website/NC_OneMap/website/NC_OneMap/viewer.aspviewer.asp
Dissemination : example Dissemination : example 22
IntegratedIntegratedOceanOceanObservingObservingSystemSystemhurricanehurricanedemo demo http://www.openioos.orghttp://www.openioos.org
Some Lessons :Some Lessons :Interoperability Interoperability DemonstrationsDemonstrations Communication is key : an open source attitude
Mailing lists, bulletin boards, and wiki’s proved invaluable across internal SEACOOS projects and external projects, e.g. Interoperability Demonstrations
Successfully accomplished (capabilities were demonstrated), but had “fire-drill” aspects from perspective of technical personnel Better coordinated in Summer 2004 than previously, a more
mature effort Better adaptation of OGC WMS and WFS technologies Additional layers A better product “under the hood”
However, the push to get out products in the short-term was done at the expense of thoroughness
Process to add new products, identify sources seen as somewhat haphazard
Not the way to build an operational system
Other Obstacles / Issues QA/QC of real-time data and archives
Needs further development; compliant with national standards
Resource availability and allocation Many personnel have multiple roles Crucial expertise in a small number of people Time for documentation Need for redundancy in comm./processing streams
Data from national providers not always in a readily accessible form (often requires “screen
scrapes”)
Engaging users / getting feedback “In-reach” within the regional program as well as with
external users Track users and map results
Other SEACOOS regional DM activities
Coordination with NDBC, NWS e.g., data push from COMPS, SABSOON Stations part of NDBC network
Coordination with developing Regional Association (SECOORA) Sensor/equipment inventory Data dictionary (documentation)
Development of metadata tool/metadata system for the RCOOS (“MetaDoor” & documentation, USC)
Coordination with State Agencies
Equipment Inventory
Full Search
Query Returns: DB records
Query Returns: Map
User Interface ComponentsQuick Search – by variable measured
Equipment InventoryAdministrative Interface
1. Institution: Institution Name, Abbreviation, Observing Group URL, Affiliation
2. Contacts: Contact Name, Phone Number, Email Address, Contact level
3. Stations: Institutional Station ID, Station Name, Lat/Long
4. Equipment: Package Description, Manufacturer, Model, Equipment Type, Vertical Position, Power Requirement, Communication Type, Near Real-time Status, Variables Measured, Change/Calibration History, Active Status, Comments
This interface provides database access to equipment managers to add and edit the following information.
Data input and involvement from federal data Data input and involvement from federal data providers for a more accurate picture of the providers for a more accurate picture of the regional observation system (assets, gaps, regional observation system (assets, gaps, performance)performance)
Connecting equipment data as metadata to support Connecting equipment data as metadata to support observations and the QA/QC process. Exploration observations and the QA/QC process. Exploration of SensorML (XML) and MetaDoor (metadata tool of SensorML (XML) and MetaDoor (metadata tool developed at USC).developed at USC).
Improved data intake capability: automated batch Improved data intake capability: automated batch processing for new equipmentprocessing for new equipment
Expanded interactive map functionality: Zoom/PanExpanded interactive map functionality: Zoom/Pan
Equipment inventoryNext Steps
Metadoor : metadata Metadoor : metadata creationcreation
Metadoor : entry context Metadoor : entry context featuresfeatures
- Collapsible forms organized into tabbed sections
- Fields color coded by requirement need
- Help context link for each field- Listbox selections and calendars- Form presents only subfields needed
based on listbox selection
Metadoor : planned Metadoor : planned metadatametadata
support forsupport for marineXMLmarineXML sensorMLsensorML
Platforms and SensorsPlatforms and Sensors Observations and MeasurementsObservations and Measurements OGC Sensor WebOGC Sensor Web
Other metadata harvestersOther metadata harvesters OGC Catalog servicesOGC Catalog services geodata.govgeodata.gov
A friend in the businessA friend in the business Development site: Development site:
http://nautilus.baruch.sc.edu/rshttp://nautilus.baruch.sc.edu/rs Production site: http://www.seacoos.orgProduction site: http://www.seacoos.org Very active listserv dealing with mainly remote-Very active listserv dealing with mainly remote-
sensing issues: [email protected] issues: [email protected] mailto: [email protected]: [email protected] message text: subscribe remotesensingmessage text: subscribe remotesensing
Who am I?Who am I? Charlton Purvis, University of South Carolina, Charlton Purvis, University of South Carolina,
SEACOOSSEACOOS [email protected]@sc.edu Happy to help and share.Happy to help and share.