Post on 16-Dec-2015
The What, Why and How of the OGC/WMO Hydrology Domain Working Group
or
How Too Many Dave’s is Never Enough
Dr David Lemon (CSIRO)OGC/WMO Hydrology Domain Working Group Workshop JRC Ispra March 2010
Helping the World to CommunicateGeographically
However ...
• In many places we can’t asses – How much we have– Where it is– Who owns it– What it is fit for
• We can’t forecast– How much we will have– Where it will be
• We certainly can’t share information in a useful timeframe– Expect by private contract
Helping the World to CommunicateGeographically
Increased Irrigation
Fire Recovery Expanding Plantations
Changing Climate
Increasing groundwater extraction
Increasing interception
Growing Urban Demand
Environmental Flows
Global water challenges
Vertessy 2007
Helping the World to CommunicateGeographically
To address these
• We will struggle to respond to these unless we know what’s happening.
• Knowing what’s happening will be difficult without sharing and exploiting interdisciplinary data sources, models, and processes
• We need interoperability of knowledge and tools, not just the data – This is complicated by:
• the volumes of data being generated• differing semantics• institutional barriers
Helping the World to CommunicateGeographically
Standards
• For some they have a bad reputation– For Developers
• Take time to develop• Must be useful to have impact• Need to identify and travel the path to adoption
– For Implementers• Can be more difficult to implement than custom solutions• Need to identify where they are needed• Need to identity which ones to use – so many to choose from.
• These can be overcome and the benefits far outweigh the pain
Helping the World to CommunicateGeographically
Standards
• Increase Choice• Reduce Cost• Encourage market competition
– Choose based on functionality desired
– Avoid “lock in” to a proprietary architecture
• Decisions to share information and services become policy decisions
• Needed for technologies to survive the ‘Valley of Death’
Helping the World to CommunicateGeographically
Towards the Hydrology DWG
• March 2007– CUAHSI submit WaterML as OGC Discussion paper
• September 2007 (Canberra, Australia)– Meeting coordinated by NR Canada and CSIRO– 34 participants, 18 organisations, (Oceania, Nth America, Europe)– Looked at
• Current international activities• Methodologies, patterns and technologies• Futures
– Agreed time was right to move forward:• IWDIF is born.
Helping the World to CommunicateGeographically
Towards the Hydrology DWG
• Early 2008– Much discussion between CSIRO, CUAHSI, OGC and WMO
• September 2008 (Atlanta)– First ad-hoc– Presentations from CUAHSI and CSIRO– Discussion about value of group– Agreed to develop charter (CSIRO lead).
• December 2008 (Valencia)– 3 Presentations (CSIRO, GRDC)
Helping the World to CommunicateGeographically
Towards the Hydrology DWG
• Early 2009– CUAHSI and CSIRO starting working on WaterML2.0
• March 2009 (Athens)– 6 presentations (EEA, INSPIRE, WMO, GRDC, CUAHSI, CSIRO– TC agrees to Charter
The OGC\WMO Hydrology Domain Working Group is formed
Helping the World to CommunicateGeographically
Since Formation
• June 2009 (Boston)– First official meeting– Groundwater IE proposed
• September 2009 (Darmstadt)– Presentation of Water Data Exchange Formats Harmonisation
Report
• December 2009 (Mountainview)– Surface Water IE proposed
Helping the World to CommunicateGeographically
Since Formation
• March 2010 (Ispra)– 24 participants, 19 organisations, (UN, Oceania, Nth America,
Europe)– Vendors, Government, Research– Looking at
• Methodologies, patterns and technologies• Current activities• Futures
Helping the World to CommunicateGeographically
The OGC
• A non-profit, international, voluntary consensus standards organisation developing standards for geospatial and location based services
• Focussed on technology standards– Eg. Web Map Service, Web Feature Service, Observations and
Measurements, KML, SensorML, …
• 350+ members including: GIS vendors (ESRI), large software vendors (Oracle, Microsoft), government agencies (USGS, NASA, US DOD), systems integrators, …
• Key relationships with other standards groups (ISO, W3C, WMO)
• Work undertaken by interested participants through Domain and Standards Working Groups
Helping the World to CommunicateGeographically
WMO
‘... the UN system's authoritative voice on the state and behaviour of the Earth's atmosphere, its interaction with the
oceans, the climate it produces and the resulting distribution of water resources’
‘... facilitates the free and unrestricted exchange of data and information, products and services in real- or
near-real time ...’
Helping the World to CommunicateGeographically
Hydrology Domain Working Group
• A joint working group of the OGC and WMO constituted as an OGC Domain Working Group.
• Brings together interested parties to develop and promote the technology for greatly improving the way in which water information is described and shared.
• Co-chaired by representatives nominated by the OGC TC and the World Meteorological Organisation’s (WMO) Commission for Hydrology (CHy).
• Current Co-Chairs: Ilya Zaslavsky (SDSC), Ulrich Looser (GRDC) and David Lemon (CSIRO)
• > 50 Participants, > 30 Organisations
Helping the World to CommunicateGeographically
Charter
• To provide a venue and mechanism for seeking technical and institutional solutions to the challenge of describing and exchanging data describing the state and location of water resources, both above and below the ground surface.
• The path to adoption will be through OGC papers and standards, advanced to ISO where appropriate, and also through the WMO’s CHy and WIS activities.
• The Hydrology DWG will provide a means of developing candidate standards for adoption by CHy as appropriate.
Helping the World to CommunicateGeographically
© 2009 Open Geospatial Consortium, Inc.
19
Expected Outcomes• An agreed feature model (ie. what are the features of the
hydrosphere (from an information perspective) and how are they related.)
• An agreed observation model. • Agreed vocabularies, endorsed by the community, and by
WMO in particular. Agreeing on semantics is a long process, but we should be able to recommend some vocabularies
• Also: services carrying the above
Helping the World to CommunicateGeographically
Activities
• Analysis and harmonization of existing commonly used protocols and standards in the domain
• Interoperability Experiments (IEs) focused on selected sub-domains of water data (e.g. surface water, groundwater, water quality, water use, hydrologic forecasts, real time data)
• Participation in GEO Architecture Implementation Pilots (AIPs), focused on broader scenarios within the entire domain and across domains
• Submission of discussion papers and best practice papers (the latter following IE and AIP experiences)
• Collaboration with other DWGs, in particular ESS (on modeling), MetOcean (on handling gridded data), SWE (on real time data management)
© 2009 Open Geospatial Consortium, Inc.
20
Helping the World to CommunicateGeographically
© 2009 Open Geospatial Consortium, Inc.
21
Timeline• September 2009: Discussion paper on harmonization of water data exchange
formats (1st version) ; IE1 announced
• December 2009: Update on the harmonization report; Proposed WaterML 2.0 schema announced; IE1 officially launched, IE 2 announced
• June 2010: WaterML 2.0 submitted as a discussion paper ; IE2 launched (to be reported 3/11); Ontology task force started (with GRDC, CSIRO, CUAHSI, others) ; IE1 preliminary report;
• December 2010: IE1 final report, a best practice paper based on the results
• June 2011: IE2 final report, and a paper
Helping the World to CommunicateGeographically
Groundwater Interoperability Experiment(Boyan Brodaric – GCS, Nate Booth – USGS)
Helping the World to CommunicateGeographically
Groundwater is an increasingly significant source of water
Ground/water feature and observation data - distributed: many data providers (~1000 in Canada alone)
- heterogeneous: structure and content (e.g. level, flow, quality)
- massive instances: millions of wells (features/sensors) & time series (observ.)
- massive schema: thousands of properties (e.g. for quality)
- few standard services: very few OGC gr/water services
• No standard way to share groundwater information
Context
Helping the World to CommunicateGeographically
Groundwater feature and observation interoperability:
1. Advance design of WaterML 2 schema• harmonize existing schemas [WaterML (USA), GroundwaterML
(CAN), WTDF (AU), WQX (USA), OGC O&M, SensorML, …]
2. Advance fit of OGC services with groundwater data
• SOS, WFS, WMS, CSW
3. Advance groundwater data exchange between US & Canada• in cross-border Great Lakes area (Lake Superior) for water wells
and levels
Objectives
Helping the World to CommunicateGeographically
1. Advance design of WaterML 2 schema• merge or link existing schemas (WaterML, O&M, GWML)?• when to restrict: what is special about groundwater observations?• when not to restrict: does it need to be extensible?
2. Advance fit of OGC services with gr/water data
• SOS 1: not designed for massive networks (i.e. GetCapability lists each sensor)
• CSW: include individual sensors?
3. Advance GW data exchange between US & CAN• what is a groundwater observation? • 1/n sensors, 1/n time points, 1/n time series, 1/n properties?
Challenges
Helping the World to CommunicateGeographically
View US & Canadian well/sensor locations
- WMS (GetMap)
Query on one well/sensor and display data
- WMS (GetFeatureInfo)
- well: WFS (getFeature via BBOX) or
- level: SOS (GetObservation via BBOX)
- return GWML or WaterML2.0 and display as HTML
Download cross-border wells and levels
- Zoom to area of interest (find BBOX)
- well: WFS (GetFeature via BBOX and thematic filter) or
- level: SOS (GetObservation via BBOX and time filter)
- Export as GWML, WaterML2.0, Excel, or KML
Use Cases
Helping the World to CommunicateGeographically
Dec 2009: Milestone 1
- preliminary WaterML 2 schema, example instances, documents (CSIRO)
- preliminary services: SOS & WFS (GSC), SOS (USGS), CSW (SDSC)
- preliminary demo of use-case
June 2010: Milestone 2
- intermediate results at OGC TC
Dec 2010: Final Milestone
- final results
- technology demo
- change requests re: OGC specs
- wrap and report at OGC TC and AGU 2010
Dates, Workplan Outputs
Helping the World to CommunicateGeographically
“At least one of the people out there has a smarter idea about what to do with your content than you do.”
- James Boyle
Helping the World to CommunicateGeographically
The need for standards
• Sharing of hydrological data is somewhat ad-hoc, based largely on ad-hoc formats, CSV files and FTP
• Leads to very inefficient (or non-existent) transfer and sharing of data
• Sharing requires a lot of transformation effort on the behalf of the data requestor– Parse different file formats– Understand the concepts– Metadata is often limited or non-existent, requiring follow up
conversations
: JAN RESERVOIR DATA 900 AM CST THU MAR 28 1985.B JAN 0328 C DH06/HP/HW/PP/QT: STN ID POOL TAILW PCPN DISCH K CFSJSNM6 295.71 / 259.51 / 0.50 / 4.300 : BARNETTDNNM6 410.1 / 330.3 / .31 / : BAY SPRINGS L&DABEM6 190.1 / 163.6 / .17 / : ABERDEEN L&DCBUM6 163.5 / 137.1 / .39 / : COLUMBUS L&DARKM6 210.1 / / .07 / .650 : ARKABUTLASRDM6 242.8 / / .10 / .250 : SARDISENDM6 235.1 / / .37 / .100 : ENIDGRNM6 197.6 / / .70 / .100 : GRENADABMDA4 / / 0.00 / 6.393 : BLAKELY, ARDGDA4 / / .10 / 2.634 : DEGRAY, ARNARA4 / / 0.00 / 1.244 : NARROWS, AR.END
Helping the World to CommunicateGeographically
A common situation
Need flow data!
Hmm maybe Don can help…
*RING RING*
Hi Don, I need some upper Derwent flow readings for my geochemical model. Any ideas?
Don
Hmm, I’ve got one site. I’ll send it through…
10 minutes…
To: Jack01/02/09, 3.2, 3, 101/02/09, 3.1, 3, 1
10 minutes…*RING RING*
Ok. Got the data. Where is the site located?
Oh, it’s at laughing jack bridge.
Coordinates?
Ummm. (papers shuffle)147.123 -41.588
What reference system??I think it’s GDA94
Ok. What sensor is used?It’s calculated from the stream gauge reading using a rating curve..
Oh…how accurate is that?
Umm......
DON?
Hydro Jack
*CLICK*
Helping the World to CommunicateGeographically
Hydrological observations
• Measurements of – Water level, – Water temperature, – Discharge, – Sediment,– Water quality – ...
• Different sampling techniques (in-situ, laboratory etc.)
• Why do people want to share hydrological observations?– Provide data to national systems (AWRIS, CUAHSI)– Share data between agencies
• Hydro company to local water authority • Researchers (learning hydrology, modelling)
– Other domains • Climatology• Oceanography• Meteorology
Helping the World to CommunicateGeographically
The need for harmonisation
• A number of initiatives exist to develop standards for water observation data:– WaterML1.0, 1.1– WDTF, WOML– XHydro– UK Environmental Agencies time series transfer format– SANDRE– ...;
• A lot of commonalities exist between the standards– Partial duplication of effort– Inconsistencies across the standards– Hard to re-use tools
• Potential for a harmonised, re-usable information model– Define common semantics of concepts– See Document 09-124 (Harmonising standards for water observation data)
Helping the World to CommunicateGeographically
The need for a common model
Sharing can be easy; understanding is difficult.
Helping the World to CommunicateGeographically
A proposed approach
• Identify common concepts within existing hydrological data standards
• Identify use cases/patterns/structures of hydrological observation data
• Identify methodologies that can be leveraged to assist the development of a harmonised standard
• Re-use existing definitions of concepts where appropriate – don’t re-invent the wheel
Helping the World to CommunicateGeographically
Re-use existing standards
• Sensor Web Enablement– SWE Common– Sensor Observation Service– Observations & Measurements
• GML & ISO– Primitive types– Spatial
• Allows us to focus on the domain concepts rather than types that are fairly common across domains (units, spatial types etc.)
Helping the World to CommunicateGeographically
OGC Observations & Measurements
• Provides a definition of common concepts and relationships of the observation process
• A baseline from which we can develop a specific model for hydrological observations
• Appears to be the emerging best practice, used by:– Integrated Ocean Observing System (IOOS)– Climate Science Markup Language (CSML)– Global Runoff Data Centre’s (GRDC) meta data profile – Submitted to ISO
Helping the World to CommunicateGeographically
Model-driven approaches
• A model-driven approach to information modeling allows specification of model in UML and various encodings can be generated (e.g. GML XML)
• Also an emerging best practice in ISO211 for developing specifications for spatial data (ISO19101, ISO19109)
• Tools exist that facilitate the development process:– FullMoon for encoding UML into GML-compliant XML– HollowWorld to allow re-use of exisiting standards in UML– SolidGround to manage model
Helping the World to CommunicateGeographically
UK Environmental Agency – TS format
Sampling features
Time series
Helping the World to CommunicateGeographically
Sampling Features
Result type (time series)
Observed property
Procedure
Helping the World to CommunicateGeographically
Benefits of a harmonised model
• Access to a wider range of developed tools, libraries and applications – reduced duplication of effort
• Adapters for existing systems can be shared
• Access to documented and explicit definitions of concepts
• Helps pave the way for increased interoperability of water information systems
Helping the World to CommunicateGeographically
Going forward - issues
• How well do existing site/location/station/measuring point concepts fit into the model?
• How do we provide common grouping mechanisms that satisfy existing and future discovery requirements?
• How do we describe our sampling regimes?• How do we encode observation results (time series etc.)?• Managing dependencies between derived time series?• Define a core set of use cases to address
Helping the World to CommunicateGeographically
Development of a harmonised model will bring benefits to a large number of parties and will
promote improved sharing of data and meta data, by both humans and computers.
Helping the World to CommunicateGeographically
The future?
Hydro Jack
Laughing Jack Bridge 147.123 -41.588 (AGD66)Upper Derwent RiverObserved properties: River level, River Flow, Rainfall
River Flow- -Measured by Sontek Acoustic Doppler Sensor (ID232)- -Accuracy: +/- 0.02 - -Installed: 01/04/2012
Need flow data!
Helping the World to CommunicateGeographically
Objectives of IE
• Extend and complement the work already underway in GW IE– Further develop WaterML2.0 for surface water
• Test compatibility of WaterML 2.0 with existing participants infrastructure.
• Advance exchange of surface water data between Germany and France in the cross-border area of the Rhine/Rhin river.
• Advance exchange of surface water data between Mexico and USA in the cross-border area of xx.
• The data will be served by all participants using WaterML 2.0 and OGC services (SOS, WFS, WMS).
Copyright © 2008, Open Geospatial Consortium, Inc., All
Rights Reserved.
Helping the World to CommunicateGeographically
Participants (so far)Organisation Ideas for IE Contributions
IOW-Sandre Cross border quality/quantity data exchange
Data and services
52° North Service implementations
Federal Water Catchment oriented approach/ cross border data exchange
Data and services
CUAHSI Cross US/Mexico border data exchange
TBA
Kisters Check feasibility/suitability of service definitions
Consume and provide services
CSIRO Development of WaterML2.0
IMTA (Mexican Institute of Water Technology)
Data and services
Helping the World to CommunicateGeographically
Current Practices in Hydrology
• The team hydrologist’s initial specification
Helping the World to CommunicateGeographically
A use case: current practices
• Create calibration tasks
Helping the World to CommunicateGeographically
Issues with current practices
• Heavy reliance on individuals and manual steps for the ‘process’ to sustain itself – Lack of resilience, critical sections– Poor use of valuable skills (copy-paste in Excel); increased risk of
non-reproducible errors
• Dispersed information: emails, post-its, script logs, sharepoint, ‘wiki sites’, personal documents, brains, etc.
• Heterogeneity of toolsets and capabilities; islands of specialisation with manual transmission in-between– XML documents– R, Matlab, Python Scripts; Excel
• Idiosyncratic access arrangements• Basic traceability requires “off-line” documentation.
Helping the World to CommunicateGeographically
Too many Dave’s?
David Maidment (CUAHSI)
David Valentine (CUAHSI)
David Tarboton (CUAHSI)
David Kinny (BOM)
David Schell (OGC)
David Arctur (OGC)
David Thomas (WMO)
David Lemon (CSIRO)
Peter Taylor (CSIRO)
Peter Fitch (CSIRO)
Peter Gijsbers (Deltares)
Pierre Lagarde (BRGM)
Pierre Kerherve (WMO)
Helping the World to CommunicateGeographically
Public mailing list https://lists.opengeospatial.org/mailman/listinfo/hydro.dwg
Twiki http://external.opengis.org/twiki_public/bin/view/HydrologyDWG
Thank you ...