The What, Why and How of the OGC/WMO Hydrology Domain Working Group or How Too Many Dave’s is...

The What, Why and How of the OGC/WMO Hydrology Domain Working Group

or

How Too Many Dave’s is Never Enough

Dr David Lemon (CSIRO)OGC/WMO Hydrology Domain Working Group Workshop JRC Ispra March 2010

Helping the World to CommunicateGeographically

Water


However ...

• In many places we can’t asses – How much we have– Where it is– Who owns it– What it is fit for

• We can’t forecast– How much we will have– Where it will be

• We certainly can’t share information in a useful timeframe– Expect by private contract


Increased Irrigation

Fire Recovery Expanding Plantations

Changing Climate

Increasing groundwater extraction

Increasing interception

Growing Urban Demand

Environmental Flows

Global water challenges

Vertessy 2007


To address these

• We will struggle to respond to these unless we know what’s happening.

• Knowing what’s happening will be difficult without sharing and exploiting interdisciplinary data sources, models, and processes

• We need interoperability of knowledge and tools, not just the data – This is complicated by:

• the volumes of data being generated• differing semantics• institutional barriers


Standards

• For some they have a bad reputation– For Developers

• Take time to develop• Must be useful to have impact• Need to identify and travel the path to adoption

– For Implementers• Can be more difficult to implement than custom solutions• Need to identify where they are needed• Need to identity which ones to use – so many to choose from.

• These can be overcome and the benefits far outweigh the pain


Standards

• Increase Choice• Reduce Cost• Encourage market competition

– Choose based on functionality desired

– Avoid “lock in” to a proprietary architecture

• Decisions to share information and services become policy decisions

• Needed for technologies to survive the ‘Valley of Death’


The Hydrology Domain Working Group


Towards the Hydrology DWG

• March 2007– CUAHSI submit WaterML as OGC Discussion paper

• September 2007 (Canberra, Australia)– Meeting coordinated by NR Canada and CSIRO– 34 participants, 18 organisations, (Oceania, Nth America, Europe)– Looked at

• Current international activities• Methodologies, patterns and technologies• Futures

– Agreed time was right to move forward:• IWDIF is born.



• Early 2008– Much discussion between CSIRO, CUAHSI, OGC and WMO

• September 2008 (Atlanta)– First ad-hoc– Presentations from CUAHSI and CSIRO– Discussion about value of group– Agreed to develop charter (CSIRO lead).

• December 2008 (Valencia)– 3 Presentations (CSIRO, GRDC)



• Early 2009– CUAHSI and CSIRO starting working on WaterML2.0

• March 2009 (Athens)– 6 presentations (EEA, INSPIRE, WMO, GRDC, CUAHSI, CSIRO– TC agrees to Charter

The OGC\WMO Hydrology Domain Working Group is formed


Since Formation

• June 2009 (Boston)– First official meeting– Groundwater IE proposed

• September 2009 (Darmstadt)– Presentation of Water Data Exchange Formats Harmonisation

Report

• December 2009 (Mountainview)– Surface Water IE proposed


Since Formation

• March 2010 (Ispra)– 24 participants, 19 organisations, (UN, Oceania, Nth America,

Europe)– Vendors, Government, Research– Looking at

• Methodologies, patterns and technologies• Current activities• Futures


The OGC

• A non-profit, international, voluntary consensus standards organisation developing standards for geospatial and location based services

• Focussed on technology standards– Eg. Web Map Service, Web Feature Service, Observations and

Measurements, KML, SensorML, …

• 350+ members including: GIS vendors (ESRI), large software vendors (Oracle, Microsoft), government agencies (USGS, NASA, US DOD), systems integrators, …

• Key relationships with other standards groups (ISO, W3C, WMO)

• Work undertaken by interested participants through Domain and Standards Working Groups


WMO

‘... the UN system's authoritative voice on the state and behaviour of the Earth's atmosphere, its interaction with the

oceans, the climate it produces and the resulting distribution of water resources’

‘... facilitates the free and unrestricted exchange of data and information, products and services in real- or

near-real time ...’


Hydrology Domain Working Group

• A joint working group of the OGC and WMO constituted as an OGC Domain Working Group.

• Brings together interested parties to develop and promote the technology for greatly improving the way in which water information is described and shared.

• Co-chaired by representatives nominated by the OGC TC and the World Meteorological Organisation’s (WMO) Commission for Hydrology (CHy).

• Current Co-Chairs: Ilya Zaslavsky (SDSC), Ulrich Looser (GRDC) and David Lemon (CSIRO)

• > 50 Participants, > 30 Organisations


Charter

• To provide a venue and mechanism for seeking technical and institutional solutions to the challenge of describing and exchanging data describing the state and location of water resources, both above and below the ground surface.

• The path to adoption will be through OGC papers and standards, advanced to ISO where appropriate, and also through the WMO’s CHy and WIS activities.

• The Hydrology DWG will provide a means of developing candidate standards for adoption by CHy as appropriate.


© 2009 Open Geospatial Consortium, Inc.

19

Expected Outcomes• An agreed feature model (ie. what are the features of the

hydrosphere (from an information perspective) and how are they related.)

• An agreed observation model. • Agreed vocabularies, endorsed by the community, and by

WMO in particular. Agreeing on semantics is a long process, but we should be able to recommend some vocabularies

• Also: services carrying the above


Activities

• Analysis and harmonization of existing commonly used protocols and standards in the domain

• Interoperability Experiments (IEs) focused on selected sub-domains of water data (e.g. surface water, groundwater, water quality, water use, hydrologic forecasts, real time data)

• Participation in GEO Architecture Implementation Pilots (AIPs), focused on broader scenarios within the entire domain and across domains

• Submission of discussion papers and best practice papers (the latter following IE and AIP experiences)

• Collaboration with other DWGs, in particular ESS (on modeling), MetOcean (on handling gridded data), SWE (on real time data management)


20



21

Timeline• September 2009: Discussion paper on harmonization of water data exchange

formats (1st version) ; IE1 announced

• December 2009: Update on the harmonization report; Proposed WaterML 2.0 schema announced; IE1 officially launched, IE 2 announced

• June 2010: WaterML 2.0 submitted as a discussion paper ; IE2 launched (to be reported 3/11); Ontology task force started (with GRDC, CSIRO, CUAHSI, others) ; IE1 preliminary report;

• December 2010: IE1 final report, a best practice paper based on the results

• June 2011: IE2 final report, and a paper


Current Activities


Groundwater Interoperability Experiment(Boyan Brodaric – GCS, Nate Booth – USGS)


Groundwater is an increasingly significant source of water

Ground/water feature and observation data - distributed: many data providers (~1000 in Canada alone)

- heterogeneous: structure and content (e.g. level, flow, quality)

- massive instances: millions of wells (features/sensors) & time series (observ.)

- massive schema: thousands of properties (e.g. for quality)

- few standard services: very few OGC gr/water services

• No standard way to share groundwater information

Context


Sensor sites and well locations


Groundwater feature and observation interoperability:

1. Advance design of WaterML 2 schema• harmonize existing schemas [WaterML (USA), GroundwaterML

(CAN), WTDF (AU), WQX (USA), OGC O&M, SensorML, …]

2. Advance fit of OGC services with groundwater data

• SOS, WFS, WMS, CSW

3. Advance groundwater data exchange between US & Canada• in cross-border Great Lakes area (Lake Superior) for water wells

and levels

Objectives


1. Advance design of WaterML 2 schema• merge or link existing schemas (WaterML, O&M, GWML)?• when to restrict: what is special about groundwater observations?• when not to restrict: does it need to be extensible?

2. Advance fit of OGC services with gr/water data

• SOS 1: not designed for massive networks (i.e. GetCapability lists each sensor)

• CSW: include individual sensors?

3. Advance GW data exchange between US & CAN• what is a groundwater observation? • 1/n sensors, 1/n time points, 1/n time series, 1/n properties?

Challenges


View US & Canadian well/sensor locations

- WMS (GetMap)

Query on one well/sensor and display data

- WMS (GetFeatureInfo)

- well: WFS (getFeature via BBOX) or

- level: SOS (GetObservation via BBOX)

- return GWML or WaterML2.0 and display as HTML

Download cross-border wells and levels

- Zoom to area of interest (find BBOX)

- well: WFS (GetFeature via BBOX and thematic filter) or

- level: SOS (GetObservation via BBOX and time filter)

- Export as GWML, WaterML2.0, Excel, or KML

Use Cases


Dec 2009: Milestone 1

- preliminary WaterML 2 schema, example instances, documents (CSIRO)

- preliminary services: SOS & WFS (GSC), SOS (USGS), CSW (SDSC)

- preliminary demo of use-case

June 2010: Milestone 2

- intermediate results at OGC TC

Dec 2010: Final Milestone

- final results

- technology demo

- change requests re: OGC specs

- wrap and report at OGC TC and AGU 2010

Dates, Workplan Outputs


query

view

Milestone 1 Results

download


WaterML 2.0(Peter Taylor - CSIRO)


“At least one of the people out there has a smarter idea about what to do with your content than you do.”

- James Boyle


The need for standards

• Sharing of hydrological data is somewhat ad-hoc, based largely on ad-hoc formats, CSV files and FTP

• Leads to very inefficient (or non-existent) transfer and sharing of data

• Sharing requires a lot of transformation effort on the behalf of the data requestor– Parse different file formats– Understand the concepts– Metadata is often limited or non-existent, requiring follow up

conversations

: JAN RESERVOIR DATA 900 AM CST THU MAR 28 1985.B JAN 0328 C DH06/HP/HW/PP/QT: STN ID POOL TAILW PCPN DISCH K CFSJSNM6 295.71 / 259.51 / 0.50 / 4.300 : BARNETTDNNM6 410.1 / 330.3 / .31 / : BAY SPRINGS L&DABEM6 190.1 / 163.6 / .17 / : ABERDEEN L&DCBUM6 163.5 / 137.1 / .39 / : COLUMBUS L&DARKM6 210.1 / / .07 / .650 : ARKABUTLASRDM6 242.8 / / .10 / .250 : SARDISENDM6 235.1 / / .37 / .100 : ENIDGRNM6 197.6 / / .70 / .100 : GRENADABMDA4 / / 0.00 / 6.393 : BLAKELY, ARDGDA4 / / .10 / 2.634 : DEGRAY, ARNARA4 / / 0.00 / 1.244 : NARROWS, AR.END


A common situation

Need flow data!

Hmm maybe Don can help…

*RING RING*

Hi Don, I need some upper Derwent flow readings for my geochemical model. Any ideas?

Don

Hmm, I’ve got one site. I’ll send it through…

10 minutes…

To: Jack01/02/09, 3.2, 3, 101/02/09, 3.1, 3, 1

10 minutes…*RING RING*

Ok. Got the data. Where is the site located?

Oh, it’s at laughing jack bridge.

Coordinates?

Ummm. (papers shuffle)147.123 -41.588

What reference system??I think it’s GDA94

Ok. What sensor is used?It’s calculated from the stream gauge reading using a rating curve..

Oh…how accurate is that?

Umm......

DON?

Hydro Jack

*CLICK*


Hydrological observations

• Measurements of – Water level, – Water temperature, – Discharge, – Sediment,– Water quality – ...

• Different sampling techniques (in-situ, laboratory etc.)

• Why do people want to share hydrological observations?– Provide data to national systems (AWRIS, CUAHSI)– Share data between agencies

• Hydro company to local water authority • Researchers (learning hydrology, modelling)

– Other domains • Climatology• Oceanography• Meteorology


The need for harmonisation

• A number of initiatives exist to develop standards for water observation data:– WaterML1.0, 1.1– WDTF, WOML– XHydro– UK Environmental Agencies time series transfer format– SANDRE– ...;

• A lot of commonalities exist between the standards– Partial duplication of effort– Inconsistencies across the standards– Hard to re-use tools

• Potential for a harmonised, re-usable information model– Define common semantics of concepts– See Document 09-124 (Harmonising standards for water observation data)


The need for a common model

Sharing can be easy; understanding is difficult.


A proposed approach

• Identify common concepts within existing hydrological data standards

• Identify use cases/patterns/structures of hydrological observation data

• Identify methodologies that can be leveraged to assist the development of a harmonised standard

• Re-use existing definitions of concepts where appropriate – don’t re-invent the wheel


Re-use existing standards

• Sensor Web Enablement– SWE Common– Sensor Observation Service– Observations & Measurements

• GML & ISO– Primitive types– Spatial

• Allows us to focus on the domain concepts rather than types that are fairly common across domains (units, spatial types etc.)


OGC Observations & Measurements

• Provides a definition of common concepts and relationships of the observation process

• A baseline from which we can develop a specific model for hydrological observations

• Appears to be the emerging best practice, used by:– Integrated Ocean Observing System (IOOS)– Climate Science Markup Language (CSML)– Global Runoff Data Centre’s (GRDC) meta data profile – Submitted to ISO


Model-driven approaches

• A model-driven approach to information modeling allows specification of model in UML and various encodings can be generated (e.g. GML XML)

• Also an emerging best practice in ISO211 for developing specifications for spatial data (ISO19101, ISO19109)

• Tools exist that facilitate the development process:– FullMoon for encoding UML into GML-compliant XML– HollowWorld to allow re-use of exisiting standards in UML– SolidGround to manage model


UK Environmental Agency – TS format

Sampling features

Time series


Sampling Features

Result type (time series)

Observed property

Procedure


Benefits of a harmonised model

• Access to a wider range of developed tools, libraries and applications – reduced duplication of effort

• Adapters for existing systems can be shared

• Access to documented and explicit definitions of concepts

• Helps pave the way for increased interoperability of water information systems


Going forward - issues

• How well do existing site/location/station/measuring point concepts fit into the model?

• How do we provide common grouping mechanisms that satisfy existing and future discovery requirements?

• How do we describe our sampling regimes?• How do we encode observation results (time series etc.)?• Managing dependencies between derived time series?• Define a core set of use cases to address


Development of a harmonised model will bring benefits to a large number of parties and will

promote improved sharing of data and meta data, by both humans and computers.


The future?

Hydro Jack

Laughing Jack Bridge 147.123 -41.588 (AGD66)Upper Derwent RiverObserved properties: River level, River Flow, Rainfall

River Flow- -Measured by Sontek Acoustic Doppler Sensor (ID232)- -Accuracy: +/- 0.02 - -Installed: 01/04/2012

Need flow data!


Future Activities?


Surface Water IE(Peter Fitch – CSIRO)


Objectives of IE

• Extend and complement the work already underway in GW IE– Further develop WaterML2.0 for surface water

• Test compatibility of WaterML 2.0 with existing participants infrastructure.

• Advance exchange of surface water data between Germany and France in the cross-border area of the Rhine/Rhin river.

• Advance exchange of surface water data between Mexico and USA in the cross-border area of xx.

• The data will be served by all participants using WaterML 2.0 and OGC services (SOS, WFS, WMS).

Copyright © 2008, Open Geospatial Consortium, Inc., All

Rights Reserved.


Participants (so far)Organisation Ideas for IE Contributions

IOW-Sandre Cross border quality/quantity data exchange

Data and services

52° North Service implementations

Federal Water Catchment oriented approach/ cross border data exchange

Data and services

CUAHSI Cross US/Mexico border data exchange

TBA

Kisters Check feasibility/suitability of service definitions

Consume and provide services

CSIRO Development of WaterML2.0

IMTA (Mexican Institute of Water Technology)

Data and services


Model Integration and Workflows?


Current Practices in Hydrology

• The team hydrologist’s initial specification


A use case: current practices

• Input data checks



• Create calibration tasks



• Calibration runs



• Output analysis


Issues with current practices

• Heavy reliance on individuals and manual steps for the ‘process’ to sustain itself – Lack of resilience, critical sections– Poor use of valuable skills (copy-paste in Excel); increased risk of

non-reproducible errors

• Dispersed information: emails, post-its, script logs, sharepoint, ‘wiki sites’, personal documents, brains, etc.

• Heterogeneity of toolsets and capabilities; islands of specialisation with manual transmission in-between– XML documents– R, Matlab, Python Scripts; Excel

• Idiosyncratic access arrangements• Basic traceability requires “off-line” documentation.


... And Finally


Too many Dave’s?

David Maidment (CUAHSI)

David Valentine (CUAHSI)

David Tarboton (CUAHSI)

David Kinny (BOM)

David Schell (OGC)

David Arctur (OGC)

David Thomas (WMO)

David Lemon (CSIRO)

Peter Taylor (CSIRO)

Peter Fitch (CSIRO)

Peter Gijsbers (Deltares)

Pierre Lagarde (BRGM)

Pierre Kerherve (WMO)


Public mailing list https://lists.opengeospatial.org/mailman/listinfo/hydro.dwg

Twiki http://external.opengis.org/twiki_public/bin/view/HydrologyDWG

Thank you ...

https://lists.opengeospatial.org/mailman/listinfo/hydro.dwg

http://external.opengis.org/twiki_public/bin/view/HydrologyDWG

The What, Why and How of the OGC/WMO Hydrology Domain Working Group or How Too Many Dave’s is...

Documents

Transcript of The What, Why and How of the OGC/WMO Hydrology Domain Working Group or How Too Many Dave’s is...