NASH data sharing workshop

Stuart Hamilton,André Bouchard, Michael Allchin

Workshop goals

•Community-led initiative on hydrometric

data sharing

•Common Vocabulary

•Advocate for Reciprocal Altruism

•Report to WMO

State of the art in hydrological data sharing•Data sharing involves many complexities

owing to the number of systems, formats, organizations, etc.

•The big tent approach (all-encompassing)

The CUAHSI model

•Consistent, standardised infrastructure, covering

▫Observation methodologies, ▫metadata, ▫data model, ▫Search discovery query engine▫Transmission format ▫Modelling platform

WaterML 2.0

•Universal OGC / WMO standard for hydrometric and hydrometeorological data

•Consistent internal tag-based format, covering▫Observation features / sensors▫Time-variant data▫Metadata▫Associations (eg, rating curves)

To date, hydrologists and water resources professionals have often encountered hurdles in; •Finding data•Assessing fitness for purpose•Data retrieval•Format translation:

•New technologies are at the pointof offering a range of solutions foraddressing and solvingthese challenges

The Panel

•Axel Anderson, FRI/AERD•Wayne Jenkinson, CHC•Alain Pietroniro, WSC•Kevin Shook, USask•Stephanie Smith, BC Hydro•Bruno Tassone, WSC

What are the main barriers to data sharing?• Academic perspective:

• Data management challenges for short term projects• Security concerns: maintaining privacy, strategic

data,• End-user perspective:

• Search and Discovery• Data provider perspective:

• Administrative, technological challenges• Data rescue – important but costly

• Corporate perspective:• Mandate/cost

Audience response• Policy and structural issues • Research data and data management issues.

▫Data often seen as by-products rather than assets• No standards in terms of metadata.

▫Discovery extremely challenging. Metadata directories are essential. Web ‘storefront’ less human

• Balance to be struck between▫rights to data-access▫responsibility for appropriate use

• NASH could foster proper standards or practices.

Policy barriers•Academic Perspective

• No real policy barriers in academia. Rather suffer from a lack of policy. Funding agency driven.

•End-user perspective:•Data Provider perspective:▫No conceptual barriers to providing data▫Administrative and funding constraints (e.g.

CLF)•Corporate perspective:▫IT security

Audience Response

•A push to make data available needs to happen.

•Governance as a barrier (lack of policy)▫Mandate to provide data▫Let people know what is available and

where to get it- suspicion that data are ‘hidden’.

•Requirements on QA/QC▫Data life cycle constraints▫Real-time vs. Archival quality

Other barriers• Academic perspective:

▫data should be valued▫lots of data of varying quality

• End-user perspective:▫we should plan to share data from the onset

and not after the fact. ▫Data are valuable – use as an incentive.

• Data Provider perspective:▫Commitment to long term maintenance of

datasets. Governance issue. Who is the data guardian.

▫Data quality is a big issue.• Corporate perspective:

▫Accountability/liability for key data sets.

Audience Response• Lack of proper data management practices.

▫ File management rather than data management▫ Lack of standard exchange formats▫ Relation between credibility and training /standards

• Communication▫ Insufficient connections between data producers and data consumers. ▫ Increasing synergy happening in the private/public sector

• Lack of incentives to share data.• Use of policies related to public security as an incentive to share

data.• Incentives – lack of distribution leads to mistrust from clients• Need to consider the community monitoring groups in the

discussion.• US

▫ Issue of data rescue and the costs involved.▫ Issue of costs to manage existing and legacy data holdings

• A real opportunity for NASH to set guidelines and standards▫ Technology has evolved▫ Open data standards▫ The move to real-time▫ Interoperability

What makes you most excited about what you see happening ?• Academic perspective:

▫ Are addressing their data management issues and structuring their data.

▫ Interested about connectivities• End-user perspective:

▫ The possibilities of XML as a standard exchange format.▫ Idea of community initiatives to make data available.▫ The technology is progressing rapidly and is expanding the

possibilities. Access to many layers of data through GIS UIs.

• Data Provider perspective:▫ A certain maturity in the hydrometric community. The concept of

« Network of networks ». The notion of data assimilation and remote sensing. Systems are becoming more robust.

• Corporate perspective:▫ Climate – related monitoring program. ▫ Who has what and where ? ▫ Next step is creation of a data portal.

• NASH – role of putting people together.

Audience Response• Cybera – We-Hub (waterenvironmentalhub.ca)

▫Some issues but a good example.• A simple set of questions that a data provider

can fill out related to data quality• Hardest parts is setting the specs. • Concept of the data mart• Web-based & open-source systems. • Need to resolve firewall issues.• OGC family of standards (include security

specs).• Quality flagging – should look at the program done in the EU GeoWOW ! (standards for GEO data). • NASH could provide input

Audience Response (opportunities)• Which data and metadata should be made readily

available…▫ USGS : pretty much all the data that users require in a

timely manner.▫ Look at the community to drive the issue in terms of their

needs.▫ Vision : quality assured real-time flow information on the

web (and in an accessible data source). The move to real-time makes the old data production process an artifact.

• Common data access▫ To generate the change, you need to provide the tools to

facilitate that change. Tools can be costly to develop▫ Example : Water temperature provided along with

hydrometric data to enable users to better understand what is going on.

• Metadata is absolutely crucial. ▫ WMO standards a good place to start. ▫ WaterML 2.0 powerful as well.

Audience Response (opportunities)• Policy improvements ? Program activities ?

▫A suggestion to present the NASH group to the next NAT meeting in St. John’s (NL) – fall 2012

• Water data governance – tools are already available.▫How do you build on those tools with the

community ?▫Council of the federation : knocking on their door

to establish the connections.▫Alberta has a CEO of monitoring.

• Data needs to be treated as a capital asset and not just as a production tool. It should therefore be life-cycle managed.

Summary - Russell•5 themes : •Policy (more practice).•Technology / remote sensing – providing

data in areas where we don’t have data. •Standards•Awareness of technologies and standards.•Willingness to participate.

▫Two words : magic and maturity of hydrometric community. ▫Incentive / cost.

Summary – Paul W.•The issue of trust.•Hard coded push-pull world that we need

to deal with.•Willingness to discuss barriers.•Lack of access to data hurts us all.•Who owns the data ?•Fiscal problem with the management of

long-term data.•The value of data.•Our willingness to help.

Final thoughts•The model for sharing data presented by

Stephanie is interesting and could be explored by NASH.

•Sharing results and synthesis of workshop•Follow up with focussed workshops on

critical issues•A final report will be presented at WMO this

fall.•We should consider other publication options as well.•Culture shift required.

Thanks to ALL!

http://hydrographers.wikispaces.com/