Interagency Ocean Observation Committee DMAC Steering Team 4 th Meeting – November 19-20, 2013

Interagency Ocean Observation Committee

DMAC Steering Team

4th Meeting – November 19-20, 2013

Consortium for Ocean Leadership, Washington, D.C.

IOOC Interagency Ocean Observing Committee

DMAC Steering Team

DMAC ST Meeting – November 19-20, 2013

Opening remarks…

• Introductions (members/stakeholders/others)

• Review: Terms of Reference – our mission

• Recent efforts (since Feb ’13)

• Timeline for next steps

• Meeting Agenda

• Questions/Discussion


DMAC Steering Team


Introductions… Federal Members Invited Stakeholders

1. Charles Alexander (Chair) NOAA/US IOOS2. LCDR Jonathan Andrechik USCG/OSTP3. Ken Casey NOAA/NODC4. Jeff deLaBeaujardiere NOAA/NESDIS5. Mark Fornwall USGS/OBIS6. Michelle Gierach NASA/JPL7. Walter Johnson BOEM8. Erica Key NSF9. Kevin Kirby EPA10. John Lever USN11. Jeff Lillycrop USACOE12. Rich Signell USGS/Woods Hole13. Samantha Simmons MMS14. Derrick Snowden NOAA/US IOOS15. Erik Stabenau NPS/Everglades

1. Nadine Alameh OGC2. Karl Benedict UNM/ESIP3. Janet Fredericks WHOI4. Eoin Howlett RPS/ASA5. Dave Jones SCC, Inc.6. David Mann Loggerhead Inst.7. Ru Morrison NERACOOS8. Krish Narasimhan unaffiliated9. Henry Norris FL FWC10. David Parrish VIMS11. Roger Proctor IMOS (Australia)12. Kurt Schwehr Google13. Julie Thomas Scripps/SCCOOS14. Steve Uczekaj Boeing


DMAC Steering Team


DMAC ST “Terms of Reference”

• Provide the IOOC with strategic guidance on DMAC-related activities and challenges

• Maintain focus on DMAC subsystem elements (i.e. discovery, transfer, access, archive)

• Identify and solve specific data management challenges within the ocean observing realm; and

• Serve as a forum for collaboration among IOOS agencies and partners on DMAC issues

Purpose, Scope and Goals:


DMAC Steering Team


Participant Roles…

Federal members:

Non-federal stakeholders:

• represent agency data management practices and priorities

• formulate DMAC guidance and recommendations for the IOOC

• two year terms, maximum of two consecutive terms

• align DMAC guidance and standards with realistic implementation which non-federal entities are often responsible for executing

• repeated meetings with one group of individuals must be avoided


DMAC Steering Team


Review of Feb 2013 Meeting…

• USGS/Rich Signell virtual host

• OGC Interoperability Experiments (Nadine Alameh)

• US IOOS DMAC update (Derrick Snowden)

• Priority tasks for ST: in-depth discussionso Standards: accelerate adoption at IOOC member agencies

o Establish DMAC as a distributed system w/centralized catalog

o Focus on catalog services

o Metadata documentation and compliance


DMAC Steering Team


The way forward (Feb 2013)…

• Guided by but not driven by IOOS Summit priorities

• Include connections to/recommendations for global

• More systematic and routine engagement with IOOC

• Meet more often (phone/video, etc.) for shorter durations

• Focus on viable solution sets for the IOOS enterprise

• Develop a 12-18 month work plan with milestones/products

• Re-boot when milestones/products complete (new ST/chair)


DMAC Steering Team

• 3 IOOS Summit Recommendations

• 3 NOC Milestones

• ST project to “test” discovery, access, use of ocean observations



DMAC Steering Team


Our agenda…


DMAC Steering Team

IOOC Meeting – September 23, 2013

determine how easy it is to FIND, ACCESS, and USEocean observations data from IOOC member agenciesusing community best practices

1. accelerating adoption of standards2. improving overall use of metadata3. DMAC as a a federation of data discovery catalogs

Advances current ST Priorities:

Data Accessibility Testing and Evaluation Project


DMAC Steering Team



• Definition of the specific scenario(s) from which the data and service requirements will be derived

• Identification of the applicable community best practices for data format, access and documentation in the context of the requirements of the defined scenario(s)

• An assessment (documented in a report summarizing by dataset/metadata/service component) of the current ability of IOOC member agencies to fulfill the requirements defined for the scenario(s).

• Development of recommendations for strategies for closing the gap between current capabilities and those defined in the scenario specification

Objectives:


DMAC Steering Team



Notional Schedule: FY2014


DMAC Steering Team


Purpose: Exercise the IOOS distributed data discovery and publication architecture to document performance and identify needed improvements. Description: •demonstrate the ability to discover and access ocean observations via multiple scenarios on a subset of specific data types and data providers

•answering test “questions” or developing a key product will be addressed with the same path: search for the data, discover the data, and use the data

US IOOS Program’s System Integration Test


DMAC Steering Team



Notional Schedule: FY2014


DMAC Steering Team




$19K from IOOC $275-300K from IOOS

Collaborate on a “single” Systems Integration Test Project in FY2014


DMAC Steering Team


Next steps – schedule

ointegrate planning for ST and IOOS Program “Systems Integration Test” project (Oct-Nov)

oconvene DMAC ST meeting/workshop (Nov)• virtual host• “Test” project – next steps• refining responses to Summit Recommendations and NOP

Milestones

oinitiate first tasks of “Test” project (Nov-Dec)

ocomplete actionable inputs to Summit Recommendations and NOC Milestones in coordination with Biology Team and Post Summit Task Team (Dec)


DMAC Steering Team


Thank You!

Questions/Discussion


DMAC Steering Team


• 3 IOOS Summit Recommendations

• 3 NOC Milestones

• ST project to “test” discovery, access, use of ocean observations


DMAC Steering Team


IOOS Summit – Nov 2012

25 Recommendations

4 on Data Management & Communications (nos. 21, 22, 23, 24)


DMAC Steering Team


emergency events

• common data access & visualization tools

• power outages

• timeliness of data delivery

• personal-scale decision products

• IOOS response teams


DMAC Steering Team


• Move toward a common operating platform to enable direct sharing of geospatial data

• Understanding and implementing Open Geospatial Consortium standards

• Remote data staging and access via the cloud

• Systematically address potential power failure points

• For personal-scale products, provide scalable, multi-resolution data sets that also contain metadata to trigger scale-dependent constraints

• Technology Focused Contingency Planning such as a cloud-based virtual Emergency Operations Center

• Collaborate with ORR on capabilities and limits of ERMA for response planning

• Test the effectiveness of data discovery and access services for an emergency response scenario

Responses to Recommendation #21


DMAC Steering Team


• standards & protocols

• global observations

• business & industry

• satellite data


DMAC Steering Team


• Standardize a process for bringing new data types or user groups into the IOOS standards

• Implement an IOOS approved data standard and protocol list that includes IOOS branding. Existing IOOS-compatible systems in public and private sectors could seek certification, establishing a core of functional IOOS systems.

• Increased band-width across the internet combined with improved software for data extraction techniques will enable better integration of satellite data with other IOOS observations

• Evaluate current data services and interoperability via a series of discovery, access and use tests driven by one or more scenarios



DMAC Steering Team


• quality assurance/quality control

• metrics on users & usage

• functionality of data assembly centers


DMAC Steering Team


• Instead of focusing on instrument calibration, it may be more productive to encourage or even mandate reporting of error and uncertainty measurements including model outputs

• Develop a QARTOD (Quality Assurance for Real-time Oceanographic Data) software “layer” to implement recommended tests that can be widely shared and implemented

• Investigate QA/QC options, e.g. implemented locally/regionally versus at a central data assembly center

• It is not clear which “liability” issues need to be resolved

• Metrics on data access and use is essential and can be programmatically implemented not just for web sites, but also for data access to web services

• Number, type and functionality of DACs is very important including guidelines on lifespan of data availability and procedures for archiving



DMAC Steering Team


NOP Implementation Plan – April 2013

3 Milestones for IOOC agencies assigned to DMAC Steering Team

Milestone 1

“Develop, evaluate, and expand an integrated geospatial database of Federal and non-Federal, certified and non-certified ocean observation data to provide access to public information and provide extracts or contact information for privately held information (IOOC Member Agencies)”

Proposed IOOC Action: Assign to DMAC STAssess current integrated geospatial databases per this milestone (July 2013)Identify limitations to existing databases (September 2013)Suggest improvements to existing databases (November 2013)


DMAC Steering Team


• A wide assortment of new and emerging tools and technologies are broadly enabling increased discovery and access to ocean observing data

• However, significant but solvable obstacles remain including semantic (how data are described) and syntactic (data formats and communications protocols)

• Adoption of an increasingly small range of data formats are helping improve interoperability

• Non-profit organizations such as OGC and Unidata are improving open source tools that are widely used – this includes a core set of web services developed and tested through sponsored forums by OGC and enhancements to data formats such as netCDF supported by Unidata as a set of software libraries and self-describing, machine-independent data formats that support the creation, access, and sharing of array-oriented scientific data.

• Data brokering is also emerging from the NSF-sponsored EarthCube project as a process for negotiating and resolving interoperability obstacles between service-based data providers.

Responses to NOC Milestone #1

Milestone 4

“Identify the existing data services and systems, as well as the requirements to support integrated discovery and access through an information management system and integrative functions required for the management system. (IOOC Member Agencies)”

Proposed IOOC Actions: Assign to DMAC ST Assess current data services, systems, and requirements (July 2013) Identify limitations to data services, systems, and requirements

(September 2013) Suggest improvements to data services, systems, and requirements

(November 2013)


DMAC Steering Team


• Imperative to assess the readiness of interoperability for current data services, systems and requirements perhaps via a simple template to capture respective methodologies

• The assessment of interoperability will help identify commonality across organizations such as the IOOC member agencies and the types of tools and resources needed

• Key questions that could be resolved include impacts on the consumer (e.g. does the system allow subsetting? is the delivery in a framework with sufficient tools for translating or assimilating?) and discoverability (e.g. can the system be brought into a discovery tool such as a catalog service? are the data adequately described?)

• A DMAC ST-led process to census this type of information might take 12 months to provide meaningful discussion and productive results.


Milestone 5

“Identify the limitations of existing methodologies for integrating observational data, including coastal and global ocean remote and in situ data, physial and biological data, and ocean observations and socioeconomic data. (NASA, NOAA, NSF, DOI [USGS], DOE)”

Proposed IOOC Actions: Partially fulfilled by IOOS Summit Report, Bio Task Team,

and DMAC ST actions under 1 & 4 Requires additional effort by identified agencies


DMAC Steering Team


• Communicating/disseminating clearly to technical implementors the landscape of compliant data services, encodings, vocabularies, etc.

• Development/deployment of service validation tools that provide constructive, actionable feedback to service providers

• Promoting the use of integrated or distributed standard-compliant services and catalogs will be the key, including client code to facilitate this process

• Increased attention to a synchronized process for the integration of biological data with the emerging DMAC architecture – it is still problematic to fully integrate biological data into a bio-physical system because of the generally different approaches to measurement and the lack of equivalent standards for data delivery

• Cannot hope to produce reliable, quantifiable climate-change forecasts without a better understanding of how bio-physical processes integrates up to the bigger picture



DMAC Steering Team


IOOC Interagency Ocean Observation Committee

DMAC Steering Team

February 27-28, 2013 – COL, Wash, DC

Interagency Ocean Observation Committee’s

DMAC Steering Team

3rd Meeting – February 27-28, 2013

Charly Alexander – U.S. IOOS Program - Chair

Consortium for Ocean Leadership, Washington, D.C.


DMAC Steering Team


Opening remarks…

• Introductions (members/stakeholders/others)

• Terms of Reference – our mission

• Timeline – our work to date

• Update on the IOOS Summit

• The way forward

• Questions/Discussion


DMAC Steering Team

1 David Legler – NOAA/OAR & IOOC Co-Chair

2 Josh Young – COL

3 Nick Rome – COL

Introductions… Others



DMAC Steering Team


Four meetings:

• May 11-12, 2011

• January 18-19, 2012

• skipped June 2012 to work on CWP for IOOS Summit

• September 5-6, 2012

• February 27-28, 2013

Timeline – our work to date…


DMAC Steering Team


• US Army Corps of Engineers – challenges/contributions

• US DMAC status report (DMAC Plan)

• QARTOD

• Observing asset inventory

• NSF’s OOI-CI

• Standards recommendation process discussion

• Agency updates

Review of May 2011 Meeting…


DMAC Steering Team


DAY 1

•Introduction/Re-Orientation

•US IOOS DMAC Update (D. Snowden)

•NASA Data Management Challenges (M. Gierach)

•OGC Web Services (L. Bermudez)

•Update on NSF’s OOI Cyber-infrastructure (M. Arrott)

DAY 2

•U.S. IOOS QARTOD (C. Alexander/D. Snowden)

•NSF Earth Cube (J. Schopf)

•Emerging Priorities/Recommendations

•Wrap-Up/Next Steps

Review of Jan 2012 Meeting…


DMAC Steering Team


Steering Team Roles & Responsibilities

Standards & Best Practices

Clarify how findings/recommendations are processed

Design/conduct DMAC customer satisfaction survey

Recast meetings; 1-2 outcomes, breakouts, etc.

Adopt use of data catalogs as a standard tool

Industry adopt use of Sensor Model Language for all instruments

Promote uniform guidance on use of netCDF

Review of Jan 2012 Meeting…


DMAC Steering Team


Next Steps – action items (Jan 2012)

1. Post materials to updated web site (done)

2. Meeting minutes/summary report (drafts complete)

3. IOOC briefing on findings/recommendations (today)

4. IOOC designate permanent ST Chair (US IOOS recommends C. Alexander)

5. OGC analysis of survey results

6. OOI-CI: questions and conference call

7. NSF’s Earth Cube: review capabilities document

8. Recommend specific execution priorities to IOOC?

9. Potential engagement mapping priorities to IOOS summit?

10. Planning/coordination for 2nd 2012 meeting


DMAC Steering Team


What the DMAC ST needs from the IOOC

1. Guidance (criteria?) on how to proceed with next steps suggest ST moves forward on 1-2 specific priorities

suggest ST activates “tiger teams” to identify or specify a data

management protocol on a test case and IOOC helps identify participants

2. Organizational support for broader engagement on DMAC

suggest ST members identify key POCs for IOOC members to contact suggest IOOC members share ST reports, etc. with IT/data management

leadership in their organizations

3. Support for mapping DMAC priorities to upcoming IOOS Summit suggest asking ST for volunteers for an IOOS Summit DMAC team

suggest a ST member begin attending IOOS Summit planning meetings


DMAC Steering Team


DAY 1

•Icebreaker – DMAC Concept Mapping

•EPA: Virtual Host Presentation/Discussion (K.Kirby)

•Priorities for a USE CASE – orientation/assignments

•Breakout Sessions I (candidate use cases) & II (relevant DMAC priorities)

DAY 2

•IOOS Summit Update (R. Morrison)

•IOOC Request – Assessment of Summit white papers (D. Legler)

•USE CASE writing teams

•Wrap-Up/Next Steps

Review of Sept 2012 Meeting…

First – our Summit white paper…

October 22, 2012 – COL, Wash, DC

DMAC Steering Team

… and our Recommendations

•Implement IOOS standards and evolving best-practice protocols for the whole data life cycle, including metadata, quality control, discovery, and archiving.

•Build from the bottom up with an agile, use-case-driven approach instead of a multiyear, top-down design

•Develop tools that query and handle standards-based responses for major development and research languages.

•Provide training, up-to-date educational materials, and support on these protocols for providers and users.

•Ensure development of social media that enable providers and users to share code, techniques and experiences.

•Develop IOOS compliance procedures and a compliance-testing suite.

DMAC Steering Team


Use Case discussions

Group 1: Human Health & Beach Closures

Group 2: Emergency Response, Renewable Energy

Group 3: Biophysical conditions based on human impacts, Ocean Energy


DMAC Steering Team

Request from IOOC


DMAC Steering Team

1. DMAC Assessment of other Summit white papers: 34 of the 100+ papers were judged to have a DMAC component or intersection with ST white paper (reviewed 24 at ST meeting)

Request from IOOC


DMAC Steering Team

2. Possible DMAC break-out session at Summit•Lightning talks on key themes/DMAC successes•Based on DMAC correlations among white papers•Proxy for bringing one or more use cases forward•Calibrate with messages emerging in Summit proceedings/chapters•Demonstrate connections to Ocean.data.gov

Next Steps

1. Review EPA’s System of Registries (DMAC ST, on-going)

2. Get clear guidance on ability to revise or update Summit white papers (IOOC Support Office, 9/14)

3. Develop DMAC session for the IOOS Summit (Alexander/Signell/Casey, 9/21)

4. Finalize Oil Spill Case Study (emergency response) (Signell/Norris, 10/19)

5. Finalize beach closure case study (Kirby/Morrison, 10/19)

6. Finalize hypoxia case study (Thomas/Gierach, 10, 19)


DMAC Steering Team

Post-Summit UpdateERIC LINDSTROMINTERAGENCY OCEAN OBSERVATION COMMITTEETUES 22 JAN 2013

Did we achieve our original objectives?

Understand progress made over the last decade

Determine requirements needed to maintain IOOS

Develop a consensus on the way forward for the next decade

Snapshot of Recommendations

Plenary……………………………..17 Broad recommendations

Breakouts………………………….80+ Varying in scope

Community White Papers….100+ Covering everything

Types of Recommendations

Regional/national/global integration

Public-private partnership models

Communications strategies

Biological information

System Integration Pilot Projects

Modeling and Predictive Capability

Moving Forward

Summit Declaration Finalized

Community White Papers final revisions 1/25

Summit Report due mid-Feb

Draft available 1/31 (IOOC comments due 2/8)

Next twelve months – “Roll-out” Summit Recommendations and Ways Forward with communities and agencies


DMAC Steering Team


Discussion & Questions

Interagency Ocean Observation Committee DMAC Steering Team 4 th Meeting – November 19-20, 2013

Documents

Transcript of Interagency Ocean Observation Committee DMAC Steering Team 4 th Meeting – November 19-20, 2013