OOI CI Overview and Status
description
Transcript of OOI CI Overview and Status
OOI CI R2 Life Cycle Objectives Review
Aug 30 - Sep 1 2011
Ocean Observatories Initiative
1
OOI CI Overview and Status
Matthew Arrott
DMAC-STWashington DC, Jan 18-19, 2012
OOI CI R2 Life Cycle Objectives Review
Aug 30 - Sep 1 20112
Agenda
• Review of the OOI CI Construction Objectives
• Introduction of Release 2 Functional Objectives
• Computational Infrastructure Objectives
• External Observatory Integration
• What will OOI-CI mean to me
OOI CI R2 Life Cycle Objectives Review
Aug 30 - Sep 1 20113
OOI CI Construction Objectives
OOI CI R2 Life Cycle Objectives Review
Aug 30 - Sep 1 20114
OOI Project Scope
Building/deploying science infrastructure for understanding a changing water planet
• Multi Dimensional Engineering Concern• Social• Organizational• Functional• Procedural• Technical
Cyberinfrastructure fulfills the integrative mechanism across these dimensions
OOI CI R2 Life Cycle Objectives Review
Aug 30 - Sep 1 20115
The Hubble Telescope for the Oceans
OOI CI R2 Life Cycle Objectives Review
Aug 30 - Sep 1 20116
The Hubble Telescope for the Oceans
OOI CI Release 2 KickoffMay 2, 20117
Core Science & Education Capabilities
• Interactive Ocean Observing (R3 - R4)
• Interactive Ocean Modeling & Data Assimilation (R3 - R4)
•Discipline-Driven Semantic Organization of Data (R3 - R4)
•Automated Data Product Generation (R1 - R3)
• Interactive Instrument Network (R1 - R3)
• Integrated Observatory Management (R1 - R2)
•User-Driven Integration of Resource (R1 - R3)
ObservatoryActivity Model
OOI CI R2 Life Cycle Objectives ReviewAug 30 - Sep 1 2011 8
OOI Integrated Observatory Product•The fully operational research observatory will meet the following •Goals:
• Continuous observations at time scales of seconds to decades• Spatial measurements from millimeters to kilometers• Sustained operation during storms and other severe conditions• Real-time or near-real-time data as appropriate• Two-way transmission of data and remote instrument control• Power delivery to sensors between the sea surface and the seafloor• Standard plug-n-play sensor interface protocol• Autonomous underwater vehicle dock for data download/battery recharge• Access to deployment and maintenance vehicles that satisfy the needs of
specific observatories• Facilities for instrument maintenance and calibration• A management system that makes data publicly available• An effective education and outreach program
OOI CI R2 Life Cycle Objectives Review
Aug 30 - Sep 1 20119
Extending the Traditional Data Access Model
10
Multi-Scale Research Specific Ocean Laboratories
OOI Review Year 2May 16-20, 201111
Release Schedule
Inception Phase
Elaboration Phase
Construction Phase
Transition Phase
OOI Review Year 2May 16-20, 201112
Release Schedule by Subsystem
Inception Phase
Elaboration Phase
Construction Phase
Transition Phase
OOI Review Year 2May 16-20, 201113
CI Components Developed for R1
• Python Capability Container
• Java Capability Container Access Library
• Web UI Platform
• Exchange Messaging System
• Distributed State Infrastructure
• Resource Registry Framework
• Science Data Persistence and Transport Format
• Data Publish-Subscribe Framework
• Event Notification Framework
• Instrument Agent Framework
• Elastic Processing Unit
• Virtualized Cloud Management Tools
OOI Review Year 2May 16-20, 201114
Capabilities of the Managed Instrument Network
•Operate Marine Observatories
•Operate Platforms and Instruments
•Manage Instrument Lifecycle
•Test and Troubleshoot Instruments
•Acquire Data and Generate Data Products
•Search Data
•Visualize Data
•Manage the Integrated Observatory Network
OOI Review Year 2May 16-20, 201115
R1 Integrated Observatory Network Web UI
https://confluence.oceanobservatories.org/display/CIPUB/OOI+Review+2011+Demos
OOI CI R2 Life Cycle Objectives Review
Aug 30 - Sep 1 201116
Release 2 Functional Objectives
OOI CI R2 Life Cycle Objectives Review
Aug 30 - Sep 1 201117
Observatory Operations & Administration
OOI Review Year 2May 16-20, 201118
• CI Release-1: Data Distribution Network
• Provide framework for discovery and interaction based on
data-intensive users’ needs
• CI Release-2: Managed Instrument Network
• Provide support for operations and maintenance of marine
networks based on RSN and CGSN needs
• CI Release-3: On Demand Measurement Processing
• Provide dynamic analysis and visualization tools based on
science communities’ user needs
• CI Release-4: Interactive Ocean Observatory
• Provide mission control and collaboration support based on
science communities’ needs
User Experience Incremental Development Plan
Scientific CollaborationScientific Collaboration
General Science General Science UsersUsers
ScienceScience ExpertsExperts
Resource Resource ProvidersProviders
OOI Review Year 2May 16-20, 201119
UXUX
R2 Product Description: List of Use CasesUC.R2.01 Acquire External Data SourceUC.R2.02 Derive Data ProductUC.R2.03 Produce Real-Time QC DataUC.R2.04 Browse to Get Data ProductUC.R2.05 Register and Connect InstrumentUC.R2.06 Command InstrumentsUC.R2.07 Direct Instrument Access IIUC.R2.08 Manage Instrument LifecycleUC.R2.09 Activate Instrument DriverUC.R2.10 Manage Marine PlatformUC.R2.11 Operate Marine ObservatoryUC.R2.12 Deploy Agents On Remote PlatformUC.R2.13 Acquire Data From InstrumentUC.R2.14 Monitor an InstrumentUC.R2.15 Qualify Instrument InterfaceUC.R2.16 Install Instrument Automatically
UC.R2.17 Define Visualization MethodUC.R2.18 Visualize Data ProductUC.R2.19 Produce Matlab Visualization
UC.R2.20 Annotate ResourcesUC.R2.21 Transform Data in WorkflowUC.R2.22 Version ResourceUC.R2.23 Ingest Dataset SupplementUC.R2.24 Search for ResourceUC.R2.25 Advanced Resource SearchUC.R2.26 Navigate Resources and MetadataUC.R2.27 Manage Replicated ArchiveUC.R2.28 Manage Resource MetadataUC.R2.29 Integrate External Data Source
UC.R2.30 Define InteractionUC.R2.31 Define New ServiceUC.R2.32 Conduct NegotiationUC.R2.33 Enroll in an OrgUC.R2.34 Share an Org ResourceUC.R2.35 Share Affiliated Orgs' ResourcesUC.R2.36 Create an OrgUC.R2.37 Control Service InteractionsUC.R2.38 Define Resource Life CycleUC.R2.39 Manage ION UsersUC.R2.40 Monitor ION ResourcesUC.R2.41 Recover Failed ProcessUC.R2.42 Define Resource PolicyUC.R2.43 Operate Message Brokers
UC.R2.44 Put Services Anywhere EasilyUC.R2.45 Replicate Activated ServiceUC.R2.46 Operate Integrated SystemUC.R2.47 Deploy Versioned User ProcessUC.R2.48 Schedule User-Defined ProcessUC.R2.49 Deploy Distributed ProcessesUC.R2.50 Define Scaling PolicyUC.R2.51 Define Execution EngineUC.R2.52 Manage ION Processes
UC.R2.53 View Modeler-Submitted ProductsUC.R2.54 Access NEPTUNE CA Data
UC.R2.55 Manage Help TicketUC.R2.56 Monitor ION NetworkUC.R2.57 Configure Start Page
UC.R2.58 Display Arbitrary ResourceUC.R2.59 Generate New Screen
DMDM
S&AS&A
A&SA&S
COICOI
CEICEI
EOIEOIOpsOps
OOI CI R2 Life Cycle Objectives Review
Aug 30 - Sep 1 201120
Managed Instrument Network
OOI Review Year 2May 16-20, 201121
Sensor Set #1 Instrument AgentsClass # Family Instrument Class Long Name
ADCPS 16 Water Velocity Velocity_profile_600m
CTDGV 24 Conductivity, Temperature, Depth CTD_glider
CTDMO 135 Conductivity, Temperature, Depth CTD_mooring
CTDPF 30 Conductivity, Temperature, Depth CTD_profiler
DOSTA 61 Dissolved Oxygen oxygen_dissolved_stable
FDCHP 7 Direct Covariance Flux flux_direct_cov_HP
FLORD 35 Fluorometer Fluorometer_two_wavelength
METBK 18 Meteorology Meteorology_bulk
NUTNR 22 Nitrate nutrient_Nitrate
OPTAA 35 Optical Properties attenuation_absorption_optical
PCO2A 10 Partial Pressure CO2 pCO2_air-sea
PCO2W 25 Partial Pressure CO2 pCO2_water
PHSEN 33 Partial Pressure CO2 pH_stable
PREST 3 Seafloor Pressure pressure_SF_tidal
SPKIR 22 Optical Properties spectral_irradiance
VEL3D 32 Water Velocity Velocity_point_3D_turb
VELPT 21 Water Velocity Velocity_point
WAVSS 10 Surface Wave Spectra wave_spectra_surface
ZPLSG 4 Zooplankton Sonar plankton_ZP_sonar_global
OOI Review Year 2May 16-20, 201122
Sensor Set #2 Instrument AgentsClass # Family Instrument Class Long Name
BOTPT 3 Seafloor Pressure pressure_bottom_tilt
CAMDS 12 Cameras x camera_digital_still_strobe
CAMHD 1 Cameras x camera_digital_video_HD
HYDLF 5 Hydrophone Hydrophone_LF_passive
MASSP 2 Mass Spectrometer mass_spectrometer
OBSBB 3 Ocean Bottom Seismometer Seismometer_BB_triaxial_accel
OBSBK 2 Ocean Bottom Seismometer Seismometer_BB_triaxial_keck
OBSSP 8 Ocean Bottom Seismometer Seismometer_shortperiod
PPSDN 1 Seafloor Fluid and DNA DNA_particulate
RASFL 1 Seafloor Fluid and DNA watersample_chem_trace_H2S_pH
THSPH 1 Seafloor Properties Temp_H2_H2S_pH
TMPSF 1 Seafloor Properties Temperature_seafloor
TRHPH 1 Seafloor Properties Temp_resist
OOI Review Year 2May 16-20, 201123
Sensor Set #3 Instrument AgentsClass # Family Instrument Class Long Name
ADCPA 15 Water Velocity velocity_profile_mobile_asset
ADCPT 17 Water Velocity Velocity_profile_300m
CTDAV 3 Conductivity, Temperature, Depth CTD_AUV
CTDBP 18 Conductivity, Temperature, Depth CTD_bottom_pumped
DOFST 22 Dissolved Oxygen oxygen_dissolved_fastresp
FDCLP 3 Direct Covariance Flux flux_direct_cov_LP
FLORT 42 Fluorometer Fluorometer_three_wavelength
HPIES 2 Inverted Echo Sounder IES_pressure_velocity
HYDBB 6 Hydrophone Hydrophone_BB_passive
NUTR4 3 Nutrients nutrient_four_channel
OSMOI 2 Seafloor Properties watersample_chem_trace
OTISF 1 Seafloor Properties flow_benthic
PARAD 30 Optical Properties PAR
PRESF 9 Seafloor Pressure pressure_SF
VADCP 2 Water Velocity Velocity_profile_50m_turb
ZPLSC 8 Zooplankton Sonar plankton_ZP_sonar_coastal
OOI Review Year 2May 16-20, 201124
CI Architecture for Instrument Integration
25
Data Processing and Product Generation
OOI Review Year 2May 16-20, 201126
COL
OOI's Sensor Life Cycle
OOI Final OOI Final System System DesignDesign
Core Core Sensor Sensor
ListListCore Core
Sensor Sensor SpecsSpecs
Sensor WorkingGroup
Core Core Sensor Sensor RFPsRFPs
COL
CGSNRSN
Common Common Sensor Sensor
SelectionSelectionss
COL
CGSNRSN
CI
Procure Procure Common Common SensorsSensors
CGSNRSN
Develop Develop QC/Xform QC/Xform AlgorithmsAlgorithms
CGSNRSN
Develop Develop Sensor Sensor DriversDriversCI
Integrate Integrate w/ Sensor w/ Sensor
AgentAgent
Test Test w/Instrumw/Instrument Test ent Test
KitKit
CI CI Integration Integration
TestTest
System System Integration Integration
TestTestCICICICI
Encode Encode AlgorithmsAlgorithms
CI
Deploy Deploy at Seaat Sea
RSN
CGSN
OOI Review Year 2May 16-20, 201127
Instrument Development Kit
• Scheduled for Release-2
• Includes:
• Logical Test Facility
Workbench (dry testing)
• Marine Specific System
Test Facility (wet testing)
• Configurations for RSN
and CGSN observatories
• Access and Management
Portals for Interactive
access
OOI CI R2 Life Cycle Objectives Review
Aug 30 - Sep 1 201128
Computational Infrastructure
Objectives
OOI Review Year 2May 16-20, 201129
National Network Deployment (Year 3)
OOI Review Year 2May 16-20, 201130
CyberPoPs and Network Infrastructure
• Acquisition Point CyberPoPs:• Portland, OR• Woods Hole, MA
• Distribution Point CyberPoPs:• Seattle, WA• Chicago, IL• San Diego, CA• McLean, VA (optional)
• Engineering Center:• San Diego, CA
• Network Infrastructure• Dedicated 10GE loop San
Diego, Portland, Seattle,
Chicago; branch to McLean
(optional)• Dedicated 1GE connection to
Woods Hole
OOI Review Year 2May 16-20, 201131
San Diego Engineering Center
• Purchased and deployed
San Diego CyberPoP and
network equipment (see
rack drawing)
• Located in UCSD’s Atkinson
Hall server room 1101
(secure and protected)
• In use for
• Release-1 production and
QA
• Management tools
(Confluence, Jira, etc)
OOI Review Year 2May 16-20, 201132
San Diego CyberPoP and Network Equipment
OOI Review Year 2May 16-20, 201133
Acquisition Point CyberPoP Deployments (Y3)
•Sites•Portland•Woods Hole
•Function•Data acquisition from
marine observatories•Real-time data processing
and OOI data product
generation•Data preservation•High availability
OOI Review Year 2May 16-20, 201134
CyberPoP Physical Layout (Year 3 Deployments)
•Sites•Seattle•Chicago
•Function•Content distribution (web
servers, data servers,
messaging)• Interconnects with major
national and international
network providers (layer 2
peering)•Links to compute clouds
OOI CI R2 Life Cycle Objectives Review
Aug 30 - Sep 1 201135
Network Peering IntegrationChicagoSeattle
OOI CI R2 Life Cycle Objectives Review
Aug 30 - Sep 1 201136
Multi-Tier Messaging Federation
37
Multi-Site Service Network Deployment
OOI CI R2 Life Cycle Objectives Review
Aug 30 - Sep 1 201138
External Observatory Integration
OOI CI R2 Life Cycle Objectives Review
Aug 30 - Sep 1 201139
IOOS Integration Model
OOI Review Year 2May 16-20, 201140
External Observatory Integration Progress
• Identification of datasets and data sources needed to support early
adopter group (ESPRESSO/Rutgers, HiOOS/UHawaii) numerical model
workflows
•Classification of datasets by type and representation
•Development of Dataset Agents for automatic data and metadata
ingestion
•Translation of data/metadata into the Common Science Data Format
•Streaming of data packets and metadata update notifications
• Integration of real-time data sources in early-adopter on-site numerical
workflow generation processes, using the Integrated Observatory
Network Release-1
OOI Review Year 2May 16-20, 201141
External Observatory Integration
• Exemplar: MARACOOS (Rutgers University)
• Experimentally substitute part of the scientists’ data assimilation workflow preceding numerical
model execution
OOI Review Year 2May 16-20, 201142
External Observatory Integration: Master Dataset List
• Datasets
• NAVY : NRLSSC (DAP) : 1
• NOAA : NDBC SOS (HTML) : 34
• NOAA : NDBC SDF (DAP) : 1
• NOAA : PFEG (DAP) : 1
• Rutgers : tashtego (DAP) : 2
• UCSD : HFRNET (DAP) : 1
• UH : SOEST (DAP) : 1
• USGS : WaterService (HTML) : 26
OOI CI R2 Life Cycle Objectives Review
Aug 30 - Sep 1 201143
“What will OOI-CI mean to me”
OOI CI Release 2 KickoffMay 2, 201144
Regional Data Provider & Manager
• Effective mechanism to reliably at scale distribute your data
• to your user community and data management/preservation
facility
• Independently publish and receive data
• in a wide variety of syntactic formats and community vocabularies
• Publish and receive as near-real-time data streams and as historical
data sets
• Effective mechanism to reliably at scale aggregate and disseminate
data from your publishers and to your end user communities
OOI CI Release 2 KickoffMay 2, 201145
Participating Agency
•Participate as providers, managers and consumers
•Anticipated some requirements for publishing on the OOI as
a separate Agency will require coordination with NSF
•Requirements are being addressed as a part of the IOOS
OOI Integration effort
OOI CI Release 2 KickoffMay 2, 201146
How does one “join” – approval / certification?
•Open data policy network for consumers of data
•Anticipate publishers required to register and certified
• Data policies and rights management being established
•Open data policies and content limitations to science and
education data content
• Publishing requires acceptance of some form of open use policy
• Consuming requires some form of “as is”; “at your on risk”; “with
attribution of source if republished” policies
OOI CI Release 2 KickoffMay 2, 201147
Relationship with IOOS, NEPTUNE and others observatories
•External Observatory Integration subsystem within the OOI
• Specifically focused on providing bi-directional integration
interfaces to IOOS, NEPTUNE-Canada and WMO (aka GTS)
• Integration continues on into the Operations and Maintenance
phase of the OOI as a user/observatory “on boarding” capability
•Provide the training and support to communities that wish to
integrate with OOI
OOI CI Release 2 KickoffMay 2, 201148
OOI define standards for them to “plug in”?, e.g SensorML
• OOI provides a wide variety of syntactic formats and community
vocabularies by which to interface with the OOI
• OOI and the user community will be able to extend these interfaces
• Capability facilitated by architectural choice of using controlled set
of canonical data models into and out of which all data are
transformed
• New transformations are added, “plug ins”, to the network for a
new format and/or vocabulary to one of the controlled data models
OOI CI Release 2 KickoffMay 2, 201149
Will a region be able to host an instance of the OOI “stack”?
• The OOI “Integrated Observatory Network” is a federation architecture
• Comprising participants from multiple domains of authority operating from their local
domain of authority
• The OOI “stack” exists at a couple of levels:
• It is a messaging protocol
• It is a component software implementation that implements the messaging protocol
and process management containers (think modern web server) that publish and
consume services
• “Capability Containers”, can be and are written in multiple languages
• achieve their interoperability through use of the common OOI messaging protocol
• OOI message protocol and any of the OOI supplied Capability Containers will be
provided under one of the standard open source licenses
OOI CI R2 Life Cycle Objectives Review
Aug 30 - Sep 1 201150
Thanks!