Ocean Observatories Initiative EOI User Review June 16 2011 OOI Cyberinfrastructure Release 1 Scope...
-
Upload
dana-wheeler -
Category
Documents
-
view
213 -
download
0
Transcript of Ocean Observatories Initiative EOI User Review June 16 2011 OOI Cyberinfrastructure Release 1 Scope...
Ocean Observatories Initiative
EOI User ReviewJune 16 2011
OOI CyberinfrastructureRelease 1 Scope
OOI CI - EOI User Review16 June 2011
EOI User ReviewJune 16 2011
OOI Integrated ObservatoryObservatory Requirements:
Provide one integrated observatory interface to all users inside and outside the
OOI. Enable the users to investigate observations,
manage the observatory and its assets and collaborate with
each other in teams.Engineering Drivers:
A geographically distributed system of systems with
observatories at multiple scales and operational
authority. OOI-wide need for data distribution, storage,
processing and command and control.
EOI User ReviewJune 16 2011
Observatory Requirements:Enable communities of users to work with the instruments and observatory assets they need from across all OOI sites in a
uniform observatory environment. Support multiple
“virtual” observatories in parallel.
Engineering Drivers: Multiple heterogeneous
communities of users accessing shared observatory assets
across the OOI. Enable access to observatory assets based on
user authorization level and observatory policy.
Virtual Observatories
Obse
rvato
ry
Inte
rface
EOI User ReviewJune 16 2011
OOI Deployment TopologyObservatory RequirementsProvide observatory assets to
support measurements of scientific processes. Provide computational and network assets to enable real-time,
secure access and processing for the observatory users
Engineering Drivers: Proximity to observatory assets, geographic redundancy. High-bandwidth, low latency access to national and international
network peering points. Access to commercial and academic
compute clouds
EOI User ReviewJune 16 2011
National & International Observatory Integration
NEPTUNE Canada
NSF OOI Network
NOAA N-Wave Network
Global LambdaIntegrated Facility
Engineering Driver:The topology used by the
OOI program is easily integrated into national and international cyber-
networks. This will facilitate collaboration for
developing the international ocean observing network
EOI User ReviewJune 16 2011
Network Messaging Infrastructure
•Based on the Layer 2 National Network Infrastructure•Provides direct application-level messaging connectivity
between components (Capability Containers) within the Integrated Observatory Network, and at users’ sites.
EOI User ReviewJune 16 2011
What's Different about OOI?
Integrated Observatory Network Integrated Observatory Network
EOI User ReviewJune 16 2011
Event Detection and Response
Observatory Requirements:Support closed loop scientific activities.
Enable interactive and event-driven control of the observatory assets based
on interactive and real-time observations and analyses.
Engineering Drivers: Workflows accessing real-
time measurements, external data sources, real-
time processing, QC and event detection, complex numerical models leading to control of mobile and stationary observatory
assets.
EOI User ReviewJune 16 2011
Integrated Observatory Network
Integrated Observatory Network
Integrated Observatory Network: 4 Releases
R2
R1
R3R4
R4: Interactive Ocean Observatory
R2: Managed Instrument Network
R1: Data Distribution NetworkR3: On-Demand Measurement
Processing
EOI User ReviewJune 16 2011
Release 1 of ION: Data Distribution Network
•publish data to ION• register data sources
•find and browse data resources•examine data resources
• initiate data notifications and subscriptions•download data from system•get live data streams
•create user accounts•scale computing resources•perform diagnostics and monitoring• receive events without delay
'User-visible' Highlights
EOI User ReviewJune 16 2011
Release 1 of ION: Data Distribution Network
•“Ingest” data products from external data sources and experimentally from sensors in canonical and raw formats
•Characterize data sources with their metadata attributes (format, structure)
•Distribute data via streaming and DAP servers to consumers such as data analysts, numerical modelers
•Provide a platform for instrument integration development, with instrument control and sensor data acquisition
•Provide a distributed service integration and execution platform in more than one programming language
Technical Perspective
EOI User ReviewJune 16 2011
Release 1 of ION: Data Distribution Network
•Data distribution via publish/subscribe•messages received are immediately distributed•processes act on messages as soon as received (workflow!)
•Scalability: manage growing amounts of work in a graceful way•new process host (virtual machines) can be started as needed•sets up for automated compute resource management
•Fault tolerance: continue operation when part of system fails•can restart a processing unit, pick up where previous left off
•Security: protect information from misuse, theft, or corruption•basic user and policy mechanisms to enable core model
Structural Highlights
EOI User ReviewJune 16 2011
Release 1 Components
https://confluence.oceanobservatories.org/display/syseng/
CIAD+AV+Transition+to+Operations
EOI User ReviewJune 16 2011
CI Components Developed
https://confluence.oceanobservatories.org/display/syseng/
CIAD+AV+Transition+to+Operations
• Python Capability Container• Java Capability Container Access Library• Web UI Platform• Exchange Messaging System• Distributed State Infrastructure• Resource Registry Framework• Data Publish-Subscribe Framework• Event Notification Framework• Science Data Persistence and Transport Format• Instrument Agent Framework• Elastic Processing Unit• Virtualized Cloud Management Tools
EOI User ReviewJune 16 2011
Use Cases for Release 1
UC.R1.05 Synchronize State Data
UC.R1.06 Distribute Data Product
UC.R1.07 Subscribe To Data
UC.R1.08 Persist Streamed Data
UC.R1.01 Hello User
UC.R1.02 Hello Instrument
UC.R1.03 Hello Data Source
UC.R1.18 Command An Instrument
UC.R1.20 Command A Resource
UC.R1.04 Ingest and Describe Data
UC.R1.21Derive Data Product Externally (Merge Data)
UC.R1.09 Discover Resource
UC.R1.14 Use Service Anywhere
UC.R1.15 Put Services Anywhere
UC.R1.16 Scale the Processing
UC.R1.17 Replicate Service
UC.R1.25 Assure Reliability
UC.R1.28 Operate System
UC.R1.29 Monitor System
UC.R1.30 Troubleshoot System
Data
Services &Operations
Basic Connections
EOI User ReviewJune 16 2011
EOI User ReviewJune 16 2011
"Hello User" Use Case1.User registers with the system by providing an (external) identity, creating an
internal identity for user.
• Internally, user has an internal identity and a user "profile"; which contains 'life cycle state'
of the user and other user attributes such as contact information and preferences.
• User authenticates to external identity provider, in order to assert external identity
• Background: The InCommon federation facilitates the exchange of knowledge that enables
the assertion of identities across organizations within this federation. The NSF CI-Logon
project is developing cyberinfrastructure software to apply InCommon processes within CI
systems, such as OOI. A user, for instance from UCSD, will be able to use her organization
credentials to register with OOI, and subsequently use OOI credentials.
• Each user identity is tracked in the user identity repository.
2.User logs in to the system using identity credential (e.g., login ID and password).
3.User accesses the system's web user interface, which calls system services.
• Features: Status, Register Resources, Search (User, Instrument, Data, Process), Instrument
Access
4.User is assigned their default role in system
EOI User ReviewJune 16 2011
Implications of OOI Approach
• We have user identities in the system, enabling security
• We keep user attributes of interest to us (e.g., contact info)
• We assign attributes as needed (e.g., user roles)
• ION doesn't perform authentication itself, so …
• Users don't have to remember a custom OOI identity
• ION doesn't have to maintain user credentials internally
• passwords and associated security issues
• support for all the associated human issues
• software we don't have to write, or maintain
• This solves a big scaling issue
EOI User ReviewJune 16 2011
Select Identity Provider and Log On
Cornell UniversityDuke UniversityGeorgetown UniversityGoogle
EOI User ReviewJune 16 2011
Identity Provider Logs You In
We get a certificate saying who (Google says) you are.
EOI User ReviewJune 16 2011
EOI User ReviewJune 16 2011
Data Distribution Use Case1. Data is accepted at a specific Acquisition Point CyberPoP or any specific capability
container.
• External data will be accepted via an Instrument Agent or other adapter process, and go through an
acquisition process before ingested as complete data set or increment for a data set.
• Information does not have to be ingested (i.e. made available in the resource registry, normalized)
to be distributed via the Data Distribution Network
2. Once minimally described, data is sent to the Exchange by the Data Producer.
• Ideally data should be fully described before routing, not just minimally described.
• Further description of the data may already be present in the resource registry of data sets.
3. The Exchange routes data messages to all subscribers to a data stream or filter that match
certain criteria in the description of the data message
1. Matching certain criteria is also fundamental to several other use cases, including explicitly in
Version a Resource.
• Subscribers equally see data, whether internal (local or remote) or external users.
• A data store (or several different stores) is a subscriber to most forms of ingested science data, in no
way different to any other subscriber. In case there is no data store subscriber, data will be
distributed across the Data Distribution Network without any form of persistence; subscribers are
solely responsible for maintaining the information they need.
EOI User ReviewJune 16 2011
Subscribe to Data Use Case1.Data Consumer identifies material of interest.2.Variant A: Data Consumer selects existing data stream that provides
the material of interest.3.Variant B: Data Consumer selects historic data set resource …4.Data Consumer registers (through API or web form) with Subscription
service to be delivered material of interest, per a defined set of criteria.5.Data Consumer specifies subscription delivery modality with
Subscription service.6.Registration is acknowledged via message returned to Data Consumer.7.Registration is verified through inspection of the Inventory and Data
Stream Registry.8.Data messages meeting registered criteria arrive within Exchange,
where they are routed to subscribed Data Consumer.9.Event notification messages generated from event producing
processes (e.g. Instrument Agents, Data Source adapter, event aggregator), arrive within the Exchange, where they are routed to registered Data Consumer.
EOI User ReviewJune 16 2011
Implications of "Data Distribution"
• User gets data when we get data (no waiting!)
• Users or modules can subscribe to selected data
• Processing routines kick off when data arrives … workflows!
• Fundamental artifacts are data streams (not static sets)
• ION internalizes the data in its constituent elements
• Allows fully using the data's natural structure
• Can present the data according to users' needs
• All kinds of data sources: models, sensors, observatories,
people, …
• ION can keep a 'reference copy' for fast, reliable access
• The data, and products, can be well curated (good metadata)
EOI User ReviewJune 16 2011
CI System Design Engineering Milestones
• Passed Life Cycle Objectives (LCO) Review for Release-1
(Mar 2010)
• Approval of Release-1 system architecture baseline and use cases
• Demonstration of critical technologies for risk mitigation
• Passed Life Cycle Architecture (LCA) Review for Release-1 (Aug 2010)
• Approval of refined Release-1 use cases and system architecture for
Construction
• Demonstration of an end-to-end running system and a scalable architecture
• Passed 2 subsystem LCA reviews: Instrument Platform Agent Architecture,
External Observatory Integration (December 2010)
• Passed Initial Operational Capability (IOC) Review (May 2011)
• Up-to-date approved architecture and design
• Added CyberPoP and network design; user experience and user interface
designs
EOI User ReviewJune 16 2011
CI Development Progress by Subsystem
ID Subsystem# R1 L4 Req’s
Satisfied
# R1 Use Cases
Satisfied
% Design complete
SLOC# Tasks
complete
SA Sensing & Acquisition 3 (7%) 3 (75%) 39% 1,198 30
AS Analysis & Synthesis N/A N/A N/A 0 0
PP Planning & Prosecution N/A N/A N/A 0 0
DM Data Management 41 (59%) 5 (56%) 95% 23,139 110
COI Common Operating Infrastructure 31 (26%) 4 (36%) 95% 34,467 155
CEI Common Execution Infrastructure 14 (56%) 8 (100%) 77% 35,599 109
IPA Instrument and Platform Agents 14 (41%) 1 (100%) 100% 11,587 52
EOI IOOS Integration Package 1 15 (100%) 1 (100%) 100% 7,793 24
UX User Experience N/A 0 (0%) 80% 24,701 67
ITV Integration, Test and Verification N/A N/A 100% 6,984 N/A
TCN Terrestrial CyberPoPs and Network N/A N/A 100% N/A N/A
41
EOI User ReviewJune 16 2011
EOI Presentation