Post on 02-Jan-2016
description
2
• NSF reverse site visit• Refactoring and cleanup after review preparations• Coordinating Node technology changes (reducing
latency, generalizing)• Authentication• Identity management• Access control• TeraGrid Compute Node design
2011 – First half
3
• Public release of cyberinfrastructure• Synchronization, replication, authentication,
authorization, resolution, discovery, retrieval
• Coordinating Nodes• UNM, UCSB, ORC
• Member Nodes• KNB, Dryad, ORNL-DAAC, AKN, NBII, CUAHSI, replication
targets, Merritt, other Metacat, other Mercury
• Investigator Toolkit• Libraries, Web interface, Zotero, R, Morpho, DataONE
Drive
Goals for 2011
4
The Major Components
Investigator Toolkit
Web Interface Analysis, Visualization Data Management
Client LibrariesJava Python Command Line
Member Nodes
Service Interfaces
Bridge to non-DataONE Services
Data Repository
Coordinating NodesService Interfaces
Object Store Index
Resolution
RegistrationReplication
Discovery
Identifiers
MonitorPreservation
Catalog
Identity Authen
5
Investigator Toolkit SoftwareSearchPortal R Client Morpho
Client LibrariesJava Python Command Line
Member Node Software
Metacat
Coordinating Node SoftwareService Interfaces
Object Store Index
Software Delivered at Public Release
Zotero DataONE FS (Excel)
Dryad
GMN (CUASHI)
(Merritt) Preservation MonitorCatalogIdentifiers
Replication DiscoveryResolution Registration
Mendeley…
DataONE Service Programming Interface (SPI)
6
Tier 1: Public read, synchronization, search, resolveTier 2: Read with access controlTier 3: Write using client toolsTier 4: Able to operate as a replication target
• Some Member Nodes with Tier 4 support and corresponding Coordinating Node functionality required for 2011
Member Node Tiers
7
Node Tier Status Priority ReleaseKNB 4 2 1 2011ORNL DAAC 2 1 1 2011Dryad 2 ? 2 1 2011CUAHSI 2 1 1 2011AKN 4 1 1 2011Merritt 2 0 2 2011 ?Fedora ? 1 3 2012Repl. Targets (GMN) 4 2 1 2011TeraGrid 4 1 2 2012Mercury based 2 1 2 2011 ?Metacat based 4 1 3 2011?
Member Nodes
Status: 0 = Not implemented, 1=partial, 2=functional, 3=completePriority: 1 = High, 2= medium, 3= low
8
Coordinating Node ServicesService Tier Status PriorityNode registry 1 2 1Object storage 1 3 2Identifier resolution 1 3 2Synchronization 1 1 1Discovery 1 1 1Log retrieval 1 0 2Identity mapping 2 2 2Session state 2 2 2Identifier reservation 3 3 3Replication 4 0 2
Status: 0 = Not implemented, 1=partial, 2=functional, 3=completePriority: 1 = High, 2= medium, 3= low
9
Coordinating Node Components
cn_service
cn_rest cn_rest_proxy
Metacatidentity_manager
nodes_registry
synchro-nization
replic-ationindexer
SOLR Postgres
JMS
LDAP
MercurySearch
10
Component Tier Status Priority ReleaseLibclient_python 4 2 1 2011Libclient_java 4 2 1 2011R plugin 2 (3) 1 1 2011D1Drive - FUSE 2 1 1 2011D1Drive - Dokan 2 1 1 2011D1Drive - WebDAV 2 0 3 2012Mercury UI 2 2 1 2011Morpho 3 1 2 2012Excel 3 1 3 2012Zotero, Mendeley 2 1 2 2011
Investigator Toolkit
Status: 0 = Not implemented, 1=partial, 2=functional, 3=completePriority: 1 = High, 2= medium, 3= low
12
Other PiecesComponent Status Priority ReleaseDocumentation 2 1 2011
Member Node test service 1 1 2011Integration testing 2 1 2011Monitoring 1 2 2011Hardware 2 1 2011
Continuous build 2Issue tracking 3Repository 3Conferencing 2
Status: 0 = Not implemented, 1=partial, 2=functional, 3=completePriority: 1 = High, 2= medium, 3= low
14
Infrastructure will change over time. • Message between services• Behavior of services, components• Message semantics
How to accommodate change while maintaining operations?
Versions
15
Infrastructure Version: Tag identifying the overall functional capabilities, types, messages supported by the DataONE infrastructure
Component Version:Tag of a specific component indicating the current revision of the component
Example: Version XXX of Metacat supports version YYY of the DataONE infrastructure
Infrastructure + Component Versions
16
1. Changes to Types Schema2. Update test data3. Build libcommon Pyxb / JIBX 4. Propagate changes through libclient5. Propagate changes through remainder of components6. Deploy CN test7. Update test Member Nodes8. Deploy staging9. Update / synchronize content to match production10. Switch staging and production
Where to update all Member Node stacks?Where to update all Investigator Tools
DataONE Infrastructure Release
17
Release Coordinating Nodes• Touches everything deployed – MNs, ITK, monitor, ... • Likely to require content migration, synchronization• Expensive
Update Member Nodes• Vendor driven updates (e.g. core changes to Metacat)• DataONE driven updates (e.g. access policy fix)• Known contact points for updates
Update client tools• Automated version checks?• How to notify users?• How to notify third party developers?
Component Updates
18
For an infrastructure upgrade:• What is the acceptable latency between change
announcement and infrastructure back in sync?• Should DataONE maintain old versions of service
interfaces? If so how many revisions?• Some changes may not be backward compatible• Option to make old version CNs readonly while content
migrated to new systems?
Version Latency and Acceptable Delta
19
New package available with clear indication of affected components, signed off by component developersAffect CNs:
• Install on staging, process metadata updates, lock production, switch.
Affect MNs:• Notify instance administrators of update• Install updates and verify operation• Registry picks up updated services
Affect ITK:• Post update information• Notify active clients of update• Notify third party developers of changes
Administrative Process for Updates
21
July • New hardware deployed at UNM, UCSB, ORC• CN development with focus on Auth and synchronization• Integration test services
August • CN development focus on indexing, replication• Deploy CN staging to new hardware• ITK libraries updated• Full Tier 1 functionality deployed
September • Identity mapping, authentication, authorization• Evaluation of Tier 1 functionality• Support Tier 2 operations
October • Support Tier 3 and 4 operations on specific Member Nodes• CNs support replication, load testing of CNs• Evaluation of Tier 2 and initial tier 3 support
November • Final evaluation of functionality• Incorporate or schedule suggestions• Final stress and usability testing
December • Final documentation, testing• Announcement of initial release
General Timeline