Preservation of NASA’s Earth Observation Data

EOSDIS Science Operations, ESDIS Project Code 423Goddard Space Flight Center, Greenbelt, MD 20771

ESIP Federation Meeting, Santa Barbara, CA.July 9, 2009

EOSDIS and NASA’s Earth SciencePreservation Activities

•NASA’s Earth Science Data and Information System Project will maintain distributed active archives to continue servicing the science and applications communities.

–responsibilities for data management include, yet go beyond, the life of EOS missions and continue with missions recommended by Decadal Survey. –keep all data important to active science research and applications in place at the EOSDIS Data Centers.

04/20/23 2

04/20/23 3

Preservation of Earth Observation Data

• NASA has observational data that will need long term preservation

Selected data sets will be organized for the National Archives and Records Retention (NARA) based on mission end of life and other critical data needs.

Our data life cycle ends with a consolidated and organized permanent record which specifically means instrument data co-located with the product generation source code, allowing for regenerated products when necessary, documentation, submitted and accepted to NARA.

Organization

• A number of events need to occur with each instrument for a successful archive strategy, but in general:

– ESDIS will coordinate joint long term archive activities with each EOS Instrument team through appropriate working groups.

– The working group will agree on source of information, responsibility for inventory and content, standards to follow and any corresponding preparation tasks.

– ESDIS will align the long term archive plan with current NASA’s records retention schedule rules.

– The entire archive process will be a transparent activity to all involved, including the science community.

– The working group will stay active through the entire archive process to ensure goals are met.

04/20/23 4

04/20/23 5

Characterizing EOS Electronic Records

EOS Data Product categories:

Level 0 - Unprocessed instrument/payload data at full resolution.Level 1 - Instrument data with radiometric and geometric calibration and geo-referencesLevel 2 - Derived geophysical variables at the same resolution and location as the Level 1 source data.Level 3 - Variables mapped on uniform space-time grid scales

Science Mission Drivers

• EP-TOMS and ICESat 1 where chosen as the first data sets to enter the long term archive process for the reasons below:

• EP-TOMS– Data flow discontinued in Dec 2005. Currently near the end of the project,

should be forming or have identified final records.– Team members should still be available to support a joint working group (OMI is

the continuity mission – different team with some overlap in membership).– A critical national historical dataset of ozone records for NASA in support of

Clean Air Act Amendments of 1990.

• ICESat 1– Instrument may fail soon and we may suddenly need to finalize records.– The instrument is a reasonable representation of the kind of information that

needs to be organized for the archive, and the learning curve faced by most other flight and science projects concerning use of preservation standards.

04/20/23 6

The value of the OAIS Ref. Model

• A common language that enables coordinated planning and implementation by partnering agencies and institutions for permanent records retention– NOAA, USGS, NARA

• A structure for linking representation information with the data– Example – MODIS LTA Handbook– Provides a logical structure, avoiding having to rethink

what goes into an AIP

04/20/23 7

Experience using OAIS RM

• Several of the data centers have used the OAIS Reference Model when evolving, updating or re-designing their archive– NASA Langley (LARC) Atmospheric Sciences Data

Center data system evolution– PO.DAAC (JPL) using OAIS Reference Model, ISO

19115 as design constraint– SEDAC and Columbia University using OAIS RM in

implementation of COTS packages Fedora and VITAL

04/20/23 8

Atmospheric Sciences Data CenterArchive, Next Generation (or ANGē)

• ANGē’s design is based on the Open Archival Information System (OAIS) Reference Model. • ANGē development is organized by each of the primary OAIS model component.• One of primary goals of ANGē’s design is to minimize the changes required for the interfaces to any current data providers.

An additional translation layer will be developed to allow current interface protocols to interact with ANGē’s OAIS compliant services. However, current data providers may elect to modify existing interfaces to take advance of better services (e.g. automated notification of data receipt). The diagram depicts this approach using nomenclature from the OAIS reference model.

04/20/23 9

CONCEPT DIAGRAM: Layered Approach to Insulate Core Functions from Varying Provider & User Needs

PRE-INGEST

CURRENT PRODUCER CONSUMERS

Requests; Responses; SIP: Data, Content Info., Preservation Descript. Info.

Data Request DIP

Queries; Result Sets

Orders; Data Packages Requests;

Data Input

Responses

DATA SEARCH

Search Parameters

Content Info.

Archive Admin.

Queries Statistics, Diagnostics

DATA ORDER

AIP AIP

AIP

Ingest Data Management Access

Archival Storage Administration Preservation Planning AIP

DIP

DIP

SIP

SIP media

database

CORE ARCHIVE

SIP

FUTURE PRODUCER

Model for Web-Based Data Submission and Workflow

DataProducer

ProvideFilesAndDescriptions

DataDeposit

DescribeData Set

DataDescription

GrantIntellectualPropertyRights

SubmissionAgreement

AppraiseAndApproveSubmissionInformationPackage

Review

TransformValues toXMLMetadata

Transformation

ArchivalInformationPackageInDigitalRepository

Ingest

Login with ID & PasswordAuthentication

One or More Submission Sessions

Notifications and Requests

Communication

Source: Downs and Chen. Creating a Trustworthy Digital Repository for a Long-Term Archive of Interdisciplinary Data: A Case Study. 21st International CODATA Conference, 5-8 October, 2008 Kyiv, Ukraine.

Socioeconomic Data CenterSEDAC

backup

04/20/23 11

04/20/23 12

EOS Data and Information System

• Earth Observing System (EOS) Data and Information System (EOSDIS) was conceived and built in the 1990’s as the spacecraft control, data acquisition, processing system for the EOS flight missions and the archive and distribution system for all NASA Earth science data

• EOSDIS consists of several Distributed Active Archive Centers (DAACs), and Science Investigator-led Processing Systems (SIPSs), and a Core System– DAACs provide processing, archive and distribution services for

EOS data.• Each DAAC holds different kinds of scientific data that includes

atmospheric, oceanic, land use/cover, space geodesy, cryospheric, and biospheric.

• The DAACs make EOS data publically available to all users, through online query, metadata, and retrieval systems directly to the user community.

– SIPSs produce standard products from EOS instrument data and send them to the DAACs for archiving

– Core System consists of components used to support major data flows at several DAACs and SIPSs

04/20/23 13

EOS electronic records life cycle

• Mission based Earth observing research programs– Many instruments and levels (maturity) of data– Many different activity timelines, retention is event driven

• Typical period for operating instrument (~3-15 yrs)– Nominally the design life plus optimal performance period– Ongoing data acquisition, product generation, archive & distribution– Ongoing science research and applications for societal benefit– Algorithm adjustments and reprocessing from beginning of mission can

occur– Provides inputs to next research & operational satellite programs

• Mission End-of-Life period, Decommissioning (~3-5 yrs)– Final reprocessing for products from the entire mission– Final research results, inputs to next missions– Organization of mission data for permanent long term retention

• Permanent electronic records are created during both periods

04/20/23 14

Earth Science Missions

2011 2012 2013 20142007 2008 2009 20102003 2004 2005 20061999 2000 2001 20021995 1996 1997 19981991 1992 1993 1994

2011 2012 2013 20142007 2008 2009 20102003 2004 2005 20061999 2000 2001 20021995 1996 1997 19981991 1992 1993 1994

10/1984 - 2 years planned mission life

10/1978 - 10 yrs m life

ERBS (SAGE II)

ERS-1

NIMBUS-7 (TOMS I)

Meteor 3 TOMS (TOMS II)

UARS

TOPEX/Poseidon

JERS-1

OrbView-2 (SeaStar)

ERS-2

Earth Probe TOMS (TOMS III)

RadarSat 1

ADEOS I early failure

Jason-1

ACRIMSAT

Aqua

ICESat

Aura

SORCE

Landsat 7

QuikSCAT

Terra

TRMM Heritage Missions

EOS Era Missions

No Planned EOL

KEY

Ongoing Data Access

GRACE

Transition to USGS

Extended Mission Life

Planned Mission Life

ADEOS II early failurePost Mission Support

April 2007 NASA Earth Science Senior Review

Science Data Available

CERES data only

Meteor 3M (SAGE III)

EOSDIS Manages DataFor all EOS Measurements

04/20/23 15

ACRIMSAT(12/99)

Solar Output

ACRIM

SORCE (1/03)Solar Irradiance

TIM, SIM, XPSSOLSTICE

Terra (12/99)EnergyBudget

CERES

Albedo,Aerosols,

Vegetation

MISR

SurfaceImaging

MODISASTER

LowerAtmospheric

Chemistry

MOPITT

Aqua (5/02)EnergyBudget

CERES

SurfaceImaging

MODIS

AtmosphericSounders

AMSR-EAIRS/AMSU/HSB

Aura (7/04)TraceGases

TES

AtmosphericDynamics and

Chemistry

HIRDLS,MLS, OMI

Mesosphere

Stratosphere

Troposphere

TRMM(11/97)

Energy Budget

CERES, LIS

EvaporationH2O

Sea Surface Winds

OceanBiology

LandBiology

Volcanology

Ecosystem Dynamics

O3

precursor gases,

aerosols

Mission & Science 04102007.ppt

O3

ClOBrOOH

trace gasesaerosols

Land Ice and Snow Cover

Rain

QuikScat(6/99)

Sea Surface Winds

SeaWinds

Temperature Moisture

Sea Ice

Fire Occurrence

METEOR 3M(12/01)

Aerosol Trace Gases

SAGE III

CloudSat(4/06)

Cloud Properties

CPR CALIPSO(4/06)

Cloud, AerosolProperties

CALIOP

Jason (12/01)Ocean Altimetry

Poseidon/JMR/DORIS

Rain

ICESat(1/03)

Ice Topographyand Altimetry

GLAS

04/20/23 16

Tracking & Data Relay Satellite

(TDRS)

Instrument Teamsand Science

Investigator-ledProcessing System

(SIPS)

EOSSpacecraft

Direct Broadcast/Direct Readout

Stations

DirectBroadcast

(DB)

EOS Data and Operations System

(EDOS) Data Processing

White SandsComplex

(WSC)

EOS PolarGround Stations

EOSDIS and Related Data

Centers

Internet

(Search, order,

distribution)

Research

Education

Value-AddedProviders

InteragencyData Centers

InternationalPartners

EarthSystem Models

Decision Support Systems

Distribution and DataAccess

Data Acquisition Flight Operations,Data Capture,

Initial Processing,Backup Archive

Data Transport

to Data Centers/

SIPSs

Science Data Processing,Data Mgmt, Interoperable

Data Archive, & Distribution

NASA Integrated Services Network (NISN)

Mission Services

Mission Operations Science Operations

EOS Operations Center (EOC)

Mission Control