Decadal Survey

NSIDC Thoughts on DS Data Systems Workshop

Ron Weaver and Marilyn Kaminski

Purpose of this UWG Discussion

• Help us strategize data management for the DS missions and what role you (PoDAG) believe NSIDC should play

• How do we make the DS data streams work best for the cryosphere research community?

• What are our opportunities?• Who should be our partners?

DS Workshop

• In late June NASA held a workshop on Decadal Survey data management. It was attended by reps from the DS missions (e.g. IceSAT2, SMAP, DESDynI etc.), the Earth Science Data Centers, including reps from the UWGs

• From NSIDC, Kaminski and Weaver (1/2 day) and Anderson attended

• For background on Decadal Survey Missions and data management please consult:

http://nasascience.nasa.gov/earth-science/decadal-surveyshttp://dsds.nasa.gov/

Voltz

(What I heard)• Missions must integrate well• Handle Data to ECS standards• Promote Synergy• Bring in Foreign Data to DS Mission Support• Measurements are Primary• Bring Applications and Operations Communities

into fold (with research community)

Decadal Survey Missions

• Soil Moisture Active/Passive (SMAP)• ICESat-II • Deformation, Ecosystem Structure and

Dynamics of Ice (DESDynl)• Climate Absolute Radiance and Refractivity

Observatory (CLARREO)• Hyperspectral InfraRed Imager (HyspIRI)

Where do these elements intersect with NSIDC DAAC systems?

• NSIDC has scientific links to SMAP, DESDynI, ICESat II and possibly CLARREO

• If we include ASF we could solidly manage SMAP, DESDynI and ICESat II

• But we could not manage all of them!• So, where do we focus?

Possible Synergies• Missions must integrate well

– IceSat I, IceSat II, DESDynI (Lidar)IceBridge• We are involved to some extent in these• Synergy between NSIDC (Lidar) and ASF

INSAR for DESDynI

– AMSR-E, SSMIS, SMAP• We can provide supporting data to SMAP• MeaSURE program data from Frozen

Ground ESDR (Kimbell), Enhanced PM data streams (Long)

• Foreign Data– Cryosat, GCOM-W

• Measurements are primary– Science disciplinary support for sea

ice, ice sheets, snow hydrology

Synergies – Data systems

• Infrastructure for users– Searchlight and CCAWS

are in part being built to handle new data streams

• Remove culture divide– Key element of IceBridge

DM task is wider data dissemination and use

• Tie to SMAP and DESDynI data systems

How to move forward

• Integrative data strategies– Hold a passive microwave data provider workshop to

make sure we are providing supportive data to SMAP, other missions. (should this include scatterometry?)

– Expand the IceBridge instrument team integration work to include a broader set of Lidar data

– Discuss DESDynI with ASF– Meet with the DS science/instrument teams about

data issues• How do we/should we lobby HQ

Backup Slides

Notes from the Meeting

(Bolded items are of particular interest)

Voltz

(What I heard)• Missions must integrate well• Handle Data to ECS standards• Promote Synergy• Bring in Foreign Data to DS Mission Support• Measurements are Primary• Bring Applications and Operations Communities

into fold (with research community)

Maiden

Present to Near Future System Features• Coexistence of heterogeneous, distributed data

providers/information Partners• Minimal set of core standards• Support for community specific standards• Reusable software• Service Oriented Architecture• On-line archives and cross-system service invocation• Ease of innovation and technology infusion

Breakouts – Standards

• Recommended Core Standards be developed and mandated by HQ as level 1 requirements

• In the next 6 months recommend a metadata standard and a file format standard for new missions/systems

Breakouts – Data Systems Infrastructure and Interfaces

• Need to provide infrastructure for ‘users’, i.e. that meets user needs as opposed to mission needs.

• Need tools that remove culture divide between experts and novices (Currently data sets are developed to facilitate mission scientists and processing, not users)

• Expand definition of ‘decadal missions’ to include other missions and measurements from other sources besides NASA including international missions, other US agencies

Breakouts – Data Systems … page 2

• On-the-fly processing – cloud computing – parameter searching (not just granules) – virtualization of products

• Need to be careful of the concept of ownership of the data and the systems

• Consider developing a system mode that contains a center of excellence for a particular data product or product suite

• Creation of interoperable datasets• Searching by parameter will require a very robust

metadata

Searchlight

NSIDC Searchlight Project Objectives (July 2008) • Create a simple to use on-line interface(s) for cryospheric

data • Get NSIDC data (V0, ECS, other) distributed to more users • Get the user just the (subsetted/reprojected) data they

need, in the format they want, as quickly as possible • Allow users to display and interact with the data • Create a standard and flexible NSIDC framework for new

applications • Meet the base objectives of CCAWS, DADDI, CADIS, ACAP,

GCAP & The DAAC

Searchlight Performance and Design Objectives

Contributing grants and contracts include ACAP(A), CADIS(C), CCAWS(W), DADDI(D), GCAP (G) and the DAAC(T) • In the list, letters in parentheses refer to the set of grants and contracts with specific mention of the requirement in the

grant or contract proposal. For example, a `Searchlight requirement indicated with (ACW) refers to something specifically mentioned or promised in the proposal materials for ACAP, CADIS and CCAWS.

• system response times will depend on action performed: – 5-second response time for display results, or in-progress indicator – ??-second response time for data delivery (this should probably be a rate, as in seconds per file ordered, or seconds per GB of

data ordered) • the user will be able to access the system with a common web browser, with no plug-ins or special software for the user to

download; Ajax functionality is specifically in-scope; the current list of supported browsers is – Windows: IE 7.x; Firefox 3.x – Macintosh/Linux: Firefox 3.x

• system will be built with open source (C) and/or Government-funded technologies (C) (W says "preferably open source") • the user experience will be consistent; users will not be required to to know where the data are coming from, and will not

be required to learn this in order to use the system. • to maximize reuse and minimize maintenance costs, there will be a standard user interface built for each kind of information

(spatial, temporal, etc). Each of these interfaces may have different ways for the user to specify input, (e.g. specifying a date using a calendar widget vs. a text box, but there will not be 2 implementations of a calendar widget)

• the system will incorporate reasonable security measures, for example, using ssl for at least user authentication • when users navigate through the system, previously entered values will be remembered by the system

IceBridge Data Management. September 2009

19

Challenges (from IceBridge proposal work)

• Timeliness Of Data Delivery• Understanding And Responding To User Community Needs• Complexity Of Data Streams, Especially Experimental Instrumentation

– Changing Formats Of Experimental Data– Dataset Co-registration

• Appropriate Data Set And System Integration – Develop Tools And Services To Fit Complex Data– Or– Adapt Complex Data To Known Tools And Services

• Distributed (Virtual?) Data Management Systems– Who Holds What Data? Who Decides? How Does Interoperability Get

Accomplished?• Note That Archive Volume Is Not On This List