NRC Panel on Climate Variability and Change July 25, 2005 1 John Bates and Thomas Karl, NOAA’s...

NRC Panel on Climate Variability and Change July 25, 20051

John Bates and Thomas Karl, NOAA’s National Climatic Data Center

Mitch Goldberg, NOAA’s Office of Research and Applications

Climate Elements of NPOESS

1


The Myth of Sisyphus*

• The gods had condemned Sisyphus to ceaselessly rolling a rock to the top of a mountain, whence the stone would fall back of its own weight. They had thought with some reason that there is no more dreadful punishment than futile and hopeless labor. …

• …I leave Sisyphus at the foot of the mountain! One always finds one's burden again. But Sisyphus teaches the higher fidelity that negates the gods and raises rocks. He too concludes that all is well. This universe henceforth without a master seems to him neither sterile nor futile. Each atom of that stone, each mineral flake of that night filled mountain, in itself forms a world. The struggle itself toward the heights is enough to fill a man's heart. One must imagine Sisyphus happy.

* A. Camus


Outline• Defining Information Stewardship Elements

Needed for NPOESS Climate Data Records (CDRs)– NOAA’s Comprehensive Large Array data

Stewardship System (CLASS)– NOAA’s Scientific Data Stewardship (SDS)

Program

• Metrics – A Maturity Model for CDRs• NPOESS Instruments & Schedules• NOAA Priority CDRs and Research-

Operations Transitions Using a Maturity Model Approach


CLASS Goals and VisionNOAA's National Data Centers and their world-wide clientele of customers look to CLASS as the primary NOAA IT infrastructure project in which all current and future large array environmental data sets will reside. CLASS provides permanent, secure storage and safe, efficient access between the Data Centers and the customers.

– A web-based data archive and distribution system for NOAA’s environmental data

– A combined process to both reengineer legacy data storage and access systems and blend new and efficient technologies to ensure the stewardship of existing (e.g., POES, GOES, NEXRAD, in-situ) and rapidly approaching large-array data sets (e.g., NPP/NPOESS, EOS, METOP, NEXRAD).

– An aggressive plan to safeguard, enhance, expand and automate NOAA’s capability to ingest, store, quality control, preserve, and access its vast environmental data holdings.

– A focused effort to ensure the Information Technology (IT) infrastructure is in place and working, before the arrival of significantly larger and more complex environmental data (e.g., NPP/NPOESS, and GOES-R).


CLASS Challenges

• Transforming from a data warehouse to an information stewardship system

• The massive data volume conundrum• Establishing a new culture within NOAA

and DOC with respect to leadership of scientific and information stewardship of satellite CDRs

• Long-term archive and information preservation


Synergy of CLASS & SDS

CLASS + Scientific Data Stewardship = Climate Products and Information

=+

CLASS – NOAA’s new archive; Comprehensive Large-Array data Stewardship System


• NOAA’s Scientific Data Stewardship rooted in NRC dialogue and reports

• NOAA/NRC SDS leads

– Bates

– Goldberg

Scientific Data Stewardship


Key Elements of a Successful CDR Program

CDR Organizational Elements

• High-level leadership council

• Advisory council to represent climate research community and other stakeholders

• Fundamental Climate Data Record (FCDR) Teams

• Thematic Climate Data Record (TCDR) Teams

CDR Organizational Elements

• High-level leadership council

• Advisory council to represent climate research community and other stakeholders

• Fundamental Climate Data Record (FCDR) Teams

• Thematic Climate Data Record (TCDR) Teams

CDR Generation Elements

• High accuracy and stability of FCDRs

• Pre-launch characterization of sensors and lifetime monitoring

• Thorough calibration of sensors

• Well-defined criteria for TCDR selection

• Stakeholder involvement and feedback for TCDRs

• Well-defined criteria for TCDR validation

• Use of in-situ data for validation

CDR Generation Elements

• High accuracy and stability of FCDRs

• Pre-launch characterization of sensors and lifetime monitoring

• Thorough calibration of sensors

• Well-defined criteria for TCDR selection

• Stakeholder involvement and feedback for TCDRs

• Well-defined criteria for TCDR validation

• Use of in-situ data for validation

Sustaining CDR Elements

• Available resources for reprocessing CDRs as new information becomes available

• Provisions for feedback from scientific community

• Long-term commitment of resources for generation and archiving of CDRs and associated data

Sustaining CDR Elements

• Available resources for reprocessing CDRs as new information becomes available

• Provisions for feedback from scientific community

• Long-term commitment of resources for generation and archiving of CDRs and associated data

Fundamental Climate Data Record (FCDR): Time series of calibrated signals for a family of sensors together with the ancillary data used to calibrate them.

Thematic Climate Data Record (TCDR): Geophysical variables derived from FCDRs, often generated by blending satellite observations, in-situ data, and model output.


Climate Data Records

Notional Functions of Scientific Data Stewardship for Climate

Network Performance Monitoring

Observations & Metadata

Analyses and Quality Control

Feedbacks

Stewardship Teams

Archives

Climate Analyses

Reference Data Sets

(Reprocessing)

Climate Quality

Products

Scientific Data StewardshipReal time and retrospective management of climate data

Model Re-analyses

Defining Scientific Data Stewardship


Sensor DataRecords (SDRs)

Data (Direct & Remotely Sensed)

Fundamental Climate Data

Records (FCDRs)

Thematic Climate Data Records

(TCDRs)

Climate Data Records or Homogenized Time Series

Homogenization and Calibration

Time-tagged Geo-Referenced

Converted to Bio-Geophysical

Variables

EnvironmentalData Records

(EDRs)

Converted to Bio-Geophysical

Variables

Defining CDRsClimate Data Records

NOAA CDR Reprocessing - Climate Central Requirements

– For reprocessing, SDS requires an IDPS-like system, (Climate Central) to process:

– SDS interdependent with CLASS, e.g., large data set I/O

– CLASS and SDS mutually dependent

– Requires NASA & W&W NDE linkage for enterprise solution

NOAA/CLASS Users

ACQUIRE RDRs

IMPROVED SDRsMULTISATELLITE

FCDRs

PROCESS TCDRs

PROVIDEASSESSMENTSIMPROVEMENTS

NOAASDS

CLIMATECENTRAL

Products and Services

Provide FCDRs and TCDRs11

RDR SDR Multi-satellite FCDRs & TCDRs


Climate Working

Group

NOAA Science Advisory

Board


CLASS Working Group


Program Management

Research Climate Data Science Teams

FCDR Teams

Observations Scoping

Requirement Systems

C2D2 NCDC ORA

FY06EDSM $2.5M

C2D2 $1 to $1.5MGovernance and Management Structure

TCDR Teams

R&D Products and

Services Theme Areas

NOAA

(5-10%)

External

(1-2%)

NOAA, Other

Agencies External Scientific

(~90%)

NOAA Climate Program & Climate Board

NOAA Observing System Council

Operational CDR Generation & Data Mgmt.

CDR Generation

NOAA

Other Agencies

Universities

Private Sector

CLASS

NOAA IT Infrastructure

operators

Research to

Operations

Currently exists FY05 FY06

NOAA’s Scientific Data Stewardship Program

12


•MandatesMandates•Mission RequirementsMission Requirements•CapabilitiesCapabilities•Capability Gaps ID’dCapability Gaps ID’d•Alternative SolutionsAlternative Solutions•Solution SelectionSolution Selection•Acquisition &Acquisition &Operations ApprovalOperations Approval

•Climate ObservationsClimate Observations& Analysis& Analysis•Climate ForcingClimate Forcing•Climate PredictionClimate Prediction•Ecosystem ApplicationsEcosystem Applications•Regional DecisionRegional DecisionSupportSupport

MAIN PROCESSING SYSTEMMAIN PROCESSING SYSTEM• Data from multiple satellite and

instrument types• Buoy, Ship & Ice Data

NOAANOAAClimateClimateCentralCentral

NOAA Management & StaffNOAA Management & Staff

Reconfigurable & ScalableReconfigurable & ScalableAVHRR

TRMMTMI

AquaAMSR-E

NPPVIIRS/CMIS

ScienceScienceCommunityCommunity

ScientificScientificDataData

StewardshipStewardship

NOAANOAAClimateClimateProgramProgram

Community Agreed Community Agreed Standards and ProtocolsStandards and Protocols

In Situ DataIn Situ Data

CLASSCLASSDataDataUsersUsers

L1b DATADATA

L1-L3 DATAL1-L3 DATA

PPBESPPBES

NOAA Operational

Satellites

NASAResearchSatellites

Future:NPP/NPOESS

NOAA’s Scientific Data StewardshipThematic Climate Data Records (TCDRs)

Sea Surface Temperature

13


Metrics – A Maturity Model for Climate Data Records*

• Reduce difficulty and confusion in community about scientific data stewardship

• Produce an easily understood way of identifying maturity of data products and science data stewardship approaches

• Help identify areas needing improvement

* With Bruce Barkstrom, NASA


Climate Requirements

• Interagency views of data maturity and science data stewardship needs reflect wide diversity– Vocabulary– Experience– Background

• Need understandable approach

ShowingData

Effectiveness

Science Data

Society

ShowingData

Effectiveness


A Simple Maturity Model

• Represent data maturity in terms of three separate dimensions:– Scientific Maturity– Preservation Maturity– Societal Impact

• Total maturity is simply length of vector

Scientific Maturity

Preservation Maturity

Societal Impact

Maturity ofdata for use


Software Capability-Maturity Model (CCMI) Levels

1. Initial – Unpredictable results

2. Managed – Repeatable performance

3. Defined – Cross-project interoperability

4. Quantitatively Managed – Improved performance + Compliance with Federal Enterprise Architecture

5. Optimized – Rapidly configurable performance + Continuous Process Improvement


Component Maturity for Climate Data Records

• Identify key attributes of maturity in each dimension

• Develop maturity ranking for each attribute on scale of 1 to 5

• Summarize component maturity by weighting each attribute– Simplest weight = 1/Number of attributes– Develop more complex weightings after experience

with approach

• Advantage: can do much of work with simple spreadsheet


Scientific Maturity Key Attributes

• Physical Understanding of Measurement Process

• Measurement of Key Instrument Characteristics

• Public Accessibility of Data Processing

• Rigorous Validation


Key Attribute Assessment Areas - Public Accessibility of Data Processing

Key Assessment Area

Level 1 Level 2 Level 3 Level 4 Level 5

Number of Analysis

Teams/CDR

None Single Two Multiple Consensus benchmark

Number of Independent Observing

Systems/CDR

None Single Two Multiple Benchmark

Reducing Model Uncertainties:

Forcings/Feedbacks/Validation

None Single Two Three Demonstrated benchmark

Availability of technique and computer code

None Technique in one

publication

Technique in multiple

publications

Computer code available

Computer code available and used by other groups


Preservation Maturity Key Attributes

• Systematic Approach to Guaranteeing Preservation of Data Understanding

• Systematic Reduction of Threats to Preservation

• Assurance of Preservation Cost Effectiveness


Societal Benefit Key Attributes

• Bibliometric Metrics– Publications and Citations

• Scientific Community Knowledge– Data use, including interdisciplinary data fusion and

statistical studies

• Economic and Policy Utility– Market valuation increase– Reduction in time to influence policy– Benefit/Hazard Reduction resulting from data use


Some Caveats• Using a Maturity Model will be exploratory – and

iterative– No expectation we’ll get it “right” the first time through

• Community Diversity must be incorporated– Different views of data processing, calibration,

validation, need for knowledge preservation– Different vocabularies

• Deep Uncertainty needs to be incorporated– Diversity of opinions on areas of scientific controversy

and value need common framework and disciplined discussion – openness a key

– Including “societal benefit” is very difficult and risky


Key Benefits

• Allows us to develop an approach consistent with NRC Recommendations on Metrics

• Open Process– Can surface divergent needs and opinions– Can provide disciplined forum for discussion and

resolution of differences

• Periodic Evaluation is required– Incorporate new information and deeper thought– Evaluation allows new directions


NPP/NPOESS Issues• NPP in morning vs afternoon orbit?

• NPOESS possible launch delays due to cost overruns

• Climate objectives only partly recognized in NPOESS mission requirements

• Collaboration with Eumetsat METOP and European partners is critical

• Must develop additional interagency and international partnerships


IPO Developed

Visible/IR Imager Radiometer Suite (VIIRS)* X X X X (AVHRR) X

Cross-track IR Sounder (CrIS)* X X X (IASI/HIRS) X

Conical MW Imager/Sounder (CMIS)* X X X

Ozone Mapper/Profiler Suite (OMPS) X X (GOME) X

GPS Occultation Sensor (GPSOS) X (GRAS)

Space Environmental Sensor Suite (SESS) X X (SEM)

Aerosol Polarimetry Sensor (APS) X

Leveraged Advanced Technology MW Sounder (ATMS)* X X X (AMSU/MHS) X

Operational Land Imager (OLI) X

ARGOS-Data Collection System (ADCS) X X X

Search and Rescue (SARSAT) X X X X

Earth Radiation Budget Sensor (ERBS) X

Total Solar Irradiance Sensor (TSIS) X

Radar altimeter (ALT) X

Advanced Scatterometer (ASCAT) X

NPOESS InstrumentsC3

20130530

Z

C22011

1330Z

METOP2006

0930Z

NPP2009

1330?

METOP/NPP/NPOESS Payloads

* Critical instrument

C120100930

Z


Scheduled launch dates not changed

Local Eq

uato

rial C

rossin

g T

ime

Deliveries

1330

0530

0730 - 1030

FY 11 12 13 14 15 16 17 1803 08 09 1002 0704 05 06 19 20 21

N’N

Earliest Availability

C3

F20

F16

F17 F19

F18 C1 C4

C5

C6

WindSat/Coriolis

DMSP

DMSP

METOP1 2 3

EOS-Terra

EOS-Aqua Earliest Need to back-up launch

NPOESS

NPOESS

NPOESSPOES

Meteorological Satellite Transition C1 & C2 Delay Schedule

Satellite launched

POES 17

10 Year Mission Life

Potential Coverage Gap

C2

Scheduled launch dates changed

Option 3

NPP


NOAA Priority CDRs

• Scientific Data Stewardship FY06 Performance Measure – 5 Maturity level 3 Thematic CDRs– Surface temperature – land and ocean– Precipitation– Radiation budget– Temperature profiles –troposphere/stratosphere– Water vapor profiles – troposphere/stratosphere


NOAA Priority CDRs – ApproachEstablish FCDR and TCDR Teams

• Perform real-time monitoring of the observing systems

• Determine best possible calibration and characterization of each instrument during its lifetime using on-board and vicarious calibration techniques and comparisons with NWP model analyses

• Account for any satellite drift or other artifacts in the sensor record

• Account for satellite to satellite instrument differences using overlapping observations

• Create a high-quality stable, seamless FCDR time series from a series of overlapping satellite observations

• Reprocess the FCDR time series as updated instrument information or improved techniques become available based upon cost/benefit analysis

• Subsampled, intercalibrated, integrated observations from suite of sensors both in situ and remote sensing

• Develop algorithms for generation of TCDRs from FCDRs

• Process near real-time TCDRs• Validate TCDRs against ground

truth and other satellite observations

• Reprocess the data set as updated instrument information or improved algorithms become available based upon cost benefit analysis

• Apply TCDRs in climate applications activities


NOAA Priority CDRs – Interagency Collaboration

• Identify common aspects of current and future CDR portfolios with interagency partners, especially NASA

• Identify maturity levels of critical climate variables

• Develop pilot research-operations teams for a sample set

• Work and learn


Conclusions – There are Major Challenges for NOAA

• Establishing a new culture within NOAA with respect to leadership of scientific and information stewardship of satellite CDRs

• This will take time (5-10 years…we’re in year 3), resources, persistence, and quality science

• Establishing an information preservation system for satellite-based CDRs also requires the establishment of new interpersonal relationships within NOAA, between agencies, and internationally


Backup Slides


Technology

Measurements

Modeling

Decision Support

- H- M- L

- H- M- L

- H- M- L

- H- M- L

Pathfinder

Operational Precursor

Operational Mission

Instrument Incubator

Planned Improvement

InitialValidated

OSSI

Pilot

Routine Use

Policy

Technology development

starts

Pathfinder Mission launch

Operational Precursor Mission launch

Pilot Program

for decision support begins

Operational Mission launch

Technology Improvement

begins

Decision support use

demonstrated

Decision support use

routine

Measurement Maturity

Measurement Maturity Index

1 2 3 4 5 6 7


Program DriversClimate Program Essential Global Climate Variables (from CCSP & GCOS)

The following essential atmospheric variables are required over land, sea and ice:

1.1 Surface•a. Air temperature•b. Precipitation•c. Air pressure

1.2 Upper-air•a. Earth radiation budget (including solar irradiance)•b. Upper-air temperature (including MSU radiances)•c. Wind speed and direction•d. Water vapor•e. Cloud properties

1.3 Composition•a. Carbon dioxide•b. Methane•c. Ozone

*Including nitrous oxide (N2O), chloroflurorocarbons (CFCs), hydrofluorocarbons (HFCs), sulpher hexaflouride (SF6), and perfluorocarbons (PFCs).

•d. Surface radiation budget•e. Wind speed and direction• f. Water vapor

•d. Other long-lived greenhouse gases *•e. Aerosol properties

1. Atmospheric Variables

•g. Evaporation & evapotranspiration


Program DriversGlobal Climate Variables and NPOESS Products

Global Climate Variables

NPP NPOESS C1 NPOESS C2 NPOESS C3

Air Temperature *VIIRS Land Surface Temperature

CrIMSS Atmos Vertical Temp Profile

*VIIRS Land Surface Temperature

CMIS Land Surface Temperature

CMIS Atmos Vertical Temp Profile








CMIS Atmo Vertical Temp Profile

Precipitation ATMS Rainfall CMIS Precipitation Rate

CMIS Precipitation Type

ATMS Rainfall

CMIS Precipitation Rate


ATMS Rainfall

CMIS Precipitation Rate


Air Pressure

Surface Radiation Budget

VIIRS Absorbed Radiation



ERBS Downward Longwave Radiation

ERBS Downward Shortwave Radiation (Surface)


1. Atmosphere 1.1. Surface





Wind Speed & Direction

Water Vapor CrIMSS Atmos Vertical Moisture Profile

CMIS Atmos Vertical Moisture Profile

CrIMSS Atmos Vertical Moisture Profile




Evaporation & Evapotranspiration

1. Atmosphere 1.1. Surface (continued)





Earth Radiation Budget

CrIS Outgoing Longwave Radiation

VIIRS Outgoing Longwave Radiation


ERBS Outgoing Longwave Radiation (TOA)





Upper Air Temperature




CMIS Atmos Temp Moisture Profile


CMIS Atmos Temp Moisture Profile

Wind Speed and Direction

Water Vapor CrIMSS Atmos Vertical Moisture Profile






1. Atmosphere 1.2 Upper Air





Cloud Properties VIIRS Cloud Base Height

VIIRS Cloud Cover/Layers

VIIRS Cloud Effective Particle Size

VIIRS Cloud Optical Thickness

VIIRS Cloud Top Height

VIIRS Cloud Top Pressure

VIIRS Cloud Top Temperature

VIIRS Cloud Mask

VIIRS Cloud Base Height







VIIRS Cloud Mask

CMIS Cloud Base Height

APS Cloud Particle Size Distribution








VIIRS Cloud Mask









VIIRS Cloud Mask


1. Atmosphere 1.2 Upper Air (continued)





Carbon Dioxide CrIS Carbon Products

CrIS Carbon Products


Methane CrIS Carbon Products



Ozone OMPS Limb Ozone Profile

OMPS Total Ozone

OMPS Nadir Ozone Profile

CrIS Total Column Ozone

OMPS Limb Ozone Profile

OMPS Total Ozone

OMPS Nadir Ozone Profile



1. Atmosphere 1.3 Composition





Other long-lived greenhouse gases

Aerosol properties VIIRS Aerosol Optical Thickness

VIIRS Aerosol Particle Size

VIIRS Aerosol Optical Thickness


APS Aerosol Optical Thickness

APS Aerosol Refractive Index

APS Aerosol Particle Size Parameter





1. Atmosphere 1.3 Composition (continued)


Program DriversEssential Global Climate Variables (from CCSP & GCOS)

2.1 Surface•a. Sea-surface temperature•b. Sea-surface salinity•c. Sea level•d. Sea state•e. Sea ice• f. Current•g. Ocean color (for biological activity)•h. Carbon dioxide partial pressure• i. ocean surface wind & wind stress• j. Surface air temp/humidity•k. Precip (fresh water/salinity flux)• l. Evaporation•m. Fresh water flux from rivers & ice melt•n. CO2 flux across the air sea interface•o. Geothermal heat flux – ocean bottom

2. Ocean Variables

2.2 Sub-surface•a. Temperature •b. Salinity•c. Current•d. Nutrients•e. Carbon• f. Ocean tracers•g. Phytoplankton





Sea Surface Temperature

* VIIRS Sea Surface Temperature


CMIS Sea Surface Temperature





Sea Surface Salinity

Sea Level ALT Sea Surface Height

Sea State

2. Ocean Variables 2.1 Surface





Sea Ice ATMS Sea Ice Concentration

VIIRS Sea Ice Characterization


*CMIS Sea Ice Characterization

ATMS Sea Ice Concentration



ATMS Sea Ice Concentration



Current

Ocean Color VIIRS Ocean Color/Chlorophyll

VIIRS Ocean Color/Chlorophyll



Carbon Dioxide Partial Pressure

2. Ocean Variables 2.1 Surface (continued)





Ocean Surface Wind CMIS Sea Surface Winds

CMIS Sea Surface Winds

CMIS Sea Surface Winds

Wind Stress CMIS Global Sea Surface Wind Stress

CMIS Global Sea Surface Wind Stress

CMIS Global Sea Surface Wind Stress

Surface Air Temp/Humidity


















Precipitation (fresh water, salinity flux)

Evaporation

Fresh Water Flux (Rivers and Ice Melt)

CO2 Flux across air sea interface

Geothermal Heat Flux – Ocean Bottom






Temperature VIIRS Sea Surface Temperature (Bulk Temp)

VIIRS Sea Surface Temperature (Bulk Temp)



Salinity

Current

Nutrients

Carbon VIIRS Depth Integrated Particulate Organic Carbon Concentration

VIIRS Depth Integrated Particulate Organic Carbon Concentration



Ocean Tracers

Phytoplankton

2. Ocean Variables 2.2 Sub-surface


Program DriversEssential Global Climate Variables (from CCSP & GCOS)

•a. Snow cover•b. Glaciers and ice caps•c. Permafrost and

seasonally-frozen ground•d. Albedo•e. Land cover (including

vegetation type)

3. Terrestrial Variables

•f. Fraction of absorbed photosynthetically active radiation (FAPAR)

•g. Leaf area index (LAI)•h. Biomass•i. Land surface temp



Global Climate Variables NPP NPOESS C1 NPOESS C2 NPOESS C3

Snow Cover *VIIRS Snow Cover/Snow Depth

ATMS Snow Cover

*VIIRS Snow Cover/Snow Depth

CMIS Snow Cover/Snow Depth


ATMS Snow Cover



ATMS Snow Cover


Glaciers and Ice Caps

Permafrost and Seasonally Frozen Ground

Albedo VIIRS Surface Albedo

VIIRS Surface Albedo VIIRS Surface Albedo

VIIRS Surface Albedo

3. Terrestrial



Global Climate Variables NPP NPOESS C1 NPOESS C2 NPOESS C3

Land Cover *VIIRS Surface Type/Veg Cover

*VIIRS Surface Type/Veg Cover CMIS Surface Type/Veg Cover

* VIIRS Surface Type/Veg Cover

CMIS Surface Type/Veg Cover

*VIIRS Surface Type/Veg CoverCMIS Surface Type/Veg Cover

Fraction of absorbed photosynthetically active radiation (FAPAR)

Leaf Area Index VIIRS Leaf Area Index

VIIRS Leaf Area Index



Biomass

Land Surface Temperature *VIIRS Land Surface Temperature







3. Terrestrial (continued)


Need for International Partnerships Group on Earth Observations GEO

NRC Panel on Climate Variability and Change July 25, 2005 1 John Bates and Thomas Karl, NOAA’s...

Documents

Transcript of NRC Panel on Climate Variability and Change July 25, 2005 1 John Bates and Thomas Karl, NOAA’s...