Overview of Current S-NPP NOAA Unique Products …...Product and Ancillary Data RDRs, SDRs, TDRs,...
Transcript of Overview of Current S-NPP NOAA Unique Products …...Product and Ancillary Data RDRs, SDRs, TDRs,...
Overview of Current S-NPP NOAA Unique Products and an Introduction to STAR's Research to Operations
Transition Process for JPSS-1 ESPC Products
Co-authors Dr. Priyanka Roy1, Walter W. Wolf 2, Thomas King1,
Shanna Sampson1
1IMSG, 2NOAA/NESDIS/STAR
1 97th AMS Annual Meeting 2017 January 24, 2017
13th Annual Symposium on New Generation Operational Environmental Satellite Systems (13NGOESS) and the Seventh Conference on Transition of Research to
Operations Conference (7R2O)
Outline
• Introduction of the Joint Polar Satellite System (JPSS)
• Overview of NOAA/National Environmental Satellite Data and Information Service (NESDIS)’s NOAA-Unique Products (NUPs) generation process.
• Description of the NESDIS/Center for Satellite Application and Research (STAR)’s NUP transition to operations process.
• Role of the Algorithm Scientific Software Integration and
Transition Team (ASSISTT)
97th AMS Annual Meeting 2017 2 January 24, 2017
JPSS Mission
• The Joint Polar Satellite System is the new generation of polar orbiting satellite system, of which Suomi-National Polar-orbiting Partnership launched in 2011 was the first.
• The satellites and the planned instruments within JPSS are:
January 24, 2017 97th AMS Annual Meeting 2017 3
Satellite Launch Year Instruments
S-NPP 2011
Visible Infrared Imaging Radiometer Suite (VIIRS), Advanced Technology Microwave Sounder (ATMS), Ozone Mapping and Profiler Suite Limb and Nadir (OMPS-L and OMPS-N) , Cross-track Infrared Sounder (CrIS) and Clouds and the Earths Radiant Energy System –Flight Model5 (CERES-FM5)
JPSS-1 4th Qtr FY2017 VIIRS, OMPS-N, ATMS, CrIS and CERES-FM6
JPSS-2 1st Qtr FY2022 VIIRS, OMPS-N, ATMS, CrIS and Radiation Budget Instrument (RBI)
JPSS-3 FY2026 VIIRS, OMPS-N, ATMS, CrIS and RBI
JPSS-4 FY2031 VIIRS, OMPS-N, ATMS, CrIS and RBI
Source : http://www.jpss.noaa.gov/launch_schedule.html
NOAA/NESDIS’s S-NPP & JPSS-1 Products
NESDIS generates higher level products from VIIRS, ATMS, OMPS and CrIS by processing the Raw Data Records (RDRs) :
– The Interface Data Processing Segment (IDPS) • Sensor Data Records (SDRs) • Temperature Data Records (TDRs) • Environmental Data Records (EDRs)
– The NPP Data Exploitation (NDE)
• NOAA Unique Products (NUPs) – All products created within NDE are considered NUPS
• Tailored Products Both IDPS and NDE have been providing critical S-NPP data operationally since 2011.
The entire product generation and distribution system is currently being updated to provide an enterprise solution for product generation from multiple satellites (Transition to Operations (TTO) date – March 2017).
97th AMS Annual Meeting 2017 4 January 24, 2017
Overview of ESPDS Operations in JPSS Common Ground System (CGS) IDPS Block 2.0 era
97th AMS Annual Meeting 2017 5 January 24, 2017
JPSS CGS IDPS
S-NPP, JPSS-1
Svalbard/McMurdo
Environmental Satellite Processing and Distribution System (ESPDS)
NDE – Product Generation Enterprise Shared Storage
Product Distribution and Access
External and
Internal Users
National Centers for
Environmental Information
(NCEI) Archive
xDRs, NUPs, Tailored Products
NUPs
xDRs, NUPs, Tailored Products
Ancillary Data file Availability Notification
Product and Ancillary Data files
Product and Ancillary Data files
RDRs, SDRs, TDRs, EDRs
Currently, with JPSS CGS IDPS Block 1.2, NDE distributes only the NUPs and Tailored Products to users and NCEI for archive using its Product Distribution Services.
TTO is scheduled for March 2017
S-NPP Products after Block 2.0 TTO
Products that are already operationally available from NDE 1.0 – Atmospheric Vertical Moisture Profile (CrIS/ATMS) (NUCAPS) – Atmospheric Vertical Temperature Profile (CrIS/ATMS) (NUCAPS) – Infrared Ozone Profile (CrIS) (NUCAPS) – Outgoing Longwave Radiation (CrIS) (NUCAPS) – Green Vegetation Fraction (VIIRS) Global (NetCDF) – Green Vegetation Fraction (VIIRS) Regional (NetCDF) – Green Vegetation Fraction (VIIRS) Global (GRIB2) – Green Vegetation Fraction (VIIRS) Regional (GRIB2) – Vegetation Health Index Suite (VIIRS) – VIIRS Polar Winds (VIIRS) – Cloud Liquid Water (ATMS) (MiRS) – Imagery (ATMS) (MiRS) – Land Surface Emissivity (ATMS) (MiRS) – Land Surface Temperature (ATMS) (MiRS) – Moisture Profile (ATMS) (MiRS) – Rainfall Rate (ATMS) (MiRS) – Sea Ice Concentration (ATMS) (MiRS) – Snow Cover Depth (ATMS) (MiRS) – Snow Water Equivalent (ATMS) (MiRS) – Temperature Profile (ATMS) (MiRS) – Total Precipitable Water (ATMS) (MiRS) – Sea Surface Temperature (VIIRS) ACSPO – Active Fires (VIIRS)
January 24, 2017 97th AMS Annual Meeting 2017 6
* More information about these products is provided in the poster titled “JPSS Environmental Satellite Processing Center Data Products” - Bonnie Reed and T. Schott, A. Layns, and S. L. Bunin (1336-4E)
S-NPP Products after Block 2.0 TTO
Additional products that will be available with NDE 2.0 – Cloud Mask (VIIRS) – Cloud Phase (VIIRS) – Cloud Cover Layers (VIIRS) – Cloud Height (Top and Base) (VIIRS) – Cloud Top Pressure (VIIRS) – Cloud Top Temperature (VIIRS) – Cloud Type (VIIRS) – Cloud Optical Depth (VIIRS) – Cloud Particle Size Distribution (VIIRS) – Cloud Ice Water Path – Cloud Liquid Water (VIIRS) – Volcanic Ash Detection and Height (VIIRS) – Aerosol Detection (VIIRS) – Aerosol Optical Depth (VIIRS) – Aerosol Particle Size (VIIRS) – Snow Cover (VIIRS) – Ice Concentration (VIIRS) – Ice Age/Thickness (VIIRS) – Ice Surface Temperature (VIIRS)
January 24, 2017 97th AMS Annual Meeting 2017 7
* More information about these products is provided in the poster titled “JPSS Environmental Satellite Processing Center Data Products” - Bonnie Reed and T. Schott, A. Layns, and S. L. Bunin (1336-4E)
S-NPP Products after Block 2.0 TTO
Deferred Products – will become available in 2017 – Albedo (Surface) EDR (VIIRS) – Land Surface Temperature EDR (VIIRS) – Surface Reflectance EDR (VIIRS) – Vegetation Indices EDR (VIIRS) – Carbon Dioxide (CO2) (CrIS) (NUCAPS) – Carbon Monoxide (CO) (CrIS) (NUCAPS) – Methane (CH4) (CrIS) (NUCAPS) – Ocean Color/Chlorophyll (VIIRS) – OMPS Limb Profile EDR – OMPS Limb Profile SDR – Ozone Nadir Profile (OMPS-N) – Ozone Total Column (OMPS-N)
January 24, 2017 97th AMS Annual Meeting 2017 8
* More information about these products is provided in the poster titled “JPSS Environmental Satellite Processing Center Data Products” - Bonnie Reed and T. Schott, A. Layns, and S. L. Bunin (1336-4E)
STAR’s Science Algorithm Transition to Operations Process for NUPS
January 24, 2017 97th AMS Annual Meeting 2017 9
SPSRB Process
• The NUPs follow the Satellite Products and Services Review Board (SPSRB) review process for transitioning products into operations.
• The SPSRB process has been updated following information from STAR’s Enterprise Product Lifecycle (EPL) Process which is a Capability Maturity Model Integration (CMMI) Level -3 Process.
• This SPSRB process is an established and mature process for efficiently and successfully transitioning science algorithms (SAs) to operations.
• The SPSRB Process Improvement Working Group (SPIWG) oversees the process.
January 24, 2017 97th AMS Annual Meeting 2017 10
* More information about these products is provided in the presentation titled “Transitioning NOAA Satellite Products from Development to Operations” - Stacy L. Bunin and K. Shontz and B. Reed (5.6 -34B)
SPSRB Standards and Guidelines
• Standards & Guidelines – Process Review Standards – NDE Delivered Algorithm Package (DAP) Standards is an ICD between STAR
and NDE. – SPSRB Coding and Scripting Standards – Data Format Standards
• Archive Guidance – Guidance on NCEI Interactions
– NCEI uses multiple storage systems to archive data including the Comprehensive Large Array-Data Stewardship System (CLASS), which is jointly governed by NCEI, OSGS and OSPO.
– NCEI has laid down procedures to provide guidance to STAR and OSPO personnel for archiving satellite data and products adhering to the NOAA Procedure for Scientific Records Appraisal and Archive Approval (http://www.nosc.noaa.gov/docs/products/NOAA_Records_Brochure_4_pages_Dec_9.pdf ).
January 24, 2017 97th AMS Annual Meeting 2017 11
Project Documentation
• SPSRB Documentations – Generated and maintained by Development Team – Algorithm Theoretical Basis (ATBD) – External Users Manual (EUM) – Internal Users Manual (IUM) – System Maintenance Manual (SMM)
• Review Process Documentations – Generated and maintained by STAR
ASSISTT – Requirements Allocation Document (RAD) – Review Item Disposition Spreadsheet (RID)
January 24, 2017 97th AMS Annual Meeting 2017 12
Key Stakeholders for the NUPs
– NOAA JPSS Office (NJO) • Provides funding • Defines the Level 1 Requirements for the products
– Office of Satellite Ground System (OSGS) • Enables development of satellite products and their transition to operations
– Center for Satellite Applications and Research (STAR) • Develops satellite products ensuring they meet all product and process requirements
– Office of Satellite and Product Operations (OSPO) • Manages and directs the operations of the central ground facilities which ingest, process and
distribute environmental satellite data and derived products
– National Centers for Environmental Information (NCEI) • Archives the satellite data and products
– Users • NOAA users, DoD, International Partners
97th AMS Annual Meeting 2017 13 January 24, 2017
NUP TTO Process
97th AMS Annual Meeting 2017 14
Developmental Phase
Process Reviews • Requirements Review (STAR) • Preliminary Design Review (STAR) • Critical Design Review (STAR) • Unit Test Readiness Review (STAR) • Software Code Review (OSPO) • Algorithm Readiness Review (STAR)
Tasks • Initial Archive Requirements identified • Quality Monitoring Concept Defined • Long-term Maintenance Concept Defined • Development processing system defined • Initial Information Technology (IT) Security concept defined • Test case processed • Code is prepared for implementation • Final Archive requirements identified • Operational and backup processing defined
Pre-Operational Phase
Process Reviews • Operational Readiness Review (OSPO) • SPSRB Briefing (OSPO and STAR)
January 24, 2017
Tasks • Operational and backup processing capabilities in place • Final IT Security Concept Defined • Pre-operational product output evaluated & tested • Code transitions to operations; all documentation is complete • Operational and backup capabilities reach ops status
ASSISTT
• ASSISTT is the bridge between science at NOAA/NESDIS/STAR and operations. Enforce a standard
software quality. Transparently
communicate between science and operations.
Science • Algorithm Design
• Prototype • Improve • Validate
ASSISTT • Research to Ops
• Standardize • Optimize • Quality Assurance
Operations • Product Execution
• Generate • Monitor • Distribute
ALGORITHM SCIENTIFIC SOFTWARE INTEGRATION AND SYSTEM TRANSITION TEAM
15 January 24, 2017 97th AMS Annual Meeting 2017
The Problem Many scientists prefer interpreted
languages (IDL and Matlab) or even older languages (Fortran 77). Fortran 90/95 or C/C++ expertise does not always exist. Code doesn’t account for operational
concerns such as run time, memory usage, disk I/O, error checking. Code is often written by a mixture of
programmers with varying styles and abilities.
Paths are hardcoded and algorithms
assume the data they need will be in arranged in a particular data tree. Executable code makes system calls
(assuming a certain OS). Code is often not well documented. Code only works with certain
compilers. Code uses non-standard functions.
Operations is tasked only to receive, generate, and monitor products – not upgrade code to operational standards.
Scientists write code that gets transitioned to operations, but:
16 January 24, 2017 97th AMS Annual Meeting 2017
Role of STAR ASSISTT
Act as the “Middle Man” • Trace all requirements and document and update all the requirements in the
project RADs • Perform product and process QA. • Lead all reviews • Provide software guidance associated with the algorithm development and
updates • Provide Algorithm Package guidance • Test and verify DAPs before delivery to NDE • Provide reporting to management • Provide coordinated schedules and schedule oversight • Add algorithm software to STAR CM system • Deliver the preliminary and final DAPs to NDE
97th AMS Annual Meeting 2017 17 January 24, 2017
STAR ASSISTT
STAR Science Team NDE DAP DAP
ASSISTT Staff
• ASSISTT Project Manager • Integration Leads • QA Leads • Algorithm Integrators • Algorithm Change Experts • Scripting and Transition Experts • Regression and System Testers • Configuration Manager • IT Cluster Managers
18 January 24, 2017 97th AMS Annual Meeting 2017
Summary
• JPSS has established an enterprise approach for satellite product generation.
• The updated ground system is getting ready for transition to operations for generating and distributing products to users.
• STAR ASSISTT is all set to support the overall efforts with its tried and tested pathway for transitioning products swiftly and efficiently into operations.
January 24, 2017 97th AMS Annual Meeting 2017 19
Backup
January 24, 2017 97th AMS Annual Meeting 2017 20
JPSS ESPC Products Project Element Observational Parameter Project Element Observational Parameter
NUCAPS CrIS Radiance Thinning Polar Winds VIIRS Polar Winds
NUCAPS Temperature Profiles Ocean Color/Chlorophyll Ocean Color
NUCAPS Moisture Profiles Ocean Color/Chlorophyll Optical Properties - Absorption
NUCAPS Infrared Ozone Profile Ocean Color/Chlorophyll Optical Properties - Backscattering
NUCAPS CO2 Ocean Color/Chlorophyll Optical Properties – Chlorophyll Fluorescence
NUCAPS CO Ocean Color/Chlorophyll Chlorophyll
NUCAPS CH4 ACSPO Sea Surface Temperature
OLR CrIS Outgoing longwave radiation Active Fires Fire Geolocation
MiRS Land Surface Temperature Active Fires Fire Radiative Power
MiRS Cloud Liquid Water Active Fires Fire Mask
MiRS Rainfall Rate Green Vegetation Fraction Green Vegetation Fraction – Global
MiRS Sea Ice Concentration Green Vegetation Fraction Green Vegetation Fraction – Regional
MiRS Snow Cover Vegetation Health Suite Vegetation Condition Index
MiRS Snow Water Equivalent Vegetation Health Suite Temperature Condition Index
MiRS Total Precipitable Water Vegetation Health Suite Vegetation Health Index
MiRS Land Surface Emissivity
MiRS Moisture Profile
MiRS Temperature Profile
21 January 24, 2017 97th AMS Annual Meeting 2017
JPSS ESPC Products
Project Element Observational Parameter Project Element Observational Parameter
Total Ozone Total Column Ozone MTCP Tropical Cyclone Maximum Wind
Ozone Nadir Profile Ozone Nadir Profile MTCP Minimum Sea Level Pressure
Surface Reflectance Surface Reflectance MTCP Radii of 34 kt winds in NE, SE, NW, and SW
Vegetation Indices Vegetation Indices MTCP Radii of 50 kt winds in NE, SE, NW, and SW
Reformatting Toolkit CrIS Radiances (BUFR) Reformatting Toolkit SST (BUFR)
Reformatting Toolkit ATMS Radiances (BUFR) Reformatting Toolkit Aerosol Optical Depth (BUFR)
Reformatting Toolkit VIIRS Radiances (BUFR) Reformatting Toolkit VIIRS Polar Winds (BUFR)
Reformatting Toolkit Nadir Profile Ozone (BUFR) Reformatting Toolkit GVF (GRIB2)
Reformatting Toolkit Total Column Ozone (BUFR)
22 January 24, 2017 97th AMS Annual Meeting 2017