Obs-nudging-FDDA

An overview

Obs-nudging-FDDA Overview

• Obs-nudging-FDDA vs. NWP forecast systems

– Overview– Restarts and cold starts– Cycling

• RT vs. Climo and Ensembles– Real Time Four Dimensional Data Assimilation

• Has evolved to include other variations • CFDDA

– Climate FDDA• E-RTFDDA

– Ensemble RTFDDA

• Obs-nudging-FDDA inputs and outputs• Background and history within NSAP

RTFDDA: What is it?

• Real Time Four Dimensional Data Assimilation system

• 3-D plus time system combining observational analysis including obs nudging and a forecast model

• Can use either MM5 or WRF for the forecast “engine”

RTFDDA “the science” concept

• PSU/NCAR MM5 or NCEP/NCAR WRF - based Four Dimensional Data Assimilation (FDDA) system run in a rapid update cycle.– Concept of dynamically

combining observational and model data in space and time, first developed by David Stauffer and Nelson Seaman (Stauffer & Seaman 1994).

• Uses newtonian relaxation technique to “nudge” the model solution towards the observed values.

• Provides continuous analyses and short term forecasts (~48hrs) every 3hrs.

WRF/MM5-RTFDDA

Multi-scale Modeling

Cold start

t

Forecast

FDDA

New 12 - 48 h forecasts every 1 - 12 hrs, using all obs up to “now”

TAMDAR

MESONETs

GOES

Wind Profs

All WMO/GTS Radars

Etc.

ACARS

obs

Example Model Domains

• Cold Regions Test Center (CRTC)

• Nested model domains of higher resolution

• Most 4DWX applications use 3 or 4 nested domains

• Domain1 is 30km, nest resolution is 3:1 ratio in MM5

• Domain 2 is 10km• Domain 3 is 3.3km• Domain 4 is 1.1km

RTFDDA: Why is it different?

• Cold Start vs. Continuous– Most forecast models run 2-4 times per

day, with each run “starting over”. This is a cold-start, and the model equations need time to spin-up and stabilize some parameters, esp. moisture.

– RTFDDA runs “continuously”, a cold-start is only done once a week, so limited model spin-up time is involved. How often to cold-start is a system-tuning parameter, but weekly is typical for real-time systems.

Cycles and Restarts

• Obs-nudging-FDDA systems do not really run continuously, they stop and restart

• Each stop/restart is called a “cycle”• Cycles are defined in the project setup/configuration

and can be anything from 1 hour to 7 days or more. Typical real-time cycles are 3-hours

• Each cycle, the model reads a “restart” file from the previous cycle to re-initialize the model state variables, so that the model does not have to start it’s equations & variables from initialization, as it does at a cold-start time.

• By doing a restart, the cycle allows time for a forecast to be generated, and for new observations to be collected and processed for the next cycle.

Obs-nudging-FDDA cycles

• Current ATEC RTFDDA configurations run cycles every 3 hours, at 02Z, 05Z, 08Z, etc. This timing is based on data availability.

• 1, 2 or 6 hour cycle intervals are sometimes used– For rapid real-time updates (1 or 2 hour cycles)– For longer forecast output options (6 hour cycles)

• Precise cycle timing depends on – cluster compute-power– forecast length goals – data availability

• Output products generally use “the latest available” for the requested valid time– This assumes that the latest available is the most accurate– Each cycle’s output may be saved/viewed if desired

Anatomy of a CycleRTFDDA

Pre-process

Obs

FDDA Final Analysis

MM5 running

Process Add’l

Obs

Prelim + Forecast …

MM5 running…

05:00 07:0006:00 08:00

Cycle Starts Longer or shorter for different cycle intervals

Cycle OutputRTFDDA

02 03 04 05 06 07 08 09 10 11 12 13 14 15 16 17 18 19 20 21 22

Valid Time

Cyc

le

Final Analysis

Preliminary Analysis

Forecast

05Z

08Z

11Z

14Z

17Z

20Z

Cycle Time

Obs-nudging-FDDA: Why is it different?

• Observation Analysis– Most models will incorporate observations using

analysis nudging at their cold-start time only (0Z, 6Z, 12Z, etc), sometimes using a spin-up data assimilation leading up to that time

– Obs-nudging-FDDA incorporates observations as it is running in real-time, at any time they are available

• This is a very simplified look at the differences! Remember “data assimilation is obs nudging” and “cycling”…

Obs-Nudging-FDDA: Various Flavors

• RT-FDDA: RealTime FDDA– Runs with real-time cycles– Rapid (1-6hr cycles) analysis updates– Short term forecasts (24-48hrs)– High resolution nested domains

• C-FDDA: Climatology FDDA– Runs for historical time period (days or months)– Analysis only, no forecasts (generally)– High resolution nested domains– Years of climatology are combined for statistical output (mean July temperatures,

etc)• E-RTFDDA: Ensemble RT-FDDA

– Ensemble of model configurations (WRF, MM5, IC/BC variations, etc)– Real-time cycles– Analysis plus forecasts– High resolution nested domains (but usually not as much detail as single RTFDDA

system)– Member outputs are combined for statistical output (mean, variance of temperature

at Station #1, etc)• E-CFDDA: Ensemble C-FDDA

– Coming Soon! No-one has computers or disks big enough (yet!!!)

Obs-nudging-FDDA: Input Data

• All Mesoscale NWP systems need input data, they do not start from zero.

• All the variations of NSAP’s NWP systems need the same classes of input data

• Depending on what type of job you are running, and where the domains are located and the goals of your system design, etc, the input data sources will be different.

• But the similarities are greater than the differences!

Input Data: IC/BC• Obs-nudging-FDDA requires a low-resolution model to

initialize. • These data are used for Initial and Boundary Conditions

for the mesoscale model (MM5 or WRF).• RTFDDA uses NAM or GFS, generally, but other data

sources have occasionally been used.• CFDDA uses NNRP due to the long time scales needed.• A case study (run one particular day of one particular

year for special scientific study) will generally use NAM or GFS if it is available in the archive.

NAM: North American Meso Model (from NCEP)GFS: Global Forecast System Model (from NCEP)NNRP: NCAR/NCEP ReAnalysis ProductNCEP: National Centers for Environmental Prediction

Input Data: Obs• Observations are critical to the performance of any of the Obs-nudging-

FDDA systems (RTFDDA, CFDDA or E-RTFDDA).• Observation datasets for RTFDDA:

– WMO (metars, soundings) – (LDM from NOAAPort, as backup to MADIS)– Range obs (scp from ATEC range machines)– Satellite derived winds (included in MADIS)– ACARS (included in MADIS)– MesoWest (included in MADIS)– The MADIS dataset from FSL combines most of these except the ATEC

range data (or other special data, such as the Torino or Athens obs)• CFDDA uses ADP obs from the NCAR archives• None of these will cause a “failure”/abort of RTFDDA, if they are

missing, but will impact it’s accuracy and performance!

ADP: Automated Data Processing (historical obs dataset from NCEP)MADIS: Meteorological Assimilation Data Ingest SystemWMO: World Meteorological OrganizationACARS: Aircraft Communication Addressing and Reporting SystemFSL: Forecast Systems Laboratory (part of NOAA)LDM: Local Data Manager

Typical RTFDDA dataflow

data input host

ETA

GFS

WMO

Satwinds

NPN

BLP

ACARs

RAWs

MADIS

<range>

Other

NCEP

Other

Ranges

NOAA

NOAAPort

FSL

model cluster

RTFDDA cycles

Output Products

web server

Web accessible Output Products

•UGUI gifs

•NAPS

•MEDOC

•FDDA sites

data host

MDV server/files

•JVIZ. etc

Gridded Climatology Files

das-input*

range DAS

gs-c3

crtc-c3

ypg-c2atc-c2

wsmr-c2

dpg-c2

4dwx.org

range DAS

ATEC RTFDDA operationsMM5-RTFDDA --- ATC, CRTC, DPG, EPG, RTTC, WSMR, YPG

WRF-RTFDDA --- ATC, CRTC, DPG, WSMR, YPG

smac-c2

Safari requests – PMRF, etcgs-c2

Web Pages

Web PagesJVIZ

At NCAR:

At Ranges:

RTFDDA Operations

• ATEC is the largest user of RTFDDA operationally, but not the only user

• Specific operational setup may vary, depending on cluster site, user output needs, data input requirements, etc.

• The RTFDDA system on any cluster for any job will be very similar in setup and configuration

Other Operations• RTFDDA

– Pentagon Shield– Several DTRA (SLC, Athens and Torino Olympics) and NGIC

special operations – FAA winter program– UAE rainfall project– Wyoming rainfall project– Saudi rainfall project– West Africa demonstration project– IAF/TAU demonstration program– NASA NYC remote sensing study program

• CFDDA – DTRA global climatology– NGIC regional climatology tool (GCAT)– ATEC range climatology

Background and History: 4DWX and modeling

• ATEC project started in 1996/97• High resolution forecasting was a primary

component of the program• Initial capabilities included a twice-a-day MM5

forecast, plus a retrospective FDDA run on an SGI for 3 ranges

• Goal was two-fold:– A high resolution mesoscale forecast for operations,

including running coupled applications that required met/weather inputs

– A 3-D range climatology computed in “real-time”, year after year.

Background and History:RT-FDDA

• RT-FDDA system design was developed to address both needs– 3-D data assimilation system provided the “best representation of

the state of the atmosphere” over the range– Accurate, high resolution forecasts for range users:

• Forecasts• Current conditions on the range (including where there are no

instruments)• Input weather data for coupled applications such as transport/dispersion,

sound propagation, rocket trajectories, parachute drop, etc• Initial RTFDDA system operating for DPG in late 2000, deployed

incrementally to 4 other ranges by 2002• Many special operations have been supported throughout the

years• EPG and RTTC added in 2006• WRF-RTFDDA added in 2006

Background and History:Influence on System Design

Decisions• The design of the RT-FDDA cycles and the C-FDDA data

assimilation are driven by both scientific principles and user needs

• Data availability (input) and user requirements (output) drive choices in timing and system selections throughout the systems

• For example:– 3 hour cycles:

• the GOES satellite winds were originally available every 3 hours (now hourly)

• the original clusters could produce a reasonable forecast length in 3 hours (24-30 hours)

• mesoscale phenomena timescales also supported a 3-hour cycle– Data sparse regions (UAE), or users needing longer forecasts

(PMRF) would get a 6-hour cycle setup– Data rich areas, or users interested in rapid updates willing to trade

a shortened forecast length (or as computers gained computational power), would run a 1-hour cycle (a DPG-PWIDS demonstration)

Background and History: Climatology

• Climatology was an original goal of the ATEC project, and was started with an FDDA system running retrospectively each day in 1997 (for DPG)

• NGIC was the first sponsor to develop a more generalized tool to use the cluster computer power and historical climate data to run high-resolution climatologies (NGIC-GCAT)

• ATEC has an historical record of more than 10 years of FDDA and RTFDDA/final analysis data from Dugway– is it comparable to a GCAT/CFDDA job run for 10 years? – Why or why not? – Was it a reasonable goal when it was started?– Is there still benefit to archiving the real-time analysis output,

when a tool like CFDDA is available?

Example System

• www.4dwx.org/images/atc/rtfdda_gmod/model.html• There is a convention for naming and finding

RTFDDA output and webpages!– Where might the WSMR MM5 job output be found?

• A “job” generally refers to a configuration, which includes the geographic location, the cluster it’s running on, all the options selected, real-time or climo or case-study, ensemble or not,etc.

• Each job will generally produce some web graphics, and some status reports, and other optional output, depending on the configuration.

Sample Web Page output, showing various product choices

Cycle Time

“Range”

Output Valid Time

Fcst Hours

Questions

• What cluster is this system running on?

• How many domains are configured?

• What is the typical forecast length produced?

• How often is it cycling?

• Is it running MM5 or WRF?