Post on 14-Dec-2015
Ed Tollerud
In collaboration with ESRL, RFC’s, WFO’s, and CDWR
HMT-DTC Task
DTC Goals and the HMTProvide early step up for front-line
forecasters vis-à-vis ensemble usage
Demonstrate usefulness of prototype ensemble forecast systems; thereby
provide long-range guidance for ensemble forecast systems
Enhance and motivate MET development
HMT-West ensemble QPF evaluation is current focus to attain these goals
Major accomplishments - past 6 months
Online WRF ensemble verification in near-real-time:Regionalization of HMT domain30-day summariesObservational data options
Object-based (MODE/MET) verification of historical atmospheric rivers
Direct comparison between GFS and WRF ensemble mean
Online QPF verification for HMT-West
Equitable Threat Scores (ETS) for ensemble members
Forecasts initiated at 1200 UTC 1/17 2010
Ensemble mean (black) GFS (green)
30-day summary boxplots, ETS
January 2010 Large HMT domain Intensity
thresholds: .01,.1,.5,1,3 in
Planned activities - next 6 months
Revise and expand winter exercise verification demonstration website
Prepare and present preliminary results from 2009-2010 winter exercise verification
Develop probabilistic capabilities for QPF in the HMT
Continue retrospective study of MET/MODE applications to atmospheric river forecasts
Anticipated major accomplishments - next 6 months
Following features added to HMT-West winter verification exercise: 1-2 new additional baseline models (SREF,
NAM,…)Time series display of MODE attributes Displays of reliability and ROC diagrams6h gage accumulation options
Plans for AOP 2011Incorporate spatial verification statistics for
ensemblesImplement METviewer online displaysCollaborate with DET to apply MET-based
probabilistic verification techniquesDevelop procedures to assess gridded QPE
products and their impact on verificationExtend verification to additional
meteorological fieldsImplement prototype AR verification utility
Anticipated major accomplishments – AOP 2011Revised and expanded verification options
appropriate for HMT (West and East)Comprehensive suite of probabilistic
verification utilities applied to QPF for HMTObject-based verification techniques
implemented for AR forecastingReports and presentations from HMT-West
verification activities
Resource requirements for AOP 2011Staff Category FTE
Scientists 1.0
Software Engineers 0.5
Students 0.1
Scientists Software Engineers
Ed Tollerud John Halley Gotway
Tara Jensen Paul Oldenburg
CIRES associate
Randy Bullock
RAL statistician Brent Ertz
Tressa Fowler Chris Harrop
Barb Brown
Non-salary Cost
Travel $7 K
Publications $2 K
Challenges (Science and Resources)Establishing synergistic roles of DET and DTC
vis-à-vis ensemble verificationDetermining potential use and advantages for
regional and episodic verificationDeveloping appropriate down-scaling methods for baselining and comparisonImproving forecaster access to results of evaluation (ALPS? AWIPS?)Validating long-range point precipitation forecast for Tillamook for September 2011
Tara Jensen
HWT-DTC Collaboration
Collaborating with:NOAA: SPC, NSSL, HPC, AWC, ESRL, EMCUniversities: OU/CAPSNCAR: MMMUCAR: COMET
HWT-DTC Collaboration Goals• Explore a different R2O Paradigm
• “O” is Forecasters, WFOs, and Prediction Centers• Evaluate convection allowing models on leading edge
of NWP• Gather robust dataset for ensemble and deterministic
studies• Datasets and algorithms leveraged by DET
• Demonstrate utility of MET-based objective evaluation in a forecast environment• Provide near real-time evaluation during Spring Experiment• Application of MET feeds back to MET development
• Facilitate faster R2O through focused evaluation of key topics identified during Spring Experiment• Perform retrospective studies using DTCs METviewer
Major accomplishments - past 6 months
Enormous expansion of near real-time evaluation capability Evaluation of 30 models during Spring Experiment
(CAPS ensemble+3 operational baselines) 10 Deterministic and 4 Ensemble products evaluated
using traditional and spatial verification methods. Three additional research models available for
retrospective studies
DTC staff participation in each week of Spring Experiment 2010
Manuscript accepted by AMS Weather and Forecasting.(Kain et. al, 2010 on Impact of radar assimilation on short-term forecasts)
2010 HWT Spring Experiment – 30 models
CAPS SSEF Ensemble PM Mean
CAPS 1 km Model
CAPS SSEF
ARW-CN (control w/
radar assimilation)
3 km HRRR
12km NAM
CAPS SSEF ARW-C0
(control w/o radar
assimilation)
• Assimilation continues to show greater skill over no-assimilation
• CAPS Ens. Mean generally hashigher skill than 1km deterministic and Operational baselines
Planned activities - next 6 monthsComplete re-run of
Spring Experiment evaluation
Add additional probabilistic measures(Brier Skill Score, ReliabilityDiagrams, and ROC Curves)
Perform retrospective evaluation of all members/products
Probabilistic Verification ofEnsemble Products (for a Single Run)
Brier Score and decomposition
CAPS SimpleCAPS NeighborhoodSREF Simple
Anticipated major accomplishments - next 6 monthsAnalysis for presentations (Severe Local Storms
and AMS)Investigate physics perturbations on Accumulated Precip.
and ReflectivityEvaluate performance of probability products for QPFDetermine meaningful metrics from object-oriented
evaluation of Radar Echo Top
Report on impact of Radar Assimilation on 0-12 hr forecast
Targeted analysis using METviewer to facilitate 2011 Spring Experiment planning
Quality-checked and complete dataset available to collaborators for their own investigation
Plans for AOP 2011Contribute to R2O into SPC, HPC and
AWC? by providing objective evaluation for their Spring Experiment foci:SevereConvective Initiation/LightningQPF/Heavy RainEnsemble Product Evaluation
Perform retrospective studiesProvide feedback to researchers on ensemble
configuration and product generationLeverage DET Verification ModuleHelp develop link between DET and HWT
Anticipated major accomplishments – AOP 2011Near real-time objective evaluation for:
Simulated Satellite Lightning ProductQPF and Probabilistic (PQPF) fieldsReflectivity (if desired)
Evaluation products available within 30 min – 2 hours of valid time (rather than 6-18 hours)
DTC staff member participation during each week of Spring Experiment
Submit Journal article on QPF/PQPF evaluation
Resource requirements for AOP 2011Staff Category FTE
Scientists 0.54
Software Engineers 0.31
Students 0.15
Scientists Software Engineers
Tara Jensen {System Engineer}
Michelle Harrold
Paul Oldenburg
Jamie Wolff* John Halley Gotway
Isidora Jankov* Brent Ernst
Tressa Fowler*
Lisa Coco
Non-salary Cost
Travel $9 K
Workshops $1 K
Publications $1.5 K
Other (see below)
Compute Resources: HPC facility like NOAA-Jet system or NCAR Bluefire systemGoal is to use DET Verification Module.
* Spring Experiment participant
ChallengesHow to balance expectations with budget
HWT Collaboration started smallSuccess has garnered more requests and expectationBudget has not grown
How to serve the needs of additional testbedsAWC planning to start separate Aviation Weather Testbed
Indicated interest in DTC participation - How do we fund activity?HPC planning to continue using HWT facility
What if their evaluation needs expand?
Defining HWT-DTC Collaboration and DET interactionWhat is appropriate evolution of these complimentary
tasks?
Zoltan Toth
DTC Ensemble Testbed
Collaborators: NOAA/NCEP/EMC
DET website: http://www.dtcenter.org/det
Goals for setting up new DET task
Develop and maintain DET infrastructureSupport its use by community
Establish NCEP operational system as benchmark
Test and evaluate new community methods
Transition successful methods to NCEP and other agencies
Link up with ensemble work in other test-beds
Module 1: Configuration
Module 2: Initial
Perturbations
Module 4: Statistical
Post-Processing
External Input(HMT, HWT, HFIP,
etc)
Module 3: Model
Perturbations
Module 5: Product
Generation
Module 6: Verification
DET M
OD
ULES
Major accomplishments - past 6 monthsInitial plans developed for
Overall architecture of DET infrastructureEach of 6 modulesTest and Evaluation
Initial tests for establishing basic capabilities for 2 modulesConfigurationInitial perturbations
Collaboration with other DTC tasks / projectsHMT – Joint plans for testing DET & HMT ensemble
generationHWT – Joint plans for evaluation of 2011 HWT ensemble
OutreachOrganized DET Workshop and engaged with WRF Ensemble
WGActivities coordinated with NCEP/EMC via regular meetings
DET Workshop
Organized by DTC with input from WRF Ensemble WGAug. 18-19, Boulder
Excellent opportunity for connecting DET with community50 plus participants from community & agencies
Total of 6 Breakout sessions focused onConfigurationPreliminary plans for testing
Extensive set of important recommendationsTo be used for updating DET plans
WRF Ensemble WG meeting, Aug 20Additional recommendations
Planned activities - next 6 months
Complete planning for overall design and 6 modules
Develop and test initial perturbation methodsCollaborate closely with NCEP/EMC for their
next implementationComplete protocol for T&EContinue coordination / collaboration with
HMT & HWTLeverage winter 2010/11 HMT ensemble for
DET testingPrepare for joint evaluation of 2011 HWT
ensemble
Anticipated major accomplishments - next 6 monthsDetailed plan for infrastructure – overall
design plus 6 modulesBasic capability for modules 1-2
Initial tests contributing to next NCEP implementation
In collaboration with NCEPProtocol for T&E & basic capability for
verification module (#6)Joint plans / activities with HMT & HWT
Testing of generation & verification of ensemble (HMT)
Evaluation plan of ensemble (HWT)
Basic capability – Functionality tested / supported
Benchmark – Methods used in operations can be replicated
Plans for AOP 2011Benchmark for module 2
Contribute to next NCEP implementation; Replicate next NCEP configuration
T&E for module 2Use T&E protocol and verification module
Contribute to NCEP implementation for module 2 (FY11-12)Test improved method, contribute with algorithm & software
Basic capability for module 3Initial tests with NEMS compatible and WRF models
Contribute to design and evaluation of HMT & HWT ensemblesCollaborative effort
Anticipated major accomplishments – AOP 2011Benchmark for module 2
Milestone: Build initial perturbation module replicating next NCEP SREF implementation
T&E for module 2Milestone: Complete tests of new initial perturbation configuration
Contribute to NCEP implementation for module 2 (FY11-12)Milestone: Collaborate with NCEP scientists and contribute to
development of new initial perturbation methodBasic capability for module 3
Milestone: Demonstrate basic capabilities for model-related uncertainty (NEMS / WRF)
Contribute to design and evaluation of HMT & HWT ensemblesMilestones:
Test DET benchmark configuration for initial perturbation module in HMT ensemble
Contribution to evaluation of HWT ensemble
Resource requirements for AOP 2011Staff Category FTE
Scientists 2.1
Software Engineers 0.8
Scientists Software Engineers
Barbara Brown Paula McCaslin
Isidora Jankov Linda Wharton
Tara Jensen Eugene Mirvis
Ed Tollerud Ann Holmes
New Hire (DET Lead)
Non-salary Cost
Travel $8 K
Workshops $15 K
Publications $2 K
ChallengesIdentify CPU resources for computer intensive DET work
Internal GSD & NCAR resources very limitedPrepare & submit proposals for external resources
Agencies and community desire quick setup of & extensive testing by DETMismatch between funding and expectationsHow can DTC build benchmarks faster?How can DTC engage other agencies / programs / projects?
HFIP, NUOPC, NEXTGEN, FAA, AFWA, etc
Contingency plan assuming additional fundingAccelerate integrated development of & community support
for modules 4-6Basic capability for modules 4-5 – AOP11Benchmark for modules 4-6 – AOP12
DET TIMELINE – STARTING MARCH 2011AOP11 deliverables – Basic capabilities, benchmarks
Basic capability for module 3Benchmark + T&E for module 2Contribute to NCEP implementation for modules 2 (FY11-12)Contribute to design and evaluation of HMT & HWT ensembles
AOP12 deliverables – Benchmarks, T&EBenchmark + T&E for module 3Basic capability for modules 4-5T&E of community methods for module 2
AOP13 deliverablesBenchmark for modules 4 & 6Contribute to NCEP implementation for module 4 (FY13-14)T&E of community methods for all modulesUpdate benchmarks
Laurie Carson
DTC NEMS
Collaborators: NOAA/NCEP/EMC, NOAA/ESRL
DTC NEMS Goals
Define community software goals for the NEMS modeling framework and operational model configurations
Gain expertise in the NEMS framework, by participating in and assisting the on-going development, with a community support focus
Develop an O2R and R2O ops-concept that supports the needs of both the research and the operational user
Progress to-date: past 6 monthsExplore approaches and ideas to help define DTC NEMS
community software goals and timeline On-going: internal (DTC) and external (EMC, NCAR, GSD)
discussionsNEMS software status and near-term plans at EMC1-2 year plans including FIM, RapidRefresh, SREFTechnical expert meeting planned to focus on the research and
operations connections between WRF and NEMS
Develop DTC expertise in NEMS softwareOn-going: technical discussions with NEMS developers
Code repositoryTask/Work plan for Software Engineer hire
Software Engineer hire to focus on NEMS development (at EMC)starting Oct 4, 2010
Technical Meeting: October 5, 2010
Planned activities - next 6 monthsDTC Software Engineer
Will reside at EMC to further facilitate the information exchange and DTC contributions
Add in-depth knowledge of NEMS to the DTCContribute to the development of NEMS softwareBegin work on a draft code management plan
Hold a technical information exchange meetingFall 2010: DTC, NOAA-EMC, NOAA-GSD, NCAR-MMMoverlap between WRF and NEMS with respect to
community supportmaintain a connection with the broad research communityuse this as a path-finder to explore R2O and O2R
mechanisms
Anticipated major accomplishments - next 6 monthsRefine work plan for DTC-NEMS software
engineerAdd I/O layer capabilities to NEMS that will benefit
research communitiesComputing Platform Portability (systems,
compilers)Inter-operability with other community softwareUsers Guides, sample datasets, and examples of
useTechnical meeting Oct 5, 2010 in Boulder
Focus on upcoming interactions between WRF and NEMS components
Plans for AOP 2011Strengthen the foundation of DTC
expertise with the NEMS software Continue to contribute to the development
process for NEMS, with a community user focus.
Refine DTC NEMS community software goals and timeline
Develop, with EMC, a NEMS code management plan to support community software and operational system goals
Anticipated major accomplishments – AOP 2011DTC NEMS code management planContinued Technical Interchange MeetingsDTC NEMS community software plan,
including a path from R2O and O2R
Resource requirements for AOP 2011Staff Category FTE
Scientists 0.1
Software Engineers 0.75
Scientists Software Engineers
Jamie Wolff Eugene Mirvis
Laurie Carson
Non-salary Cost
Travel $12 K
Workshops $300
ChallengesEngagement of the NWP research community
is a critical component to making contributions to the operational NWP capabilities
NEMS is still under active development, so any decision to package a software release to the research community must be done with careful coordination between EMC and DTC
A fully supported community software package requires considerable resources from DTC. Is this the best use of DTC resources?