A protocol for model validation
description
Transcript of A protocol for model validation
A protocol for model validation
ACCENT/GLOREAM Workshop
Paris, France, 11-13 October 2006
Peter Builtjes, TNO-the Netherlands and FU-Berlin
Paris, France, 11-13 October 2006ACCENT/GLOREAM Workshop - A protocol for model validation2
I) Introduction
II) A first lay-out of a protocol
III) Testing of the protocol
IV) Discussion
Paris, France, 11-13 October 2006ACCENT/GLOREAM Workshop - A protocol for model validation3
I) Introduction
• Model validation/evaluation/testing: comparison between
calculated and observed concentrations/depositions
• Observed concentrations should be accurate (the instrument)
and spatial representative (in balance with the model grid)
Paris, France, 11-13 October 2006ACCENT/GLOREAM Workshop - A protocol for model validation4
Spatial representative
• Ask the expert, rural, Educated guess
• Analysis of concentration patterns in time
• More stations in one grid
• Field study with for example passive samplers
• Determination by modelling-data assimilation
• For regional scale modelling, using rural stations, +/- 20 % ?
Paris, France, 11-13 October 2006ACCENT/GLOREAM Workshop - A protocol for model validation5
Recent model intercomparison and validation studies: EUROTRAC, EMEP-review, EURO-DELTA
Recommendation:
• No model validation for just one model
• Combine model intercomparison with model validation
Paris, France, 11-13 October 2006ACCENT/GLOREAM Workshop - A protocol for model validation6
EURODELTA
PM2.5 concentrations over Europe, preliminary/confidential first results
Paris, France, 11-13 October 2006ACCENT/GLOREAM Workshop - A protocol for model validation7
Carlos Borrego, AIR4EU
Total Model Uncertainty =
Model uncertainty + Input data uncertainty + Variability
Recommended Quality Indicators:
• Correlation Coefficient
• Fractional Bias
• RMSE (RPE)
• NMSE
Paris, France, 11-13 October 2006ACCENT/GLOREAM Workshop - A protocol for model validation8
Model intercomparison and validation
• City Delta and Euro Delta: JRC-graphical tool
• EMEP model intercomparison: TNO-tool
Paris, France, 11-13 October 2006ACCENT/GLOREAM Workshop - A protocol for model validation9
Taylor diagram for mean Summer ozone
Fine-scale and coarse-scale models
Paris, France, 11-13 October 2006ACCENT/GLOREAM Workshop - A protocol for model validation10
Compared to EMEP-review
O3 DAY TIME SUMMER 1999 obs. mean mod. mean residue RMSE corr. σ-ratio*
EMEP_v1.0 83.36 85.39 19.15 24.78 0.45 0.66
MATCH_v1.0 83.36 95.78 20.49 25.61 0.57 0.56
LOTOS_v1.0 83.36 78.29 20.90 26.28 0.47 0.87
LOTOS-EUROS 83.36 80.16 18.91 24.36 0.57 0.98
SO4 Year 2001 obs. mean mod. mean residue RMSE corr. σ-ratio*
EMEP_v1.0 2.30 2.17 1.23 1.87 0.57 1.31
MATCH_v1.0 2.34 2.78 1.29 1.84 0.62 1.26
LOTOS_v1.0 2.34 3.33 1.92 2.96 0.46 1.77
LOTOS-EUROS 2.34 1.87 1.23 1.81 0.50 0.97
NO3 Year 2001 obs. mean mod. mean residue RMSE corr. σ-ratio*
EMEP_v1.0 3.33 3.74 2.03 2.98 0.61 1.42
MATCH_v1.0 3.33 2.88 1.46 2.04 0.61 0.83
LOTOS_v1.0 3.33 3.26 1.94 2.71 0.36 0.89
LOTOS-EUROS 3.33 3.63 1.84 2.52 0.57 1.11
Paris, France, 11-13 October 2006ACCENT/GLOREAM Workshop - A protocol for model validation11
In TNO:
Quick Scan based on EMEP review tool to test different
model versions of the LOTOS-EUROS model + input:
Required QA/QC
Paris, France, 11-13 October 2006ACCENT/GLOREAM Workshop - A protocol for model validation12
II) A first lay-out of a protocol
Based on discussions at the ACCENT Workshop on Model
Benchmarking and Quality Assurance
Thessaloniki, 29/30 May 2006
Items of a protocol:
a) Define the purpose of the model and of the validation
Which output should be validated?
Example: Hourly ozone or annual averaged Benzene
Paris, France, 11-13 October 2006ACCENT/GLOREAM Workshop - A protocol for model validation13
b) Identify the processes required in the model
Is aerosol chemistry required or not?
c) Define the horizontal and vertical resolution, and the time scale
of the output
d) Concerning the input data, decide which data should be fixed
Example: are emissions taken as they are given?
Paris, France, 11-13 October 2006ACCENT/GLOREAM Workshop - A protocol for model validation14
e) Concerning observations, decide about QA/QC and spatial representativity
d) Quality indicators should be defined, including a threshold below which the model performance will be considered as inadequate
For daily max O3, the correlation coefficient between calculated
and observed should be more than 0.5, based on previous studies
e) Sensitivity runs should be defined for key processes, or key input data
Paris, France, 11-13 October 2006ACCENT/GLOREAM Workshop - A protocol for model validation15
III) Testing of the protocol
An attempt
a) Purpose of the validation for O3 daily max Summer, over Europe
b) Processes required: Only gasphase chemistry, like EMEP,
RADM, CBM4
(box model validation needed, see Poppe 1996)
Paris, France, 11-13 October 2006ACCENT/GLOREAM Workshop - A protocol for model validation16
c) Horizontal resolution : 25 x 25 km2, or 50 x 50 km2
(for discussion!!)Vertical resolution: 20 layers upto the lower stratosphere,or 5 layers upto 5 km (for discussion!!)Time resolution: hourly, not for discussion
d) Fixed input data:• Anthropogenic and biogenic emissions• Meteorology, prognostic and diagnostic• Boundary conditions, MOZART/TM5, or Logan• Landuse data base
Paris, France, 11-13 October 2006ACCENT/GLOREAM Workshop - A protocol for model validation17
e) Observations: EMEP only, rural stationsUncertainty +/- 20 % ???
f) Quality indicators: Based on experience in Euro-Delta/EMEP review
g) Sensitivity runs: for example:• Biogenic emissions• Reactivity of anthropogenic VOC-emissions• Cloud cover• Dry deposition, also over sea
Paris, France, 11-13 October 2006ACCENT/GLOREAM Workshop - A protocol for model validation18
IV) Discussion/Statements
Always combine model validation with model intercomparison
Make ensemble approach and data assimilation an integral part of model validation
Work towards toolkit : JRC + TNO-EMEP review
The proposed protocol - seems to work for daymax O3 - should be
tested in several projects and by several groups, and based on experience improved upon
And finally being “accepted”