Uncertain models and modelling uncertainty Marian Scott Dept of Statistics, University of Glasgow...
-
Upload
patrick-keating -
Category
Documents
-
view
221 -
download
3
Transcript of Uncertain models and modelling uncertainty Marian Scott Dept of Statistics, University of Glasgow...
![Page 1: Uncertain models and modelling uncertainty Marian Scott Dept of Statistics, University of Glasgow EMS workshop, Nottingham, April 2004.](https://reader035.fdocuments.net/reader035/viewer/2022062511/5515cff0550346d46f8b45ae/html5/thumbnails/1.jpg)
Uncertain models and modelling uncertainty
Marian ScottDept of Statistics, University of Glasgow
EMS workshop, Nottingham, April 2004
![Page 2: Uncertain models and modelling uncertainty Marian Scott Dept of Statistics, University of Glasgow EMS workshop, Nottingham, April 2004.](https://reader035.fdocuments.net/reader035/viewer/2022062511/5515cff0550346d46f8b45ae/html5/thumbnails/2.jpg)
Outline of presentation
Model building and testing- is the environment special? Statistical models vs physical/process based models What is sensitivity/uncertainty analysis? Quantifying and apportioning variation in model and
data. General comments- relevance and implementation.
![Page 3: Uncertain models and modelling uncertainty Marian Scott Dept of Statistics, University of Glasgow EMS workshop, Nottingham, April 2004.](https://reader035.fdocuments.net/reader035/viewer/2022062511/5515cff0550346d46f8b45ae/html5/thumbnails/3.jpg)
All models are wrong but some are useful
(and some are more useful than others)
(All data are useful, but some are more varied than others.)
![Page 4: Uncertain models and modelling uncertainty Marian Scott Dept of Statistics, University of Glasgow EMS workshop, Nottingham, April 2004.](https://reader035.fdocuments.net/reader035/viewer/2022062511/5515cff0550346d46f8b45ae/html5/thumbnails/4.jpg)
Questions we ask about models
Is the model valid? Are the assumptions
reasonable? Does the model make
sense based on best scientific knowledge
Is the model credible? Do the model predictions
match the observed data?
How uncertain are the results?
What is a good model?Simple, realistic, efficient, useful, reliable, valid etc
![Page 5: Uncertain models and modelling uncertainty Marian Scott Dept of Statistics, University of Glasgow EMS workshop, Nottingham, April 2004.](https://reader035.fdocuments.net/reader035/viewer/2022062511/5515cff0550346d46f8b45ae/html5/thumbnails/5.jpg)
Statistical models
Always includes an term to describe random variation
Empirical Descriptive and predictive Model building goal: simplest model which is
adequate used for inference
![Page 6: Uncertain models and modelling uncertainty Marian Scott Dept of Statistics, University of Glasgow EMS workshop, Nottingham, April 2004.](https://reader035.fdocuments.net/reader035/viewer/2022062511/5515cff0550346d46f8b45ae/html5/thumbnails/6.jpg)
Physical/process based models
Uses best scientific knowledge May not explicitly include , or any random
variation Descriptive and predictive Goal may not be simplest model Not used for inference
![Page 7: Uncertain models and modelling uncertainty Marian Scott Dept of Statistics, University of Glasgow EMS workshop, Nottingham, April 2004.](https://reader035.fdocuments.net/reader035/viewer/2022062511/5515cff0550346d46f8b45ae/html5/thumbnails/7.jpg)
Models
Mathematical (deterministic/process based) models tend
to be complex to ignore important sources of uncertaintyStatistical models tend to be empirical To ignore much of the
biological/physical/chemical knowledge
![Page 8: Uncertain models and modelling uncertainty Marian Scott Dept of Statistics, University of Glasgow EMS workshop, Nottingham, April 2004.](https://reader035.fdocuments.net/reader035/viewer/2022062511/5515cff0550346d46f8b45ae/html5/thumbnails/8.jpg)
Stages in modelling
Design and conceptualisation:– Visualisation of structure– Identification of processes (variable selection)– Choice of parameterisation
Fitting and assessment– parameter estimation (calibration)– Goodness of fit
![Page 9: Uncertain models and modelling uncertainty Marian Scott Dept of Statistics, University of Glasgow EMS workshop, Nottingham, April 2004.](https://reader035.fdocuments.net/reader035/viewer/2022062511/5515cff0550346d46f8b45ae/html5/thumbnails/9.jpg)
Model evaluation tools
Graphical procedures % variation explained in response Statistical model comparisons (F-tests,
ANOVA, GLRT)
well designed for statistical models, but what of the physical, process-driven models?– Comparability to measurements
![Page 10: Uncertain models and modelling uncertainty Marian Scott Dept of Statistics, University of Glasgow EMS workshop, Nottingham, April 2004.](https://reader035.fdocuments.net/reader035/viewer/2022062511/5515cff0550346d46f8b45ae/html5/thumbnails/10.jpg)
The story of randomness and uncertainty
Randomness as the source of variability
– A source of variation, different animals range over different territory, eat different sources of ….
The effect is that we cannot be certain
Uncertainty due to lack of knowledge
– conflicting evidence
– ignorance
– effects of scale
– lack of observations Uncertainty due to variability
– Natural randomness
– behavioural variability
![Page 11: Uncertain models and modelling uncertainty Marian Scott Dept of Statistics, University of Glasgow EMS workshop, Nottingham, April 2004.](https://reader035.fdocuments.net/reader035/viewer/2022062511/5515cff0550346d46f8b45ae/html5/thumbnails/11.jpg)
Effect of uncertainties
Uncertainty in model quantities/parameters/
inputs Uncertainty about model
form Uncertainty about model
completeness
Lack of observations contribute to
– uncertainties in input data– parameter uncertainties
Conflicting evidence contributes to
– uncertainty about model form
– Uncertainty about validity of assumptions
Making it difficult to judge how good a model is!!
![Page 12: Uncertain models and modelling uncertainty Marian Scott Dept of Statistics, University of Glasgow EMS workshop, Nottingham, April 2004.](https://reader035.fdocuments.net/reader035/viewer/2022062511/5515cff0550346d46f8b45ae/html5/thumbnails/12.jpg)
Modelling tools - SA/UA
Sensitivity analysis
determining the amount and kind of change produced in the model predictions by a change in a model parameter
Uncertainty analysis
an assessment/quantification of the uncertainties associated with the parameters, the data and the model structure.
![Page 13: Uncertain models and modelling uncertainty Marian Scott Dept of Statistics, University of Glasgow EMS workshop, Nottingham, April 2004.](https://reader035.fdocuments.net/reader035/viewer/2022062511/5515cff0550346d46f8b45ae/html5/thumbnails/13.jpg)
Modellers conduct SA to determine
(a) if a model resembles the system or processes under study,
(b) the factors that mostly contribute to the output variability,
(c) the model parameters (or parts of the model itself) that are insignificant,
(d) if there is some region in the space of input factors for which the model variation is maximum,
and (e) if and which (group of) factors interact with each
other.
![Page 14: Uncertain models and modelling uncertainty Marian Scott Dept of Statistics, University of Glasgow EMS workshop, Nottingham, April 2004.](https://reader035.fdocuments.net/reader035/viewer/2022062511/5515cff0550346d46f8b45ae/html5/thumbnails/14.jpg)
SA flow chart (Saltelli, Chan and Scott, 2000)
![Page 15: Uncertain models and modelling uncertainty Marian Scott Dept of Statistics, University of Glasgow EMS workshop, Nottingham, April 2004.](https://reader035.fdocuments.net/reader035/viewer/2022062511/5515cff0550346d46f8b45ae/html5/thumbnails/15.jpg)
Design of the SA experiment
Simple factorial designs (one at a time) Factorial designs (including potential
interaction terms) Fractional factorial designs Important difference: design in the context of
computer code experiments – random variation due to variation in experimental units does not exist.
![Page 16: Uncertain models and modelling uncertainty Marian Scott Dept of Statistics, University of Glasgow EMS workshop, Nottingham, April 2004.](https://reader035.fdocuments.net/reader035/viewer/2022062511/5515cff0550346d46f8b45ae/html5/thumbnails/16.jpg)
SA techniques
Screening techniques– O(ne) A(t) T(ime), factorial, fractional factorial
designs used to isolate a set of important factors
Local/differential analysis Sampling-based (Monte Carlo) methods Variance based methods
– variance decomposition of output to compute sensitivity indices
![Page 17: Uncertain models and modelling uncertainty Marian Scott Dept of Statistics, University of Glasgow EMS workshop, Nottingham, April 2004.](https://reader035.fdocuments.net/reader035/viewer/2022062511/5515cff0550346d46f8b45ae/html5/thumbnails/17.jpg)
Screening
screening experiments can be used to identify the parameter subset that controls most of the output variability with low computational effort.
![Page 18: Uncertain models and modelling uncertainty Marian Scott Dept of Statistics, University of Glasgow EMS workshop, Nottingham, April 2004.](https://reader035.fdocuments.net/reader035/viewer/2022062511/5515cff0550346d46f8b45ae/html5/thumbnails/18.jpg)
Screening methods
Vary one factor at a time (NOT particularly recommended)
Morris OAT design (global)– Estimate the main effect of a factor by computing a
number r of local measures at different points x1,…,xr in the input space and then average them.
– Order the input factors
![Page 19: Uncertain models and modelling uncertainty Marian Scott Dept of Statistics, University of Glasgow EMS workshop, Nottingham, April 2004.](https://reader035.fdocuments.net/reader035/viewer/2022062511/5515cff0550346d46f8b45ae/html5/thumbnails/19.jpg)
Local SA
Local SA concentrates on the local impact of the factors on the model. Local SA is usually carried out by computing partial derivatives of the output functions with respect to the input variables.
The input parameters are varied in a small interval around a nominal value. The interval is usually the same for all of the variables and is not related to the degree of knowledge of the variables.
![Page 20: Uncertain models and modelling uncertainty Marian Scott Dept of Statistics, University of Glasgow EMS workshop, Nottingham, April 2004.](https://reader035.fdocuments.net/reader035/viewer/2022062511/5515cff0550346d46f8b45ae/html5/thumbnails/20.jpg)
Global SA
Global SA apportions the output uncertainty to the uncertainty in the input factors, covering their entire range space.
A global method evaluates the effect of xj while all other xi,ij are varied as well.
![Page 21: Uncertain models and modelling uncertainty Marian Scott Dept of Statistics, University of Glasgow EMS workshop, Nottingham, April 2004.](https://reader035.fdocuments.net/reader035/viewer/2022062511/5515cff0550346d46f8b45ae/html5/thumbnails/21.jpg)
How is a sampling (global) based SA implemented?
Step 1: define model, input factors and outputs
Step 2: assign p.d.f.’s to input parameters/factors and if necessary covariance structure. DIFFICULT
Step 3:simulate realisations from the parameter pdfs to generate a set of model runs giving the set of output values.
![Page 22: Uncertain models and modelling uncertainty Marian Scott Dept of Statistics, University of Glasgow EMS workshop, Nottingham, April 2004.](https://reader035.fdocuments.net/reader035/viewer/2022062511/5515cff0550346d46f8b45ae/html5/thumbnails/22.jpg)
Choice of sampling method
S(imple) or Stratified R(andom) S(ampling)– Each input factor sampled independently many times from
marginal distbns to create the set of input values (or randomly sampled from joint distbn.)
Expensive (relatively) in computational effort if model has many input factors, may not give good coverage of the entire range space
L(atin) H(ypercube) S(sampling)– The range of each input factor is categorised into N equal
probability intervals, one observation of each input factor made in each interval.
![Page 23: Uncertain models and modelling uncertainty Marian Scott Dept of Statistics, University of Glasgow EMS workshop, Nottingham, April 2004.](https://reader035.fdocuments.net/reader035/viewer/2022062511/5515cff0550346d46f8b45ae/html5/thumbnails/23.jpg)
SA -analysis
At the end of the computer experiment, data is of the form (yij, x1i,x2i,….,xni), where x1,..,xn are the realisations of the input factors.
Analysis includes regression analysis (on raw and ranked values), standard hypothesis tests of distribution (mean and variance) for sub-samples corresponding to given percentiles of x and Analysis of Variance.
![Page 24: Uncertain models and modelling uncertainty Marian Scott Dept of Statistics, University of Glasgow EMS workshop, Nottingham, April 2004.](https://reader035.fdocuments.net/reader035/viewer/2022062511/5515cff0550346d46f8b45ae/html5/thumbnails/24.jpg)
Some ‘new’ methods of analysis
Measures of importance
VarXi(E(Y|Xj =xj))/Var(Y)
HIM(Xj) =yiyi’/N
Sobol sensitivity indices Fourier Amplitude Sensitivity test (FAST)
![Page 25: Uncertain models and modelling uncertainty Marian Scott Dept of Statistics, University of Glasgow EMS workshop, Nottingham, April 2004.](https://reader035.fdocuments.net/reader035/viewer/2022062511/5515cff0550346d46f8b45ae/html5/thumbnails/25.jpg)
So far so good
but how useful are these techniques in some real life problems?
Are there other complicating factors?
Do statisticians have too simple/complex a view of the world?
![Page 26: Uncertain models and modelling uncertainty Marian Scott Dept of Statistics, University of Glasgow EMS workshop, Nottingham, April 2004.](https://reader035.fdocuments.net/reader035/viewer/2022062511/5515cff0550346d46f8b45ae/html5/thumbnails/26.jpg)
Common features of environmental modelling and observations
Knowledge of the processes creating the observational record may be incomplete
The observational records may be incomplete (observed often irregularly in space and time)
involve extreme events involve quantification of risk
![Page 27: Uncertain models and modelling uncertainty Marian Scott Dept of Statistics, University of Glasgow EMS workshop, Nottingham, April 2004.](https://reader035.fdocuments.net/reader035/viewer/2022062511/5515cff0550346d46f8b45ae/html5/thumbnails/27.jpg)
Issues and purpose of analysis
Global and local pollutant mapping from Chernobyl
Global carbon cycle – greenhouse gases, CO2 levels and global warming
Ocean modelling
Air pollution modelling (local and regional scale)
Chronologies for past environment studies
Decision making- Which areas should be restricted?
Prediction-What is the trend in temperature? Predict its level in 2050?
Decision making-is it safe to eat fish?
Regulatory- Have emission control agreements reduced air pollutants?
Understanding -when did things happen in the past
![Page 28: Uncertain models and modelling uncertainty Marian Scott Dept of Statistics, University of Glasgow EMS workshop, Nottingham, April 2004.](https://reader035.fdocuments.net/reader035/viewer/2022062511/5515cff0550346d46f8b45ae/html5/thumbnails/28.jpg)
Questions we ask about observations
Do they result from observational or designed; laboratory or field experiments?
What scale are they collected over (time and space)? Are they representative? Are they qualitative or quantitative? How are they connected to processes, how well
understood are these connections? How varied are they?
![Page 29: Uncertain models and modelling uncertainty Marian Scott Dept of Statistics, University of Glasgow EMS workshop, Nottingham, April 2004.](https://reader035.fdocuments.net/reader035/viewer/2022062511/5515cff0550346d46f8b45ae/html5/thumbnails/29.jpg)
Example 1: are atmospheric SO2 concentrations declining?
Measurements made at a monitoring station over a 20 year period: processes involve meteorology (local and long-range, source distribution, chemistry of sulphur)
Complex statistical model developed to describe the pattern, the model portions the variation to ‘trend’, seasonality, residual variation
Main objective
![Page 30: Uncertain models and modelling uncertainty Marian Scott Dept of Statistics, University of Glasgow EMS workshop, Nottingham, April 2004.](https://reader035.fdocuments.net/reader035/viewer/2022062511/5515cff0550346d46f8b45ae/html5/thumbnails/30.jpg)
so2 monitored in GB02
observations
so2
0 50 100 150 200 250
02
46
81
0
![Page 31: Uncertain models and modelling uncertainty Marian Scott Dept of Statistics, University of Glasgow EMS workshop, Nottingham, April 2004.](https://reader035.fdocuments.net/reader035/viewer/2022062511/5515cff0550346d46f8b45ae/html5/thumbnails/31.jpg)
Plot of so2 against time, monitored in GB02Lines = Model 3
months
so2
1980 1985 1990 1995
02
46
81
0
![Page 32: Uncertain models and modelling uncertainty Marian Scott Dept of Statistics, University of Glasgow EMS workshop, Nottingham, April 2004.](https://reader035.fdocuments.net/reader035/viewer/2022062511/5515cff0550346d46f8b45ae/html5/thumbnails/32.jpg)
SO4 in air, monitored at Lough Navar (GB06)
observations
SO
4 in
air
0 50 100 150
0.0
0.5
1.0
1.5
2.0
2.5
![Page 33: Uncertain models and modelling uncertainty Marian Scott Dept of Statistics, University of Glasgow EMS workshop, Nottingham, April 2004.](https://reader035.fdocuments.net/reader035/viewer/2022062511/5515cff0550346d46f8b45ae/html5/thumbnails/33.jpg)
Example 2
Discovery of radioactive particles on the foreshore of a nuclear facility since 1983
Is the rate of finds falling off? Are the particle characteristics changing with time? Processes: transport in the marine environment,
chemistry of the particles in the sea, interaction with source
What can we infer about the size of the source and its distribution?
![Page 34: Uncertain models and modelling uncertainty Marian Scott Dept of Statistics, University of Glasgow EMS workshop, Nottingham, April 2004.](https://reader035.fdocuments.net/reader035/viewer/2022062511/5515cff0550346d46f8b45ae/html5/thumbnails/34.jpg)
Log activity and trend
Date
logact
ivity
20.0
17.5
15.0
12.5
10.0
7.5
5.0
Accuracy MeasuresMAPE 11.8851MAD 1.4229MSD 3.8787
VariableActualFits
Trend Analysis Plot for logactivityLinear Trend Model
Yt = 14.9899 - 0.00712072*t
![Page 35: Uncertain models and modelling uncertainty Marian Scott Dept of Statistics, University of Glasgow EMS workshop, Nottingham, April 2004.](https://reader035.fdocuments.net/reader035/viewer/2022062511/5515cff0550346d46f8b45ae/html5/thumbnails/35.jpg)
Trend in number of finds
year
number of finds
2002200019981996199419921990198819861984
25
20
15
10
5
0
Accuracy MeasuresMAPE 108.951MAD 4.025MSD 28.222
VariableActualFits
Trend Analysis Plot for number of findsLinear Trend Model
Yt = 14.7476 - 0.401299*t
![Page 36: Uncertain models and modelling uncertainty Marian Scott Dept of Statistics, University of Glasgow EMS workshop, Nottingham, April 2004.](https://reader035.fdocuments.net/reader035/viewer/2022062511/5515cff0550346d46f8b45ae/html5/thumbnails/36.jpg)
Cumulative number of finds
1612840
200
150
100
50
0
1612840
Scatterplot of cumulative finds pre 1998 and post 1997
![Page 37: Uncertain models and modelling uncertainty Marian Scott Dept of Statistics, University of Glasgow EMS workshop, Nottingham, April 2004.](https://reader035.fdocuments.net/reader035/viewer/2022062511/5515cff0550346d46f8b45ae/html5/thumbnails/37.jpg)
Example 3: how well should models agree?
6 ocean models (process based-transport, sedimentary processes, numerical solution scheme, grid size) used to predict the dispersal of a pollutant
Results to be used to determine a remediation policy
The models differ in their detail and also in their spatial scale
![Page 38: Uncertain models and modelling uncertainty Marian Scott Dept of Statistics, University of Glasgow EMS workshop, Nottingham, April 2004.](https://reader035.fdocuments.net/reader035/viewer/2022062511/5515cff0550346d46f8b45ae/html5/thumbnails/38.jpg)
Model agreement
Three different sites (local, regional and global relative to a source)
6 different models Level of agreement (high
values are poor).
site 1site 2site 3
654321
6
5
4
3
2
1
0
Modelle
vel o
f agr
eem
ent
Sensitivity measures for each model
![Page 39: Uncertain models and modelling uncertainty Marian Scott Dept of Statistics, University of Glasgow EMS workshop, Nottingham, April 2004.](https://reader035.fdocuments.net/reader035/viewer/2022062511/5515cff0550346d46f8b45ae/html5/thumbnails/39.jpg)
Predictions of levels of cobalt-60
Different models, same input data
Predictions vary by considerable margins
Magnitude of variation a function of spatial distribution of sites
tiwtistcwtcsbiwbisbcwbcs
250
150
50
Simulation condition
CV
(%)
CV(%) for location 7
tiwtistcwtcsbiwbisbcwbcs
250
150
50
Simulation condition
CV
(%)
CV(%) for location 8
tiwtistcwtcsbiwbisbcwbcs
250
150
50
Simulation condition
CV
(%)
CV(%) for location 9
tiwtistcwtcsbiwbisbcwbcs
250
150
50
Simulation condition
CV
(%)
CV(%) for location 10
tiwtistcwtcsbiwbisbcwbcs
250
150
50
Simulation condition
CV
(%)
CV(%) for location 11
![Page 40: Uncertain models and modelling uncertainty Marian Scott Dept of Statistics, University of Glasgow EMS workshop, Nottingham, April 2004.](https://reader035.fdocuments.net/reader035/viewer/2022062511/5515cff0550346d46f8b45ae/html5/thumbnails/40.jpg)
Environmental modelling
Modelling may involve– Understanding and handling variation– Dealing with unusual observations– Dealing with missing observations– Evaluating uncertainties
![Page 41: Uncertain models and modelling uncertainty Marian Scott Dept of Statistics, University of Glasgow EMS workshop, Nottingham, April 2004.](https://reader035.fdocuments.net/reader035/viewer/2022062511/5515cff0550346d46f8b45ae/html5/thumbnails/41.jpg)
How well should the model reproduce the data?
anecdotal comments ‘agreement between model and measurement better than 1 (2 ) orders of magnitude is acceptable’.
But this needs to be moderated by the measurement variation and uncertainties
It also depends on the purpose (model fit for purpose)
![Page 42: Uncertain models and modelling uncertainty Marian Scott Dept of Statistics, University of Glasgow EMS workshop, Nottingham, April 2004.](https://reader035.fdocuments.net/reader035/viewer/2022062511/5515cff0550346d46f8b45ae/html5/thumbnails/42.jpg)
How can SA/UA help?
SA/UA have a role to play in all modelling stages:– We learn about model behaviour and ‘robustness’ to
change;– We can generate an envelope of ‘outcomes’ and
see whether the observations fall within the envelope;
– We can ‘tune’ the model and identify reasons/causes for differences between model and observations
![Page 43: Uncertain models and modelling uncertainty Marian Scott Dept of Statistics, University of Glasgow EMS workshop, Nottingham, April 2004.](https://reader035.fdocuments.net/reader035/viewer/2022062511/5515cff0550346d46f8b45ae/html5/thumbnails/43.jpg)
On the other hand - Uncertainty analysis
Parameter uncertainty– usually quantified in form of a distribution.
Model structural uncertainty– more than one model may be fit, expressed as a
prior on model structure.
Scenario uncertainty– uncertainty on future conditions.
![Page 44: Uncertain models and modelling uncertainty Marian Scott Dept of Statistics, University of Glasgow EMS workshop, Nottingham, April 2004.](https://reader035.fdocuments.net/reader035/viewer/2022062511/5515cff0550346d46f8b45ae/html5/thumbnails/44.jpg)
Tools for handling uncertainty
Parameter uncertainty– Probability distributions and Sensitivity analysis
Structural uncertainty– Bayesian framework– one possibility to define a discrete set of models,
other possibility to use a Gaussian process
![Page 45: Uncertain models and modelling uncertainty Marian Scott Dept of Statistics, University of Glasgow EMS workshop, Nottingham, April 2004.](https://reader035.fdocuments.net/reader035/viewer/2022062511/5515cff0550346d46f8b45ae/html5/thumbnails/45.jpg)
Conclusions
The world is rich and varied in its complexity Modelling is an uncertain activity
Model assessment is a difficult process SA/UA are an important tools in model assessment The setting of the problem in a unified Bayesian
framework allows all the sources of uncertainty to be quantified, so a fuller assessment to be performed.
![Page 46: Uncertain models and modelling uncertainty Marian Scott Dept of Statistics, University of Glasgow EMS workshop, Nottingham, April 2004.](https://reader035.fdocuments.net/reader035/viewer/2022062511/5515cff0550346d46f8b45ae/html5/thumbnails/46.jpg)
Challenges
Some challenges: different terminologies in different subject areas. need more sophisticated tools to deal with multivariate
nature of problem. challenges in describing the distribution of input
parameters. challenges in dealing with the Bayesian formulation of
structural uncertainty for complex models. Computational challenges in simulations for large
and complex computer models with many factors.