Assessment of uncertainty of

environmental models based on the equifinality thesis

Keith BevenLancaster University, UK

A Manifesto for the Equifinality Thesis

• One aim of environmental models is to achieve a single true description of governing processes (reality) - especially for predicting impacts of change

• Difficult to achieve in applications to places that are unique in their characteristics and where (nonlinear) predictions are subject to input, observation, and model structural errors

• There may instead be many descriptions that are compatible with current understanding and available observations

• The concept of the single description may remain a philosophical axiom or theoretical aim but is impossible to achieve in practice

• So we must accept that there may be many feasible descriptions, or a concept of equifinality, as the basis for a new approach

A Manifesto for the Equifinality Thesis

• There is no optimum model - can only assess the likelihood of a model being acceptable

• Different acceptable models will produce different predictions - especially of impacts of future change

• Should therefore assess the resulting uncertainty in predictions

• Should be prepared to revise predictions as new data become available

• Models should be rejected if shown to be non-behavioural (hypothesis testing by model rejection?)

So what if we accept multiple possible models and parameter sets…… <M(θ)1, M(θ)2, … M(θ)N > ???

Equifinality: an empirical result

Ks (cm day-1)

0.0001

0.0002

0.0003

0.0004

0.0005

0.0006

0 0.02 0.04 0.06 0.08 0.1

0.0001

0.0002

0.0003

0.0004

0.0005

0.0006

0 0.02 0.04 0.06 0.08 0.1 0.12 0.14

0.0001

0.0002

0.0003

0.0004

0.0005

0.0006

1.5 1.7 1.9 2.1 2.3 2.5

0.0001

0.0002

0.0003

0.0004

0.0005

0.0006

0.1 1 10 100 1000

n

r (cm-1)

L

L

L

L

Fitting van Genuchten parameters in modelling recharge

after Binley and Beven, Groundwater, 2002

Equifinality: an empirical result

Fitting parameters of the MAGIC geochemistry model

after Page et al., Water, Soil and Air Pollution, 2003

Equifinality: an empirical result

Fitting parameters of the Penman-Monteith equation in predicting patch scale latent and sensible heat fluxes

after Schulz and Beven, Hydrological Processes, 2003

Uncertainty is not only Statistics

• Classical approach to uncertainty of treating total model error in linear (or log linear) form

O(X, t) = Ô{M(θ,β)} + ε(X,t)

• Implicit assumption that the model is correct (or can be made to be correct through another additive term) and that any error in the inputs and other boundary conditions cascades through the model linearly

• Full power of linear statistics can then be used to estimate

L[O(X,t)|θ]

• and posterior distribution of parameters, θ

Uncertainty is not only Statistics

• But environmental models are generally nonlinear, are subject to important uncertainties in input and boundary conditions and (we suspect) suffer from model structural error

• More interested in L[M(θ)|O(X,t)]

• Additive total error may be difficult to justify but may be an approximation that will often be useful

• BUT…… from the single total error series we cannot easily deconstruct the sources of uncertainty in the modelling process

Sources of subjectivity in assessing model error

• Choice of models to be considered (including processes to be included, closure and boundary conditions),

• Choice of ranges (prior distributions) of parameter values to be considered

• Choice of input data (and input data errors) with which to drive the model (including future scenarios for prediction)

• Choice of error model, performance measure(s) or allowable error to be considered acceptable (taking account of scale, heterogeneity and incommensurability effects)

We like to think that we are objective and scientific in modelling but this is not entirely the case. There are always choices to be made...

Equifinality and the Modelling Process

• Take a (thoughtful) sample of all possible models (structures + parameter sets)

• Evaluate those models in terms of both understanding and observations in a particular application

• Reject those models that are non-behavioural (but note that there may be a scale problem in comparing model predictions and observations)

• Devise testable hypotheses to allow further models to be rejected

• [If all models rejected, revise model structures……]

This is the essence of the GLUE methodology(Bayesian priors and likelihood functions based on additive error as special case where strong assumptions justified)

Published Applications of GLUE

• Rainfall-runoff modelling

• Stochastic rainfall models

• Radar Rainfall calibration

• Modelling catchment geochemistry

• Modelling flood frequency and flood inundation

• Modelling river dispersion

• Modelling soil erosion

• Modelling land surface to atmosphere fluxes

• Modelling atmospheric deposition and critical loads

• Modelling groundwater capture zones

• Modelling groundwater recharge

• Modelling water stress cavitation and tree death

• Modelling forest fires

Generalised Likelihood Uncertainty Estimation (GLUE)

Note 1: it is the parameter set in combination with the given input and boundary condition data that gives a behavioural model - complex parameter interactions may mean that marginal distributions and global covariances have little relevance

Note 2: implicit treatment of complex errors in likelihood weighting of simulations (effectively assuming that prediction errors for any behavioural model in prediction will be “similar” to conditioning periods)

Note 3: prediction limits are conditional on choicesNote 4: depends on having sufficient sample to find

upper limit of performance

Deconstructing total model error

• Extended GLUE methodology– insist on model providing predictions within range of

“effective observation error” of evaluation variables

– effective observation error constructed to take account of scale dependencies and incommensurability (may be dependent on model implementation)

– models providing predictions outside range are rejected as non-behavioural (multiple models can be included in same formalism but possible that all models may be rejected)

– success may depend on allowing realisations of error in input and boundary condition data

Incorporating Observational Errors into GLUE

• Initial condition errors generally poorly known and are difficult to assess (no means of direct measurement) but effect will die out as simulation proceeds (eventually)

• Input and boundary condition data errors generally poorly known because of measurement technique limitations and will be processed nonlinearly through the model

• Model structural error is ubiquitous and difficult to assess - claims to physical realism do not exclude possibility of structural error but want models that are consistent with effective observational error

• Effective observational error may be difficult to assess because of scale, heterogeneity and incommensurability effects - but provides a critical limit for model rejection

Predictive distribution over all behavioural models: (A)

predictions encompass new observation

0

0.2

0.4

0.6

0.8

1

1.2 1.4 1.6 1.8 2 2.2

Predictive distribution over all behavioural models: (B)

predictions do not encompass new observation

0

0.2

0.4

0.6

0.8

1

1.2 1.4 1.6 1.8 2 2.2

………But models are still behavioural (or can be rejected on basis of new

observations)

Bukmoongol Catchment – Location

Bukmoongol – Stage/Discharge Errors

Concrete Weir has three sharp-crested rectangle sections – 2 only active atpeak discharges

-10.0

-8.0

-6.0

-4.0

-2.0

0.0

-7.0 -5.0 -3.0 -1.0log(Stage Reading) m

log

(Disch

arg

e) m

3/sec

Observed Data

Regression Fit

90% Confidence Limits

0.0

0.2

0.4

0.6

0.8

1.0

0.0 0.1 0.2 0.3 0.4 0.5 0.6Stage Reading m

Dis

cha

rge

m3/s

ec

Observed Data

Regression Fit

90% Confidence Limits

+/- 20%

Rainfall Magnitude Errors

0.0

0.2

0.4

26/07 31/07 05/08 10/08 15/08 20/08 25/08Date Time

Dis

cha

rge

0

40

80

120

Ra

infa

ll

Discharge

0.0

0.5

1.0

-5.0 0.0 5.0Rainfall Error %

Cu

mu

lativ

e P

Event 1

0.0

0.5

1.0

-5.0 0.0 5.0Rainfall Error %

Cu

mu

lativ

e P

Event 3

0.0

0.5

1.0

-5.0 0.0 5.0Rainfall Error %

Cu

mu

lativ

e P

Event 2

Results – Model Structural Error?

Log

(Dis

ch

arg

e)

Sta

nd

ard

Z V

alu

e(S

cale

d P

M2)

IC errors

Timing ErrorsAnd under prediction

ModelRecessionError

Future prospects

• A possible way forward – To think about moving from evaluations that are based on ‘lumped’ additive error terms to a framework that considers error source terms individually

• Generic technique, simple to implement in an uncertainty framework (GLUE) and based on explicit assumptions– even if assumptions may be imprecise (e.g. “effective measurement error”)

• Allows for equifinality of model structures, input realisations and parameter sets in producing behavioural models - interaction of input errors and model structure may be important

• Allows for evaluation of model structures by rejection (hypothesis testing) (see philosophy paper, Proc. Roy. Soc., 2002)

Future prospects: model structural error

• But what if these tests reveal that all models tried are non-behavioural and should be rejected?

• ?? Ensure that model space has been searched adequately

• ?? Relax rejection criteria

• ?? Add a model inadequacy term to compensate for structural error - but will be time variable and will have additional parameters for each model realisation, and how much correction should be allowed?

• ?? Find a better model - uncertainty estimation should NOT remove the need for creative and constructive modelling

and if you might possibly still want to read more…...

• Beven, K J, 2000, Uniqueness of place and process representations in hydrological modelling, Hydrology and Earth System Sciences, 4(2), 203-213.

• Beven, K. J., 2002, Towards an alternative blueprint for a physically-based digitally simulated hydrologic response modelling system, Hydrol. Process., 16(2), 189-206

• Beven, K. J., 2002, Towards a coherent philosophy for environmental modelling, Proc. Roy. Soc. Lond., A458, 2465-2484 (comment by Philippe Baveye and reply still to appear)

• Beven, K J and Young, P C, 2003, Comment on Bayesian Recursive Parameter Estimation for Hydrologic Models by M Thiemann et al. Water Resources Research, 39(5), doi: 10.1029/2001WR001183

www.es.lancs.ac.uk/hfdg/glue.html