Data Assimilation

Tristan Quaife, Philip Lewis

What is Data Assimilation?

A working definition:

The set techniques the combine data with some underlying process model to provide optimal estimates of the true state and/or parameters of that model.

What is Data Assimilation?

It is not just model inversion.

But could be seen as a process constraint on inversion (e.g. a temporal constraint)

e.g. Use EO data to constrain estimates of terrestrial C fluxes

Terrestrial EO data: no direct constraint on C fluxes

Combine with model

Data Assimilation is Bayesian

• Bayes’ theorem:

P(A|B) =P(B|A) P(A)

P(B)

What does DA aim to do?

Use all available information about The underlying model The observations The observation operator

Including estimates of uncertainty and the current state of the system

To provide a best estimate of the true state of the system with quantified uncertainty

Kalman Filter DA: MODIS LAI product

Data assimilation into DALEC ecological model

Lower-level product DA

Ensure consistency between model and observations

Assimilate low-level products (surface reflectance)

Uncertainty better quantified

Need to build observation operator relating model state (e.g. LAI) to reflectance

Example of Oregon (MODIS DA)

Quaife et al. 2008, RSE

Modelled vs. observed reflectance

Red NIRNote BRF shape in red: can’t simulate with 1-D canopy (GORT here)

NEP results

No assimilation

Assimilating MODIS

(red/NIR)

DALEC model calibrated from flux measurements at tower site 1

Integrated flux predictions

Flux (gC.m-2)

Assimilated data

3yr totalStandardDeviation

NEP

No assimilation 240.2 212.2

MODIS B1 & B2 373.0 151.3

Williams et al. (2005)

406.0 27.8

GPP

No assimilation 1646.4 834.5

MODIS B1 & B2 2620.3 96.8

Williams et al. (2005)

2170.3 18.1

Flux (gC.m-2)

Assimilated data

TotalStandardDeviation

NEP

Assimilation exc. snow

373.0 151.3

Assimilation inc. snow

404.8 129.6

Williams et al. (2005)

406.0 27.8

GPP

Assimilation exc. snow

2620.3 96.8

Assimilation inc. snow

2525.6 42.7

Williams et al. (2005)

2170.3 18.1

Mean NEP for 2000-2002

15 65

gC/m2/year

4.5 km

Flux Tower

Spatial average = 50.9

Std. dev. = 9.7

(gC/m2/year)

NEP – Site2 (intermediate) parameters, with/without DA

Model running at Site 2, Oregon

Site 1 model EO-calibrated at site 2NEP observations from Site 2

Shows ability to spatialise

Data assimilation

Low-level DA can be effective

‘easier’ data uncertainties

Can be applied to multiple observation types

Requires Observation operator(s)

RT models

Requires other uncertainties

Ecosystem Model

Driver (climate)

Observation operator

Specific issues in land EOLDAS

No spatial transfer of information Require full spatial coverage Atmosphere dealt with by an instantaneous retrieval

(i.e. no transport model) All state vector members influence observations

We are not interested in other variables!

Sequential Smoothers Variational

Nominal classification of DA

Kalman Filter Variants - EKF

Ensemble Kalman Filter Variants – Unscented EnKF

Particle filters Lots of different types true MCMC technique

Sequential methods

• Propagation step:

x = Mx-

P = MP-MT + Q• Analysis step:

x* = x + K( y – Hx )

K = PHT( HPHT+R )-1

The Kalman filter

State vectorModel

Covariancematrix

Stochastic forcingKalman

gainObservation

vectorObservation covariance

matrix

Observation operator

The Kalman Filter

• Linear process model

• Linear observation operator

• Assumes normally distributed errors

• Propagation step:

x = m(x-)

P = M'P-M'T + Q• Analysis step:

x* = x + K( y – h(x) )

K = PH'T( H'PH'T+R )-1

The Extended Kalman filter

Jacobian matrix

Jacobian matrix

The Extended Kalman Filter

• Linear process model

• Linear observation operator

• Assumes normally distributed errors

• Problem with divergence

• Propagation step:

X = m(X-) + Q

no explicit error propagation• Analysis step:

X*= X + K( D – HX )

K = PHT( HPHT+R )-1

The Ensemble Kalman filter

State vector ensemble

Perturbed observations

The Ensemble Kalman Filter

• P estimated from X

• Non linear observations using augmentation:

Xa = h(X) X

The Ensemble Kalman Filter

No assimilation

Assimilating MODIS surface

reflectance bands 1 and 2

The Ensemble Kalman Filter

• Avoids use of Jacobian matrices

• Assumes normally distributed errors

– Some degree of relaxation of this assumption

• Augmentation assumes local linearisation

Particle Filters

• Propagation step:

X = m(X-) + Q

• Analysis step:

e.g. Metropolis-Hastings

Particle Filters

Particle Filters

No available observations

Particle Filters

• Fully Bayesian

– No underlying assumptions about distributions

• Theoretically the most appealing choice of sequential technique, but…

• Our analysis show little difference with EnKF

• Potentially requires larger ensemble

– But comparing 1:1 is faster than EnKF

Sequential techniques

• General considerations:

– Designed for real time systems

– Only consider historical observations

– Only assimilates observations in single time step

– Can lead to artificial high frequency components

• Extension of sequential techniques

• All observations effect every time step

• Analogous to weighting on observations

– [ smoothing-convolution / regularisation ]

• Difficult to apply in rapid change areas

Smoothers

Smoothers - regularisation

x = (HTR-1H + λ2BTB)-1HTR-1y

B is the required constraint. It imposes:

Bf = 0and the scalar λ is a weighting on the constraint.

Constraint matrix

Lagrange multiplier

Regularisation

Regularisation

Quaife and Lewis (2010) Temporal constraints on linear

BRDF model parameters. IEEE TGRS, in press.

Regularisation

• Lots of literature on the selection of λ

– Cross validation etc

• Permits insight into the form of Q

Variational techniques

• Expressed as a cost function

• Uses numerical minimisation

• Gradient descent requires differential

• Traditionally used for initial conditions

– But parameters may also be adjusted

3DVAR

J(x) = ( x-x- ) P-1 ( x-x- )T +

( y-h(x) ) R-1 ( y-h(x) )T

Background

Observations

3DVAR

• No temporal propagation of state vector

– OK for zero order approximations

– Unable to deal with phenology

4DVAR

J(x) = ( x-x- ) P-1 ( x-x- )T +

( y-h(xi) ) R-1 ( y-h(xi) )TΣi

Time varying state vector

Variational techniques

• Parameters constant over time window

• Non smooth transitions

• Assumes normal error distribution

• Size of time window?

• For zero-order case 3DVAR = 4DVAR

– 4DVAR for use with phenology model

• Absence of Q - propagation of P?

Building an EOLDAS

• Lewis et al. (RSE submitted)

• Sentinel-2

EOLDAS

Assimilation

Assume model

Uncertainty known

EOLDAS

Base level noise

Cross validation

Cross validation

EOLDAS

Cross validation

Double noise

EOLDAS

Double noise

Conclusions - technique

• DA is optimal way to combine observations and model

• Range of options available for DA

• Sequential

• Smoothers

• Variational

• Require understanding of relative uncertainties of model and observations

• Require way of linking observations and model state

• Observation operator (e.g. RT)

References

• P. Lewis et al. (2010 submitted) RSE, An EOLDAS

• T. Quaife, P. Lewis, M. DE Kauwe, M. Williams, B. Law, M. Disney, P. Bowyer (2008), Assimilating Canopy Reflectance data into an Ecosystem Model with an Ensemble Kalman Filter, Remote Sensing of Environment, 112(4),1347-1364.

• T. Quaife and P. Lewis (2010) Temporal constraints on linear BRF model parameters IEEE Transactions on Geoscience and Remote Sensing doi: 10.1109/TGRS.2009.2038901

• http://www.ecmwf.int/newsevents/training/rcourse_notes/DATA_ASSIMILATION/ASSIM_CONCEPTS/Assim_concepts11.html

• http://www.cs.unc.edu/~welch/kalman/