An Introduction to Ensemble Kalman Filtering · An Introduction to Ensemble Kalman Filtering...
Transcript of An Introduction to Ensemble Kalman Filtering · An Introduction to Ensemble Kalman Filtering...
6/18/07
ring
tion (DAReS)
7
Anderson: NSF August, 2006 1
An Introduction toEnsemble Kalman Filte
Jeffrey AndersonNCAR Data Assimilation Research Sec
AMS NWP/WAF, 26 June, 200
6/18/07
ng
ith Ensembles
Anderson: NSF August, 2006 2
Overview
1. The Data Assimilation Problem
2. A Bayesian View of Ensemble Kalman Filteri
3. Challenges to Ensemble Filters
4. Adaptive Ensemble Filter Algorithms
5. Model and Observing System Development w
6/18/07
:
Anderson: NSF August, 2006 3
The Data Assimilation Problem
6/18/07
:
_________________________
Anderson: NSF August, 2006 4
The Data Assimilation Problem
__________________________________________________
A (Physical) system:Atmosphere, coupled climate system...
6/18/07
:
_________________________
___________________
odel ‘state vector’.).
Anderson: NSF August, 2006 5
The Data Assimilation Problem
__________________________________________________
A (Physical) system:Atmosphere, coupled climate system...
______________________________________A model of the physical system:
Represents system with discrete vector: the mApproximates time evolution of system (poorly
6/18/07
:
_________________________
___________________
odel ‘state vector’.).___________________
tion error.ace).licated.
Anderson: NSF August, 2006 6
The Data Assimilation Problem
__________________________________________________
A (Physical) system:Atmosphere, coupled climate system...
______________________________________A model of the physical system:
Represents system with discrete vector: the mApproximates time evolution of system (poorly
______________________________________Observations of the system:
Have a (sometimes poor) estimate of observaCould be sparse and irregular in time (and spRelation to model ‘state vector’ may be compMay have very low information content.
6/18/07
all three pieces:
Anderson: NSF August, 2006 7
Data Assimilation increases information about
6/18/07
all three pieces:___________
Anderson: NSF August, 2006 8
Data Assimilation increases information about______________________________________(Physical) system:
Estimates of state (analyses, posteriors...).Initial conditions for forecasts.Enhanced (physical) understanding.
6/18/07
all three pieces:___________
___________
l ‘parameters’).
Anderson: NSF August, 2006 9
Data Assimilation increases information about______________________________________(Physical) system:
Estimates of state (analyses, posteriors...).Initial conditions for forecasts.Enhanced (physical) understanding.
______________________________________Model:
Estimates of model errors.Relative characteristics of different models.Improved model (find good values for mode
6/18/07
all three pieces:___________
___________
l ‘parameters’).___________
servations.ased information.
Anderson: NSF August, 2006 10
Data Assimilation increases information about______________________________________(Physical) system:
Estimates of state (analyses, posteriors...).Initial conditions for forecasts.Enhanced (physical) understanding.
______________________________________Model:
Estimates of model errors.Relative characteristics of different models.Improved model (find good values for mode
______________________________________Observations:
Estimates of observation errors.Information content of existing or planned obObserving system designs that provide incre
6/18/07
Forecasters
ember ENKF
Anderson: NSF August, 2006 11
Ensemble Filter Products now Available to
Environment Canada Operational GEM 16 m
6/18/07
Forecasters
KF
Anderson: NSF August, 2006 12
Ensemble Filter Products now Available to
University of Washington WRF En
6/18/07
Forecasters
vailable.
d below.
ve additional issues.
Anderson: NSF August, 2006 13
Ensemble Filter Products now Available to
1. Many other ensemble forecast products are a
2. Most suffer from the same challenges outline
3. Other ensemble generation methods may ha
6/18/07
ring.
le.
rsion in the valley)
variable.re.
Anderson: NSF August, 2006 14
An intr oduction to Ensemble Filte
1: Single variable and observation of that variabLet’s think of it as temperature at SLC.
(Slides are for a mid-winter ski day...inve
2: Single observed variable, single unobservedSLC temperature, Park City temperatu
That’s all there is... (without loss of generality).
6/18/07
, C.
n C and B.
AC) p A C( )x)p x C( )dx
---------------------------------
4 6
Anderson: NSF August, 2006 15
Bayes rule:
A: Prior estimate based on all previous informationB: An additional observation.
: Posterior (updated estimate) based o
p A BC( )p B AC( ) p A C( )
p B C( )------------------------------------------
p B(p∫ B(
----------= =
−6 −4 −2 0 20
0.05
0.1
0.15
0.2 Prior PDF
Pro
babi
lity
p A BC( )
6/18/07
tion, C.
n C and B.
AC) p A C( )x)p x C( )dx
---------------------------------
4 6
Obs. Likelihood
Anderson: NSF August, 2006 16
Bayes rule:
A: Prior estimate based on all previous informaB: An additional observation.
: Posterior (updated estimate) based o
p A BC( )p B AC( ) p A C( )
p B C( )------------------------------------------
p B(p∫ B(
----------= =
−6 −4 −2 0 20
0.05
0.1
0.15
0.2 Prior PDF
Pro
babi
lity
p A BC( )
6/18/07
tion, C.
n C and B.
AC) p A C( )x)p x C( )dx
---------------------------------
4 6
Obs. Likelihood
Anderson: NSF August, 2006 17
Bayes rule:
A: Prior estimate based on all previous informaB: An additional observation.
: Posterior (updated estimate) based o
p A BC( )p B AC( ) p A C( )
p B C( )------------------------------------------
p B(p∫ B(
----------= =
−6 −4 −2 0 20
0.05
0.1
0.15
0.2 Prior PDF
Pro
babi
lity
Product (Numerator)
p A BC( )
6/18/07
ion, C.
n C and B.
AC) p A C( )x)p x C( )dx
--------------------------------
4 6
Obs. Likelihood
.)
Anderson: NSF August, 2006 18
Bayes rule:
A: Prior estimate based on all previous informatB: An additional observation.
: Posterior (updated estimate) based o
p A BC( )p B AC( ) p A C( )
p B C( )------------------------------------------
p B(p∫ B(
-----------= =
−6 −4 −2 0 20
0.05
0.1
0.15
0.2 Prior PDF
Pro
babi
lity
Normalization (Denom
p A BC( )
6/18/07
tion, C.
n C and B.
AC) p A C( )x)p x C( )dx
---------------------------------
4 6
Obs. Likelihood
.)
rior
Anderson: NSF August, 2006 19
Bayes rule:
A: Prior estimate based on all previous informaB: An additional observation.
: Posterior (updated estimate) based o
p A BC( )p B AC( ) p A C( )
p B C( )------------------------------------------
p B(p∫ B(
----------= =
−6 −4 −2 0 20
0.05
0.1
0.15
0.2 Prior PDF
Pro
babi
lity
Normalization (Denom
Poste
p A BC( )
6/18/07
ghout
Anderson: NSF August, 2006 20
Consistent Color Scheme Throu
Green = Prior
Red = Observation
Blue = Posterior
6/18/07
s.
AC) p A C( )x)p x C( )dx
---------------------------------
2 4
bs. Likelihood
Anderson: NSF August, 2006 21
Bayes rule:
This product is closed for Gaussian distribution
p A BC( )p B AC( ) p A C( )
p B C( )------------------------------------------
p B(p∫ B(
----------= =
−4 −2 00
0.2
0.4
0.6
Prior PDF
Pro
babi
lity O
6/18/07
s.
AC) p A C( )x)p x C( )dx
---------------------------------
2 4
bs. Likelihood
Anderson: NSF August, 2006 22
Bayes rule:
This product is closed for Gaussian distribution
p A BC( )p B AC( ) p A C( )
p B C( )------------------------------------------
p B(p∫ B(
----------= =
−4 −2 00
0.2
0.4
0.6
Prior PDF
Pro
babi
lity O
Posterior PDF
6/18/07
dµ2 and
Anderson: NSF August, 2006 23
Product of two Gaussians:
Product of d-dimensional normals with meansµ1 ancovariance matricesΣ1 andΣ2 is normal.
N µ1 Σ1,( )N µ2 Σ2,( ) cN µ Σ,( )=
6/18/07
dµ2 and
Anderson: NSF August, 2006 24
Product of two Gaussians:
Product of d-dimensional normals with meansµ1 ancovariance matricesΣ1 andΣ2 is normal.
Covariance:
Mean:
N µ1 Σ1,( )N µ2 Σ2,( ) cN µ Σ,( )=
Σ Σ11– Σ2
1–+( ) 1–=
µ Σ11– Σ2
1–+( ) 1– Σ11– µ1 Σ2
1– µ2+( )=
6/18/07
dµ2 and
1 Σ2+ ) 1– µ2 µ1–( )]
Anderson: NSF August, 2006 25
Product of two Gaussians:
Product of d-dimensional normals with meansµ1 ancovariance matricesΣ1 andΣ2 is normal.
Covariance:
Mean:
Weight:
We’ll ignore the weight since we immediatelynormalize products to be PDFs.
N µ1 Σ1,( )N µ2 Σ2,( ) cN µ Σ,( )=
Σ Σ11– Σ2
1–+( ) 1–=
µ Σ11– Σ2
1–+( ) 1– Σ11– µ1 Σ2
1– µ2+( )=
c1
2Π( )d 2⁄ Σ1 Σ2+ 1 2⁄--------------------------------------------------- 12--- µ2 µ1–( )T Σ([–
exp=
6/18/07
dµ2 and
exponentials.
1 Σ2+ ) 1– µ2 µ1–( )]
Anderson: NSF August, 2006 26
Product of two Gaussians:
Product of d-dimensional normals with meansµ1 ancovariance matricesΣ1 andΣ2 is normal.
Covariance:
Mean:
Weight:
Easy to derive for 1D (d=1); just do products of
N µ1 Σ1,( )N µ2 Σ2,( ) cN µ Σ,( )=
Σ Σ11– Σ2
1–+( ) 1–=
µ Σ11– Σ2
1–+( ) 1– Σ11– µ1 Σ2
1– µ2+( )=
c1
2Π( )d 2⁄ Σ1 Σ2+ 1 2⁄--------------------------------------------------- 12--- µ2 µ1–( )T Σ([–
exp=
6/18/07
.
le.th’.
AC) p A C( )x)p x C( )dx
---------------------------------
2 4
Anderson: NSF August, 2006 27
Bayes rule:
Ensemble filters:Prior is available as finite sample
Don’t know much about properties of this sampMay naively assume it is random draw from ‘tru
p A BC( )p B AC( ) p A C( )
p B C( )------------------------------------------
p B(p∫ B(
----------= =
−4 −2 00
0.2
0.4
0.6
Prior Ensemble
Pro
babi
lity
6/18/07
uous likelihood?
to sample.
AC) p A C( )x)p x C( )dx
---------------------------------
2 4
Anderson: NSF August, 2006 28
Bayes rule:
How can we take product of sample with contin
Fit a continuous (Gaussian for now) distribution
p A BC( )p B AC( ) p A C( )
p B C( )------------------------------------------
p B(p∫ B(
----------= =
−4 −2 00
0.2
0.4
0.6
Prior Ensemble
Pro
babi
lity
Prior PDF
6/18/07
ly always Gaussian).
e methods below.obs. likelihood.
AC) p A C( )x)p x C( )dx
---------------------------------
2 4
bs. Likelihood
Anderson: NSF August, 2006 29
Bayes rule:
Observation likelihood usually continuous (near
If Obs. Likelihood isn’t Gaussian, can generalizFor instance, can fit set of Gaussian kernels to
p A BC( )p B AC( ) p A C( )
p B C( )------------------------------------------
p B(p∫ B(
----------= =
−4 −2 00
0.2
0.4
0.6
Prior Ensemble
Pro
babi
lity
Prior PDF
O
6/18/07
is Gaussian.
AC) p A C( )x)p x C( )dx
---------------------------------
2 4
bs. Likelihood
Anderson: NSF August, 2006 30
Bayes rule:
Product of prior Gaussian fit and Obs. likelihood
Computing continuous posterior is simple.BUT, need to have a SAMPLE of this PDF.
p A BC( )p B AC( ) p A C( )
p B C( )------------------------------------------
p B(p∫ B(
----------= =
−4 −2 00
0.2
0.4
0.6
Prior Ensemble
Pro
babi
lity
Prior PDF
O
Posterior PDF
6/18/07
nclear.rform., etc.
2 3
Anderson: NSF August, 2006 31
Sampling Posterior PDF:
There are many ways to do this.
Exact properties of different methods may be uTrial and error still best way to see how they peWill interact with properties of prediction models
−2 −1 0 10
0.2
0.4
0.6Posterior PDF
Pro
babi
lity
6/18/07
2 4
Anderson: NSF August, 2006 32
Ensemble Filter Algorithms:
Ensemble Adjustment (Kalman) Filter.
−4 −2 00
0.2
0.4
0.6
Prior Ensemble
Pro
babi
lity
6/18/07
2 4
Anderson: NSF August, 2006 33
Ensemble Filter Algorithms:
Ensemble Adjustment (Kalman) Filter.
Again, fit a Gaussian to sample.−4 −2 00
0.2
0.4
0.6
Prior Ensemble
Pro
babi
lity
Prior PDF
6/18/07
2 4
bs. Likelihood
Anderson: NSF August, 2006 34
Ensemble Filter Algorithms:
Ensemble Adjustment (Kalman) Filter.
Compute posterior PDF.−4 −2 00
0.2
0.4
0.6
Prior Ensemble
Pro
babi
lity
Prior PDF
O
Posterior PDF
6/18/07
.2 4
Anderson: NSF August, 2006 35
Ensemble Filter Algorithms:
Ensemble Adjustment (Kalman) Filter.
Use deterministic algorithm to ‘adjust’ ensemble−4 −2 00
0.2
0.4
0.6
Pro
babi
lity
Posterior PDF
6/18/07
.f posterior.
2 4
Anderson: NSF August, 2006 36
Ensemble Filter Algorithms:
Ensemble Adjustment (Kalman) Filter.
Use deterministic algorithm to ‘adjust’ ensembleFirst, ‘shift’ ensemble to have exact mean o
−4 −2 00
0.2
0.4
0.6
Pro
babi
lity
Posterior PDF
Mean Shifted
6/18/07
.f posterior.t variance of posterior.
2 4
Anderson: NSF August, 2006 37
Ensemble Filter Algorithms:
Ensemble Adjustment (Kalman) Filter.
Use deterministic algorithm to ‘adjust’ ensembleFirst, ‘shift’ ensemble to have exact mean oSecond, use linear contraction to have exac
−4 −2 00
0.2
0.4
0.6
Pro
babi
lity
Posterior PDF
Mean Shifted
Variance Adjusted
6/18/07
le size.
s ensemble mean,
2 4
Anderson: NSF August, 2006 38
Ensemble Filter Algorithms:
Ensemble Adjustment (Kalman) Filter.
i = 1,..., ensemb
p is prior, u is update (posterior), overbar iσ is standard deviation.
−4 −2 00
0.2
0.4
0.6
Pro
babi
lity
Posterior PDF
Mean Shifted
Variance Adjusted
xiu
xip
xp–( ) σu σp⁄( )⋅ xu+=
6/18/07
sitioned or weighted.
2 4
Anderson: NSF August, 2006 39
Ensemble Filter Algorithms:
Ensemble Adjustment (Kalman) Filter.
Bimodality maintained, but not appropriately poNo problem with random outliers.
−4 −2 00
0.2
0.4
0.6
Pro
babi
lity
Posterior PDF
Mean Shifted
Variance Adjusted
6/18/07
rved variablee at Park City.
ngle variable.
nal variable.
ditional variable.
itional variables.
Anderson: NSF August, 2006 40
2: Single observed variable, single unobseSLC temperature, temperatur
So far, have known observation likelihood for si
Now, suppose model state vector has an additio
Will examine how ensemble methods update ad
Basic method generalizes to any number of add
Related to Kalman filter in subtle ways.
6/18/07
variables
e that all we knowr joint distribution.
ariable is observed.C temperature)should happen toerved variable?rk City temp.)
like a nastyersion...
Anderson: NSF August, 2006 41
Ensemble filters: Updating additional prior state
Assumis prio
One v(SL
What unobs
(Pa
Looks inv
3
3.5
4
4.5
5
Uno
bser
ved
Sta
te V
aria
ble
−2 0 2 4Observed Variable
6/18/07
variables
e that all we knowr joint distribution.
ariable is observed.
e observedle with one ofus methods.
Anderson: NSF August, 2006 42
Ensemble filters: Updating additional prior state
Assumis prio
One v
Updatvariabprevio
33.5
44.5
5
Uno
bs.
−2 0 2 4Observed Variable
6/18/07
variables
e that all we knowr joint distribution.
ariable is observed.
e observedle with one ofus methods.
Anderson: NSF August, 2006 43
Ensemble filters: Updating additional prior state
Assumis prio
One v
Updatvariabprevio
33.5
44.5
5
Uno
bs.
−2 0 2 4Observed Variable
6/18/07
variables
e that all we knowr joint distribution.
ariable is observed.
e observedle with one ofus methods.
Anderson: NSF August, 2006 44
Ensemble filters: Updating additional prior state
Assumis prio
One v
Updatvariabprevio
33.5
44.5
5
Uno
bs.
−2 0 2 4Observed Variable
6/18/07
variables
e that all we knowr joint distribution.
ariable is observed.
ute increments fornsemble memberserved variable.
Anderson: NSF August, 2006 45
Ensemble filters: Updating additional prior state
Assumis prio
One v
Compprior eof obs
33.5
44.5
5
Uno
bs.
−2 0 2 4Observed Variable
Increments
6/18/07
variables
e that all we knowr joint distribution.
ariable is observed.
ute increments fornsemble memberserved variable.
Anderson: NSF August, 2006 46
Ensemble filters: Updating additional prior state
Assumis prio
One v
Compprior eof obs
33.5
44.5
5
Uno
bs.
−2 0 2 4Observed Variable
Increments
6/18/07
variables
e that all we knowr joint distribution.
ariable is observed.
ute increments fornsemble memberserved variable.
Anderson: NSF August, 2006 47
Ensemble filters: Updating additional prior state
Assumis prio
One v
Compprior eof obs
33.5
44.5
5
Uno
bs.
−2 0 2 4Observed Variable
Increments
6/18/07
variables
e that all we knowr joint distribution.
ariable is observed.
ute increments fornsemble memberserved variable.
Anderson: NSF August, 2006 48
Ensemble filters: Updating additional prior state
Assumis prio
One v
Compprior eof obs
33.5
44.5
5
Uno
bs.
−2 0 2 4Observed Variable
Increments
6/18/07
variables
e that all we knowr joint distribution.
ariable is observed.
ute increments fornsemble memberserved variable.
Anderson: NSF August, 2006 49
Ensemble filters: Updating additional prior state
Assumis prio
One v
Compprior eof obs
33.5
44.5
5
Uno
bs.
−2 0 2 4Observed Variable
Increments
6/18/07
variables
e that all we knowr joint distribution.
ariable is observed.
only incrementsntees that ifvation had not on observedle, unobservedle is unchangedy desirable).
Anderson: NSF August, 2006 50
Ensemble filters: Updating additional prior state
Assumis prio
One v
Usingguaraobserimpacvariabvariab(highl
33.5
44.5
5
Uno
bs.
−2 0 2 4Observed Variable
Increments
6/18/07
variables
e that all we knowr joint distribution.
hould theerved variable beted?
hoice: least squares
alent to linearsion.
as assumingal prior.
Anderson: NSF August, 2006 51
Ensemble filters: Updating additional prior state
Assumis prio
How sunobsimpac
First c
Equivregres
Samebinorm
3
3.5
4
4.5
5
Uno
bser
ved
Sta
te V
aria
ble
−2 0 2 4Observed Variable
Increments
6/18/07
variables
joint priorution of twoles.
hould theerved variable beted?
hoice: least squares
by findingleastes fit.
Anderson: NSF August, 2006 52
Ensemble filters: Updating additional prior state
Have distribvariab
How sunobsimpac
First c
Beginsquar3
3.5
4
4.5
5
Uno
bser
ved
Sta
te V
aria
ble
−2 0 2 4Observed Variable
Increments
6/18/07
variables
joint priorution of twoles.
regress theved variableents ontoents for theerved variable.
alent to first finding of increment inpace.
Anderson: NSF August, 2006 53
Ensemble filters: Updating additional prior state
Have distribvariab
Next, obserincremincremunobs
Equivimagejoint s
3
3.5
4
4.5
5
Uno
bser
ved
Sta
te V
aria
ble
−2 0 2 4Observed Variable
Increments
6/18/07
variables
joint priorution of twoles.
regress theved variableents ontoents for theerved variable.
alent to first finding of increment inpace.
Anderson: NSF August, 2006 54
Ensemble filters: Updating additional prior state
Have distribvariab
Next, obserincremincremunobs
Equivimagejoint s
3
3.5
4
4.5
5
Uno
bser
ved
Sta
te V
aria
ble
−2 0 2 4Observed Variable
Increments
6/18/07
variables
joint priorution of twoles.
regress theved variableents ontoents for theerved variable.
alent to first finding of increment inpace.
Anderson: NSF August, 2006 55
Ensemble filters: Updating additional prior state
Have distribvariab
Next, obserincremincremunobs
Equivimagejoint s
3
3.5
4
4.5
5
Uno
bser
ved
Sta
te V
aria
ble
−2 0 2 4Observed Variable
Increments
6/18/07
variables
joint priorution of twoles.
regress theved variableents ontoents for theerved variable.
alent to first finding of increment inpace.
Anderson: NSF August, 2006 56
Ensemble filters: Updating additional prior state
Have distribvariab
Next, obserincremincremunobs
Equivimagejoint s
3
3.5
4
4.5
5
Uno
bser
ved
Sta
te V
aria
ble
−2 0 2 4Observed Variable
Increments
6/18/07
variables
joint priorution of twoles.
regress theved variableents ontoents for theerved variable.
alent to first finding of increment inpace.
Anderson: NSF August, 2006 57
Ensemble filters: Updating additional prior state
Have distribvariab
Next, obserincremincremunobs
Equivimagejoint s
3
3.5
4
4.5
5
Uno
bser
ved
Sta
te V
aria
ble
−2 0 2 4Observed Variable
Increments
6/18/07
variables
joint priorution of twoles.
ssion: Equivalent toding image ofent in joint space.
projecting frompace ontoerved priors.
, multiply by priorle correlation.
Anderson: NSF August, 2006 58
Ensemble filters: Updating additional prior state
Have distribvariab
Regrefirst finincrem
Then joint sunobs
Finallysamp
3
3.5
4
4.5
5
Uno
bser
ved
Sta
te V
aria
ble
−2 0 2 4Observed Variable
Increments
6/18/07
variables
joint priorution of twoles.
ssion: Equivalent toding image ofent in joint space.
projecting frompace ontoerved priors.
, multiply by priorle correlation.
Anderson: NSF August, 2006 59
Ensemble filters: Updating additional prior state
Have distribvariab
Regrefirst finincrem
Then joint sunobs
Finallysamp
3
3.5
4
4.5
5
Uno
bser
ved
Sta
te V
aria
ble
−2 0 2 4Observed Variable
Increments
6/18/07
variables
joint priorution of twoles.
ssion: Equivalent toding image ofent in joint space.
projecting frompace ontoerved priors.
, multiply by priorle correlation.
Anderson: NSF August, 2006 60
Ensemble filters: Updating additional prior state
Have distribvariab
Regrefirst finincrem
Then joint sunobs
Finallysamp
3
3.5
4
4.5
5
Uno
bser
ved
Sta
te V
aria
ble
−2 0 2 4Observed Variable
Increments
6/18/07
variables
joint priorution of twoles.
ssion: Equivalent toding image ofent in joint space.
projecting frompace ontoerved priors.
, multiply by priorle correlation.
Anderson: NSF August, 2006 61
Ensemble filters: Updating additional prior state
Have distribvariab
Regrefirst finincrem
Then joint sunobs
Finallysamp
3
3.5
4
4.5
5
Uno
bser
ved
Sta
te V
aria
ble
−2 0 2 4Observed Variable
Increments
6/18/07
variables
joint priorution of twoles.
ssion: Equivalent toding image ofent in joint space.
projecting frompace ontoerved priors.
, multiply by priorle correlation.
Anderson: NSF August, 2006 62
Ensemble filters: Updating additional prior state
Have distribvariab
Regrefirst finincrem
Then joint sunobs
Finallysamp
3
3.5
4
4.5
5
Uno
bser
ved
Sta
te V
aria
ble
−2 0 2 4Observed Variable
Increments
6/18/07
variables
ave an updatedrior) ensemble forobserved variable.
Anderson: NSF August, 2006 63
Ensemble filters: Updating additional prior state
Now h(postethe un
3
3.5
4
4.5
5
Uno
bser
ved
Sta
te V
aria
ble
−2 0 2 4Obs.
6/18/07
variables
ave an updatedrior) ensemble forobserved variable.
Gaussians showsean and variancehanged.
Anderson: NSF August, 2006 64
Ensemble filters: Updating additional prior state
Now h(postethe un
Fittingthat mhave c
3
3.5
4
4.5
5
Uno
bser
ved
Sta
te V
aria
ble
Prior State Fit
−2 0 2 4Obs.
6/18/07
variables
ave an updatedrior) ensemble forobserved variable.
Gaussians showsean and variancehanged.
features of theistribution mayave changed.
Anderson: NSF August, 2006 65
Ensemble filters: Updating additional prior state
Now h(postethe un
Fittingthat mhave c
Otherprior dalso h
3
3.5
4
4.5
5
Uno
bser
ved
Sta
te V
aria
ble
Prior State Fit
Posterior Fit
−2 0 2 4Obs.
6/18/07
variables
CAL POINT:
impact onerved variable is a linearsion, can do thisENDENTLY for
umber oferved variables!
also do many atusing matrixra as in traditionaln Filter.
Anderson: NSF August, 2006 66
Ensemble filters: Updating additional prior state
CRITI
SinceunobssimplyregresINDEPany nunobs
Couldonce algebKalma
3
3.5
4
4.5
5
Uno
bser
ved
Sta
te V
aria
ble
Prior State Fit
Posterior Fit
−2 0 2 4Obs.
6/18/07
nd observations
ference Equation:
(1)
(2)
(nice, not essential).
(3)
(4)
t:
(5)
tk t0≥>
Anderson: NSF August, 2006 67
Phase 3: Generalize to geophysical models a
Dynamical system governed by (stochastic) Dif
Observations at discrete times:
Observational error white in time and Gaussian
Complete history of observations is:
Goal: Find probability distribution for state at time
dxt f xt t,( )= G xt t,( )dβt+ t 0≥,
yk h xk tk,( )= vk k;+ 1 2 … tk 1+;, ,=
vk N 0 Rk,( )→
Yτ yl tl τ≤;{ }=
p x t Yt,( )
6/18/07
nd observations
Difference Equation.
(6)
(7)
(8)
ator:
(9)
1)
------
x
Anderson: NSF August, 2006 68
Phase 3: Generalize to geophysical models a
State between observation times obtained fromNeed to update state given new observation:
Apply Bayes rule:
Noise is white in time (3) so:
Integrate numerator to get normalizing denomin
p x tk Ytk,( ) p x tk yk Ytk 1–
,,( )=
p x tk Ytk,( )
p yk xk Ytk 1–,( ) p x tk Ytk –
,(
p yk Ytk 1–( )
-----------------------------------------------------------------------=
p yk xk Ytk 1–,( ) p yk xk( )=
p yk Ytk 1– p yk x( ) p x tk Ytk 1–
,( )d∫=
6/18/07
nd observations
(10)
t x and y are vectors.
each observation.
state vector.
1)
)dξ---------
Anderson: NSF August, 2006 69
Phase 3: Generalize to geophysical models a
Probability after new observation:
Exactly analogous to earlier derivation except tha
EXCEPT, no guarantee we have prior sample for
SO, let’s make sure we have priors by ‘extending’
p x tk Ytk,
p yk x( ) p x tk Ytk –
,(
p yk ξ( ) p ξ tk Ytk 1–,(∫
---------------------------------------------------------=
6/18/07
nd observations
on vector.
(2)
ues of observations.
mple of state vector x.
observations.
1 tk t0≥>
Anderson: NSF August, 2006 70
Phase 3: Generalize to geophysical models a
Extending the state vector to joint state-observati
Recall:
Applying h to x at a given time gives expected val
Get prior sample of obs. by applying h to each sa
Let z = [x, y] be the combined vector of state and
yk h xk tk,( )= vk k;+ 1 2 … tk +;, ,=
6/18/07
nd observations
(10.ext)
Anderson: NSF August, 2006 71
Phase 3: Generalize to geophysical models a
NOW, we have a prior for each observation:
p z tk Ytk,
p yk z( ) p z tk Ytk 1–
,( )
p yk ξ( ) p ξ tk Ytk 1–,( )dξ∫
------------------------------------------------------------------=
6/18/07
nd observations
ons in set yk?
of those in set j.
rmalizations.
yk1
yk2 … yk
s, , ,{ }=
Anderson: NSF August, 2006 72
Phase 3: Generalize to geophysical models a
One more issue: how to deal with many observati
Let yk be composed of s subsets of observations:
Observational errors for obs. in set i independent
Then:
Can rewrite (10.ext) as series of products and no
yk
p yk z( ) p yki z( )
i 1=
s
∏=
6/18/07
nd observations
ons in set yk?
equentially.
y independent error sequentially.
a.
e obs. error covariance.ated space.
dent errors!
Anderson: NSF August, 2006 73
Phase 3: Generalize to geophysical models a
One more issue: how to deal with many observati
Implication: can assimilate observation subsets s
If subsets are scalar (individual obs. have mutualldistributions), can assimilate each observation
If not, have two options:1. Repeat everything above with matrix algebr
2. Do singular value decomposition; diagonalizAssimilate observations sequentially in rotRotate result back to original space.
Good news: Most geophysical obs. have indepen
6/18/07
re)ilable.
Anderson: NSF August, 2006 74
Applying an Ensemble Filter.
Ensemble stateestimate after usingprevious observation(analysis).
Ensemble state attime of next obser-vation (prior).
tk tk+1
1. Use model to advanceensemble (3 members heto time at which next observation becomes ava
****
6/18/07
=h(x), byember.
servationsments withed errors canequentially.
Anderson: NSF August, 2006 75
Applying an Ensemble Filter.
2. Get prior ensemble sample of observation, yapplying forward operator h to each ensemble m
Theory: obfrom instruuncorrelatbe done s
y
****
h hh
6/18/07
ibution
Anderson: NSF August, 2006 76
Applying an Ensemble Filter.
3. Getobserved valueandobservational error distrfrom observing system.
y
****
h hh
6/18/07
mbleation errors).
y
ce betweenrs of ensem-imarily increment.
Anderson: NSF August, 2006 77
Applying an Ensemble Filter.
4. Findincrement for each prior observation ense(this is a scalar problem for uncorrelated observ
y
****
h hh Note: Differen
different flavoble filters is probservation in
6/18/07
ariable to linearlyble increments.
y
impact oftion increments onte variable can be sequentially!
Anderson: NSF August, 2006 78
Applying an Ensemble Filter.
5. Use ensemble samples of y and each state vregress observation increments onto state varia
y
****
h hh
Theory:observaeach stahandled
6/18/07
variable are updated,bservation...
y
tk+2
Anderson: NSF August, 2006 79
Applying an Ensemble Filter.
6. When all ensemble members for each state have a new analysis. Integrate to time of next o
y
****
h hh
tk
6/18/07
aotic Model.
on in red.
mble in green.
all three state
r variance = 4.0.
ber ensembles.
Anderson: NSF August, 2006 80
Some Fun with the Lorenz-63 3-Variable Ch
Observati
Prior ense
Observingvariables.
Obs. erro
4 20-mem−20
0
20 −20
0
2010
20
30
40
6/18/07
aotic Model.
on in red.
mble in green.
Anderson: NSF August, 2006 81
Some Fun with the Lorenz-63 3-Variable Ch
Observati
Prior ense
−20
0
20 −20
0
2010
20
30
40
6/18/07
aotic Model.
on in red.
mble in green.
Anderson: NSF August, 2006 82
Some Fun with the Lorenz-63 3-Variable Ch
Observati
Prior ense
−100
10−20
0
2010
20
30
40
6/18/07
aotic Model.
on in red.
mble in green.
Anderson: NSF August, 2006 83
Some Fun with the Lorenz-63 3-Variable Ch
Observati
Prior ense
−100
10−20
0
2010
20
30
40
6/18/07
aotic Model.
on in red.
mble in green.
is passing throughble region.
Anderson: NSF August, 2006 84
Some Fun with the Lorenz-63 3-Variable Ch
Observati
Prior ense
Ensembleunpredicta
−100
10−20
0
2010
20
30
40
6/18/07
aotic Model.
on in red.
mble in green.
semble heads forthe rest for the
Anderson: NSF August, 2006 85
Some Fun with the Lorenz-63 3-Variable Ch
Observati
Prior ense
Part of enone lobe, other.
−100
10−20
0
2010
20
30
40
6/18/07
aotic Model.
on in red.
mble in green.
is not linear here.
regression might be.
Anderson: NSF August, 2006 86
Some Fun with the Lorenz-63 3-Variable Ch
Observati
Prior ense
The prior
Standard pretty bad
.
−100
10−20
0
2010
20
30
40
6/18/07
aotic Model.
on in red.
mble in green.
is not linear here.
er hand...
ntrive examples
like this notin real assimilations.
Anderson: NSF August, 2006 87
Some Fun with the Lorenz-63 3-Variable Ch
Observati
Prior ense
The prior
On the oth
Hard to cothis bad.
Behavior apparent
−100
10−20
0
2010
20
30
40
6/18/07
o errors.
y
tk+2
pling Error;ian Assumption
Error;inearelation
Anderson: NSF August, 2006 88
Basic filter implementation is subject t
y
****
h hh
tk
1. Model Error
2. h errors;Representativeness
4. SamGauss
5. SamplingAssuming LStatistical R
3. ‘Gross’ Obs. Errors
6/18/07
se unobservedariable is known to
related to set ofved variables.
erved variable remainnged.
servations be oftic wind velocity.
variable isity temperature.
Anderson: NSF August, 2006 89
Regression sampling error and filter divergence
Suppostate vbe unobser
Unobsshoulduncha
Let obAntarc
State Park C
−3
−2
−1
0
1
2
3
Uno
bser
ved
Sta
te V
aria
ble SD=0.88
MN=0.12
−2 0 2Observed Variable
6/18/07
se unobservedariable is known to
related to set ofved variables.
samples from jointution will haveero correlationcted |corr| = 0.19 samples).
ne observation,. variable mean andhange.
Anderson: NSF August, 2006 90
Regression sampling error and filter divergence
Suppostate vbe unobser
Finitedistribnon-z(expefor 20
After ounobsS.D. c
−3
−2
−1
0
1
2
3
Uno
bser
ved
Sta
te V
aria
ble SD=0.88
MN=0.12
SD=0.82MN=−0.27
After Obs. 1
Sample Correl. = 0.49
−2 0 2Observed Variable
6/18/07
se unobservedariable is known to
related to set ofved variables.
erved variableremain unchanged
erved means a random walk asobs. are used.
Anderson: NSF August, 2006 91
Regression sampling error and filter divergence
Suppostate vbe unobser
Unobsshould
Unobsfollowmore
−3
−2
−1
0
1
2
3
Uno
bser
ved
Sta
te V
aria
ble SD=0.88
MN=0.12
SD=0.62MN=0.50
After Obs. 21
Sample Correl. = −0.24
−2 0 2Observed Variable
6/18/07
se unobservedariable is known to
related to set ofved variables.
erved variableremain unchanged
erved standardion is persistentlyased.
ted change in |SD|ative for any non-ample correlation!
Anderson: NSF August, 2006 92
Regression sampling error and filter divergence
Suppostate vbe unobser
Unobsshould
Unobsdeviatdecre
Expecis negzero s
−3
−2
−1
0
1
2
3
Uno
bser
ved
Sta
te V
aria
ble SD=0.88
MN=0.12
SD=0.49MN=0.36
After Obs. 41
Sample Correl. = 0.01
−2 0 2Observed Variable
6/18/07
se unobservedariable is known to
related to set ofved variables.
erved variableremain unchanged
erved standardion is persistentlyased.
ted change in |SD|ative for any non-ample correlation!
Anderson: NSF August, 2006 93
Regression sampling error and filter divergence
Suppostate vbe unobser
Unobsshould
Unobsdeviatdecre
Expecis negzero s
−3
−2
−1
0
1
2
3
Uno
bser
ved
Sta
te V
aria
ble SD=0.88
MN=0.12
SD=0.40MN=0.12
After Obs. 61
Sample Correl. = 0.26
−2 0 2Observed Variable
6/18/07
se unobservedariable is known to
related to set ofved variables.
erved variableremain unchanged
erved standardion is persistentlyased.
ted change in |SD|ative for any non-ample correlation!
Anderson: NSF August, 2006 94
Regression sampling error and filter divergence
Suppostate vbe unobser
Unobsshould
Unobsdeviatdecre
Expecis negzero s
−3
−2
−1
0
1
2
3
Uno
bser
ved
Sta
te V
aria
ble SD=0.88
MN=0.12
SD=0.26MN=0.14
After Obs. 81
Sample Correl. = 0.25
−2 0 2Observed Variable
6/18/07
se unobservedariable is known to
related to set ofved variables.
ates of unobs.e too confident
rogressively lesst to any meaningfulvations.
sult can be thatingful obs. aretially ignored.
Anderson: NSF August, 2006 95
Regression sampling error and filter divergence
Suppostate vbe unobser
Estimbecom
Give pweighobser
End remeanessen
−3
−2
−1
0
1
2
3
Uno
bser
ved
Sta
te V
aria
ble SD=0.88
MN=0.12
SD=0.19MN=0.03
After Obs. 101
Sample Correl. = −0.29
−2 0 2Observed Variable
6/18/07
hows expectedte value of sample
ation vs. trueation.
decrease withle size and for largeorrelations|.
Anderson: NSF August, 2006 96
Regression sampling error and filter divergence
Plot sabsolucorrelcorrel
Errorssamp|real c
0 0.5 10
0.2
0.4
0.6
0.8
1
True Correlation
Exp
ecte
d |S
ampl
e C
orre
latio
n|
10 Members20 Members40 Members80 Members
6/18/07
g error:
is smallce in priors.
n between
r and correct for it.
Anderson: NSF August, 2006 97
Ways to deal with regression samplin
1. Ignore it: if number of unrelated observationsand there is some way of maintaining varian
2. Use larger ensembles to limit sampling error.
3. Use additionala priori information about relatioobservations and state variables.
4. Try to determine the amount of sampling erro
6/18/07
g error:
tion between
e from observation.olynomial.
2000?)
Anderson: NSF August, 2006 98
Ways to deal with regression samplin
3. Use additional a priori information about relaobservations and state variables.
Atmospheric assimilation problems.Weight regression as function of horizontaldistancGaspari-Cohn: 5th order compactly supported p
−2000 −1000 0 10000
0.5
1
Distance from Observation (Km
Reg
ress
ion
Wei
ght
6/18/07
g error:
tion between
variable pairs.
2000?)
Anderson: NSF August, 2006 99
Ways to deal with regression samplin
3. Use additional a priori information about relaobservations and state variables.
Can use other functions to weight regression.Unclear whatdistance means for some obs./stateReferred to asLOCALIZATION.
−2000 −1000 0 10000
0.5
1
Distance from Observation (Km
Reg
ress
ion
Wei
ght
6/18/07
g error:
r and correct for it:
le correlation.
rge sample correl.
easure sampling error.
with a sample removed.
efficients and weight.
Anderson: NSF August, 2006 100
Ways to deal with regression samplin
4. Try to determine the amount of sampling erro
A. Could weight regressions based on sampLimited success in tests.For small true correlations, can still get la
B. Do bootstrap with sample correlation to mLimited success.Repeatedly compute sample correlation
C. Use hierarchical Monte Carlo.Have a ‘sample’ of samples.Compute expected error in regression co
6/18/07
g error: ensembles.
N-member ensem-
s. increments for.
s. / state pair:mples of regression
β.y inβ implies staterements should be
regression confi-r,α.
Anderson: NSF August, 2006 101
Ways to deal with regression samplin4C. Use hierarchical Monte Carlo: ensemble of
y
****
H HH
y
tk+2
tk
y
****
H HH
y
tk+2
tk
M independentN-memberEnsembles
M groups of bles.
Compute obeach group
For given ob1. Have M sa
coefficient,2. Uncertaint
variable increduced.
3. Compute dence facto
β1
βΜ
RegressionConfidenceFactor,α
6/18/07
g error:
ensembles.
es 4 groups of 20.
efficient,βi.
inimizes:
te increments).
Anderson: NSF August, 2006 102
Ways to deal with regression samplin
4C. Use hierarchical Monte Carlo: ensemble of
Split ensemble into M independent groups.For instance, 80 ensemble members becom
With M groups get M estimates of regression co
Find regression confidence factorα (weight) that m
Minimizes RMS error in the regression (and sta
αβi β j–[ ]2
i 1 i j≠,=
M∑
j 1=
M∑
6/18/07
g error:
ensembles.
t regression byα.
has repeatedations, cante sample mean or
n statistics forα.
can be used inquent assimilationscalization.
mple mean regression)
Anderson: NSF August, 2006 103
Ways to deal with regression samplin
4C. Use hierarchical Monte Carlo: ensemble of
Weigh
If one observgeneramedia
Meanαsubseas a lo
α is function of M and (sample SD /sa
0 1 2 3 4 50
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
Q: Ratio of sample standard deviation to mean
Re
gre
ssio
n C
on
fide
nce
Fa
cto
r, α
Group Size 2Group Size 4Group Size 8Group Size 16
Q Σβ β⁄=
6/18/07
ssure Obs. at 20N, 60E
60 80 100
an factor level 3
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
0 60 80 100
s section at row 18
Anderson: NSF August, 2006 104
Localization in GCM can be very complex. Surface Pre
20 40 60 80 100−10
0
10
20
30
40
50
60u mean factor level 1
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
20 40 60 80 100−10
0
10
20
30
40
50
60u mean factor level 2
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
20 40−10
0
10
20
30
40
50
60u me
20 40 60 80 100−10
0
10
20
30
40
50
60u mean factor level 4
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
20 40 60 80 100−10
0
10
20
30
40
50
60u mean factor level 5
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
20 40
0.2
0.4
0.6
0.8
1cros
PS to U
6/18/07
rors
independentlyt ongoing.
emble filters...
riance inflation,rior uncertainty
s more impact.
ation’: only letct a set oftate variables.
othly decrease0 as function of
Anderson: NSF August, 2006 105
Dealing With Ensemble Filter Er
y
****
h hh
y
tk+2
tk
1. Model Error
2. h errors;Representativeness
4. Sampling Error;Gaussian Assumption
5. Sampling Error;Assuming LinearStatistical Relation
Fix 1, 2, 3HARD bu
Often, ens
1-4: CovaIncrease pto give ob
5. ‘Localizobs. impa‘nearby’ s
Often smoimpact to distance.
3. Gross Obs. Errors
6/18/07
ance Inflation
=> ‘true’ distribution.
−1 0
Anderson: NSF August, 2006 106
Model/Filter Error; Filter Divergence and Vari
1. History of observations and physical system
−4 −3 −20
0.5
1
Pro
babi
lity
"TRUE" Prior PDF
6/18/07
ance Inflation
=> ‘true’ distribution.ufficient prior variance.
fident, obs. ignored−1 0
Anderson: NSF August, 2006 107
Model/Filter Error; Filter Divergence and Vari
1. History of observations and physical system 2. Sampling error, some model errors lead to ins
3. Can lead to ‘filter divergence’: prior is too con−4 −3 −20
0.5
1
Pro
babi
lity
"TRUE" Prior PDF
Variance Deficient PDF
6/18/07
ance Inflation
=> ‘true’ distribution.ufficient prior variance.
ase spread of prior
.
−1 0
) x+
Anderson: NSF August, 2006 108
Model/Filter Error; Filter Divergence and Vari
1. History of observations and physical system 2. Sampling error, some model errors lead to ins
3. Naive solution is Variance inflation: just incre
4. For ensemble member i,
−4 −3 −20
0.5
1
Pro
babi
lity
"TRUE" Prior PDF
Variance Deficient PDF
inflate xi( ) λ xi x–(=
6/18/07
ance Inflation
=> ‘true’ distribution.
2 4
Anderson: NSF August, 2006 109
Model/Filter Error; Filter Divergence and Vari
1. History of observations and physical system .
−4 −2 00
0.2
0.4
0.6
0.8
Pro
babi
lity
"TRUE" Prior PDF
6/18/07
ance Inflation
=> ‘true’ distribution.ift in entire distribution.
RTAIN2 4
n (from model)
Anderson: NSF August, 2006 110
Model/Filter Error; Filter Divergence and Vari
1. History of observations and physical system 2. Most model errors also lead to erroneous sh
3. Again, prior can be viewed as being TOO CE−4 −2 00
0.2
0.4
0.6
0.8
Pro
babi
lity
"TRUE" Prior PDF Error in Mea
6/18/07
ance Inflation
=> ‘true’ distribution.ift in entire distribution.
RTAIN
or it directly
2 4
n (from model)
ariance Inflated
Anderson: NSF August, 2006 111
Model/Filter Error; Filter Divergence and Vari
1. History of observations and physical system 2. Most model errors also lead to erroneous sh
3. Again, prior can be viewed as being TOO CE4. Inflating can ameliorate this5. Obviously, if we knew E(error), we’d correct f
−4 −2 00
0.2
0.4
0.6
0.8
Pro
babi
lity
"TRUE" Prior PDF Error in Mea
V
6/18/07
n
e assimilation
els.
ations can ‘blow up’.
e top of AGCMs.
nd error.
Anderson: NSF August, 2006 112
Physical Space Variance Inflatio
Inflate all state variables by same amount befor
Capabilities:
1. Can be very effective for a variety of mod
2. Can maintain linear balances.
3. Stays on local flat manifolds.
4. Simple and inexpensive.
Liabilities:
1. State variables not constrained by observ
For instance unobserved regions near th
2. Magnitude ofλ normally selected by trial a
6/18/07
orenz-63
ence
vation in red.
nsemble in green.
Anderson: NSF August, 2006 113
Physical space covariance inflation in L
Observation outside prior: danger of filter diverg
Obser
Prior e−20
0
20 −20
0
2010
20
30
40
6/18/07
orenz-63
divergence avoided
vation in red.
nsemble in green.
semble in magenta.
Anderson: NSF August, 2006 114
Physical space covariance inflation in L
After inflating, observation is in prior cloud: filter
Obser
Prior e
Inflated en
−20
0
20 −20
0
2010
20
30
40
6/18/07
orenz-63
vation in red.
nsemble in green.
Anderson: NSF August, 2006 115
Physical space covariance inflation in L
Prior distribution is significantly ‘curved’
Obser
Prior e−20
0
20 −20
0
2010
20
30
40
6/18/07
orenz-63
o be off attractor.to transient off-ehavior or...
w-up’.
vation in red.
nsemble in green.
semble in magenta.
Anderson: NSF August, 2006 116
Physical space covariance inflation in L
Inflated prior outside attractor. Posterior will alsCan lead attractor b
Model ‘blo
Obser
Prior e
Inflated en
−20
0
20 −20
0
2010
20
30
40
6/18/07
Tolerate Errors.
ystem.th observations., and filter error.
Anderson: NSF August, 2006 117
Adjunct Algorithms Developed for DART Can
1. Adaptive Error Tolerant Filters.Automatically detect error in assimilation sAdd uncertainty when model disagrees wiCan deal with LARGE model, observation
6/18/07
rror Tolerant Filter
-observed inconsistency
.
posed to be unbiased.
2 4
hood
S.D.n
obs2
Anderson: NSF August, 2006 118
Variance inflation for Observations: An Adaptive E
1. For observed variable, have estimate of prior
2. Expected(prior mean - observation) =
Assumes that prior and observation are supIs it model error or random chance?
−4 −2 00
0.2
0.4
0.6
0.8
Pro
babi
lity
Prior PDF
S.D.
Obs. Likeli
Expected Separatio
Actual 4.714 SDs
σprior2 σ+
6/18/07
rror Tolerant Filter
-observed inconsistency
.
rior and observation.
2 4
hood
S.D.tion
obs2
Anderson: NSF August, 2006 119
Variance inflation for Observations: An Adaptive E
1. For observed variable, have estimate of prior
2. Expected(prior mean - observation) =
3. Inflating increases expected separation.Increases ‘apparent’ consistency between p
−4 −2 00
0.2
0.4
0.6
0.8
Pro
babi
lity
Prior PDF Obs. Likeli
Inflated S.D.Expected Separa
Actual 3.698 SDs
σprior2 σ+
6/18/07
Tolerate Errors.
ystem.th observations., and filter error.
emble sampling errors.
problems.
Anderson: NSF August, 2006 120
Adjunct Algorithms Developed for DART Can
1. Adaptive Error Tolerant Filters.Automatically detect error in assimilation sAdd uncertainty when model disagrees wiCan deal with LARGE model, observation
2. Hierarchical filters detect and avoid small ensEnsemble of ensembles for tuning period.Limit impact of observations as required.Eliminate unnecessary calculation.
Can apply filters without tuning to large
6/18/07
all three pieces:
l system.
or characteristics.
ion.
ar tangent replacement).
Anderson: NSF August, 2006 121
Assimilation increases information about _________________________________________________
1. Get an improved estimate of state of physica
Initial conditions for forecasts.Includes time evolution and ‘balances’.High quality analyses (re-analyses).
_________________________________________________
2. Get better estimates of observing system err
Estimate value of existing or planned observations.Design observing systems that provide increased informat
_________________________________________________
3. Improve model of physical system.
Evaluate model systematic errors.Forward and backward sensitivity analysis (adjoint and lineSelect appropriate values for model parameters.
6/18/07
ary, 2003
ge spread).
inds.
reanalysis.
radiances.
Anderson: NSF August, 2006 122
DART/CAM NWP Assimilation: Janu
Model: CAM 3.1 T85L26.
Initialized from a climatological distribution (hu
Observations: Radiosondes, ACARS, Satellite W
Subset of observations used in NCAR/NCEP
Compare to NCEP operational, T254L64, uses
6/18/07
Af ter 6 hours.
CAMstarts withclimatology!Nearly zonal.
Anderson: NSF August, 2006 123
NCEP
DART/CAM
Difference.
6/18/07
Af ter 1 day.
Anderson: NSF August, 2006 124
NCEP
DART/CAM
Difference.
6/18/07
Af ter 3 days.
CAM gainszonalstructure.
Anderson: NSF August, 2006 125
NCEP
DART/CAM
Difference.
6/18/07
Af ter 7 days.
NHconverged.SH poorlyobserved.
Anderson: NSF August, 2006 126
NCEP
DART/CAM
Difference.
6/18/07
perature RMS
Hemisphere
NWP system.
1.8 2 2.2 2.4 2.6 2.8MPERATURE K
S Error: Northern Hemisphere
NCEP Opnl.T85 CAM
Anderson: NSF August, 2006 127
6-Hour Forecast Observation Space Tem
Tropics Northern
DART/CAM competitive with operational
0.9 1 1.1 1.2 1.3 1.4 1.5
100
200
300
400
500
600
700
800
900
1000
Pre
ssu
re (
hP
a)
TEMPERATURE K
6−Hour Forecast RMS Error: Tropics
NCEP Opnl.T85 CAM
1 1.2 1.4 1.6
100
200
300
400
500
600
700
800
900
1000P
ressu
re (
hP
a)
TE
6−Hour Forecast RM
6/18/07
all three pieces:
l system.
or characteristics.
ion.
ar tangent replacement).
Anderson: NSF August, 2006 128
Assimilation increases information about _________________________________________________
1. Get an improved estimate of state of physica
Initial conditions for forecasts.Includes time evolution and ‘balances’.
High quality analyses (re-analyses)._________________________________________________
2. Get better estimates of observing system err
Estimate value of existing or planned observations.Design observing systems that provide increased informat
_________________________________________________
3. Improve model of physical system.
Evaluate model systematic errors.Forward and backward sensitivity analysis (adjoint and lineSelect appropriate values for model parameters.
6/18/07
-CHEM model.
T CO observations.
ted by Kevin Raeder.
Anderson: NSF August, 2006 129
High-quality analysis of CO in Finite Volume CAM
Assimilate standard observations plus MOPIT
Work by Ave Arellano and Peter Hess suppor
6/18/07
Anderson: NSF August, 2006 1306/18/07
all three pieces:
l system.
or characteristics.
ations.ion.
ar tangent replacement).
Anderson: NSF August, 2006 131
Assimilation increases information about _________________________________________________
1. Get an improved estimate of state of physica
Initial conditions for forecasts.Includes time evolution and ‘balances’.High quality analyses (re-analyses).
_________________________________________________
2. Get better estimates of observing system err
Estimate value of existing or planned observDesign observing systems that provide increased informat
_________________________________________________
3. Improve model of physical system.
Evaluate model systematic errors.Forward and backward sensitivity analysis (adjoint and lineSelect appropriate values for model parameters.
6/18/07
rvtions in WRF
eric electric field.
low earth satellite.
Anderson: NSF August, 2006 132
Assimilating GPS Radio Occultation ObseaAssimilated as refractivity along beam path.Complicated function of T, Q, P and ionosph
Get a sounding as GPS satellite sets relative to
6/18/07
rvtions in WRF
satellite.
300 320
0, 2003
Anderson: NSF August, 2006 133
Assimilating GPS Radio Occultation Obsea
Weather Research and Forecasting Model.Regional Weather Prediction model.Configured for CONUS domain, 50 km grid.
Several hundred profiles available from CHAMP
180 200 220 240 260 2800
10
20
30
40
50
60
70
80
Latit
ude
(N)
Longitude (E)
GPS RO locations in CONUS domain, Jan 1−1
6/18/07
rvtions in WRF
close to GPS profiles.
adiosonde profiles.
M, GFDL, GFS...
Anderson: NSF August, 2006 134
Assimilating GPS Radio Occultation Obsea
Evaluating Impact of GPS Observations.
Case 1: Assimilate radiosondes EXCEPT thoseCase 2: Also assimilate GPS profiles.
Look at reduction in error from close (unused) r
NOTE: Identical code allows assimilation in CA
6/18/07
Errors in WRFt pair).
0 0.5 1 1.5 mean & rms error (g/kg)
an 1−10, 2003
rad+gpsrad
Anderson: NSF August, 2006 135
GPS Radio Occultation Impact on T and QEach plot displays bias (left pair) and RMS (righRed curves include GPS: reduced bias and RMS.
−1 0 1 2 3 4
300
400
500
600
700
800
900
Pre
ssu
re(h
Pa
)
T analysis mean & rms error (K)
Jan 1−10, 2003
−1 −0.5
300
400
500
600
700
800
900
Pre
ssu
re(h
Pa
)
Q analysis
J
6/18/07
all three pieces:
l system.
or characteristics.
ased information.
ar tangent replacement).
Anderson: NSF August, 2006 136
Assimilation increases information about _________________________________________________
1. Get an improved estimate of state of physica
Initial conditions for forecasts.Includes time evolution and ‘balances’.High quality analyses (re-analyses).
_________________________________________________
2. Get better estimates of observing system err
Estimate value of existing or planned observations.
Design observing systems that provide incre_________________________________________________
3. Improve model of physical system.
Evaluate model systematic errors.Forward and backward sensitivity analysis (adjoint and lineSelect appropriate values for model parameters.
6/18/07
all three pieces:
l system.
or characteristics.
ion.
ar tangent replacement).
Anderson: NSF August, 2006 137
Assimilation increases information about _________________________________________________
1. Get an improved estimate of state of physica
Initial conditions for forecasts.Includes time evolution and ‘balances’.High quality analyses (re-analyses).
_________________________________________________
2. Get better estimates of observing system err
Estimate value of existing or planned observations.Design observing systems that provide increased informat
_________________________________________________
3. Improve model of physical system.
Evaluate model systematic errors.Forward and backward sensitivity analysis (adjoint and lineSelect appropriate values for model parameters.
6/18/07
omparisons.perature Bias
.5 0 0.5 1
rth America
_TEMPERATURE bias (C)
guess
analysis
.5 0 0.5 1
rth America
_TEMPERATURE bias (C)
guess
analysis
Anderson: NSF August, 2006 138
Example of low-resolution assimilation cCAM spectral vs. FV for January, 2003: Tem
SpectralT21
FiniteVolume2x2.5
−1.5 −1 −0.5 0 0.5 1
100
200
300
400
500
600
700
800
900
1000
Tropics
Pre
ssu
re (
hP
a)
RADIOSONDE_TEMPERATURE bias (C)
guess
analysis
−1.5 −1 −0
100
200
300
400
500
600
700
800
900
1000
No
Pre
ssu
re (
hP
a)
RADIOSONDE
−1.5 −1 −0.5 0 0.5 1
100
200
300
400
500
600
700
800
900
1000
Tropics
Pre
ssu
re (
hP
a)
RADIOSONDE_TEMPERATURE bias (C)
guess
analysis
−1.5 −1 −0
100
200
300
400
500
600
700
800
900
1000
No
Pre
ssu
re (
hP
a)
RADIOSONDE
6/18/07
all three pieces:
l system.
or characteristics.
ion.
near tangent proxy).
Anderson: NSF August, 2006 139
Assimilation increases information about _________________________________________________
1. Get an improved estimate of state of physica
Initial conditions for forecasts.Includes time evolution and ‘balances’.High quality analyses (re-analyses).
_________________________________________________
2. Get better estimates of observing system err
Estimate value of existing or planned observations.Design observing systems that provide increased informat
_________________________________________________
3. Improve model of physical system.
Evaluate model systematic errors.Forward/backward sensitivity analysis (adjoint/liSelect appropriate values for model parameters.
6/18/07
ANY forecast ord analysis quantities
of adjoint andiods.
ns, or functions thereof.
mperature.a temperatures.
semble.
Anderson: NSF August, 2006 140
Ensemble Sensitivity Analysis
Can compute correlation (covariance) betweenanalysis quantity and ALL other forecast anor functions thereof at any time lag.
Can get same information as unlimited numberlinear tangent integrations over arbitrary per
Explore relations between variables, observatio
Example 1: Base point is 500 hPa mid-latitude teLook at impact on evolution of 500hP
Similar to linear tangent integration.Significant correlations from 20 member T85 en
6/18/07
t equivalent)
225
K
−1
−0.5
0
0.5
1
00 hPa Temperature
Anderson: NSF August, 2006 141
Forward in Time Sensitivity (Linear Tangen
+
degrees east
degr
ees
nort
h
135 18030
40
50
60
70
Time lag 00 hours: 500 hPa Temperature to 5
6/18/07
t equivalent)
00 hPa Temperature
225
K
−1
−0.5
0
0.5
1
Anderson: NSF August, 2006 142
Forward in Time Sensitivity (Linear Tangen
Time lag 06 hours: 500 hPa Temperature to 5
+
degrees east
degr
ees
nort
h
135 18030
40
50
60
70
6/18/07
t equivalent)
00 hPa Temperature
225
K
−1
−0.5
0
0.5
1
Anderson: NSF August, 2006 143
Forward in Time Sensitivity (Linear Tangen
Time lag 12 hours: 500 hPa Temperature to 5
+
degrees east
degr
ees
nort
h
135 18030
40
50
60
70
6/18/07
t equivalent)
00 hPa Temperature
225
K
−1
−0.5
0
0.5
1
Anderson: NSF August, 2006 144
Forward in Time Sensitivity (Linear Tangen
Time lag 18 hours: 500 hPa Temperature to 5
+
degrees east
degr
ees
nort
h
135 18030
40
50
60
70
6/18/07
t equivalent)
00 hPa Temperature
225
K
−1
−0.5
0
0.5
1
Anderson: NSF August, 2006 145
Forward in Time Sensitivity (Linear Tangen
Time lag 24 hours: 500 hPa Temperature to 5
+
degrees east
degr
ees
nort
h
135 18030
40
50
60
70
6/18/07
t equivalent)
00 hPa Temperature
225
K
−1
−0.5
0
0.5
1
Anderson: NSF August, 2006 146
Forward in Time Sensitivity (Linear Tangen
Time lag 30 hours: 500 hPa Temperature to 5
+
degrees east
degr
ees
nort
h
135 18030
40
50
60
70
6/18/07
ANY forecast ord analysis quantities
of adjoint andiods.
ns, or functions thereof.
onal velocity.emperature.
Anderson: NSF August, 2006 147
Ensemble Sensitivity Analysis
Can compute correlation (covariance) betweenanalysis quantity and ALL other forecast anor functions thereof.
Can get same information as unlimited numberlinear tangent integrations over arbitrary per
Explore relations between variables, observatio
Example 2: Base point is 500 hPa mid-latitude zLook at impact of previous 500 hPa t
Compare to an adjoint integration.
6/18/07
uivalent)
0 hPa Temperature
225
K
−1
−0.5
0
0.5
1
Anderson: NSF August, 2006 148
Backward in Time Sensitivity (Adjoint eq
Time lag -00 hours: 500 hPa Zonal Velocity to 50
+
degrees east
degr
ees
nort
h
135 18030
40
50
60
70
6/18/07
uivalent)
0 hPa Temperature
225
K
−1
−0.5
0
0.5
1
Anderson: NSF August, 2006 149
Backward in Time Sensitivity (Adjoint eq
Time lag -06 hours: 500 hPa Zonal Velocity to 50
+
degrees east
degr
ees
nort
h
135 18030
40
50
60
70
6/18/07
uivalent)
0 hPa Temperature
225
K
−1
−0.5
0
0.5
1
Anderson: NSF August, 2006 150
Backward in Time Sensitivity (Adjoint eq
Time lag -12 hours: 500 hPa Zonal Velocity to 50
+
degrees east
degr
ees
nort
h
135 18030
40
50
60
70
6/18/07
uivalent)
0 hPa Temperature
225
K
−1
−0.5
0
0.5
1
Anderson: NSF August, 2006 151
Backward in Time Sensitivity (Adjoint eq
Time lag -18 hours: 500 hPa Zonal Velocity to 50
+
degrees east
degr
ees
nort
h
135 18030
40
50
60
70
6/18/07
uivalent)
0 hPa Temperature
225
K
−1
−0.5
0
0.5
1
Anderson: NSF August, 2006 152
Backward in Time Sensitivity (Adjoint eq
Time lag -24 hours: 500 hPa Zonal Velocity to 50
+
degrees east
degr
ees
nort
h
135 18030
40
50
60
70
6/18/07
uivalent)
0 hPa Temperature
225
K
−1
−0.5
0
0.5
1
Anderson: NSF August, 2006 153
Backward in Time Sensitivity (Adjoint eq
Time lag -30 hours: 500 hPa Zonal Velocity to 50
+
degrees east
degr
ees
nort
h
135 18030
40
50
60
70
6/18/07
all three pieces:
l system.
or characteristics.
ion.
ar tangent replacement).
ters.
Anderson: NSF August, 2006 154
Assimilation increases information about _________________________________________________
1. Get an improved estimate of state of physica
Initial conditions for forecasts.Includes time evolution and ‘balances’.High quality analyses (re-analyses).
_________________________________________________
2. Get better estimates of observing system err
Estimate value of existing or planned observations.Design observing systems that provide increased informat
_________________________________________________
3. Improve model of physical system.
Evaluate model systematic errors.Forward and backward sensitivity analysis (adjoint and line
Select appropriate values for model parame
6/18/07
Data Assimilation.
AM assimilation of wave drag effi- parameter.
ic values are noised be 0).
ciency< ~4 sug- by modelers.
ar convection.r ‘Wrong Reason’.
LL model problems
Anderson: NSF August, 2006 155
Climate Model Parameter Estimation via Ensemble
T21 Cgravityciency
Ocean(shoul
0< effigested
Positive values over NH land expected.Problem: large negative values over tropical land neMay reduce wind bias in tropical troposphere, but fo
Assimilation tries to use free parameter to fix A
0 90 180 270
−45
0
45
−10 −5 0 5 10
6/18/07
DART)
DART.
Anderson: NSF August, 2006 156
Data Assimilation Research Testbed (
Software to do everything here (and more) is in
Requires F90 compiler, Matlab.
Available from www.image.ucar.edu/DAReS/.
6/18/07
Anderson: NSF August, 2006 157