7/30/2019 Eece 522 Notes_28 Ch_12b
1/20
1
12.6 Sequential LMMSE EstimationSame kind if setting as for Sequential LS
Fixed number of parameters (but here they are modeled as random)
Increasing number of data samples
][][][ nnn wHx +=Data Model:
(n+1)1x[n] = [x[0] x[n]]T
p1unknown PDF
known mean & cov
(n+1)1w[n] = [w[0] w[n]]T
unknown PDF
known mean & cov
Cw must be diagonal
with elements 2n
& w are uncorrelated
=][
]1[
][n
n
n Th
H
H
(n+1)pknown
Goal: Given an estimate ]1[ n based on x[n 1], when new
data samplex[n] arrives, update the estimate to ][ n
7/30/2019 Eece 522 Notes_28 Ch_12b
2/20
2
Development of Sequential LMMSE Estimate
Our Approach Here: Use vector space ideas to derive solution
for DC Level in White Noise then write down general
solution.][][ nwAnx +=
For convenience Assume bothA and w[n] have zero mean
Givenx[0] we can find the LMMSE estimate
]0[]0[}])[{(
])}[({]0[
]}0[{
]}0[{22
2
220xx
nwAE
nwAAEx
xE
AxEA
A
A
+=
+
+=
=
Now we seek to sequentially update this estimate
with the info fromx[1]
7/30/2019 Eece 522 Notes_28 Ch_12b
3/20
3
From Vector Space View: A
x[0]
x[1]0A
1A
First projectx[1] ontox[0] to get ]0|1[x
Estimate new data
given old data
Prediction!
Notation: the estimate
at 1 based on 0 Use Orthogonality Principle
]0|1[]1[]1[~ xxx = is to x[0]
This is the new, non-redundant info provided by datax[1]It is called the innovation
x[0]x[1]0A
]1[~x
7/30/2019 Eece 522 Notes_28 Ch_12b
4/20
4
Find Estimation Update by ProjectingA onto Innovation
]1[~]}1[~{
]}1[~{]1[~
]1[~
]1[~
,]1[~]1[~
]1[~]1[~
, 221 xxE
xAEx
x
xAx
x
x
xAA
===
Gain: k1
Recall Property: Two Estimates from data just add:
]0[]1[~ xx
]1[~
10
101
xkA
AAA
+=
+=
[ ]]0|1[]1[ 101 xxkAA +=
x[0]
x[1]0
A
]1[~x1A
1APredicted
New Data
New
DataOld
EstimateGain
I nnovat ion is Old Dat a
7/30/2019 Eece 522 Notes_28 Ch_12b
5/20
5
The Innovations Sequence
The Innovations Sequence is
Key to the derivation & implementation of Seq. LMMSE
A sequence of orthogonal (i.e., uncorrelated) RVs
Broadly significant in Signal Processing and Controls
]0[x
{ }],2[~],1[~],0[~ xxx
]0|1[]1[ xx
]1|2[]2[ xx
Means: Based on
ALL data up to n = 1
(inclusive)
7/30/2019 Eece 522 Notes_28 Ch_12b
6/20
6
General Sequential LMMSE Estimation
Initialization No Data Yet! Use Prior Information
}{ 1 E= Est imat e
CM =1 MMSE Mat r ix
Update Loop For n = 0, 1, 2,
nnTnn
nnn
hMh
hMk
12
1
+
=
Gain Vect or Calculat ion
[ ]11 ][ += nTnnnn nx hk Est imat e Updat e
[ ] 1= nTnnn MhkIM MMSE Mat r ix Updat e
7/30/2019 Eece 522 Notes_28 Ch_12b
7/20
7
Sequential LMMSE Block Diagram
][][][ nnn wHx +=
=
][
]1[][
n
nn
Th
HH
Data Model
kn
21 ,, nnn hM
Compute
Gain
][~ nx
z-1
1
n
+
+
x[n]
Tnh
1]1|[ = n
Tnnnx h
+
InnovationObservation
11][ + n
Tnnn nx hk
n
DelayUpdatedEstimate
Previous
EstimatenhPredicted
Observation
Exact Same St r uct ur e as f or Sequent ial Linear LS!!
7/30/2019 Eece 522 Notes_28 Ch_12b
8/20
8
Comments on Sequential LMMSE Estimation
1. Same structure as for sequential linear LS. BUT they solve
the estimation problem under very different assumptions.
2. No matrix inversion required So computationally Efficient
3. Gain vector kn weighs confidence in new data (2n) against allprevious data (Mn-1)
when previous data is better, gain is small dont use new data much
when new data is better, gain is large new data is heavily used
4. If you know noise statistics 2n and observation rows hnTover
the desired range ofn:
Can run MMSE Matrix Recursion without data measurements!!! This provides a Predictive Performance Analysis
7/30/2019 Eece 522 Notes_28 Ch_12b
9/20
9
12.7 Examples Wiener FilteringDuring WWII, Norbert Wiener developed the mathematical ideas
that led to the Wiener filter when he was working on ways toimprove anti-aircraft guns.
He posed the problem in C-T form and sought the best linear
filter that would reduce the effect of noise in the observed A/C
trajectory.
He modeled the aircraft motion as a wide-sense stationaryrandom process and used the MMSE as the criterion for
optimality. The solutions were not simple and there were many
different ways of interpreting and casting the results.
The results were difficult for engineers of the time to understand.
Others (Kolmogorov, Hopf, Levinson, etc.) developed these ideas
for the D-T case and various special cases.
7/30/2019 Eece 522 Notes_28 Ch_12b
10/20
10
Weiner Filter: Model and Problem Statement
Signal Model: x[n] =s[n] + w[n]
Observed: Noisy Signal
Model as WSS, Zero-MeanCxx = Rxx
covariance matrix
correlation
matrix}{ TE xxRxx =
}}){})({{( TEEE xxxxCxx =
Desired Signal
Model as WSS, Zero-MeanCss = Rss
NoiseModel as WSS, Zero-Mean
Cww = Rww
Same if zero-mean
Problem Statement: Processx[n] using a linear filter to providea de-noised version of the signal that has minimum MSE
relative to the desired signal
LMMSE Problem!
7/30/2019 Eece 522 Notes_28 Ch_12b
11/20
11
Filtering, Smoothing, Prediction
Terminology for three different ways to cast the Wiener filter problem
Filtering Smoothing Prediction
Given:x[0],x[1], ,x[n] Given:x[0],x[1], ,x[N-1] Given:x[0],x[1], ,x[N-1]
Find: ]1[,],1[],0[ Nsss 0,][ >+ llNxFind: ][ ns Find:
xCC xxx 1 =
x[n]
]0[s
2
1 n
]1[s
]2[s
]3[s
3
x[n]
2
1
x[n]
]5[x
2
1 4 5
Note!!
n n3 3
]0[
s ]1[
s ]2[s ]3[s
All t hr ee solved using General LMMSE Est .
7/30/2019 Eece 522 Notes_28 Ch_12b
12/20
7/30/2019 Eece 522 Notes_28 Ch_12b
13/20
13
Comments on Filtering: FIR Wiener
xaxRRr
a
Twwss
Tss
T
ns =+= ### $### %&
1)(~][
[ ]
[ ]T
nn
Tnnn
aaa
nhhh
01
)()()(
...
][...]1[]0[
=
=h
=
=n
k
n knxkhns
0
)( ][][][
Wiener Filter as
Time-Varying FIR
Filter Causal!
Length Grows!Wiener - Hopf Filt er ing Equat ions
[ ]
T
ssssssss
sswwss
nrrr
xx
][...]1[]0[
)(
=
=+
r
rhRR
R## $## %&
=
][
]1[
]0[
][
]1[
]0[
]0[]1[][
]1[]0[]1[
][]1[]0[
)(
)(
)(
Toeplitz&Symmetricnr
r
r
nh
h
h
rnrnr
nrrr
nrrr
ss
ss
ss
n
n
n
xxxxxx
xxxxxx
xxxxxx
''
###### $###### %& "
'(''
"
"
In Principle: Solve WHF Eqs for filter h at each nIn Practice: Use Levinson Recursion to Recursively Solve
7/30/2019 Eece 522 Notes_28 Ch_12b
14/20
14
Comments on Filtering: IIR Wiener
Can Show: as n Wiener filter becomes Time-InvariantThus: h(n)[k] h[k]
Then the Wiener-Hopf Equations become:
,1,0][][][0
==
=
llrklrkh ssk
xx
and these are solved using so-called Spectral Factorization
And the Wiener Filter becomes IIR Time-Invariant:
=
=0
][][][
k
knxkhns
7/30/2019 Eece 522 Notes_28 Ch_12b
15/20
15
Revisit the FIR Wiener: Fixed Length L
=
=1
0
][][][L
k
knxkhns
]6[s
The way the Wiener filter was formulated above, the length of filter
grew so that the current estimate was based on all the past data
Reformulate so that current estimate is based on onlyL most recent
data: x[3] x[4] x[5] x[6] x[7] x[8] x[9]
]7[s]8[s
Wiener - Hopf Filt er ing Equat ions f or WSS Pr ocess w/ Fixed FI R
[ ]Tssssssss
sswwss
nrrr
xx
][...]1[]0[
)(
=
=+
r
rhRRR
## $## %&
=
][
]1[
]0[
][
]1[
]0[
]0[]1[][
]1[]0[]1[
][]1[]0[
Toeplitz&Symmetric
nr
r
r
nh
h
h
rnrnr
nrrr
nrrr
ss
ss
ss
xxxxxx
xxxxxx
xxxxxx
''
####### $####### %&"
'(''
"
"
Solve W- H Filt er ing Eqs ONCE f or f ilt er h
7/30/2019 Eece 522 Notes_28 Ch_12b
16/20
16
Comments on Smoothing: FIR Smoother
WxxRRRsW
=+=
### $### %&1
)( wwssss Each row ofW like a FIR Filter Time-Varying Non-Causal!
Block-Based
To interpret this Consider N=1 Case:
]0[1
]0[]0[]0[
]0[]0[
SNRLow,0SNRHigh,1
xSNR
SNRx
rr
rs
wwss
ss
#$#%&
+=
+=
7/30/2019 Eece 522 Notes_28 Ch_12b
17/20
17
Comments on Smoothing: IIR Smoother
Estimates[n] based on {,x[1],x[0],x[1],}
=
=
k
knxkhns ][][][Time-Invariant &
Non-Causal IIR Filter
The Wiener-Hopf Equations become:
Pww( f)H(f) 0 whenPss(f)
7/30/2019 Eece 522 Notes_28 Ch_12b
18/20
18
Relationship of Prediction to AR Est. & Yule-WalkerWiener-Hopf Prediction Equations
[ ]Txxxxxxxxxxxx
Nlrlrlr ]1[...]1[][ ++==
r
rhR
+
+=
]1[
]1[][
][
]1[]0[
]0[]2[]1[
]2[]0[]1[]1[]1[]0[
Toeplitz&Symmetric
Nlr
lrlr
nh
hh
rNrNr
NrrrNrrr
xx
xx
xx
xxxxxx
xxxxxx
xxxxxx
''
######## $######## %&"
'(''
""
For l=1 we get EXACTLY the Yule-Walker Eqs used in
Ex. 7.18 to solve for the ML estimates of the AR parameters!!
FIR Prediction Coefficients are estimated AR parms
Recall: we first estimatedthe ACF lags rxx[k] using the data
Then used the estimates to find estimates of the AR parameters
xxxx rhR =
7/30/2019 Eece 522 Notes_28 Ch_12b
19/20
19
Relationship of Prediction to Inverse/Whitening Filter
)(1
1
za
u[k] x[k]
)(za
ARModel
][ kx
+
Inverse Filter: 1a(z)
FIR Pred.
Signal Observed
u[k]
White NoiseWhite Noise
1-Step PredictionImagination
&
Modeling
Physical Reality
R lt f 1 St P di ti F AR(3)
7/30/2019 Eece 522 Notes_28 Ch_12b
20/20
20
Results for 1-Step Prediction: For AR(3)
0 20 40 60 80 100-6
-4
-2
0
2
4
Sample Index, k
SignalValue
Signal Prediction
Error
At each kwe predictx[k] using past 3 samples
Application to Data Compression
Smaller Dynamic Range of Error gives More Efficient Binary Coding(e.g., DPCM Differential Pulse Code Modulation)