The Variational EM Algorithm for On-line Identification of Extended
Transcript of The Variational EM Algorithm for On-line Identification of Extended
The Variational EM Algorithm for On-lineIdentification of Extended AR Models
Václav Šmídl Anthony Quinn
UTIA, Czech Academy of Science Trinity College Dublin, Ireland
March 22, ICASSP 05
Václav Šmídl, Anthony Quinn The Variational EM for Extended AR Models March 22, ICASSP 05 1 / 1
Outline
Václav Šmídl, Anthony Quinn The Variational EM for Extended AR Models March 22, ICASSP 05 2 / 1
Univariate AR model
AR model:
dt = −a1dt−1 − a2dt−2 − . . .− aqdt−q + σet , et ∼ N (0, 1)
with
AR parameters a = [a1, . . . , aq]′, and σ ,
regressor x t = [dt−1, . . . , dt−q]′,
history Dt = [d1, d2, . . . , dt ] ,
Observation model:
f (dt |a, σ, Dt−1) = f (dt |a, σ, x t) = N(−a′x t , σ
2) .
Václav Šmídl, Anthony Quinn The Variational EM for Extended AR Models March 22, ICASSP 05 3 / 1
Recursive Bayesian Inference
Conjugacy:
f (a, σ|Dt) ∝ f (dt |a, σ, x t) f (a, σ|Dt−1) ,
same functional form
f (a, σ|Dt) = f (a, σ|s (Dt))
st = s (Dt) is time-invariant function generating sufficient statistics.
BTf (a, σ|Dt−1) f (a, σ|Dt )
f (dt |a, σ, Dt−1)
st−1 st
dt
update
Václav Šmídl, Anthony Quinn The Variational EM for Extended AR Models March 22, ICASSP 05 4 / 1
Recursive Bayesian Inference
Conjugacy:
f (a, σ|Dt) ∝ f (dt |a, σ, x t) f (a, σ|Dt−1) ,
same functional form
f (a, σ|Dt) = f (a, σ|s (Dt))
st = s (Dt) is time-invariant function generating sufficient statistics.
BTf (a, σ|Dt−1) f (a, σ|Dt )
f (dt |a, σ, Dt−1)
st−1 st
dt
update
Václav Šmídl, Anthony Quinn The Variational EM for Extended AR Models March 22, ICASSP 05 4 / 1
Recursive Bayesian Inference
Conjugacy:
f (a, σ|Dt) ∝ f (dt |a, σ, x t) f (a, σ|Dt−1) ,
same functional form
f (a, σ|Dt) = f (a, σ|s (Dt))
st = s (Dt) is time-invariant function generating sufficient statistics.
BTf (a, σ|Dt−1) f (a, σ|Dt )
f (dt |a, σ, Dt−1)
st−1 st
dt
update
Václav Šmídl, Anthony Quinn The Variational EM for Extended AR Models March 22, ICASSP 05 4 / 1
Recursive Bayesian Inference of AR parameters
Observation model for AR:
f (dt |a, σ, x t) = N(−a′x t , σ
2) ,
Conjugate distribution is Normal-inverse-Gamma, N iG:
f (a, σ|Dt) = N iG (Vt , νt)
sufficient statistics, st = {Vt , νt}:
Vt = Vt−1 +
[dt
x t
] [dt x t
]=
[vd,d ;t v ′
x ,d ;tv x ,d ;t Vx ,x ;t
],
νt = νt−1 + 1,
Moments:Ef (a|Dt ) [a] = V−1
x ,x ;tv x ,d ;t .
Václav Šmídl, Anthony Quinn The Variational EM for Extended AR Models March 22, ICASSP 05 5 / 1
Extending AR model
AR Extended AR (EAR)
etσ
a2
a1
aq
z−1
z−2
z−q
x2;t
x1;t
xq;t
dtdtet
σg−1
0
g1
g2
gq
a1
a2
aq
yt
x1;t
x2;t
xq;t
z−1
a g dtetσ yt
x t
Posterior:
f (a, σ|Dt) = N iG (Vt , νt) ,Vt = Vt−1 + z tz ′
tνt = νt−1 + 1
z t = [dt , dt−1, dt−2, . . . , dt−q]′ z t = g ′ (Dt) =[g0 (Dt) , g1 (Dt−1) , . . . , gq (Dt−1)]
′
Václav Šmídl, Anthony Quinn The Variational EM for Extended AR Models March 22, ICASSP 05 6 / 1
Mixture-based Extension of AR (MEAR)
Relaxing the assumption of known time-variant transformation.Treating g as time-variant discrete random variable:
g → g t ∈ {g1, g2, . . . , g c} ⇔ l t ∈ {l 1, l 2, . . . , l c}
The observation model is
f (dt |a, σ, l t , Dt−1) =∏c
i=1 f (dt |a, σ, x i;t , g i)li;t ,
f (li;t) = αi ,∑c
i=1αi = 1, i = 1, . . . , c,
f (dt |a, σ, Dt−1) =∑c
i=1 αi f (dt |a, σ, x i;t , g i),
EAR MEAR
a g dtetσ yt
x t
g2AR
g c
etσ
dt
α
g1
Václav Šmídl, Anthony Quinn The Variational EM for Extended AR Models March 22, ICASSP 05 7 / 1
Conjugacy Lost and Conjugacy Regained
Bayes’ rule:
f (a, σ, l t |Dt) ∝ f (dt |a, σ, l t , Dt−1) f (l t) f (a, σ|Dt−1) ,
Conditional independence approximation:
f (a, σ, l t |Dt) ≈ f (a, σ, l t |Dt) = f (a, σ|Dt) f (l t |Dt) ,
Then:
f (a, σ|Dt) f (l t |Dt) ∝ f (dt |a, σ, l t , Dt−1) f (l t) f (a, σ|Dt−1) .
BTf (a, σ|Dt−1)
Pf (dt |a, σ, l t , Dt−1)
f (a, σ|Dt )Pf (a, σ, l t |Dt )
approx.
Václav Šmídl, Anthony Quinn The Variational EM for Extended AR Models March 22, ICASSP 05 8 / 1
Restoring conjugacy in mixture-type models
Traditionally, conditional independence was achieved heuristically:
f (a, σ, l t |Dt) ≈ f (a, σ|Dt) f (l t |Dt) ,
f (l t |Dt) =∫
a,σf (a, σ, l t |Dt) dadσ,
f (a, σ|Dt) = f(
a, σ|Dt , l t
).
Quasi-Bayes: l t = Ef (l t |Dt )[l t ] ,
Viterbi-like: l t = arg maxl t f (l t |Dt) .
We favour free-form functional optimization.
Václav Šmídl, Anthony Quinn The Variational EM for Extended AR Models March 22, ICASSP 05 9 / 1
The Variational Bayes (VB) method
Two ingredients:
1 Conditional independence assumption:
f (a, σ, l t |Dt) = f1 (a, σ|Dt) f2 (l t |Dt) ,
2 Minimization of the KL divergence,
f1 (·) , f2 (·) = arg minf1 ,f2
KL(
f (a, σ|Dt) f (l t |Dt) ||f (a, σ, l t |Dt))
,
Necessary condition for minimum:
f1 (a, σ|Dt) ∝ exp(
Ef2(l t |Dt )[ln (f (a, σ, l t , Dt))]
),
f2 (l t |Dt) ∝ exp(
Ef1(a,σ|Dt )[ln (f (a, σ, l t , Dt))]
).
Solved iteratively via the Variational EM (VEM) algorithm.
Václav Šmídl, Anthony Quinn The Variational EM for Extended AR Models March 22, ICASSP 05 10 / 1
The Variational Bayes (VB) method
Two ingredients:
1 Conditional independence assumption:
f (a, σ, l t |Dt) = f1 (a, σ|Dt) f2 (l t |Dt) ,
2 Minimization of the KL divergence,
f1 (·) , f2 (·) = arg minf1 ,f2
KL(
f (a, σ|Dt) f (l t |Dt) ||f (a, σ, l t |Dt))
,
Necessary condition for minimum:
f1 (a, σ|Dt) ∝ exp(
Ef2(l t |Dt )[ln (f (a, σ, l t , Dt))]
),
f2 (l t |Dt) ∝ exp(
Ef1(a,σ|Dt )[ln (f (a, σ, l t , Dt))]
).
Solved iteratively via the Variational EM (VEM) algorithm.
Václav Šmídl, Anthony Quinn The Variational EM for Extended AR Models March 22, ICASSP 05 10 / 1
The Variational Bayes (VB) method
Two ingredients:
1 Conditional independence assumption:
f (a, σ, l t |Dt) = f1 (a, σ|Dt) f2 (l t |Dt) ,
2 Minimization of the KL divergence,
f1 (·) , f2 (·) = arg minf1 ,f2
KL(
f (a, σ|Dt) f (l t |Dt) ||f (a, σ, l t |Dt))
,
Necessary condition for minimum:
f1 (a, σ|Dt) ∝ exp(
Ef2(l t |Dt )[ln (f (a, σ, l t , Dt))]
),
f2 (l t |Dt) ∝ exp(
Ef1(a,σ|Dt )[ln (f (a, σ, l t , Dt))]
).
Solved iteratively via the Variational EM (VEM) algorithm.
Václav Šmídl, Anthony Quinn The Variational EM for Extended AR Models March 22, ICASSP 05 10 / 1
VB-observation model
f1 (a, σ|Dt) ∝ exp(
Ef (l t |Dt )
[ln
(f (dt |a, σ, l t) f (l t) f (a, σ|Dt−1)
)]),
f1 (a, σ|Dt) ∝ red}f (dt |a, σ)f (a, σ|Dt−1)
VB the generates a partial VB-observation model for each variable.
Václav Šmídl, Anthony Quinn The Variational EM for Extended AR Models March 22, ICASSP 05 11 / 1
VB-observation model
f1 (a, σ|Dt) ∝ exp(
Ef (l t |Dt )
[ln
(f (dt |a, σ, l t) f (l t) f (a, σ|Dt−1)
)]),
f1 (a, σ|Dt) ∝ red}f (dt |a, σ)f (a, σ|Dt−1)
VB the generates a partial VB-observation model for each variable.
Václav Šmídl, Anthony Quinn The Variational EM for Extended AR Models March 22, ICASSP 05 11 / 1
VB-observation model
f1 (a, σ|Dt) ∝ exp(
Ef (l t |Dt )
[ln
(f (dt |a, σ, l t) f (l t) f (a, σ|Dt−1)
)]),
f1 (a, σ|Dt) ∝ red}f (dt |a, σ)f (a, σ|Dt−1)
VB the generates a partial VB-observation model for each variable.
BTf (a, σ|Dt−1) f (a, σ|Dt )
f1(dt |a, σ, Dt−1)
BTf (l t |Dt−1) f (l t |Dt )
f2(dt |l t , Dt−1)
Václav Šmídl, Anthony Quinn The Variational EM for Extended AR Models March 22, ICASSP 05 11 / 1
VB-observation model
f1 (a, σ|Dt) ∝ exp(
Ef (l t |Dt )
[ln
(f (dt |a, σ, l t) f (l t) f (a, σ|Dt−1)
)]),
f1 (a, σ|Dt) ∝ red}f (dt |a, σ)f (a, σ|Dt−1)
VB the generates a partial VB-observation model for each variable.
BTf (a, σ|Dt−1) f (a, σ|Dt )
f1(dt |a, σ, Dt−1)
BTf (l t |Dt−1) f (l t |Dt )
f2(dt |l t , Dt−1)
Observation models interact with posteriors via moments.
Václav Šmídl, Anthony Quinn The Variational EM for Extended AR Models March 22, ICASSP 05 11 / 1
VB for MEAR
VB-posterior distributions:
f (a, σ|Dt) = N iG (Vt , νt) ,
f (lt |Dt) = Mu (w t) ,
with statistics expressed implicitly:
Vt = Vt−1 +c∑
i=1
wi;tz i;tz ′i;t
νt = νt−1 + 1,
wi;t ∝ |Ji;t |exp[− 1
2σ z ′j;t
[−1, a′
]′ [−1, a′
]z j;t (1)
− 12 z ′
j;tV−1x ,x ;tz j;t
],
Solved iteratively.Note: Statistics Vt is updated by rank-c structure.
Václav Šmídl, Anthony Quinn The Variational EM for Extended AR Models March 22, ICASSP 05 12 / 1
VB for MEAR
VB-posterior distributions:
f (a, σ|Dt) = N iG (Vt , νt) ,
f (lt |Dt) = Mu (w t) ,
with statistics expressed implicitly:
Vt = Vt−1 +c∑
i=1
wi;tz i;tz ′i;t
νt = νt−1 + 1,
wi;t ∝ |Ji;t |exp[− 1
2σ z ′j;t
[−1, a′
]′ [−1, a′
]z j;t (1)
− 12 z ′
j;tV−1x ,x ;tz j;t
],
Solved iteratively.Note: Statistics Vt is updated by rank-c structure.
Václav Šmídl, Anthony Quinn The Variational EM for Extended AR Models March 22, ICASSP 05 12 / 1
VB for MEAR
VB-posterior distributions:
f (a, σ|Dt) = N iG (Vt , νt) ,
f (lt |Dt) = Mu (w t) ,
with statistics expressed implicitly:
Vt = Vt−1 +c∑
i=1
wi;tz i;tz ′i;t
νt = νt−1 + 1,
wi;t ∝ |Ji;t |exp[− 1
2σ z ′j;t
[−1, a′
]′ [−1, a′
]z j;t (1)
− 12 z ′
j;tV−1x ,x ;tz j;t
],
Solved iteratively.Note: Statistics Vt is updated by rank-c structure.
Václav Šmídl, Anthony Quinn The Variational EM for Extended AR Models March 22, ICASSP 05 12 / 1
MEAR with time-variant weights
Consider Markov model of label evolution:
f (lt |lt−1, Q) =∏c
i,j=1 q li;t lj;t−1
i,j ,
with unknown transition matrix Q.Conditional independence is imposed between the weights:
f (a, σ, lt , lt−1|Dt) ≈ f (a, σ|Dt) f (lt |Dt) f (lt−1|Dt) ,
Posterior distributions are again Multinomial on both labels, lt , lt−1,and Dirichlet on Q.
Václav Šmídl, Anthony Quinn The Variational EM for Extended AR Models March 22, ICASSP 05 13 / 1
MEAR with time-variant parameters
Since conjugacy was restored using VB, the technique of forgettingcan be used for dealing with time-variant parameters.
f (at , σt |Dt−1) ∝[f (at−1, σt−1|Dt−1)
]φ
at−1 → at
σt−1 → σt
[f (at , σt)
]1−φ
In combination with time-variant weights we use two approximationsof time-update step of stochastic filtering: (i) VB for weights, (ii)forgetting for a and σ.
BT f (at , σt |Dt )
f1(dt |a, σ, Dt−1)
φ
f (at , σt )
f (at−1, σt−1|Dt−1)
f (at , σt |Dt−1)
Václav Šmídl, Anthony Quinn The Variational EM for Extended AR Models March 22, ICASSP 05 14 / 1
Outlier corrupted AR
Outlier:
yt = −ax t + σet ,
dt = yt + ξt , ξt =
{het
0
Transformations:g1 : z t = [dt , . . . , dt−q]′,g2 :z t = 1
h [dt , . . . , dt−q]′,g i : i = 3 . . . qz t = [dt , . . . , yt−i , . . . dt−q]
′,
VB result
55 60 65 700
1
time (t )
w4,
t
0
1
w3,
t
0
1
w2,
t
0
1
w1,
t
component weights (first outlier)
60 70 80 90− 2
− 1
0
1
time (t )
Signals and reconstructions
QB result
55 60 65 700
1
time (t )
w4,
t
0
1
w3,
t
0
1
w2,
t
0
1
w1,
t
60 70 80 90− 2
− 1
0
1
time (t )
Václav Šmídl, Anthony Quinn The Variational EM for Extended AR Models March 22, ICASSP 05 15 / 1
Burst noise corrupted AR
Burst noise:
yt = −ax t + σet ,
dt = yt + ξt , ξt ∼ N (0, h) ,
with unknown h.Transformations:g1 : z t = [dt , . . . , dt−q]′,g i : z t = 1
κi[dt , x i;t ],
where κi and x i;t are from Kalmanfilter.Experiment was performed withforgetting factor φ = 0.95.
1100 1200 1300 1400 1500
−0.4
−0.2
0
0.2
Unc
orru
pted
Speach segment 1
1100 1200 1300 1400 1500
−0.4
−0.2
0
0.2
Cor
rupt
ed
1100 1200 1300 1400 1500
−0.4
−0.2
0
0.2
VB
reco
nstru
ctio
n1100 1200 1300 1400 1500
−0.4
−0.2
0
0.2
QB
reco
nstru
ctio
n
1100 1200 1300 1400 1500
−0.4
−0.2
0
0.2
time (t)
VL
reco
nstru
ctio
n
Václav Šmídl, Anthony Quinn The Variational EM for Extended AR Models March 22, ICASSP 05 16 / 1
Conclusion
Mixture-based extension of the AR for dealing with unknowntransformations was presented.
The Variational Bayes technique was used to achieve conjugateupdate of parameter statistics
Applied to reconstruction of speech corrupted by burst noise.
Václav Šmídl, Anthony Quinn The Variational EM for Extended AR Models March 22, ICASSP 05 17 / 1
Spanning
The chosen transformation (filter-bank) forms nodes of a simplex ofpossible dyads. Update by a linear combination of these dyads allowsus to span interior of the simplex.
desired behaviour undesired behaviourg2g1
g3
g2g1
g3
This should be remembered when designing a filter-bank.
Václav Šmídl, Anthony Quinn The Variational EM for Extended AR Models March 22, ICASSP 05 18 / 1
Computational flow
w tO.P.
w t−1
11−z−1
11−z−1
z−1
g2
z2;tO.P.
z2;t z′2;t
g c
zc;tO.P.
zc;t z′c;t
......
g1
z1;tO.P.
z1;t z′1;t
dt
w2;tEq. (1)
um
x
...
wc;t
...
w1;tEq. (1)
Eq. (1)
Φt
Vt
Reminiscent of multiple model approach. In our approach, however,we propagate statistics.
Václav Šmídl, Anthony Quinn The Variational EM for Extended AR Models March 22, ICASSP 05 19 / 1