Component Analysis for SP Component Analysis Methods for ...
16
1 Component Analysis Methods for SP Component Analysis Methods for Signal Processing Fernando De la Torre ([email protected] ) Component Analysis Methods for SP Component Analysis for SP • Computer Vision & Image Processing – Structure from motion. – Spectral graph methods for segmentation. – Appearance and shape models. – Fundamental matrix estimation and calibration. – Compression. – Classification. – Dimensionality reduction and visualization. • Signal Processing – Spectral estimation, system identification (e.g. Kalman filter), sensor array processing (e.g. cocktail problem, eco cancellation), blind source separation, … • Computer Graphics – Compression (BRDF), synthesis,… • Speech, bioinformatics, combinatorial problems. Component Analysis Methods for SP Component Analysis for PR • Computer Vision & Image Processing – Structure from motion. – Spectral graph methods for segmentation. – Appearance and shape models. – Fundamental matrix estimation and calibration. – Compression. – Classification. – Dimensionality reduction and visualization. • Signal Processing – Spectral estimation, system identification (e.g. Kalman filter), sensor array processing (e.g. cocktail problem, eco cancellation), blind source separation, … • Computer Graphics – Compression (BRDF), synthesis,… • Speech, bioinformatics, combinatorial problems. Structure from motion Component Analysis Methods for SP Component Analysis for PR • Computer Vision & Image Processing – Structure from motion. – Spectral graph methods for segmentation. – Appearance and shape models. – Fundamental matrix estimation and calibration. – Compression. – Classification. – Dimensionality reduction and visualization. • Signal Processing – Spectral estimation, system identification (e.g. Kalman filter), sensor array processing (e.g. cocktail problem, eco cancellation), blind source separation, … • Computer Graphics – Compression (BRDF), synthesis,… • Speech, bioinformatics, combinatorial problems. Spectral graph methods for segmentation.
Transcript of Component Analysis for SP Component Analysis Methods for ...
1
Com
ponent Analysis
Meth
ods for SP
Co
mp
on
en
t A
naly
sis
Meth
od
s
for
Sig
nal P
rocessin
g
Fern
ando D
e la T
orre
(fto
rre@
cs.c
mu.e
du)
Com
ponent Analysis
Meth
ods for SP
Com
ponent Analy
sis
for SP
•C
om
pute
r Vis
ion &
Im
age P
rocessin
g–
Structu
re fro
m m
otion.
–Spectral gra
ph m
eth
ods for segm
enta
tion.
–Appeara
nce a
nd s
hape m
odels
.
–Fundam
enta
l m
atrix
estim
ation a
nd c
alib
ration.
–C
om
pre
ssio
n.
–C
lassific
ation.
–D
imensio
nalit
y reduction a
nd v
isualiz
ation.
•Sig
nal Pro
cessin
g–
Spectral estim
ation, syste
m identification (e.g
. Kalm
an filt
er), sensor
array p
rocessin
g (e.g
. cockta
il pro
ble
m, eco c
ancella
tion), b
lind s
ourc
e
separa
tion, …
•C
om
pute
r G
raphic
s–
Com
pre
ssio
n (BR
DF), s
ynth
esis
,…
•Speech, bio
info
rmatics, com
bin
ato
rial pro
ble
ms.
Com
ponent Analysis
Meth
ods for SP
Com
ponent Analy
sis
for PR
•
Com
pute
r Vis
ion &
Im
age P
rocessin
g–
Structu
re fro
m m
otion.
–Spectral gra
ph m
eth
ods for segm
enta
tion.
–Appeara
nce a
nd s
hape m
odels
.
–Fundam
enta
l m
atrix
estim
ation a
nd c
alib
ration.
–C
om
pre
ssio
n.
–C
lassific
ation.
–D
imensio
nalit
y reduction a
nd v
isualiz
ation.
•Sig
nal Pro
cessin
g–
Spectral estim
ation, syste
m identification (e.g
. Kalm
an filt
er), sensor
array p
rocessin
g (e.g
. cockta
il pro
ble
m, eco c
ancella
tion), b
lind s
ourc
e
separa
tion, …
•C
om
pute
r G
raphic
s–
Com
pre
ssio
n (BR
DF), s
ynth
esis
,…
•Speech, bio
info
rmatics, com
bin
ato
rial pro
ble
ms.
Structu
re fro
m m
otion
Com
ponent Analysis
Meth
ods for SP
Com
ponent Analy
sis
for PR
•
Com
pute
r Vis
ion &
Im
age P
rocessin
g–
Structu
re fro
m m
otion.
–Spectral gra
ph m
eth
ods for segm
enta
tion.
–Appeara
nce a
nd s
hape m
odels
.
–Fundam
enta
l m
atrix
estim
ation a
nd c
alib
ration.
–C
om
pre
ssio
n.
–C
lassific
ation.
–D
imensio
nalit
y reduction a
nd v
isualiz
ation.
•Sig
nal Pro
cessin
g–
Spectral estim
ation, syste
m identification (e.g
. Kalm
an filt
er), sensor
array p
rocessin
g (e.g
. cockta
il pro
ble
m, eco c
ancella
tion), b
lind s
ourc
e
separa
tion, …
•C
om
pute
r G
raphic
s–
Com
pre
ssio
n (BR
DF), s
ynth
esis
,…
•Speech, bio
info
rmatics, com
bin
ato
rial pro
ble
ms.
Spectral gra
ph m
eth
ods for segm
enta
tion.
2
Com
ponent Analysis
Meth
ods for SP
Com
ponent Analy
sis
for PR
•
Com
pute
r Vis
ion &
Im
age P
rocessin
g–
Structu
re fro
m m
otion.
–Spectral gra
ph m
eth
ods for segm
enta
tion.
–Appeara
nce a
nd s
hape m
odels
.
–Fundam
enta
l m
atrix
estim
ation a
nd c
alib
ration.
–C
om
pre
ssio
n.
–C
lassific
ation.
–D
imensio
nalit
y reduction a
nd v
isualiz
ation.
•Sig
nal Pro
cessin
g–
Spectral estim
ation, syste
m identification (e.g
. Kalm
an filt
er), sensor
array p
rocessin
g (e.g
. cockta
il pro
ble
m, eco c
ancella
tion), b
lind s
ourc
e
separa
tion, …
•C
om
pute
r G
raphic
s–
Com
pre
ssio
n (BR
DF), s
ynth
esis
,…
•Speech, bio
info
rmatics, com
bin
ato
rial pro
ble
ms.
Appeara
nce a
nd s
hape m
odels
Com
ponent Analysis
Meth
ods for SP
Com
ponent Analy
sis
for PR
•
Com
pute
r Vis
ion &
Im
age P
rocessin
g–
Structu
re fro
m m
otion.
–Spectral gra
ph m
eth
ods for segm
enta
tion.
–Appeara
nce a
nd s
hape m
odels
.
–Fundam
enta
l m
atrix
estim
ation a
nd c
alib
ration.
–C
om
pre
ssio
n.
–C
lassific
ation.
–D
imensio
nalit
y reduction a
nd v
isualiz
ation.
•Sig
nal Pro
cessin
g–
Spectral estim
ation, syste
m identification (e.g
. Kalm
an filt
er), sensor
array p
rocessin
g (e.g
. cockta
il pro
ble
m, eco c
ancella
tion), b
lind s
ourc
e
separa
tion, …
•C
om
pute
r G
raphic
s–
Com
pre
ssio
n (BR
DF), s
ynth
esis
,…
•Speech, bio
info
rmatics, com
bin
ato
rial pro
ble
ms.
cockta
il pro
ble
m
Com
ponent Analysis
Meth
ods for SP
Com
ponent Analy
sis
(IC
A)
Sound
Sourc
e 1
Sound
Sourc
e 2
Mix
ture
1
Mix
ture
2
Outp
ut 1
Outp
ut 2
I C A
Sound
Sourc
e 3
Mix
ture
3
Outp
ut 3
Com
ponent Analysis
Meth
ods for SP
Com
ponent Analy
sis
for PR
•
Com
pute
r Vis
ion &
Im
age P
rocessin
g–
Structu
re fro
m m
otion.
–Spectral gra
ph m
eth
ods for segm
enta
tion.
–Appeara
nce a
nd s
hape m
odels
.
–Fundam
enta
l m
atrix
estim
ation a
nd c
alib
ration.
–C
om
pre
ssio
n.
–C
lassific
ation.
–D
imensio
nalit
y reduction a
nd v
isualiz
ation.
•Sig
nal Pro
cessin
g–
Spectral estim
ation, syste
m identification (e.g
. Kalm
an filt
er), sensor
array p
rocessin
g (e.g
. cockta
il pro
ble
m, eco c
ancella
tion), b
lind s
ourc
e
separa
tion, …
•C
om
pute
r G
raphic
s–
Com
pre
ssio
n (BR
DF), s
ynth
esis
,…
•Speech, bio
info
rmatics, com
bin
ato
rial pro
ble
ms.
3
Com
ponent Analysis
Meth
ods for SP
Why C
om
ponent Analy
sis
for SP?
•Learn
fro
m h
igh d
imensio
nal data
and few
sam
ple
s.
–U
sefu
l fo
r dim
ensio
nalit
y reduction.
featu
res
sam
ple
s
•Effic
ient m
eth
ods O
( d
n< <n
2 )
(Everitt,1
984)
•Easy to incorp
ora
te
–R
obustn
ess to n
ois
e, m
issin
g d
ata
, outlie
rs (de la T
orre &
Bla
ck, 2003a)
–In
variance to g
eom
etric
tra
nsfo
rmations (de la T
orre &
Bla
ck, 2003b; de la
Torre &
Nguyen,2
007)
–N
on-lin
earities (Kern
el m
eth
ods)
(Scholkopf & S
mola
,2002; Shaw
e-T
aylo
r &
Cristianin
i,2004)
–Pro
babilistic (la
tent variable
models
)
–M
ulti-fa
cto
rial (tensors
) (P
aate
ro &
Tapper, 1
994 ;O
’Leary
& P
ele
g,1
983;
Vasile
scu &
Terz
opoulo
s,2
002; Vasile
scu &
Terz
opoulo
s,2
003)
–Exponential fa
mily
PC
A (G
ord
on,2
002; C
ollins e
t al. 0
1)
Com
ponent Analysis
Meth
ods for SP
Are
CA M
eth
ods P
opula
r/U
sefu
l/U
sed?
•Still
work
to d
o
–R
esults 1
-10
of about
65,3
00,0
00
for
"B
ritn
ey S
pears
".
•About 20%
of C
VPR
-06 p
apers
use C
A.
•G
oogle
:–
Results 1
-10
of about
1,8
70,0
00
for
"p
rin
cip
alco
mp
on
en
tan
aly
sis
".
–R
esults 1
-10
of about
506,0
00
for
"in
dep
en
den
tco
mp
on
en
tan
aly
sis
".
–R
esults 1
-10
of about
273,0
00
for
"lin
ear
dis
cri
min
an
tan
aly
sis
".
–R
esults 1
-10
of about
46,1
00
for
"n
eg
ati
ve
matr
ix
facto
rizati
on
".
–R
esults 1
-10
of about
491,0
00
for
"kern
elm
eth
od
s".
Com
ponent Analysis
Meth
ods for SP
Outlin
e•
Introduction
•G
enera
tive m
odels
–Princip
al C
om
ponent Analysis
(PC
A).
–N
on-n
egative M
atrix
Facto
rization (N
MF).
–In
dependent C
om
ponent Analysis
(IC
A).
–M
ultid
imensio
nal Scalin
g (M
DS).
•D
iscrim
inative m
odels
–Lin
ear D
iscrim
inantAnalysis
(LD
A).
–O
riente
d C
om
ponent Analysis
(O
CA).
–C
anonic
al C
orrela
tion A
nalysis
(C
CA).
•Sta
ndard
exte
nsio
ns o
f lin
ear m
odels
–Kern
el m
eth
ods.
–Late
nt variable
models
.
–Tensor fa
cto
rization
Com
ponent Analysis
Meth
ods for SP
Princip
al C
om
ponent Analy
sis
(PC
A)
•PC
Afinds
the
directions
of
maxim
um
variation
of
the
data
based
on
linearcorrela
tion.
•PC
Adecorrela
tes
the
origin
alvariable
s.
(Pears
on, 1901; H
ote
lling, 1933;M
ard
ia e
t al., 1979; Jolliffe, 1986; D
iam
anta
ras, 1996)
4
Com
ponent Analysis
Meth
ods for SP
Princip
al C
om
ponent Analy
sis
(PC
A)
kcc
c+
++
+≈
......
21
µ
kb
bb
21
[]
T nn
µ1
BC
dd
dD
+≈
=...
21
d=d= pixelspixels
nn= images
= images
1××
××
ℜ∈
ℜ∈
ℜ∈
ℜ∈
dn
k
kd
nd
µC
BD
•Assum
ing
0m
ean
data
,th
ebasis
Bth
atpre
serv
eth
em
axim
um
variation
ofth
esig
nalis
giv
en
by
the
eig
envecto
rsof
DD
T.
BΛ
BD
D=
Td
d
Com
ponent Analysis
Meth
ods for SP
Snap-s
hot M
eth
od &
SVD
•If d
>>n (e.g
. im
ages 1
00*1
00 v
s. 300 s
am
ple
s) no D
DT.
•D
DTand D
TD
have the s
am
e e
igenvalu
es (energ
y) and
rela
ted e
igenvecto
rs (by D
).
•B
is a
lin
ear com
bin
ation o
f th
e d
ata
!
•[α
,L]=
eig
(DTD
) B
=D
α(d
iag(d
iag(L
)))
-0.5
ΛD
αD
Dα
DD
DD
αB
BΛ
BD
DT
TT
T=
==
SVD
PC
A
diagonal
nn
nk
kd
T
ΣI
VV
IU
U
VΣ
U
VU
ΣD
TT
==
ℜ∈
ℜ∈
ℜ∈
=×
××
•SVD
facto
rizes the d
ata
matrix
Das:
Λ=
=
ℜ∈
ℜ∈
=×
×
TT
CC
IB
B
CB
BC
D
nk
kd
TT
UU
ΛD
D=
TT
VV
ΛD
D=
(Beltra
mi, 1
873; Schm
idt, 1
907; G
olu
b &
Loan, 1989)
(Sirovic
h, 1987)
Com
ponent Analysis
Meth
ods for SP
PC
A/S
VD
in C
om
pute
r Vis
ion
•PC
A/S
VD
has b
een a
pplie
d to:
–R
ecognitio
n (
eig
enfa
ces:T
urk
& P
entland, 1991; Sirovic
h &
Kirby,
1987; Leonard
is &
Bis
chof, 2
000; G
ong e
t al., 2000; M
cKenna e
t al., 1997a)
–Para
mete
rized m
otion m
odels
(Yacoob &
Bla
ck, 1999; Bla
ck e
t al., 2000; Bla
ck,
1999; Bla
ck &
Jepson, 1998)
–Appeara
nce/s
hape m
odels
(C
oote
s &
Taylor, 2
001; C
oote
s e
t al., 1998;Pentland
et al., 1994; Jones &
Poggio
, 1998; C
asia
& S
cla
roff, 1999; Bla
ck &
Jepson, 1998; Bla
nz &
Vetter, 1
999; C
oote
s e
t al., 1995; M
cKenna e
t al., 1997;de la T
orre e
t al., 1998b; de la
Torre e
t al., 1998b)
–D
ynam
ic a
ppeara
nce m
odels
(Soatto e
t al., 2001; R
ao, 1997; O
rrio
ls &
Bin
efa
, 2001; G
ong e
t al., 2000)
–Structu
re fro
m M
otion (
Tom
asi & K
anade, 1992; Bre
gle
r et al., 2000;Stu
rm &
Triggs, 1996; Bra
nd, 2001)
–Illu
min
ation b
ased reconstruction (
Haya
kaw
a, 1994)
–Vis
ual serv
oin
g (
Mura
se &
Naya
r, 1
995; M
ura
se &
Naya
r, 1
994)
–Vis
ual correspondence (
Zhang e
t al., 1995; Jones &
Malik
, 1992)
–C
am
era
motion e
stim
ation
(Hartle
y, 1
992; H
artle
y & Z
isserm
an, 2000)
–Im
age w
ate
rmark
ing (
Liu
& T
an, 2000)
–Sig
nal pro
cessin
g (
Moonen &
de M
oor, 1
995)
–N
eura
l appro
aches (
Oja
, 1982; Sanger, 1
989; Xu, 1993)
–Bilinear m
odels
(Tenenbaum
& F
reem
an, 2000; M
arim
ont & W
andell,
1992)
–D
irect exte
nsio
ns
(Welling e
t al., 2003; Penev &
Atick, 1996)
Com
ponent Analysis
Meth
ods for SP
Error Function for PC
A
(Eckard
t & Y
oung, 1936; G
abriel & Z
am
ir, 1979; Bald
i & H
orn
ik, 1989; Shum
et al.,
1995; de la T
orre &
Bla
ck, 2003a)
•N
ot uniq
ue s
olu
tion:
•To o
bta
in s
am
e P
CA s
olu
tion R
has to s
atisfy
:
•R
is c
om
pute
d a
s a
genera
lized k
×k e
igenvalu
e p
roble
m.
Λ=
=
==
−
TT
CC
IB
B
CR
CB
RB
ˆˆ
ˆˆ
ˆˆ
1
kk×
−ℜ
∈=
RB
CC
BR
R1
()
11
−−
Λ=
BR
BR
CC
TT
(de la T
orre, 2006)
F
n i
ii
EB
CD
Bc
dC
B,
−=
−=∑ =1
2 21
)(
•PC
A m
inim
izes the follo
win
g C
ON
VE
Xfu
nction.
5
Com
ponent Analysis
Meth
ods for SP
“In
terc
orr
ela
tio
ns a
mo
ng
vari
ab
les a
re t
he b
an
e o
f th
e
mu
ltiv
ari
ate
researc
her’
s s
tru
gg
le
for
mean
ing
”C
oo
ley a
nd
Lo
hn
es, 1971
Com
ponent Analysis
Meth
ods for SP
Part-b
ased R
epre
senta
tion
�The firin
g rate
s o
f neuro
ns a
re n
ever negative.
�In
dependent re
pre
senta
tions.
NM
F &
IC
A
Com
ponent Analysis
Meth
ods for SP
Non-n
egative M
atrix
Facto
rization
•Positiv
e facto
rization.
•Leads to p
art-b
ased repre
senta
tion.
0||
||)
(≥
−=
CB
,B
CD
CB
,F
E
Com
ponent Analysis
Meth
ods for SP
Nonnegative F
acto
rization
•M
ultip
licative a
lgorith
m c
an b
e inte
rpre
ted a
s
dia
gonally
rescale
d g
radie
nt descent.
∑−
=≥
≥ij
ijijd
F2
0,
0)
(min
BC
CB
ij
ij
ijij
)(
)(
BV
B
DB
CC
T
T
←Inference:
Learning:
ij
T
ij
T
ijij
)(
)( B
CC
DC
BB
←
Derivatives:
ijij
ij
F)
()
(C
BB
CB
C
TT
−=
∂∂
ij
T
ij
T
ij
F)
()
(D
CB
CC
B−
=∂∂
(Lee &
Seung, 1999;L
ee &
Seung, 2000)
6
Com
ponent Analysis
Meth
ods for SP
Independent C
om
ponent Analy
sis
•W
e n
eed m
ore
than s
econd o
rder sta
tistics to repre
sent
the s
ignal.
Com
ponent Analysis
Meth
ods for SP
ICA
•Look for s
i th
at are
independent.
•PC
A fin
ds u
ncorrela
ted v
ariable
s, th
e independent
com
ponents
have n
on G
aussia
n d
istrib
utions.
•U
ncorrela
ted E
(sis
j)= E
(si)E(s
j)
•In
dependent E
(g(s
i)f(s
j))=
E(g
(si))E
(f(s
j)) fo
r any n
on-
linear f,g
1−≈
=≈
=B
WW
DS
CB
CD
(Hyv
rinen e
t al., 2001)
PC
AIC
A
Com
ponent Analysis
Meth
ods for SP
ICA v
s P
CA
Com
ponent Analysis
Meth
ods for SP
Many o
ptim
ization c
rite
ria
•M
inim
ize h
igh o
rder m
om
ents
: e.g
. kurtosis
kurt(W
) = E
{s4} -3
(E{s
2}) 2
•M
any o
ther in
form
ation c
rite
ria.
∑∑
==
+−
n i
i
n i
ii
S1
1
)(c
Bc
dSpars
eness (e.g
. S=| |)
(Olh
ausen &
Fie
ld, 1996)
(Chennubhotla &
Jepson, 2001b;Zou e
t al., 2005; dAspre
mont et al., 2004;)
•Als
o a
n e
rror fu
nction:
•O
ther spars
e P
CA.
7
Com
ponent Analysis
Meth
ods for SP
Basis
of natu
ral im
ages
Com
ponent Analysis
Meth
ods for SP
Denois
ing
Origin
al
image
Nois
y Im
age
(30%
nois
e)
Denois
e
(Wie
ner filter)
ICA
Com
ponent Analysis
Meth
ods for SP
Multid
imensio
nal Scalin
g (M
DS)
•M
DS takes a
matrix
of pair-w
ise d
ista
nces a
nd
finds a
n e
mbeddin
g that pre
serv
es the
inte
rpoin
t dis
tances.
Com
ponent Analysis
Meth
ods for SP
MD
S(II)
mapping
Optim
ize w
.r.t
yi
8
Com
ponent Analysis
Meth
ods for SP
MD
S (III)
Com
ponent Analysis
Meth
ods for SP
Outlin
e•
Introduction
•G
enera
tive m
odels
–Princip
al C
om
ponent Analysis
(PC
A).
–N
on-n
egative M
atrix
Facto
rization (N
MF).
–In
dependent C
om
ponent Analysis
(IC
A).
–M
ultid
imensio
nal Scalin
g (M
DS)
•D
iscrim
inative m
odels
–Lin
ear D
iscrim
inantAnalysis
(LD
A).
–O
riente
d C
om
ponent Analysis
(O
CA).
–C
anonic
al C
orrela
tion A
nalysis
(C
CA).
•Sta
ndard
exte
nsio
ns o
f lin
ear m
odels
–Kern
el m
eth
ods.
–Late
nt variable
models
.
–Tensor fa
cto
rization
Com
ponent Analysis
Meth
ods for SP
Linear Discrim
inant Analysis (LDA)
•O
ptim
al lin
ear dim
ensio
nalit
y reduction if cla
sses a
re
Gaussia
n w
ith e
qual covariance m
atrix
.
BΛ
SB
SB
SB
BS
BB
bb
t
t
TT
J=
=|
|
||
)(
∑∑
==
−−
=C i
C j
T
ji
ji
b
11
))(
(µ
µµ
µS
(Fis
her, 1
938;M
ard
ia e
t al., 1979; Bis
hop, 1995)
∑ =
==
n i
T ii
T
t
1
dd
DD
S
T
ji
c j
C i
ji
w
i
)()
(1
1
µd
µd
S−
−=∑∑
==
Com
ponent Analysis
Meth
ods for SP
Oriente
d C
om
ponent Analy
sis
(O
CA)
•G
enera
lized e
igenvalu
e p
roble
m:
•b
ocais
ste
ere
d b
y the d
istrib
ution o
f nois
e.
OCA
T
OCA
T OCA
noise
OCA
signal b
Σb
bΣ
b
λk
ek
ib
Σb
Σ=
sig
nal
nois
e
OCA
b
9
Com
ponent Analysis
Meth
ods for SP
33
OC
A for Face R
ecognitio
n
k
T
k
T k
ek
i
bΣ
b
bΣ
b11
j
T j
j
T j
ei
bΣ
b
bΣ
b22
Com
ponent Analysis
Meth
ods for SP
Canonic
al C
orrela
tion A
naly
sis
•PC
A independently a
nd g
enera
l m
appin
g
•Sig
nals
dependent sig
nals
with s
mall
energ
y c
an b
e lost.
PC
AP
CA
Com
ponent Analysis
Meth
ods for SP
Canonic
al C
orrela
tion A
naly
sis
(C
CA)
•Learn
rela
tions b
etw
een m
ultip
le d
ata
sets
? (e.g
. find
featu
res in o
ne s
et re
late
d to a
noth
er data
set)
•G
iven tw
o s
ets
, C
CA fin
ds the p
air
of directions w
xand
wyth
at m
axim
ize the c
orrela
tion
betw
een the p
roje
ctions (assum
e z
ero
mean d
ata
)
•Severa
l w
ays o
f optim
izin
g it:
•An s
tationary
poin
t of r is
the s
olu
tion to C
CA.
T y
TT y
T x
TT x
y
TT x
Yw
Yw
Xw
Xw
Yw
Xw
=ρ
nd
nd
and
××
ℜ∈
ℜ∈
21
YX
=ℜ
∈
=ℜ
∈
=+
×+
+×
+
yxd
dd
d
T
Td
dd
d
T
T
www
YY
0
0X
XΒ
0Y
X
YX
0A
)(
)(
)(
)(
21
21
21
21
,
Βw
Aw
λ=
(Mard
ia e
t al., 1979; Borg
a)
Com
ponent Analysis
Meth
ods for SP
Dynam
ic C
anonic
al C
orrela
tion A
naly
sis
10
Com
ponent Analysis
Meth
ods for SP
Robot lo
caliz
ation w
ith C
anonic
al
Correla
tion A
naly
sis
(Skocaj & L
eonard
is, 2000)
Com
ponent Analysis
Meth
ods for SP
Com
ponent Analysis
Meth
ods for SP
Outlin
e•
Introduction
•G
enera
tive m
odels
–Princip
al C
om
ponent Analysis
(PC
A).
–M
ultid
imensio
nal Scalin
g (M
DS)
–N
on-n
egative M
atrix
Facto
rization (N
MF).
–In
dependent C
om
ponent Analysis
(IC
A).
•D
iscrim
inative m
odels
–Lin
ear D
iscrim
inantAnalysis
(LD
A).
–O
riente
d C
om
ponent Analysis
(O
CA).
–C
anonic
al C
orrela
tion A
nalysis
(C
CA).
•Sta
ndard
exte
nsio
ns o
f lin
ear m
odels
–Kern
el m
eth
ods.
–Late
nt variable
models
.
–Tensor fa
cto
rization
Com
ponent Analysis
Meth
ods for SP
Lin
ear m
eth
ods a
re n
ot enough
•W
hen d
ata
poin
ts s
it o
n a
non-lin
ear m
anifold
–W
e w
on’t fin
d a
good l
inearm
appin
g fro
m the d
ata
poin
ts to a
pla
ne, because there
isn’t a
ny
–In
the e
nd, lin
ear m
eth
ods d
o n
oth
ing m
ore
than
rota
te/tra
nsla
te/s
cale
data
11
Com
ponent Analysis
Meth
ods for SP
Kern
el M
eth
ods
•The
kern
eldefines
an
implic
itm
appin
g(u
sually
hig
hdim
ensio
naland
non-lin
ear)
from
inputto
featu
respace,so
the
data
becom
es
linearly
separa
ble
.
•C
om
puta
tion
inth
efe
atu
respace
can
be
costly
because
itis
(usually
)hig
hdim
ensio
nal
–The
featu
respace
isty
pic
ally
infinite-d
imensio
nal!
Feature space
Input space
),
,(
),
,(
),
(3
21
2 22 1
21
21
zz
zx
xx
xx
x=
+→
Com
ponent Analysis
Meth
ods for SP
Kern
el M
eth
ods
•Suppose φ
(.) is
giv
en a
s follo
ws
•An inner pro
duct in
the featu
re s
pace is
•So, if w
e d
efine the k
ern
el fu
nction a
s follo
ws, th
ere
is n
o
need to c
arry o
ut φ(
.) e
xplic
itly
•This
use o
f kern
el fu
nction to a
void
carryin
g o
ut φ(
.)
explic
itly is k
now
n a
s the k
ern
el tric
k. In
any lin
ear
alg
orith
m that can b
e e
xpre
ssed b
y inner pro
ducts
can b
e
made n
onlin
ear by g
oin
g to the featu
re s
pace
Com
ponent Analysis
Meth
ods for SP
Lin
ear m
eth
ods n
ot enough
•Learn
ing a
non-lin
ear re
pre
senta
tion for cla
ssific
ation
Com
ponent Analysis
Meth
ods for SP
Kern
el PC
A(S
cholkopf et al., 1998)
12
Com
ponent Analysis
Meth
ods for SP
Kern
el PC
A
•Eig
envecto
rs o
f th
e c
ov. M
atrix
in featu
re s
pace.
•Eig
envecto
rs lie
in the s
pan o
f data
in featu
re s
pace.
∑ =
ΦΦ
=n i
ii
n1
T )(
)(
1d
dC
λ1
1b
bC
=
∑ =
Φ=
n i
ii
1
1)
(db
α
λα
Kα=
(Scholkopf et al., 1998)
λα
α])
([
),
()
(1
11
1
∑∑
∑=
==
Φ=
Φn i
i
n i
ij
ii
n j
iK
nd
dd
d
Com
ponent Analysis
Meth
ods for SP
Late
nt Variable
Models
Com
ponent Analysis
Meth
ods for SP
Facto
r Analy
sis
•A G
aussia
n d
istrib
ution o
n the c
oeffic
ients
and n
ois
e is
added to P
CA�
Facto
r Analysis
.
•In
fere
nce
(Row
eis
& G
hahra
mani, 1
999;T
ippin
g &
Bis
hop, 1999a)
Ψ+
=−
−=
=Ψ
Ψ=
Ψ+
==
++
=
TT
d
k
E
diag
Np
Np
Np
BB
µd
µd
d0,
cη
Bc
µd
Bc
dI
0,
cc
ηB
cµ
d
))
)(((
)cov(
),...,
,(
)|
()
(
),
|(
),
|(
)|
()
(
21
ηη
η(Mard
ia e
t al., 1979)
11
1
)(
)(
)(
),
|(
)(
−−
−
Ψ+
=
−Ψ
+=
=
BB
IV
µd
BB
Bm
Vm
cd|
c
T
TT
Np
PC
A reconstruction low
error.
FA h
igh reconstruction e
rror (low
lik
elih
ood).
),
(d
cp
Join
tly
Gaussia
n
Com
ponent Analysis
Meth
ods for SP
Ppca
•If P
PC
A.
•If is
equiv
ale
nt to
PC
A.
TT
TT
BB
BB
BB
11
)(
)(
0−
−=
Ψ+
→ε
d
TE
Iη
ηε
==
Ψ)
( 0→
ε
•Pro
babilistic v
isual le
arn
ing (
Moghaddam
& P
entland, 1997;)
∑
=
Σ
=
Σ
==
−
−
=
−−
+−
−−
Σ−
−
∏∫
=
−−
2
)(
2
)(
1
21
221
212
)(
)(
)(
21
212
)(
)(
21
)2(
)2(
)2(
)2(
)(
)(
)(
2
1
2
11
kd
k i
i
d
c
dd
ee
ee
dp
pp
k iii
TT
πρλ
ππ
π
ρε
λε
dµ
dI
BB
µd
µd
µd
T
cc
c|d
d
i
T
id
Bc
=
13
Com
ponent Analysis
Meth
ods for SP
Tensor Facto
rization
Com
ponent Analysis
Meth
ods for SP
Tensor fa
ces
(Vasile
scu &
Terz
opoulo
s, 2002; Vasile
scu &
Terz
opoulo
s, 2003)
views
illuminations
expressions
people
Com
ponent Analysis
Meth
ods for SP
Eig
enfa
ces
•Facia
l im
ages (id
entity
change)
•Eig
enfa
ces b
ases v
ecto
rs c
aptu
re the v
ariability
in facia
l
appeara
nce (do n
ot decouple
pose, illum
ination, …
)
Com
ponent Analysis
Meth
ods for SP
Data
Org
aniz
ation
•Linear/PCA: D
ata
Mat
rix
–R
pix
els
x im
ages
–a matrix of imag
e vectors
•Multilinear: Data Tensor
–R
people
x v
iews x
illu
ms x
expre
ss x
pix
els
–N-dimen
sion
al matrix
–28
peo
ple, 45 imag
es/person
–5 view
s, 3 illuminations,
3 expression
s pe
r pe
rson
exil
vpp
,,
,i
Illuminations
Vie
ws
D
DPixels
Ima
ges
D
14
Com
ponent Analysis
Meth
ods for SP
N-M
ode S
VD
Alg
orith
m
N = 3
pixels
xexpress
xillum
s.x
view
s x
people
x5
1U
UU
UU
. ...
Z ZZZ
D DDD4
32
=
Com
ponent Analysis
Meth
ods for SP
PCA:
TensorFaces:
Com
ponent Analysis
Meth
ods for SP
Ten
sorF
aces
Mea
n Sq. Err. = 4
09.1
5
3 illum
+ 11 pe
ople param
.
33 basis vectors
PC
A
Mea
n Sq. Err. = 8
5.75
33 param
eters
33 basis vectors
Strate
gic
Data
Com
pre
ssio
n =
Perc
eptu
al Q
ualit
y
Ori
gin
al
176 ba
sis vectors
Ten
sorF
aces
6 illum
+ 11 pe
ople param
.
66 basis vectors
•TensorF
aces d
ata
reduction in illu
min
ation s
pace p
rim
arily
degra
des illu
min
ation e
ffects
(cast shadow
s, hig
hlig
hts
)
•PC
A h
as lower mean square error but higher perceptual error
Com
ponent Analysis
Meth
ods for SP
Th
an
ks
CA
15
Com
ponent Analysis
Meth
ods for SP
Bib
liogra
phy
Com
ponent Analysis
Meth
ods for SP
Com
ponent Analysis
Meth
ods for SP
Com
ponent Analysis
Meth
ods for SP
16
Com
ponent Analysis
Meth
ods for SP
Com
ponent Analysis
Meth
ods for SP
Com
ponent Analysis
Meth
ods for SP
Bib
liogra
phy
Zhou F
., D
e la T
orre F
. and H
odgin
s J
. (2
008)
"Ali
gn
ed
Clu
ste
r
An
aly
sis
fo
r T
em
po
ral
Se
gm
en
tati
on
of
Hu
ma
n M
oti
on
“ IEEE
Conference on Automatic Face and Gestures Recognition, September,
2008.
De la T
orre, F. and N
guye
n, M
. (2008)
“P
ara
me
teri
ze
d K
ern
el
Pri
nc
ipa
l C
om
po
ne
nt
An
aly
sis
: T
he
ory
an
d A
pp
lic
ati
on
s t
o
Su
pe
rvis
ed
an
d U
ns
up
erv
ise
d I
ma
ge
Ali
gn
me
nt“
IEEE Conference
on Computer Vision and Pattern Recognition, June, 2008.
