Post Nonlinear Independent Subspace Analysis

Post Nonlinear Independent SubspaceAnalysis

Zoltán Szabó, Barnabás Póczos,Gábor Szirtes and András Lorincz

Neural Information Processing Group,Department of Information Systems,

Eötvös Loránd University,Budapest, Hungary

ICANN 2007

Z. Szabó, B. Póczos, G. Szirtes and A. Lorincz PNL ISA

Post Nonlinear Independent Subspace Analysis

Cocktail party problem:

independent groups of people / music bands,

nonlinear (post nonlinear) mixing.


PNL ISA Equations

The PNL ISA model:

x(t) = f[As(t)]. (1)

Assumptions (s = [s1; . . . ; sM ] ∈ RMd = R

D):source s is d-independent: I(s1, . . . , sM) = 0,s(t) ∈ R

D is i.i.d. in time t ,A ∈ R

D×D invertible and ‘mixing’, that is: A = [Aij ∈ Rd×d ],

∀ i ⇒ ∃(j , k) : Aij and Aik are invertible.f : R

D → RD invertible, acts component-wise.

Goal: s.


Ambiguites of PNL ISA

PNL mixing structure⇒ mirror demixing [s = Wg(x)].

Question: d-independence of s⇒ true s has been found?

⇓

Yes: PNL ISA separability theorem.


Separability; PNL-ISA Ambiguities withLocally-Constant Nonzero C2 Densities

Theorem

Supposing that:

A, W: invertible and ’mixing’ matrices,

s: (i) existing covariance matrix, (ii) somewhere locallyconstant, C2 density function,

h = g ◦ f is a component-wise bijection, with analyticalcoordinate functions.

In this case, if e := [e1; . . . ; eM ] = Wh(As) is d-independentwith somewhere locally constant density function, then:

e recovers the hidden source (up to ISA ambiguities +constant translation within subspaces).


Separability⇒ PNL ISA algorithm

Sketch:1 Estimate g = f−1:

d-dependent Central Limit Theorem

⇓

As is asymptotically Gaussian (D →∞)

⇓

g: ’gaussianization’ transformation

2 Estimate W: linear ISA to g(x).


Test databases (s)

I.i.d. tests:1 3D-geom (d = 3, M = 6),2 celebrities (d = 2, M = 10),3 letters (d = 2, M ≤ 50).

Non-i.i.d. test: IFS (self-similar structures; d = 2, M = 9).


Simulations

Performance index: Amari-index (r ∈ [0, 1]) to measure theblock-permutation matrix property of the linearapproximation of

s− E [s] ∈ RD 7→ s := Wg[f(As)]− E [s] ∈ R

D.

Components of the PNL ISA algorithm:gaussianization based on ranks of samples,ISA by joint f-decorrelation (JFD).

Simulation parameters:goodness: average of 50 random (A, s, f) runs,mixing matrix A: random orthogonal,coordinate-wise distortions: fi(z) = ci [aiz + tanh(biz)] + di .


Illustrations-1: r(T )

Amari-index as a function of

the sample number (T ): 3D-geom, celebrites, IFS.

dimensionality of the problem (↔ M): letters.

1 2 5 10 20 50 10010

−3

10−2

10−1

100

Number of samples (T)

Am

ari−

inde

x (r

)

3D−geom, celebrities, IFS

3D−geomcelebritiesIFS

x1031 2 5 10 20 50 100

10−3

10−2

10−1

100

Number of samples (T)

Am

ari−

inde

x (r

)

letters

M=2M=3M=4M=10M=20M=50

x103

Power-law decline: r(T ) ∝ T−c (c > 0), as Dր.


Illustration-1: demo (T = 100, 000)

−8 −6 −4 −2 0 2 4 6 8−6

−4

−2

0

2

4

6

8

Functions fi


Illustrations-2: D (T = 10, 000)

Estimation of D = dim(s) - in the background: Dx = 2D.

Gaussianization→ ordered eigenvalues of cov [g(x)].

Results: average over 50 random runs (A, f).

0 5 10 15 20 25 30 35 400

0.5

1

1.5

2

2.5

3Distribution of Eigenvalues: celebrities (3D−geom, IFS)

Eig

enva

lues

: Mea

n ±

Dev

iatio

n

Number of Eigenvalue2 3 4

0

0.5

1

1.5

2

2.5

3

Number of Components (M)

Eig

enva

lues

: Mea

n ±

Dev

iatio

n

Distribution of Eigenvalues: letters


Summary

PNL ISA problem

Separability of PNL ISA

⇓ (← d-dependent Central Limit Theorem)

PNL ISA = gaussianization + ISA

Simulations:Estimation error vs. sample number:

power-law decline, as Dր.Possibility to estimate the dimension of the hidden source.

The dimensions of the hidden sources: can also beestimated using the ISA Separation Theorem [Szabó et al.,JMLR 8 (2007), 1063-1095] . . .


Thank you for the attention!


Post Nonlinear Independent Subspace Analysis

Documents

Transcript of Post Nonlinear Independent Subspace Analysis