Enhancing Tensor Subspace

Learning by Element Rearrangement

1

Dong XU

School of Computer Engineering

Nanyang Technological Universityhttp://www.ntu.edu.sg/home/dongxu

dongxu@ntu.edu.sg

Outline

• Summary of our recent works on Tensor (or Bilinear) Subspace Learning

• Element Rearrangement for Tensor (or Bilinear) Subspace Learning

What is Tensor?

Tensors are arrays of numbers which transform in certain ways under coordinate transformations.

1m

2m3m

1m

Vector Matrix 3rd-order Tensor

2m

1m

1 2 3m m m X R1 2m mX R1mxR

Bilinear (or 2D) Subspace Learning: each image is represented as a 2nd-order tensor (i.e., a matrix)

Tensor Subspace Learning (more general case): each image is represented as a higher order tensor

2

1

m

ij ik kjk

Y X U

100

100

100

100

100

100

.

.

.

100

10

100

10

100

10

100

10

100

10

100

10

=

=

=

.

.

.

Definition of Mode-k Product

(100)

2 )'(10m2 (100)m

2m(100)

1m(100)

1m

2 )'(10m

1m

2 (100)m

1(100)m

3 (40)m

2 )'(10m

2 (100)m

k U Y XNotation:

Product for two Matrices

Original Matrix

New Matrix

= 1(100)m

3 (40)m

2 )'(10m

Y XUProjection

Matrix

Original Tensor

New Tensor

Projection Matrix

Projection: high-dimensional space -> low-dimensional space

Reconstruction: low-dimensional space -> high-dimensional space

Definition of Mode-k Flattening

...1(100)m

2 3 (100*40)m m

Tensor Matrix

1m

2 (100)m

1(100)m

3 (40)m

Potential Assumption in Previous Tensor-based Subspace Learning:

Intra-tensor correlations: Correlations along column vectors of mode-k flattened matrices.

Data Representation in Dimensionality Reduction

Vector Matrix 3rd-order Tensor

High Dimension

Low Dimension

Examples

PCA, LDA Rank-1 Decomposition, 2001

A. Shashua and A. Levin,Tensorface, 2002

M. Vasilescu andD. Terzopoulos,

Our WorkXu et al., 2005

Yan et al., 2005…

...

...

...

Filtered Image Video SequenceGray-level Image

What is Gabor Features?Gabor features can improve recognition performance in comparison to grayscale features. Chengjun Liu T-IP, 2002

Gabor Wavelet Kernels

Eight Orientations

Five

Scales

Input: Grayscale

Image Output: 40 Gabor-filtered

Images

…

Why Represent Image Objects as Tensors instead of Vectors?

Natural Representation Gray-level Images (2D structure) Videos (3D structure) Gabor-filtered Images (3D structure)

Enhance Learnability in Real Application Curse of Dimensionality (Gabor-filtered image: 100*100*40 -> Vector: 400,000)

Small sample size problem (less than 5,000 images in common face databases)

Reduce Computation Cost

... ...

Concurrent Subspace Analysis (CSA) as an Example(Criterion: Optimal Reconstruction)

1m

100

40

100

1U

2U

The reconstructed sample

Input sample

Projection Matrices?

Sample in Low- dimensional space

3U

1m

10

10

10

Dimensionality Reduction

1m

100

40

100

Reconstruction

D. Xu, S. Yan, Lei Zhang, H. Zhang et al., CVPR 2005 and T-CSVT 2008 3

1

* 31

21 1 1 3 3 3

|

( | )

arg min || ... ||k k

k k

i iiU

U

U U U U

X X

Objective Function:

Tensorization - New Research Direction: Other Related Works

• Discriminant Analysis with Tensor Representation (DATER): CVPR 2005 and T-IP 2007

• Coupled Kernel-based Subspace Learning (CKDA): CVPR 2005 • Rank-one Projections with Adaptive Margins (RPAM): CVPR 2006 and T-SMC-B

2007• Enhancing Tensor Subspace Learning by Element Rearrangement: CVPR 2007

and T-PAMI 2009• Discriminant Locally Linear Embedding with High Order Tensor Data (DLLE/T):

T-SMC-B 2008• Convergent 2D Subspace Learning with Null Space Analysis (NS2DLDA): T-

CSVT 2008• Semi-supervised Bilinear Subspace Learning: T-IP 2009• Applications in Human Gait Recognition

– CSA+DATER: T-CSVT 2006– Tensor Marginal Fisher Analysis (TMFA): T-IP 2007

Note: Other researchers also published several papers along this direction!!!

Tensorization - New Research Direction Tensorization in Graph Embedding Framework

Direct Graph Embedding

1minT

T

y B yy Ly

Original PCA & LDA,ISOMAP, LLE,

Laplacian Eigenmap

Linearization

PCA, LDA, LPP, MFA

wXy T

Kernelization

KPCA, KDA, KMFA

)( iii xw

Tensorization

CSA, DATER, TMFA

nnii wwwy 2

21

1X

Type

Formulation

Example

S. Yan, D. Xu, H. Zhang et al., CVPR, 2005 and T-PAMI,2007

Google Citation: 174 (until 15-Sep-2009)

Element Rearrangement: Motivations

• The success of tensor-based subspace learning relies on the redundancy among the unfolded vector

• However, such correlation/redundancy is usually not strong for real data

• Our Solution: Element rearrangement is employed as a preprocessing step to increase the intra-tensor correlations for existing tensor subspace learning methods

12

Motivations-Continued

Sets of highlycorrelated pixels

Columns of highlycorrelated pixels

Element Rearrangement

Low correlation High correlation

Intra-tensor correlations: Correlations among the features within certain tensor dimensions, such as rows, columns and Gabor features…

Problem Definition

• The task of enhancing correlation/redundancy among 2nd–order tensor is to search for a pixel rearrangement operator R, such that

14

* 2

,1

arg min{ min || || }N

R T R Ti iR U V

i

R X UU X VV

1. is the rearranged matrix from sample2. The column numbers of U and V are predefined

iXRiX

After the pixel rearrangement, we can use the rearranged tensors as input for tensor subspace learning algorithms!

Solution to Pixel Rearrangement Problem

15

Compute reconstructed matrices

1, 1 1 1 1

nRRec T Ti n n n i n nX U U X V V

Optimize operator R

2,

1

arg min || ||N

R Recn i i nR

i

R X X

Optimize U and V

n=n+1

Initialize U0, V0

2

,1

( , ) arg min || ||n n

NR RT T

n n i iU Vi

U V X UU X VV

2 2, 1 1 1 1

1 1

: || || || ||n n n

N NR R RRec T Ti i n i n n i n n

i i

Note X X X U U X V V

• It is Integer Programming problem

Step for Optimizing R

16

* 2,

1

arg min || ||N

R Reci i nR

i

R X X

,

min .

1: 0 1; 2 : 1; 3 : 1

pq pqR

p q

pq pq pqp q

c R st

R R R

2,

1

| ( ) ( ) |N

Recpq i i n

i

where c X p X q

1. Linear programming problem in Earth Mover’s Distance (EMD) has integer solution.

2. We constrain the rearrangement within spatially local neighborhood or feature-based neighborhood for speedup.

p

Original matrix

Reconstructed matrix

q

pqc

Sender

Receiver

Convergence Speed

17

Rearrangement Results

18

Reconstruction Visualization

19

Reconstruction Visualization

20

Classification Accuracy

Summary

.

• Our papers published in CVPR 2005 are the first works to address dimensionality reduction with the image objects represented as high-order tensors of arbitrary order.

• Our papers published in CVPR 2005 opens a new research direction. We also published a series of works along this direction.

• Element arrangement can further improve data compression performance and classification accuracy.