Adaptive Offset Subspace Self-Organizing Map with an Application

to Handwritten Digit Recognition

Huicheng Zheng, Pádraig Cunningham and Alexey Tsymbal

August 20th, 2006

Thanks to the MUSCLE Internal Fellowship (http://www.muscle-noe.org).

MDM/KDD’06

2

Outline• Introduction• Learning self-organized subspaces

– Adaptive Subspace Self-Organizing Map (ASSOM)

• Learning self-organized linear manifolds– Adaptive Offset Subspace Self-Organizing Map

(AOSSOM)

• Experiments– Cluster separating– Handwritten digit recognition

• Conclusions

3

Fundamental Requirements in MDMAmong others:

• Dimension reduction

• Visualization

The Self-Organizing Map (SOM) meets these requirements and finds its use in MDM [J.-C. Lamirel et al., 2000] [R. Honda et al., 2000]

4

Learning Self-Organized Subspaces• Pattern invariance classes pattern subspaces

– For example: Translation-invariant learning

• Combining subspace learning and SOM [Kohonen. T., et al., 1997]: Adaptive Subspace Self-Organizing Map (ASSOM)

rectangles

circles

triangles

……

5

Applications of the ASSOM to MDM

• Invariant-feature learning[Kohonen, T., et al., 1997]

• Speech processing[Hase, H., et al., 1996]

• Texture segmentation[Ruiz del Solar, J., 1998]

• Image retrieval[De Ridder, D., et al., 2000]

• Image classification[Zhang, B., et al., 1999]

6

• The ASSOM does not learn linear manifolds adequately.

• Faces viewed from different perspectives, variants of handwritten digits can be handled by linear manifolds [Liu, Z. Q., 2001] [Hinton, G. E., et al., 1997].

Learning Linear Manifolds

7

Learning Self-Organized Linear Manifolds

• Principal Component Analysis Self-Organizing Map (PCASOM) [López-Rubio, E., et al., 2004]– Combining PCA with SOM– Computationally expensive

• Adaptive Manifold Self-Organizing Map (AMSOM) [Liu, Z. Q., 2001]– Attaching mean vectors to the ASSOM– No given objective function

• Adaptive Offset Subspace Self-Organizing Map (AOSSOM) [Proposed]– Attaching offset vectors to the ASSOM– Well defined objective function

8

Outline• Introduction• Learning self-organized subspaces

– Adaptive Subspace Self-Organizing Map

• Learning self-organized linear manifolds– Adaptive Offset Subspace Self-Organizing Map

• Experiments– Cluster separating– Handwritten digit recognition

• Conclusions

9

Adaptive Subspace Self-Organizing Map

The module arrays in ASSOM

Rectangular topology

Hexagonal topology

1b 2b Hb

Q

x

1Tbx

2ˆ )(L jx

2Tbx Hbx

T

A module representing the subspace L(j)

c

i

j

ci

j

10

Competition and Adaptation

• Repeatedly:– Competition: Winning module of

– Adaptation: For the winner and the modules i in its neighborhood

– Orthonormalize the basis vectors

Ss

jsc j

2)(ˆmaxarg )(L

x

)()(ˆ

)()()()(),(

)(

)(T)()(

,ss

tsthtts

i

ihi

cihc

xx

bx

L

)1()( tihb

)()( tihb

)(),()( )(,

)( stst ihc

ih xb

)(sx

Sss ),(x

11

Outline• Introduction• Learning self-organized subspaces

– Adaptive Subspace Self-Organizing Map

• Learning self-organized linear manifolds– Adaptive Offset Subspace Self-Organizing Map

• Experiments– Cluster separating– Handwritten digit recognition

• Conclusions

12

Adaptive Offset Subspace Self-Organizing Map (AOSSOM): Linear Manifold Learning

• defines a set of linear manifold {Ai}

• Learning of can be done by defining a criterion function:

– defines an MSE criterion.

– defines a spatial interaction between neighboring sites in the AOSSOM.

• Using stochastic gradient descent:

Ii Ss

ic dPshE

iXXx )()(~ 2)(

A

)()(1

)( ,...,, iH

ii bbr

)()(1

)( ,...,, iH

ii bbr

:, 'hh )()( ),()( )()()()('' tttt i

h

ii

h

ih brbb

Ss

si

2)(~

Ax

)(ich

13

Outline• Introduction• Learning self-organized subspaces

– Adaptive Subspace Self-Organizing Map

• Learning self-organized linear manifolds– Adaptive Offset Subspace Self-Organizing Map

• Experiments– Cluster separating– Handwritten digit recognition

• Conclusions

14

Cluster Separating by Learning Linear Manifolds (1)

Nearly optimal parameters were used in all the experiments.

15

Cluster Separating by Learning Linear Manifolds (2)

Instability was observed for the AMSOM when initial learning rate . A larger value has been used to alleviate the problem.

10 5.10

16

• Four random clusters with 2-D Gaussian distributions.• “AMSOM 1” uses the neighborhood-decreasing scheme

proposed in [Liu, Z. Q., 2001]: A Gaussian function with the FWHM

• “AMSOM 2” uses the same neighborhood function as the AOSSOM: A Gaussian function with the FWHM

Cluster Separating by Learning Linear Manifolds (3)

)exp()( 0 F

twtw

tT

Twtw

99)( 0

17

Cluster Separating by Learning Linear Manifolds (3)

Boldface texts emphasize the cases where the AOSSOM worked the best (8 out of 10 times).

Observation of projection error decreasing for D=1.

18

Outline• Introduction• Learning self-organized subspaces

– Adaptive Subspace Self-Organizing Map

• Learning self-organized linear manifolds– Adaptive Offset Subspace Self-Organizing Map

• Experiments– Cluster separating– Handwritten digit recognition

• Conclusions

19

Application of the AOSSOM to Handwritten Digit Recognition

• Large variety of handwriting styles

• It is hard for hand-crafted feature extractor to cope with these variations.

• Variations create embedded manifolds in the feature space.

• Piece-wise linear approximation of embedded manifolds.

20

Handwritten Digit Image Database

• MNIST database composed of 60,000 training digits and 10,000 test digits [LeCun, Y., et al., 1998].

• Some examples:

21

MNIST Learning with the AOSSOM

• One AOSSOM trained for each digit class

• For an input test digit image x, each AOSSOM Mk reconstructs an output .

kMx̂

22

MNIST Learning Performance Summary

H: Linear manifold dimension

W: No. of rows and columns of AOSSOM

23

Outline• Introduction• Learning self-organized subspaces

– Adaptive Subspace Self-Organizing Map

• Learning self-organized linear manifolds– Adaptive Offset Subspace Self-Organizing Map

• Experiments– Cluster separating– Handwritten digit recognition

• Conclusions

24

Conclusions

• The AOSSOM permits to learn organized linear manifolds with a well-defined criterion function.

• Higher robustness is observed in the experiments with the AOSSOM than with the AMSOM.

• The AOSSOM is well suited for MDM tasks where data structures are best handled by piece-wise linear manifolds.

25

References• De Ridder, D., et al., 2000: The adaptive subspace map for image

description and image database retrieval. SSPR&SPR.

• Hase, H., et al., 1996: Speech signal processing using Adaptive Subspace SOM (ASSOM). Technical Report NC95-140, The Inst. of Electronics, Information and Communication Engineers, Tottori University, Koyama, Japan.

• Hinton, G. E., et al., 1997: Modeling the manifolds of images of handwritten digits. IEEE Trans. Neural Networks, 8(1):65–74.

• Honda, R., et al., 2000: Semantic indexing and temporal rule discovery for time-series satellite images. In S. J. Simoff and O. R. Zaïane, editors, Proc. First Int. Workshop on Multimedia Data Mining, in conjunction with Sixth ACM SIGKDD Int. Conf. on Knowledge Discovery and Data Mining, pp. 82–90, Boston, MA, USA.

• Kohonen, T., et al., 1997: Self-Organized formation of various invariant-feature filters in the adaptive-subspace SOM. Neural Computation 9(6).

26

References• Lamirel, J.-C., et al., 2000: A Self Organizing Map (SOM) extended

model for information discovery in a digital library context. In S. J. Simoff and O. R. Zaïane, editors, Proc. First Int. Workshop on Multimedia Data Mining, in conjunction with Sixth ACM SIGKDD Int. Conf. on Knowledge Discovery and Data Mining, pp. 60–66, Boston, MA, USA.

• LeCun, Y., et al., 1998: Gradient-based learning applied to document recognition. Proc. IEEE, 86(11):2278–2324.

• Liu, Z. Q., 2001: Retrieving faces using Adaptive Subspace Self-Organising Map. In Proc. International Symposium on Intelligent Multimedia, Video and Speech Processing, pp. 377–380, Hong Kong.

• López-Rubio, E., et al., 2004: A Principal Components Analysis Self-Organizing Map. Neural Networks, 17(2), pp. 261–270.

• Ruiz del Solar, J., 1998: Texsom: texture segmentation using Self-Organizing Maps. Neurocomputing 21(1–3).

• Zhang, B., et al., 1999: Handwritten digit recognition by adaptive-subspace self-organizing map (ASSOM). IEEE Trans. on Neural Networks 10(4).

27

Thanks and questions?