Research Activities at Computer Vision and Image Understanding Group

Florida State University

Xiuwen Liu

Florida State Vision GroupDepartment of Computer Science

Florida State University

http://fsvision.cs.fsu.edu

Outline

Motivations• Some applications of computer vision techniques

Computer Vision and Image Understanding Group

Some of the research projects

Contact information

Introduction

An image patch represented by hexadecimals

Introduction - continued

Fundamental problem in computer vision• Given a matrix of numbers representing an image, or a

sequence of images, how to generate a perceptually meaningful description of the matrix?

– An image can be a color image, gray level image, or other format such as remote sensing images

– A two-dimensional matrix represents a signal image– A three-dimensional matrix represents a sequence of images

A video sequence is a 3-D matrix A movie is also a 3-D matrix

Introduction - continued

Introduction - continued

Why do we want to work on this problem?• It is very interesting theoretically

– It involves many disciplines to develop a computational model for the problem

• It is the key component to understand and model intelligence

– Note that 50% of the brain is devoted to vision• It has many practical applications

– Internet applications– Movie-making applications– Military applications

Computer Vision Applications

No hands across America• sponsored by Delco Electronics, AssistWare Technology,

and Carnegie Mellon University• Navlab 5 drove from Pittsburgh, PA to San Diego, CA,

using the RALPH computer program. • The trip was 2849 miles of which 2797 miles were driven

automatically with no hands – Which is 98.2%

Computer Vision Applications – continued

Computer Vision Applications – continued

DARPA Grant Challenge: http://www.darpa.mil/grandchallenge/index.htm

Computer Vision Applications – continued

Military applications• Automated target recognition

Computer Vision Applications – continued

Computer Vision Applications – continued

Extracted hydrographic regions

Computer Vision Applications – continued

Medical image analysis• Characterize different types of tissues in medical images

for automated medical image analysis

Computer Vision Applications – continued

Computer Vision Applications – continued

Biometrics• From faces, fingerprints, iris patterns .....• It has many applications such as security, ATM

withdrawal, credit card managements .....

Computer Vision Applications – cont.

Computer Vision Applications – continued

Content-based image retrieval has become an active research area to meet the needs of searching images on the web in a meaningful way• Color histogram has been widely used

Content-Based Image Retrieval – cont.

Vision-Based Image Morphing

Vision-Based Image Morphing - continued

Computer Vision and Image Understanding Group

Faculty: Xiuwen Liu, Anuj Srivastava, Washington Mio, Eric Klassen

Goals: Develop and implement effective image understanding algorithms and systems for images and videos from multi modalities including visible, infrared, and range sensors

Approaches: Learning-based vision algorithms, statistical modeling of objects, computational modeling

and analysis of textures, statistical modeling of shapes, stochastic optimization, inference algorithms on manifolds, and Bayesian inference

Research Projects

The group offers a wide range of research possibilities• Implementation projects• Development of new applications• Development of new algorithms• Theoretical and mathematical analysis of algorithms

Implementation Projects

These projects involve implementing proven ideas and algorithms on specific datasets with specific interface and programming language constraints• For example, Haitao Wu implemented a

graphical user interface for a face recognition algorithm we have as his Masters project

• Yu Wang implemented a web-based interface for a content-based image retrieval algorithm

A Real-time Recognition/Tracking System

Content-based Image Retrieval

Image Query System by Yu Wang

Future Implementation Possibilities

Implement a Java-based system for face detection

Implement a Java-based system for learning Implement and improve web-based systems for

content-based image and video retrieval

Generic Image Modeling

How can we characterize all these images perceptually?

Spectral Histogram Representation

Spectral histogram

• Given a bank of filters F(), = 1, …, K, a spectral histogram is defined as the marginal distribution of filter responses

)I(*)(I )()( vFv

v

vIzδzH ))((|I|

1)( )()(I

),,,( )(I

)2(I

)1(II

KHHHH

Spectral Histogram Representation - continued

Choice of filters • Laplacian of Gaussian filters• Gabor filters• Gradient filters• Intensity filter

LoG filter Gabor filter

Spectral Histogram Representation - continued

A Texture Synthesis Example

A white noise image was transformed to a perceptually similar texture by matching the spectral histogram

Average spectral histogram error

Texture Synthesis Examples - continued

A random texture image

Observed image Synthesized image

Texture Synthesis Examples - continued

An image with periodic structures

Observed image Synthesized image

Texture Synthesis Examples - continued

A mud image with some animal foot prints

Mud image

Synthesized image

Texture Synthesis Examples - continued

A random texture image with elements

Observed image

Synthesized image

Object Synthesis Examples As in texture synthesis, we start from a random image In addition, similar object images are used as boundary conditions in that

the corresponding pixel values are not updated during sampling process

Object Synthesis Examples - continued

Object Synthesis Examples - continued

Principal Component Analysis

Eigen Values of 400 Eigen Vectors

Principal Component Analysis - continued

Original Image Reconstructed using 50 PCs

Reconstructed using 200 PCs

Principal Component Analysis - continued

Principal Component Analysis - continued

Difference Between Reconstruction and Sampling

Reconstruction is not sufficient to show the adequacy of a representation and sampling from the set of images with same representation is more informational

Face detection based on spectral representations

Face detection is to detect all instances of faces in a given image

Each image window is represented by its spectral histogram• A support vector machine is trained on training faces• Then the trained support vector machine is used to classify

each image window in an input image More results at http://fsvision.fsu.edu/face-detection

Face detection - continued

Face detection - continued

Face detection - continued

Rotation invariant face detection

Rotation invariant face detection - continued

Linear representations

Linear representations are widely used in appearance-based object recognition applications• Simple to implement and analyze• Efficient to compute• Effective for many applications

dT RIUUI ),(

Standard Linear Representations

Principal Component Analysis• Designed to minimize the reconstruction error on the training set• Obtained by calculating eigenvectors of the co-variance matrix

Fisher Discriminant Analysis• Designed to maximize the separation between means of each class• Obtained by solving a generalized eigen problem

Independent Component Analysis• Designed to maximize the statistical independence among coefficients

along different directions• Obtained by solving an optimization problem with some object function

such as mutual information, negentropy, ....

Standard Linear Representations - continued

Standard linear representations are sub optimal for recognition applications• Evidence in the literature [1][2]• A toy example

– Standard representations give the worst recognition performance

Proposed Approach

Optimal Component Analysis (OCA)• Derive a performance function that is related to the

recognition performance• Formulate the problem of finding optimal representations

as an optimization one on the Grassmann manifold• Use MCMC stochastic gradient algorithm for

optimization

Performance Measure

It must have continuous directional derivatives It must be related to the recognition performance It can be computed efficiently Based on the nearest neighbor classifier

• However, it can be applied to other classifiers as it forms clusters of images from the same class that far from clusters from other classes

• See an example for support vector machines

Performance Measure - continued

Suppose there are C classes to be recognized• Each class has ktrain training images • It has kcross cross validation images

Performance Measure - continued

h is a monotonically increasing and bounded function• We used h(x) = 1/(1+exp(-2x)• Note that when , F(U) is exactly the recognition performance using

the nearest neighbor classifier Some examples of F(U) along some directions

Performance Measure - continued

F(U) depends on the span of U but is invariant to change of basis• In other words, F(U)=F(UO) for any orthonormal matrix

O• The search space of F(U) is the set of all the subspaces,

which is known as the Grassmann manifold– It is not a flat vector space and gradient flow must take the

underlying geometry of the manifold into account; see [3] [4] [5] for related work

Deterministic Gradient Flow - continued

Gradient at [J] (first d columns of n x n identity matrix)

Deterministic Gradient Flow - continued

Gradient at U: Compute Q such that QU=J

Deterministic gradient flow on Grassmann manifold

Stochastic Gradient and Updating Rules

Stochastic gradient is obtained by adding a stochastic component

Discrete updating rules

MCMC Simulated Annealing Optimization Algorithm

Let X(0) be any initial condition and t=01. Calculate the gradient matrix A(Xt)

2. Generate d(n-d) independent realizations of wij’s

3. Compute Y (Xt+1) according to the updating rules

4. Compute F(Y) and F(Xt) and set dF=F(Y)- F(Xt)

5. Set Xt+1 = Y with probability min{exp(dF/Dt),1}

6. Set Dt+1 = Dt / and set t=t+1

7. Go to step 1

The Toy Example

The following result on the toy example shows the effectiveness of the algorithm• The following figure shows the recognition performance of Xt and

F(Xt)

ORL Face Dataset

Experimental Results on ORL Dataset

Here the size of image is 92 x 112, d = 5 (subspace)• Comparison using gradient, stochastic gradient, and the proposed

technique with different initial conditions

PCA ICA FDA

Results on ORL Dataset - continued

With respect to d and ktrain

d=3ktrain=5

d=10ktrain=5

d=20ktrain=5

d=5ktrain=1

d=5ktrain=2

d=5ktrain=8

Results on CMU PIE Dataset

Here we used part of the CMU PIE dataset• There are 66 subjects• Each subject has 21 pictures under different lighting conditions

-X0=PCA-d=10

-X0=ICA-d=10

-X0=FDA-d=5

Some Comparative Results on ORL Comparison where performance on cross validation images is

maximized• In other words, the comparison is to show the best performance linear

representations can achieve• PCA – black dotted; ICA – red dash-dotted; FDA – green dashed; OCA – blue solid

Some Comparative Results on ORL - continued

Comparison where the performance on the training is optimized• In other words, it is a fair comparison• PCA – black dotted; ICA – red dash-dotted; FDA – green dashed; OCA – blue solid

PROBABILITY MODELS FOR IMAGE ANALYSIS

Empirical Studies Indicate PatternsEmpirical Studies Indicate Patterns

Need models that:Need models that:

• are low-dimensional (computationally tractable)are low-dimensional (computationally tractable)

• are accurate models of (real) observed clutterare accurate models of (real) observed clutter

• support the observed patternssupport the observed patterns

Histogram of x-derivativeHistogram of x-derivative

BESSEL K FORM

A Parametric Family:A Parametric Family:

•Image statistics (under spectral decompositions) exhibit non Image statistics (under spectral decompositions) exhibit non Gaussian statistics.Gaussian statistics.

•This density explains the non-Gaussian and heavy-tail This density explains the non-Gaussian and heavy-tail nature of observed image statistics.nature of observed image statistics.

•The parameters p and c are easily estimated from the data The parameters p and c are easily estimated from the data using sample variance and kurtosis.using sample variance and kurtosis.

•This model is derived from first principles.This model is derived from first principles.

K is the modified Bessel function of third kindK is the modified Bessel function of third kind

MODELING SUCCESS

ObservedObserved Bessel KBessel K

Original ImageOriginal Image Gabor FilterGabor Filter

Filtered ImageFiltered Image

Statistics of Filtered ImageStatistics of Filtered Image

SHAPE ANALYSIS

•Represent shapes as elements of infinite-dimensional manifoldsRepresent shapes as elements of infinite-dimensional manifolds

•Analyze shapes using geometry of that manifoldAnalyze shapes using geometry of that manifold

-- connect shapes using -- connect shapes using geodesic pathsgeodesic paths on the manifold on the manifold

-- quantify shape differences using -- quantify shape differences using geodesic lengthsgeodesic lengths

-- compute shape statistics (-- compute shape statistics (meanmean, variance), variance)

•Applications:Applications:

-- clustering of objects according to shapes (-- clustering of objects according to shapes (learninglearning))

-- shape based recognition of objects (-- shape based recognition of objects (recognitionrecognition))

-- predicting shapes of partially-obscured objects (-- predicting shapes of partially-obscured objects (completioncompletion))

Basic Idea:Basic Idea: Given two shapes (far left and far right), we Given two shapes (far left and far right), we connect them using a geodesic path on the connect them using a geodesic path on the shape manifold. shape manifold.

Second ShapeSecond ShapeFirst ShapeFirst Shape

Eight shapes along geodesic pathEight shapes along geodesic path

GEODESIC PATHS ON SHAPESGEODESIC PATHS ON SHAPES

ExampleExample

Fish shapes taken from Surrey databaseFish shapes taken from Surrey database

MEAN SHAPES

Four Sample ShapesFour Sample Shapes

Their Mean ShapeTheir Mean Shape

CLUSTERING OF SHAPES

Results:Results: 7 resulting clusters, each row is a cluster 7 resulting clusters, each row is a cluster

3D Model-Based Recognition

Medical Image Analysis

Advances in medical imaging provide many new opportunities and challenges for computer vision research

Automated medical image analysis

Medical Image Analysis - continued

Medical Image Analysis - continued

Medical Image Analysis - continued

Medical Image Analysis - continued

Video Sequence Analysis and Summary Motion analysis based on correspondence Video stream-based surveillance Video summary

Courses

Most Relevant Courses • CAP 5638 Pattern Recognition (Spring 2004)• CAP 5415 Principles and Algorithms of Computer Vision

– Fall 2004• CAP 6417 Theoretical Foundations of Computer Vision

– STA 5106 Computational Methods in Statistics I – STA 5107 Computational Methods in Statistics I I– Seminars and advanced studies

Related Courses• CAP 5615 Artificial Neural Networks• CAP 5600 Artificial Intelligence• CAP 5xxx Machine Learning

CAP 5638 Pattern Recognition

It will be offered Spring 2004• Tuesday and Thursday 6:45-8:00 PM• At Love 103• The course ref #: 07842 • http://www.cs.fsu.edu/~liux/courses/cap5638/

It will cover• The basics for pattern recognition

– Neural networks• Machine learning algorithms• Applications in data mining, pattern discovery, artificial intelligence,

and security, It should be interesting to anyone interested in more

intelligent computer learning algorithms

Funding of the Group

National Science Foundation• DMS • CISE IIS• FRG• ACT

National Imagery and Mapping Agency• NGA – National Geo-spatial Intelligence Agency

Army Research Office

Summary

Florida State Vision group offers many interesting research topics/projects• Efficient represent for generic images• Computational models for object recognition and image

classification• Medical image analysis• Motion/video sequence analysis and modeling

• They are challenging• They are interesting

Contact Information

• Name Xiuwen Liu• Web site at http://fsvision.fsu.edu http://www.cs.fsu.edu/~liux• Email at liux@cs.fsu.edu• Office at LOV 166• Phone 644-0050