Figure-centric averages

Antonio Torralba & Aude Oliva (2002)Averages: Hundreds of images containing a person are averaged to reveal regularities

in the intensity patterns across all the images.

More by Jason Salavon

More at: http://www.salavon.com/

“100 Special Moments” by Jason Salavon

Why blurry?

Computing Means

Two Requirements:• Alignment of objects• Objects must span a subspace

Useful concepts:• Subpopulation means• Deviations from the mean

Images as Vectors

=

m

n

n*m

Vector Mean: Importance of Alignment

=

m

n

n*m

=

n*m

½ + ½ = mean image

How to align faces?

http://www2.imm.dtu.dk/~aam/datasets/datasets.html

Shape Vector

=

43Provides alignment!

Average Face

1. Warp to mean shape2. Average pixels

http://graphics.cs.cmu.edu/courses/15-463/2004_fall/www/handins/brh/final/

Objects must span a subspace

(1,0)

(0,1)

(.5,.5)

Example

Does not span a subspace

mean

Subpopulation meansExamples:

• Happy faces

• Young faces

• Asian faces

• Etc.

• Sunny days

• Rainy days

• Etc.

• Etc.

Average male

Average female

Deviations from the mean

--

==

Image X Mean X

X = X - X

Deviations from the mean

+=

+ 1.7=

X

X = X - X

Manipulating Facial Appearance through Shape and Color

Duncan A. Rowland and David I. Perrett

St Andrews University

IEEE CG&A, September 1995

Face ModelingCompute average faces

(color and shape)

Compute deviations between male and female (vector and color differences)

Changing gender

Deform shape and/or color of an input face in the direction of “more female” original shape

color both

Enhancing gender

more same original androgynous more opposite

Changing age

Face becomes “rounder” and “more textured” and “grayer”

original shape

color both

Back to the Subspace

Linear Subspace: convex combinations

m

iiiXaX

1

Any new image X can beobtained as weighted sum of stored “basis” images.

Our old friend, change of basis!What are the new coordinates of X?

The Morphable Face Model

The actual structure of a face is captured in the shape vector S = (x1, y1, x2, …, yn)T, containing the (x, y) coordinates of the n vertices of a face, and the appearance (texture) vector T = (R1, G1, B1, R2, …, Gn, Bn)T, containing the color values of the mean-warped face image.

Shape S

Appearance T

The Morphable face model

Again, assuming that we have m such vector pairs in full correspondence, we can form new shapes Smodel and new appearances Tmodel as:

If number of basis faces m is large enough to span the face subspace then:

Any new face can be represented as a pair of vectors

(1, 2m)T and (1, 2m)T !

m

iiimodel a

1

SS

m

iiimodel b

1

TT

Using 3D Geometry: Blinz & Vetter, 1999

show SIGGRAPH video

Computer Science

Erik Learned-Miller

Joint Alignment: What’s It Good For?

26Learned-Miller

Congealing (CVPR 2000, PAMI 2006)

27Learned-Miller

Five Applications

Image factorizations• For transfer learning, learning from one example

Alignment for Data Pooling• 3D MR registration• EEG registration

Artifact removal• Magnetic resonance bias removal

Improvements to recognition algorithms• Alignment before recognition

Defining anchor points for registration• Find highly repeatable regions for future registrations

28Learned-Miller

Congealing

Process of joint “alignment” of sets of arrays (samples of continuous fields).

3 ingredients• A set of arrays in some class• A parameterized family of continuous transformations• A criterion of joint alignment

29Learned-Miller

Congealing Binary Digits

3 ingredients• A set of arrays in some class:

• Binary images• A parameterized family of continuous transformations:

• Affine transforms• A criterion of joint alignment:

• Entropy minimization

30Learned-Miller

Criterion of Joint Alignment Minimize sum of pixel stack

entropies by transforming each image.

A pixel stack

31Learned-Miller

ObservedImage

“Latent Image”

Transform

(Previous work by Grenander,, Frey and Jojic.)

An Image Factorization

32Learned-Miller

QuickTime™ and aTIFF (LZW) decompressor

are needed to see this picture. A pixel stack

33Learned-Miller

The Independent Pixel Assumption Model assumes independent pixels A poor generative model:

• True image probabilities don’t match model probabilities.• Reason: heavy dependence of neighboring pixels.

However! This model is great for alignment and separation of causes!

• Why? • Relative probabilities of “better aligned” and “worse aligned” are

usually correct. Once components are separated, a more accurate (and

computationally expensive) model can be used to model each component.

34Learned-Miller

Before After

Each pair implicitly creates a sample of the transform T.

Congealing

A transform

35Learned-Miller

Character Models

LatentImages

Transforms

Image KernelDensity Estimator

(or other estimator)

Transform KernelDensity Estimator

(CVPR 2003)

Latent ImageProbability Density

for Zeroes

P(IL)

TransformProbability Density

for Zeroes

P(T)Con

geal

ing

36Learned-Miller

How do we line up a new image?

Sequence of successively “sharper” models

…

step 0 step 1 step N

…

Take one gradient step with respect to each model.

37Learned-Miller

Digit Models from One Example