Searching and Browsing Video in Face Space

Lee Begeja Zhu Liu

Video and Multimedia Technologies Research

Page 2

Face Oriented Video Browsing

• Challenge - non linguistic browsing

• Browse a video using faces

• Anchorpersons in news broadcasts

• Main casts in movies

• Hosts and guests in talk shows

Face Finding

• Face Detection

• Find a face

• Face Recognition

• Find a specific face

• Face Clustering

• Find a set of similar faces

Face Clustering

What’s in a Face ?

Feature extraction - 50 Features

•9 – 3 color moments in Luv space

• Moments – mean, variance, skew

• Luv – L -luminance; u,v–chrominance

•24 – Gabor textures – 3 scales x 4 directions, mean and std dev

•17 – Edge detection histogram in 16 bins across the 2∏ polar coordinate space; with one bin for non-edge pixels

Face Clustering

•Torso region alone: The face dissimilarity is defined as the torso region distance, TD.

•Torso region and Icon region: The face dissimilarity is defined as weighted summation of torso and icon region distance, α∙TD + (1- α)∙ID, where α is the weighting factor.

•Torso region and Face region: The face dissimilarity is defined as the minimum of the torso region distance and face distance based on eigenface projection, min(TD, FD).

• Icon region alone: The face dissimilarity is defined as the icon region distance, ID.

Video Browsing Interface

Page 7

Performance Metrics

•Average Cluster Purity (ACP) – perfect ACP of 1.0 means each cluster only contains faces from one person.

•Average Face (Class) Purity (AFP) – perfect AFP of 1.0 would have all the faces of one person appearing in one cluster.

•Analogous to precision(ACP) vs. recall(AFP)

Results

Face dissimilarity Video 1 AFP ACP

Video 2 AFP ACP

Icon region alone 0.45 0.61

Torso region alone 0.47 0.82 0.57 0.92

Torso + Face regions (α=0.5)

0.51 0.83 0.70 0.97

Torso + Face regions (eigenface)

0.54 0.92 0.72 1

Future Work

•Working with Sumit Chopra to incorporate dimensionality reduction (DrLIM)

• Face Search/Clustering across programs

•Discussions with Patrick Haffner on using SVMs for Face Recognition

•Do specific face recognition (Obama, Leno)

•Search for multiple faces within a frame

• Improve Face Detection

•Include user generated video in our results

Additional Slides

ThatcherEffect

Gabor textures•3 scales x 4 directions

Directions

Scales

Eigenface

Eigenface approach is a PCA (Principal Component Analysis) method, in which a small set of characteristic pictures are used to describe the variation between face images.

Recognition is performed by projecting a new image onto the subspace spanned by the eigenfaces and then classifying the face by comparing its position in the face space with the positions of known individuals.

Informally, eigenfaces are a set of "standardized face ingredients", derived from analysis of many pictures of faces. Any human face can be considered to be a combination of these standard faces. For example, a face might be composed of the average face plus 20% from eigenface 1, 35% from eigenface 2, and -12% from eigenface 3.

Eigenfaces