MITRE Corporation

Pose Correction for Automatic Facial Recognition

Team: Elliot Godzich, Dylan Marriner, Emily Myers-Stanhope, Emma Taborsky (PM), Heather Williams

Liaisons: Josh Klontz ’10 and Mark BurgeAdvisor: Zachary Dodds

• Fraud detection• Aid distribution• Law enforcement• National security 

• Algorithmic identification of faces from images• Commercial systems exist; MITRE is building a

U.S. system for flexibility and security• Unobtrusive relative to other biometric

techniques, but with similar applications:

Automated Facial Recognition

Privacy Concerns

• Off-pose images are a significant challenge for automated facial recognition

• Many current algorithms, including MITRE's, do not include pose correction

Pose Correction

Pose Correction• Our approach to pose-correction involves finding

and matching facial features in different images

• Feature-finding and shape transformation, are also useful for other image-processing tasks

research: use and extend existing approaches

implement: within MITRE's existing codebase

test: using MITRE's test scaffolding and databases

Problem Statement

Our goal is to research, implement, and test a pose correction library that

improves MITRE's existing facial recognition system.

Average of Synthetic Exact

Filters

Active Shape Model

Pose-correction pipeline

Pixels Features Shape

ASEF ASM

• Facial features, or landmarks, can support both recognition and pose-correction

• Features are based on spatial geometry and/or appearance

Features

ASEF filter creation

training image (with known right-

eye location)

human-designed synthetic output

For each training image we create a synthetic output with the correct position of the feature, e.g., the right eye.

ASEF filter creation

training image (with known right-

eye location)

human-designed synthetic output

filter transforming the image at left into

the image at right

We want to create a filter that exactly transforms a training image into the desired synthetic output

* =

ASEF filter creation

In the Fourier domain, we want

where Synthetic, Image, and Filter are the 2D Fourier transforms of the synthetic output, image, and filter.

Complex division thus provides the filter:

ASEF filter creationWe take the average of all of the synthetic exact

filters to define, here, a final right-eye filter

We average 517 filters like this…

ASEF filter creationWe take the average of all of the synthetic exact

filters to define, here, a final right-eye filter

We average 517 filters like this…

…to obtain the final filter?

ASEF filter creationWe take the average of all of the synthetic exact

filters to define, here, a final right-eye filter

We average 517 filters like this…

…to obtain the final filter.

ASEF filter application

The filter’s strongest response is most right-eye-ey location in the image

Unfiltered image Filtered image

We apply the filter in the Fourier domain; the peak in the spatial domain is a first estimate of the feature location

Final output

Gallery

 

Error Images within that error

< .01 26.3 %

< .02 63.9 %

< .05 86.1 %

< .1 87.7 %

ASEF resultsMany images' eyes are found quite accurately,

but there are also some dramatic outliers:

Units are fraction of interocular distance

Percentage of pictures

Influence of ASEF’s Gaussian s

Radius, s = 2px Radius, s = 25pxRadius, s = 15px

synt

heti

c o

utpu

tsA

SEF

filt

ers

Radius, s = 20px

ASEF tradeoffsTesting changes in Gaussian radii (s)

the opposite tradeoff

more accurate localization – and

more outliers

left eye error (units of interocular distance)

Radius, s = 5px

left eye error (units of interocular distance)

ASEF improvementsUsing spatial heuristics as weights

Unweighted filtered image

Spatially weighted filtered image

1.0 * original 0.5 * originaloriginal

Without weighting With weighting

ASEF improvementsUsing spatial heuristics as weights

right eye error (units of interocular distance)

right eye error (units of interocular distance)

left

eye

err

or these clusters show mis-identifying the

left or right eye

Average of Synthetic Exact

Filters

Active Shape Model

Pose-correction pipeline

Pixels Features Shape

ASEF ASM

Active Shape Models (ASM)

• Describe classes of objects with varying shapes

geometric arrangement of facial features: eyes, nose, …

• Each shape is a set of points

• ASM trains on a training set of shapes, creating a statistical model of the variation within that shape-family.

ASM, step 1: Procrustes fitting

Procrustes analysis determines a scaling, rotation, and translation that best align a family of shapes.

training data (hundreds of faces) mean face (not necessarily angry)

We use this approach to align all of the training faces and extract the mean face.

ASM, step 2: Estimating face space

We use the most descriptive eigenvectors to describe the allowable shape domain.

ASM uses principal components analysis to build a model of representative transformations of a face

s = 0 (mean face)

-3s +3s

Independent face-shape axes

ASM, step 3: Transforming facesWe can apply realistic transformations to the

mean face along face space’s eigenvectors.

Second semester plans1) Multi-resolution and weighted ASEF feature finding

2) Adding pixel appearance to the ASM shape models

3) Implementing pose-correction techniques (for pixels)

shape space: yaw

First approach: apply ASM's transformations to generate poses at desired values of pitch and yaw.

Project Work Clinic Deliverables Due DateJanuary Winter break

Spring Semester Begins: 1/17

Phase III Presentation 1/17/2012Implement AAM, continue improving ASEF, research and select pose correction methods

Final Report & Poster

February Begin implementing selected pose correction methods, combine ASEF and ASM

March

Spring Break: 3/9-18Spring Break

Continue work on pose correction

AprilFinal Report Draft of Poster Design 4/2/2012Revise FR, Final Pres Draft 1 of Final Report 4/10/2012

Final Touches Final Report Review 4/12/2012Feature Freeze 4/13/2012Draft of Final Report 4/18/2012Draft of Final Presentation

4/23/2012

May

Finals: 5/3-4Projects Day 5/1/2012Final Report 5/4/2012

Spring ScheduleMITRE clinic, spring 2012 schedule

Questions?

Average of Synthetic Exact

Filters

Active Shape Model

Pixels Features Shape

ASEF ASM

Gallery

Gallery

 

Second semester plans

The spring term will focus on researching and implementing landmark-based pose correction techniques.

First approach: apply transformations given by ASM to generate poses at varying degrees of pitch and yaw.

yaw

pitch

Error Without log transform< .01 26.3 %

< .02 63.9 %

< .05 86.1 %

< .1 87.7 %

ASEF results

Comparing image pre-processing techniques

Error With log transform< .01 25.4 %

< .02 61.3 %

< .05 83.7 %

< .1 85.6 %

Fraction of interocular distance

Percentage of pictures

AAM adds color or grayscale information to ASM’s model. AAM can generate photorealistic faces, not just geometrically realistic ones.

Active Appearance Models (AAM)

Shown here are faces generated by varying the central face’s

appearance parameters by ±3 s along two appearance axes.

from T.F. Cootes, G.J. Edwards, and C.J. Taylor, Active Appearance Models

old pipeline

new pipeline

Face-recognition pipeline

Face detection

Recognition

Landmarking

Pose correction

Input image

Output ID

Fall term’s focus

Spring term’s focus

Next Steps

Improving ASEF:We will experiment with image processing techniques and weighting based on expected pose and image complexity

Extending ASM: We will implement Active Appearance Models to extend face pose-generation to face image-generation.

Implementing Pose Correction: ASEF and ASM provide a baseline approach: namely, transforming a query image to a standard face pose

Pixels

Features

Shape

Automated Facial Recognition• Use of computers to identify faces from images• Commercial systems exist, but MITRE is developing a

system specifically for the US for flexibility and security• Unobtrusive relative to other biometric techniques, but

with similar applications:

Motivation: Uses for Biometrics

• Law enforcement and national security • Fraud detection• Aid distribution• Social networking

Error Percent of Identifications< .01 0.263610315186246

< .02 0.638968481375358

< .05 0.861031518624642

< .1 0.876790830945559

Error Percent of Identifications< .01 0.253581661891117

< .02 0.613180515759312

< .05 0.836676217765043

< .1 0.859598853868195

Without cosine window

With cosine window

ASEF improvementsMapping

Last semester

This semester

Face-recognition pipeline

old pipeline

new pipeline

Face-recognition pipeline

Face detection

Recognition

Landmarking

Pose correction

Input image

Output ID

Training Data Average Face

ASM, step 2: Mean-face finding

We use this approach to align all of the training faces and thus find the mean face.

We got this… ?

Centered! ASEF’s right-eye filter in the spatial domain

Face-recognition pipeline

Face-recognition pipeline

Pixels

Landmarks

Shape Model

Average of Synthetic Exact Filters (ASEF)

Active Appearance Model (AAM)

Landmarking algorithms

ASEF filter creation

For each training image we create a synthetic output with the correct position of the feature, e.g., the right eye.

training image (with known right-eye location)

human-designed synthetic output

• Our approach to pose-correction involves finding and matching facial features in different images

Pose Correction

With dots.

Average of Synthetic Exact Filters (ASEF)

Active Shape Model (ASM)

Landmarking algorithms

Pixels

Features

Shape

old pipeline

Face-recognition pipeline

Face detection

Recognition

Input image

Output ID

Maybe we

don't use

this slide

at all?

old pipeline

new pipeline

Face-recognition pipeline

Face detection

Recognition

Landmarking

Pose correction

Input image

Output ID

Maybe we

don't use

this slide

at all?

MITRE Corporation

Pose Correction for Automatic Facial Recognition

Team: Elliot Godzich, Dylan Marriner, Emily Myers-Stanhope, Emma Taborsky (PM), Heather Williams

Liaisons: Josh Klontz ’10 and Mark BurgeAdvisor: Zachary Dodds

No dots at

all?