S. Kevin Zhou Siemens Corporate Research, Inc. … Kevin Zhou.pdfApril 23, 2005 @ WOCC S. Kevin...
Transcript of S. Kevin Zhou Siemens Corporate Research, Inc. … Kevin Zhou.pdfApril 23, 2005 @ WOCC S. Kevin...
April 23, 2005 @ WOCC S. Kevin Zhou, SCR
Unconstrained Face Recognition and Analysis
S. Kevin Zhou
Siemens Corporate Research, Inc.
April 23, 2005 @ WOCC S. Kevin Zhou, SCR
Roadmap to Unconstrained Face Recognition and Analysis
• Introduction
• Selected Approaches– Face recognition across illumination.– Face recognition across illumination and pose.– Video-based face recognition.– Age Estimation.
April 23, 2005 @ WOCC S. Kevin Zhou, SCR
Roadmap to Unconstrained Face Recognition and Analysis
• Introduction
• Selected Approaches– Face recognition across illumination.– Face recognition across illumination and pose.– Video-based face recognition.– Age Estimation.
April 23, 2005 @ WOCC S. Kevin Zhou, SCR
Why Face Recognition and Analysis?
• Application.– Non-intrusive biometric.– Homeland security, law enforcement, surveillance.– Virtual reality, HCI, multimedia.
• Theory.– Interdisciplinary: Image/video processing,
mathematics, physics, vision, statistics and learning, psychophysics, neuroscience, etc.
April 23, 2005 @ WOCC S. Kevin Zhou, SCR
State-Of-The-Art
• Current FR systems work well ONLY under controlled situations.– Neutral expression, no makeup (Intrinsic). – Frontal illumination, frontal view (Extrinsic).– Mugshot of good quality.
• Apply pattern recognition techs. to face image.– Appearance-based: Subspace methods
• PCA [Turk & Pentland, 91], LDA [Belhumeur et al., 97].• Local feature analysis (LFA) [Penev & Atick 96], ICA• Neural network, evolutionary computing, genetic algorithm
– Feature-based:• Elastic graph matching [Lades et al., ’93].
April 23, 2005 @ WOCC S. Kevin Zhou, SCR
Unconstrained Face Recognition and Analysis
• Motivation: deal with unconstrained conditions– Intrinsic variations: expression, makeup, aging.– Extrinsic variations: illumination and pose.– Surveillance video.– Age-related: Aging process, age estimation.– Expression and animation.
• Feasible approaches– Combine pattern recognition with variation modeling– Face modeling and animation– Utilized video characteristics– Statistical learning
April 23, 2005 @ WOCC S. Kevin Zhou, SCR
Roadmap to Unconstrained Face Recognition and Analysis
• Introduction
• Selected Approaches– Face recognition across illumination.– Face recognition across illumination and pose.– Video-based face recognition.– Age Estimation.
* S. Zhou, R. Chellappa, and D. Jacobs, “Characterization of human faces under illumination variations using rank, integrability, and symmetry constraints,” European Conf. on Computer Vision, 2004.
April 23, 2005 @ WOCC S. Kevin Zhou, SCR
Illumination affects appearance
* Courtesy of Prof. David Jacobs.
April 23, 2005 @ WOCC S. Kevin Zhou, SCR
Approach
• Generalized photometric stereo.– Describes all possible human face images under all
possible illumination conditions.– Combines a physical illumination model with
statistical regularity in the human class.– Derive an illumination-invariant signature for robust
face recognition under illumination variation.
April 23, 2005 @ WOCC S. Kevin Zhou, SCR
Key Derivations of Generalized Photometric Stereo
[ ])(
)( ,...,, )...(
1313
21
2211
×××
×
⊗=⊗=
+++==
sfWsfTTT
sTTTTsh
mmd
m
mmnd fff
Statistical regularity in identity
Lambertian illumination model
April 23, 2005 @ WOCC S. Kevin Zhou, SCR
FR Across Illumination: Recognition Results
Training set Yale Yale (m=10)
Vetter(m=100)
Method
Average Recognition Rate
Eigenface Generalized Photometric Stereo
Generalized Photometric Stereo
35% 67% 93%
April 23, 2005 @ WOCC S. Kevin Zhou, SCR
Roadmap to Unconstrained Face Recognition and Analysis
• Introduction to unconstrained face recognition.
• Selected Approaches– Face recognition across illumination.– Face recognition across illumination and pose.– Video-based face recognition.– Age Estimation.
* S. Zhou and R. Chellappa, “Image-based face recognition under illumination and pose variations,”Journal of Optical Society of America (JOSA), Feb., 2005.
April 23, 2005 @ WOCC S. Kevin Zhou, SCR
Appearances under illumination and pose variation
l16 l15 l13 l21 l12 l11 l08 l06 l10 l18 l04 l02
c22
c02c37c05c27c29c11c14c34
Illumination
Pose
• 68 objects, 12 lights, 9 poses.
April 23, 2005 @ WOCC S. Kevin Zhou, SCR
Approach
• Illuminating light field– Describes all possible human face images under all
possible illumination conditions and at all possible poses.
– Extends generalized photometric stereo to handle pose variation.
– Derives an illumination- and pose-invariant signature for robust face recognition under illumination and pose variations.
April 23, 2005 @ WOCC S. Kevin Zhou, SCR
Illuminating Light Field (ILF)[Zhou & Chellappa JOSA’05]
• The concept of light field (LF).–
–
– f : illumination- and pose-invariant.
hvs
Ls
)(L 13131 ×××× ⊗= sfWsmmVdVd
)(1 sfWh s ⊗=×vv
d
April 23, 2005 @ WOCC S. Kevin Zhou, SCR
FR Across Illumination and Pose: Recognition Results
Across illuminations Across poses
Illumination variation is easier to handle than pose variation!!
April 23, 2005 @ WOCC S. Kevin Zhou, SCR
Roadmap to Unconstrained Face Recognition and Analysis
• Introduction
• Selected Approaches– Face recognition across illumination.– Face recognition across illumination and pose.– Video-based face recognition.– Age Estimation.
* S. Zhou, V. Krueger, and R. Chellappa, “Probabilistic recognition of human faces from video,” Computer Vision and Image Understanding (special issue on Face Recognition), Vol. 91, pp. 214-245, August 2003.
April 23, 2005 @ WOCC S. Kevin Zhou, SCR
Video presents challenges and chances
• Requires solving both tracking and recognition.• Appearance variation.• Poor image quality.• Multiple frames with temporal continuity.
April 23, 2005 @ WOCC S. Kevin Zhou, SCR
Tracking-then-Recognition v.s. Tracking-and-Recognition Approaches
Tracking-then-recognition Tracking-and-recognition
Essentially still-image-based face recognition
Simultaneous tracking-and-recognition
Utilize temporal information for tracking only
Utilize temporal information for tracking and recognition
Recognition performance relies on tracking accuracy
Improves tracking accuracy and recognition performance
Probabilistic, interpretable
April 23, 2005 @ WOCC S. Kevin Zhou, SCR
Time Series State Space Model
• Motion equation: • Identity equation: • Observation equation:
ttt g u+= − )( 1θθ
1−= tt nn
tnttt tvyh +Ι== };T{ θ
Video frame
?};T{ ttt θyh =
nI
ty
April 23, 2005 @ WOCC S. Kevin Zhou, SCR
Model Solution
• Posterior distribution:: posterior recognition density.: posterior tracking density.
• Particle filter with efficient computation.
)|,( :0 tttnp yθ
)|( :0 ttnp y)|( :0 ttp yθ
April 23, 2005 @ WOCC S. Kevin Zhou, SCR
Tracking Accuracy and Recognition Result
• NIST database – Case 1: Pure tracking using a Laplacian density.– Case 2: Tracking-then-recognition using an IPS density. – Case 3: Tracking-and-recognition using a combined density.
Case Case 1 Case 2 Case 3Tracking Accuracy
87% NA 100%
Recognition within top 1
NA 57% 93%
Recognition within top 3
NA 83% 100%
* Courtesy of the HumanID project
April 23, 2005 @ WOCC S. Kevin Zhou, SCR
Roadmap to Unconstrained Face Recognition and Analysis
• Introduction to unconstrained face recognition.
• Selected Approaches– Face recognition across illumination.– Face recognition across illumination and pose.– Video-based face recognition.– Age Estimation.
* S. Zhou et al., “Image based regression using boosting method,” Submitted.
April 23, 2005 @ WOCC S. Kevin Zhou, SCR
What is Image Based Regression?
• Regression or function approximation– Given an input image , infer or approximate an
output that is associated with the image .
• Age estimation:
x
age=)(xy
)(xyx
April 23, 2005 @ WOCC S. Kevin Zhou, SCR
State-Of-The-Art: Data-Driven Approach
• Nonparametric regression (NPR)– Smoothed k-NN regressor
• Kernel ridge regression (KRR)– Hyperplane in RKHS
• Support vector regression (SVR)– Single output, ε-insensitive loss function
• Boosting regression– Using boosting method– Not data-driven
∑∑==
==N
nnn
N
nnnk
1
T
1
)()(),()( xxwxxwxg φφ
∑=
∝=N
nnnnn hww
1
);,()( );()()( xxxxyxxg
∑<
=
=NI
innii
kwg1
),()( xxx
Hxhxhxg ∈=∑ )( ;)()( mm mmα
April 23, 2005 @ WOCC S. Kevin Zhou, SCR
Challenges: Appearance Variation
• Appearance variation– Inter-object variation.– Extrinsic variations: camera, geometry, lighting, etc.– Alignment/background.
• Treatment of appearance variation– Data-driven approach: Kernel function is global
and sensitive to appearance variation.
– Boosting approach: Feature function is local and robust to appearance variation.
),( nk xx
)(xh m
April 23, 2005 @ WOCC S. Kevin Zhou, SCR
Boosting
• Boosting [Freund & Schapire’95][Friedman et al., AS’00]
– AdaBoost is the state-of-the-art classification method.– Ensemble method: Combines weak learners into a
strong learner using an additive form:
– Selects weak learners (or features) from the dictionary set.
• Three elements of boosting– (a) Loss function or error model – (b) Dictionary set– (c) Selection algorithm
Hxhxhxg ∈=∑ )( ;)()( mm mmα
))(),(( xyxgL
{ })( xhH =
April 23, 2005 @ WOCC S. Kevin Zhou, SCR
Dictionary Set
• Primitives: 1-D decision stump [Viola & Jones, CVPR’01]
feature simple :)( ;otherwise ;1
)( if ;1)( x
xx f
pθpfh
⎩⎨⎧−
≥+=
p = -1θ
f(x)
h(x)
-1
+1
April 23, 2005 @ WOCC S. Kevin Zhou, SCR
Result: Age estimation
• Variations– Pose, illumination, expression, beard, moustache,
spectacle, etc.
• Performance (1002 images, 800 training/202 testing, 5-fold CV)
April 23, 2005 @ WOCC S. Kevin Zhou, SCR
Visual Tracking
* S. Zhou, et al., “Visual tracking and recognition using appearance-adaptive models in particle filters,” IEEE Trans. on Image Processing, November 2004.
April 23, 2005 @ WOCC S. Kevin Zhou, SCR
THANKS for Listening!!!
[email protected]* Shaohua Kevin Zhou, http://www.cfar.umd.edu/~shaohua/