Dog Breed Classification Using Part Localization
Jiongxin Liu1, Angjoo Kanazawa2, David Jacobs2, and Peter Belhumeur1
1 Columbia University 2 University of Maryland
Fine-grained classification[Nilsback and Zisserman ’08]
[Parkhi et al ’12]
[Kumar et al ‘12]
[Branson et al ‘10]
Related work• Dense feature extraction:– Mine discriminative region with random forests [Yao et al
’11]– Multiple Kernel Learning [Nilsback and Zisserman ’08]– Post-segmentation [Parkhi and Zisserman ’12]
• Pose-normalized appearance:– Birdlets [Farrell et al ’11]
Related work• Dense feature extraction:– Mine discriminative region with random forests [Yao et al
’11]– Multiple Kernel Learning [Nilsback and Zisserman ’08]– Post-Segmentation [Parkhi and Zisserman ’12]
• Pose-normalized appearance:– Birdlets [Farrell et al ’11]
Generic sampling of features contains more noise than useful
information for fine-grained classification!
Same breed or not?Entlebucher Mountain Dog Greater Swiss Mountain Dog
NO!!
Key insight: Differences in common parts are more informative
Entlebucher Mountain Dog Greater Swiss Mountain Dog
Localize parts based on a non-parameteric method by [Belhumeur et al ‘11]
“Columbia dogs with parts” dataset133 breeds, 8351 images
Low inter-breed variationNorfolk Terrier or Cairn Terrier?
High intra-breed variationBoth labrador retriever
Innumerable Poses
Diverse Appearances
Varying geometry of parts
Overview of the system1. Face Detection 2. Part Detection 3. Feature Extraction and ear localization
4. One vs All classification
Pipeline 1: Dog Face Detection
Keep the 5 highest scoring windows
Pipeline 2: Localize Parts
Idea: From the “fit” to K most similar exemplars weighted by the
detector output, take the most probable part
location
Detector responsesPart locations
Review: Consensus of Exemplars
Local Part Detectors Part LocalizationExemplar Selection
...
Slide from Neeraj Kumar
RANSAC-like Exemplar Selection1. Repeat r times:
a. Choose random exemplar kb. Choose 2 random modes of local detector outputs D={di} on queryc. Find similarity transform t that aligns exemplar to these pointsd. Evaluate match of all i face parts for this (k,t) pair:
e. Add (k,t) pair to list of possible exemplars, ranked by score
2. Take top M (k,t) pairs for determining global configuration
Part detector probabilityat this (aligned) location
Probability of thisconfiguration givendetector outputs
Slide from Neeraj Kumar
Final Part LocalizationFor each face part i:
a. Compute distribution of this part from all M aligned exemplarsb. For each of the top M aligned exemplars [(k,t) pairs]:
Multiply normalized local detector outputs with global distribution of part computed from exemplars to get scores at each pixel location
c. Add all scores together to get final scores at each pixel and choose max
Slide from Neeraj Kumar
Pipeline 2: Localize Parts
From K most similar exemplars and the detector output, take the most probable part location
Detector responses
Difference between current part location and that of exemplar
Part locations
Pipeline 3: Infer ears using detected parts
With r(=10) exemplars from each breed
Pipeline 3: Infer ears using detected parts
With r(=10) exemplars from each breed
Pipeline 4: Classification
Extract SIFT at part locations for each breed+color histogram one vs all linear SVM classifier
Qualitative Results: Successful
Qualitative Results: Failures
Results: ROC curves
Available in iTunes now
Take a Picture
By tapping the nose
Get the breed!
Browse Dog Breeds
Thank you!!
Top Related