Diagnosing Error in Object Detectors
description
Transcript of Diagnosing Error in Object Detectors
![Page 1: Diagnosing Error in Object Detectors](https://reader036.fdocuments.net/reader036/viewer/2022062501/5681611b550346895dd073c0/html5/thumbnails/1.jpg)
Diagnosing Error in Object Detectors
Department of Computer Science
University of Illinois at Urbana-Champaign (UIUC)
Derek HoiemYodsawalai Chodpathumwan
Qieyun Dai
Work supported in part by NSF awards IIS-1053768 and IIS-0904209, ONR MURI Grant N000141010934, and a research award from Google
![Page 2: Diagnosing Error in Object Detectors](https://reader036.fdocuments.net/reader036/viewer/2022062501/5681611b550346895dd073c0/html5/thumbnails/2.jpg)
Object detection is a collection of problems
DistanceShapeOcclusion Viewpoint
Intra-class Variation for “Airplane”
![Page 3: Diagnosing Error in Object Detectors](https://reader036.fdocuments.net/reader036/viewer/2022062501/5681611b550346895dd073c0/html5/thumbnails/3.jpg)
Object detection is a collection of problems
Localization ErrorBackground
Dissimilar Categories
Similar Categories
Confusing Distractors for “Airplane”
![Page 4: Diagnosing Error in Object Detectors](https://reader036.fdocuments.net/reader036/viewer/2022062501/5681611b550346895dd073c0/html5/thumbnails/4.jpg)
How to evaluate object detectors?• Average Precision (AP)
– Good summary statistic for quick comparison– Not a good driver of research
• We propose tools to evaluate – where detectors fail– potential impact of particular improvements
Typical evaluation through comparison of AP numbers
figs from Felzenszwalb et al. 2010
![Page 5: Diagnosing Error in Object Detectors](https://reader036.fdocuments.net/reader036/viewer/2022062501/5681611b550346895dd073c0/html5/thumbnails/5.jpg)
Detectors Analyzed as Examples on VOC 2007
Deformable Parts Model (DPM)
Felzenszwalb et al. 2010 (v4)
• Sliding window• Mixture of HOG templates with
latent HOG parts
Multiple Kernel Learning (MKL)
Vedaldi et al. 2009
• Jumping window• Various spatial pyramid bag of
words features combined with MKL
x xx
![Page 6: Diagnosing Error in Object Detectors](https://reader036.fdocuments.net/reader036/viewer/2022062501/5681611b550346895dd073c0/html5/thumbnails/6.jpg)
Top false positives: Airplane (DPM)
3
27 37
1
4
5
30
332
6
7
Other Objects11%
Background27%
Similar Objects33%
Bird, Boat, Car
Localization29%
Impact of Removing/Fixing FPs
AP = 0.36
![Page 7: Diagnosing Error in Object Detectors](https://reader036.fdocuments.net/reader036/viewer/2022062501/5681611b550346895dd073c0/html5/thumbnails/7.jpg)
Top false positives: Dog (DPM)
Similar Objects50%
Person, Cat, Horse
1 6 16 42 5
8 22
Background23%
93
10
Localization17%
Impact of Removing/Fixing FPs
Other Objects10%
AP = 0.03
![Page 8: Diagnosing Error in Object Detectors](https://reader036.fdocuments.net/reader036/viewer/2022062501/5681611b550346895dd073c0/html5/thumbnails/8.jpg)
Top false positives: Dog (MKL)
Similar Objects74%
Cow, Person, Sheep, Horse
Background4%
Localization17%
Other Objects5%
AP = 0.17
Impact of Removing/Fixing FPs
Top 5 FP
![Page 9: Diagnosing Error in Object Detectors](https://reader036.fdocuments.net/reader036/viewer/2022062501/5681611b550346895dd073c0/html5/thumbnails/9.jpg)
Summary of False Positive Analysis
DPM v4(FGMR 2010)
MKL (Vedaldi et al. 2009)
![Page 10: Diagnosing Error in Object Detectors](https://reader036.fdocuments.net/reader036/viewer/2022062501/5681611b550346895dd073c0/html5/thumbnails/10.jpg)
Analysis of object characteristics
Additional annotations for seven categories: occlusion level, parts visible, sides visible
Occlusion Level
![Page 11: Diagnosing Error in Object Detectors](https://reader036.fdocuments.net/reader036/viewer/2022062501/5681611b550346895dd073c0/html5/thumbnails/11.jpg)
Normalized Average Precision• Average precision is sensitive to number of
positive examples
• Normalized average precision: replace variable Nj with fixed N
Precision= TruePositivesTruePositives+FalsePositives
TruePositives=Recall∗𝑁 𝑗Number of object examples in subset j
![Page 12: Diagnosing Error in Object Detectors](https://reader036.fdocuments.net/reader036/viewer/2022062501/5681611b550346895dd073c0/html5/thumbnails/12.jpg)
Object characteristics: Aeroplane
![Page 13: Diagnosing Error in Object Detectors](https://reader036.fdocuments.net/reader036/viewer/2022062501/5681611b550346895dd073c0/html5/thumbnails/13.jpg)
Object characteristics: AeroplaneOcclusion: poor robustness to occlusion, but little impact on overall performance
Easier (None) Harder (Heavy)
![Page 14: Diagnosing Error in Object Detectors](https://reader036.fdocuments.net/reader036/viewer/2022062501/5681611b550346895dd073c0/html5/thumbnails/14.jpg)
Size: strong preference for average to above average sized airplanes
Object characteristics: Aeroplane
Easier Harder
X-SmallSmallX-LargeMediumLarge
![Page 15: Diagnosing Error in Object Detectors](https://reader036.fdocuments.net/reader036/viewer/2022062501/5681611b550346895dd073c0/html5/thumbnails/15.jpg)
Aspect Ratio: 2-3x better at detecting wide (side) views than tall views
Object characteristics: Aeroplane
TallX-TallMediumWideX-Wide
Easier (Wide) Harder (Tall)
![Page 16: Diagnosing Error in Object Detectors](https://reader036.fdocuments.net/reader036/viewer/2022062501/5681611b550346895dd073c0/html5/thumbnails/16.jpg)
Sides/Parts: best performance = direct side view with all parts visible
Object characteristics: Aeroplane
Easier (Side) Harder (Non-Side)
![Page 17: Diagnosing Error in Object Detectors](https://reader036.fdocuments.net/reader036/viewer/2022062501/5681611b550346895dd073c0/html5/thumbnails/17.jpg)
Summarizing Detector Performance
Avg. Performance of Best Case
Avg. Performance of Worst Case
Avg. Overall Performance
DPM (v4): Sensitivity and Impact
![Page 18: Diagnosing Error in Object Detectors](https://reader036.fdocuments.net/reader036/viewer/2022062501/5681611b550346895dd073c0/html5/thumbnails/18.jpg)
DPM (FGMR 2010)MKL (Vedaldi et al. 2009)
occlusion trunc size view part_visaspect
Sensitivity
Impact
Summarizing Detector PerformanceBest, Average, Worst Case
![Page 19: Diagnosing Error in Object Detectors](https://reader036.fdocuments.net/reader036/viewer/2022062501/5681611b550346895dd073c0/html5/thumbnails/19.jpg)
DPM (FGMR 2010)MKL (Vedaldi et al. 2009)
occlusion trunc size view part_visaspect
Occlusion: high sensitivity, low potential impact
Summarizing Detector PerformanceBest, Average, Worst Case
![Page 20: Diagnosing Error in Object Detectors](https://reader036.fdocuments.net/reader036/viewer/2022062501/5681611b550346895dd073c0/html5/thumbnails/20.jpg)
DPM (FGMR 2010)MKL (Vedaldi et al. 2009)
occlusion trunc size view part_visaspect
MKL more sensitive to size
Summarizing Detector PerformanceBest, Average, Worst Case
![Page 21: Diagnosing Error in Object Detectors](https://reader036.fdocuments.net/reader036/viewer/2022062501/5681611b550346895dd073c0/html5/thumbnails/21.jpg)
DPM (FGMR 2010)MKL (Vedaldi et al. 2009)
occlusion trunc size view part_visaspect
DPM more sensitive to aspect
Summarizing Detector PerformanceBest, Average, Worst Case
![Page 22: Diagnosing Error in Object Detectors](https://reader036.fdocuments.net/reader036/viewer/2022062501/5681611b550346895dd073c0/html5/thumbnails/22.jpg)
Conclusions
• Most errors that detectors make are reasonable– Localization error and confusion with similar objects– Misdetection of occluded or small objects
• Large improvements in specific areas (e.g., remove all background FPs or robustness to occlusion) has small impact in overall AP– More specific analysis should be standard
• Our code and annotations are available online– Automatic generation of analysis summary from standard
annotations
www.cs.illinois.edu/homes/dhoiem/publications/detectionAnalysis_eccv12.tar.gz
![Page 23: Diagnosing Error in Object Detectors](https://reader036.fdocuments.net/reader036/viewer/2022062501/5681611b550346895dd073c0/html5/thumbnails/23.jpg)
Thank you!
Similar Objects74%
Cow, Person, Sheep, Horse
Background4%
Localization17%
Other Objects5%
AP = 0.17
Impact of Removing/Fixing FPs
Top 5 FP
Top Dog False Positives
www.cs.illinois.edu/homes/dhoiem/publications/detectionAnalysis_eccv12.tar.gz
![Page 24: Diagnosing Error in Object Detectors](https://reader036.fdocuments.net/reader036/viewer/2022062501/5681611b550346895dd073c0/html5/thumbnails/24.jpg)