AgendaIntroductionBag-of-words modelsVisual words with spatial locationPart-based modelsDiscriminative methodsSegmentation and recognitionRecognition-based image retrievalDatasets & Conclusions

Classifier based methodsObject detection and recognition is formulated as a classification problem. and a decision is taken at each window about if it contains a target object or not.Where are the screens?The image is partitioned into a set of overlapping windows

Discriminative methods106 examplesNearest neighborNeural networksSupport Vector Machines and KernelsConditional Random Fields

Nearest Neighbors106 examplesShakhnarovich, Viola, Darrell 2003Difficult due to high intrinsic dimensionality of images- lots of data needed- slow neighbor lookupTorralba, Fergus, Freeman 2008

Multi-layer Hubel-Wiesel architecturesNeural networksLeCun, Bottou, Bengio, Haffner 1998Rowley, Baluja, Kanade 1998Hinton & Salakhutdinov 2006Ranzato, Huang, Boureau, LeCun 2007

Riesenhuber & Poggio 1999Serre, Wolf, Poggio. 2005Mutch & Lowe 2006 Biologically inspired

Support Vector MachinesHeisele, Serre, Poggio, 2001Face detectionPyramid Match KernelCombining Multiple KernelsVarma & Roy 2007Bosch, Munoz, Zisserman 2007 Grauman & Darrell 2005Lazebnik, Schmid, Ponce 2006

Conditional Random FieldsKumar & Hebert 2003Quattoni, Collins, Darrell 2004More in Segmentation section

A simple algorithm for learning robust classifiersFreund & Shapire, 1995Friedman, Hastie, Tibshhirani, 1998

Provides efficient algorithm for sparse visual feature selectionTieu & Viola, 2000Viola & Jones, 2003

Easy to implement, not requires external optimization tools.

Boosting

A simple object detector with Boosting Download Toolbox for manipulating dataset Code and dataset

Matlab code Gentle boosting Object detector using a part based model

Dataset with cars and computer monitorshttp://people.csail.mit.edu/torralba/iccv2005/

Boosting Boosting fits the additive modelby minimizing the exponential lossTraining samplesThe exponential loss is a differentiable upper bound to the misclassification error.

Weak classifiers The input is a set of weighted training samples (x,y,w)

Regression stumps: simple but commonly used in object detection.

Four parameters:b=Ew(y [x> q])a=Ew(y [x< q])xfm(x)q

From images to features:A myriad of weak detectorsWe will now define a family of visual features that can be used as weak classifiers (weak detectors)

Takes image as input and the output is binary response.The output is a weak detector.

A myriad of weak detectors

Yuille, Snow, Nitzbert, 1998Amit, Geman 1998Papageorgiou, Poggio, 2000Heisele, Serre, Poggio, 2001Agarwal, Awan, Roth, 2004Schneiderman, Kanade 2004 Carmichael, Hebert 2004

Weak detectorsTextures of textures Tieu and Viola, CVPR 2000

Every combination of three filters generates a different featureThis gives thousands of features. Boosting selects a sparse subset, so computations on test time are very efficient. Boosting also avoids overfitting to some extend.

Haar waveletsHaar filters and integral imageViola and Jones, ICCV 2001

The average intensity in the block is computed with four sums independently of the block size.

Haar waveletsPapageorgiou & Poggio (2000)Polynomial SVM

Edges and chamfer distanceGavrila, Philomin, ICCV 1999

Edge fragmentsWeak detector = k edge fragments and threshold. Chamfer distance uses 8 orientation planesOpelt, Pinz, Zisserman, ECCV 2006

Histograms of oriented gradients Dalal & Trigs, 2006 Shape contextBelongie, Malik, Puzicha, NIPS 2000 SIFT, D. Lowe, ICCV 1999

Weak detectorsPart based: similar to part-based generative models. We create weak detectors by using parts and voting for the object center location

Car modelScreen modelThese features are used for the detector on the course web site.

Weak detectorsFirst we collect a set of part templates from a set of training objects.Vidal-Naquet, Ullman, Nature Neuroscience 2003

Weak detectorsWe now define a family of weak detectors as:==Better than chance*

Weak detectorsWe can do a better job using filtered imagesStill a weak detectorbut better than before**===

Example: screen detectionFeature output

Example: screen detectionFeature outputThresholded outputWeak detectorProduces many false alarms.

Example: screen detectionFeature outputThresholded outputStrong classifier at iteration 1

Example: screen detectionFeature outputThresholded outputStrongclassifierSecond weak detectorProduces a different set of false alarms.

Example: screen detection+Feature outputThresholded outputStrongclassifierStrong classifier at iteration 2

Example: screen detection+Feature outputThresholded outputStrongclassifierStrong classifier at iteration 10

Example: screen detection+Feature outputThresholded outputStrongclassifierAdding featuresFinalclassificationStrong classifier at iteration 200

We want the complexity of the 3 features classifier with the performance of the 100 features classifier:Cascade of classifiersFleuret and Geman 2001, Viola and Jones 2001

3 features30 features100 featuresSelect a threshold with high recall for each stage.

We increase precision using the cascade

Some goals for object recognition Able to detect and recognize many object classesComputationally efficientAble to deal with data starving situations:Some training samples might be harder to collect than othersWe want on-line learning to be fast

Shared featuresIs learning the object class 1000 easier than learning the first?

Can we transfer knowledge from one object to another?Are the shared properties interesting by themselves?

Shared featuresScreen detectorCar detectorFace detector Independent binary classifiers:Torralba, Murphy, Freeman. CVPR 2004. PAMI 2007

50 training samples/class29 object classes2000 entries in the dictionary

Results averaged on 20 runsError bars = 80% intervalKrempp, Geman, & Amit, 2002Torralba, Murphy, Freeman. CVPR 2004 Shared featuresClass-specific features

Generalization as a function of object similaritiesNumber of training samples per classNumber of training samples per classArea under ROCArea under ROCK = 2.1K = 4.8Torralba, Murphy, Freeman. CVPR 2004. PAMI 2007

Sharing patchesBart and Ullman, 2004

For a new class, use only features similar to features that where good for other classes:Proposed Dog features

Sharing transformationsMiller, E., Matsakis, N., and Viola, P. (2000). Learning from one example through shared densities on transforms. In IEEE Computer Vision and Pattern Recognition.

Transformations are sharedand can be learnt from other tasks.

Some references on multiclassCaruana 1997Schapire, Singer, 2000Thrun, Pratt 1997Krempp, Geman, Amit, 2002E.L.Miller, Matsakis, Viola, 2000Mahamud, Hebert, Lafferty, 2001Fink 2004LeCun, Huang, Bottou, 2004Holub, Welling, Perona, 2005

*****************************Strees: the main part is not to discriminate among classes. The hard part is to detect the objects when inmersed in clutter.*****