Learning sparse representations to restore, classify, and sense images and videosGuillermo Sapiro

University of MinnesotaSupported by NSF, NGA, NIH, ONR, DARPA, ARO, McKnight Foundation

Learning Sparsity*Martin DuarteRodriguezRamirez

Lecumberry

Learning Sparsity

Learning Sparsity* OverviewIntroductionDenoising, Demosaicing, InpaintingMairal, Elad, Sapiro, IEEE-TIP, January 2008

Learn multiscale dictionaries Mairal, Elad, Sapiro, SIAM-MMS, April 2008

Sparsity + Self-similarityMairal, Bach, Ponce, Sapiro, Zisserman, pre-print.

Incoherent dictionaries and universal codingRamirez, Lecumberry, Sapiro, June 2009, pre-print

Learning to classifyMairal, Bach, Ponce, Sapiro, Zisserman, CVPR 2008, NIPS 2008Rodriguez and Sapiro, pre-print, 2008.

Learning to sense sparse signalsDuarte and Sapiro, pre-print, May 2008, IEEE-TIP to appear

Learning Sparsity

Learning Sparsity*Introduction I: Sparse and Redundant Representations

Webster Dictionary: Of few and scattered elements

Learning Sparsity

Learning Sparsity* Restoration by Energy Minimization y : Given measurements x : Unknown to be recoveredRestoration/representation algorithms are often related to the minimization of an energy function of the formBayesian type of approach

What is the prior? What is the image model?

Learning Sparsity

Learning Sparsity* The Sparseland Model for Images M

Learning Sparsity

Learning Sparsity* What Should the Dictionary D Be?

Learning Sparsity

Learning Sparsity*Introduction II: Dictionary Learning

Learning Sparsity

Learning Sparsity* Measure of Quality for DField & Olshausen (96)Engan et. al. (99)Lewicki & Sejnowski (00)Cotter et. al. (03)Gribonval et. al. (04)Aharon, Elad, & Bruckstein (04) Aharon, Elad, & Bruckstein (05)Ng et al. (07)Mairal, Sapiro, Elad (08)

Learning Sparsity

Learning Sparsity* The KSVD Algorithm General Aharon, Elad, & Bruckstein (`04)

Learning Sparsity

Learning Sparsity*Show me the pictures

Learning Sparsity

Learning Sparsity* Change the Metric in the OMP

Learning Sparsity

Learning Sparsity* Non-uniform noise

Learning Sparsity

Learning Sparsity* Example: Non-uniform noise

Learning Sparsity

Learning Sparsity* Example: Inpainting

Learning Sparsity

Learning Sparsity* Example: Demoisaic

Learning Sparsity

Learning Sparsity* Example: Inpainting

Learning Sparsity

Learning Sparsity*Not enough fun yet?: Multiscale Dictionaries

Learning Sparsity

Learning Sparsity*Learned multiscale dictionary

Learning Sparsity

Learning Sparsity*

Learning Sparsity

Learning Sparsity*Color multiscale dictionaries

Learning Sparsity

Learning Sparsity*Example

Learning Sparsity

Learning Sparsity*Video inpainting

Learning Sparsity

Extending the Models Learning Sparsity*

Learning Sparsity

Universal Coding and Incoherent Dictionaries

ConsistentImproved generalization propertiesImproved active set computationImproved coding speedImproved reconstructionSee poster by Ramirez and Lecumberry

Learning Sparsity*

Learning Sparsity

Sparsity + Self-similarity=Group SparsityCombine the two of the most successful models for images

Mairal, Bach. Ponce, Sapiro, Zisserman, pre-print, 2009 Learning Sparsity*

Learning Sparsity

Learning to Classify

Learning Sparsity*Global Dictionary

Learning Sparsity

Learning Sparsity*Barbara

Learning Sparsity

Learning Sparsity*Boat

Learning Sparsity

Learning Sparsity*Digits

Learning Sparsity

Which dictionary? How to learn them?Multiple reconstructive dictionary? (Payre)

Single reconstructive dictionary? (Ng et al, LeCunn et al.)

Dictionaries for classification!

See also Winn et al., Holub et al., Lasserre et al., Hinton et al. for joint discriminative/generative probabilistic approaches

Learning Sparsity*

Learning Sparsity

Learning Sparsity*Learning multiple reconstructive and discriminative dictionariesWith J. Mairal, F. Bach, J. Ponce, and A. Zisserman, CVPR 08, NIPS 08

Learning Sparsity

Learning Sparsity*Texture classification

Learning Sparsity

Semi-supervised detection learningMIT -- Learning Sparsity*

Learning Sparsity

Learning Sparsity*Learning a Single Discriminative and Reconstructive Dictionary

Exploit the representation coefficients for classificationInclude this in the optimizationClass supervised simultaneous OMPWith F. Rodriguez

Learning Sparsity

Learning Sparsity*Digits images: Robust to noise and occlusions

Learning Sparsity

Learning Sparsity*Supervised Dictionary LearningWith J. Mairal, F. Bach, J. Ponce, and A. Zisserman, NIPS 08

Learning Sparsity

Learning to Sense Sparse Images

MotivationCompressed sensing (Candes &Tao, Donoho, et al.)SparsityRandom samplingUniversalityStabilityShall the sensing be adapted to the data type?Yes! (Elad, Peyre, Weiss et al., Applebaum et al, this talk).Shall the sensing and dictionary be learned simultaneously?

Learning Sparsity*

Learning Sparsity

Some formulas.

Learning Sparsity*+ RIP (Identity Gramm Matrix)

Learning Sparsity

Design the dictionary and sensing togetherLearning Sparsity*

Learning Sparsity

Just Believe the Pictures Learning Sparsity*

Learning Sparsity

Just Believe the PicturesLearning Sparsity*

Learning Sparsity

Just Believe the PicturesLearning Sparsity*

Learning Sparsity

Learning Sparsity*ConclusionsState-of-the-art denoising results for still (shared with Dabov et al.) and videoGeneralVectorial and multiscale learned dictionariesDictionaries with internal structureDictionary learning for classificationSee also Szlam and Sapiro, ICML 2009See also Carin et al, ICML 2009Dictionary learning for sensing

A lot of work still to be done!

Learning Sparsity

Please do not use the wrong dictionaries12 M pixel image7 million patchesLARS+online learning: ~8 minutes

Mairal, Bach, Ponce, Sapiro, ICML 2009

Learning Sparsity*

Learning Sparsity

Learning Sparsity*

Learning Sparsity