Adriana pres - Department of Computer [Fei-Fei 2005] L. Fei-Fei and P. Perona. A Bayesian Hi hi l M

download Adriana pres - Department of Computer [Fei-Fei 2005] L. Fei-Fei and P. Perona. A Bayesian Hi hi l M

of 45

  • date post

    10-Jul-2018
  • Category

    Documents

  • view

    214
  • download

    0

Embed Size (px)

Transcript of Adriana pres - Department of Computer [Fei-Fei 2005] L. Fei-Fei and P. Perona. A Bayesian Hi hi l M

  • Scene RecognitionScene Recognition

    Adriana KovashkaAdriana KovashkaUTCS, PhD studentUTCS, PhD student

  • ProblemProblemProblemProblem

    StatementStatement Distinguish between different types of scenes

    Applications Applications Matching human perception Understanding the environment

    Indexing of images / videoR b ti Robotics

    Graphics In painting In-painting

  • BackgroundBackgroundBackgroundBackground

    Definition of a sceneDefinition of a scene [A] scene is mainly characterized as a place

    in which we can move [Oliva 2001]in which we can move [Oliva 2001] Assumptions

    Human categorization Human categorization Approaches

    Parsing of the scene as a whole, or in parts

  • Coast [Oliva 2001]

    Mountain [Oliva 2001]

  • Inside City [Oliva 2001]

    Street [Oliva 2001]

  • Kitchen [Lazebnik 2006]

    Industrial [Lazebnik 2006]

  • Scene?Scene?Scene?Scene?

  • Scene?Scene?Scene?Scene?

  • Urban or natural?Urban or natural?Urban or natural?Urban or natural?

  • Urban or natural?Urban or natural?Urban or natural?Urban or natural?

  • Spatial Envelope [Oliva 2001]Spatial Envelope [Oliva 2001]Spatial Envelope [Oliva 2001]Spatial Envelope [Oliva 2001]

    Inspiration from human perceptionInspiration from human perception Naturalness, openness, roughness

    Expansion ruggedness Expansion, ruggedness Holistic, no recognition of objects Three levels

    cars and people vs. street vs. urban environment

  • Spatial Envelope (contd)Spatial Envelope (contd)Spatial Envelope (cont d)Spatial Envelope (cont d)

    Scene modelingScene modeling Discrete Fourier Transform

    Windowed Fourier Transform[Oliva 2001]

    [Oliva 2001]

  • Spatial Envelope (contd)Spatial Envelope (contd)Spatial Envelope (cont d)Spatial Envelope (cont d)

    Principal Components AnalysisPrincipal Components Analysis

    [Oliva 2001]

  • Spatial Envelope (contd)Spatial Envelope (contd)Spatial Envelope (cont d)Spatial Envelope (cont d)

    [Oliva 2001]

  • Spatial Envelope (contd)Spatial Envelope (contd)Spatial Envelope (cont d)Spatial Envelope (cont d)

    Properties of the spatial envelopeProperties of the spatial envelope Discriminant spectral template (DST)

    Relates spectral components to properties of the Relates spectral components to properties of the spatial envelope

    Parameter d learned through matching of feature vectors and property values

    Windowed discriminant spectral template (WDST)(WDST)

  • Spatial Envelope (contd)Spatial Envelope (contd)Spatial Envelope (cont d)Spatial Envelope (cont d)

    [Oliva 2001]

  • Spatial Envelope (contd)Spatial Envelope (contd)Spatial Envelope (cont d)Spatial Envelope (cont d)

    [Oliva 2001]

  • Spatial Envelope (contd)Spatial Envelope (contd)Spatial Envelope (cont d)Spatial Envelope (cont d)

    ResultsResults Scene properties

    [Oliva 2001]

  • Spatial Spatial Envelope Envelope (contd)(contd)

    [Oliva 2001]

  • Spatial Spatial Envelope Envelope (contd)(contd)

    [Oliva 2001]

  • Spatial Envelope (contd)Spatial Envelope (contd)Spatial Envelope (cont d)Spatial Envelope (cont d)

    ClassificationClassification K-nn 4 out of 7 neighbors g

    picked by humans

    [Oliva 2001]

  • Spatial Envelope (contd)Spatial Envelope (contd)Spatial Envelope (cont d)Spatial Envelope (cont d)

    StrengthsStrengths Higher-level descriptions

    Low dimensionality Low dimensionality Invariance to object composition

    Weak local information Weak local information Weaknesses

    Significant number of human labels

  • [Oliva 2001]

  • Spatial Pyramid [Lazebnik 2006]Spatial Pyramid [Lazebnik 2006]Spatial Pyramid [Lazebnik 2006]Spatial Pyramid [Lazebnik 2006]

    Global locally orderlessGlobal, locally orderless Bag-of-features

    E t i f P id M t h K l i 2 d Extension of Pyramid Match Kernel in 2-d Regular clustering of features

  • Spatial Pyramid (contd)Spatial Pyramid (contd)Spatial Pyramid (cont d)Spatial Pyramid (cont d)

    [Grauman 2005] as quoted in [Lazebnik 2006][Lazebnik 2006]

    [Lazebnik 2006][Lazebnik 2006]

  • Spatial Pyramid (contd)Spatial Pyramid (contd)Spatial Pyramid (cont d)Spatial Pyramid (cont d)

    [Lazebnik 2006]

  • Spatial Pyramid (contd)Spatial Pyramid (contd)Spatial Pyramid (cont d)Spatial Pyramid (cont d)

    [Lazebnik 2006]

  • Spatial Pyramid (contd)Spatial Pyramid (contd)Spatial Pyramid (cont d)Spatial Pyramid (cont d)

    ResultsResults SVM classification

    Scene recognition Scene recognition

    [Lazebnik 2006]]

    65.2% for [Fei-Fei 2005]

  • Spatial Pyramid (contd)Spatial Pyramid (contd)

    [Lazebnik 2006]

  • Spatial Pyramid (contd)Spatial Pyramid (contd)Spatial Pyramid (cont d)Spatial Pyramid (cont d)

    Object recognitionObject recognition

    [Lazebnik 2006]

  • Spatial Pyramid (contd)Spatial Pyramid (contd)Spatial Pyramid (cont d)Spatial Pyramid (cont d)

    StrengthsStrengths Reasonable dimensionality

    Locally orderless Locally orderless Dense representation

    Robust to failures at individual levels Robust to failures at individual levels Weaknesses

    No invariability to composition of image Not robust to clutter

  • Scene CompletionScene Completion http://graphics.cs.cmu.edu/projects/scene-completion/[Hays 2007]

    Input image Scene Descriptor Image Collection

    200 matches20 completionsContext matching

    + blendingHays and Efros, SIGGRAPH 2007

  • [Oliva 2001]

  • Topic Models [FeiTopic Models [Fei--Fei 2005]Fei 2005]Topic Models [FeiTopic Models [Fei Fei 2005]Fei 2005]

    Bayesian hierarchical modelBayesian hierarchical model Intermediate representations

    B f f t Bag-of-features 4 ways to extract regions 2 types of features

  • Topic Models (contd)Topic Models (contd)Topic Models (cont d)Topic Models (cont d)

    [Fei-Fei 2005]

  • Hierarchical Bayesian text modelsHierarchical Bayesian text models[Fei-Fei 2005]

    beach

    Latent Dirichlet Allocation (LDA)

    wN

    c zN

    D

    Fei-Fei et al. ICCV 2005

  • Topic Models (contd)Topic Models (contd)Topic Models (cont d)Topic Models (cont d) distribution of

    l l b lclass labels parameter

    (estimated by EM) c class label distribution of

    themes for image z theme x patchx patch parameter

    (estimated by EM)

    [Fei-Fei 2005]

  • Topic Models (contd)Topic Models (contd)Topic Models (cont d)Topic Models (cont d)

    CodebookCodebook 174 codewords

    [Fei-Fei 2005][Fei-Fei 2005]

  • Topic Models (contd)Topic Models (contd)

    [Fei-Fei 2005]

  • TopicTopicTopic Topic Models Models (contd)(contd)(cont d)(cont d)

    Results

    [Fei-Fei 2005]

  • Topic Models (contd)Topic Models (contd)Topic Models (cont d)Topic Models (cont d)

    StrengthsStrengths Unsupervised

    Invariant to composition Invariant to composition Weaknesses

    No geometry Matches of themes to categories No correspondence to semantic categories

  • ComparisonComparisonComparisonComparison

    Global vs localGlobal vs. local Spatial Envelope, Spatial Pyramid

    Topic Models Topic Models Viewpoint / location biases vs. invariability

    S Spatial Pyramid Topic Models, Spatial Envelope

  • Comparison (contd)Comparison (contd)Comparison (cont d)Comparison (cont d)

    Intermediate representationsIntermediate representations Spatial Envelope, Topic Models

    Supervision vs no supervision Supervision vs. no supervision Spatial Envelope

    S Topic Models, Spatial Pyramid Object recognition?

  • DiscussionDiscussionDiscussionDiscussion

    Object recognition vs scene recognitionObject recognition vs. scene recognition Global approaches

    Spatial Pyramid scenes vs objects results Spatial Pyramid, scenes vs. objects results Bag-of-features

    Use of scene recognition Use of scene recognition Ambiguous scenes Human recognition of scenes

    Importance

  • ReferencesReferencesReferencesReferences[Fei-Fei 2005] L. Fei-Fei and P. Perona. A Bayesian

    Hi hi l M d l f L i N t l SHierarchical Model for Learning Natural Scene Categories. CVPR 2005.

    [Grauman 2005] K. Grauman and T. Darrell. The Pyramid M t h K l Di i i ti Cl ifi ti ith S t fMatch Kernel: Discriminative Classification with Sets of Image Features. ICCV 2005.

    [Hays 2007] J. Hays and A.A. Efros. Scene completion using millions of photographs SIGGRAPH 2007using millions of photographs. SIGGRAPH 2007.

    [Lazebnik 2006] S. Lazebnik, C. Schmid, and J. Ponce. Beyond Bags of Features: Spatial Pyramid Matching for Recognizing Natural Scene Categories CVPR 2006Recognizing Natural Scene Categories. CVPR 2006.

    [Oliva 2001] A. Oliva and A. Torralba. Modeling the Shape of the Scene: a Holistic Representation of the Spatial Envelope IJCV 2001Envelope. IJCV 2001.