Learning Deep Features for Scene Recognition using Places Database · 2017-10-27 · Learning Deep...

20
Learning Deep Features for Scene Recognition using Places Database Bolei Zhou, Agata Lapedriza, Jianxiong Xiao, Antonio Torralba, Aude Oliva NIPS2014 Bora Çelikkale

Transcript of Learning Deep Features for Scene Recognition using Places Database · 2017-10-27 · Learning Deep...

Page 1: Learning Deep Features for Scene Recognition using Places Database · 2017-10-27 · Learning Deep Features for Scene Recognition using Places Database Bolei Zhou, Agata Lapedriza,

Learning Deep Features for Scene Recognition using Places Database

Bolei Zhou, Agata Lapedriza, Jianxiong Xiao, Antonio Torralba, Aude Oliva NIPS2014

Bora Çelikkale

Page 2: Learning Deep Features for Scene Recognition using Places Database · 2017-10-27 · Learning Deep Features for Scene Recognition using Places Database Bolei Zhou, Agata Lapedriza,

INTRODUCTION

Human Visual Recognition

Samples world several times / sec

~millions images within a year

Page 3: Learning Deep Features for Scene Recognition using Places Database · 2017-10-27 · Learning Deep Features for Scene Recognition using Places Database Bolei Zhou, Agata Lapedriza,

INTRODUCTION

Primate Brain

Hierarchical organization in layers of increasing processing complexity

Inspired CNNs

Page 4: Learning Deep Features for Scene Recognition using Places Database · 2017-10-27 · Learning Deep Features for Scene Recognition using Places Database Bolei Zhou, Agata Lapedriza,

PROBLEM & MOTIVATION

Obj Classification have obtained astonishing performanace with large databases (ImageNet)

Iconic images do not contain the richness and diversity of visual info in scenes

Page 5: Learning Deep Features for Scene Recognition using Places Database · 2017-10-27 · Learning Deep Features for Scene Recognition using Places Database Bolei Zhou, Agata Lapedriza,

CONTRIBUTIONS

Scene-centric database 60x larger than SUN

Comparison metrics for scene datasets:Density, Diversity

Page 6: Learning Deep Features for Scene Recognition using Places Database · 2017-10-27 · Learning Deep Features for Scene Recognition using Places Database Bolei Zhou, Agata Lapedriza,

SCENE DATASETS

Scene15 (Lazebnik et al. 2006)

15 categories

~3000 imgs

MIT Indoor67 (Quatham & Torralba 2009)

67 categories of indoor places

15.620 imgs

SUN (Xiao et al. 2010)

397 (well-sampled) categories

130.519 imgs

Places (Zhou et al. 2014)

476 categories

7.076.580 imgs

Page 7: Learning Deep Features for Scene Recognition using Places Database · 2017-10-27 · Learning Deep Features for Scene Recognition using Places Database Bolei Zhou, Agata Lapedriza,

PLACES DATASET

Same categories from SUN

696 popular adjectives in Eng

Google Images

Bing Images

Flickr

>40M imgs are downloaded

1

Page 8: Learning Deep Features for Scene Recognition using Places Database · 2017-10-27 · Learning Deep Features for Scene Recognition using Places Database Bolei Zhou, Agata Lapedriza,

PLACES DATASET

PCA-based duplicate removal across SUN

2 Places & SUN have different images

Allows to combine Places & SUN

Page 9: Learning Deep Features for Scene Recognition using Places Database · 2017-10-27 · Learning Deep Features for Scene Recognition using Places Database Bolei Zhou, Agata Lapedriza,

PLACES DATASET

Annotations (with AMT)

Questions (eg: is this a living room?)

Two round setup:1. Default answer is NO2. Default answer is YES

Imgs shown / round: 750 + 60 from SUN for control

3

Take >90% accuracy

Page 10: Learning Deep Features for Scene Recognition using Places Database · 2017-10-27 · Learning Deep Features for Scene Recognition using Places Database Bolei Zhou, Agata Lapedriza,

COMPARISON METRICS

Relative Density

Page 11: Learning Deep Features for Scene Recognition using Places Database · 2017-10-27 · Learning Deep Features for Scene Recognition using Places Database Bolei Zhou, Agata Lapedriza,

COMPARISON METRICS

Relative Density

Images have more similar neighbors

NN of a1

NN of b1

Page 12: Learning Deep Features for Scene Recognition using Places Database · 2017-10-27 · Learning Deep Features for Scene Recognition using Places Database Bolei Zhou, Agata Lapedriza,

COMPARISON METRICS

Relative Diversity

Simpson Index: two random individual belong to same specie

NN of a1 NN of b

1

Page 13: Learning Deep Features for Scene Recognition using Places Database · 2017-10-27 · Learning Deep Features for Scene Recognition using Places Database Bolei Zhou, Agata Lapedriza,

EXPERIMENTS

Density & Diversity Comparison (AMT)

1 Relative diversity vs. relative density per each category and dataset

Show 12 pairs of images

Workers select the most similar pair

Diversity: pairs are chosen random for each db

Density: 5th NN (avoid near duplicates) is chosen as pair with GIST

Page 14: Learning Deep Features for Scene Recognition using Places Database · 2017-10-27 · Learning Deep Features for Scene Recognition using Places Database Bolei Zhou, Agata Lapedriza,

EXPERIMENTS

Cross Dataset Generalization

2 Training and testing across different datasets

ImageNet-CNN and linear SVM

Page 15: Learning Deep Features for Scene Recognition using Places Database · 2017-10-27 · Learning Deep Features for Scene Recognition using Places Database Bolei Zhou, Agata Lapedriza,

EXPERIMENTS

Comparison with Hand-designed Features

3

Page 16: Learning Deep Features for Scene Recognition using Places Database · 2017-10-27 · Learning Deep Features for Scene Recognition using Places Database Bolei Zhou, Agata Lapedriza,

EXPERIMENTS

Training CNN for Scene Recognition

2,5M imgs from 205 categories, on AlexNet 4

Page 17: Learning Deep Features for Scene Recognition using Places Database · 2017-10-27 · Learning Deep Features for Scene Recognition using Places Database Bolei Zhou, Agata Lapedriza,

PLACES-CNNs

Hybrid-AlexNet

Places + ImageNet 3.5M imgs, 1183 categoriesAccuracy = 0.5230 on validation set

Places205-GoogLeNet (on 205 categories)

Accuracy: top1 = 0.5567, top5 = 0.8541 on validation set

Places205-VGG16 (on 205 categories)

Accuracy: top1 = 0.5890, top5 = 0.8770 on validation set

Page 18: Learning Deep Features for Scene Recognition using Places Database · 2017-10-27 · Learning Deep Features for Scene Recognition using Places Database Bolei Zhou, Agata Lapedriza,

PLACES2 DATASET

400+ unique scene categories

>10M images

AlexNet top1 accuracy: 43.0%

VGG16 top1 accuracy: 47.6%

Page 19: Learning Deep Features for Scene Recognition using Places Database · 2017-10-27 · Learning Deep Features for Scene Recognition using Places Database Bolei Zhou, Agata Lapedriza,

DEMO

http://places.csail.mit.edu/demo.html

http://places2.csail.mit.edu/demo.html

http://places.csail.mit.edu/demo.html
http://places2.csail.mit.edu/demo.html
Page 20: Learning Deep Features for Scene Recognition using Places Database · 2017-10-27 · Learning Deep Features for Scene Recognition using Places Database Bolei Zhou, Agata Lapedriza,

THANK YOU


Theme 3: Places Art Vocabulary - The Getty · Theme 3: Places Art Vocabulary background—the part of a scene that appears most distant; what is behind the foreground and middle ground

Theme 3: Places Art Vocabulary - The Getty · Theme 3: Places Art Vocabulary background—the part of a scene that appears most distant; what is behind the foreground and middle ground

The Mountaineer Pla~ers - Forest Theater · Dramatic Publishing, Woodstock, Illinois . ScENE 1: SCENE 2: ScENE 3: SCENE 4: SCENE 5: ScENE 6: ScENE 7: ScENE 8: SCENE 9: SCENE 10: ScENE

The Mountaineer Pla~ers - Forest Theater · Dramatic Publishing, Woodstock, Illinois . ScENE 1: SCENE 2: ScENE 3: SCENE 4: SCENE 5: ScENE 6: ScENE 7: ScENE 8: SCENE 9: SCENE 10: ScENE

Look No Deeper: Recognizing Places from Opposing Viewpoints … · 2019-02-21 · Look No Deeper: Recognizing Places from Opposing Viewpoints under Varying Scene Appearance using

Look No Deeper: Recognizing Places from Opposing Viewpoints … · 2019-02-21 · Look No Deeper: Recognizing Places from Opposing Viewpoints under Varying Scene Appearance using

Learning Deep Features for Scene Recognition using …papers.nips.cc/paper/5349-learning-deep-features-for...Learning Deep Features for Scene Recognition using Places Database Bolei

Learning Deep Features for Scene Recognition using …papers.nips.cc/paper/5349-learning-deep-features-for...Learning Deep Features for Scene Recognition using Places Database Bolei

Mise en-scene/Cave scene

Mise en-scene/Cave scene

manoa.hawaii.edumanoa.hawaii.edu/.../uploads/1973.A-Streetcar-Named-Desire.pdf · Scene Scene Scene Scene Scene Scene 11: 111: Vil: Scene Vlll: Scene IX: ... "A Streetcar Named Desire"

manoa.hawaii.edumanoa.hawaii.edu/.../uploads/1973.A-Streetcar-Named-Desire.pdf · Scene Scene Scene Scene Scene Scene 11: 111: Vil: Scene Vlll: Scene IX: ... "A Streetcar Named Desire"

Scene Composition ( mis -en-scene)

Scene Composition ( mis -en-scene)

Places: An Image Database for Deep Scene Understandingplaces.csail.mit.edu/places2_arxiv.pdf ·  · 2016-10-10based duplicate removal was conducted within each scene ... top ranked

Places: An Image Database for Deep Scene Understandingplaces.csail.mit.edu/places2_arxiv.pdf ·  · 2016-10-10based duplicate removal was conducted within each scene ... top ranked

Going Places Additional Activities Scene 1 German With English Instructions This project has been funded with support from the European Commission. This.

Going Places Additional Activities Scene 1 German With English Instructions This project has been funded with support from the European Commission. This.

Transfer Learning for Visual Scene Understanding€¦ · Image: adaptedfrom[Zhou,Khosla,Lapedriza,Oliva,Torralba. "Learning Deep Features for Discriminative Localization" CVPR2016],

Transfer Learning for Visual Scene Understanding€¦ · Image: adaptedfrom[Zhou,Khosla,Lapedriza,Oliva,Torralba. "Learning Deep Features for Discriminative Localization" CVPR2016],

Scene by scene final

Scene by scene final

Trading Places - Angus JournalThe notion fits the restaurants just as well. For starters, they hold different customer bases: 40, the more adult couples scene, and Trading Places:

Trading Places - Angus JournalThe notion fits the restaurants just as well. For starters, they hold different customer bases: 40, the more adult couples scene, and Trading Places:

Learning Deep Features for Discriminative Localization - arXiv · Learning Deep Features for Discriminative Localization Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, Antonio

Learning Deep Features for Discriminative Localization - arXiv · Learning Deep Features for Discriminative Localization Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, Antonio

dlomusicaltheatre.comdlomusicaltheatre.com/programs/CalamityJane.pdf · Phone 442-0770 Tlcowtað Wtuoic Sltop, 9"tc. 1214 E. Main Scene l. Scene 2. Scene 3. Scene 4. Scene 5. Scene

dlomusicaltheatre.comdlomusicaltheatre.com/programs/CalamityJane.pdf · Phone 442-0770 Tlcowtað Wtuoic Sltop, 9"tc. 1214 E. Main Scene l. Scene 2. Scene 3. Scene 4. Scene 5. Scene

FICHE TECHNIQUE modifiee5 - pianiste et compositrice de … · FICHE TECHNIQUE SALLE : Tribune télescopique à banquettes (environ 83 places) SCENE : de forme trapézoïdale Ouverture

FICHE TECHNIQUE modifiee5 - pianiste et compositrice de … · FICHE TECHNIQUE SALLE : Tribune télescopique à banquettes (environ 83 places) SCENE : de forme trapézoïdale Ouverture

Knowledge Guided Disambiguation for Large-Scale Scene ... · remarkable progress on scene recognition, partially due to these recent large-scale scene datasets, such as the Places

Knowledge Guided Disambiguation for Large-Scale Scene ... · remarkable progress on scene recognition, partially due to these recent large-scale scene datasets, such as the Places

Learning deep features for scene recognition using Places database

Learning deep features for scene recognition using Places database

Learning Deep Features for Scene Recognition using Places ...vision.princeton.edu/projects/2014/places/paper.pdfan accuracy of 90% or higher on these control images. After the two

Learning Deep Features for Scene Recognition using Places ...vision.princeton.edu/projects/2014/places/paper.pdfan accuracy of 90% or higher on these control images. After the two

Emotion Recognition in Context · 2017. 5. 31. · Emotion Recognition in Context Ronak Kosti∗, Jose M. Alvarez†, Adria Recasens‡, Agata Lapedriza∗ Universitat Oberta de Catalunya∗

Emotion Recognition in Context · 2017. 5. 31. · Emotion Recognition in Context Ronak Kosti∗, Jose M. Alvarez†, Adria Recasens‡, Agata Lapedriza∗ Universitat Oberta de Catalunya∗

Australia Scene by Scene

Australia Scene by Scene