A Zero-Shot Framework for Sketch Based Image Retrieval ...sungeui/IR_F18/... · A Zero-Shot...
Transcript of A Zero-Shot Framework for Sketch Based Image Retrieval ...sungeui/IR_F18/... · A Zero-Shot...
![Page 1: A Zero-Shot Framework for Sketch Based Image Retrieval ...sungeui/IR_F18/... · A Zero-Shot Framework for Sketch Based Image Retrieval [ECCV `18] Doheon Lee 20183398 2018. 11. 26](https://reader035.fdocuments.net/reader035/viewer/2022081523/5fd4348722c9ef233b283e85/html5/thumbnails/1.jpg)
A Zero-Shot Framework for Sketch Based Image Retrieval [ECCV `18]
Doheon Lee20183398
2018. 11. 26
CS688 Paper Presentation 2
![Page 2: A Zero-Shot Framework for Sketch Based Image Retrieval ...sungeui/IR_F18/... · A Zero-Shot Framework for Sketch Based Image Retrieval [ECCV `18] Doheon Lee 20183398 2018. 11. 26](https://reader035.fdocuments.net/reader035/viewer/2022081523/5fd4348722c9ef233b283e85/html5/thumbnails/2.jpg)
2
Review : Adversarial Metric Learning
● Metric● Measure similarity between two images
● Mathematical measurements are not intuitive.
● Generating hard negative using GAN.● Better than using existing data for metric
learning
![Page 3: A Zero-Shot Framework for Sketch Based Image Retrieval ...sungeui/IR_F18/... · A Zero-Shot Framework for Sketch Based Image Retrieval [ECCV `18] Doheon Lee 20183398 2018. 11. 26](https://reader035.fdocuments.net/reader035/viewer/2022081523/5fd4348722c9ef233b283e85/html5/thumbnails/3.jpg)
3
Table of Contents
● Introduction
● Background
● Main Contribution
● Experiment & Result
![Page 4: A Zero-Shot Framework for Sketch Based Image Retrieval ...sungeui/IR_F18/... · A Zero-Shot Framework for Sketch Based Image Retrieval [ECCV `18] Doheon Lee 20183398 2018. 11. 26](https://reader035.fdocuments.net/reader035/viewer/2022081523/5fd4348722c9ef233b283e85/html5/thumbnails/4.jpg)
Introduction
![Page 5: A Zero-Shot Framework for Sketch Based Image Retrieval ...sungeui/IR_F18/... · A Zero-Shot Framework for Sketch Based Image Retrieval [ECCV `18] Doheon Lee 20183398 2018. 11. 26](https://reader035.fdocuments.net/reader035/viewer/2022081523/5fd4348722c9ef233b283e85/html5/thumbnails/5.jpg)
5
Image Retrieval
● Text based image retrieval● Search image by textual description
● Content based image retrieval● Search image similar to query image
● Sketch-based Image Retrieval (SBIR)
![Page 6: A Zero-Shot Framework for Sketch Based Image Retrieval ...sungeui/IR_F18/... · A Zero-Shot Framework for Sketch Based Image Retrieval [ECCV `18] Doheon Lee 20183398 2018. 11. 26](https://reader035.fdocuments.net/reader035/viewer/2022081523/5fd4348722c9ef233b283e85/html5/thumbnails/6.jpg)
6
Problems in Coarse Evaluation
● SBIR is usually used for fine-grained IR.● Current methods are focused on class –based
retrieval.
● Shape or attributed-based retrieval are important.
![Page 7: A Zero-Shot Framework for Sketch Based Image Retrieval ...sungeui/IR_F18/... · A Zero-Shot Framework for Sketch Based Image Retrieval [ECCV `18] Doheon Lee 20183398 2018. 11. 26](https://reader035.fdocuments.net/reader035/viewer/2022081523/5fd4348722c9ef233b283e85/html5/thumbnails/7.jpg)
7
Problems in Coarse Evaluation
● Get credit when fetches an image in same class.● No need to match outlines and shape
● Simply learning a class specific mapping
Query Images
![Page 8: A Zero-Shot Framework for Sketch Based Image Retrieval ...sungeui/IR_F18/... · A Zero-Shot Framework for Sketch Based Image Retrieval [ECCV `18] Doheon Lee 20183398 2018. 11. 26](https://reader035.fdocuments.net/reader035/viewer/2022081523/5fd4348722c9ef233b283e85/html5/thumbnails/8.jpg)
8
Fine-grained Evaluation
● Evaluate by comparing the estimated rank.● Annotating rank list by human.
Human biased and requires human labor
Coarse-grained evaluation in the
zero-shot setting.
![Page 9: A Zero-Shot Framework for Sketch Based Image Retrieval ...sungeui/IR_F18/... · A Zero-Shot Framework for Sketch Based Image Retrieval [ECCV `18] Doheon Lee 20183398 2018. 11. 26](https://reader035.fdocuments.net/reader035/viewer/2022081523/5fd4348722c9ef233b283e85/html5/thumbnails/9.jpg)
Related Work
![Page 10: A Zero-Shot Framework for Sketch Based Image Retrieval ...sungeui/IR_F18/... · A Zero-Shot Framework for Sketch Based Image Retrieval [ECCV `18] Doheon Lee 20183398 2018. 11. 26](https://reader035.fdocuments.net/reader035/viewer/2022081523/5fd4348722c9ef233b283e85/html5/thumbnails/10.jpg)
10
Zero-shot Learning
● Learning to recognize images of novel classes
Apple
Banana
Training Set Test Set
???
???
![Page 11: A Zero-Shot Framework for Sketch Based Image Retrieval ...sungeui/IR_F18/... · A Zero-Shot Framework for Sketch Based Image Retrieval [ECCV `18] Doheon Lee 20183398 2018. 11. 26](https://reader035.fdocuments.net/reader035/viewer/2022081523/5fd4348722c9ef233b283e85/html5/thumbnails/11.jpg)
11
Zero-shot Learning
Slide from ECCV 16 ZSL Tutorial
![Page 12: A Zero-Shot Framework for Sketch Based Image Retrieval ...sungeui/IR_F18/... · A Zero-Shot Framework for Sketch Based Image Retrieval [ECCV `18] Doheon Lee 20183398 2018. 11. 26](https://reader035.fdocuments.net/reader035/viewer/2022081523/5fd4348722c9ef233b283e85/html5/thumbnails/12.jpg)
12
Variational Autoencoder
● Find latent features from data
● Encoder● Encodes data (x) to latent variable (z)
● Decoder● decodes latent variable (z) to data(x)
![Page 13: A Zero-Shot Framework for Sketch Based Image Retrieval ...sungeui/IR_F18/... · A Zero-Shot Framework for Sketch Based Image Retrieval [ECCV `18] Doheon Lee 20183398 2018. 11. 26](https://reader035.fdocuments.net/reader035/viewer/2022081523/5fd4348722c9ef233b283e85/html5/thumbnails/13.jpg)
Main Contribution
![Page 14: A Zero-Shot Framework for Sketch Based Image Retrieval ...sungeui/IR_F18/... · A Zero-Shot Framework for Sketch Based Image Retrieval [ECCV `18] Doheon Lee 20183398 2018. 11. 26](https://reader035.fdocuments.net/reader035/viewer/2022081523/5fd4348722c9ef233b283e85/html5/thumbnails/14.jpg)
14
Main Contribution
● Proposed a new benchmark for zero-shot SBIR
● Proposed a generative approach for the SBIR task
![Page 15: A Zero-Shot Framework for Sketch Based Image Retrieval ...sungeui/IR_F18/... · A Zero-Shot Framework for Sketch Based Image Retrieval [ECCV `18] Doheon Lee 20183398 2018. 11. 26](https://reader035.fdocuments.net/reader035/viewer/2022081523/5fd4348722c9ef233b283e85/html5/thumbnails/15.jpg)
15
New Benchmark
● Modified “Sketchy” dataset● Dataset contains images with 6 sketch each
● 125 classes : 104 train, 21 test
![Page 16: A Zero-Shot Framework for Sketch Based Image Retrieval ...sungeui/IR_F18/... · A Zero-Shot Framework for Sketch Based Image Retrieval [ECCV `18] Doheon Lee 20183398 2018. 11. 26](https://reader035.fdocuments.net/reader035/viewer/2022081523/5fd4348722c9ef233b283e85/html5/thumbnails/16.jpg)
16
New Benchmark
● Current SBIR works are class-based.
![Page 17: A Zero-Shot Framework for Sketch Based Image Retrieval ...sungeui/IR_F18/... · A Zero-Shot Framework for Sketch Based Image Retrieval [ECCV `18] Doheon Lee 20183398 2018. 11. 26](https://reader035.fdocuments.net/reader035/viewer/2022081523/5fd4348722c9ef233b283e85/html5/thumbnails/17.jpg)
17
Generative Model for ZS-SBIR
● Sketch gives a basic outline of the image.● Additional details are generated from the latent
prior vector
● Training by sketch-image pairs to model probability density function: 𝒑(𝒙𝒊𝒎𝒈|𝒙𝒔𝒌𝒆𝒕𝒄𝒉; 𝜽)
● The trained result can generate image features.
x: features
![Page 18: A Zero-Shot Framework for Sketch Based Image Retrieval ...sungeui/IR_F18/... · A Zero-Shot Framework for Sketch Based Image Retrieval [ECCV `18] Doheon Lee 20183398 2018. 11. 26](https://reader035.fdocuments.net/reader035/viewer/2022081523/5fd4348722c9ef233b283e85/html5/thumbnails/18.jpg)
18
Conditional VAE
● Variational lower bound for p(x)
● Conditional probability 𝒑(𝒙𝒊𝒎𝒈|𝒙𝒔𝒌𝒆𝒕𝒄𝒉)
q: variational distribution (Gaussian)
![Page 19: A Zero-Shot Framework for Sketch Based Image Retrieval ...sungeui/IR_F18/... · A Zero-Shot Framework for Sketch Based Image Retrieval [ECCV `18] Doheon Lee 20183398 2018. 11. 26](https://reader035.fdocuments.net/reader035/viewer/2022081523/5fd4348722c9ef233b283e85/html5/thumbnails/19.jpg)
19
Conditional VAE
● Regularization loss for preserving latent alignments of the sketch
Generated
feature
![Page 20: A Zero-Shot Framework for Sketch Based Image Retrieval ...sungeui/IR_F18/... · A Zero-Shot Framework for Sketch Based Image Retrieval [ECCV `18] Doheon Lee 20183398 2018. 11. 26](https://reader035.fdocuments.net/reader035/viewer/2022081523/5fd4348722c9ef233b283e85/html5/thumbnails/20.jpg)
20
Conditional Adversarial AE
● Using GAN model replaced KL-Divergence term.● Network Minimize loss
● Discriminator maximize following terms
E: encoder
![Page 21: A Zero-Shot Framework for Sketch Based Image Retrieval ...sungeui/IR_F18/... · A Zero-Shot Framework for Sketch Based Image Retrieval [ECCV `18] Doheon Lee 20183398 2018. 11. 26](https://reader035.fdocuments.net/reader035/viewer/2022081523/5fd4348722c9ef233b283e85/html5/thumbnails/21.jpg)
Experiment & Result
![Page 22: A Zero-Shot Framework for Sketch Based Image Retrieval ...sungeui/IR_F18/... · A Zero-Shot Framework for Sketch Based Image Retrieval [ECCV `18] Doheon Lee 20183398 2018. 11. 26](https://reader035.fdocuments.net/reader035/viewer/2022081523/5fd4348722c9ef233b283e85/html5/thumbnails/22.jpg)
22
Experiment benchmark
● The experiments are done in proposed zero-shot benchmark
● Features are generated from decoder part.● Sampled features are clustered using K-means.
Cluster Center
VGG-16 features
![Page 23: A Zero-Shot Framework for Sketch Based Image Retrieval ...sungeui/IR_F18/... · A Zero-Shot Framework for Sketch Based Image Retrieval [ECCV `18] Doheon Lee 20183398 2018. 11. 26](https://reader035.fdocuments.net/reader035/viewer/2022081523/5fd4348722c9ef233b283e85/html5/thumbnails/23.jpg)
23
Result
(VGG-16)
Deep
Sketch
Hashing
![Page 24: A Zero-Shot Framework for Sketch Based Image Retrieval ...sungeui/IR_F18/... · A Zero-Shot Framework for Sketch Based Image Retrieval [ECCV `18] Doheon Lee 20183398 2018. 11. 26](https://reader035.fdocuments.net/reader035/viewer/2022081523/5fd4348722c9ef233b283e85/html5/thumbnails/24.jpg)
24
Result
Preserved
Attribute