Representation Learning by Learning to Count - UCF CRCV · 2019-03-26 · Learning to Count...
Transcript of Representation Learning by Learning to Count - UCF CRCV · 2019-03-26 · Learning to Count...
![Page 1: Representation Learning by Learning to Count - UCF CRCV · 2019-03-26 · Learning to Count ★Least effort bias Easy to satisfy loss if network always outputs zero. 11 ★Solution?](https://reader033.fdocuments.net/reader033/viewer/2022060317/5f0c5bb47e708231d4350174/html5/thumbnails/1.jpg)
Representation Learning by Learning to CountPresented by: Muhammad Tayyab
1
![Page 2: Representation Learning by Learning to Count - UCF CRCV · 2019-03-26 · Learning to Count ★Least effort bias Easy to satisfy loss if network always outputs zero. 11 ★Solution?](https://reader033.fdocuments.net/reader033/viewer/2022060317/5f0c5bb47e708231d4350174/html5/thumbnails/2.jpg)
Background
★ Supervised representation learning
Feature Extraction Classifier Dog
Object classificationObject detectionSemantic segmentation
2https://www.youtube.com/watch?v=ZmaXDb9akEI
![Page 3: Representation Learning by Learning to Count - UCF CRCV · 2019-03-26 · Learning to Count ★Least effort bias Easy to satisfy loss if network always outputs zero. 11 ★Solution?](https://reader033.fdocuments.net/reader033/viewer/2022060317/5f0c5bb47e708231d4350174/html5/thumbnails/3.jpg)
Background
★ Supervised representation learning
Feature Extraction Classifier Dog
3
Require human annotation● Costly● Error prone● Time consuming
![Page 4: Representation Learning by Learning to Count - UCF CRCV · 2019-03-26 · Learning to Count ★Least effort bias Easy to satisfy loss if network always outputs zero. 11 ★Solution?](https://reader033.fdocuments.net/reader033/viewer/2022060317/5f0c5bb47e708231d4350174/html5/thumbnails/4.jpg)
Background
★ Self-supervised representation learning
Feature Extraction Pseudo task Solve task
Doesn’t require human annotation● Cheap● Scalable
4
![Page 5: Representation Learning by Learning to Count - UCF CRCV · 2019-03-26 · Learning to Count ★Least effort bias Easy to satisfy loss if network always outputs zero. 11 ★Solution?](https://reader033.fdocuments.net/reader033/viewer/2022060317/5f0c5bb47e708231d4350174/html5/thumbnails/5.jpg)
★ Good representations
Background
5
Representation hyperspace
![Page 6: Representation Learning by Learning to Count - UCF CRCV · 2019-03-26 · Learning to Count ★Least effort bias Easy to satisfy loss if network always outputs zero. 11 ★Solution?](https://reader033.fdocuments.net/reader033/viewer/2022060317/5f0c5bb47e708231d4350174/html5/thumbnails/6.jpg)
Background
★ Representation learning
Dog
6http://cv-tricks.com/cnn/understand-resnet-alexnet-vgg-inception/
![Page 7: Representation Learning by Learning to Count - UCF CRCV · 2019-03-26 · Learning to Count ★Least effort bias Easy to satisfy loss if network always outputs zero. 11 ★Solution?](https://reader033.fdocuments.net/reader033/viewer/2022060317/5f0c5bb47e708231d4350174/html5/thumbnails/7.jpg)
Background
★ Representation learning
Dog
7
![Page 8: Representation Learning by Learning to Count - UCF CRCV · 2019-03-26 · Learning to Count ★Least effort bias Easy to satisfy loss if network always outputs zero. 11 ★Solution?](https://reader033.fdocuments.net/reader033/viewer/2022060317/5f0c5bb47e708231d4350174/html5/thumbnails/8.jpg)
Background
★ Representation learning
Cat
8
![Page 9: Representation Learning by Learning to Count - UCF CRCV · 2019-03-26 · Learning to Count ★Least effort bias Easy to satisfy loss if network always outputs zero. 11 ★Solution?](https://reader033.fdocuments.net/reader033/viewer/2022060317/5f0c5bb47e708231d4350174/html5/thumbnails/9.jpg)
★ Idea: Sum of predicted patchwise feature count should be similar to the image feature count
Similarity based on counting
9
![Page 10: Representation Learning by Learning to Count - UCF CRCV · 2019-03-26 · Learning to Count ★Least effort bias Easy to satisfy loss if network always outputs zero. 11 ★Solution?](https://reader033.fdocuments.net/reader033/viewer/2022060317/5f0c5bb47e708231d4350174/html5/thumbnails/10.jpg)
★ Formulation:○ Downsampling operator D.○ Tiling operator Tj.
Learning to Count
★ Loss
10
![Page 11: Representation Learning by Learning to Count - UCF CRCV · 2019-03-26 · Learning to Count ★Least effort bias Easy to satisfy loss if network always outputs zero. 11 ★Solution?](https://reader033.fdocuments.net/reader033/viewer/2022060317/5f0c5bb47e708231d4350174/html5/thumbnails/11.jpg)
Learning to Count
★ Least effort bias○ Easy to satisfy loss if network always outputs zero.
11
★ Solution?○ Contrastive loss○ Learn features useful for discrimination.
![Page 12: Representation Learning by Learning to Count - UCF CRCV · 2019-03-26 · Learning to Count ★Least effort bias Easy to satisfy loss if network always outputs zero. 11 ★Solution?](https://reader033.fdocuments.net/reader033/viewer/2022060317/5f0c5bb47e708231d4350174/html5/thumbnails/12.jpg)
Learning to Count
Contrastive loss:
12
![Page 13: Representation Learning by Learning to Count - UCF CRCV · 2019-03-26 · Learning to Count ★Least effort bias Easy to satisfy loss if network always outputs zero. 11 ★Solution?](https://reader033.fdocuments.net/reader033/viewer/2022060317/5f0c5bb47e708231d4350174/html5/thumbnails/13.jpg)
★ Observation
Experiments
Average response of trained network on the ImageNet validation set13
![Page 14: Representation Learning by Learning to Count - UCF CRCV · 2019-03-26 · Learning to Count ★Least effort bias Easy to satisfy loss if network always outputs zero. 11 ★Solution?](https://reader033.fdocuments.net/reader033/viewer/2022060317/5f0c5bb47e708231d4350174/html5/thumbnails/14.jpg)
Experiments
★ Transfer Learning
14
Cat
Classification
Cat
Detection Segmentation
http://cs231n.stanford.edu/slides/2017/cs231n_2017_lecture11.pdf
![Page 15: Representation Learning by Learning to Count - UCF CRCV · 2019-03-26 · Learning to Count ★Least effort bias Easy to satisfy loss if network always outputs zero. 11 ★Solution?](https://reader033.fdocuments.net/reader033/viewer/2022060317/5f0c5bb47e708231d4350174/html5/thumbnails/15.jpg)
★ Transfer Learning
Experiments
Evaluation of transfer learning on PASCAL15
![Page 16: Representation Learning by Learning to Count - UCF CRCV · 2019-03-26 · Learning to Count ★Least effort bias Easy to satisfy loss if network always outputs zero. 11 ★Solution?](https://reader033.fdocuments.net/reader033/viewer/2022060317/5f0c5bb47e708231d4350174/html5/thumbnails/16.jpg)
★ Transfer Learning
Experiments
16
ImageNet classification with a linear classifier
![Page 17: Representation Learning by Learning to Count - UCF CRCV · 2019-03-26 · Learning to Count ★Least effort bias Easy to satisfy loss if network always outputs zero. 11 ★Solution?](https://reader033.fdocuments.net/reader033/viewer/2022060317/5f0c5bb47e708231d4350174/html5/thumbnails/17.jpg)
Experiments
★ Places data set○ By MIT○ 10,624,928 Images○ 434 Classes
17
![Page 18: Representation Learning by Learning to Count - UCF CRCV · 2019-03-26 · Learning to Count ★Least effort bias Easy to satisfy loss if network always outputs zero. 11 ★Solution?](https://reader033.fdocuments.net/reader033/viewer/2022060317/5f0c5bb47e708231d4350174/html5/thumbnails/18.jpg)
○ 434 Classes
Experiments
18
![Page 19: Representation Learning by Learning to Count - UCF CRCV · 2019-03-26 · Learning to Count ★Least effort bias Easy to satisfy loss if network always outputs zero. 11 ★Solution?](https://reader033.fdocuments.net/reader033/viewer/2022060317/5f0c5bb47e708231d4350174/html5/thumbnails/19.jpg)
Experiments
19
Places classification with a linear classifier
★ Transfer Learning
![Page 20: Representation Learning by Learning to Count - UCF CRCV · 2019-03-26 · Learning to Count ★Least effort bias Easy to satisfy loss if network always outputs zero. 11 ★Solution?](https://reader033.fdocuments.net/reader033/viewer/2022060317/5f0c5bb47e708231d4350174/html5/thumbnails/20.jpg)
★ Ablation studies
Experiments
20
![Page 21: Representation Learning by Learning to Count - UCF CRCV · 2019-03-26 · Learning to Count ★Least effort bias Easy to satisfy loss if network always outputs zero. 11 ★Solution?](https://reader033.fdocuments.net/reader033/viewer/2022060317/5f0c5bb47e708231d4350174/html5/thumbnails/21.jpg)
★ Ablation studies
Experiments
21
As an error metric, we use the first term in the loss function normalized by the average of the norm of the feature vector. More precisely, the error when the network is trained with the i-th downsampling style and tested on the j-th one is
![Page 22: Representation Learning by Learning to Count - UCF CRCV · 2019-03-26 · Learning to Count ★Least effort bias Easy to satisfy loss if network always outputs zero. 11 ★Solution?](https://reader033.fdocuments.net/reader033/viewer/2022060317/5f0c5bb47e708231d4350174/html5/thumbnails/22.jpg)
★ Ablation studies
Experiments
22
Learning the downsampling style
![Page 23: Representation Learning by Learning to Count - UCF CRCV · 2019-03-26 · Learning to Count ★Least effort bias Easy to satisfy loss if network always outputs zero. 11 ★Solution?](https://reader033.fdocuments.net/reader033/viewer/2022060317/5f0c5bb47e708231d4350174/html5/thumbnails/23.jpg)
★ Counting
Experiments
23
Image croppings of increasing size. The number of visual primitives should increase going from left to right
Counting evaluation on ImageNet
![Page 24: Representation Learning by Learning to Count - UCF CRCV · 2019-03-26 · Learning to Count ★Least effort bias Easy to satisfy loss if network always outputs zero. 11 ★Solution?](https://reader033.fdocuments.net/reader033/viewer/2022060317/5f0c5bb47e708231d4350174/html5/thumbnails/24.jpg)
Experiments
24Examples of activating/ignored images for ImageNet test set
Least Activating Im
agesMos
t Act
ivat
ing
Imag
es
![Page 25: Representation Learning by Learning to Count - UCF CRCV · 2019-03-26 · Learning to Count ★Least effort bias Easy to satisfy loss if network always outputs zero. 11 ★Solution?](https://reader033.fdocuments.net/reader033/viewer/2022060317/5f0c5bb47e708231d4350174/html5/thumbnails/25.jpg)
Experiments
25Examples of activating/ignored images for COCO test set
Least Activating Im
agesMos
t Act
ivat
ing
Imag
es
![Page 26: Representation Learning by Learning to Count - UCF CRCV · 2019-03-26 · Learning to Count ★Least effort bias Easy to satisfy loss if network always outputs zero. 11 ★Solution?](https://reader033.fdocuments.net/reader033/viewer/2022060317/5f0c5bb47e708231d4350174/html5/thumbnails/26.jpg)
Experiments
26Nearest neighbor retrievals for ImageNet
Query Images Matches
![Page 27: Representation Learning by Learning to Count - UCF CRCV · 2019-03-26 · Learning to Count ★Least effort bias Easy to satisfy loss if network always outputs zero. 11 ★Solution?](https://reader033.fdocuments.net/reader033/viewer/2022060317/5f0c5bb47e708231d4350174/html5/thumbnails/27.jpg)
Experiments
27Nearest neighbor retrievals for COCO
Query Images Matches
![Page 28: Representation Learning by Learning to Count - UCF CRCV · 2019-03-26 · Learning to Count ★Least effort bias Easy to satisfy loss if network always outputs zero. 11 ★Solution?](https://reader033.fdocuments.net/reader033/viewer/2022060317/5f0c5bb47e708231d4350174/html5/thumbnails/28.jpg)
Experiments
28Blocks of the 8 most activating images for 4 neurons for ImageNet
Neu
ron
1N
euro
n 3
Neuron 4
Neuron 2
![Page 29: Representation Learning by Learning to Count - UCF CRCV · 2019-03-26 · Learning to Count ★Least effort bias Easy to satisfy loss if network always outputs zero. 11 ★Solution?](https://reader033.fdocuments.net/reader033/viewer/2022060317/5f0c5bb47e708231d4350174/html5/thumbnails/29.jpg)
Experiments
29Blocks of the 8 most activating images for 4 neurons for COCO
Neu
ron
1N
euro
n 3
Neuron 4
Neuron 2
![Page 30: Representation Learning by Learning to Count - UCF CRCV · 2019-03-26 · Learning to Count ★Least effort bias Easy to satisfy loss if network always outputs zero. 11 ★Solution?](https://reader033.fdocuments.net/reader033/viewer/2022060317/5f0c5bb47e708231d4350174/html5/thumbnails/30.jpg)
Thank you!
30