End to-end convolutional network for saliency prediction
-
Upload
xavier-giro -
Category
Technology
-
view
168 -
download
2
Transcript of End to-end convolutional network for saliency prediction
![Page 1: End to-end convolutional network for saliency prediction](https://reader030.fdocuments.net/reader030/viewer/2022032504/55c1f696bb61eb086d8b46a3/html5/thumbnails/1.jpg)
End-to-end convolutional network for saliency prediction
Junting Pan Xavier Giró-i-Nieto
Slides online@DocXavi
Large-scale Scene Understanding (LSUN)
Challenge 2015
http://bit.ly/juntingnet
![Page 2: End to-end convolutional network for saliency prediction](https://reader030.fdocuments.net/reader030/viewer/2022032504/55c1f696bb61eb086d8b46a3/html5/thumbnails/2.jpg)
2
Financial supportTechnical support
Albert Gil Josep Pujal
ACKNOWLEDGMENTS
![Page 3: End to-end convolutional network for saliency prediction](https://reader030.fdocuments.net/reader030/viewer/2022032504/55c1f696bb61eb086d8b46a3/html5/thumbnails/3.jpg)
3
LSUN SALIENCY CHALLENGE: A Déjà vu ?
John Markoff, “Scientists see promise in deep learning Programs”, The New York Times (Nov2012).
Photo: Keith Penner
![Page 4: End to-end convolutional network for saliency prediction](https://reader030.fdocuments.net/reader030/viewer/2022032504/55c1f696bb61eb086d8b46a3/html5/thumbnails/4.jpg)
4
LSUN SALIENCY CHALLENGE: A Déjà vu ?
[Mohedano’14]
![Page 5: End to-end convolutional network for saliency prediction](https://reader030.fdocuments.net/reader030/viewer/2022032504/55c1f696bb61eb086d8b46a3/html5/thumbnails/5.jpg)
5
LSUN SALIENCY CHALLENGE: A Déjà vu ?
![Page 6: End to-end convolutional network for saliency prediction](https://reader030.fdocuments.net/reader030/viewer/2022032504/55c1f696bb61eb086d8b46a3/html5/thumbnails/6.jpg)
6
RELATED WORK: Deep Saliency
Kümmerer, Matthias, Lucas Theis, and Matthias Bethge. "Deep Gaze I: Boosting Saliency Prediction with Feature Maps Trained on ImageNet." arXiv preprint arXiv:1411.1045 (2014).
![Page 7: End to-end convolutional network for saliency prediction](https://reader030.fdocuments.net/reader030/viewer/2022032504/55c1f696bb61eb086d8b46a3/html5/thumbnails/7.jpg)
7
RELATED WORK: Deep Saliency
Vig, Eleonora, Michael Dorr, and David Cox. "Large-scale optimization of hierarchical features for saliency prediction in natural images." Computer Vision and Pattern Recognition (CVPR), 2014 IEEE Conference on. IEEE, 2014.
![Page 8: End to-end convolutional network for saliency prediction](https://reader030.fdocuments.net/reader030/viewer/2022032504/55c1f696bb61eb086d8b46a3/html5/thumbnails/8.jpg)
8
RELATED WORK: Fully convolutional
Long, Jonathan, Evan Shelhamer, and Trevor Darrell. "Fully convolutional networks for semantic segmentation." Computer Vision and Pattern Recognition (CVPR), 2015 IEEE Conference on. IEEE, 2015.
![Page 9: End to-end convolutional network for saliency prediction](https://reader030.fdocuments.net/reader030/viewer/2022032504/55c1f696bb61eb086d8b46a3/html5/thumbnails/9.jpg)
9
RELATED WORK: Image Classification
CaffeNet
ARCHITECTURE[Khrizevsky’12]
DATA[Deng’09]
FRAMEWORK[Jia’14]
![Page 10: End to-end convolutional network for saliency prediction](https://reader030.fdocuments.net/reader030/viewer/2022032504/55c1f696bb61eb086d8b46a3/html5/thumbnails/10.jpg)
10
SALIENCY PREDICTION: JuntingNet
JuntingNet
![Page 11: End to-end convolutional network for saliency prediction](https://reader030.fdocuments.net/reader030/viewer/2022032504/55c1f696bb61eb086d8b46a3/html5/thumbnails/11.jpg)
11
SALIENCY PREDICTION: JuntingNet
JuntingNet
DATAiSun [Xu’15]
SALICON [Jiang’15]
![Page 12: End to-end convolutional network for saliency prediction](https://reader030.fdocuments.net/reader030/viewer/2022032504/55c1f696bb61eb086d8b46a3/html5/thumbnails/12.jpg)
12
SALIENCY PREDICTION: Data
TRAIN VALIDATION TEST
SALICON [Jiang’15] 10,000 5,000 5,000
iSun [Xu’15] 6,000 926 2,000
CAT2000 [Borji’15] 2,000 - 2,000
MIT300 [Judd’12] 300 . -
LargeScale
![Page 13: End to-end convolutional network for saliency prediction](https://reader030.fdocuments.net/reader030/viewer/2022032504/55c1f696bb61eb086d8b46a3/html5/thumbnails/13.jpg)
13
SALIENCY PREDICTION: JuntingNet
JuntingNet
ARCHITECTURE[Pan’15]
DATAiSun [Xu’15]
SALICON [Jiang’15]
![Page 14: End to-end convolutional network for saliency prediction](https://reader030.fdocuments.net/reader030/viewer/2022032504/55c1f696bb61eb086d8b46a3/html5/thumbnails/14.jpg)
14
SALIENCY PREDICTION: Architecture
![Page 15: End to-end convolutional network for saliency prediction](https://reader030.fdocuments.net/reader030/viewer/2022032504/55c1f696bb61eb086d8b46a3/html5/thumbnails/15.jpg)
15
SALIENCY PREDICTION: Architecture
End to end + regression = JuntingNet
![Page 16: End to-end convolutional network for saliency prediction](https://reader030.fdocuments.net/reader030/viewer/2022032504/55c1f696bb61eb086d8b46a3/html5/thumbnails/16.jpg)
16
SALIENCY PREDICTION: Architecture
Resize
96x96
Upsample + filter
4608 = 48x48
2D map
![Page 17: End to-end convolutional network for saliency prediction](https://reader030.fdocuments.net/reader030/viewer/2022032504/55c1f696bb61eb086d8b46a3/html5/thumbnails/17.jpg)
17
SALIENCY PREDICTION: JuntingNet
JuntingNet
ARCHITECTURE[Pan’15] (soon)
DATAiSun [Xu’15]
SALICON [Jiang’15]
FRAMEWORK[Bergstra’10][Bastien’12]
![Page 18: End to-end convolutional network for saliency prediction](https://reader030.fdocuments.net/reader030/viewer/2022032504/55c1f696bb61eb086d8b46a3/html5/thumbnails/18.jpg)
18
SALIENCY PREDICTION: Framework
Tutorial by Daniel Nouri (*) on regression for facial points for Kaggle.
(*) Daniel Nouri, “Using convolution networks to detect facil points” (Dec 2014).
on Lasagne
![Page 19: End to-end convolutional network for saliency prediction](https://reader030.fdocuments.net/reader030/viewer/2022032504/55c1f696bb61eb086d8b46a3/html5/thumbnails/19.jpg)
19
SALIENCY PREDICTION: Training
Data augmentation with horizontal mirroring.
![Page 20: End to-end convolutional network for saliency prediction](https://reader030.fdocuments.net/reader030/viewer/2022032504/55c1f696bb61eb086d8b46a3/html5/thumbnails/20.jpg)
20
SALIENCY PREDICTION: Training
Loss function Mean Square Error (MSE)
Weight initialization Gaussian distribution
Learning rate 0.03 to 0.0001
Mini batch size 128
Training time 7h (SALICON) / 3h (iSUN)
Acceleration Sigmoid + nesterov momentum 0.9
Regularisation Maxout norm
GPU NVidia GTX 980
![Page 21: End to-end convolutional network for saliency prediction](https://reader030.fdocuments.net/reader030/viewer/2022032504/55c1f696bb61eb086d8b46a3/html5/thumbnails/21.jpg)
21
RESULTS: Qualitative (iSUN)
JuntingNetGround TruthPixels
![Page 22: End to-end convolutional network for saliency prediction](https://reader030.fdocuments.net/reader030/viewer/2022032504/55c1f696bb61eb086d8b46a3/html5/thumbnails/22.jpg)
22
RESULTS: Qualitative (iSUN)
JuntingNetGround TruthPixels
![Page 23: End to-end convolutional network for saliency prediction](https://reader030.fdocuments.net/reader030/viewer/2022032504/55c1f696bb61eb086d8b46a3/html5/thumbnails/23.jpg)
23
RESULTS: Quantitative (iSUN)
![Page 24: End to-end convolutional network for saliency prediction](https://reader030.fdocuments.net/reader030/viewer/2022032504/55c1f696bb61eb086d8b46a3/html5/thumbnails/24.jpg)
24
RESULTS: Qualitative (SALICON)
JuntingNetGround TruthPixels
![Page 25: End to-end convolutional network for saliency prediction](https://reader030.fdocuments.net/reader030/viewer/2022032504/55c1f696bb61eb086d8b46a3/html5/thumbnails/25.jpg)
25
RESULTS: Qualitative (SALICON)
JuntingNetGround TruthPixels
![Page 26: End to-end convolutional network for saliency prediction](https://reader030.fdocuments.net/reader030/viewer/2022032504/55c1f696bb61eb086d8b46a3/html5/thumbnails/26.jpg)
26
RESULTS: Quantitative (SALICON)
![Page 27: End to-end convolutional network for saliency prediction](https://reader030.fdocuments.net/reader030/viewer/2022032504/55c1f696bb61eb086d8b46a3/html5/thumbnails/27.jpg)
27
RESULTS: Publications by end of June
http://bit.ly/juntingnet
![Page 28: End to-end convolutional network for saliency prediction](https://reader030.fdocuments.net/reader030/viewer/2022032504/55c1f696bb61eb086d8b46a3/html5/thumbnails/28.jpg)
28
Thank you LSUN ! Thank you Boston !
http://bit.ly/juntingnetSlides online @DocXavi