![AnOverviewof TransferLearning - Nanjing Universitylamda.nju.edu.cn/conf/mlss2014/(X(1)S(kcfwaxyqgaqzl2...JMLR 2009] • Transfer$learning$for$ classificaon,$and$ regression$problems.$](https://static.fdocuments.net/doc/165x107/5b05aa8b7f8b9ad1768bb2ef/anoverviewof-transferlearning-nanjing-x1skcfwaxyqgaqzl2jmlr-2009-transferlearningfor.jpg)
Saurabh Gupta - From Captions to Visual Concepts and Back · 2018-11-22 · PHR AP NN VB$ JJ DT...
1
PHR AP NN VB JJ DT PRP IN Oth All All Classifica(on (AlexNet) 39 28 37 37 26 32 25 36 27 Classifica(on (VGG) 45 31 37 40 30 34 26 41 31 MIL (AlexNet) 46 29 40 38 26 32 22 41 30 MIL (VGG) 52 33 44 39 29 34 24 46 34 Human Agreement 64 35 36 43 32 34 32 53 From Captions to Visual Concepts and Back Hao Fang*, Saurabh Gupta*, Forrest Iandola*, Rupesh Srivastava*, Li Deng, Piotr Dollár, Jianfeng Gao, Xiaodong He, Margaret Mitchell, John C. PlaK, C. Lawrence Zitnick, Geoffrey Zweig 1. Word Detection MS COCO Caption Test Server Ablation Study Human Study Results 2. Sentence Generation 3. Sentence Re-Ranking crowd woman camera Purple holding cat #1 A woman holding a camera in a crowd. 3. Sentence ReRanking A purple camera with a woman. A woman holding a camera in a crowd. ... A woman holding a cat. 2. Sentence GeneraWon woman, crowd, cat, camera, holding, purple 1. Word DetecWon Camera p w i =1 - Y j 2b i (1 - p w ij ) p w ij = 1 1 + exp (-v t w φ(b ij ) - u w ) CNN FC6, FC7, FC8 as fully convoluWonal layers MIL SpaWal class probability maps Per class probability Image Results Analysis words Language Model A woman holding woman holding camera crowd A woman holding a camera in a crowd. purple cat Word Probability Attribute Conditioning Objec(ve: Cap(on probability: Relevance: A woman holding a camera in a crowd. Overview We present a novel approach for automaWcally generaWng image descripWons using: • MulWple Instance Learning (MIL) for visually detecWng words • A maximum entropy language model • Sentence ranking using MERT and a Deep MulWmodal Similarity Model (DMSM) Pipeline Multiple Instance Learning (MIL) SoftMax: Noisy Or: Rerank the mbest sentences using Minimum Error Rate Training (MERT). Ranking is based on the following features: DMSM Unique CapWons Seen in Training Human 99.4 4.8 kNearest Neighbor 36.6 100 LSTM / RNN Style 33.1 60.3 Our 47.0 30.0 0 5 10 15 20 25 30 35 40 BLEU BLEU vs. NN Similarity GIST fc7 fc7-fine ME-DMSM Fewer Similar NNs More Similar NNs = human > human >= human kNearest Neighbor 22.1 5.5 27.6 Our 26.2 7.8 34.0 System PPLX BLEU METEOR = human > human >= human UncondiWoned 24.1 1.2 6.8 Shuffled Human 1.7 7.3 Baseline 20.9 16.9 18.9 9.9 2.4 12.3 Baseline + score 20.2 20.1 20.5 16.9 3.9 20.8 Baseline + score +DMSM 20.2 21.1 20.7 18.7 4.6 23.3 Baseline + score + DMSM [m] 19.2 23.3 22.2 VGG + score [m] 18.1 23.6 22.8 VGG + score + DMSM [m] 18.1 25.7 23.6 26.2 7.8 34.0 Human wriKen capWon 19.3 24.1 two ride baseball cat red look 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 Recall Precision man person tennis bed boy road elephant sky kite sidewalk bike ski bottle railroad rug pier apartment Nouns
Transcript of Saurabh Gupta - From Captions to Visual Concepts and Back · 2018-11-22 · PHR AP NN VB$ JJ DT...
PHR AP
NN VB JJ DT PRP IN Oth All All
Classifica(on (AlexNet) 39 28 37 37 26 32 25 36 27
Classifica(on (VGG) 45 31 37 40 30 34 26 41 31
MIL (AlexNet) 46 29 40 38 26 32 22 41 30
MIL (VGG) 52 33 44 39 29 34 24 46 34
Human Agreement 64 35 36 43 32 34 32 53 -‐
From Captions to Visual Concepts and Back
Hao Fang*, Saurabh Gupta*, Forrest Iandola*, Rupesh Srivastava*, Li Deng, Piotr Dollár, Jianfeng Gao, Xiaodong He, Margaret Mitchell, John C. PlaK, C. Lawrence Zitnick, Geoffrey Zweig
1. Word Detection MS COCO Caption Test Server
Ablation Study Human Study
Results
2. Sentence Generation
3. Sentence Re-Ranking
crowd woman
camera
Purple
holding
cat
#1 A woman holding a camera in a crowd.
3. Sentence Re-‐Ranking
A purple camera with a woman. A woman holding a camera in a crowd.
... A woman holding a cat.
2. Sentence GeneraWon
woman, crowd, cat, camera, holding,
purple
1. Word DetecWon
Camera
pwi = 1�Y
j2bi
(1� pwij)
pwij =1
1 + exp (�vtw�(bij)� uw)
CNN FC6, FC7, FC8 as fully convoluWonal layers
MIL
SpaWal class probability maps
Per class probability
Image
Results
Analysis
words
Language Model A woman
holding
woman holding
camera crowd
A woman holding a camera in a crowd.
purple cat Word Probability
Attribute Conditioning
Objec(ve:
Cap(on probability:
Relevance:
A woman holding a camera in a crowd.
Overview We present a novel approach for automaWcally generaWng image descripWons using:
• MulWple Instance Learning (MIL) for visually detecWng words
• A maximum entropy language model • Sentence ranking using MERT and a Deep MulWmodal
Similarity Model (DMSM)
Pipeline
Multiple Instance Learning (MIL)
SoftMax:
Noisy Or:
Re-‐rank the m-‐best sentences using Minimum Error Rate Training (MERT). Ranking is based on the following features:
DMSM
Unique CapWons
Seen in Training
Human 99.4 4.8
k-‐Nearest Neighbor 36.6 100
LSTM / RNN Style 33.1 60.3
Our 47.0 30.0
0
5
10
15
20
25
30
35
40
BLE
U
BLEU vs. NN Similarity
GIST fc7 fc7-fine ME-DMSM
Fewer Similar NNs More Similar NNs
= human
> human
>= human
k-‐Nearest Neighbor 22.1 5.5 27.6
Our 26.2 7.8 34.0
System PPLX BLEU METEOR = human > human >= human UncondiWoned 24.1 1.2 6.8 Shuffled Human -‐ 1.7 7.3 Baseline 20.9 16.9 18.9 9.9 2.4 12.3 Baseline + score 20.2 20.1 20.5 16.9 3.9 20.8 Baseline + score +DMSM 20.2 21.1 20.7 18.7 4.6 23.3 Baseline + score + DMSM [m] 19.2 23.3 22.2 VGG + score [m] 18.1 23.6 22.8 VGG + score + DMSM [m] 18.1 25.7 23.6 26.2 7.8 34.0 Human wriKen capWon -‐ 19.3 24.1
two ride baseball
cat red look
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
Recall
Pre
cisi
on
man
person
tennis
bed
boy
road
elephant
sky
kite
sidewalk
bike
ski
bottle
railroad
rug
pier
apartment
Nouns
![ML with Tensorflow-new - GitHub Pages · Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 7 -68 27 Jan 2016 Case Study: AlexNet [Krizhevsky et al. 2012] Full (simplified) AlexNet](https://static.fdocuments.net/doc/165x107/5f85e5d623d48f15c659d6e2/ml-with-tensorflow-new-github-pages-fei-fei-li-andrej-karpathy-justin.jpg)
![Deep Learning Acceleration via Low Precision Computingmlforsystems.org/assets/slides/isca2019/MLfor... · are fast moving targets for computing platform optimization. AlexNet [39],](https://static.fdocuments.net/doc/165x107/5fd8e0e324b5ce4d23196684/deep-learning-acceleration-via-low-precision-c-are-fast-moving-targets-for-computing.jpg)
