ML with Tensorflow-new - GitHub Pages › ml › lec12.pdf · 2017-10-02 · Fei-Fei Li & Andrej...

27
Lecture 12 RNN Sung Kim <[email protected]> http://hunkim.github.io/ml/

Transcript of ML with Tensorflow-new - GitHub Pages › ml › lec12.pdf · 2017-10-02 · Fei-Fei Li & Andrej...

Page 1: ML with Tensorflow-new - GitHub Pages › ml › lec12.pdf · 2017-10-02 · Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 10 -16 8 Feb 2016 Recurrent Neural Network x RNN

Lecture 12 RNN

Sung Kim <[email protected]>http://hunkim.github.io/ml/

Page 2: ML with Tensorflow-new - GitHub Pages › ml › lec12.pdf · 2017-10-02 · Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 10 -16 8 Feb 2016 Recurrent Neural Network x RNN

Sequence data

• We don’t understand one word only

• We understand based on the previous words + this word. (time series)

• NN/CNN cannot do this

Page 3: ML with Tensorflow-new - GitHub Pages › ml › lec12.pdf · 2017-10-02 · Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 10 -16 8 Feb 2016 Recurrent Neural Network x RNN

Sequence data

• We don’t understand one word only

• We understand based on the previous words + this word. (time series)

• NN/CNN cannot do this

http://colah.github.io/posts/2015-08-Understanding-LSTMs/

Page 4: ML with Tensorflow-new - GitHub Pages › ml › lec12.pdf · 2017-10-02 · Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 10 -16 8 Feb 2016 Recurrent Neural Network x RNN

http://colah.github.io/posts/2015-08-Understanding-LSTMs/

Page 5: ML with Tensorflow-new - GitHub Pages › ml › lec12.pdf · 2017-10-02 · Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 10 -16 8 Feb 2016 Recurrent Neural Network x RNN

Lecture 10 - 8 Feb 2016Fei-Fei Li & Andrej Karpathy & Justin JohnsonFei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 10 - 8 Feb 201613

Recurrent Neural Network

x

RNN

Page 6: ML with Tensorflow-new - GitHub Pages › ml › lec12.pdf · 2017-10-02 · Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 10 -16 8 Feb 2016 Recurrent Neural Network x RNN

Lecture 10 - 8 Feb 2016Fei-Fei Li & Andrej Karpathy & Justin JohnsonFei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 10 - 8 Feb 201614

Recurrent Neural Network

x

RNN

yusually want to predict a vector at some time steps

Page 7: ML with Tensorflow-new - GitHub Pages › ml › lec12.pdf · 2017-10-02 · Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 10 -16 8 Feb 2016 Recurrent Neural Network x RNN

Lecture 10 - 8 Feb 2016Fei-Fei Li & Andrej Karpathy & Justin JohnsonFei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 10 - 8 Feb 201615

Recurrent Neural Network

x

RNN

yWe can process a sequence of vectors x by applying a recurrence formula at every time step:

new state old state input vector at some time step

some functionwith parameters W

Page 8: ML with Tensorflow-new - GitHub Pages › ml › lec12.pdf · 2017-10-02 · Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 10 -16 8 Feb 2016 Recurrent Neural Network x RNN

Lecture 10 - 8 Feb 2016Fei-Fei Li & Andrej Karpathy & Justin JohnsonFei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 10 - 8 Feb 201616

Recurrent Neural Network

x

RNN

yWe can process a sequence of vectors x by applying a recurrence formula at every time step:

Notice: the same function and the same set of parameters are used at every time step.

Page 9: ML with Tensorflow-new - GitHub Pages › ml › lec12.pdf · 2017-10-02 · Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 10 -16 8 Feb 2016 Recurrent Neural Network x RNN

/HFWXUH����� ��)HE�����)HL�)HL�/L��$QGUHM�.DUSDWK\��-XVWLQ�-RKQVRQ)HL�)HL�/L��$QGUHM�.DUSDWK\��-XVWLQ�-RKQVRQ /HFWXUH����� ��)HE�������

�9DQLOOD��5HFXUUHQW�1HXUDO�1HWZRUN

[

511

\

7KH�VWDWH�FRQVLVWV�RI�D�VLQJOH�³KLGGHQ´�YHFWRU�K�

Page 10: ML with Tensorflow-new - GitHub Pages › ml › lec12.pdf · 2017-10-02 · Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 10 -16 8 Feb 2016 Recurrent Neural Network x RNN

http://colah.github.io/posts/2015-08-Understanding-LSTMs/

Page 11: ML with Tensorflow-new - GitHub Pages › ml › lec12.pdf · 2017-10-02 · Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 10 -16 8 Feb 2016 Recurrent Neural Network x RNN

Lecture 10 - 8 Feb 2016Fei-Fei Li & Andrej Karpathy & Justin JohnsonFei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 10 - 8 Feb 201618

Character-levellanguage modelexample

Vocabulary:[h,e,l,o]

Example trainingsequence:“hello”

x

RNN

y

Page 12: ML with Tensorflow-new - GitHub Pages › ml › lec12.pdf · 2017-10-02 · Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 10 -16 8 Feb 2016 Recurrent Neural Network x RNN

Lecture 10 - 8 Feb 2016Fei-Fei Li & Andrej Karpathy & Justin JohnsonFei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 10 - 8 Feb 201619

Character-levellanguage modelexample

Vocabulary:[h,e,l,o]

Example trainingsequence:“hello”

Page 13: ML with Tensorflow-new - GitHub Pages › ml › lec12.pdf · 2017-10-02 · Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 10 -16 8 Feb 2016 Recurrent Neural Network x RNN

Lecture 10 - 8 Feb 2016Fei-Fei Li & Andrej Karpathy & Justin JohnsonFei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 10 - 8 Feb 201620

Character-levellanguage modelexample

Vocabulary:[h,e,l,o]

Example trainingsequence:“hello”

Page 14: ML with Tensorflow-new - GitHub Pages › ml › lec12.pdf · 2017-10-02 · Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 10 -16 8 Feb 2016 Recurrent Neural Network x RNN

Lecture 10 - 8 Feb 2016Fei-Fei Li & Andrej Karpathy & Justin JohnsonFei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 10 - 8 Feb 201620

Character-levellanguage modelexample

Vocabulary:[h,e,l,o]

Example trainingsequence:“hello”

Page 15: ML with Tensorflow-new - GitHub Pages › ml › lec12.pdf · 2017-10-02 · Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 10 -16 8 Feb 2016 Recurrent Neural Network x RNN

Lecture 10 - 8 Feb 2016Fei-Fei Li & Andrej Karpathy & Justin JohnsonFei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 10 - 8 Feb 201620

Character-levellanguage modelexample

Vocabulary:[h,e,l,o]

Example trainingsequence:“hello”

Page 16: ML with Tensorflow-new - GitHub Pages › ml › lec12.pdf · 2017-10-02 · Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 10 -16 8 Feb 2016 Recurrent Neural Network x RNN

Lecture 10 - 8 Feb 2016Fei-Fei Li & Andrej Karpathy & Justin JohnsonFei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 10 - 8 Feb 201620

Character-levellanguage modelexample

Vocabulary:[h,e,l,o]

Example trainingsequence:“hello”

Page 17: ML with Tensorflow-new - GitHub Pages › ml › lec12.pdf · 2017-10-02 · Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 10 -16 8 Feb 2016 Recurrent Neural Network x RNN

Lecture 10 - 8 Feb 2016Fei-Fei Li & Andrej Karpathy & Justin JohnsonFei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 10 - 8 Feb 201621

Character-levellanguage modelexample

Vocabulary:[h,e,l,o]

Example trainingsequence:“hello”

Page 18: ML with Tensorflow-new - GitHub Pages › ml › lec12.pdf · 2017-10-02 · Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 10 -16 8 Feb 2016 Recurrent Neural Network x RNN

Lecture 10 - 8 Feb 2016Fei-Fei Li & Andrej Karpathy & Justin JohnsonFei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 10 - 8 Feb 201621

Character-levellanguage modelexample

Vocabulary:[h,e,l,o]

Example trainingsequence:“hello”

Page 19: ML with Tensorflow-new - GitHub Pages › ml › lec12.pdf · 2017-10-02 · Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 10 -16 8 Feb 2016 Recurrent Neural Network x RNN

RNN applications

• Language Modeling

• Speech Recognition

• Machine Translation

• Conversation Modeling/Question Answering

• Image/Video Captioning

• Image/Music/Dance Generation

http://jiwonkim.org/awesome-rnn/

https://github.com/TensorFlowKR/awesome_tensorflow_implementations

Page 20: ML with Tensorflow-new - GitHub Pages › ml › lec12.pdf · 2017-10-02 · Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 10 -16 8 Feb 2016 Recurrent Neural Network x RNN

Lecture 10 - 8 Feb 2016Fei-Fei Li & Andrej Karpathy & Justin JohnsonFei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 10 - 8 Feb 20166

Recurrent Networks offer a lot of flexibility:

Vanilla Neural Networks

Page 21: ML with Tensorflow-new - GitHub Pages › ml › lec12.pdf · 2017-10-02 · Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 10 -16 8 Feb 2016 Recurrent Neural Network x RNN

Lecture 10 - 8 Feb 2016Fei-Fei Li & Andrej Karpathy & Justin JohnsonFei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 10 - 8 Feb 20167

Recurrent Networks offer a lot of flexibility:

e.g. Image Captioningimage -> sequence of words

Page 22: ML with Tensorflow-new - GitHub Pages › ml › lec12.pdf · 2017-10-02 · Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 10 -16 8 Feb 2016 Recurrent Neural Network x RNN

Lecture 10 - 8 Feb 2016Fei-Fei Li & Andrej Karpathy & Justin JohnsonFei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 10 - 8 Feb 20168

Recurrent Networks offer a lot of flexibility:

e.g. Sentiment Classificationsequence of words -> sentiment

Page 23: ML with Tensorflow-new - GitHub Pages › ml › lec12.pdf · 2017-10-02 · Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 10 -16 8 Feb 2016 Recurrent Neural Network x RNN

Lecture 10 - 8 Feb 2016Fei-Fei Li & Andrej Karpathy & Justin JohnsonFei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 10 - 8 Feb 20169

Recurrent Networks offer a lot of flexibility:

e.g. Machine Translationseq of words -> seq of words

Page 24: ML with Tensorflow-new - GitHub Pages › ml › lec12.pdf · 2017-10-02 · Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 10 -16 8 Feb 2016 Recurrent Neural Network x RNN

Lecture 10 - 8 Feb 2016Fei-Fei Li & Andrej Karpathy & Justin JohnsonFei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 10 - 8 Feb 201610

Recurrent Networks offer a lot of flexibility:

e.g. Video classification on frame level

Page 25: ML with Tensorflow-new - GitHub Pages › ml › lec12.pdf · 2017-10-02 · Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 10 -16 8 Feb 2016 Recurrent Neural Network x RNN

Multi-Layer RNN

Page 26: ML with Tensorflow-new - GitHub Pages › ml › lec12.pdf · 2017-10-02 · Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 10 -16 8 Feb 2016 Recurrent Neural Network x RNN

Training RNNs is challenging

• Several advanced models- Long Short Term Memory (LSTM)- GRU by Cho et al. 2014

Page 27: ML with Tensorflow-new - GitHub Pages › ml › lec12.pdf · 2017-10-02 · Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 10 -16 8 Feb 2016 Recurrent Neural Network x RNN

Next

RNN in TensorFlow


Deep Visual-Semantic Alignments for Generating …...Deep Visual-Semantic Alignments for Generating Image Descriptions Andrej Karpathy Li Fei-Fei Department of Computer Science, Stanford

Deep Visual-Semantic Alignments for Generating …...Deep Visual-Semantic Alignments for Generating Image Descriptions Andrej Karpathy Li Fei-Fei Department of Computer Science, Stanford

Convolution Layer...Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 7 -13 27 Jan 2016 32 32 3 Convolution Layer 32x32x3 image 5x5x3 filter 1 number: the result of taking a dot

Convolution Layer...Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 7 -13 27 Jan 2016 32 32 3 Convolution Layer 32x32x3 image 5x5x3 filter 1 number: the result of taking a dot

Lecture 8: Spatial Localization and Detectioncs231n.stanford.edu/slides/2016/winter1516_lecture8.pdf · Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 8 - 1 1 Feb 2016 Lecture

Lecture 8: Spatial Localization and Detectioncs231n.stanford.edu/slides/2016/winter1516_lecture8.pdf · Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 8 - 1 1 Feb 2016 Lecture

Lecture 5: Training Neural Networks, Part I - GitHub Pages · Widrow and Hoff, ~1960: Adaline/Madaline * Original slides borrowed from Andrej Karpathy and Li Fei-Fei, Stanford cs231n

Lecture 5: Training Neural Networks, Part I - GitHub Pages · Widrow and Hoff, ~1960: Adaline/Madaline * Original slides borrowed from Andrej Karpathy and Li Fei-Fei, Stanford cs231n

Lecture 3: Loss functions and Optimizationvision.stanford.edu/.../slides/2016/winter1516_lecture3.pdfFei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 3 - 10 11 Jan 2016 Suppose:

Lecture 3: Loss functions and Optimizationvision.stanford.edu/.../slides/2016/winter1516_lecture3.pdfFei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 3 - 10 11 Jan 2016 Suppose:

Obj2Text: Generating Visually Descriptive Language from ...text. bitrary images has received considerable atten-tion (Vinyals et al.,2015;Karpathy and Fei-Fei, 2015;Mao et al.,2015).

Obj2Text: Generating Visually Descriptive Language from ...text. bitrary images has received considerable atten-tion (Vinyals et al.,2015;Karpathy and Fei-Fei, 2015;Mao et al.,2015).

Administrative - Stanford Universitycs231n.stanford.edu/slides/2016/winter1516_lecture15.pdfFei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 15 -Lecture 15 - 16 7 Mar 2016 And

Administrative - Stanford Universitycs231n.stanford.edu/slides/2016/winter1516_lecture15.pdfFei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 15 -Lecture 15 - 16 7 Mar 2016 And

Lecture 7: Convolutional Neural Networks - Stanford …cs231n.stanford.edu/slides/2016/winter1516_lecture7.pdfLecture 7: Convolutional Neural Networks Fei-Fei Li & Andrej Karpathy

Lecture 7: Convolutional Neural Networks - Stanford …cs231n.stanford.edu/slides/2016/winter1516_lecture7.pdfLecture 7: Convolutional Neural Networks Fei-Fei Li & Andrej Karpathy

Case Study: LeNet-5€¦ · Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 7 -60 27 Jan 2016 Case Study: LeNet-5 [LeCun et al., 1998] Conv filters were 5x5, applied at stride

Case Study: LeNet-5€¦ · Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 7 -60 27 Jan 2016 Case Study: LeNet-5 [LeCun et al., 1998] Conv filters were 5x5, applied at stride

Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 14 ...vision.stanford.edu/teaching/cs231n/slides/2016/... · Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 14 - Lecture

Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 14 ...vision.stanford.edu/teaching/cs231n/slides/2016/... · Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 14 - Lecture

Lecture 2: Image Classification pipeline - Artificial Intelligencevision.stanford.edu/teaching/cs231n/slides/2015/lecture2.pdf · Fei-Fei Li & Andrej Karpathy Lecture 2 - 32 7 Jan

Lecture 2: Image Classification pipeline - Artificial Intelligencevision.stanford.edu/teaching/cs231n/slides/2015/lecture2.pdf · Fei-Fei Li & Andrej Karpathy Lecture 2 - 32 7 Jan

Lecture 12 - Stanford University CS231n: Convolutional ...cs231n.stanford.edu/slides/2016/winter1516_lecture12.pdf · Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 12 - 22

Lecture 12 - Stanford University CS231n: Convolutional ...cs231n.stanford.edu/slides/2016/winter1516_lecture12.pdf · Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 12 - 22

FOIL it! Find One mismatch between Image and Language ......by using ‘neuraltalk’ (Karpathy and Fei-Fei, 2015) loss 11 Andrej Karpathy and Fei-Fei Li “Deep Visual-Semantic Alignments

FOIL it! Find One mismatch between Image and Language ......by using ‘neuraltalk’ (Karpathy and Fei-Fei, 2015) loss 11 Andrej Karpathy and Fei-Fei Li “Deep Visual-Semantic Alignments

DenseCap: Fully Convolutional Localization Networks for Dense … · DenseCap: Fully Convolutional Localization Networks for Dense Captioning Justin Johnson Andrej Karpathy Li Fei-Fei

DenseCap: Fully Convolutional Localization Networks for Dense … · DenseCap: Fully Convolutional Localization Networks for Dense Captioning Justin Johnson Andrej Karpathy Li Fei-Fei

DenseCap: Fully Convolutional Localization Networks for ... · DenseCap: Fully Convolutional Localization Networks for Dense Captioning Justin Johnson Andrej Karpathy Li Fei-Fei Department

DenseCap: Fully Convolutional Localization Networks for ... · DenseCap: Fully Convolutional Localization Networks for Dense Captioning Justin Johnson Andrej Karpathy Li Fei-Fei Department

VISUALIZING AND UNDERSTANDING RECURRENT NEURAL NETWORKS Presented By: Collin Watts Wrritten By: Andrej Karpathy, Justin Johnson, Li Fei-fei.

VISUALIZING AND UNDERSTANDING RECURRENT NEURAL NETWORKS Presented By: Collin Watts Wrritten By: Andrej Karpathy, Justin Johnson, Li Fei-fei.

ML with Tensorflow-new - GitHub Pageshunkim.github.io/ml/lec12.pdf · e.g. Machine Translation seq of words -> seq of words. Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture

ML with Tensorflow-new - GitHub Pageshunkim.github.io/ml/lec12.pdf · e.g. Machine Translation seq of words -> seq of words. Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture

DenseCap: Fully Convolutional ... - cv-foundation.org · DenseCap: Fully Convolutional Localization Networks for Dense Captioning Justin Johnson∗ Andrej Karpathy∗ Li Fei-Fei Department

DenseCap: Fully Convolutional ... - cv-foundation.org · DenseCap: Fully Convolutional Localization Networks for Dense Captioning Justin Johnson∗ Andrej Karpathy∗ Li Fei-Fei Department

Lecture 11 - Stanford Universitycs231n.stanford.edu/slides/2016/winter1516_lecture11.pdf · Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 11 - 47 17 Feb 2016 The power of

Lecture 11 - Stanford Universitycs231n.stanford.edu/slides/2016/winter1516_lecture11.pdf · Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 11 - 47 17 Feb 2016 The power of

Lecture 6: Training Neural Networks, Part 2vision.stanford.edu/teaching/cs231n/slides/2016/... · Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 6 - 3 25 Jan 2016 Mini-batch

Lecture 6: Training Neural Networks, Part 2vision.stanford.edu/teaching/cs231n/slides/2016/... · Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 6 - 3 25 Jan 2016 Mini-batch