ML with Tensorflow lab - GitHub Pageshunkim.github.io/ml/lab11.pdf · 2017. 10. 2. · Fei-Fei Li &...

22
Lab 11 CNN Sung Kim <[email protected]> http://hunkim.github.io/ml/

Transcript of ML with Tensorflow lab - GitHub Pageshunkim.github.io/ml/lab11.pdf · 2017. 10. 2. · Fei-Fei Li &...

Page 1: ML with Tensorflow lab - GitHub Pageshunkim.github.io/ml/lab11.pdf · 2017. 10. 2. · Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 7 -55 27 Jan 2016 1 1 2 4 5 6 7 8 3 2

Lab 11 CNN

Sung Kim <[email protected]>http://hunkim.github.io/ml/

Page 2: ML with Tensorflow lab - GitHub Pageshunkim.github.io/ml/lab11.pdf · 2017. 10. 2. · Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 7 -55 27 Jan 2016 1 1 2 4 5 6 7 8 3 2

https://github.com/nlintz/TensorFlow-Tutorials

Page 3: ML with Tensorflow lab - GitHub Pageshunkim.github.io/ml/lab11.pdf · 2017. 10. 2. · Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 7 -55 27 Jan 2016 1 1 2 4 5 6 7 8 3 2

CNN

http://parse.ele.tue.nl/cluster/2/CNNArchitecture.jpg

Page 4: ML with Tensorflow lab - GitHub Pageshunkim.github.io/ml/lab11.pdf · 2017. 10. 2. · Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 7 -55 27 Jan 2016 1 1 2 4 5 6 7 8 3 2

MNIST 28x28x1 image

32 filters (3x3x1)

Convolutional layers

Page 5: ML with Tensorflow lab - GitHub Pageshunkim.github.io/ml/lab11.pdf · 2017. 10. 2. · Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 7 -55 27 Jan 2016 1 1 2 4 5 6 7 8 3 2

28x28x1 image

32 filters (3x3x1)w=tf.Variable(tf.random_normal([3,3,1,32], stddev=0.01))

Convolutional layers

Page 6: ML with Tensorflow lab - GitHub Pageshunkim.github.io/ml/lab11.pdf · 2017. 10. 2. · Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 7 -55 27 Jan 2016 1 1 2 4 5 6 7 8 3 2

28x28x1 image

32 filters (3x3x1)w=tf.Variable(tf.random_normal([3,3,1,32], stddev=0.01))

Convolutional layers

Page 7: ML with Tensorflow lab - GitHub Pageshunkim.github.io/ml/lab11.pdf · 2017. 10. 2. · Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 7 -55 27 Jan 2016 1 1 2 4 5 6 7 8 3 2

28x28x1 image

32 filters (3x3x1)activation maps

(?, ?, 32)

Convolutional layers

Page 8: ML with Tensorflow lab - GitHub Pageshunkim.github.io/ml/lab11.pdf · 2017. 10. 2. · Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 7 -55 27 Jan 2016 1 1 2 4 5 6 7 8 3 2

ReLU

Page 9: ML with Tensorflow lab - GitHub Pageshunkim.github.io/ml/lab11.pdf · 2017. 10. 2. · Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 7 -55 27 Jan 2016 1 1 2 4 5 6 7 8 3 2

Pooling layer (sampling)

conv layer

l1 = tf.nn.max_pool(c1, ksize=[1, 2, 2, 1], strides=[1, 2, 2, 1], padding='SAME')

Page 10: ML with Tensorflow lab - GitHub Pageshunkim.github.io/ml/lab11.pdf · 2017. 10. 2. · Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 7 -55 27 Jan 2016 1 1 2 4 5 6 7 8 3 2

Lecture 7 - 27 Jan 2016Fei-Fei Li & Andrej Karpathy & Justin JohnsonFei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 7 - 27 Jan 201655

1 1 2 4

5 6 7 8

3 2 1 0

1 2 3 4

Single depth slice

x

y

max pool with 2x2 filters and stride 2 6 8

3 4

MAX POOLING

Page 11: ML with Tensorflow lab - GitHub Pageshunkim.github.io/ml/lab11.pdf · 2017. 10. 2. · Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 7 -55 27 Jan 2016 1 1 2 4 5 6 7 8 3 2

Pooling layer (sampling)

conv layer

l1 = tf.nn.max_pool(c1, ksize=[1, 2, 2, 1], strides=[1, 2, 2, 1], padding='SAME')

28x28x32

? x ? x 32

Page 12: ML with Tensorflow lab - GitHub Pageshunkim.github.io/ml/lab11.pdf · 2017. 10. 2. · Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 7 -55 27 Jan 2016 1 1 2 4 5 6 7 8 3 2

Tensor("MaxPool:0", shape=(?, 14, 14, 32), dtype=float32)

Shape not sure? Print tensor

Tensor("Conv2D:0", shape=(?, 28, 28, 32), dtype=float32)

Page 13: ML with Tensorflow lab - GitHub Pageshunkim.github.io/ml/lab11.pdf · 2017. 10. 2. · Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 7 -55 27 Jan 2016 1 1 2 4 5 6 7 8 3 2

X = trX.reshape(-1, 28, 28, 1)

Page 14: ML with Tensorflow lab - GitHub Pageshunkim.github.io/ml/lab11.pdf · 2017. 10. 2. · Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 7 -55 27 Jan 2016 1 1 2 4 5 6 7 8 3 2

X = trX.reshape(-1, 28, 28, 1)

dropout

Page 15: ML with Tensorflow lab - GitHub Pageshunkim.github.io/ml/lab11.pdf · 2017. 10. 2. · Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 7 -55 27 Jan 2016 1 1 2 4 5 6 7 8 3 2

CNN

http://parse.ele.tue.nl/cluster/2/CNNArchitecture.jpg

Page 16: ML with Tensorflow lab - GitHub Pageshunkim.github.io/ml/lab11.pdf · 2017. 10. 2. · Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 7 -55 27 Jan 2016 1 1 2 4 5 6 7 8 3 2

Fully connected net

Page 17: ML with Tensorflow lab - GitHub Pageshunkim.github.io/ml/lab11.pdf · 2017. 10. 2. · Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 7 -55 27 Jan 2016 1 1 2 4 5 6 7 8 3 2

cost and optimization

https://www.tensorflow.org/versions/r0.8/api_docs/python/train.html#RMSPropOptimizer

Page 18: ML with Tensorflow lab - GitHub Pageshunkim.github.io/ml/lab11.pdf · 2017. 10. 2. · Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 7 -55 27 Jan 2016 1 1 2 4 5 6 7 8 3 2

Other TF optimizers

https://www.tensorflow.org/versions/r0.8/api_docs/python/train.html#RMSPropOptimizer

Page 19: ML with Tensorflow lab - GitHub Pageshunkim.github.io/ml/lab11.pdf · 2017. 10. 2. · Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 7 -55 27 Jan 2016 1 1 2 4 5 6 7 8 3 2

Train and testing

Page 20: ML with Tensorflow lab - GitHub Pageshunkim.github.io/ml/lab11.pdf · 2017. 10. 2. · Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 7 -55 27 Jan 2016 1 1 2 4 5 6 7 8 3 2

Train and testing 0 128

128 256 256 384 384 512

Page 21: ML with Tensorflow lab - GitHub Pageshunkim.github.io/ml/lab11.pdf · 2017. 10. 2. · Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 7 -55 27 Jan 2016 1 1 2 4 5 6 7 8 3 2

Train and testing

Page 22: ML with Tensorflow lab - GitHub Pageshunkim.github.io/ml/lab11.pdf · 2017. 10. 2. · Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 7 -55 27 Jan 2016 1 1 2 4 5 6 7 8 3 2

https://github.com/nlintz/TensorFlow-Tutorials


Lecture 12 - Stanford Universityvision.stanford.edu/.../cs231n/slides/2016/winter1516_lecture12.pdf · Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 12 ... Fei-Fei Li & Andrej

Lecture 12 - Stanford Universityvision.stanford.edu/.../cs231n/slides/2016/winter1516_lecture12.pdf · Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 12 ... Fei-Fei Li & Andrej

Li- Jia Li, Richard Socher , Li Fei-Fei

Li- Jia Li, Richard Socher , Li Fei-Fei

FEI - Dominio FEI 01 en-us

FEI - Dominio FEI 01 en-us

Fei-Fei Li & Justin Johnson & Serena Yeung Lecture 2 ...cs231n.stanford.edu/slides/2017/cs231n_2017_lecture2.pdf · All pixels change when ... Fei-Fei Li & Justin Johnson & Serena

Fei-Fei Li & Justin Johnson & Serena Yeung Lecture 2 ...cs231n.stanford.edu/slides/2017/cs231n_2017_lecture2.pdf · All pixels change when ... Fei-Fei Li & Justin Johnson & Serena

ML with Tensorflow - GitHub Pageshunkim.github.io/ml/lec5.pdf · Linear regression •We know Y is 0 or ...

ML with Tensorflow - GitHub Pageshunkim.github.io/ml/lec5.pdf · Linear regression •We know Y is 0 or ...

Fei-Fei Li & Justin Johnson & Serena Yeung Lecture 4 ...cs231n.stanford.edu/slides/2018/cs231n_2018_lecture04.pdf · Fei-Fei Li & Justin Johnson & Serena Yeung Lecture 4 - April 12,

Fei-Fei Li & Justin Johnson & Serena Yeung Lecture 4 ...cs231n.stanford.edu/slides/2018/cs231n_2018_lecture04.pdf · Fei-Fei Li & Justin Johnson & Serena Yeung Lecture 4 - April 12,

Lab 6-1: Q Network - GitHub Pageshunkim.github.io/ml/RL/rl06-l1.pdfPlaying Atari with Deep Reinforcement Learning - University of Toronto by V Mnih et al. Algorithm Playing Atari with

Lab 6-1: Q Network - GitHub Pageshunkim.github.io/ml/RL/rl06-l1.pdfPlaying Atari with Deep Reinforcement Learning - University of Toronto by V Mnih et al. Algorithm Playing Atari with

Course Summary LING 572 Fei Xia 03/06/07. Outline Problem description General approach ML algorithms Important concepts Assignments What’s next?

Course Summary LING 572 Fei Xia 03/06/07. Outline Problem description General approach ML algorithms Important concepts Assignments What’s next?

Lecture 7: DQN - GitHub Pageshunkim.github.io/ml/RL/rl07.pdf · Lecture 7: DQN Reinforcement Learning with TensorFlow&OpenAI Gym Sung Kim Q-function Approximation:

Lecture 7: DQN - GitHub Pageshunkim.github.io/ml/RL/rl07.pdf · Lecture 7: DQN Reinforcement Learning with TensorFlow&OpenAI Gym Sung Kim Q-function Approximation:

Lab 2: Playing OpenAI Gym Games - GitHub Pageshunkim.github.io/ml/RL/rl-l02.pdf · Lab 2: Playing OpenAI Gym Games Reinforcement Learning with TensorFlow&OpenAI Gym Sung Kim

Lab 2: Playing OpenAI Gym Games - GitHub Pageshunkim.github.io/ml/RL/rl-l02.pdf · Lab 2: Playing OpenAI Gym Games Reinforcement Learning with TensorFlow&OpenAI Gym Sung Kim

Descriptors II - courses.cs.washington.edu• Fei-Fei & Perona, 2005 • Interest point detector • Csurka et al. 2004 • Fei-Fei & Perona, 2005 • Sivic et al. 2005 • Other methods

Descriptors II - courses.cs.washington.edu• Fei-Fei & Perona, 2005 • Interest point detector • Csurka et al. 2004 • Fei-Fei & Perona, 2005 • Sivic et al. 2005 • Other methods

FEI -8 FEI 0.5 - 10 14.0 16.0 8.0 10.0 [RoHS2 (AWG) (22-20) · 14.0 16.0 8.0 10.0 [rohs2 (awg) (22-20) 5- (20-19) (18) (16) fei o. fei fei fei fei 75-10 16.0 14.0 16.0 14.0 16.0 10.0

FEI -8 FEI 0.5 - 10 14.0 16.0 8.0 10.0 [RoHS2 (AWG) (22-20) · 14.0 16.0 8.0 10.0 [rohs2 (awg) (22-20) 5- (20-19) (18) (16) fei o. fei fei fei fei 75-10 16.0 14.0 16.0 14.0 16.0 10.0

ML with Tensorflow-new - GitHub Pageshunkim.github.io/ml/lec11.pdf · 2017-10-02 · Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 7 -68 27 Jan 2016 Case Study: AlexNet [Krizhevsky

ML with Tensorflow-new - GitHub Pageshunkim.github.io/ml/lec11.pdf · 2017-10-02 · Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 7 -68 27 Jan 2016 Case Study: AlexNet [Krizhevsky

Lecture3 xing fei-fei

Lecture3 xing fei-fei

Lab 6-2: Q Network for Cart Pole - GitHub Pageshunkim.github.io/ml/RL/rl06-l2.pdf · 2017-10-02 · import numpy as np import tensor flow as tf import gym gym. make( CartPole—vØ'

Lab 6-2: Q Network for Cart Pole - GitHub Pageshunkim.github.io/ml/RL/rl06-l2.pdf · 2017-10-02 · import numpy as np import tensor flow as tf import gym gym. make( CartPole—vØ'

Lecture7 xing fei-fei

Lecture7 xing fei-fei

Fituici FEI

Fituici FEI

ML with Tensorflow - GitHub Pageshunkim.github.io/ml/lec0.pdf · Goals •Basic understanding of machine learning algorithms Linear regression, Logistic regression (classification)-Neural

ML with Tensorflow - GitHub Pageshunkim.github.io/ml/lec0.pdf · Goals •Basic understanding of machine learning algorithms Linear regression, Logistic regression (classification)-Neural

Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 14 ...vision.stanford.edu/teaching/cs231n/slides/2016/... · Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 14 - Lecture

Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 14 ...vision.stanford.edu/teaching/cs231n/slides/2016/... · Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 14 - Lecture

Lab 7: DQN 1 (NIPS 2013) - GitHub Pageshunkim.github.io/ml/RL/rl07-l1.pdf · Lab 7: DQN 1 (NIPS 2013) Reinforcement Learning with TensorFlow&OpenAI Gym Sung Kim

Lab 7: DQN 1 (NIPS 2013) - GitHub Pageshunkim.github.io/ml/RL/rl07-l1.pdf · Lab 7: DQN 1 (NIPS 2013) Reinforcement Learning with TensorFlow&OpenAI Gym Sung Kim