Convolutional Neural Networkyfwang/courses/cs181b/notes/CNN.pdf · PR , ANN, & ML Signal Generation...
Transcript of Convolutional Neural Networkyfwang/courses/cs181b/notes/CNN.pdf · PR , ANN, & ML Signal Generation...
![Page 1: Convolutional Neural Networkyfwang/courses/cs181b/notes/CNN.pdf · PR , ANN, & ML Signal Generation •Resting potential •Charge difference across neuron membrane approximately](https://reader035.fdocuments.net/reader035/viewer/2022081617/6025025fa12f8b7d723cae19/html5/thumbnails/1.jpg)
Convolutional Neural Network
Can the network be simplified by considering the properties of images?
![Page 2: Convolutional Neural Networkyfwang/courses/cs181b/notes/CNN.pdf · PR , ANN, & ML Signal Generation •Resting potential •Charge difference across neuron membrane approximately](https://reader035.fdocuments.net/reader035/viewer/2022081617/6025025fa12f8b7d723cae19/html5/thumbnails/2.jpg)
Artificial Neural Networks
• Connectionist, PDP, etc. models
• A biologically-inspired approach for
• intelligent computing machines
• massive parallelism
• distributed computing
• learning, generalization, adaptivity
• Tolerant of fault, uncertainty, imprecise info
![Page 3: Convolutional Neural Networkyfwang/courses/cs181b/notes/CNN.pdf · PR , ANN, & ML Signal Generation •Resting potential •Charge difference across neuron membrane approximately](https://reader035.fdocuments.net/reader035/viewer/2022081617/6025025fa12f8b7d723cae19/html5/thumbnails/3.jpg)
Von Neumann computer
Biological neural systems
Processor complex, high speed, few
simple, low speed, many
Memory separate from processor,
non-content addressable
integrated into processor, content addressable
Computing centralized, sequential stored programs
distributed, parallel self-learning
Reliability vulnerable fault tolerant
Compared to Von Neumann
![Page 4: Convolutional Neural Networkyfwang/courses/cs181b/notes/CNN.pdf · PR , ANN, & ML Signal Generation •Resting potential •Charge difference across neuron membrane approximately](https://reader035.fdocuments.net/reader035/viewer/2022081617/6025025fa12f8b7d723cae19/html5/thumbnails/4.jpg)
PR , ANN, & ML
Biological Neural Networks• soma (cell body)
• dendrites (receivers)
• axon (transmitters)
• synapses (connection points, axon-soma, axon-dendrite, axon-axon)
• Chemicals (neurotransmitters)
• neurons
• each makes about connections
• with an operating speed of a few milliseconds
• one-hundred-step rule
1011
10 103 4~
Axon hillock
![Page 5: Convolutional Neural Networkyfwang/courses/cs181b/notes/CNN.pdf · PR , ANN, & ML Signal Generation •Resting potential •Charge difference across neuron membrane approximately](https://reader035.fdocuments.net/reader035/viewer/2022081617/6025025fa12f8b7d723cae19/html5/thumbnails/5.jpg)
PR , ANN, & ML
Signal Generation• Resting potential
• Charge difference across neuron membrane approximately –70mV
• Graded potential • Stimulus across synapses of post-synaptic neuron
• Action potential• If accumulation of graded potential across neuron
membrane over a short period of time is higher than ~15mV, action potential is generated and propagated across axon
• Same form and amplitude regardless of stimulus, signal by frequency rather than amplitude
![Page 6: Convolutional Neural Networkyfwang/courses/cs181b/notes/CNN.pdf · PR , ANN, & ML Signal Generation •Resting potential •Charge difference across neuron membrane approximately](https://reader035.fdocuments.net/reader035/viewer/2022081617/6025025fa12f8b7d723cae19/html5/thumbnails/6.jpg)
PR , ANN, & ML
Signal Generation
15mV
![Page 7: Convolutional Neural Networkyfwang/courses/cs181b/notes/CNN.pdf · PR , ANN, & ML Signal Generation •Resting potential •Charge difference across neuron membrane approximately](https://reader035.fdocuments.net/reader035/viewer/2022081617/6025025fa12f8b7d723cae19/html5/thumbnails/7.jpg)
PR , ANN, & ML
Computational Neuron Model (McCulloch and Pitts)
x2
xn
x1 w1
w2
wn
1
( )n
i i
i
O w x b
b11),tanh()(
10,1
1)(
xx
ex
x
- time dependency?
- frequency response?
![Page 8: Convolutional Neural Networkyfwang/courses/cs181b/notes/CNN.pdf · PR , ANN, & ML Signal Generation •Resting potential •Charge difference across neuron membrane approximately](https://reader035.fdocuments.net/reader035/viewer/2022081617/6025025fa12f8b7d723cae19/html5/thumbnails/8.jpg)
What does a neuron do mathematically?
• W and b determine a “basis” function
• Unlike Fourier bases, these bases are learned to fit the data
• Pattern finder or classifier
• W and b determine a partitioning hyperplane
• They separate high-dimensional feature spaces into regions
• Pattern classifier
• Again, the partitioning hyperplanes are learned
![Page 9: Convolutional Neural Networkyfwang/courses/cs181b/notes/CNN.pdf · PR , ANN, & ML Signal Generation •Resting potential •Charge difference across neuron membrane approximately](https://reader035.fdocuments.net/reader035/viewer/2022081617/6025025fa12f8b7d723cae19/html5/thumbnails/9.jpg)
Why CNN for Image
• Some patterns are much smaller than the whole image
A neuron does not have to see the whole image to discover the pattern.
“beak” detector
Connecting to small region with less parameters
![Page 10: Convolutional Neural Networkyfwang/courses/cs181b/notes/CNN.pdf · PR , ANN, & ML Signal Generation •Resting potential •Charge difference across neuron membrane approximately](https://reader035.fdocuments.net/reader035/viewer/2022081617/6025025fa12f8b7d723cae19/html5/thumbnails/10.jpg)
Why CNN for Image
• The same patterns appear in different regions.
“upper-left beak” detector
“middle beak”detector
They can use the same set of parameters.
Do almost the same thing
![Page 11: Convolutional Neural Networkyfwang/courses/cs181b/notes/CNN.pdf · PR , ANN, & ML Signal Generation •Resting potential •Charge difference across neuron membrane approximately](https://reader035.fdocuments.net/reader035/viewer/2022081617/6025025fa12f8b7d723cae19/html5/thumbnails/11.jpg)
Why CNN for Image
• Subsampling the pixels will not change the object
subsampling
bird
bird
We can subsample the pixels to make image smaller
Less parameters for the network to process the image
![Page 12: Convolutional Neural Networkyfwang/courses/cs181b/notes/CNN.pdf · PR , ANN, & ML Signal Generation •Resting potential •Charge difference across neuron membrane approximately](https://reader035.fdocuments.net/reader035/viewer/2022081617/6025025fa12f8b7d723cae19/html5/thumbnails/12.jpg)
The whole CNN
Fully Connected Feedforward network
cat dog ……Convolution
Max Pooling
Convolution
Max Pooling
Flatten
Can repeat many times
Activation
Activation
![Page 13: Convolutional Neural Networkyfwang/courses/cs181b/notes/CNN.pdf · PR , ANN, & ML Signal Generation •Resting potential •Charge difference across neuron membrane approximately](https://reader035.fdocuments.net/reader035/viewer/2022081617/6025025fa12f8b7d723cae19/html5/thumbnails/13.jpg)
The whole CNN
Convolution
Max Pooling
Convolution
Max Pooling
Flatten
Can repeat many times
Some patterns are much smaller than the whole image
The same patterns appear in different regions.
Subsampling the pixels will not change the object
Property 1
Property 2
Property 3
![Page 14: Convolutional Neural Networkyfwang/courses/cs181b/notes/CNN.pdf · PR , ANN, & ML Signal Generation •Resting potential •Charge difference across neuron membrane approximately](https://reader035.fdocuments.net/reader035/viewer/2022081617/6025025fa12f8b7d723cae19/html5/thumbnails/14.jpg)
The whole CNN
Fully Connected Feedforward network
cat dog ……Convolution
Max Pooling
Convolution
Max Pooling
Flatten
Can repeat many times
![Page 15: Convolutional Neural Networkyfwang/courses/cs181b/notes/CNN.pdf · PR , ANN, & ML Signal Generation •Resting potential •Charge difference across neuron membrane approximately](https://reader035.fdocuments.net/reader035/viewer/2022081617/6025025fa12f8b7d723cae19/html5/thumbnails/15.jpg)
CNN – Convolution
1 0 0 0 0 1
0 1 0 0 1 0
0 0 1 1 0 0
1 0 0 0 1 0
0 1 0 0 1 0
0 0 1 0 1 0
6 x 6 image
1 -1 -1
-1 1 -1
-1 -1 1
Filter 1
-1 1 -1
-1 1 -1
-1 1 -1
Filter 2
……
Those are the network parameters to be learned.
Matrix
Matrix
Each filter detects a small pattern (3 x 3).
Property 1
![Page 16: Convolutional Neural Networkyfwang/courses/cs181b/notes/CNN.pdf · PR , ANN, & ML Signal Generation •Resting potential •Charge difference across neuron membrane approximately](https://reader035.fdocuments.net/reader035/viewer/2022081617/6025025fa12f8b7d723cae19/html5/thumbnails/16.jpg)
CNN – Convolution
1 0 0 0 0 1
0 1 0 0 1 0
0 0 1 1 0 0
1 0 0 0 1 0
0 1 0 0 1 0
0 0 1 0 1 0
6 x 6 image
1 -1 -1
-1 1 -1
-1 -1 1
Filter 1
3 -1
stride=1
![Page 17: Convolutional Neural Networkyfwang/courses/cs181b/notes/CNN.pdf · PR , ANN, & ML Signal Generation •Resting potential •Charge difference across neuron membrane approximately](https://reader035.fdocuments.net/reader035/viewer/2022081617/6025025fa12f8b7d723cae19/html5/thumbnails/17.jpg)
CNN – Convolution
1 0 0 0 0 1
0 1 0 0 1 0
0 0 1 1 0 0
1 0 0 0 1 0
0 1 0 0 1 0
0 0 1 0 1 0
6 x 6 image
1 -1 -1
-1 1 -1
-1 -1 1
Filter 1
3 -3
If stride=2
We set stride=1 below
![Page 18: Convolutional Neural Networkyfwang/courses/cs181b/notes/CNN.pdf · PR , ANN, & ML Signal Generation •Resting potential •Charge difference across neuron membrane approximately](https://reader035.fdocuments.net/reader035/viewer/2022081617/6025025fa12f8b7d723cae19/html5/thumbnails/18.jpg)
CNN – Convolution
1 0 0 0 0 1
0 1 0 0 1 0
0 0 1 1 0 0
1 0 0 0 1 0
0 1 0 0 1 0
0 0 1 0 1 0
6 x 6 image
1 -1 -1
-1 1 -1
-1 -1 1
Filter 1
3 -1 -3 -1
-3 1 0 -3
-3 -3 0 1
3 -2 -2 -1
stride=1
Property 2
![Page 19: Convolutional Neural Networkyfwang/courses/cs181b/notes/CNN.pdf · PR , ANN, & ML Signal Generation •Resting potential •Charge difference across neuron membrane approximately](https://reader035.fdocuments.net/reader035/viewer/2022081617/6025025fa12f8b7d723cae19/html5/thumbnails/19.jpg)
CNN – Convolution
1 0 0 0 0 1
0 1 0 0 1 0
0 0 1 1 0 0
1 0 0 0 1 0
0 1 0 0 1 0
0 0 1 0 1 0
6 x 6 image
3 -1 -3 -1
-3 1 0 -3
-3 -3 0 1
3 -2 -2 -1
-1 1 -1
-1 1 -1
-1 1 -1
Filter 2
-1 -1 -1 -1
-1 -1 -2 1
-1 -1 -2 1
-1 0 -4 3
Do the same process for every filter
stride=1
4 x 4 image
FeatureMap
![Page 20: Convolutional Neural Networkyfwang/courses/cs181b/notes/CNN.pdf · PR , ANN, & ML Signal Generation •Resting potential •Charge difference across neuron membrane approximately](https://reader035.fdocuments.net/reader035/viewer/2022081617/6025025fa12f8b7d723cae19/html5/thumbnails/20.jpg)
CNN – Colorful image
1 0 0 0 0 1
0 1 0 0 1 0
0 0 1 1 0 0
1 0 0 0 1 0
0 1 0 0 1 0
0 0 1 0 1 0
1 0 0 0 0 1
0 1 0 0 1 0
0 0 1 1 0 0
1 0 0 0 1 0
0 1 0 0 1 0
0 0 1 0 1 0
1 0 0 0 0 1
0 1 0 0 1 0
0 0 1 1 0 0
1 0 0 0 1 0
0 1 0 0 1 0
0 0 1 0 1 0
1 -1 -1
-1 1 -1
-1 -1 1Filter 1
-1 1 -1
-1 1 -1
-1 1 -1Filter 2
1 -1 -1
-1 1 -1
-1 -1 1
1 -1 -1
-1 1 -1
-1 -1 1
-1 1 -1
-1 1 -1
-1 1 -1
-1 1 -1
-1 1 -1
-1 1 -1Colorful image
![Page 21: Convolutional Neural Networkyfwang/courses/cs181b/notes/CNN.pdf · PR , ANN, & ML Signal Generation •Resting potential •Charge difference across neuron membrane approximately](https://reader035.fdocuments.net/reader035/viewer/2022081617/6025025fa12f8b7d723cae19/html5/thumbnails/21.jpg)
1 0 0 0 0 1
0 1 0 0 1 0
0 0 1 1 0 0
1 0 0 0 1 0
0 1 0 0 1 0
0 0 1 0 1 0
image
convolution
-1 1 -1
-1 1 -1
-1 1 -1
1 -1 -1
-1 1 -1
-1 -1 1
1x
2x
……
36x
……
1 0 0 0 0 1
0 1 0 0 1 0
0 0 1 1 0 0
1 0 0 0 1 0
0 1 0 0 1 0
0 0 1 0 1 0
Convolution v.s. Fully Connected
Fully-connected
![Page 22: Convolutional Neural Networkyfwang/courses/cs181b/notes/CNN.pdf · PR , ANN, & ML Signal Generation •Resting potential •Charge difference across neuron membrane approximately](https://reader035.fdocuments.net/reader035/viewer/2022081617/6025025fa12f8b7d723cae19/html5/thumbnails/22.jpg)
1 0 0 0 0 1
0 1 0 0 1 0
0 0 1 1 0 0
1 0 0 0 1 0
0 1 0 0 1 0
0 0 1 0 1 0
6 x 6 image
1 -1 -1
-1 1 -1
-1 -1 1
Filter 11:
2:
3:
…
7:
8:
9:…
13:
14:
15:
…
Only connect to 9 input, not fully connected
4:
10:
16:
1
0
0
0
0
1
0
0
0
0
1
1
3
Less parameters!
![Page 23: Convolutional Neural Networkyfwang/courses/cs181b/notes/CNN.pdf · PR , ANN, & ML Signal Generation •Resting potential •Charge difference across neuron membrane approximately](https://reader035.fdocuments.net/reader035/viewer/2022081617/6025025fa12f8b7d723cae19/html5/thumbnails/23.jpg)
1 0 0 0 0 1
0 1 0 0 1 0
0 0 1 1 0 0
1 0 0 0 1 0
0 1 0 0 1 0
0 0 1 0 1 0
1 -1 -1
-1 1 -1
-1 -1 1
Filter 1
1:
2:
3:
…
7:
8:
9:…
13:
14:
15:
…
4:
10:
16:
1
0
0
0
0
1
0
0
0
0
1
1
3
-1
Shared weights
6 x 6 image
Less parameters!
Even less parameters!
![Page 24: Convolutional Neural Networkyfwang/courses/cs181b/notes/CNN.pdf · PR , ANN, & ML Signal Generation •Resting potential •Charge difference across neuron membrane approximately](https://reader035.fdocuments.net/reader035/viewer/2022081617/6025025fa12f8b7d723cae19/html5/thumbnails/24.jpg)
The whole CNN
Fully Connected Feedforward network
cat dog ……Convolution
Max Pooling
Convolution
Max Pooling
Flatten
Can repeat many times
![Page 25: Convolutional Neural Networkyfwang/courses/cs181b/notes/CNN.pdf · PR , ANN, & ML Signal Generation •Resting potential •Charge difference across neuron membrane approximately](https://reader035.fdocuments.net/reader035/viewer/2022081617/6025025fa12f8b7d723cae19/html5/thumbnails/25.jpg)
CNN – Max Pooling
3 -1 -3 -1
-3 1 0 -3
-3 -3 0 1
3 -2 -2 -1
-1 1 -1
-1 1 -1
-1 1 -1
Filter 2
-1 -1 -1 -1
-1 -1 -2 1
-1 -1 -2 1
-1 0 -4 3
1 -1 -1
-1 1 -1
-1 -1 1
Filter 1
![Page 26: Convolutional Neural Networkyfwang/courses/cs181b/notes/CNN.pdf · PR , ANN, & ML Signal Generation •Resting potential •Charge difference across neuron membrane approximately](https://reader035.fdocuments.net/reader035/viewer/2022081617/6025025fa12f8b7d723cae19/html5/thumbnails/26.jpg)
CNN – Max Pooling
1 0 0 0 0 1
0 1 0 0 1 0
0 0 1 1 0 0
1 0 0 0 1 0
0 1 0 0 1 0
0 0 1 0 1 0
6 x 6 image
3 0
13
-1 1
30
2 x 2 image
Each filter is a channel
New image but smaller
Conv
MaxPooling
![Page 27: Convolutional Neural Networkyfwang/courses/cs181b/notes/CNN.pdf · PR , ANN, & ML Signal Generation •Resting potential •Charge difference across neuron membrane approximately](https://reader035.fdocuments.net/reader035/viewer/2022081617/6025025fa12f8b7d723cae19/html5/thumbnails/27.jpg)
The whole CNN
Convolution
Max Pooling
Convolution
Max Pooling
Can repeat many times
A new image
The number of the channel is the number of filters
Smaller than the original image
3 0
13
-1 1
30Activation
Activation
![Page 28: Convolutional Neural Networkyfwang/courses/cs181b/notes/CNN.pdf · PR , ANN, & ML Signal Generation •Resting potential •Charge difference across neuron membrane approximately](https://reader035.fdocuments.net/reader035/viewer/2022081617/6025025fa12f8b7d723cae19/html5/thumbnails/28.jpg)
The whole CNN
Fully Connected Feedforward network
cat dog ……Convolution
Max Pooling
Convolution
Max Pooling
Flatten
A new image
A new image
![Page 29: Convolutional Neural Networkyfwang/courses/cs181b/notes/CNN.pdf · PR , ANN, & ML Signal Generation •Resting potential •Charge difference across neuron membrane approximately](https://reader035.fdocuments.net/reader035/viewer/2022081617/6025025fa12f8b7d723cae19/html5/thumbnails/29.jpg)
Flatten
3 0
13
-1 1
30 Flatten
3
0
1
3
-1
1
0
3
Fully Connected Feedforward network
![Page 30: Convolutional Neural Networkyfwang/courses/cs181b/notes/CNN.pdf · PR , ANN, & ML Signal Generation •Resting potential •Charge difference across neuron membrane approximately](https://reader035.fdocuments.net/reader035/viewer/2022081617/6025025fa12f8b7d723cae19/html5/thumbnails/30.jpg)
CNN Demo - Mnist
• Hand-written digits
• 60,000 training samples and 10,000 test samples
• 28x28 binary images
![Page 31: Convolutional Neural Networkyfwang/courses/cs181b/notes/CNN.pdf · PR , ANN, & ML Signal Generation •Resting potential •Charge difference across neuron membrane approximately](https://reader035.fdocuments.net/reader035/viewer/2022081617/6025025fa12f8b7d723cae19/html5/thumbnails/31.jpg)
CNN Demo – CiFar10• 10 classes (airplane, auto, bird, cat, dog, etc.)
• 50,000 training samples and 10,000 test samples
• 32x32 color images
![Page 32: Convolutional Neural Networkyfwang/courses/cs181b/notes/CNN.pdf · PR , ANN, & ML Signal Generation •Resting potential •Charge difference across neuron membrane approximately](https://reader035.fdocuments.net/reader035/viewer/2022081617/6025025fa12f8b7d723cae19/html5/thumbnails/32.jpg)
Only modified the network structure and input format (vector -> 3-D tensor)
CNN for Mnist
Convolution
Max Pooling
Convolution
Max Pooling
input
1 x 28 x 28
25 x 26 x 26
25 x 13 x 13
50 x 11 x 11
50 x 5 x 5Flatten
1250
Fully Connected
Feedforward network
output
Activation
Activation
![Page 33: Convolutional Neural Networkyfwang/courses/cs181b/notes/CNN.pdf · PR , ANN, & ML Signal Generation •Resting potential •Charge difference across neuron membrane approximately](https://reader035.fdocuments.net/reader035/viewer/2022081617/6025025fa12f8b7d723cae19/html5/thumbnails/33.jpg)
Loss Function
• Ground truth – 1-hot vector of dimension nx1• N: number of categories
• 1: true category (dog, cat, etc.)
• CNN output • Softmax post processing to make a probability
distribution
• Loss:• Sum over mini-batch
• 2-norm error (doesn’t work)
• Cross entropy
![Page 34: Convolutional Neural Networkyfwang/courses/cs181b/notes/CNN.pdf · PR , ANN, & ML Signal Generation •Resting potential •Charge difference across neuron membrane approximately](https://reader035.fdocuments.net/reader035/viewer/2022081617/6025025fa12f8b7d723cae19/html5/thumbnails/34.jpg)
Softmax Function
• Exponent CNN output
• Normalized by sum of exponents
• Output is a probability distribution
• Accentuate positives and suppress negatives
![Page 35: Convolutional Neural Networkyfwang/courses/cs181b/notes/CNN.pdf · PR , ANN, & ML Signal Generation •Resting potential •Charge difference across neuron membrane approximately](https://reader035.fdocuments.net/reader035/viewer/2022081617/6025025fa12f8b7d723cae19/html5/thumbnails/35.jpg)
Entropy (Information Theory)
• Number of bits required to encode a lexicon
• Amount of info (surprise)
• Large Pi -> small –log(Pi) -> short codeword
• Small pi -> large –log(pi) -> large codeword
• Compared to ASCII (8-bit per char)
• Entropy is largest when all pi are the same (most uncertain)
![Page 36: Convolutional Neural Networkyfwang/courses/cs181b/notes/CNN.pdf · PR , ANN, & ML Signal Generation •Resting potential •Charge difference across neuron membrane approximately](https://reader035.fdocuments.net/reader035/viewer/2022081617/6025025fa12f8b7d723cae19/html5/thumbnails/36.jpg)
• Use a codebook designed for one distribution for another
• Not symmetric, hence, not a distance function
• q(x) > p(x) => x has a short codeword => with small p(x)
• q(x) > p(x) => x has a long codeword => with large p(x)
• Smallest when p==q
Cross Entropy
![Page 37: Convolutional Neural Networkyfwang/courses/cs181b/notes/CNN.pdf · PR , ANN, & ML Signal Generation •Resting potential •Charge difference across neuron membrane approximately](https://reader035.fdocuments.net/reader035/viewer/2022081617/6025025fa12f8b7d723cae19/html5/thumbnails/37.jpg)
Only modified the network structure and input format (vector -> 3-D tensor)
CNN in Keras
Convolution
Max Pooling
Convolution
Max Pooling
input1 x 28 x 28
25 x 26 x 26
25 x 13 x 13
50 x 11 x 11
50 x 5 x 5
How many parameters for each filter?
How many parameters for each filter?
9
225
![Page 38: Convolutional Neural Networkyfwang/courses/cs181b/notes/CNN.pdf · PR , ANN, & ML Signal Generation •Resting potential •Charge difference across neuron membrane approximately](https://reader035.fdocuments.net/reader035/viewer/2022081617/6025025fa12f8b7d723cae19/html5/thumbnails/38.jpg)
Only modified the network structure and input format (vector -> 3-D tensor)
CNN in Keras
Convolution
Max Pooling
Convolution
Max Pooling
input
1 x 28 x 28
25 x 26 x 26
25 x 13 x 13
50 x 11 x 11
50 x 5 x 5Flatten
1250
Fully Connected Feedforward network
output
![Page 39: Convolutional Neural Networkyfwang/courses/cs181b/notes/CNN.pdf · PR , ANN, & ML Signal Generation •Resting potential •Charge difference across neuron membrane approximately](https://reader035.fdocuments.net/reader035/viewer/2022081617/6025025fa12f8b7d723cae19/html5/thumbnails/39.jpg)
Backpropagation Learning rule
1x 2x 3x 4x
1z 2z
w jk
Wij
1y 2y 3y
kx
w jk
jy
Wij
iz
1O2O
iO
k
u
kjk
u
j xwnet
)()( k
u
kjk
u
j
u
j xwgnetgy
j k
u
kjkij
j
u
jij
u
i xwgWyWNET )(
))(()()( k
u
kjk
j
ij
j
u
jij
u
i
u
i xwgWgyWgNETgz
![Page 40: Convolutional Neural Networkyfwang/courses/cs181b/notes/CNN.pdf · PR , ANN, & ML Signal Generation •Resting potential •Charge difference across neuron membrane approximately](https://reader035.fdocuments.net/reader035/viewer/2022081617/6025025fa12f8b7d723cae19/html5/thumbnails/40.jpg)
Cost function
2
,
2
,
))(((2
1)(
2
1)(
k
u
kjk
j
ij
iu
u
i
u
i
iu
u
i xwgWgOzOE wW,
wkl
Wij E
![Page 41: Convolutional Neural Networkyfwang/courses/cs181b/notes/CNN.pdf · PR , ANN, & ML Signal Generation •Resting potential •Charge difference across neuron membrane approximately](https://reader035.fdocuments.net/reader035/viewer/2022081617/6025025fa12f8b7d723cae19/html5/thumbnails/41.jpg)
What does machine learn?
http://newsneakernews.wpengine.netdna-cdn.com/wp-content/uploads/2016/11/rihanna-puma-creeper-velvet-release-date-02.jpg
球鞋
美洲獅
![Page 42: Convolutional Neural Networkyfwang/courses/cs181b/notes/CNN.pdf · PR , ANN, & ML Signal Generation •Resting potential •Charge difference across neuron membrane approximately](https://reader035.fdocuments.net/reader035/viewer/2022081617/6025025fa12f8b7d723cae19/html5/thumbnails/42.jpg)
42
Visualizing Convolutional Networks
[1] Zeiler, Matthew D., and Rob Fergus. "Visualizing and understanding convolutional networks." European Conference on Computer Vision. Springer International Publishing, 2014.
Deconvolutional NetworkMap activations back to the input
pixel space, show what input pattern
originally caused a given activation.
Steps Compute activations at a
specific layer.
Keep one activation and set all
others to zero.
Unpooling and deconvolution
Construct input image
![Page 43: Convolutional Neural Networkyfwang/courses/cs181b/notes/CNN.pdf · PR , ANN, & ML Signal Generation •Resting potential •Charge difference across neuron membrane approximately](https://reader035.fdocuments.net/reader035/viewer/2022081617/6025025fa12f8b7d723cae19/html5/thumbnails/43.jpg)
43
Visualizing Convolutional Networks
[1] Zeiler, Matthew D., and Rob Fergus. "Visualizing and understanding convolutional networks." European Conference on Computer Vision. Springer International Publishing, 2014.
Layer1
![Page 44: Convolutional Neural Networkyfwang/courses/cs181b/notes/CNN.pdf · PR , ANN, & ML Signal Generation •Resting potential •Charge difference across neuron membrane approximately](https://reader035.fdocuments.net/reader035/viewer/2022081617/6025025fa12f8b7d723cae19/html5/thumbnails/44.jpg)
44
Visualizing Convolutional Networks
[1] Zeiler, Matthew D., and Rob Fergus. "Visualizing and understanding convolutional networks." European Conference on Computer Vision. Springer International Publishing, 2014.
Layer2
![Page 45: Convolutional Neural Networkyfwang/courses/cs181b/notes/CNN.pdf · PR , ANN, & ML Signal Generation •Resting potential •Charge difference across neuron membrane approximately](https://reader035.fdocuments.net/reader035/viewer/2022081617/6025025fa12f8b7d723cae19/html5/thumbnails/45.jpg)
45
Visualizing Convolutional Networks
[1] Zeiler, Matthew D., and Rob Fergus. "Visualizing and understanding convolutional networks." European Conference on Computer Vision. Springer International Publishing, 2014.
Layer3
![Page 46: Convolutional Neural Networkyfwang/courses/cs181b/notes/CNN.pdf · PR , ANN, & ML Signal Generation •Resting potential •Charge difference across neuron membrane approximately](https://reader035.fdocuments.net/reader035/viewer/2022081617/6025025fa12f8b7d723cae19/html5/thumbnails/46.jpg)
46
Visualizing Convolutional Networks
[1] Zeiler, Matthew D., and Rob Fergus. "Visualizing and understanding convolutional networks." European Conference on Computer Vision. Springer International Publishing, 2014.
Layer4
![Page 47: Convolutional Neural Networkyfwang/courses/cs181b/notes/CNN.pdf · PR , ANN, & ML Signal Generation •Resting potential •Charge difference across neuron membrane approximately](https://reader035.fdocuments.net/reader035/viewer/2022081617/6025025fa12f8b7d723cae19/html5/thumbnails/47.jpg)
47
Visualizing Convolutional Networks
[1] Zeiler, Matthew D., and Rob Fergus. "Visualizing and understanding convolutional networks." European Conference on Computer Vision. Springer International Publishing, 2014.
Layer5
![Page 48: Convolutional Neural Networkyfwang/courses/cs181b/notes/CNN.pdf · PR , ANN, & ML Signal Generation •Resting potential •Charge difference across neuron membrane approximately](https://reader035.fdocuments.net/reader035/viewer/2022081617/6025025fa12f8b7d723cae19/html5/thumbnails/48.jpg)
More Application: Playing Go
Network (19 x 19 positions)
Next move
19 x 19 vector
Black: 1
white: -1
none: 0
19 x 19 vector
Fully-connected feedforward network can be used
But CNN performs much better.
19 x 19 matrix (image)
![Page 49: Convolutional Neural Networkyfwang/courses/cs181b/notes/CNN.pdf · PR , ANN, & ML Signal Generation •Resting potential •Charge difference across neuron membrane approximately](https://reader035.fdocuments.net/reader035/viewer/2022081617/6025025fa12f8b7d723cae19/html5/thumbnails/49.jpg)
More Application: Playing Go
CNN
CNN
record of previous plays
Target:“天元” = 1else = 0
Target:“五之 5” = 1else = 0
Training: 黑: 5之五 白:天元 黑:五之5 …
![Page 50: Convolutional Neural Networkyfwang/courses/cs181b/notes/CNN.pdf · PR , ANN, & ML Signal Generation •Resting potential •Charge difference across neuron membrane approximately](https://reader035.fdocuments.net/reader035/viewer/2022081617/6025025fa12f8b7d723cae19/html5/thumbnails/50.jpg)
Why CNN for playing Go?
• Some patterns are much smaller than the whole image
• The same patterns appear in different regions.
Alpha Go uses 5 x 5 for first layer
![Page 51: Convolutional Neural Networkyfwang/courses/cs181b/notes/CNN.pdf · PR , ANN, & ML Signal Generation •Resting potential •Charge difference across neuron membrane approximately](https://reader035.fdocuments.net/reader035/viewer/2022081617/6025025fa12f8b7d723cae19/html5/thumbnails/51.jpg)
Why CNN for playing Go?
• Subsampling the pixels will not change the object
Alpha Go does not use Max Pooling ……
Max Pooling How to explain this???
![Page 52: Convolutional Neural Networkyfwang/courses/cs181b/notes/CNN.pdf · PR , ANN, & ML Signal Generation •Resting potential •Charge difference across neuron membrane approximately](https://reader035.fdocuments.net/reader035/viewer/2022081617/6025025fa12f8b7d723cae19/html5/thumbnails/52.jpg)
More Application: Speech
Time
Freq
ue
ncy
Spectrogram
CNN
Image
The filters move in the frequency direction.
![Page 53: Convolutional Neural Networkyfwang/courses/cs181b/notes/CNN.pdf · PR , ANN, & ML Signal Generation •Resting potential •Charge difference across neuron membrane approximately](https://reader035.fdocuments.net/reader035/viewer/2022081617/6025025fa12f8b7d723cae19/html5/thumbnails/53.jpg)
More Application: Text
Source of image: http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.703.6858&rep=rep1&type=pdf
?