Convolutional Neural Networks in MATLABsanum.github.io/2019/img/CNNTalkMATLAB.pdf · Convolutional...
Transcript of Convolutional Neural Networks in MATLABsanum.github.io/2019/img/CNNTalkMATLAB.pdf · Convolutional...
![Page 1: Convolutional Neural Networks in MATLABsanum.github.io/2019/img/CNNTalkMATLAB.pdf · Convolutional Neural Networks in MATLAB 28 March 2019. Welcome • Danielle Winter Application](https://reader033.fdocuments.net/reader033/viewer/2022053023/60541c317594fa4ad67876f4/html5/thumbnails/1.jpg)
Convolutional Neural Networks in MATLAB
28 March 2019
![Page 2: Convolutional Neural Networks in MATLABsanum.github.io/2019/img/CNNTalkMATLAB.pdf · Convolutional Neural Networks in MATLAB 28 March 2019. Welcome • Danielle Winter Application](https://reader033.fdocuments.net/reader033/viewer/2022053023/60541c317594fa4ad67876f4/html5/thumbnails/2.jpg)
Welcome
• Danielle Winter
Application Engineer – AI and Data Science in Engineering
• Adri van Nieuwkerk
Business Development Consultant - Education
Copyright 2018 Opti-Num Solutions (Pty) Ltd 2
![Page 3: Convolutional Neural Networks in MATLABsanum.github.io/2019/img/CNNTalkMATLAB.pdf · Convolutional Neural Networks in MATLAB 28 March 2019. Welcome • Danielle Winter Application](https://reader033.fdocuments.net/reader033/viewer/2022053023/60541c317594fa4ad67876f4/html5/thumbnails/3.jpg)
Why are we here today?
Copyright 2018 Opti-Num Solutions (Pty) Ltd 3
Shell
Musashi Seimitsu Industry Co.,Ltd.
![Page 4: Convolutional Neural Networks in MATLABsanum.github.io/2019/img/CNNTalkMATLAB.pdf · Convolutional Neural Networks in MATLAB 28 March 2019. Welcome • Danielle Winter Application](https://reader033.fdocuments.net/reader033/viewer/2022053023/60541c317594fa4ad67876f4/html5/thumbnails/4.jpg)
Why are we here today?
• Feature extraction is painful
Copyright 2018 Opti-Num Solutions (Pty) Ltd 4
![Page 5: Convolutional Neural Networks in MATLABsanum.github.io/2019/img/CNNTalkMATLAB.pdf · Convolutional Neural Networks in MATLAB 28 March 2019. Welcome • Danielle Winter Application](https://reader033.fdocuments.net/reader033/viewer/2022053023/60541c317594fa4ad67876f4/html5/thumbnails/5.jpg)
Why are we here today?
Copyright 2018 Opti-Num Solutions (Pty) Ltd 5
• Deep learning is getting more accurate
![Page 6: Convolutional Neural Networks in MATLABsanum.github.io/2019/img/CNNTalkMATLAB.pdf · Convolutional Neural Networks in MATLAB 28 March 2019. Welcome • Danielle Winter Application](https://reader033.fdocuments.net/reader033/viewer/2022053023/60541c317594fa4ad67876f4/html5/thumbnails/6.jpg)
Why are we here today?
Copyright 2018 Opti-Num Solutions (Pty) Ltd 6
• Deep Learning is Cool!
![Page 7: Convolutional Neural Networks in MATLABsanum.github.io/2019/img/CNNTalkMATLAB.pdf · Convolutional Neural Networks in MATLAB 28 March 2019. Welcome • Danielle Winter Application](https://reader033.fdocuments.net/reader033/viewer/2022053023/60541c317594fa4ad67876f4/html5/thumbnails/7.jpg)
Agenda
• Introduction to concepts
– Deep Learning checklist
– Convolutional Neural Networks and layer architecture
• CNNs for non-image applications
• Some challenges and how to mitigate them
• Take Aways
Copyright 2018 Opti-Num Solutions (Pty) Ltd 7
![Page 8: Convolutional Neural Networks in MATLABsanum.github.io/2019/img/CNNTalkMATLAB.pdf · Convolutional Neural Networks in MATLAB 28 March 2019. Welcome • Danielle Winter Application](https://reader033.fdocuments.net/reader033/viewer/2022053023/60541c317594fa4ad67876f4/html5/thumbnails/8.jpg)
Agenda
• Introduction to concepts
– Deep Learning checklist
– Convolutional Neural Networks and layer architecture
• CNNs for non-image applications
• Some challenges and how to mitigate them
• Take Aways
Copyright 2018 Opti-Num Solutions (Pty) Ltd 8
![Page 9: Convolutional Neural Networks in MATLABsanum.github.io/2019/img/CNNTalkMATLAB.pdf · Convolutional Neural Networks in MATLAB 28 March 2019. Welcome • Danielle Winter Application](https://reader033.fdocuments.net/reader033/viewer/2022053023/60541c317594fa4ad67876f4/html5/thumbnails/9.jpg)
Deep Learning Workflow
Copyright 2018 Opti-Num Solutions (Pty) Ltd 9
validation data
testing data
DataNetwork Architecture• Pretrained• Transfer Learning• From Scratch
Training Options• Algorithm• Learning Rate• Mini-Batch• Validation• Environment
Data
Training options
Network Layers
Deep Learning Checklist
Car Tree Dog Cat Bus Duck
![Page 10: Convolutional Neural Networks in MATLABsanum.github.io/2019/img/CNNTalkMATLAB.pdf · Convolutional Neural Networks in MATLAB 28 March 2019. Welcome • Danielle Winter Application](https://reader033.fdocuments.net/reader033/viewer/2022053023/60541c317594fa4ad67876f4/html5/thumbnails/10.jpg)
Deep Learning Workflow
Copyright 2018 Opti-Num Solutions (Pty) Ltd 10
validation data testing data
![Page 11: Convolutional Neural Networks in MATLABsanum.github.io/2019/img/CNNTalkMATLAB.pdf · Convolutional Neural Networks in MATLAB 28 March 2019. Welcome • Danielle Winter Application](https://reader033.fdocuments.net/reader033/viewer/2022053023/60541c317594fa4ad67876f4/html5/thumbnails/11.jpg)
Convolutional Neural Networks
Copyright 2018 Opti-Num Solutions (Pty) Ltd 11
![Page 12: Convolutional Neural Networks in MATLABsanum.github.io/2019/img/CNNTalkMATLAB.pdf · Convolutional Neural Networks in MATLAB 28 March 2019. Welcome • Danielle Winter Application](https://reader033.fdocuments.net/reader033/viewer/2022053023/60541c317594fa4ad67876f4/html5/thumbnails/12.jpg)
CNN Architecture – Input Layer
Copyright 2018 Opti-Num Solutions (Pty) Ltd 12
imageInputLayer([5,5])Image size: m*n*a
Colour image: a = 3Multispectral image: a = many
![Page 13: Convolutional Neural Networks in MATLABsanum.github.io/2019/img/CNNTalkMATLAB.pdf · Convolutional Neural Networks in MATLAB 28 March 2019. Welcome • Danielle Winter Application](https://reader033.fdocuments.net/reader033/viewer/2022053023/60541c317594fa4ad67876f4/html5/thumbnails/13.jpg)
CNN Architecture – Middle Layers
• Convolution of spatial filters with input image
Copyright 2018 Opti-Num Solutions (Pty) Ltd 13
convolution2dLayer([2,2],50)
![Page 14: Convolutional Neural Networks in MATLABsanum.github.io/2019/img/CNNTalkMATLAB.pdf · Convolutional Neural Networks in MATLAB 28 March 2019. Welcome • Danielle Winter Application](https://reader033.fdocuments.net/reader033/viewer/2022053023/60541c317594fa4ad67876f4/html5/thumbnails/14.jpg)
CNN Architecture – Middle Layers
• Convolution of spatial filters with input image
Copyright 2018 Opti-Num Solutions (Pty) Ltd 14
convolution2dLayer([2,2],50) activations(net, X, layer)
![Page 15: Convolutional Neural Networks in MATLABsanum.github.io/2019/img/CNNTalkMATLAB.pdf · Convolutional Neural Networks in MATLAB 28 March 2019. Welcome • Danielle Winter Application](https://reader033.fdocuments.net/reader033/viewer/2022053023/60541c317594fa4ad67876f4/html5/thumbnails/15.jpg)
CNN Architecture – Middle Layers
Copyright 2018 Opti-Num Solutions (Pty) Ltd 15
reluLayer maxPooling2dLayer([3,3])
𝑓 𝑥 = ቊ𝑥, 𝑥 ≥ 00, 𝑥 < 0
![Page 16: Convolutional Neural Networks in MATLABsanum.github.io/2019/img/CNNTalkMATLAB.pdf · Convolutional Neural Networks in MATLAB 28 March 2019. Welcome • Danielle Winter Application](https://reader033.fdocuments.net/reader033/viewer/2022053023/60541c317594fa4ad67876f4/html5/thumbnails/16.jpg)
CNN Architecture – Final Layers
Copyright 2018 Opti-Num Solutions (Pty) Ltd 16
fullyConnectedLayer(2) softmaxLayer()
classificationLayer()
![Page 17: Convolutional Neural Networks in MATLABsanum.github.io/2019/img/CNNTalkMATLAB.pdf · Convolutional Neural Networks in MATLAB 28 March 2019. Welcome • Danielle Winter Application](https://reader033.fdocuments.net/reader033/viewer/2022053023/60541c317594fa4ad67876f4/html5/thumbnails/17.jpg)
Agenda
• Introduction to concepts
– Deep Learning checklist
– Convolutional Neural Networks and layer architecture
• CNNs for non-image applications
• Some challenges and how to mitigate them
• Take Aways
Copyright 2018 Opti-Num Solutions (Pty) Ltd 17
![Page 18: Convolutional Neural Networks in MATLABsanum.github.io/2019/img/CNNTalkMATLAB.pdf · Convolutional Neural Networks in MATLAB 28 March 2019. Welcome • Danielle Winter Application](https://reader033.fdocuments.net/reader033/viewer/2022053023/60541c317594fa4ad67876f4/html5/thumbnails/18.jpg)
CNNs for Non-Image Applications
Copyright 2018 Opti-Num Solutions (Pty) Ltd 18
![Page 19: Convolutional Neural Networks in MATLABsanum.github.io/2019/img/CNNTalkMATLAB.pdf · Convolutional Neural Networks in MATLAB 28 March 2019. Welcome • Danielle Winter Application](https://reader033.fdocuments.net/reader033/viewer/2022053023/60541c317594fa4ad67876f4/html5/thumbnails/19.jpg)
CNNs for Non-Image Applications
Copyright 2018 Opti-Num Solutions (Pty) Ltd 19
![Page 20: Convolutional Neural Networks in MATLABsanum.github.io/2019/img/CNNTalkMATLAB.pdf · Convolutional Neural Networks in MATLAB 28 March 2019. Welcome • Danielle Winter Application](https://reader033.fdocuments.net/reader033/viewer/2022053023/60541c317594fa4ad67876f4/html5/thumbnails/20.jpg)
Agenda
• Introduction to concepts
– Deep Learning checklist
– Convolutional Neural Networks and layer architecture
• CNNs for non-image applications
• Some challenges and how to mitigate them
• Take Aways
Copyright 2018 Opti-Num Solutions (Pty) Ltd 20
![Page 21: Convolutional Neural Networks in MATLABsanum.github.io/2019/img/CNNTalkMATLAB.pdf · Convolutional Neural Networks in MATLAB 28 March 2019. Welcome • Danielle Winter Application](https://reader033.fdocuments.net/reader033/viewer/2022053023/60541c317594fa4ad67876f4/html5/thumbnails/21.jpg)
Some Challenges
• Building layer architecture
Copyright 2018 Opti-Num Solutions (Pty) Ltd 21
![Page 22: Convolutional Neural Networks in MATLABsanum.github.io/2019/img/CNNTalkMATLAB.pdf · Convolutional Neural Networks in MATLAB 28 March 2019. Welcome • Danielle Winter Application](https://reader033.fdocuments.net/reader033/viewer/2022053023/60541c317594fa4ad67876f4/html5/thumbnails/22.jpg)
Some Challenges
• Building layer architecture
• Training Time
– Use GPU
Copyright 2018 Opti-Num Solutions (Pty) Ltd 22
![Page 23: Convolutional Neural Networks in MATLABsanum.github.io/2019/img/CNNTalkMATLAB.pdf · Convolutional Neural Networks in MATLAB 28 March 2019. Welcome • Danielle Winter Application](https://reader033.fdocuments.net/reader033/viewer/2022053023/60541c317594fa4ad67876f4/html5/thumbnails/23.jpg)
Some Challenges
• Building layer architecture
• Training Time
– Use GPU
• Getting better performance
– Learning rate
– Epochs
– Amount of data
Copyright 2018 Opti-Num Solutions (Pty) Ltd 23
NaNs/ spike in
loss?
↑
learn rate
↓ learn rate
Loss ↓ at end of
training?
↑ Epochs↓ learn rate at plateau
Training loss >
validation loss?Augment data,
↓ epochs
YN
Y
Y
N
![Page 24: Convolutional Neural Networks in MATLABsanum.github.io/2019/img/CNNTalkMATLAB.pdf · Convolutional Neural Networks in MATLAB 28 March 2019. Welcome • Danielle Winter Application](https://reader033.fdocuments.net/reader033/viewer/2022053023/60541c317594fa4ad67876f4/html5/thumbnails/24.jpg)
Agenda
• Introduction to concepts
– Deep Learning checklist
– Convolutional Neural Networks and layer architecture
• CNNs for non-image applications
• Some challenges and how to mitigate them
• Take Aways
Copyright 2018 Opti-Num Solutions (Pty) Ltd 24
![Page 25: Convolutional Neural Networks in MATLABsanum.github.io/2019/img/CNNTalkMATLAB.pdf · Convolutional Neural Networks in MATLAB 28 March 2019. Welcome • Danielle Winter Application](https://reader033.fdocuments.net/reader033/viewer/2022053023/60541c317594fa4ad67876f4/html5/thumbnails/25.jpg)
Take-Aways
• Growing the Deep Learning Community in South Africa
– Application-based research
• Deep Learning in MATLAB
– Deep Learning Onramp Tutorial
– MATLAB Onramp Tutorial
Copyright 2018 Opti-Num Solutions (Pty) Ltd 25
![Page 26: Convolutional Neural Networks in MATLABsanum.github.io/2019/img/CNNTalkMATLAB.pdf · Convolutional Neural Networks in MATLAB 28 March 2019. Welcome • Danielle Winter Application](https://reader033.fdocuments.net/reader033/viewer/2022053023/60541c317594fa4ad67876f4/html5/thumbnails/26.jpg)
Q&A
Copyright 2018 Opti-Num Solutions (Pty) Ltd 26
Questions & Answers