Course Presentation Segmentation - Virginia Techjbhuang/teaching/ece... · What is Semantic...
Transcript of Course Presentation Segmentation - Virginia Techjbhuang/teaching/ece... · What is Semantic...
![Page 1: Course Presentation Segmentation - Virginia Techjbhuang/teaching/ece... · What is Semantic Segmentation ECE 6554: Topic Presentation Xiaolong Li Goal: Partition the image into semantically](https://reader034.fdocuments.net/reader034/viewer/2022042405/5f1e3649785aef1bc97f4f97/html5/thumbnails/1.jpg)
ECE 6554: Topic PresentationXiaolong Li
Semantic Segmentation
UC
LA:h
ttps:
//goo
.gl/i
mag
es/I0
VTi2
![Page 2: Course Presentation Segmentation - Virginia Techjbhuang/teaching/ece... · What is Semantic Segmentation ECE 6554: Topic Presentation Xiaolong Li Goal: Partition the image into semantically](https://reader034.fdocuments.net/reader034/viewer/2022042405/5f1e3649785aef1bc97f4f97/html5/thumbnails/2.jpg)
Semantic Image Segmentation with Deep Convolutional Nets and Fully Connected CRFs. L.-C. Chen, G. Papandreou, I. Kokkinos, K. Murphy, A. L. Yuille. ICLR 2015
Fully Convolutional Networks for Semantic Segmentation. J. Long, E. Shelhamer, and T. Darrell, CVPR 2015
OUTLINE
Paper to talk about:
Semantic Segmentation
Why?
ECE 6554: Topic PresentationXiaolong Li
![Page 3: Course Presentation Segmentation - Virginia Techjbhuang/teaching/ece... · What is Semantic Segmentation ECE 6554: Topic Presentation Xiaolong Li Goal: Partition the image into semantically](https://reader034.fdocuments.net/reader034/viewer/2022042405/5f1e3649785aef1bc97f4f97/html5/thumbnails/3.jpg)
What is Semantic Segmentation
ECE 6554: Topic PresentationXiaolong Li
‘Lena’ lena mirror
![Page 4: Course Presentation Segmentation - Virginia Techjbhuang/teaching/ece... · What is Semantic Segmentation ECE 6554: Topic Presentation Xiaolong Li Goal: Partition the image into semantically](https://reader034.fdocuments.net/reader034/viewer/2022042405/5f1e3649785aef1bc97f4f97/html5/thumbnails/4.jpg)
What is Semantic Segmentation
ECE 6554: Topic PresentationXiaolong Li
Goal: Partition the image into semantically meaningfulparts, and classify each part ——>Patch-wiseRecognizing and delineating objects in an image Classifying each pixel in the image ——>Pixel-wise
![Page 5: Course Presentation Segmentation - Virginia Techjbhuang/teaching/ece... · What is Semantic Segmentation ECE 6554: Topic Presentation Xiaolong Li Goal: Partition the image into semantically](https://reader034.fdocuments.net/reader034/viewer/2022042405/5f1e3649785aef1bc97f4f97/html5/thumbnails/5.jpg)
Why Semantic Segmentation?
ECE 6554: Topic PresentationXiaolong Li
To let robots segment objects so that they can graspthem
http
s://g
oo.g
l/im
ages
/6xA
QAM
![Page 6: Course Presentation Segmentation - Virginia Techjbhuang/teaching/ece... · What is Semantic Segmentation ECE 6554: Topic Presentation Xiaolong Li Goal: Partition the image into semantically](https://reader034.fdocuments.net/reader034/viewer/2022042405/5f1e3649785aef1bc97f4f97/html5/thumbnails/6.jpg)
ECE 6554: Topic PresentationXiaolong Li
Why Semantic Segmentation?
Useful tool for editing images, visual effects
CVF
X Le
ctur
e1:
http
s://w
ww.
yout
ube.
com
/wat
ch?v
=rE-
hVty
tT-I
![Page 7: Course Presentation Segmentation - Virginia Techjbhuang/teaching/ece... · What is Semantic Segmentation ECE 6554: Topic Presentation Xiaolong Li Goal: Partition the image into semantically](https://reader034.fdocuments.net/reader034/viewer/2022042405/5f1e3649785aef1bc97f4f97/html5/thumbnails/7.jpg)
ECE 6554: Topic PresentationXiaolong Li
Why Semantic Segmentation?
Autonomous Driving, to differentiate pedestrian and background
Citydataset
![Page 8: Course Presentation Segmentation - Virginia Techjbhuang/teaching/ece... · What is Semantic Segmentation ECE 6554: Topic Presentation Xiaolong Li Goal: Partition the image into semantically](https://reader034.fdocuments.net/reader034/viewer/2022042405/5f1e3649785aef1bc97f4f97/html5/thumbnails/8.jpg)
Fully Convolutional Networks for Semantic Segmentation. J. Long, E. Shelhamer, and T. Darrell, CVPR 2015
ECE 6554: Topic PresentationXiaolong Li
![Page 9: Course Presentation Segmentation - Virginia Techjbhuang/teaching/ece... · What is Semantic Segmentation ECE 6554: Topic Presentation Xiaolong Li Goal: Partition the image into semantically](https://reader034.fdocuments.net/reader034/viewer/2022042405/5f1e3649785aef1bc97f4f97/html5/thumbnails/9.jpg)
Fully Convolutional Networks for Semantic Segmentation. J. Long, E. Shelhamer, and T. Darrell, CVPR 2015
Usual convolutional networks
Fully convolutional networks
ECE 6554: Topic PresentationXiaolong Li
![Page 10: Course Presentation Segmentation - Virginia Techjbhuang/teaching/ece... · What is Semantic Segmentation ECE 6554: Topic Presentation Xiaolong Li Goal: Partition the image into semantically](https://reader034.fdocuments.net/reader034/viewer/2022042405/5f1e3649785aef1bc97f4f97/html5/thumbnails/10.jpg)
ECE 6554: Topic PresentationXiaolong Li
2015
, Ber
kele
y Vi
sion
: http
://tu
toria
l.caf
fe.b
erke
leyv
isio
n.or
g/
To understand “Fully Convolutional”
![Page 11: Course Presentation Segmentation - Virginia Techjbhuang/teaching/ece... · What is Semantic Segmentation ECE 6554: Topic Presentation Xiaolong Li Goal: Partition the image into semantically](https://reader034.fdocuments.net/reader034/viewer/2022042405/5f1e3649785aef1bc97f4f97/html5/thumbnails/11.jpg)
ECE 6554: Topic PresentationXiaolong Li
2015
, Ber
kele
y Vi
sion
: http
://tu
toria
l.caf
fe.b
erke
leyv
isio
n.or
g/
To understand “Fully Convolutional”
A typical CNN
![Page 12: Course Presentation Segmentation - Virginia Techjbhuang/teaching/ece... · What is Semantic Segmentation ECE 6554: Topic Presentation Xiaolong Li Goal: Partition the image into semantically](https://reader034.fdocuments.net/reader034/viewer/2022042405/5f1e3649785aef1bc97f4f97/html5/thumbnails/12.jpg)
ECE 6554: Topic PresentationXiaolong Li
To understand “Fully Convolutional”A classification CNN
A FCN
2015
, Ber
kele
y Vi
sion
: http
://tu
toria
l.caf
fe.b
erke
leyv
isio
n.or
g/
![Page 13: Course Presentation Segmentation - Virginia Techjbhuang/teaching/ece... · What is Semantic Segmentation ECE 6554: Topic Presentation Xiaolong Li Goal: Partition the image into semantically](https://reader034.fdocuments.net/reader034/viewer/2022042405/5f1e3649785aef1bc97f4f97/html5/thumbnails/13.jpg)
FCN: segmentation that combines layers of hierarchy and refines the spatial precision of the output.
ECE 6554: Topic PresentationXiaolong Li
![Page 14: Course Presentation Segmentation - Virginia Techjbhuang/teaching/ece... · What is Semantic Segmentation ECE 6554: Topic Presentation Xiaolong Li Goal: Partition the image into semantically](https://reader034.fdocuments.net/reader034/viewer/2022042405/5f1e3649785aef1bc97f4f97/html5/thumbnails/14.jpg)
Segmentation Architecture
1. ILSVRC classifiers, in-network up sampling and a pixel-wise loss.
2. Add skips between layers to fuse coarse, semantic and local, appearance
3. Dense predictions, pixel-wise prediction
ECE 6554: Topic PresentationXiaolong Li
![Page 15: Course Presentation Segmentation - Virginia Techjbhuang/teaching/ece... · What is Semantic Segmentation ECE 6554: Topic Presentation Xiaolong Li Goal: Partition the image into semantically](https://reader034.fdocuments.net/reader034/viewer/2022042405/5f1e3649785aef1bc97f4f97/html5/thumbnails/15.jpg)
Some Tricks
ECE 6554: Topic PresentationXiaolong Li
skip layers
![Page 16: Course Presentation Segmentation - Virginia Techjbhuang/teaching/ece... · What is Semantic Segmentation ECE 6554: Topic Presentation Xiaolong Li Goal: Partition the image into semantically](https://reader034.fdocuments.net/reader034/viewer/2022042405/5f1e3649785aef1bc97f4f97/html5/thumbnails/16.jpg)
Some Tricks
ECE 6554: Topic PresentationXiaolong Li
skip layers refinement
![Page 17: Course Presentation Segmentation - Virginia Techjbhuang/teaching/ece... · What is Semantic Segmentation ECE 6554: Topic Presentation Xiaolong Li Goal: Partition the image into semantically](https://reader034.fdocuments.net/reader034/viewer/2022042405/5f1e3649785aef1bc97f4f97/html5/thumbnails/17.jpg)
Interpolation
1. Up-sampling is performed in-network for end-to-end learning by
back-propagation from the pixel wise loss.
2. The deconvolution filter in such a layer can be learned.
ECE 6554: Topic PresentationXiaolong Li
Some Tricks
![Page 18: Course Presentation Segmentation - Virginia Techjbhuang/teaching/ece... · What is Semantic Segmentation ECE 6554: Topic Presentation Xiaolong Li Goal: Partition the image into semantically](https://reader034.fdocuments.net/reader034/viewer/2022042405/5f1e3649785aef1bc97f4f97/html5/thumbnails/18.jpg)
Some results:
PASCAL VOC
NYUDv2
ECE 6554: Topic PresentationXiaolong Li
![Page 19: Course Presentation Segmentation - Virginia Techjbhuang/teaching/ece... · What is Semantic Segmentation ECE 6554: Topic Presentation Xiaolong Li Goal: Partition the image into semantically](https://reader034.fdocuments.net/reader034/viewer/2022042405/5f1e3649785aef1bc97f4f97/html5/thumbnails/19.jpg)
Conclusion
1. Fine-tuning from classification to segmentation gives reasonable
predictions for each net.
2. Learning through up-sampling combined with
the skip layer fusion to be more effective and efficient
ECE 6554: Topic PresentationXiaolong Li
![Page 20: Course Presentation Segmentation - Virginia Techjbhuang/teaching/ece... · What is Semantic Segmentation ECE 6554: Topic Presentation Xiaolong Li Goal: Partition the image into semantically](https://reader034.fdocuments.net/reader034/viewer/2022042405/5f1e3649785aef1bc97f4f97/html5/thumbnails/20.jpg)
Semantic Image Segmentation with Deep Convolutional Nets and Fully Connected CRFs. L.-C. Chen, G. Papandreou, I. Kokkinos, K. Murphy, A. L. Yuille. ICLR 2015
ECE 6554: Topic PresentationXiaolong Li
![Page 21: Course Presentation Segmentation - Virginia Techjbhuang/teaching/ece... · What is Semantic Segmentation ECE 6554: Topic Presentation Xiaolong Li Goal: Partition the image into semantically](https://reader034.fdocuments.net/reader034/viewer/2022042405/5f1e3649785aef1bc97f4f97/html5/thumbnails/21.jpg)
Paper’s main idea
1.Use CNN to generate a rough prediction of segmentation (smooth, blurry heat map)
2.Refine this prediction with a conditional random field (CRF)
ECE 6554: Topic PresentationXiaolong Li
![Page 22: Course Presentation Segmentation - Virginia Techjbhuang/teaching/ece... · What is Semantic Segmentation ECE 6554: Topic Presentation Xiaolong Li Goal: Partition the image into semantically](https://reader034.fdocuments.net/reader034/viewer/2022042405/5f1e3649785aef1bc97f4f97/html5/thumbnails/22.jpg)
Why are CNNs insufficient?
Good for high-level vision tasks like classification, bad for low level tasks like segmentation.
• Problem: subsampling• Problem: spatial invariance (shared kernel weights)
Solution: fully connected CRF
ECE 6554: Topic PresentationXiaolong Li
![Page 23: Course Presentation Segmentation - Virginia Techjbhuang/teaching/ece... · What is Semantic Segmentation ECE 6554: Topic Presentation Xiaolong Li Goal: Partition the image into semantically](https://reader034.fdocuments.net/reader034/viewer/2022042405/5f1e3649785aef1bc97f4f97/html5/thumbnails/23.jpg)
Holes’ algorithms
Solution: fully connected CRF
Solution: fully connected CRF
ECE 6554: Topic PresentationXiaolong Li
![Page 24: Course Presentation Segmentation - Virginia Techjbhuang/teaching/ece... · What is Semantic Segmentation ECE 6554: Topic Presentation Xiaolong Li Goal: Partition the image into semantically](https://reader034.fdocuments.net/reader034/viewer/2022042405/5f1e3649785aef1bc97f4f97/html5/thumbnails/24.jpg)
Solution: fully connected CRF
CRF
Randomly choose points and give initial label
ECE 6554: Topic PresentationXiaolong Li
![Page 25: Course Presentation Segmentation - Virginia Techjbhuang/teaching/ece... · What is Semantic Segmentation ECE 6554: Topic Presentation Xiaolong Li Goal: Partition the image into semantically](https://reader034.fdocuments.net/reader034/viewer/2022042405/5f1e3649785aef1bc97f4f97/html5/thumbnails/25.jpg)
CRF Energy Function
ECE 6554: Topic PresentationXiaolong Li
![Page 26: Course Presentation Segmentation - Virginia Techjbhuang/teaching/ece... · What is Semantic Segmentation ECE 6554: Topic Presentation Xiaolong Li Goal: Partition the image into semantically](https://reader034.fdocuments.net/reader034/viewer/2022042405/5f1e3649785aef1bc97f4f97/html5/thumbnails/26.jpg)
Global Map
ECE 6554: Topic PresentationXiaolong Li
![Page 27: Course Presentation Segmentation - Virginia Techjbhuang/teaching/ece... · What is Semantic Segmentation ECE 6554: Topic Presentation Xiaolong Li Goal: Partition the image into semantically](https://reader034.fdocuments.net/reader034/viewer/2022042405/5f1e3649785aef1bc97f4f97/html5/thumbnails/27.jpg)
Comparison to state-of-the-art
ECE 6554: Topic PresentationXiaolong Li
![Page 28: Course Presentation Segmentation - Virginia Techjbhuang/teaching/ece... · What is Semantic Segmentation ECE 6554: Topic Presentation Xiaolong Li Goal: Partition the image into semantically](https://reader034.fdocuments.net/reader034/viewer/2022042405/5f1e3649785aef1bc97f4f97/html5/thumbnails/28.jpg)
Comparison to state-of-the-art
ECE 6554: Topic PresentationXiaolong Li
![Page 29: Course Presentation Segmentation - Virginia Techjbhuang/teaching/ece... · What is Semantic Segmentation ECE 6554: Topic Presentation Xiaolong Li Goal: Partition the image into semantically](https://reader034.fdocuments.net/reader034/viewer/2022042405/5f1e3649785aef1bc97f4f97/html5/thumbnails/29.jpg)
Comparison to state-of-the-art
ECE 6554: Topic PresentationXiaolong Li
![Page 30: Course Presentation Segmentation - Virginia Techjbhuang/teaching/ece... · What is Semantic Segmentation ECE 6554: Topic Presentation Xiaolong Li Goal: Partition the image into semantically](https://reader034.fdocuments.net/reader034/viewer/2022042405/5f1e3649785aef1bc97f4f97/html5/thumbnails/30.jpg)
Successful Cases
ECE 6554: Topic PresentationXiaolong Li
![Page 31: Course Presentation Segmentation - Virginia Techjbhuang/teaching/ece... · What is Semantic Segmentation ECE 6554: Topic Presentation Xiaolong Li Goal: Partition the image into semantically](https://reader034.fdocuments.net/reader034/viewer/2022042405/5f1e3649785aef1bc97f4f97/html5/thumbnails/31.jpg)
Failure Cases
ECE 6554: Topic PresentationXiaolong Li
![Page 32: Course Presentation Segmentation - Virginia Techjbhuang/teaching/ece... · What is Semantic Segmentation ECE 6554: Topic Presentation Xiaolong Li Goal: Partition the image into semantically](https://reader034.fdocuments.net/reader034/viewer/2022042405/5f1e3649785aef1bc97f4f97/html5/thumbnails/32.jpg)
Conclusion
• Modify the CNN architecture to become less spatially invariant.
• Use the CNN to compute a rough score map.
• Use a fully connected CRF to sharpen the score
ECE 6554: Topic PresentationXiaolong Li
![Page 33: Course Presentation Segmentation - Virginia Techjbhuang/teaching/ece... · What is Semantic Segmentation ECE 6554: Topic Presentation Xiaolong Li Goal: Partition the image into semantically](https://reader034.fdocuments.net/reader034/viewer/2022042405/5f1e3649785aef1bc97f4f97/html5/thumbnails/33.jpg)
Experiments
Intel Xeon E5-2670
NVIDIA GPU
Caffe
VOC_FCN_32s
Python
Cuda8.0
ECE 6554: Topic PresentationXiaolong Li
![Page 34: Course Presentation Segmentation - Virginia Techjbhuang/teaching/ece... · What is Semantic Segmentation ECE 6554: Topic Presentation Xiaolong Li Goal: Partition the image into semantically](https://reader034.fdocuments.net/reader034/viewer/2022042405/5f1e3649785aef1bc97f4f97/html5/thumbnails/34.jpg)
load image, switch to BGR, subtract mean, and make dims C x H x W for Caffe
Data_preparation
ECE 6554: Topic PresentationXiaolong Li
![Page 35: Course Presentation Segmentation - Virginia Techjbhuang/teaching/ece... · What is Semantic Segmentation ECE 6554: Topic Presentation Xiaolong Li Goal: Partition the image into semantically](https://reader034.fdocuments.net/reader034/viewer/2022042405/5f1e3649785aef1bc97f4f97/html5/thumbnails/35.jpg)
Experiment
ECE 6554: Topic PresentationXiaolong Li
26.862607 1.238836
![Page 36: Course Presentation Segmentation - Virginia Techjbhuang/teaching/ece... · What is Semantic Segmentation ECE 6554: Topic Presentation Xiaolong Li Goal: Partition the image into semantically](https://reader034.fdocuments.net/reader034/viewer/2022042405/5f1e3649785aef1bc97f4f97/html5/thumbnails/36.jpg)
Experiment
ECE 6554: Topic PresentationXiaolong Li
39.570141 1.738234
![Page 37: Course Presentation Segmentation - Virginia Techjbhuang/teaching/ece... · What is Semantic Segmentation ECE 6554: Topic Presentation Xiaolong Li Goal: Partition the image into semantically](https://reader034.fdocuments.net/reader034/viewer/2022042405/5f1e3649785aef1bc97f4f97/html5/thumbnails/37.jpg)
Experiment
ECE 6554: Topic PresentationXiaolong Li
32.238836 1.238836
![Page 38: Course Presentation Segmentation - Virginia Techjbhuang/teaching/ece... · What is Semantic Segmentation ECE 6554: Topic Presentation Xiaolong Li Goal: Partition the image into semantically](https://reader034.fdocuments.net/reader034/viewer/2022042405/5f1e3649785aef1bc97f4f97/html5/thumbnails/38.jpg)
Experiment
ECE 6554: Topic PresentationXiaolong Li
39.570141 1.5334832
![Page 39: Course Presentation Segmentation - Virginia Techjbhuang/teaching/ece... · What is Semantic Segmentation ECE 6554: Topic Presentation Xiaolong Li Goal: Partition the image into semantically](https://reader034.fdocuments.net/reader034/viewer/2022042405/5f1e3649785aef1bc97f4f97/html5/thumbnails/39.jpg)
Experiment
ECE 6554: Topic PresentationXiaolong Li
27.895173 1.239234
![Page 40: Course Presentation Segmentation - Virginia Techjbhuang/teaching/ece... · What is Semantic Segmentation ECE 6554: Topic Presentation Xiaolong Li Goal: Partition the image into semantically](https://reader034.fdocuments.net/reader034/viewer/2022042405/5f1e3649785aef1bc97f4f97/html5/thumbnails/40.jpg)
Conclusion
ECE 6554: Topic PresentationXiaolong Li
1.Their network is very fast even when dealing with high resolutionimage, and GPU is at least 20 times faster than CPU.
2. The algorithms show good performance towards images when the objects are either well-separated or overlapped with each other 3. The background of image like sky, grass has a big influence on the segmentation.
Better performance could be expected with their FCN_8s, and detailed performance on validation dataset needs to be checked.
![Page 41: Course Presentation Segmentation - Virginia Techjbhuang/teaching/ece... · What is Semantic Segmentation ECE 6554: Topic Presentation Xiaolong Li Goal: Partition the image into semantically](https://reader034.fdocuments.net/reader034/viewer/2022042405/5f1e3649785aef1bc97f4f97/html5/thumbnails/41.jpg)
ECE 6554: Topic PresentationXiaolong Li
Thanks