Fully Convolutional Networks for Semantic Segmentationai.ms.mff.cuni.cz/~sui/leibl.pdf · 2018. 1....

Fully Convolutional Networks for SemanticSegmentation

Marek Leibl

Center for Machine Perception, Department of Cybernetics, FEL CTU

March 30, 2017

Marek Leibl (Center for Machine Perception, Department of Cybernetics, FEL CTU)Biomedical Images Recognition March 30, 2017 1 / 28

Presentation Outline

Introduction ... convolutional neural networks

Semantic Segmentation ... fully convolutional neural networks

Conditional Random Fields ... and recurrent neural networks

Applications ... drosophila eggs segmentation

Convolutional Neural Networks

Convolutional Neural Networks - Introduction

Convolutional Neural Networks (CNNs)

layered artificial neural networkinspired by organization of the animal visual cortexlocal connectivity

Deep Convolutional Neural Networks

many convolutional layers often combined with max-pooling andrelu

CNNs - Convolution

Convolution = building block of CNNs

G[i, j] =∑u

H[u, v] · F [i− u, j − v]

CNNs - Example

CNNs - Structure

Neurons in each layer organized into 3D matrixes

2 spatial dimensions1 dimension for channels (features)

Local connectivity

each neuron relatively small number of input connections ...receptive field

Shared weights

weights are invariant to translations in spatial dimensionsreduced search space and better robustness

CNNs - Building blocks

Convolutional layer

trainable weightstypically followed by Relu

Relu (rectified linear unit)

f(x) = max(0, x)introduces non-linearity to CNNeasy to compute, no saturation

Max-pooling

reducing spatial size

Softmax

normalized exponentialfunctionf(x)j = exj∑

CNNs - Deep CNN architectures for image classification

GoogLeNet (2015)

9 Inception modulesover 100 layerstrained on a few GPUswithin a week

Microsoft ResNet (2015)

152 layersresidual blocks→ easier to optimizetrained on an 8 GPU machinefor two to three weeks

Semantic Segmentation

Semantic Segmentation vs Image Classification

Image classification: determine class for a given image

super human performance on some tasks

Semantic segmentation: determine class for each pixel

harder problem

Fully Convolutional Neural Netwoks

Fully Convolutional Neural Netwok

no fully connected layeroutput: pixel-wise prediction

Training Fully Convolutional NNs

needed label for each pixel (ground truth)need to be manually segmented by human→ training set is usually expensive

Fully Convolutional Neural Netwoks: Architectures

SegNet (2015)

deep encoder-decoder architectureUniversity of Cambridge

U-Net (2015)

U-shaped architecturebiomedical image segmentation

Fully Convolutional Neural Netwoks: SegNet

SegNet (2015)

deep encoder-decoder architectureUniversity of Cambridgehttp://mi.eng.cam.ac.uk/projects/segnet/

Fully Convolutional Neural Netwoks: U-Net

U-Net (2015)

U-shaped architecturebiomedical image segmentation

Conditional Random Fields&

Convolutional Neural Networks

Drawbacks of fully connected CNNs

CNNs ouperforms “classical” methods onmajority of semantic segmenations tasks

they still fail in some cases

Cross entropy loss

does not considerglobal consistencycan only use informationwithin its receptive field

→ “holes” in the segmentation map→ inconsistent segmentation

E = −∑x

pxclog(yxc)

Conditional Random Fields: Introduction

Conditional Random Field (CRF)

probabilistic graphical modeltwo types of variables:

observations (e.g. original image)random variables (e.g. pixel-wise segmentation)

used for structured predictione.g. semantic segmentationthese days outperformed by CNNs

CRF as a post-processing methodto improve CNN output

Conditional Random Fields: Formal Definition

Markov Random Field (MRF)

G = (V,E) undirected graphY = {Yv}v∈V

Yv|YN(v) ⊥⊥ YV \N [v]|YN(v)

Conditional Random Field (CRF)G = (V,E) undirected graph

V = {1, ..., N} ... pixelsE ... define neighbors

I = {Ii}N1 ... observation for each pixelX = {Xi}N1 ... random variable for each pixelX|I is a Markov Random Field (MRF)→ P (X = x|I) = 1

Z(I)exp {−E(x|I)}

Conditional Random Fields Postprocessing

Goal: maximize P (X = x|I) = 1Z(I)exp {−EI(x)}

I ... CNN segmentationX ... final segmentation

→ minimize energy function:

EI(x) =∑

i ψu(xi) +∑

i<j ψp(xi, xj)NP-hard for general graphs

→ Mean Field Approximation

CRF: Mean Field Iteration as RNN

CRF as RNN

Drosophila Eggs Segmentaion

Problem Definition

Binary segmentation of microscopy scans

classes: the eggs and the background

75 manually annotated images of drosophila eggs

4000 unlabeled images

U-Net architecture

U-shaped Convolutional Neural Networkthe usual convolution-pooling path followed by the upsampling pathoutput: segmentation map of the input image

Results

Thank You!

http://cs231n.stanford.edu/

https://adeshpande3.github.io/adeshpande3.github.io/The-9-Deep-Learning-Papers-You-Need-To-Know-About.html

http://www.robots.ox.ac.uk/ szheng/CRFasRNN.html

https://www.tensorflow.org

Fully Convolutional Networks for Semantic Segmentationai.ms.mff.cuni.cz/~sui/leibl.pdf · 2018. 1....

Documents

Transcript of Fully Convolutional Networks for Semantic Segmentationai.ms.mff.cuni.cz/~sui/leibl.pdf · 2018. 1....

UC Berkeley Fully Convolutional Networks for Semantic ... · BoxSup: Exploiting Bounding Boxes to Supervise Convolutional Networks for Semantic Segmentation. Dai et al. 2015. FCNs

SEMANTIC SEGMENTATION OF AERIAL IMAGERY VIA MULTI … · SEMANTIC SEGMENTATION OF AERIAL IMAGERY VIA MULTI-SCALE SHUFFLING CONVOLUTIONAL NEURAL NETWORKS WITH DEEP SUPERVISION Kaiqiang

Convolutional Neural Networksimft.ftn.uns.ac.rs/ssip2017/wp-content/uploads/... · Convolutional Neural Networks for Semantic ... Shelhamer, Darrell, Fully Convolutional Networks

Multiscale Hierarchical Convolutional Networks · Multiscale Hierarchical Convolutional Networks and code is available online using TensorFlow and Keras1. 2. Deep Convolutional Networks

Convolutional Neural Networks Analyzed via Convolutional ... · PDF fileConvolutional Neural Networks Analyzed via Convolutional Sparse Coding ... Deep Learning, Convolutional Neural

Expectation-Maximization Attention Networks for Semantic ...openaccess.thecvf.com/content_ICCV_2019/papers/Li... · Semantic segmentation. Fully convolutional network (FCN) [22] based

Convolutional Neural Networks

SCENE SEMANTIC SEGMENTATION FROM INDOOR · PDF fileSCENE SEMANTIC SEGMENTATION FROM ... we propose an Encode-Decoder Fully Convolutional Networks for RGB-D image classification. ...

Convolutional Networks

ScribbleSup: Scribble-Supervised Convolutional Networks for ...openaccess.thecvf.com/content_cvpr_2016/papers/Lin...ScribbleSup: Scribble-Supervised Convolutional Networks for Semantic

Fully Convolutional Networks

Deep Convolutional Neural Networks - Semantic Scholar · entropy Article A Framework for Designing the Architectures of Deep Convolutional Neural Networks Saleh Albelwi * and Ausif

Convolutional Neural Networks Analyzed via Convolutional ...yromano/posters/ml_csc_poster.pdf · Convolutional Neural Networks ... Recently, convolutional sparse coding (CSC) ...

Learning to Extract Semantic Structure From Documents ...Learning to Extract Semantic Structure from Documents Using Multimodal Fully Convolutional Neural Networks Xiao Yang‡, Ersin

Convolutional Networks: Motivationsrihari/CSE676/9.2 CNN-Motivation.pdf · Deep Learning Srihari Overview of Convolutional Networks • Convolutional networks, also known as Convolutional

SemanticFusion: Dense 3D Semantic Mapping with ...wp.doc.ic.ac.uk/bjm113/.../2017/07/SemanticFusion... · SemanticFusion: Dense 3D Semantic Mapping with Convolutional Neural Networks

Deep Neural Networks Convolutional Networks II

Fully Convolutional Networks for Semantic Segmentation [1] › ~yjlee › teaching › ecs289g... · Fully Convolutional Networks for Semantic Segmentation [1] Jonathan Long, Evan

SemanticFusion: Dense 3D Semantic Mapping with ... · SemanticFusion: Dense 3D Semantic Mapping with Convolutional Neural Networks John McCormac, Ankur Handa, Andrew Davison, and

Fully Convolutional Adaptation Networks for Semantic ...zhaofanqiu.deepfun.club/papers/CVPR18SEG.pdf · street scenes) on semantic segmentation and our proposal achieves superior