CROSS MODALITY SYNTHESIS (MRI TO CT)

Yu Zhao, zhaoyu.hust@gmail.com

Background

¨ Radiotherapy treatment planning:¤ Target delineation, treatment response assessment,

adaptive therapies, etc.

¨ MR images are necessary: superior soft-tissue contrast, etc. Missing: electron density info, provided by CT images è CT-MRI coregistration: 1. misalignment; 2. workflow, patients and costs pressure.

¨ MRI-only simulation workflow: providing both info on tumor volume and location and electron density information.

Background

¨ CT-MR registration VS MR-only simulation

Background

¨ MRI images: in-phase (IP), out-of-phase (OOP): sequences correspond to paired MRI gradient echo (GRE) sequences obtained with the same repetition time (TR) but with two different echo time (TE)values.

¨ The main application of the IP-OOP sequences is to identify pathological (microsopic) fat content of tissues in the abdomen by showing signal intensities drop on the OOP images compared to the IP images.

Background

fat only = in-phase - opposed phase water only = in-phase + opposed phase

Methods

¨ 1. Style Transfer ¨ 2. Image Analogy ¨ 3. Deep Learning Nets

Style

Content

Synthetic

1. Style Transfer

1. Style Transfer

¨ A Neural Algorithm of Artistic StyleLeon A. Gatys, Alexander S. Ecker, Matthias Bethge

v 1). A new way to visualize feature. Use gradient decent to update the image till it generate similar feature response.

v 2). content reconstructionv 3). style reconstruction

A schematic of the style loss.

A schematic of the content loss.

1. Style Transfer

1. Style Transfer

¨ Feature extraction nets for losses

Content features at different level

1. Style Transfer

Feature visualization

¨ Content loss

¨ The content loss is the (scaled, squared) Euclidean distance between feature representations of the content and combination images.

¨ draw the content feature from block2_conv2 è structural detail

¨ Style loss

¨ Gram matrix¨ The terms of this matrix are proportional to the covariances

of corresponding sets of features, and thus capturesinformation about which features tend to activate together.By only capturing these aggregate statistics across theimage, they are blind to the specific arrangement of objectsinside the image. This is what allows them to captureinformation about style independent of content.

¨ The style loss is then the (scaled, squared) Frobenius norm ofthe difference between the Gram matrices of the style andcombination images.

¨ Total loss

¨ Total loss= content loss + style loss + variation loss (a regularization term that encourages spatial smoothness)

¨ Regression

¨ Gradient Decent

¨ Reproduction results Style Image

Content Images

2. Using Image Analogies:

¨ http://www.mrl.nyu.edu/publications/image-analogies/analogies-fullres.pdf

3. Deep Learning Nets

U-Net: Convolutional Networks for Biomedical Image Segmentation

3. Deep Learning Nets

U-Net: Convolutional Networks for Biomedical Image Segmentation

3. Deep Learning Nets

pk(x) ≈ 1 for the k that has the maximum activation ak(x) and pk(x) ≈ 0 for all other k.

U-Net: Convolutional Networks for Biomedical Image Segmentation

3. Deep Learning Nets

U-Net: Convolutional Networks for Biomedical Image Segmentation

3. Deep Learning Nets

U-Net: Convolutional Networks for Biomedical Image Segmentation

Data augmentation is essential to teach the network the desired invariance and robustness properties, when only few training samples are available. Especially random elastic deformations of the training samples seem to be the key concept to train a segmentation network with very few annotated images. We generate smooth deformations using random displacement vectors on a coarse 3 by 3 grid. The displacements are sampled from a Gaussian distribution with 10 pixels standard deviation.

3. Deep Learning Nets

U-Net: Convolutional Networks for Biomedical Image Segmentation

3. Deep Learning Nets

V-Net: Fully Convolutional Neural Networks for Volumetric Medical Image Segmentation

3. Deep Learning Nets

¨ Deep learning for MRI-CT synthesis for radiotherapy¤ perfectly registered CT-MR pairs ¤ Memory issue for training of a standard 3D deep

network

¨ Multi-view Deep Convolutional Neural Networks ¤ Effective cost function for misalignments¤ Multi-view combination for memory efficiency without

losing 3D context

3. Deep Learning Nets

1). Unet based structure:

U-Net: Convolutional Networks for Biomedical Image Segmentation U-Net: Our modified version

3. Deep Learning Nets

3. Deep Learning Nets

3. Deep Learning Nets

¨ Misalignment between Ground Truth and Inputs --Max Pooled loss:

¨ 1). Maxpooling before calculating actual loss

3. Deep Learning Nets

¨ Misalignment between Ground Truth and Inputs --Max Pooled loss:

A schematic explanation of the impact of mis-registration to intensity transformation. (a) Image1 (Modality 1), (b) A perfectly registered Image2

(Modality 2), (c) Image 2 with rigid mis-alignment, (d) Image 2 with non-rigid mis-alignment. Triangles in (a)-(d) represent the same object. Dashed lines in (c)

and (d) denote the locations of the perfectly registered Image2.

3. Deep Learning Nets

¨ Misalignment between Ground Truth and Inputs --Max Pooled loss:

¨ 2). using L1 distance rather than L2 as L1 encourages less blurring (Image-to-Image Translation with Conditional Adversarial Networks, Phillip Isola, et al, 2016)

3. Deep Learning Nets

¨ Final loss: Maxpooling Hinge-like Huber loss

! " =

0, " < 0.0112"

*, 0.01 < " < +

+ " − 12+ , -.ℎ012340

3. Deep Learning Nets

¨ 3D models (whole volume in)

3. Deep Learning Nets 2.5D: 2D at different views è combine

¨ Axial view: good but with serious stitching blurs¨ Sagittal view: overall good¨ Coronal view: very bad

3. Deep Learning Nets 2.5D: 2D at different views è combine

¨ Combine:

Summary

¨ Multi-channel Multi-view net with hinge-like smL1 loss