Mapping cortical change in Alzheimer's disease, brain development ...
Neuroimage Analysis for Automated Brain Disease...
Transcript of Neuroimage Analysis for Automated Brain Disease...
Neuroimage Analysis
for Automated Brain Disease Diagnosis
University of North Carolina at Chapel Hill
http://mingxia.web.unc.edu/
07-17-2019
Mingxia Liu
Healthy Brain Alzheimer’s
Ventricle
Sulcus
Gyrus
Sulcus
Gyrus
Healthy Brain Alzheimer’s
Healthy Brain vs. Alzheimer’s
Background
Background
• Alzheimer’s Disease (AD)
– A progressive disease
Normal
Control
Mild Cognitive
Impairment (MCI)Alzheimer’s
Disease
Bra
inH
ealth
Time
Stable MCI (sMCI)Progressive MCI (pMCI)
• Calling Need
– Developing computer-aided methods for MCI/AD diagnosis
• Structural Magnetic Resonance Imaging (MRI)
• FDG-Positron Emission Tomography (PET)
• Cerebrospinal Fluid (CSF) ‐‐‐ Aβ42, t‐tau and p‐tau
Biomarkers for early diagnosis of AD and MCI
MRI PET CSF
CSF
Brain
Dura
Multi-modal data
Background
M. Liu, etc., Landmark-based Deep Multi-Instance Learning for Brain Disease Diagnosis, Medical Image Analysis, 2018.
M. Liu, etc. Joint Classification and Regression via Deep Multi-task Multi-channel Learning for Alzheimer’s Disease Diagnosis. IEEE
Trans. on Biomedical Engineering, 2018.
fMRIPETsMRI
Neuroimaging
Data
Brain Disease Diagnosis – Typical Pipeline
Image
Preprocessing
Feature
Extraction/Selection
Classifier
Learning
Machine Learning & Deep Learning
Challenges in Computer-aided Disease Diagnosis
• Effective feature representation of neuroimages
• Missing multi-modal data
• Heterogeneous data at different imaging sites
Outline
• Missing Data
• Multi-modal Data Fusion
• Domain Adaptation
Multi-modal Neuroimage
CSFPETsMRI
Single-modal Neuroimage
• Structural MRI (sMRI)
sMRI
Part I. Single-modal Neuroimage Analysis
• Structural MRI based Brain Disease Diagnosis
Anatomical Landmarks for Structural MRI
• Landmark-based Deep Representation of sMRI
C. Lian, M. Liu, J. Zhang, and D. Shen. IEEE Trans. on Pattern Analysis and Machine Intelligence, 2019.
M. Liu, J. Zhang, C. Lian, and D. Shen. IEEE Trans. on Cybernetics, 2019.
M. Liu, J. Zhang, E. Adeli, and D. Shen. IEEE Trans. on Biomedical Engineering, 2019.
M. Liu, J. Zhang, E. Adeli, and D. Shen. Medical Image Analysis, 2018.
M. Liu, J. Zhang, D. Nie, P.T. Yap, and D. Shen. IEEE Journal of Biomedical and Health Informatics, 2018.
J. Zhang, M. Liu, and D. Shen. IEEE Trans. on Image Processing, 2017.
...
Training Data
Patch-Level Feature Learning
Convolutional Neural
Network 1
Landmark Discovery
Disease
ClassificationTesting Data
Landmark-based Patch Extraction
Convolutional Neural
Network P
...
Image Retrieval
...Deep Feature
Representation 1
Deep Feature
Representation P
Patch-based
Features
MR Image Pre-processing
...
Landmark Detection
Landmark
Definition
...
Patch Extraction
...
Training Data
Patch-Level Feature Learning
Convolutional Neural
Network 1
Landmark Discovery
Disease
ClassificationTesting Data
Landmark-based Patch Extraction
Convolutional Neural
Network P
...
Image Retrieval
...Deep Feature
Representation 1
Deep Feature
Representation P
Patch-based
Features
MR Image Pre-processing
...
Landmark Detection
Landmark
Definition
...
...
Training Data
Patch-Level Feature Learning
Convolutional Neural
Network 1
Landmark Discovery
Disease
ClassificationTesting Data
Landmark-based Patch Extraction
Convolutional Neural
Network P
...
Image Retrieval
...Deep Feature
Representation 1
Deep Feature
Representation P
Patch-based
Features
MR Image Pre-processing
...
Landmark Detection
Landmark
Definition
...
Landmark Discovery
...
Training Data
Patch-Level Feature Learning
Convolutional Neural
Network 1
Landmark Discovery
Disease
ClassificationTesting Data
Landmark-based Patch Extraction
Convolutional Neural
Network P
...
Image Retrieval
...Deep Feature
Representation 1
Deep Feature
Representation P
Patch-based
Features
MR Image Pre-processing
...
Landmark Detection
Landmark
Definition
...
Test MRI
...
Training Data
Patch-Level Feature Learning
Convolutional Neural
Network 1
Landmark Discovery
Disease
ClassificationTesting Data
Landmark-based Patch Extraction
Convolutional Neural
Network P
...
Image Retrieval
...Deep Feature
Representation 1
Deep Feature
Representation P
Patch-based
Features
MR Image Pre-processing
...
Landmark Detection
Landmark
Definition
...
Landmark Detection
...
Training Data
Patch-Level Feature Learning
Convolutional Neural
Network 1
Landmark Discovery
Disease
ClassificationTesting Data
Landmark-based Patch Extraction
Convolutional Neural
Network P
...
Image Retrieval
...Deep Feature
Representation 1
Deep Feature
Representation P
Patch-based
Features
MR Image Pre-processing
...
Landmark Detection
Landmark
Definition
...
Training MRIs
Pre-processed
MR Images
…
…
CNN
Deep Multi-channel Convolutional
Neural Network (CNN)
DiseaseClassification
Clinical ScorePrediction
* Featured Article of IEEE Journal of Biomedical and Health Informatics, 2018
Anatomical Landmark-based Deep Network
00.01 p-values
1,740 landmarks via group
comparison between AD and NCTop 50 landmarks
Anatomical Landmark-based Deep Network
M. Liu, J. Zhang, E. Adeli, and D. Shen. Medical Image Analysis, 2018.
M. Liu, J. Zhang, D. Nie, P.T. Yap, and D. Shen. IEEE Journal of Biomedical and Health Informatics, 2018.
Concatenated
sMRI
FC8-32 FC8-32FC8-32 …
FC9-32L
Conv1-
32@3×3×3
Conv2-
32@3×3×3
Max-pooling
Conv3-
64@2×2×2
Conv4-
64@2×2×2
Max-pooling
Conv5-
128@2×2×2
Conv6-
128@2×2×2
Max-pooling
Conv1-
32@3×3×3
Conv2-
32@3×3×3
Max-pooling
Conv3-
64@2×2×2
Conv4-
64@2×2×2
Max-pooling
Conv5-
128@2×2×2
Conv6-
128@2×2×2
Max-pooling
FC7-128
…
FC7-128
Conv1-
32@3×3×3
Conv2-
32@3×3×3
Max-pooling
Conv3-
64@2×2×2
Conv4-
64@2×2×2
Max-pooling
Conv5-
128@2×2×2
Conv6-
128@2×2×2
Max-pooling
Patch 1 Patch 2 … Patch L
Landmark-based Deep Network
FC7-128
FC10-8L
FC11-2
Class Label
Soft-max
Global
Image-level
Representation
Local
Patch-level
Representation
Global
Representation?
M. Liu, J. Zhang, E. Adeli, and D. Shen. Medical Image Analysis, 2018.
M. Liu, J. Zhang, D. Nie, P.T. Yap, and D. Shen. IEEE Journal of Biomedical and Health Informatics, 2018.
0.431
Ove
rall
Ac
cu
rac
y
0.404
0.46
7
0.486 0.487
0.518
0.325
Co
rre
lati
on
Co
eff
icie
nt
0.289
0.468
0.492
0.538
0.567
Classification Results for AD vs. sMCI vs. pMCI vs. NC Regression Results for MMSE
Results of Classification and Regression
M. Liu, J. Zhang, E. Adeli, and D. Shen. Medical Image Analysis, 2018.
M. Liu, J. Zhang, D. Nie, P.T. Yap, and D. Shen. IEEE Journal of Biomedical and Health Informatics, 2018.
End-to-end Disease Diagnosis with sMRI
• Hierarchical Fully Convolutional network
C. Lian, M. Liu, J. Zhang, and D. Shen. IEEE Trans. on Pattern Analysis and Machine Intelligence, 2019.
M. Liu, J. Zhang, C. Lian, and D. Shen. IEEE Trans. on Cybernetics, 2019.
M. Liu, J. Zhang, E. Adeli, and D. Shen. IEEE Trans. on Biomedical Engineering, 2019.
M. Liu, J. Zhang, E. Adeli, and D. Shen. Medical Image Analysis, 2018.
M. Liu, J. Zhang, D. Nie, P.T. Yap, and D. Shen. IEEE Journal of Biomedical and Health Informatics, 2018.
J. Zhang, M. Liu, and D. Shen. IEEE Trans. on Image Processing, 2017.
15
Hierarchical Network for ROI Identification
• Hierarchical Fully Convolutional Network (H-FCN)
– Automatically and identify disease-related ROIs in the whole sMR image
– Jointly learn multi-scale features and construct a classification model
C. Lian, M. Liu, J. Zhang, and D. Shen. IEEE Trans. on Pattern Analysis and Machine Intelligence, 2019.
C. Lian, M. Liu*, L. Wang, and D. Shen. MICCAI 2019.
Hierarchical Network for ROI Identification
Input:
sMRI
C. Lian, M. Liu, J. Zhang, and D. Shen. IEEE Trans. on Pattern Analysis and Machine Intelligence, 2019.
C. Lian, M. Liu*, L. Wang, and D. Shen. MICCAI 2019.
Input:
sMRI
1) Location
proposals
Hierarchical Network for ROI Identification
C. Lian, M. Liu, J. Zhang, and D. Shen. IEEE Trans. on Pattern Analysis and Machine Intelligence, 2019.
C. Lian, M. Liu*, L. Wang, and D. Shen. MICCAI 2019.
1) Location
proposals
Input:
sMRI
2) Patch-level sub-
networks (PSN) (shared
weights)
Hierarchical Network for ROI Identification
PSN
PSN
Class_P
64 64 64128 12832
C. Lian, M. Liu, J. Zhang, and D. Shen. IEEE Trans. on Pattern Analysis and Machine Intelligence, 2019.
C. Lian, M. Liu*, L. Wang, and D. Shen. MICCAI 2019.
PSN
1) Location
proposals
… …
PSN
PSN
PSN
PSN
PSN
PSN
PSN
Input:
sMRI
2) Patch-level sub-
networks (PSN) (shared
weights)
Hierarchical Network for ROI Identification
PSN
Class_P
64 64 64128 12832
C. Lian, M. Liu, J. Zhang, and D. Shen. IEEE Trans. on Pattern Analysis and Machine Intelligence, 2019.
C. Lian, M. Liu*, L. Wang, and D. Shen. MICCAI 2019.
…
64
Conv_R Class_R
…
PSN
PSN
PSN
PSN
PSN
PSN
PSN
PSN
2) Patch-level sub-
networks (PSN) (shared
weights)
3) Region-level
sub-networks
1) Location
proposals
Input:
sMRI
Hierarchical Network for ROI Identification
PSN
Class_P
64 64 64128 12832
C. Lian, M. Liu, J. Zhang, and D. Shen. IEEE Trans. on Pattern Analysis and Machine Intelligence, 2019.
C. Lian, M. Liu*, L. Wang, and D. Shen. MICCAI 2019.
…
64
Conv_R Class_R
…
64
Conv_R Class_R
…
PSN
PSN
PSN
PSN
PSN
PSN
PSN
PSN
2) Patch-level sub-
networks (PSN) (shared
weights)
3) Region-level
sub-networks
1) Location
proposals
Input:
sMRI
Hierarchical Network for ROI Identification
PSN
Class_P
64 64 64128 12832
C. Lian, M. Liu, J. Zhang, and D. Shen. IEEE Trans. on Pattern Analysis and Machine Intelligence, 2019.
C. Lian, M. Liu*, L. Wang, and D. Shen. MICCAI 2019.
…
64
Conv_S
Class_S
64
Conv_R Class_R
64
Conv_R Class_R
… …
Output:
Class label
PSN
PSN
PSN
PSN
PSN
PSN
PSN
PSN
3) Region-level
sub-networks
4) Subject-level
sub-network
1) Location
proposals
Input:
sMRI
2) Patch-level sub-
networks (PSN) (shared
weights)
Classification (1 × 1 × 1 Conv)
4 × 4 × 4 Conv
1 × 1 × 1 Conv
Channel concatenation
Region-level Conv
2 × 2 × 2 max pooling
3 × 3 × 3 Conv
Spatial concatenation
Subject-level Conv
Potentially pruned sub-networks
Skipped connection Conv: Convolution
PSN
Class_P
64 64 64128 12832
Hierarchical Network for ROI Identification
C. Lian, M. Liu, J. Zhang, and D. Shen. IEEE Trans. on Pattern Analysis and Machine Intelligence, 2019.
C. Lian, M. Liu*, L. Wang, and D. Shen. MICCAI 2019.
Network Pruning: To remove less informative regions
Identified Voxel-level Discriminative Locations
1
2
3
1 2 3
1 2 3
1
2
3 4
3 4
3 4
1 2
1 2
1 2 3
1 2 3
1 2 3
1 2 3
1
2 3
1 2
1 2
12
12
3
1 2 3
1 2 3
12
3
(a) AD Subject #1 (b) AD Subject #2 (c) AD Subject #3
(d) AD Subject #4 (e) AD Subject #5 (f) AD Subject #6
C. Lian, M. Liu, J. Zhang, and D. Shen. IEEE Trans. on Pattern Analysis and Machine Intelligence, 2019.
C. Lian, M. Liu*, L. Wang, and D. Shen. MICCAI 2019.
Identified Patch-level Discriminative Locations
Sagittal View Axial View Coronal View 3D View
C. Lian, M. Liu, J. Zhang, and D. Shen. IEEE Trans. on Pattern Analysis and Machine Intelligence, 2019.
C. Lian, M. Liu*, L. Wang, and D. Shen. MICCAI 2019.
Identified Region-level Discriminative Locations
Sagittal View Axial View Coronal View 3D View
C. Lian, M. Liu, J. Zhang, and D. Shen. IEEE Trans. on Pattern Analysis and Machine Intelligence, 2019.
C. Lian, M. Liu*, L. Wang, and D. Shen. MICCAI 2019.
0.5
0.6
0.7
0.8
0.9
1
Patch Region Subject
AC
C
0.5
0.6
0.7
0.8
0.9
1
Patch Region Subject
AU
C
H-FCN before network pruning H-FCN after network pruning
Results of AD vs. NC classification obtained by patch-, region-, and subject-level sub-
networks in H-FCN without /with network pruning
Classification Results
C. Lian, M. Liu, J. Zhang, and D. Shen. IEEE Trans. on Pattern Analysis and Machine Intelligence, 2019.
C. Lian, M. Liu*, L. Wang, and D. Shen. MICCAI 2019.
1) Global subject-level representation is more useful
2) Network pruning promotes the classification performance
Outline
• Missing Data
• Multi-modal Data Fusion
• Domain Adaptation
Multi-modal Neuroimage
CSFPETsMRI
Single-modal Neuroimage
• Structural MRI (sMRI)
sMRI
Part II. Multi-modal Neuroimage Analysis
• Multi-modality Fusion for Disease Diagnosis
• Imaging Synthesis for Missing Modalities
• Domain Adaptation for Multi-site Data
M. Liu, J. Zhang, P.-T. Yap, and D. Shen. MICCAI, 2016.
M. Liu, J. Zhang, P.-T. Yap, and D. Shen. Medical Image Analysis, 2017.
M. Liu, Y. Gao, P.-T. Yap, and D. Shen. IEEE Journal of Biomedical and Health Informatics, 2018.
Multi-modality Fusion for Disease Diagnosis
• Hypergraph Learning with Missing Data
30
Multi-View Data Grouping
• Given 3 modalities (i.e., MRI, PET and CSF), 6 views are constructed
according to the availability of different modalities
Multi-Modality Data
CSFPET MRI
Multi-ViewData Grouping
View 1
View 2
View 3
View 4
View 5
View 6
PET
CSF
MRI
Missing Data
* MICCAI Young Scientist Award Nomination, 2016
* MICCAI Travel Award, 2016
31
Multi-View Data Grouping
• Given 3 modalities (i.e., MRI, PET and CSF), 6 views are constructed
according to the availability of different modalities
Multi-Modality Data
CSFPET MRI
Multi-ViewData Grouping
View 1
View 2
View 3
View 4
View 5
View 6
PET
CSF
MRI
Missing Data
* MICCAI Young Scientist Award Nomination, 2016
* MICCAI Travel Award, 2016
• Construct multiple hypergraphs, with each hypergraph corresponding to
a specific view
Multi-Modality Data
CSFPET MRI
Multi-ViewData Grouping
View 1
View 2
View 3
View 4
View 5
View 6
Sparse
Representation
Sparse
Representation
Sparse Representation based Hypergraph Construction
Sparse
Representation
Sparse
Representation
Sparse
Representation
Sparse
Representation
PET
CSF
MRI
Missing Data
M. Liu, J. Zhang, P.-T. Yap, and D. Shen. MICCAI, 2016.
M. Liu, J. Zhang, P.-T. Yap, and D. Shen. Medical Image Analysis, 2017.
M. Liu, Y. Gao, P.-T. Yap, and D. Shen. IEEE Journal of Biomedical and Health Informatics, 2018.
Hypergraph Construction
• A view-aligned hypergraph classification model to capture the coherence
among different views
View-Aligned
Hypergraph Classification
View-Aligned Hypergraph Classification
Multi-Modality Data
CSFPET MRI
Multi-ViewData Grouping
View 1
View 2
View 3
View 4
View 5
View 6
Sparse
Representation
Sparse
Representation
Sparse Representation based Hypergraph Construction
Sparse
Representation
Sparse
Representation
Sparse
Representation
Sparse
Representation
PET
CSF
MRI
Missing Data
M. Liu, J. Zhang, P.-T. Yap, and D. Shen. MICCAI, 2016.
M. Liu, J. Zhang, P.-T. Yap, and D. Shen. Medical Image Analysis, 2017.
M. Liu, Y. Gao, P.-T. Yap, and D. Shen. IEEE Journal of Biomedical and Health Informatics, 2018.
• Multi-view label fusion strategy (weighted voting)
Multi-View
Label Fusion
Multi-Modality Data
CSFPET MRI
Multi-ViewData Grouping
View 1
View 2
View 3
View 4
View 5
View 6
Sparse
Representation
Sparse
Representation
Sparse Representation based Hypergraph Construction
Sparse
Representation
Sparse
Representation
Sparse
Representation
Sparse
Representation
View-Aligned
Hypergraph Classification
PET
CSF
MRI
Missing Data
M. Liu, J. Zhang, P.-T. Yap, and D. Shen. MICCAI, 2016.
M. Liu, J. Zhang, P.-T. Yap, and D. Shen. Medical Image Analysis, 2017.
M. Liu, Y. Gao, P.-T. Yap, and D. Shen. IEEE Journal of Biomedical and Health Informatics, 2018.
Multi-view Label Fusion
Label Space
𝐱𝑛1MRI
𝐱𝑛1PET 𝐱𝑛1
CSF
𝑓𝑛1MRI
𝑓𝑛1PET
𝑓𝑛1CSF
𝑓𝑛2MRI
𝑓𝑛2PET
𝐱𝑛2MRI𝐱𝑛2
PET
View-Aligned Constraint
Coherence among views
View-Aligned Regularizer
M. Liu, J. Zhang, P.-T. Yap, and D. Shen. MICCAI, 2016.
M. Liu, J. Zhang, P.-T. Yap, and D. Shen. Medical Image Analysis, 2017.
M. Liu, Y. Gao, P.-T. Yap, and D. Shen. IEEE Journal of Biomedical and Health Informatics, 2018.
Missing CSF
MRI
PET
Subject #2
CSF
MRI
PET
Subject #1
36
𝑚𝑖𝑛𝐅, 𝛂, {𝐖𝑚}𝑚=1
𝑀 𝑚=1𝑀 𝛀𝑚 (𝐟𝑚 − 𝐲) 2
2
+𝜆 𝑚=1𝑀 𝐖𝑚
𝐹2
+𝜇 𝑛=1𝑁 𝑚=1
𝑀 𝑝=1𝑀 𝛺𝑛,𝑛
𝑚 𝛺𝑛,𝑛𝑝
𝑓𝑛𝑚 − 𝑓𝑛
𝑝 2
+ 𝑚=1𝑀 𝛼𝑚 2 𝐟𝑚 T 𝐋𝑚𝐟𝑚
𝑠. 𝑡. 𝑚=1𝑀 𝛼𝑚 = 1, ∀𝛼𝑚 ≥ 0;
𝑖=1𝑁𝑒𝑚
𝑊𝑖,𝑖𝑚 = 1, ∀𝑊𝑖,𝑖
𝑚 ≥ 0.
Step 3: View-Aligned Hypergraph Classification
Formulation of view-aligned hypergraph classification (VAHC)
View-aligned Regularizer
Hypergraph Laplacian matrix , and .
M. Liu, J. Zhang, P.-T. Yap, and D. Shen. MICCAI, 2016.
M. Liu, J. Zhang, P.-T. Yap, and D. Shen. Medical Image Analysis, 2017.
M. Liu, Y. Gao, P.-T. Yap, and D. Shen. IEEE Journal of Biomedical and Health Informatics, 2018.
Learned Weights for Views
Learned weights for views in four classification tasks
AD vs. NC MCI vs. NC pMCI vs. sMCI pMCI vs. NC
0.0
0.1
0.2
0.3
0.4
0.5
0.6
We
igh
ts
MRI PET CSF PET+MRI
MRI+CSF PET+MRI+CSF
M. Liu, J. Zhang, P.-T. Yap, and D. Shen. MICCAI, 2016.
M. Liu, J. Zhang, P.-T. Yap, and D. Shen. Medical Image Analysis, 2017.
M. Liu, Y. Gao, P.-T. Yap, and D. Shen. IEEE Journal of Biomedical and Health Informatics, 2018.
Using all 3 modalities (MRI+PET+CSF) achieves the best results
Experimental Results
pMCI vs. sMCI classification
ACC SEN SPE BAC PPV NPV AUC
50
60
70
80
90
100
Re
su
lts (
%)
(c) pMCI vs. sMCI classification
Zero KNN EM SVD VAHL
M. Liu, J. Zhang, P.-T. Yap, and D. Shen. MICCAI, 2016.
M. Liu, J. Zhang, P.-T. Yap, and D. Shen. Medical Image Analysis, 2017.
M. Liu, Y. Gao, P.-T. Yap, and D. Shen. IEEE Journal of Biomedical and Health Informatics, 2018.
Imaging Synthesis for Missing Modalities
• Deep Learning based Automatic PET Synthesis from sMRI
Y. Pan, M. Liu*, C. Lian, Y. Xia, and D. Shen. MICCAI, 2019.
Y. Pan, M. Liu, C. Lian, T. Zhou, Y. Xia, and D. Shen. MICCAI, 2018.
Y. Pan, M. Liu*, C. Lian, L. Yue, S. Xiao, Y. Xia and D. Shen. ISMRM, 2019
Pre-processed
MR and PET
Images
Stage 2:
Brain Disease Classification
Stage 1:
PET Image Synthesis
based on sMRI
Synthesizing Missing PET based on MRI for Brain Disease Diagnosis
Hybrid Cycle-GAN for Missing PET Synthesis
Y. Pan, M. Liu, C. Lian, T. Zhou, Y. Xia, and D. Shen. MICCAI, 2018
Y. Pan, M. Liu*, C. Lian, L. Yue, S. Xiao, Y. Xia and D. Shen. ISMRM, 2019
• Among 800+ subjects in ADNI, all subjects have sMRI, while
only half of them have FDG-PET scans.
MRI PET
Hybrid cycle-consistent generative adversarial network (HGAN) to impute missing PET scans based on sMRI
Synthetic PET
Synthetic MRI3232
Residual Net
Block (RNB)
Real PET
1 128 64 32 16
𝑫𝑷
112864
3216
𝑫𝑴
1 16 32 32 16 8
𝑮𝑴
⋯
RN
B
RN
B
8 16 32
𝑮𝑷
⋯
RN
B
RN
B
32 16 1
3×3×3 Convolution
7×7×7 Convolution
3×3×3 Deconvolution
4×4×4 Convolution
Addition
Real MRI
6
6
1/01/0
𝐗𝑀
𝐗𝑃
𝐺1 𝐗𝑀
𝐺2 𝐗𝑃
Hybrid Cycle-GAN for Missing PET Synthesis
Y. Pan, M. Liu, C. Lian, T. Zhou, Y. Xia, and D. Shen. MICCAI, 2018
Y. Pan, M. Liu*, C. Lian, L. Yue, S. Xiao, Y. Xia and D. Shen. ISMRM, 2019
Synthetic PET ImagesP
ET
(RID
:5016)
Y. Pan, M. Liu, C. Lian, T. Zhou, Y. Xia, and D. Shen. MICCAI, 2018
Y. Pan, M. Liu*, C. Lian, L. Yue, S. Xiao, Y. Xia and D. Shen. ISMRM, 2019
(a) GAN (b) CGAN (c) VGAN (d) HGAN (Ours) (e) Ground Truth
PE
T(R
ID:
4352)
Synthetic MRI ScansM
RI
(RID
:5016)
Y. Pan, M. Liu, C. Lian, T. Zhou, Y. Xia, and D. Shen. MICCAI, 2018
Y. Pan, M. Liu*, C. Lian, L. Yue, S. Xiao, Y. Xia and D. Shen. ISMRM, 2019
(a) GAN (b) CGAN (c) VGAN (d) HGAN (Ours) (e) Ground Truth
MR
I(R
ID:
4352)
PET
(Real + Synthetic)
MRI (Real)
24
2424
24
24
24
24
2424
24
2424
…
3232 32 6464 64
Sub-network 1
DCM DCM DCM
1616 16
Sub-network 2
3232 32 6464 64
DCM DCM DCM
1616 16
Concatenation
32
8K …
Clinical scores at
four time-points
8
32
8
128
32
BL
M06
M24
M12
Fully-connectedDown-sampling Copy 3×3×3 Convolution 2×2×2 Max-poolingChannel concatenation
Hybrid Cycle-GAN for Missing PET Synthesis
Landmark-based Deep Network for Brain Disease Classification using MRI and PET (Real+Synthetic)
Y. Pan, M. Liu, C. Lian, T. Zhou, Y. Xia, and D. Shen. MICCAI, 2018
Y. Pan, M. Liu*, C. Lian, L. Yue, S. Xiao, Y. Xia and D. Shen. ISMRM, 2019
PET
(Real + Synthetic)
MCI conversion prediction with complete MRI
and complete (after imputation) PET
Hybrid Cycle-GAN for Missing PET Synthesis
Y. Pan, M. Liu, C. Lian, T. Zhou, Y. Xia, and D. Shen. MICCAI, 2018
Y. Pan, M. Liu*, C. Lian, L. Yue, S. Xiao, Y. Xia and D. Shen. ISMRM, 2019
How to generating classification-oriented
PET/MRI scans for diagnosis?
Feature-consistent GAN for Joint PET Synthesis and Classification
1128643216
𝐷
3×3×3 Conv
7×7×7 Conv
3×3×3 Deconv
4×4×4 Conv
Addition
Difference
32 32
RNB
1163232168
𝐺
RN
B
RN
B ⋯
6
𝐺𝑀
𝐺𝑃
𝐷𝑃
𝐷𝑀 𝔏𝑔
𝔏𝑔
𝐹𝑃
𝐹𝑀 𝔏c
𝔏c
FG
AN
Synthetic
MRI
Real
PET
Synthetic
PET
Real
MRI
Synthetic
PET
Real
PET
1 2
K
1
2
K
⋮1/0
𝑙2
64643216 64
64643216
𝔏c
64
Feature-consistent
Component 𝐹𝑃
Cosin
e K
ern
el
Y. Pan, M. Liu*, C. Lian, Y. Xia, and D. Shen. MICCAI, 2019
Feature-consistent
Component 𝐹𝑃
30
50
70
90
AUC ACC SEN SEP
ROI LMF LDMIL DSNN (Ours)
MCI conversion prediction with complete MRI
and complete (after imputation) PET
Feature-consistent GAN for Joint PET Synthesis and Classification
Y. Pan, M. Liu*, C. Lian, Y. Xia, and D. Shen. MICCAI, 2019
Generating task-oriented PET scans helps boost performance
Domain Adaptation for Multi-site Data
• Low-rank Representation for Multi-site Data Adaptation
M. Wang, D. Zhang, J. Huang, D. Shen, and M. Liu*, and D. Shen. MICCAI, 2018
Low-rank Representation for Domain Adaptation
ABIDE: 17 imaging sites with resting-state fMRI data
Scanners
ScanningParameters
PopulationNoise Level
ImageContrast
M. Wang, D. Zhang, J. Huang, D. Shen, and M. Liu*, and D. Shen. MICCAI, 2018
*Best Poster Award, MICS, 2019
50
…
P2
P1
PS
P
New Representation
Latent Representation Space
Linear
Representation
Site T
Site S
Site 1
Site 2
Low-rank Representation for Domain Adaptation
M. Wang, D. Zhang, J. Huang, D. Shen, and M. Liu*, and D. Shen. MICCAI, 2018
Mapping to a
common latent spaceRepresenting source
data using target data
Source Domains
Target Domain
*Best Poster Award, MICS, 2019
51
Low-rank Representation for Domain Adaptation
Formulation of multi-center low-rank representation (maLRR)
min𝐉,𝐏,𝐏𝑖,𝐙𝑖,𝐄𝑆𝑖 ,𝐄𝑃𝑖 ,𝐅𝑖𝐉 ∗ + 𝑖=1
𝐾 𝐅𝑖 ∗ + 𝛼 𝐄𝑆𝑖 1+ 𝛽 𝐄𝑃𝑖 1
s. t. 𝐏𝑖𝐗𝑆𝑖 = 𝐏𝐗𝑇𝐙𝑖 + 𝐄𝑆𝑖 ,
𝐏𝑖= 𝐏 + 𝐄𝑃𝑖 , 𝑖 = 1,… , 𝐾
𝐏 = 𝐉, 𝐙𝑖 = 𝐅𝑖 , 𝐏𝐏𝑇 = 𝐈1) Mapping to a latent space
2) Representing source data
using target data
M. Wang, D. Zhang, J. Huang, D. Shen, and M. Liu*, and D. Shen. MICCAI, 2018
*Best Poster Award, MICS, 2019
52
Low-rank Representation for Domain Adaptation
Performance of different methods in the task of Autism vs. NC classification, with
NYU as the target domain and the other four sites as the source domains
ABIDE: 5 imaging sites with resting-state fMRI data
M. Wang, D. Zhang, J. Huang, D. Shen, and M. Liu*, and D. Shen. MICCAI, 2018
*Best Poster Award, MICS, 2019
Outline
• Missing Data
• Multi-modal Data Fusion
• Domain Adaptation
Multi-modal Neuroimage
CSFPETsMRI
Single-modal Neuroimage
• Structural MRI (sMRI)
sMRI
Acknowledge
Dr. Dinggang Shen
Dr. Daoqiang Zhang
Dr. Pew-Thian Yap
Dr. Jun Zhang
Dr. Chunfeng Lian
Dr. Ling Yue
Dr. Jing Zhang
Dr. Aimei Dong
Dr. Bo Wang
Collaborators
Dr. Ehsan Adeli
Dr. Yue Gao
Dr. Biao Jie
Dr. Tao Zhou
Dr. Yong Xia
Visiting Scholars and Students
Mr. Mingliang Wang
Mr. Yongsheng Pan
Mr. Dongren Yao
Mr. Jiashuang Huang
Thanks for Your Attention!
http://mingxia.web.unc.edu/
Mingxia Liu