Artificial Intelligence (AI) in Medicine · § To innovate, develop, and apply artificial...
Transcript of Artificial Intelligence (AI) in Medicine · § To innovate, develop, and apply artificial...
Radiation Oncology
Artificial Intelligence (AI) in Medicine:
Challenges and Opportunities
Steve Jiang, Ph.D.
Barbara Crittenden Professor in Cancer ResearchDirector, Medical Artificial Intelligence and
Automation (MAIA) LabVice Chair, Department of Radiation Oncology
Radiation Oncology
AI Is Changing The World
2 © Steve Jiang, Ph.D., MAIA Lab, 2020
Radiation Oncology
AI Is Going to Transform Healthcare
3 © Steve Jiang, Ph.D., MAIA Lab, 2020
http://time.com/5556339/artificial-intelligence-robots-medicine/
Radiation Oncology
AI Research at MAIA Lab
4
Radiation Oncology
Medical Artificial Intelligence and Automation LabMAIA Lab
5 © Steve Jiang, Ph.D., MAIA Lab, 2019
Since 07/2017
Radiation Oncology
§ To innovate, develop, and apply artificial intelligence technologies to empower clinicians, especially those with less experience or limited resources, for improved patient care, from three aspects:
§ Improving accuracy by retrieving hidden information from patient data/images and performing better than (most) physicians when there is ground truth– Precise diagnosis and better clinical decision making
§ Improving efficiency by automating clinical procedures and saving physicians’ time in front of computers– Re-humanize medicine
§ Transferring expertise to less experienced physicians by learning from experienced physicians when there is no ground truth– Reduce healthcare disparities
6 © Steve Jiang, Ph.D., MAIA Lab, 2019
Mission of MAIA Lab
Radiation Oncology
§ To empower clinicians with AI– Not trying to replace
clinicians with AI
§ To automate and augment clinical procedures with AI– Not focused on
informatics/data science
§ To reduce healthcare disparities with AI– The most impactful
application of AI in medicine
7 © Steve Jiang, Ph.D., MAIA Lab, 2020
Research Focuses of MAIA Lab
Radiation Oncology
Medical Image Segmentation§ Most organ segmentation problems are easy§ We focus on – Auto segmentation of challenging organs– Auto delineation of gross tumors and clinical targets
8 © Steve Jiang, Ph.D., MAIA Lab, 2017
Radiation Oncology
AI for Radiotherapy Treatment Planning§ Supervised learning to learn the relationship between patient
anatomy and optimal radiation dose§ Reinforcement learning to explore the large hyperparameter
space for treatment plan optimization§ Imitation learning to learn from experienced treatment planners– Similar to how to train AI playing video games or driving cars
9 © Steve Jiang, Ph.D., MAIA Lab, 2020
Nguyen, …, Jiang, arXiv:1805.10397, 2018; Phys Med Biol. 64(6):065020, 2019.
GT Pred
Radiation Oncology
Dose Prediction – Feasibility Study (9/2017)
© Dan Nguyen, Ph.D. and Steve Jiang, Ph.D., MAIA Lab, 201710
PTV
Bladder
L FemHeadR FemHead
Rectum
Body
Nguyen, …, Jiang, Sci Rep. 9(1):1076, 2019.
Radiation Oncology © Dan Nguyen, Ph.D. and Steve Jiang, Ph.D., MAIA Lab, 201811
Dose Prediction – H&N VMAT w/ HD U-NET
Nguyen, …, Jiang, Phys Med Biol. 64(6):065020, 2019.
Radiation Oncology © Chenyang Shen, Ph.D. and Xun Jia, Ph.D., MAIA Lab, 2018
HDR Planning w/ DRL Based Organ Weight Tuning
12
Shen,..., Jia, Phys Med Biol. 64(11):115013, 2019.
Radiation Oncology13
IMRT Planning w/ DRL Based Hyper-Parameter Tuning
© Chenyang Shen, Ph.D. and Xun Jia, Ph.D., MAIA Lab, 2020
Radiation Oncology
Dose Calculation using Deep Learning§ Deep learning can learn the difference between a simple and a sophisticated dose
calculation method– scatter dose and inhomogeneity effort
§ A completely different system is good for secondary dose check§ It can be used when real-time efficiency and reasonable accuracy are needed– Intermediate step dose calculation during plan optimization– Plan casting for online ART
14
HD Unet
© Penelope Xing, Ph.D. and Steve Jiang, Ph.D., MAIA Lab, 2018
Xing, …, Jiang, Med Phys 47(2)753-758, 2020
Radiation Oncology
– 120 lung cases in Eclipse (72 training/18 validation/30 testing)– Non-coplanar 3D CRT, 3D conformal arc, IMRT, and VMAT plans– Rx dose: 24 Gy to 60 Gy– Energy: 6 MV, 10 MV, 6xFFF, and 10xFFF
Dose Conversion: From AAA to Acuros XB
15 © Penelope Xing, Ph.D. and You Zhang Ph.D., MAIA Lab, 2018
Xing, Zhang, …, Jiang, JACMP DOI: 10.1002/acm2.12937, 2020
Radiation Oncology16
Convert PB Dose to MC Dose for Proton RT§ MGH Data: Pencil Beam (XiO) à Monte Carlo (TOPAS)
© Chao Wu and Steve Jiang, MAIA Lab, 2019
Predict doseComposite doseRotate BackRotate
Rotate Rotate Back
HD-UNET
Beam1
Beam2
HN Liver Prostate Lung
Number of patients 90 93 75 32
Number of beams 726 218 260 91
Pencil Beam vs MC (73.3±6.3) % (79.2±5.1) % (73.3±2.7) % (65.4±5.3) %
Predicted vs MC (92.8±2.9) % (92.7±2.9) % (99.6±0.3) % (89.7±3.8) %Gamma index (1%/1 mm)
Radiation Oncology
AI for Beam Orientation Optimization (BOO)§ Develop an AlphaGo type of DL algorithm – reinforcement learning (RL) policy network – Monte Carlo Tree Search (MCTS)
§ Go movements à CyberKnife robot sequence
© Dan Nguyen, Ph.D. and Steve Jiang, Ph.D., MAIA Lab, 2018
Ogunmolu, ..., Jiang, AAPM, 2018.
17
Radiation Oncology
DeepBOO V1: Policy Network Trained w/ Column Generation
© Dan Nguyen, Ph.D. and Steve Jiang, Ph.D., MAIA Lab, 201818
Sadeghnejad Barkousaraie, …, Nguyen, Med Phys 47(3)880-897, 2020.
Use column generation (CG) to train a supervised learning (SL) policy network
298⁰
174⁰
36⁰
242⁰
84⁰86⁰
212⁰240⁰
306⁰354⁰
Radiation Oncology19 © Azar Sadeghnejad Barkousaraie, Ph.D., and Dan Nguyen, Ph.D., MAIA Lab, 2019
DeepBoo V2: Guided Monte Carlo Tree Search§ Policy network quickly generates a sampling probability at each level§ Tree builds its own policy over time– Search policy is updated as a weighted sum of NN policy and tree policy
Radiation Oncology20 © Azar Sadeghnejad Barkousaraie, Ph.D., and Dan Nguyen, Ph.D., MAIA Lab, 2020
DeepBoo V3: w/o the need of pre-dose calculation
§ DNN is updating during the tree search after exploring each leaf§ No prior dose calculation is needed:– Using a pre-trained network to measure the quality of the treatment plan
Self Improving Tree
Back propagate
Deep BOO (v1)
Radiation Oncology
Convert CT Image to Ventilation Image§ Generating functional lung ventilation image from anatomical
4D CT images using CNN
21
Input Predicted Truth Predicted on CT
© Yuncheng Zhong, Ph.D. and Jing Wang, Ph.D., MAIA Lab, 2018
Zhong, …, Wang, Med Phys 2019 doi: 10.1002/mp.13421, arXiv:1808.06982
Radiation Oncology
AI for Cervical Lymph Node Malignancy Prediction
Chen,…, Wang, PMB, vol. 64, 075011 (13pp), 2019
q There is often uncertainty about the malignant potential of lymph nodes (LNs) in head and neck cancer
q INRT- AIR: A Prospective Phase II Study of Involved Nodal Radiation Therapy Using AI-Based Radiomics for Head and Neck Squamous Cell Carcinoma (NCT03953976 PI: David Sher)
q Predict the risk of malignancy in otherwise benign-appearing lymph nodes, using pathology as ground truth
q First clinical trial in rad onc to use AI to improve physician targetingq Already enrolled > 30 patients in 4 monthsq Expected phase III clinical trial comparing AI-based involved nodal
radiation with standard-of-care radiotherapy
© Jing Wang, Ph.D. And David Sher, M.D., MAIA Lab, 2018
Radiation Oncology
§ Infrastructure in EC building (BLE)§ Tracking patients, clinical staff, and device § Applications in clinic– A touchpad checklist before treatment to ensure correct steps are taken
and correct devices are used – Auto recording of chart round/seminar attendance – Exam room workflow management
© Steve Jiang, Ph.D., MAIA Lab, 2019
AI for Smart Clinic
23Iqbal, …, Jiang, arXiv:1711.08149, 2017Tang, …, Jiang, arXiv:1907.10554, 2019
Radiation Oncology
Implementing AI in Clinical Practice:Challenges and Solutions
24
Radiation Oncology
§ Dataset size– Dataset size is often limited for deep learning
§ Lateral data heterogeneity– Your AI model may not work for others
§ Lack of ground truth– Medicine is still an art in many cases
§ Model bias– AI model is as good as people/data learns from
§ Longitudinal data variation– Model performance will decline with time
§ Clinical data quality– Not all clinical data are created with high quality
§ Data curation and cleaning– Very expensive to curate clinical data
§ Model explainability– Model explainability is required for AI in medicine
§ Model robustness– Adversarial attacks are possible
25 © Steve Jiang, Ph.D., MAIA Lab, 2020
General Problems for AI in Medicine
Radiation Oncology
§ Distributions in latent space can be different for different institutions, machines, scanners, clinicians, etc
§ Model trained w/ Dataset A may perform poorly for Dataset B§ Not always the more data the better results
26 © Steve Jiang, Ph.D., MAIA Lab, 2019
Challenge: Data Heterogeneity (lateral/spatial variation)
Radiation Oncology27 © Steve Jiang, Ph.D., MAIA Lab, 2019
Breast and Skin Cancer Prediction (UK Biobank data)
Ghorbani and Zou, Data Shapley: Equitable Valuation of Data for Machine Learning, arXiv:1904.02868
Radiation Oncology
§ Distributions in latent space can be different for different institutions, machines, scanners, clinicians, etc
§ Model trained w/ Dataset A may perform poorly for Dataset B§ Not always the more data the better results
28 © Steve Jiang, Ph.D., MAIA Lab, 2020
Challenge: Data Heterogeneity (lateral/spatial variation)
§ What many people do– Train and test the model with your own data– No idea about model generalizability
§ What we are promoting– Train the model with your data and test it with others data– If works, show some generalizability. Enough?– How to demonstrate the model generalizability efficiently and sufficiently– Universal model (works anywhere anytime for anybody)?
Radiation Oncology
AI for Mammography Breast Cancer Screening
McKinney, Sieniek, Godbole, Godwin et al, Nature 577, 89–94(2020) (Google Health etc)
§ Trained the modelon UK datasets (25,856 women, AUC 0.889)
§ Tested on an USA dataset (3,097 women, AUC 0.810)
§ Outperformed 6 radiologists
§ “We provide evidence of the ability of the system to generalize from the UK to the USA.”– Most images were acquired with devices of a same vendor
© Steve Jiang, Ph.D., MAIA Lab, 202029
Radiation Oncology
§ Distributions in latent space can be different for different institutions, machines, scanners, clinicians, etc
§ Model trained w/ Dataset A may perform poorly for Dataset B§ Not always the more data the better results
30 © Steve Jiang, Ph.D., MAIA Lab, 2020
Challenge: Data Heterogeneity (lateral/spatial variation)
§ What many people do– Train and test the model with your own data– No idea about model generalizability
§ What we are promoting– Train the model with your data and test it with others data– If works, show some generalizability. Enough?– How to demonstrate the model generalizability efficiently and sufficiently– Universal model (works anywhere anytime for anybody)?
§ What is our solution– Train the model with your data. When deploy the model, automated
transfer learning with a small local dataset of end users– Model commissioning. Different models for different users.
Radiation Oncology
Dose Prediction - Model Generalizability
31 © Roya Kandalan and Steve Jiang,, MAIA Lab, 2019
Radiation Oncology
CBCT to CT Conversion – Model Generalizability
© Xiao Liang and Steve Jiang, Ph.D., MAIA Lab, 201832
Source dataset
Target dataset
§ Test dataset: Prostate 1§ Same vendor§ Different anatomical sites
§ Test dataset: Prostate 2§ Different vendor§ Different anatomical sites
Liang,..., Jiang, arXiv:1810.13350, 2018; PhysMed Biol, doi: 10.1088, 2019.
Radiation Oncology
§ Medicine is still an art in many cases§ Evidence based medicine and clinical guidelines only
give physicians the floor not the ceiling§ There is room for physicians to exercise their own
judgements§ Variation exists in physicians’ clinical practice§ There is often no ground truth to tell which one is
the best § How AI can help?
33 © Steve Jiang, Ph.D., MAIA Lab, 2019
Challenge: Lack of Ground Truth
à Personalized AI for Physicians
Radiation Oncology
§ Prostate gland has been surgically removed
§ Residual tumor cells are not visible in CT or MRI images
§ Not an object with defined boundary by image contrast
34 © Anjali Balagopal and Steve Jiang, Ph.D., MAIA Lab, 2019
Project: Post-op Prostate CTV Segmentation
§ Currently, manually contoured by physicians following clinical guidelines (such as RTOG) by considering – relationship with nearby organs– pathology report– pre-operative MRI – individual patient characteristics– variability in anatomy and co-morbidities
§ Large variation among physicians– Personal judgements for individual patients based on training
background, experience, and personal preference
Radiation Oncology35 © Anjali Balagopal and Steve Jiang, Ph.D., MAIA Lab, 2019
Anatomy-guided multi-task network (AG-MTN) for CTV segmentation
A deep learning framework
Project: Post-op Prostate CTV Segmentation
Radiation Oncology36
CTV Uncertainty Maps using Monte Carlo Dropout
© Anjali Balagopal, Dan Nguyen, and Steve Jiang, MAIA Lab, 2020
Radiation Oncology
COMPARISON WITH RESIDENT CONTOURS§ Residents contoured the CTV on test patients
§ Resident contours compared to the Physician contours using volume overlap
37
58%
63%
68%
73%
78%
83%
88%
93%
Physician vs Model Physician vs Resident
© Anjali Balagopal, Dan Nguyen, and Steve Jiang, MAIA Lab, 2020
Radiation Oncology38
CLINICAL EVALUATION BY PHYSICIANS§ 30 test patients contours were evaluated by experienced practicing physicians
§ Patients were completely anonymized
§ Physicians were asked to evaluate two contours - original physician contour and the AI contour and score each of them from 1-4
4 – Acceptable without changes3 -- Acceptable with minor changes2 -- Acceptable with major changes1 – Unacceptable
§ Physicians were also asked to guess which of the two contours was the physician contour.
© Anjali Balagopal, Dan Nguyen, and Steve Jiang, MAIA Lab, 2020
Radiation Oncology
CLINICAL EVALUATION RESULT§ For patients originally contoured and evaluated by the same physician – Intraobserver
evaluation, • Clinically accepted contours scored an average of 3.4 while AI contours scored 3.2• Able to identify 60% of the physician contours
§ For patients originally contoured by one physician and evaluated by a different physician – Interobserver evaluation, • Clinically accepted contours scored an average of 3.07 while AI contours scored
3.29• Able to correctly identify 25% of the physician contours
§ None of the contours were found to be clinically unacceptable
§ Statistical equivalence test showed that AI model contours and clinically accepted contours are equivalent.
39© Anjali Balagopal, Dan Nguyen, and Steve Jiang, MAIA Lab, 2020
Radiation Oncology40 © Anjali Balagopal and Steve Jiang, Ph.D., MAIA Lab, 2019
Personalized AI for Physicians
85.1
0%
82.8
0%
82.2
0%
82.2
0%
89.0
0%
84.8
0%
84.7
0% 85.8
0%
PHY1 PHY2 PHY3 PHY4
General segmentation model Physician Style adapted model
Radiation Oncology41 © Anjali Balagopal and Steve Jiang, Ph.D., MAIA Lab, 2019
Personalized AI for Physicians
340 post-op prostate cancer patients contoured by 4 physicians at UTSW
0 40 80 120
Phy 1
Phy 2
Phy 3
Phy 4
Style classification model (~90% accuracy for 4 styles)
§ Proposed workflow– Train a CTV segmentation model using whatever data you have– When deploying the model, find physicians’ contouring styles from the
local data – Automatically adapt the initial model to each style– When segmenting a new patient, choose the model for the desired style
Radiation Oncology
§ AI is only as good as the people and data it learns from § Biases exist in medicine (race, gender, income, etc), but AI can
make these biases automated and invisible§ Bias in AI is mainly from the biased training data– Patient cohort/patient geographical spread/genetic background/disease
incidence/data labeling/data-acquisition/data-curation method– Training data is a subset of a heterogeneous dataset (IMB Watson Oncology )
– Training data doesn’t include enough samples for a particular subset– Training data include a pre-existing bias (Low-income patients do poorly for X
treatment à AI suggests no X treatment for them)
§ How to design AI so that it is fair to all patients/clinicians– Identify sources of inequity/De-bias training data– Develop algorithms that are robust to skews in data– Concepts of local models and model commissioning may be helpful – Regularly monitor AI output and downstream consequences
42 © Steve Jiang, Ph.D., MAIA Lab, 2019
Challenge: Bias
Radiation Oncology
§ Typically pediatric radiotherapy involves 25-30 episodes of GA§ Pediatric Radiation Oncology with Movie Induced Sedation Effect§ Radiation and movie are paused if motion exceeds tolerance§ DL-based motion detection and prediction à bias issue
© Steve Jiang, Ph.D., MAIA Lab, 2020
PROMISE Project
43
Radiation Oncology
§ Due to the progress of the medical technologies, clinical data used for AI model training will become obsolete
§ Model trained today may not work well some time later§ Model performance will decay with time§ Each trained AI model has a half life§ Static models à Evolving models§ We need to continuously QA the deployed AI model§ We need to continuously update the deployed model– How to update the model automatically with new data– How to avoid/minimize human intervention for data curation– How to QA the updated model– How to deal with regulatory issues– … …
44 © Steve Jiang, Ph.D., MAIA Lab, 2019
Challenge: Data Evolution (longitudinal/temporal variation)
Radiation Oncology
§ Deep Learning is a black box§ Explainability is important for AI in medicine§ It is still controversial § For some people, we don't need to understand how it works– Complex systems: too many parameters and relationships– Beyond human's comprehension - humans can only handle 5 or 6 things
in their heads at once– End users don’t really care what's under the hood as long as its
performance is good and robust
45 © Steve Jiang, Ph.D., MAIA Lab, 2019
Challenge: Explainability (I)
Radiation Oncology
§ For many others, we need to understand how it works– It is not about knowing everything– It is about knowing enough for your downstream tasks– Correlations are often not sufficient to take action– Causality decreases the risk of the action
§ Methods for understanding how DL works– Attribution methods• E.g., which part of image leads to the diagnosis
– Local approaches• Local approximation and thus explainable
– Attention mechanism• Which part of the model is associated with a given class
– Distillation• Simpler and more interpretable models are used to approximate the behavior of complex models
– Causal methods46 © Steve Jiang, Ph.D., MAIA Lab, 2019
Challenge: Explainability (II)
Radiation Oncologyhttps://www.utsouthwestern.edu/labs/maia/