Post on 16-Dec-2015
Medical
J. Michael Fitzpatrick, Department of Electrical Engineering and Computer ScienceVanderbilt University, Nashville, TN
Course on Medical Image Registration, Nov 3-Nov 24, 2008Institute für Robotic, Leibniz UniversitätHannover, Germany
Image RegistrationImage Registration
Schedule
Nov 3: Overview of Medical Image Registration
Nov 10: Point-based, rigid registration
Nov 17: Intensity-based registration
Nov 24: Non-rigid registration
Computed Tomography (1972)
Siemens CT Scanner (Somatom AR)
3D Cross-sectional Image
“voxels” (“volume elements”)
Magnetic Resonance Imaging
GE MR Scanner (Signa 1.5T)
Positron Emission Tomography
GE PET Scanner
Physician has 3 or more views.
CT(bone)
MR(wet tissue)
PET(biologicalactivity)
Combining multiple images requires image registration
Image Registration: Definition
Determination of corresponding points in two different views
Motion relative to the scanners can be three-dimensional.
Slice orientations vary widely. transverse sagittal coronal
Views may be very different.
But all orientations and all views can be combined if we have the 3D
point mapping.
Combining Registered Images = “Image Fusion”
MR + PETCT + MRCT MR PET
Rigid Registration: Definition
Rigid Registration = Registration using a “rigid” transformation
Rigid Transformation
Rigid Non-rigid
Distances between all points remain constant.
6 degrees of freedom
Nonrigid Transformationscan be very complex!
[Thompson, 1996]
Non-rigid example
Registration Dichotomy
• “Retrospective” methods (nothing attached to patient before imaging)
Match anatomical features: e.g., surfaces Maximize similarity of intensity patterns
• “Prospective” methods (something attached to patient before imaging)
Non-invasive: Match skin markers Invasive: Match bone-implanted markers
Most Common Approaches
• Intensity-based* (not for surgical guidance)
• Surface-based (requires identified surfaces)
• Point-based (requires identified points)
• Stereotactic frames (for surgical guidance)
*Sometimes called “voxel-based”
The Most Successful Intensity-Based Method:
Mutual Information
2D Intensity Histogram (Hill94)
CT
MRCT intensity
MR
inte
nsity
Misregistration Blurs It
0 cm 2 cm 5 cm
MR
CT
MR
PET
Hill, 1994
• A measure of histogram sharpness • Most popular “intensity” method • Assumes a search method is available• Stochastic, multiresolution search common• Requires a good starting pose• May not find global optimum• Not useful for surgical guidance
Mutual Information(Viola, Collignon, 1996)
Example: Mutual Information
Studholme, Hill, Hawkes,
1996, “Automated
3D registration of MR and CT images of the head”, MIA,
1996
(Open movie with
QuickTime)
The Most Successful Surface-Based Method:
The Iterative Closest-Point Algorithm
• Minimizes a positive distance function• Assumes surfaces have been delineated• Guaranteed to converge• Requires a good starting pose• May not find global optimum• Can be used for surgical guidance
Iterative Closest-Point Method(Besl and McKay, 1992)
Start with two surfaces
Reorient one (somehow)
Reorient one (somehow)
Reorient one (somehow)
Pick points on moving surface
Pick points on moving surface
Remove moving surface
Points become proxy for surface
Find closest points on stationary surface
Measure the total distance
Remove stationary surface
Points become proxy for surface
Register point sets (rigid)
Register point sets (rigid)
Restore stationary surface
Find (new) closest points
Find (new) closest points
Remove stationary surface
Remove stationary surface
Register Points
Register Points, and so on…
Iterative Closest-Point Algorithm:
• Find closest points• Measure total distance• Register points
Stop when distance change is small.
ICP: Image-to-Image
Dawant et al.
ICP: Image to Patient
• The BrainLab VectorVision surgical guidance system uses surface-based registration.
ICP requires surface delineation, which is a problem in Image Segmentation
Example: Level Set Segmen-
tation (Dawant
et al.)
http://www.vuse.vanderbilt.edu/~dawant/levelset_examples/
The fiducial marker is used in prospective registration for image-guided surgery.
The Most Common Application of The Point-based Method:
The Fiducial Marker
Image-Guided Surgery
...and the other is the patient.
One view is an image....
Just another image registration problem.
Acustar™
Allen, Maciunas, Fitzpatrick, and
Galloway
1988-1995 (J&J Z-Kat)
are implanted into the skull.
Posts
[Maurer, et al., TMI, 1997]
Acustar™
Allen, Maciunas, Fitzpatrick, and
Galloway
1988-1995 (J&J Z-Kat)
[Maurer, et al., TMI, 1997]
Liquid in marker
shows up
in image
Divot cap is localizable
in OR
Acustar™
Allen, Maciunas, Fitzpatrick, and
Galloway
1988-1995 (J&J Z-Kat)
[Maurer, et al., TMI, 1997]
Marker center and cap center occupy the same position relative
to the post
Acustar™
Allen, Maciunas, Fitzpatrick, and
Galloway
1988-1995 (J&J Z-Kat)
[Maurer, et al., TMI, 1997]
Marker center and cap center occupy the same position relative
to the post
Find corresponding
“fiducial” points
Point-based, Rigid Registration
View 2= “Space” 2
View 1= “Space” 1
Rigid transformation
Align corresponding
fiducials“targets” are also aligned
Find all corresponding
“fiducial” points
Measures of Error
View 1
Registered Views
View 2
Fiducial Localization Error (FLE)
Target Registration Error (TRE)Fiducial Registration
Error (FRE)
The Most Successful Point-based Method (by far!):
Minimization of Sum of Squares of
Fiducial Registration Errors
• Minimizes a positive distance function• Most popular point method • Assumes points have been localized• Guaranteed to converge• Does not require a good starting pose• Always finds global optimum• Can be used for surgical guidance
Minimization of Sum of FRE2
(Shönemann, Farrell, 1966)
Accuracy: State of the Art
The best accuracy is probably achieved for the head…
Retrospective Registration of Head: Image-to-Image
Median Maximum
CT-MR : 0.6 mm 3.0 mm
PET-MR: 2.5 mm 6.0 mm
TRE
Prospective Registration of Head: mean TRE ≤ 1 mm (CT)
[Hill, JCAT, 1998, Maurer, TMI, 1997]
Error Theory for Minimization of
Mean-square FRE
End of Overview
How to Do
Minimization of Sum of Squares of
Fiducial Registration Errors
Sum of Squares: Step 1
N
N
i
i
yy
xx
yyyxxx iiii ~ ; ~ :pointsCentered""
Center the points:
Centered
xy
Step 2 (Shönemann, Farrell, 1966)
)det(VU ,1 1 diag
where, D
0
),,diag(Λ
where
,Λ :SVD
~~
t
3 2 1
3 2 1
,D
UVR
IVVUU
VUH
H
t
tt
t
ii
tyx
Determine the Rotation: Centered
Centered and Rotated
Step 3 (Farrell, 1966)
R t y x
Determine the Translation:
Rx
ytx
y
Before rotation After rotation, but before translation
Error Analysis
Start with Assumptions about FLE
Independent, normal, isotropic, zero mean
Space 1 Space 2
“Effective” FLE
22221 FLEFLEFLE
1FLE
FLE
2FLESpace 1 Space 2
FRE Statistics: Sibson 1979
22
2
22
2
FLE)21(FRE
DOF. 63with
square-chi is thewhere
,FRE
:FLE To
/N
N
c
O
Approximate Solution:
Configuration doesn’t
matter!
22
2
222
2
FLE)21(FRE
DOF. 63with
square-chi is thewhere
,3/FLEFRE
:FLE To
/N
N
N
O
2
2 2 22 1 2 3
2 2 21 2 3
1TRE 1 / 3 FLE
d d d
N f f f
Principal axesConfiguration does
matter.
d1d2d3
[Fitzpatrick, West, Maurer, TMI, ’98]
TRE statistics, 1998Approximate Solution:
Got to here Nov 10, 2008
4mm3mm
2mm
1mm
2mm
1mm
FRE = 1mm
TRE for
FLEof
1mm
Marker Placement
[West et al., Neurosurgery, April, 2001]
A distribution would be better
<TRE2>
TRE295% level
Pro
babi
lity
den
sity
And what about direction?
TRE statistics, 2001
.directions
orthogonal along
components
t independen denote
3,2,1 where
,0, TRE i
i
N i
Approximate Solution:TRE1
TRE2TRE3
[Fitzpatrick and West., TMI, Sep 2001]
Some Remaining Problems
Isotropic Scaling
[Actually now solved: Batchelor, West, Fitzpatrick, Proc. of Med. Im. Undstnd. & Anal. ,
Jul 2002]
Anisotropic Scaling
(Iterative Solution Only)
Register M points sets simultaneouslyView 1 View 2
;
View 3 View M
The “Generalized” Procrustes Problem
(Iterative Solution Only)
Anisotropic FLE
(Iterative Solutions Only)
Other Unsolved Problems
• What is the statistical effect on TRE of dropping or adding a fiducial?
• Does anisotropy in FLE always, sometimes, or never makes TRE worse?
• How do we configure markers on a given surface so as to minimize TRE over a given region?
• Is there a correlation between FRE and TRE?It’ solved: There is no correlation!
Fitzpatrick, SPIE Medical Imaging Symposium, to be presented Feb 2009.
• Extension to perspective transformations.
• Extension to surface matching.
Other Unsolved Problems (cont.)
Rigid Registration of the Head
State of the Art
CT MR-T1 MR-T2
Finding Points = “Localization”
Acustar v. Leibinger:Leibinger Grows Up!
Retrospective Registration of Head Images: The State of the Art
Median Maximum (Acustar)
Best CT-MR : 0.6 mm 3.0 mm (0.5 mm)
Poor CT-MR: 5.4 mm 61 mm (0.5 mm)
Best PET-MR: 2.5 mm 6.0 mm (1.7 mm)
Poor PET-MR: 5.3 mm 15 mm (1.7 mm)
And how do we know?…
Retrospective Image Regstration Evaluation
Access: 150+ participants in 20 countries
Evaluation: 57 participants in 17 countries
External siteVanderbilt
1995-2007
End
Additional slides follow
Categories within error prediction
• Number of point sets: Two or more
• Scaling: Isotropic or anisotropic
• Point-wise weighting: equal or unequal
• Anisotropic weighting
• Cost function: squared error or other
• Point-wise FLE: equal or unequal
• Spatial FLE: isotropic or anisotropic...
Key: Approximate, Negligible progress
Anisotropic Scaling
R, t = rotation, translationwi
2= point weightingS = diag( sx , sy , sz )
N
iiii RSwN
22)/1( ytx
Given {xi yi wi} find R, t, S to minimize mean FRE2
.for
),,(FRE min
minimizes
that Sfor Search
i
2
R,
i
i
S
R
xx
xtt
Iterative Algorithm:
sy
sz
sx
Searchspace
Problem Statement:
Scaling: Anisotropic II
R, t = rotation, translationwi
2= point weightingS = diag( sx , sy , sz )
N
iiii SRwN
22)/1( ytx
Given {xi yi wi} find R, t, S to minimize mean FRE2
),(FRE
minimizes
that , ,for Search
:In harder thamuch
gly)(surprisin is II
2 t
t
RS,
RS
Iterative Algorithm:Problem Statement:
Spatial Weighting
R, t = rotation, translationwi
2= point weightingS = diag( sx , sy , sz )A = diag( ax , ay , az )
N
iiii RSAwN
22 )()/1( ytx
Given {xi yi wi} find R, t, S to minimize mean FRE2
[96] vTrendafilo &Chu
[91] Swane &Koschat
Iterative Algorithm:Problem Statement:
Batchelor and
Fitzpatrick [2000]
Partial Solution:
Generalized Procrustes Problem
Cost function Iterative method(only)
Add Isotropic Scaling
Approximate Solution:
22
2
222
FLE3
71FRE
DOF 73 has
.3/FLEFRE
:) To
N
N
N
O( 2
FRE2 = sum of squared fiducial
registration errors
FRE: Generalized + Scaling
Approximate Solution:
22
2
22
2
FLE3
71FRE
DOF. 7)-1)(3N-(M has
.13FLE
FRE :) To
N
MN
O( 2
FRE2 = sum of squared fiducial
registration errors
TRE statistics with scaling
2
To
TRE
West and Fitzpatrick [2001]
2O
Approximate Solution:
TRE2 = target
registration error
Applications of TREStatistics
Surgical Paths
Radiation Isodose
Contours
Error Bounds
Probe Design
Tip = “target”
IREDs are fiducials
FLE
TRE
ix22i TREFLEFRE 2
Fiducial-Specific FRE
1x Poor fiducial alignment tends to occur where target registration is good!!
2x
Four Solution Methods
1/21. Square Root:
2. SVD: , where
3. Quaternion:
(a) Form 4x4 matrix from elements of .
(b) Find eigenvector of with largest eigenvalue.
(c) Elements of
t t
t t
R H HH
R VU H UDV
Q H
Q
R
q
1 2 3 4 are quadratic in , , , .
4. Dual-number quaternion: [Walker91].
q q q q
( All work equally well [Eggert91]! )
.~~on only depends and easy, is tyxt ii2iwHR
Generalized Procrustes Problem
)()()()()(
2)()(2
)()()(
)(
where
minimize to
}1 |,,{ find
},1 1 , |{
each, points of sets For
mmi
mmmi
M
m
M
mk
N
i
ki
mii
mmm
m
SR
w
MmSR
MN,mi
NM
i
txx
xx
t
x
(We’ve already done it for M=2.)
Problem Statement: Illustration:
Generalized Procrustes Problem
neglible. are changes until
. where
,,, 1 ,ˆ
minimize to,, Find
.)/1(ˆ
: thisIterate
. Start with
)()()()()(
2)(2
*)()()(
)(
)()(
mmi
mmmi
N
ii
mii
mmm
M
m
mi
mi
mi
SR
Mmw
SR
Mi
txx
xx
t
xx
xx
Iterative Algorithm: Illustration:
*Subject to S(m) normalization
Approximation Method
.
:0 , if sameoutput that Note 2
., :FLEs small Assume (1)
2
)()(
2)()(
1
21
iii
truetrue
itrue
itrue
i
iii
IR
R
gxy
t
gtzy
fzx
(due to Sibson, 1979)
Approximation Method (cont.)
orderhigher dropping ,in Expand (3)
)(TRE
)(FRE
) (
22
22
)1(
O
O
O
RIR
t
FRE Statistics
N
iii
i
RN
Ni
2
R,
2
21
1min FRE
for Statistics :Find
and
,,1for }{ :Given
ytx
z
t
Problem Statement: Approximate Solution:
22
2
222
21
22
FLE)21(FRE
freedom. of degrees
63 with ddistribute
square-chi is thewhere
,/FRE
:) To
/N
N
N
O( 2
TRE statistics with scaling
N
iii
s
i
RsN
Rs
Ni
2
,R,
2
2
1
1min FRE
when ,TRE
for Statistics :Find
. and ,
,,1for }{ :Given
ytx
ytx
x
t
Problem Statement:
2
To
TRE
West, Fitzpatrick,
and Batchelor [2001]
2O
Approximate Solution:
What do “solved” and “unsolved” mean?
• “Solved”, working definition: Reduced to solving algebraic equations Iterative algorithm that converges to solution Approximate solution accurate to
• “Unsolved”: Not solved
)(O
Point-wise weighting: Equal or Unequal(We’ve just looked at this one.)
R, t = rotation, translationwi
2= point weighting
N
iiii RwN
22)/1( ytx
Given {xi yi wi} find R, t to minimize mean FRE2
Problem Statement:
See previousslides again!
Solution:
1. Performing a Registration
xi = point in “from” set; yi = point in “to” set.t = translation vector.R = 3x3 rotation matrix (therefore RtR = I ).
Rxi + t
N
iiii RwN
22)/1( ytx
Given {xi yi wi } find R, tto minimize mean FRE2 xi yi
( usually wi=1)
a.k.a. The “Orthogonal Procrustes Problem”Problem Statement:
2. Predicting Registration Error
View 1
Registered Views
View 2
Input---•fiducial positions
•target position, r•FLE distribution
Output---statistics for
TRE
r
Output---statistics for
FRE
Isotropic Scaling
iii
iiii
w
Rw
s
R,
s
xx
yx
t
~~
~~
(2)
Find
1Set (1)
2
2
R, t = rotation, translationwi
2= point weightings = isotropic scaling
N
iiii RswN
22)/1( ytx
Given {xi yi wi} find R, t, s to minimize mean FRE2
Problem Statement: Solution:
AcknowledgementsBenoit M. Dawant, PhD, EECS
Robert L. Galloway, PhD, BME
William C. Chapman, MD, Surgery
Jeannette L. Herring, PhD, EECS
Jim Stefansic, PhD, Psychology
Diane M. Muratore, MS, BME
David M. Cash, MS, BME
Steve Hartman, MS, BME
W. Andrew Bass, BME
NSF NIH
Matthew Wang, PhD, IBMJay B. West, PhD, Accuray, Inc.
Derek L. G. Hill, PhD Kings CollegeCalvin R. Maurer, Jr., PhD, Stanford U.
What could we choose to optimize?
• Mean-square “Fiducial Registration Error” (FRE2) Known as the “Orthogonal Procrustes Problem” in
statistics since 1950s.
• Robust estimators (median, M-estimators) Less sensitive to “outliers”
Color key: Major problems solved, Much less done