Image-guided therapy and medical robotics tutorial using a LEGO Mindstorms NXT robot and 3D Slicer

Danielle F. Pace, B.CmpH

Ron Kikinis, M.D.

Nobuhiko Hata, Ph.DSurgical Planning Laboratory,

Brigham and Women’s Hospital andHarvard Medical School

Surgical Planning Laboratory, Harvard Medical School and Brigham and Women’s HospitalAugust 2007 -2-

Funding Information

This tutorial was made possible by grant numbers 5U41RR019703, 5P01CA067165 and 5U54EB005149 from the National Institutes of Health (NIH). Its contents are solely the responsibility of the authors and do not necessarily represent the official views of the NIH. This tutorial was also in part supported by National Science Foundation (NSF) grant number 9731748 and the Center for Integration of Medicine and Innovative Technology (CIMIT).

Surgical Planning Laboratory, Harvard Medical School and Brigham and Women’s HospitalAugust 2007 -3-

Thanks to• Terry Peters, Ph.D (Robarts Research Institute,

University of Western Ontario)• Steven Canvin (The LEGO Group)• Cory Walker (NXT++ Developer)• G. Wade Johnson (Device::USB Developer)• Steve Pieper, Ph.D (Surgical Planning

Laboratory)• Haiying Liu (Surgical Planning Laboratory)• Junichi Tokuda, Ph.D (Surgical Planning

Laboratory)• Christoph Ruetz (Surgical Planning Laboratory)• Philip Mewes (Surgical Planning Laboratory)• The entire LEGO Mindstorms community

Surgical Planning Laboratory, Harvard Medical School and Brigham and Women’s HospitalAugust 2007 -4-

Goals of the LEGO IGT and medical robotics tutorial

• To emphasize Slicer3’s support for image-guided therapy (IGT) using medical robots

• To demonstrate the typical steps of an image-guided therapy or medical robotics procedure in a hands-on manner

The example procedure that we will use to do this is a needle biopsy.

Surgical Planning Laboratory, Harvard Medical School and Brigham and Women’s HospitalAugust 2007 -5-

An Example of IGT in Action: Needle Biopsy• In suspected cancer cases (such as for

prostate or breast cancer) a needle biopsy is often performed as part of the diagnostic process.

• A needle is used toextract small pieces oftissue for analysis.

Goal: Hit the tumour so that it can be detected!

(image from R. Alterovitz, K. Goldberg and A. Okamura, Proceedings of the 2005 IEEE International Conference on Robotics and Automation, Barcelona, Spain, April 2005,

pp. 1652-1657)

Surgical Planning Laboratory, Harvard Medical School and Brigham and Women’s HospitalAugust 2007 -6-

Medical Robotics• Medical robots are

increasingly popular for procedures such as needle biopsy in prostate cancer

• In medical robotics, we use the same steps as in IGT: imaging, planning, registration, tracking and navigation

• To the right is a prostate biopsy robot that operates within an MRI machine (being tested on a melon!)

(JHU/SPL MRI prostate robot)

Surgical Planning Laboratory, Harvard Medical School and Brigham and Women’s HospitalAugust 2007 -7-

Tutorial Supplies

One LEGO Mindstorms NXT robotics kit One LEGO Deluxe Brick Box Two pom-poms CT volume of the phantom 3D Slicer LEGO tutorial module A Linux computer with root access Two sheets of white paper and tape Assembly instructions for the robot and

phantom Phantom placement guide

Surgical Planning Laboratory, Harvard Medical School and Brigham and Women’s HospitalAugust 2007 -8-

IGT Workflow

z

x

y

Step 1:Imaging

Step 2:Planning

Step 3:Registration

Step 4:Tracking and

Navigation

(-22, 84, 41)

z

xy ICS

PCS

(image modified from S. Haker et al., Top Magn. Reson. Imaging 16(5),

October 2005, pp. 355-368)

(image from http://www.ncigt.org/ourwork/core_mrguidedtherapy.html)

Surgical Planning Laboratory, Harvard Medical School and Brigham and Women’s HospitalAugust 2007 -9-

1. Set up the Robot and PhantomPosition the phantom relative to the LEGO robot such that:1.The needle can

reach both pom-pom targets

2.The registration pillars will be in view of the LEGO robot’s ultrasonic sensor

Surgical Planning Laboratory, Harvard Medical School and Brigham and Women’s HospitalAugust 2007 -10-

2. Imaging - Load the CT Volume of the Phantom

Surgical Planning Laboratory, Harvard Medical School and Brigham and Women’s HospitalAugust 2007 -11-

3. Connect Slicer to the robot

Slicer3 connects to the robot via a USB cable

Surgical Planning Laboratory, Harvard Medical School and Brigham and Women’s HospitalAugust 2007 -12-

4. RegistrationThe position of the phantom relative to the LEGO robot is not known.

We must determine the relationship between the image coordinate system (ICS) and the patient (robot) coordinate system (PCS).

We must perform registration!

Surgical Planning Laboratory, Harvard Medical School and Brigham and Women’s HospitalAugust 2007 -13-

The registration procedure

1. Find the eight fiducials in the patient (robot) coordinate system (PCS)

2. Find the eight fiducials in the image coordinate system (ICS)

3. Use a landmark registration algorithm to determine the rigid transformation from the image coordinate system to the patient (robot) coordinate system:

PCS3x1 = R3x3 • ICS3x1 + t3x1

Surgical Planning Laboratory, Harvard Medical School and Brigham and Women’s HospitalAugust 2007 -14-

The eight fiducials

1 2

3 4

5 6

7 8

back

front(towards

robot)

Surgical Planning Laboratory, Harvard Medical School and Brigham and Women’s HospitalAugust 2007 -15-

Finding the fiducials in the PCS• Two views of the patient (robot) coordinate system:

z

y

xz

y

x

Surgical Planning Laboratory, Harvard Medical School and Brigham and Women’s HospitalAugust 2007 -16-

Finding the fiducials in the PCS

• The LEGO robot’s ultrasonic sensor detects distance to the nearest obstacle.

• We will find the phantom by using the robot’s ultrasonic sensor to find the “registration pillars” on the phantom.

four “registration pillars”

Surgical Planning Laboratory, Harvard Medical School and Brigham and Women’s HospitalAugust 2007 -17-

Finding the fiducials in the PCS

y

x

• Use the ultrasonic sensor to find the registration pillars: the 4 angles shown

• Average the 4 angles to find

• We know d because the robot’s needle can hit both pom-poms

• Use , d and knowledge of the phantom’s structure to find the eight fiducials on the phantom

PCS

d

Surgical Planning Laboratory, Harvard Medical School and Brigham and Women’s HospitalAugust 2007 -18-

Finding the fiducials in the PCS

y

• The robotic arm moves from left to right while polling the ultrasonic sensor for distance and the motor’s rotation sensor for the number of degrees

• This occurs twice - once for the “top” pillars and once for the “bottom” pillars.

Surgical Planning Laboratory, Harvard Medical School and Brigham and Women’s HospitalAugust 2007 -19-

Finding the fiducials in the PCS

y

•Profiles of the number of degrees that the motor has turned vs. distance are generated for the top and the bottom swipes

•The local minima represent the registration pillars and are automatically detected

Surgical Planning Laboratory, Harvard Medical School and Brigham and Women’s HospitalAugust 2007 -20-

Finding the fiducials in the ICS

Select the corresponding fiducials in the image coordinate system by clicking on the points on the CT volume.

Surgical Planning Laboratory, Harvard Medical School and Brigham and Women’s HospitalAugust 2007 -21-

5. Preoperative Planning - select the target coordinate• The target coordinate is the location

where you would like the tip of the robot’s needle to be at the end of the “biopsy”.

• Let’s try to have the robot biopsy the red pom-pom.

• We need to find the center of the red pom-pom on the CT volume.

Surgical Planning Laboratory, Harvard Medical School and Brigham and Women’s HospitalAugust 2007 -22-

5. Planning - select the target coordinate

Click on the target coordinate: the center of the red pom-pom.

Surgical Planning Laboratory, Harvard Medical School and Brigham and Women’s HospitalAugust 2007 -23-

6) Execute the biopsy

The target in the patient (robot) coordinate system

Transformation from the ICS to the PCS

The final needle position in the patient (robot) coordinate system

Tracking is done through targeting during the robot’s movement

The display of the final needle position in the PCS provides Navigation

Surgical Planning Laboratory, Harvard Medical School and Brigham and Women’s HospitalAugust 2007 -24-

After completing this tutorial:• You have seen an example of medical robotics

within Slicer3• You have learned about the following steps in

IGT and medical robotics in a hands-on manner:ImagingPreoperative planningTargeting and trackingNavigationRegistration

• Tutorial website:http://wiki.na-mic.org/Wiki/index.php/LEGO_IGT_and_ Medical_Robotics_Tutorial

Surgical Planning Laboratory, Harvard Medical School and Brigham and Women’s HospitalAugust 2007 -25-

Additional references• Learn about image registration by reading:

• B. Zitová and J. Flusser. Image Registration Methods: A Survey. Image and Vision Computing, 21:977-1000, 2003.

• Brainstorm about the components of the total error that occur from the selection of the target to the final position of the robot needle tip. How could you measure those errors?• Start by learning about error in registration by

reading: J.B. West, M.J. Fitzpatrick, S.A. Toms, C.R. Maurer, and R.J. Maciunas. Fiducial Point Placement and the Accuracy of Point-based, Rigid Body Registration. Neurosurgery, 48(4):810-817, 2001.