of 13

  • date post

  • Category


  • view

  • download


Embed Size (px)

Transcript of AUGMENTED REALITY WITH X-RAY LOCALIZATION FOR Papers/2001 SGH Augmented reality...  AUGMENTED...


Authors: Yeo Seng Jin, FRCS(Ed), FAMS Yung Shing Wai Kwoh Chee Keong, PhD, MSc Seah Evan, B A Sc Wong Thong Seng, BEng Ng Wan Sing, PhD, DIC Lond., MEng (NUS'pore) Teo Ming Yeong, SM, B.Sc(Hons)

Attribute: The work is the result of collaboration of the Department of Orthopaedic Surgery, Singapore General Hospital and the Computer Integrated Medical Intervention Laboratory, Nanyang Technological University.

Acknowledgment: We are very grateful to Mr. Robert Ng, Manager of Department of Experimental Surgery of Singapore General Hospital for making the arrangements for our experiments and assist us in use the C-arm fluoroscopic machine in the mortuary.

Corresponding Address: A/P Kwoh Chee Keong

Division of Computing Systems, School of Computer Engineering

Nanyang Technological University

Blk N4 #2A-32

Nanyang Avenue

Singapore 639798

Tel: (65) 790 6057

Fax: (65) 792 6559

Meeting: The paper was presented in the Third Annual NTU-SGH Biomedical Engineering Symposium.

ABSTRACT An approach for the localization of acetabular prosthesis cup placement during total hip replacement (THR) surgery, which is based on only one X-ray image is described. The purpose of this project is to assist the surgeon in placing the hip-prosthesis cup at the right orientation. The ultimate aim is to use the procedure intraoperatively. From X-ray images, the 2D coordinates of points in images is picked and the 3D world coordinates of the hip can be calculated using a mathematical model. The method has been applied on mock bone and cadaver trials and gave satisfactory result in finding the center of the acetabulum cup and the desired orientation of implant insertion (45 of abduction and 15-20 of anteversion) for implanting the acetabular component. The calculated information is then integrated to into a new augmented reality system to provide real-time fusion of video and virtual information for online, real-time visualisations during actual clinical procedures.

Keywords: image-guided surgery, computer aided surgery, augmented reality, X-ray localization, total hip replacement, femoral implant, orthopedics surgery, image intensifier, distortion and calibration. Camera tracking

1. INTRODUCTION The primary motivation of this research project is to set up an Augmented Reality System for Therapy (ART) for the purpose of Total Hip Replacement. Total hip replacement (THR) is an operation in which, the damaged hip (sometimes due to arthritis and sometimes due to damage caused by an accident) is replaced with an artificial hip, which consists of the Acetabulum cup and Femoral Stem. This operation involves a number of steps to complete the THR. Our primary aim is to provide assistance to place the acetabulum cup at the right orientation.

In Total Hip Replacement operation, the patient is covered up with drapes. This makes it difficult for the surgeon to accurately determine the position of the limb. As surgeons do not have the information of the correct orientation of acetabular cup, they implant the acetabular cup based on his experience without tracking and localizing equipment. This limitation may causes dislocation of the artificial hip and revision of surgery has become necessary. The incidence of dislocation following primary total hip replacement (THR) surgery is between 2-6% and even higher following revisions [1][2].

The aim of this project is to equip the surgeon with X-ray eyes to see through the drape. This is achieved via an augmented reality system. The system will therefore be able to assist the surgeon to place the tools and hence the prosthesis at the correct position and orientation to achieve best clinical outcome. Since the patient is completely covered up and therefore Augmented Reality (AR) comes in useful to reveal whats not visible directly by overlays the computer-synthesized images onto the users real world views.

Augmented Reality has been applied in many fields such as medical application, military training, engineering design, manufacturing, architecture, maintenance and repair (as shown in Figure 1.1).

(i) Mechanical (b) Interior Design, only the phone is real

(c) Exterior Construction [3] (d) Breast needle biopsy [4]

Figure 1.1: Different Applications of Augmented Reality

2. System overview AR needs internal image to superimpose over the real video image. There are various internal imaging techniques for medical purpose, such as X-ray, fluoroscope (which is low intensity X-ray), ultrasound scanner, computed tomography (CT) and magnetic resonance imaging (MRI). For THR the present technique to localize the hip is CT. The images are taken preoperatively for diagnosis and planning [3]. During operation, if CT is used, users will be exposed to high amount of radiation. Hence, in our system of X-ray localization of hip, in order to minimize the radiation exposure, we must minimize the number of X-ray images to be taken. So, we explore a system with one or two X-ray images to determine the coordinates of the points on the screen and then calculate 3D world coordinates of the hip without actual reconstruction of the 3D model. We also developed a localization & multi-modal image registration method for this application [3].

The advantages of the system are: First, the resulting system is cost effective because of the use of few X-ray images instead of continuous CT. Second, radiation dosage both to surgeons and patients can be reduced drastically. Third, the new calibration method does not taking geometric model into consideration. Figure 2.1 shows the system overview of our system and the first prototype of the augmented images. The potential of the system lies in direct, fully immersive, real-time multisensory fusion of real and virtual information data stream into online, real-time visualisations available during actual clinical procedures.


Image Overlay Unit


Tracking Unit

See-through Display


Figure 2.1 shows the system overview of our system and the first prototype of the augmented images. In this

demonstration, we modeled the alignment tool and the corrected line of action for THR. To present the information to the user, we augment the image and present as one image to the user.

The overall system can be subdivided into Image Intensifier sub-system, Tracking unit sub-system, Video Cameras sub-system and Image Overlay Unit. In this paper we will first focus our discussion on Image Intensifier sub-system where we take X-ray images and determine all the important parameters and information to determine the cup size and the ideal line of action and orientation in the given images. With the known information, we then passed the necessary information and markers locations for the tracking unit sub-system for tracking. Finally, these information are used to register and fused the generated images to the real-time video.

3. Image Intensifier sub-system In the Image Intensifier sub-system, the main objective is to come up with the image intensifier (II) distortion calibration method and X-ray localization technique for THR. The following summarized the procedure for THR in our system will give reader a better understanding of the sequence of actions.

Placing 6 marker on the pelvis with at least one of the marker is out of plane

Get image using C-arm X-ray machine.

Find 3D coordinates of markers in world coordinate with ancillary device.

Find 2D coordinates of these markers on the screen for the images in screen coordinates.

Develop the transformation between 3D and 2D coordinates system.

Select at least three points on the pelvis periphery on screen projection and find its center on 2D and hence on 3D.

Show the desired orientation of the tool by rotating first 45 around Z-axis (abduction) and then 20 around X-axis (anteversion).

Pass the information to 3D tracking device.

Determine the 3D position and orientation of the pelvic at each frame.

Generate the correct graphics and fused them with stereo video inputs.

3.1 The image intensifier (II) distortion calibration method When fluoroscopy is used, we must ensure that a rectangular grid appears as it is in the X-ray. If not correction must be made to restore the capture images. It is important to note that in the actual usage, it is not always possible to use standard gantry angles for oblique fields, particularly where conformal planning (to confirm the size, orientation of an organism etc ) is employed. Hence, distortion calibration must take into consideration of this variable.

Some well known sources of distortion as reported in papers are as followed. First, the projection of the X-ray image onto the curved surface of the image intensifier front end. Second, the electron optics of the image intensifier, interactions with external magnetic fields and the video component of the fluoroscopic system (including the optical coupling between the output phosphor screen of the image intensifier and the video camera). The most visually apparent of these is the pincushion effect of the projection of the X-ray image onto the curved surface of the image intensifier (I.I) front end (Figure 3.1 (a)). Rotation and S distortion introduced by the electron optics of the image intensifier and interaction with external magnetic fields (specifically the earths magnetic field) is shown in (Figure 3.1 (b)).

Figure 3.1 (a) The pincushion effect of the projection of the X-ray image onto the curved surface of the image intensifier (I.I) front end. (b) Rotati