Inside-Out Tracking for Flexible Hand-heldNuclear Tomographic Imaging

Philipp Matthies1,2, Chen-Hsuan Shih3, Jakob Vogel4, Alvaro Sanchez5, TobiasLasser1, Benjamin Frisch1, Michael Friebe2,6, and Nassir Navab1,7

1 Computer Aided Medical Procedures, Technische Universitat Munchen2 Institute for Advanced Study, Technische Universitat Munchen

3 Department of Management, Technology and Economics, ETH Zurich, Switzerland4 Paul Scherrer Institut, Villigen, Switzerland

5 Kitware Inc., USA6 Institute of Medical Engineering, Otto-von-Guericke-Universitat Magdeburg

7 Computer Aided Medical Procedures, Johns Hopkins University

Abstract. Tracking systems are used in many medical scenarios forthe localization of devices or the patient, for example in intra-operativenuclear imaging. The infrared tracking system commonly used for thispurpose is outside-in and often suffers from line of sight issues. To alle-viate these, we present the combination of an inside-out tracking tech-nique with a hand-held mini gamma camera and an image reconstructionpipeline to provide 3D SPECT-like images in a compact flexible setupsuitable for interventions. As a proof-of-concept we show first resultson phantoms mimicking common nuclear medicine procedures, sentinellymph node biopsy in thyroid and breast cancer. Additionally, we showa first clinical result of this procedure on a breast cancer patient. Theresults achieved show comparable performance to the standard outside-in tracking approach, while easing the interventional procedure in termsof hardware and line of sight requirements.

1 Introduction

Single Photon Emission Computed Tomography (SPECT) provides 3D imagesof the distribution of a gamma-emitting radionuclide. SPECT is used in a widerange of diagnostic applications such as bone, brain, thyroid, infection, myocar-dial perfusion or tumor imaging [1]. For interventional or intra-operative appli-cations, flexible nuclear imaging systems have been introduced [2, 3] based ontracked mobile gamma detectors and subsequent tomographic reconstructions.The systems used in clinical applications, such as guidance of breast [4], head andneck [5], or melanoma [6] sentinel lymph node (SLN) biopsy, rely on an external(outside-in) infrared tracking system. This approach is limited by the necessityof external cameras, increasing the spatial and financial overhead of the system,but also by the need for a long and direct line of sight from the tracking systemto the tracked object, which cannot be easily guaranteed in cramped surgicalenvironments.

2

We suggest inside-out tracking as an alternative optical tracking method,where a camera placed on the detector observes features of the surroundingenvironment to provide location information. By applying a 2D marker-basedtracking algorithm, as suggested by [7] or [8], it is possible to observe the po-sition and orientation of the camera in relation to the tracking target, in ourcase a cube with 2D augmented reality (AR) markers attached to each side. Ina medical environment, similar approaches have been implemented by Rafii-Tariet al. [9] for an ultrasound guidance system using a transducer-mounted camerato create 3D panorama images relative to skin markers, and by Magaraggia et al.[10] for a video camera-based solution for screw fixation guidance. Also, a com-mercial solution for instrument guidance based on a stereo camera is availablefor ultrasound- and CT-guided procedures [11].

In the following we will present the suggested setup combining inside-outtracking with tomographic image reconstruction, along with proof-of-conceptphantom experiments and a first clinical result of a breast cancer patient.

2 Materials and Methods

The general system setup involving the marker cube and the mobile gamma de-tector with attached video camera is illustrated in fig. 1. Details of the imagereconstruction pipeline, marker-based tracking and coordinate system calibra-tion are outlined in the following.

Fig. 1: Schematic image of system setup. Black arrows mark the coordinate sys-tem transformations between the video camera (cam), calibrator (calib), gammadetector (detec), volume of interest (voi), and patient target (target).

2.1 Image reconstruction

We aim to reconstruct the distribution of radioactivity, a signal f : V → [0,∞)mapping a volume V to non-negative numbers (the activity). We discretize V into

3

voxels, where x = (xi)i∈I denotes the activity in each voxel i. Let further m =(mj)j∈J denote the actual measurements of each detector j, and let p(i, j) denotethe detection probability of detecting an emission from voxel i in detector j(measured as in [2]). Then we can formulate the Poisson log-likelihood as

L(x|m) = mj log

(∑i

xi · p(i, j)

)−∑i

xi · p(i, j).

A reconstruction is then performed by computing a maximum likelihood esti-mator

arg maxx

L(x|m)

using the maximum likelihood expectation maximization method (MLEM, [12]).This is computed iteratively for each component i ∈ {1, . . . , |I|} using the start-ing value x0 = 1 ∈ R|I| and iteration index k ∈ N,

xk+1i =

xki∑

j p(i, j)·∑j

mj · p(i, j)∑i xi · p(i, j)

.

2.2 Marker based tracking

The position of the gamma camera with respect to volume V is contained in themeasurement probabilities p(i, j). Unlike outside-in tracking, where an externalsystem tracks the gamma camera, the device needs to track its own positionwith respect to the world in our inside-out approach.

We propose to use an optical system, and mounted a wide-angle video-cameraonto the gamma camera, see fig. 1. The world position is established via a markercube, showing different black-and-white marker patterns on the sides. Such acube can easily be positioned in the field of view of the camera, and can bemanufactured to be sterilizable [13]. Furthermore, it is quite likely to observemore than one of the markers at a time, thus making tracking more robust.

Having calibrated the optical and distortion parameters of the video cam-era [14, 15], we use the ArUco tracking framework [8] for obtaining a locationand orientation in three-dimensional space for each of the cube’s visible markerswith respect to the optical camera center. The transformation from the videocamera to the cube targetTcam (cf. fig. 1) is then computed as mean of all de-tected transformations, individually extended from the surfaces to the cube’scenter.

2.3 Coordinate systems calibration

By default, the marker-based tracking system gives the transformation from thevideo camera’s center to the markers. For reconstruction, however, the locationof the gamma camera with respect to the volume of interest, detecTvoi, is required,and thus knowledge of the transformation

detecTcam = detecTcalib · calibTcam

4

between the optical and the gamma camera’s coordinate systems, see the dashedarrows in fig. 1.

In order to obtain this mapping, a custom-printed calibrator tool was at-tached to the gamma camera, see fig. 1. Thanks to the known dimensions ofthis tool, and the known position of an optical marker affixed to it, we knowdetecTcalib directly. To recover the full transformation detecTcam, we measure thetransformation from video camera center to marker calibTcam using the imagesobtained from the video camera while the calibrator tool is mounted, and insertit into the full chain.

During normal operation, the calibrator tool is removed again, in order notto obstruct the field of view. The video camera tracks the marker cube, andcomputes the location of the gamma camera with respect to world coordinatesas

detecTtarget = detecTcalib · calibTcam · (targetTcam)−1.

The volume of interest can be selected interactively by the user, thus givingvoiTtarget, and the full transformation from volume of interest to the gammacamera, as required for tomographic reconstruction, computes as

detecTvoi = detecTtarget · (voiTtarget)−1.

3 Experiments

Data was acquired using a CrystalCam mini gamma camera (Crystal Photonics,Germany) and a Hero3 video camera (GoPro, USA). As a proof-of-concept, twophantoms were scanned repeatedly, mimicking common nuclear medicine proce-dures: sentinel lymph node biopsy in thyroid and breast cancer. Additionally, afirst breast cancer patient was scanned. After tomographic reconstruction, theresulting image was visualized in an augmented reality view superimposed ontothe video camera image.

3.1 Phantom experiments

The breast lymph node phantom consists of two hollow spheres (volumes of3.5ml and 0.6ml each filled with a 3MBq solution of Tc99m ) in a water-bath, cf.fig. 2. This resembles a common sentinel lymph node scenario in breast cancer.

The custom-printed thyroid phantom contains 4 chambers of different sizesthat can be filled with radioactivity or water to simulate hot or cold nodules ofdifferent size, cf. fig. 2. The dimensions of the structure are comparable to theaverage human thyroid. The two spheres in the left lobe of the thyroid were filledwith a 0.6MBq solution of Tc99m , the two spheres in the right lobe were filledwith water. The thyroid itself was filled with a 60kBq solution of Tc99m . Theconcentrations resemble typical uptake values of thyroid patients in our hospital.

Both phantoms were scanned manually (freehand) by multiple operators foran average duration of 2.5min. The breast lymph node phantom was mainlyscanned from two sides, whereas the thyroid phantom was placed in the neck of

5

an upper body doll and scanned in a semi-circular fashion with angular varia-tions, cf. fig. 2. Tomographic reconstruction was performed using a discretizedvolume of 44×34×20 voxels for the breast lymph node phantom and 40×40×30voxels for the thyroid phantom, both with an isotropic size of 3mm3, and 10 it-erations of MLEM.

For both phantoms, we also acquired tracking data using the outside-intracking provided by a Polaris Vicra infrared tracking system (Northern Dig-ital, Canada) in parallel to the inside-out tracking data for evaluation purposes.

(a) (b) (c)

(d) (e) (f)

Fig. 2: Images showing (a) the breast lymph node phantom, (b) a reconstructionof the hot nodes in an augmented reality view superimposed on the video image,(c) a rendering of the thyroid phantom, (d) the scanning process of the thyroidphantom in a doll, (e) a reconstruction of the thyroid phantom (hot nodules) inan augmented reality view superimposed on the video image, and (f) a slice ofthe reconstruction of the thyroid phantom (red lines marking the ground truth).

3.2 Patient experiment

As a first clinical test, a breast cancer patient at our clinic was scanned. Thepatient was diagnosed to undergo sentinel lymph node biopsy, and we scannedthe axilla for 2.1min with the marker cube placed on the patient’s breast, cf.fig. 3. After our scan a conventional scintigraphy for localization of lymph nodesfor surgery was performed, in which the physician marked the positions of thelymph nodes on the skin, cf. fig. 3. Tomographic reconstruction was performed

6

using a discretized volume of 44×44×30 voxels with an isotropic size of 3mm3,and 10 iterations of MLEM.

4 Results

In case of the hot-spot phantom, the ground-truth distance between the twospheres is 5.5cm. In our results, we discard obvious reconstruction artifacts,and consider the distance between the centers of mass of the respective recon-structed hot-spots. We have scanned the phantom three times, but rejected oneacquisition due to bad operator performance – one of the hotspots was not cov-ered sufficiently well in order to reconstruct it at all. In the other two scans,we obtained a mean hot-spot-distance of 5.45cm using our inside-out trackingapproach, and 5.75cm for the outside-in system.

For the thyroid phantom, we consider the two hot nodules known to be3.3cm apart. We take the same approach as described for the hot-spot-phantomto compute the reconstructed distances. In five scans, we obtained a mean hot-nodule-distance of 3.22cm for our inside-out tracking, and 3.26cm for the outside-in system.

The result of our patient experiment is shown in fig. 3. After reconstructionof a three-dimensional SPECT image, approximate distances have been takenfrom the planar scintigraphic image, and the three-dimensional SPECT imagehas been rotated to show a perspective comparable to the scintigraphy. Ourreconstruction is reasonable and coincides with the diagnosis of the attendingphysician.

5 Discussion and Conclusion

Our experiments show a performance quite comparable to the standard systemwith its outside-in tracking system. In our phantom experiments, we have beenable to recover the distance between the respective hot regions to high precision.More importantly, we have obtained a ‘correct’ reconstruction in our patient ex-periment, substantiated by the physician’s diagnosis. Both tracking approachescan thus be considered to yield results of equivalent quality.

Judging the two approaches from the workflow-perspective, however, theinside-out approach has clear advantages: The tracking target, the cube withthe fiducial markers, can be easily positioned in a place where the physician’selbow room is unaffected, in the same field of view required by the gamma cam-era, and at a short distance. In particular, there is no need for the operatorto care about a free line of sight between an over-head system and the camerain his hand, thus making the measurement process simpler – everything is inperspective!

Furthermore, the marker cube can be adapted to medical requirements interms of size, sterilizability, weight, among other aspects. Even in a standardsetting, more than just one side of the cube will be tracked by the camera, thusmaking the approach quite robust, and further marker cubes could be added

7

Fig. 3: (a) Visualization of reconstruction result in real-time augmented realityview. The patient target (1) is tracked and the volume of interest is renderedas a white box (2) in front of the gamma camera (3). (b) Reconstructed imagefrom a perspective comparable to the (c) scintigraphy.

without trouble. Similarly, our prototype approach of mounting the camerason top of each other can be improved considerably by designing a dedicatedcombined optical-/gamma-camera, thus adapting the device optimally to clinicalpractice.

Apart from engineering questions, future work will need to go into the detailsof marker tracking, to guarantee assured tracking in the relevant sphere of ac-tion. This includes questions about minimal and maximal distances, limitationsimposed by lighting conditions, marker designs, and others. First preliminaryexperiments have already shown considerable robustness.

In summary, we presented a system combining inside-out tracking with ahand-held mini gamma camera and tomographic reconstruction. We were ableto demonstrate the feasibility of our suggested approach in phantom experimentsas well as in a first patient test.

8

Acknowledgements This work was partially funded by the DFG cluster of excel-lence MAP, the TUM Institute for Advanced Study (funded by the German Ex-cellence Initiative), and the Bayerische Forschungsstiftung (project RoBildOR).We would like to thank Alexander Schoch for the integration of the ArUco trackerin the software framework.

References

1. Cherry, S.R., Sorenson, J.A., Phelps, M.E.: Physics in Nuclear Medicine. ElsevierHealth Sciences (April 2012)

2. Matthies, P., Gardiazabal, J., Okur, A., Vogel, J., Friebe, M., Lasser, T., Navab,N.: Mini gamma cameras for intra-operative nuclear tomographic reconstruction.Medical Image Analysis (2014)

3. Wendler, T., Hartl, A., Lasser, T., Traub, J., Daghighian, F., Ziegler, S., Navab, N.:Towards Intra-operative 3D Nuclear Imaging: Reconstruction of 3D RadioactiveDistributions Using Tracked Gamma Probes. In: Springer LNCS 4792 (MICCAI).(2007) 909–917

4. Bluemel, C., Schnelzer, A., Okur, A., Ehlerding, A., Paepke, S., Scheidhauer, K.,Kiechle, M.: Freehand spect for image-guided sentinel lymph node biopsy in breastcancer. European Journal of Nuclear Medicine and Molecular Imaging 40(11)(2013) 1656–1661

5. Mandapathil, M., Teymoortash, A., Heinis, J., Wiegand, S., Gldner, C., Hoch,S., Roeler, M., Werner, J.A.: Freehand spect for sentinel lymph node detectionin patients with head and neck cancer: first experiences. Acta Oto-laryngologica134(1) (2014) 100–104

6. Rieger, A., Saeckl, J., Belloni, B., Hein, R., Okur, A., Scheidhauer, K., Wendler,T., Traub, J., Friess, H., Martignoni, M.: First experiences with navigated radio-guided surgery using freehand spect. Case Rep Oncol. 4(2) (2011) 420–425

7. Olson, E.: AprilTag: A robust and flexible visual fiducial system. In: Proceedingsof the IEEE International Conference on Robotics and Automation (ICRA), IEEE(May 2011) 3400–3407

8. Garrido-Jurado, S., Munoz-Salinas, R., Madrid-Cuevas, F.J., Marin-Jimenez, M.J.:Automatic generation and detection of highly reliable fiducial markers under oc-clusion. Pattern Recognition 47(6) (June 2014) 2280–2292

9. Rafii-Tari, H., Abolmaesumi, P., Rohling, R.: Panorama ultrasound for guidingepidural anesthesia: a feasibility study. In: IPCAI’11: Proceedings of the Secondinternational conference on Information processing in computer-assisted interven-tions, Springer-Verlag (June 2011)

10. Magaraggia, J., Kleinszig, G., Wei, W., Weiten, M., Graumann, R., Angelopoulou,E., Hornegger, J.: On the accuracy of a video-based drill-guidance solution fororthopedic and trauma surgery: preliminary results. In Yaniv, Z.R., Holmes, D.R.,eds.: SPIE Medical Imaging, SPIE (March 2014) 903610–903610–9

11. Clear Guide Medical: http://clearguidemedical.com/ (accessed 6/3/2016).

12. Shepp, L.A., Vardi, Y.: Maximum Likelihood Reconstruction for Emission Tomog-raphy. Transactions on Medical Imaging MI-1(2) (1982) 113–122

13. Nicolau, S.A., Pennec, X., Soler, L., Ayache, N.: A complete augmented realityguidance system for liver punctures: First clinical evaluation. Medical Imaging andAugmented Reality, Proceedings 3749 (2005) 539–547

9

14. Zhang, Z.: Flexible Camera Calibration By Viewing a Plane From Unknown Ori-entations. In: Proceedings of the Seventh IEEE International Conference on Com-puter Vision (ICCV). Volume 1., IEEE (1999) 666–673

15. Bouguet, J.Y.: Camera Calibration Toolbox for Matlab http://www.vision.

caltech.edu/bouguetj/calib_doc/ (accessed 6/3/2016).