FIST: Fast Interactive Segmentation of Tumors

Sebastian Steger and Georgios Sakas

Fraunhofer IGD,Department of Cognitive Computing & Medical Imaging,

Fraunhoferstrasse 5, 64283 Darmstadt, GermanySebastian.Steger@igd.fraunhofer.de

Abstract. Automatic segmentation methods for tumors are typicallyonly suitable for a specific type of tumor in a specific imaging modal-ity and sometimes lack in accuracy whereas manual tumor segmenta-tion achieves the desired results but is very time consuming. Interactivesegmentation however speeds up the process while still being able tomaintain the accuracy of manual segmentation.

This paper presents a novel method for fast interactive segmentationof tumors (called FIST) from medical images, which is suitable for allsomewhat spherical tumors in any 3d medical imaging modality. The userclicks in the center of the tumor and a belief propagation based iterativeadaption process is initiated, thereby considering image gradients as wellas local smoothness priors of the surface. During that process, instantvisual feedback is given, enabling to intervene in the adaption processby sketching parts of the contour in any cross section.

The approach has successfully been applied to the segmentation ofliver tumors in CT datasets. Satisfactory results could be achieved in 15.21seconds on the average. Further trials on oropharynx tumors, liver tu-mors and the prostate from MR images as well as lymph nodes and thebladder from CT volumes demonstrate the generality of the presentedapproach.

Keywords: tumor segmentation, interactive segmentation, belief prop-agation.

1 Introduction

The accurate 3d segmentation of tumors in medical images is required for anal-ysis, therapy planning and visualization. An example application scenario isthe preoperative target definition for radio frequency ablation of tumors in theabdomen, e.g. liver tumors.

The size and shape of tumors can vary tremendously. However for many typesof tumors – including liver tumors – the shape is typically somewhat spherical.The appearance in prevalent imaging modalities also varies from type to type andfrom instance to instance. Whereas some tumors show a strong image gradientat the boundary and therefore can easily be segmented with high accuracy, theboundaries of other tumors are hardly visible or even not at all in any common

H. Yoshida et al. (Eds.): Abdominal Imaging 2011, LNCS 7029, pp. 125–132, 2012.c© Springer-Verlag Berlin Heidelberg 2012

126 S. Steger and G. Sakas

imaging modality. In those cases the boundary of a tumor can only be guessedby an experienced observer based on indirect signs like irregularities in anatomydue to tumor growth. Even for some tumors with seemingly visible boundaries,the actual tumor may be larger and can only be guessed based on the experienceof the observer.

Whereas manual contouring is very time consuming and therefore prone toerrors due to fatigue of the operator, automatic or semi-automatic segmentationtechniques may in some cases not yield in the required accuracy. Furthermoreautomatic and semi-automatic segmentation methods are typically tailored andfine-tuned to a very specific segmentation problem (e.g. [5]). A generalization todifferent imaging modalities or different types of structures is in many cases notpossible.

Interactive segmentation does usually not suffer from these problems. For asuccessful fast interactive segmentation the following key requirements must bemet:

– The algorithm shall adapt to user interaction in the sense that a man-ual delineation always overrules any automatization. This guarantees thataccuracy identical to manual segmentation can be achieved.

– The smaller the amount of interaction is, the faster a segmentation can beachieved.

– All interactions shall be performed using a user friendly GUI. The usershall receive instant visual feedback from the GUI during the segmentationprocess.

– The method shall be as general as possible, meaning that it should performwell across different types of tumors in different imaging modalities.

Many interactive segmentation techniques exist. Popular methods are based onseeded region growing [1], interactive graph cuts [4], simple interactive objectextraction [7] or binary partition trees [11]. A comparative evaluation of thoseinteractive segmentation algorithms can be found in [9]. Furthermore interactivemethods based on the adaption of contours have been presented [8].

In this work a novel interactive 3d segmentation method for more or lessspheric structures such as tumors is presented. It has successfully been appliedto the segmentation of liver tumors from CT and MR images as well as to otherstructures.

2 Segmentation Process

The workflow of the segmentation process from the user’s perspective is as fol-lows: In a cross section view of a 3d medical volume the user initially sets aseed point in the approximate center of the target structure (Fig. 1(a)). Aninitial surface around that point purely based on image features is shown (Fig.1(b)). In many cases this initial contour is rather discontinuous. Within very fewseconds this contour iteratively adapts in order to meet a compromise betweensmoothness and image criteria (Fig. 1(c)). During this adaption process the user

FIST: Fast Interactive Segmentation of Tumors 127

(a) Click on seed point (0.0) (b) Initial surface (1.1) (c) Adaption (2.6)

(d) User interaction (4.6) (e) Adaption (cont.) (6.6) (f) Final Result (7.9)

Fig. 1. Interactive segmentation workflow (duration in seconds)

may navigate through the volume and observe the adaption process. In case heor she is not satisfied with the adaption, the user can intervene by manuallysketching parts of the boundary in any cross section (Fig. 1(d)). The adaptionprocess then incorporates this input as a constraint in the subsequent iterations(Fig. 1(e)). The user must stop the process as soon as he or she is satisfied withthe segmentation result (Fig. 1(f)).

3 Adaption Algorithm

3.1 Data Structure

The target structures are assumed to be somewhat spherical. For such objects atleast one central point – the seed point – exists, which can be directly connectedto each point on the object’s surface without intersecting the surface elsewhere.Thus the surface can uniquely be described by assigning a single radius to eachmember of a set ofm angles uniformly covering all possible angles in the 3d space(see Fig. 2): [0, ..,m− 1] �→ R

+, or [0, ..,m− 1] �→ N in the discrete version. Verysimilar data structures are used for the segmentation of gliomas from MRI [6]and lymph nodes from CT images [2].

128 S. Steger and G. Sakas

Fig. 2. Rays are sent out from a given seed point. Segmentation is given by an assign-ment of a radius to each ray. It is obtained jointly by globally minimizing an energyfunction using belief propagation.

An energy is assigned to each possible configuration r ∈ Nm, a low energy for

likely surfaces and high energy for unlikely surfaces. It depends on the image, theuser input and local shape priors. The actual segmentation process then comesdown to an energy optimization problem.

3.2 Energy Functional

The energy to be minimized is given by

E(r) =

i<m∑

i=0

⎛

⎝c(ri · ai) + λ ·∑

j∈N(i)

c(ri, rj)

⎞

⎠ , (1)

where λ is a balancing weight, ai denotes a unit vector in the direction of rayi, the unary term c(x) represents a cost for a single point and the pairwiseterm c(ri, rj) represents a cost for the transition between the assignment oftwo neighboring rays. The neighborhood is denoted by N(i) and includes allneighboring rays.

Image Based. The image based contribution to the unary cost term c(x) maydepend on many different image properties. The more is known about the targetstructure, the more specific this term can be. For example if a typical grayvalue range was known, the term could penalize configurations in which parts ofinterior are not within that range, as seen in [12]. Furthermore the term couldincorporate gray value homogeneity in the interior and/or exterior or even learntgray value profiles.

FIST: Fast Interactive Segmentation of Tumors 129

All of these examples have in common that something about the structurehas to be known in advance and thus generality is lost. Instead we only deployan image gradient based cost which is given by

c(x) = max(0, gmax − |∇(Gσ ∗ I(x))|), (2)

where gmax is the maximal gradient strength, ∇ is the gradient operator, ∗ isthe convolution operator and Gσ is a gaussian kernel. Please note that the costsare bound to [0, gmax] in order to not favor very large gradients potentially notbelonging to the object’s boundaries. The gaussian kernel is used to suppressnoise. A non local minimum suppression technique can additionally be used inorder to favor the exact object boundary in areas that all have a larger imagegradient than gmax.

Shape Preserving Cost. To ensure a smooth contour, similar radii of neigh-boring rays shall be preferred. This is reflected by the shape preserving cost:

cshape(i, j) =rmax

n· |ri, rj |α , (3)

where ri and rj represent the radii of two neighboring rays, rmax

n is the distancebetween two neighboring radii and α allows to over proportionally penalize largedifferences of neighboring radii.

User Input. As soon as the user draws a contour the unary costs are adaptedin the following way: For all drawn points, the closest ray with the direction aand the euclidean distance d to the seed point is computed. Then the costs areset to

c(d · a) ={0 d = d∞ otherwise

. (4)

This ensures that the optimal solution – the segmentation surface – will gothrough the manually drawn points. Due to the shape preserving cost, similarradii will be assigned to the surrounding rays.

Optimization. In [2], a gradient descent like approach was successfully used ina similar optimization problem. However, due to the interactive change of theenergy functional, the optimal solution may ”jump” from one to another localoptimum and thus local optimization approaches are not suitable.

Since the energy given by equation (1) depends on a unary term and pairwiseconnections between direct neighbors only, the energy minimization problem issuited to be minimized by the belief propagation algorithm [10] or the graphcut algorithm [3]. Even though the latter showed slightly superior optimizationperformance in some cases [13], we decided to use the former as the interactiveadaption of the energy functional can easily be incorporated.

130 S. Steger and G. Sakas

Fig. 3. Segmentation of liver tumors from CT images

(a) CT: Lymph node (b) CT: Enlarged nodes (c) CT: Bladder

(d) MRI: Liver tumor (e) MRI: Tumor in theoropharynx

(f) MRI: Prostate

Fig. 4. Application to different somewhat spherical structures from different imagingmodalities

FIST: Fast Interactive Segmentation of Tumors 131

The general idea of the algorithm is that messages are iteratively passed fromnodes to neighboring nodes thereby inferring the most likely assignment. In ourcase we use the algorithm to jointly infer the most likely radius for each ray.Messages are passed between neighboring rays. After each iteration the mostlikely assignment is computed and the corresponding surface is shown in thegraphical user interface, giving the operator instant feedback and the chance tointervene.

4 Experiments

The presented approach has been applied to the segmentation of 24 liver tu-mors from 3 CT images. The operator clicked in the center of the lesion andthen interacted with the system until the expected contour was shown. In allcases this was possible because the boundaries of all tumors could be directlyconnected from the seed point without intersecting the contour elsewhere. Forspherical tumors almost no interaction was required, whereas for others with cor-ners required heavy interaction. The interactive segmentation of a single tumortook 15.21 seconds on the average. The standard deviation was 16.64 seconds.Figure 3 shows some sample results.

In order to test the generality of the presented interactive segmentation tech-nique, it was applied to various somewhat spherical structures from differentimaging modalities: Lymph nodes and the urinal bladder from CT as well asthe prostate, liver tumors and tumors in the oropharynx from MR images. Allof these structures could be satisfactorily segmented. Results can be seen inFigure 4.

5 Conclusion

This paper presented a novel method for fast interactive segmentation of tumors(FIST). From all possible surfaces around a user defined seed point, the mostlikely surface is computed by minimizing an energy functional using belief prop-agation. The energy of a surface depends on the image gradient and the localsmoothness. During the adaption process the user gets instant visual feedbackand is able to interactively set specific points of the surface in all three orthogo-nal cross sections. The approach has successfully been tested on liver tumors inCT images as well as on other kind of structures in CT and MR images.

While achieving similar accuracy than manual segmentation, it is a lot faster(on the average only 15.21 seconds for each entity) and therefore feasible to use itin clinical routine, for example for target definition in abdominal intervention. Incontrast to automated segmentation techniques, FIST is not limited to a specifictype of structure in a single imaging modality, but can be used on any somewhatspherical in any 3d imaging modality. Furthermore the user has a major impacton the segmentation result, so that even tumors with only little contrast can stillbe segmented based on the operator’s experience.

132 S. Steger and G. Sakas

Future work includes research on further image based and shape preservingcosts in order to require less user interaction to obtain segmentation results evenfaster. This may however sacrifice generality.

Acknowledgements. This work is partially funded within the NeoMark project(FP7-ICT-2007-2-224483) by the European Commission.

References

1. Adams, R., Bischof, L.: Seeded region growing. IEEE Transactions on PatternAnalysis and Machine Intelligence 16(6), 641–647 (1994)

2. Barbu, A., Suehling, M., Xu, X., Liu, D., Zhou, S., Comaniciu, D.: AutomaticDetection and Segmentation of Axillary Lymph Nodes. In: Jiang, T., Navab, N.,Pluim, J.P.W., Viergever, M.A. (eds.) MICCAI 2010, Part I. LNCS, vol. 6361,pp. 28–36. Springer, Heidelberg (2010)

3. Boykov, Y., Veksler, O., Zabih, R.: Fast approximate energy minimizationvia graph cuts. IEEE Transactions on Pattern Analysis and Machine Intelli-gence 23(11), 1222–1239 (2001)

4. Boykov, Y.Y., Jolly, M.P.: Interactive graph cuts for optimal boundary & regionsegmentation of objects in n-d images. In: Proc. Eighth IEEE Int. Conf. ComputerVision, ICCV 2001, vol. 1, pp. 105–112 (2001)

5. Dornheim, L., Dornheim, J., Rossling, I., Monch, T.: Model-based segmentation ofpathological lymph nodes in ct data, vol. 7623, p. 76234V. SPIE (2010)

6. Egger, J., Bauer, M., Kuhnt, D., Carl, B., Kappus, C., Freisleben, B., Nimsky, C.:Nugget-Cut: A Segmentation Scheme for Spherically- and Elliptically-Shaped 3dObjects. In: Goesele, M., Roth, S., Kuijper, A., Schiele, B., Schindler, K. (eds.)DAGM 2010. LNCS, vol. 6376, pp. 373–382. Springer, Heidelberg (2010)

7. Friedland, G., Jantz, K., Rojas, R.: Siox: simple interactive object extraction instill images. In: Proc. Seventh IEEE Int. Multimedia Symp. (2005)

8. Kass, M., Witkin, A., Terzopoulos, D.: Snakes: Active contour models. Interna-tional Journal of Computer Vision 1(4), 321–331 (1988)

9. McGuinness, K., O’Connor, N.E.: A comparative evaluation of interactive segmen-tation algorithms. Pattern Recognition 43(2), 434–444 (2010)

10. Pearl, J.: Probabilistic Reasoning in Intelligent Systems: Networks of PlausibleInference. Morgan Kaufmann Publishers Inc., San Francisco (1988)

11. Salembier, P., Garrido, L.: Binary partition tree as an efficient representation forimage processing, segmentation, and information retrieval 9(4), 561–576 (2000)

12. Steger, S., Erdt, M.: Lymph node segmentation in ct images using a size invari-ant mass spring model. In: 10th IEEE International Conference on InformationTechnology and Applications in Biomedicine (ITAB) pp. 1–4 (2010)

13. Tappen, M.F., Freeman, W.T.: Comparison of graph cuts with belief propagationfor stereo, using identical mrf parameters. In: Proc. Ninth IEEE Int. ComputerVision Conf., pp. 900–906 (2003)