3D Visualization Method for Maxillofacial Surgical Planning by Using X-ray and Photo Images

Young-In Kim*, Jung-Hyun Park**, Chang-Hun Kim*

Dept. of Computer Science & Engineering, Korea University*,

Dept. of Oral&Maxillofacial Surgery, Dental College, Yonsei University**

{yikim,chkim}@cgvr.korea.ac.kr*, goodputt@hotmail.com**

Abstract

This paper describes a prototype system for predicting of human facial shape and

visualizing of realistic 3D images after maxillofacial surgery for patient with facial

deformities from three input images consisting of lateral Xray, lateral photo and

frontal photo. The basic idea is to join a accurate predicting variation of facial shape

with a 3D visualizing postsurgical facial shape. Firstly, we draw out the important

parts of face, contours and control points in the profile of a patient from Xray image.

Secondly, when a surgeon designates beforehand the change of the direction and

magnitude of a maxillary bones (the lower and upper jawbone) at plan step, the

system forms the postsurgical image using warping method and predicting function

which calculate the change of movement of a soft tissue. The predicting function is

based on the clinical study obtained from 100 patients and the warping method is

implemented considering anatomic relation of facial soft tissue. Finally we generate

the three dimensional images of postsurgical facial shape using the deformation

estimating method which predict the 3dimensional movement of vertices on face.

Keywords: Digital Surgery, Medical Visualization, Surgical Planning System

1 Introduction

Digital dental surgery planning system has many advantages in that of predicting

the postsurgical morphology and appearance of human face, or solving the problem

of a fair facial surface before the actual surgery is carried out. Early works [4,6,7]

need expensive equipment and high computation cost because it require the

complex and laborious processing steps of mapping the soft tissue structure to the

3D skull model which is constructed from 2D CT or MRI data. More over these

systems are not adequate for maxillofacial surgery because subtle malformation

can strongly affect the appearance of a face and these systems is hard to handle to

surgeon. Thus typically, the facial operation surgeon draws the patients predicted

profile using X-ray to give at least a 2D appearance of the future face. In this thesis,

we propose the facial surgery planning system, which computes accurate 2D lateral

facial contour and realistic 3D picture of the post surgical shape.

Our system consists of two modules, one is about the virtual surgery step and the

other is 3D visualization step. In virtual surgery step, we use lateral X-ray of patient

and predicting function for computing predicted soft tissue profile after maxillofacial

surgery. In 3D visualization step, we use two photos, which are frontal and lateral

images of patient, and deformation estimating method for generating 3D

appearance of post surgical facial shape. Figure1 shows the simple procedure of our

surgery predicting system.

a) b) c)

Figure 1: Illustration of system flow for maxillofacial surgery planning.

a) Lateral photo and X-ray image of the presurgical face. b) 2D postsurgical image

after maxillofacial surgical simulation. c) 3D postsurgical appearance.

The remainder of this paper is structured as follow: In section 2 we describe the

motivation of our research. And we give an overview of the system in section 3.

Section 4 we explain the principles of the predicting function, which is used for

virtual surgery. Section5 describes the deformation estimating method of soft

tissue for generating 3D appearance. Finally, we demonstrate and discuss the

proposed system with experimental results obtained from the X-ray image and

frontal and lateral facial photos in section 6.

2 Motivation

There has been much research in the field of facial surgery simulation. Most

facial surgery simulation systems used today in 3D medical image data like CT or

MRI. These 3D medical image data requires high cost and complex preprocessing

for surgical simulation. So we choose the planar medical image data, X-ray image,

traditionally it has been used in clinical area. Early works such as [9] restricted

themselves to 2D lateral image generation of the malformed patients. So we

propose the system which requires a simple and cheap X-ray image data and

generates 3D appearance of postsurgical facial shape.

3 System Overview

This section explains on different procedures, data acquisition and preprocessing

steps, virtual surgery step, and 3D visualization step in the chart of Figure 2. Our

data sources consist of X-ray image, frontal and lateral facial photos.

First of all, an initial facial contour is extracted from the X-ray image and control

points is created on a lateral contour.

Secondly, we simulate the maxillofacial surgery system, which enable to compute

the variations between the presurgical profile and postsurgical profile. As this result

postsurgical profile contour is created and it is warped with the lateral facial photo.

Then we warp the lateral photo to do the virtual surgery result.

Finally, we execute the third pass procedure, 3D postsurgical image generation. In

our system, we individualize the generic facial model and predict the postsurgical

facial images on frontal view. Figure2 shows the overall flow structure of our dental

surgery planning system.

Figure 2: System overview.

3.1 Data Acquisition and Preprocessing

The first pass procedure of our virtual surgery system is the data acquisition and

preprocessing step. In this step, we prepare two 2D wire frame templates composed

of feature points with predefined relation for front and side views and draw out the

facial profile on X-ray image and align facial lateral photo with it.

Because X-ray image maintains the 1:1 ratio of the surgery area, the lateral photo

is transformed by fixed profile on X-ray. Figure 3 shows the result after malformed

profile contours extraction and adaptation of them. These images can be aligned by

rotation and translation and scaling.

a) b)

Figure 3. Facial contour extraction. a) extracted facial profile on X-ray image. b)

Adaptation profile to X-ray image and photo.

3.2 Virtual Surgery Operation

This section explains the process of virtual operation necessary to accomplish

facial surgery simulations. First, we cut the malformed mandible contour and move

it to correct it. Next, the variations of the control points on facial soft tissue contours

between presurgery and postsurgery are computed by predicting function. Finally,

non-control points consisting of soft tissue contours are interpolated by linear

interpolation method.

Figure 4 shows the process of virtual operation. Figure 4a represents the cut and

moved malformed mandible contour. Figure 4b is the result from calculating

movement of the control points on soft tissue. Figure 4c shows the linear

interpolating movements of non-control points on soft-tissue contours.

a) b) c)

Figure 4. Maxillofacial surgery on the hard tissue contour. a) cutting and moving

the mandible. b) calculating the movements of control point on soft tissue. c) linear

interpolating the movements of non-control point on soft tissue contours.

This system makes synthetic postoperative image of post-operation by image

warping technique. Image warping process consists of triangulation and color

interpolation. Delaunay triangulation used on this system defines local areas on

lateral photo, each area is reorganized by virtual surgery, and color information in

area is interpolated.

Figure 5 shows this process about generating warped postsurgical image. Figure

5b shows the divided regions by triangulation and Figure 5d shows the warped

result of the modified model by virtual surgery.

a) b) c) d)

Figure 5. Virtual postsurgical facial image generation. a) Input data : lateral facial

photo. b) Triangulation of input image. c) Virtual operation d) Warped image of the

modified model by virtual surgery

3.3 3D postsurgical image generation

In this section, we explain the process for visualizing 3D postsurgical image

generation. First of all, we make individual face to visualize 3D images. After

preprocessing, we get two wire templates on front and side views and use them to

modify a generic facial model. In this paper, generic facial model with 915 vertices

is modified to make an individualized smooth facial surface. Figure 6 shows the

process of individualizing a generic facial model.

Figure 6. Individualization of generic facial model

4 Computation of the Soft-Tissue Variation

In this paper, we use the predicting function to calculate the variation of

postsurgical soft tissue profile. As the variation of human soft tissue can’t be

generally computed by linear equations, we suppose that it can be computed by

non-linear equations. To formulate our predicting function, we define 8 control

points on hard tissue and 10 control points on soft tissue. Figure 7 shows these

control points. Control points on hard tissue consist of A(ANS), B(A), C(MxI), D(MxM),

E(MnI), F(B), G(Pg), H(Me) and control points on soft tissue consist of A’(Pn), B’(Sn),

C’(A), D’(Ls), E’(Stms), F’(Stmi), G’(Li), H’(B’), I(Pg’), J’(Me’).

Figure 7. Facial control points on hard tissue and soft tissue.

This predicting function is based on the clinical study obtained from 100 patients.

For measuring the variation between presurgical profile and postsurgical profile, we

define eq. 1 as following.

, Eq. 1

The horizontal variation on soft tissue A’ is formulated as following.

Hard control points

Soft control points

Eq. 2

As the preceding equation 2 , other horizontal variation on control points from B’

to H’ is formulated. Equation 3 explains the linear regression for finding values of

.

Eq. 3

:estimated value, :input value , :parameter , :the number of data, : the number of

parameter

Using hard-tissue location changed from virtual surgery produce the soft-tissue

change by soft-tissue movement prediction functions. Each partial function for a

feature point on soft-tissue makes the movement based on multiple feature points

on hard-tissue. Table 1 shows the example of two partial functions.

Table1 Two functions about hard and soft tissue relation

Soft-tissue Hard-tissue

VPn 0.44409 * ANS – 0.27588 * vA + 0.05905

VB’ 0.65773 * vMe – 0.13109 * hMe + 0.21406 * hMnM – 1.31458

11 soft tissue control points predicted by that of hard tissue control point is as

small as this system represent the smooth facial outline. So, additional points of soft

tissue is defined as follows:

Eq. 4

, where p is the control point coordinate, w is the maximum length effected by p,

and p’ is the point on soft-tissue. [Figure 4] shows that this system determines

coordinate of the feature point on soft-tissue by prediction function, and interpolate

the additional points by equation 4.

Maxillofacial surgeon generally says that an error tolerance on maxillofacial

surgery is less than 2mm. In figure 8, we represent the comparison of result on

actual surgery with estimated surgery. You can see that the differences between this

two graphs are not greater than 2 mm.

Figure 8. The comparison of result between actual surgery and estimated surgery.

5 Deformation Estimating Method

Our system makes frontal postsurgical facial image by the deformation estimating

method. 3D visualization process consists of the individualization of a generic facial

model and the calculation of 3dimensional movement of soft tissue on face. For

calculating the movement of control points, we define the deformation estimating

method. First, we measure 3 dimensional variations of control points between soft

tissue and hard tissue from CT data. Figure 8 shows the 3 dimensional control points

of the soft tissue. The number of 3 dimensional control points is seven, Pn, Sn, A, Ls,

Li, B’, Pg’ , except Stms, Stmi, Me.

a) b) c)

Figure 9. 3 dimensional control points from CT data. a) soft tissue control points on

lateral view b) control points on frontal view. c) control points from –90° to +90° on

sectional view.

Secondly, we calculate the 3 dimensional variations on facial soft tissue. For this

computation, an axis of coordinates is established on individual facial model. We

use the following equation to compute the postsurgical soft tissue control points. In

this equation 5, P(x,y,z) is a control point on soft tissue contour before virtual

maxillofacial surgery and P’(x’,y’,z’) is computed position of it after virtual

maxillofacial surgery. P’’(x’’,y’’,z’’) in figure 10 is a temporary point projected on xy-

plane a point P(x,y,z).

, , . Eq. 5.

a) b)

Figure 10. Computing variation of 3D difference before and after maxillofacial

Z

surgery. a) lateral comutation model. b) sectional computation model.

6 Results

The goal was to predict the facial shape after procedure in maxillofacial surgery.

Figure 11-12 show the shape of soft tissue after maxillofacial surgery. The

calculations were carried out on a Intel Pentium-III 700 processor, 256MB RAM,

Microsoft Windows NTTM Workstation 4.0 and our system was implemented by using

Microsoft Visual C++TM 6.0 and SGI OpenGL. We can compare a estimated

postsurgical image with a actual postsurgical image. (See Figure11). We can find the

high similarity of two images.

a) b) c)

Figure 11. Comparison simulated results with actual surgery. a) Input data: presurgical images. b)

Estimated postsurgical images. c) Actual postsurgical image

a) b) c) d) f)

f)

Figure 12. 3D images after maxillofacial surgery. a,b) presurgical frontal and lateral

pictures of patient. c) Estimated postsurgical facial image. d,e) Actual postsurgical

frontal and lateral photo. f) 3d visualization of estimated postsurgical result.

7 Conclusion

We present a system which enables us to predict the deformations of the facial

shape after surgical procedures. Our system can generate a high realistic and

accurate postsurgical images by predicting function which obtained from 100

clinical studies. In our system, maxillofacial surgeon can easily extract the outline

profile and predict the postsurgical facial shape.

8 Reference

[1] Haider, A. Md., Takahashi, E. and Kaneko, T., Automatic Reconstruction of 3D

Human Face from CT and Color Photographs, IEICE Trans. Inf. & Syst..,

vol.E81-D, No.9, pp.1287-1293, Sep. 1999.

[2] Haider, A. Md., Takahashi, E. and Kaneko, T., A 3D face reconstruction

method from CT image and color photographs, IEICE Trans. Inf. & Syst..,

Vol.E81-E, No.10, pp.1095-1102, Oct. 1998.

[3] Hounsfield, G.N., and Ambrose, J.A. Computerized transverse axial scanning

tomography. British Journal of Radiology, pp.1016-1022, 1973

[4] Koch, R.M., Gross, M.H., Carls, F.R., von Büren, D.F., Fankhauser, G. and

Parish, Y.I.H., Simulating facial surgery using finite element models,

Computer Graphics(SIGGRAPH’96 Proceedings), pp.421-429, Aug. 1996

[5] Lee, W-S., Thalmann, N.M., Fast head modeling for animation, Image and

Vision Computing, pp.355-364, 2000.

[6] Xia, J., Wang, D., Samman, N., Yeung, R.W.K., Tideman, H., Computer-

assisted three-dimensional surgical planning and simulation: 3D color facial

model generation, International journal of oral & maxillofacial surgery,

Vol.29, No.1, pp.20-10, Feb. 2000.

[7] Xia, J., Ip, H.H.S., Samman, N., Wang, D., Kot, C.S.B., Yeung, R.W.K.,

Tideman, H., Computer-assisted three-dimensional surgical planning and

simulation: 3D virtual osteotomy, International journal of oral & maxillofacial

surgery, Vol.29, No.1, pp.11-17, Feb. 2000.

[8] Dr.Pss,(http://www.bit.co.kr/medical- i nfo/html/ drpss.htm )

[9] QuickCeph Systems, (http://www.quickceph.com/)