Reconstruction of 3D Face Surface from Slices: A Literature Survey

Mahmudul Hasan

CPSC 601.20: Biometric TechnologiesDepartment of Computer Science, University of Calgary2500 University Drive NW, Calgary, AB T2N 1N4, Canada

mhasan@cpsc.ucalgary.ca

Table of Contents

Introduction & Background

3D Face Surface ReconstructionApplications of 3D Face Surface Reconstruction

Existing Methods of 3D Face Surface ReconstructionStatistical Approach to Shape from ShadingArea Matching Based on Belief PropagationAnalysis-by-Synthesis Technique Based on 3D Morphable ModelShape Estimation Based on a Set of Feature Point LocationsMinimum Variance Estimation of 3D Face Shape

Comparative Study & Performance AnalysesFindings & Conclusions

IntroductionThe main objective of this study was to perform a comparative analysis of the existing 3D face surface reconstruction algorithmsin terms of their basic methodologies and performance issues.

In addition, this study also focuses on some general 3D surface reconstruction algorithms which can contribute in thereconstruction of 3D faces.

This study has categorized the existing algorithms based on theirrequirement of prior knowledge about the class of solutions.

A detailed comparative study is presented based on the advantages, limitations and areas of application of the studied 3Dface surface reconstruction techniques.

Introduction (cont.)Face recognition has recently received significant attention as one of the most successful applications of image analysis and understanding, especially during the past several years. At least two reasons account for this trend: the first is the wide range of commercial and law enforcement applications, and the second is the availability of feasibletechnologies after 30 years of research [1].

The problem of machine recognition of human faces continues to attract researchers from disciplines such as image processing, pattern recognition, neural networks, computer vision, computer graphics, andpsychology [1].

Even though current machine recognition systems have reached a certain level of maturity, their success is limited by the conditions imposed by many real applications. For example, recognition of face images acquired in an outdoor environment with changes inillumination and/or pose remains a largely unsolved problem [1].

Introduction (cont.)

One of the major findings of Face Recognition Vendor Test (FRVT) 2002 [2] was that the three-dimensional morphable models andnormalization increase the performance of face recognition.

3D model-based methods [3, 4, 5, 6] provide potential solutions topose invariant face recognition.

3D face models are usually derived from laser scanned 3D heads(range data) or reconstructed using shape from shading [7].

Introduction (cont.)Realistic looking facial modeling and animation is one of the mostinteresting and difficult problems in computer graphics [8].

So far, in most of the popular commercially available tools, the 3-D facial models are obtained not directly from images but by laser-scanning of people’s faces [8].

These scanners are usually expensive and a number of hoursof work is required prior to animating the model [8].

To avoid the shortcoming of laser-scan based face modeling, image based face modeling methods have received significant attentions in the past several years [8]. Some of these methodsreconstruct the 3D face from one or more 2D face images (slices).

Background: Existing MethodsIn 1996, J.J. Atick et al presented a technique for recovering 3D face shape from a single 2D image using only the shading information i.e.solving the shape-from-shading problem [3].

In 2004, D. Onofrio et al proposed a method that determines correspondences between surface patches on different views of a face through a modeling of disparity maps based on Markov Random Fields(MWFs) [8].

In 2004, V. Blanz et al presented an algorithm based on a set of feature point locations which produces high-resolution shape estimates of the 3Dface from a 2D face image [9, 10].

In 2006, V. Blanz et al presented an algorithm based on an analysis-by-synthesis technique that estimates shape and pose by fully reproducingthe appearance of the face in the image [10].

In 2006, Z. Zhang et al proposed a minimum variance estimation framework for 3D face reconstruction from multiple views and a new 3D surface reconstruction algorithm based on deformable subdivision mesh[11].

Background: General 3D Surface Reconstruction Algorithms

In 1994, D. Shiwei et al proposed a method where the range image is segmented into regions corresponding to the surface patches on objects. Then, algebraic surfaces are fitted to the range points in these regions bysolving a generalized eigenvector problem [12].

In 1996, G. Barequet et al presented an algorithm which reconstructs a solid model given a series of planar cross-sections. The main contribution of this work was the use of knowledge obtained during the interpolation of neighboring layers while attempting to interpolate a particular layer [13].

In 2002, S.F. Frisken et al presented an efficient method for estimating 3D Euclidian distance field from 2D range images which can be used by many existing algorithms that reconstructs 3D models from range data[14].

Few other 3D surface reconstruction exist which are based on givensample points [15, 16, 17] and labeled image regions [18].

3D Face Surface Reconstruction

The goal of 3D face surface reconstruction is to reconstruct a 3D facegiven one or more 2D face images.

The key approach which has been used to solve this problem is to use a 3D template which is then deformed to represent the target face in thedatabase.

For most of the existing algorithms, the template is built such a way that it is deformable to represent all the existing faces in a particulardatabase.

A great challenge is to deform the reconstructed 3D face to represent the target face in the database when the input 2D images have variations inillumination, pose, and facial expression.

Applications of 3D Face Surface Reconstruction

The reconstructed 3D face can be used to register a face in thedatabase or for the purpose of face recognition.

3D face surface reconstruction has some interesting applications in animation and face recognition where a single view of a person can be used to generate the new views to any pose [3].

It also potentially has some biomedical applications; for example, it can be used to design custom masks for facial burn victims from pre-burn photos. These masks are mostly designed using laser scans of person’s face which are expensive, not convenient, and not always feasible forburn victims [3].

Statistical Approach to Shape from Shading (A1)

This technique can recover 3D face shape from a single 2D face imageusing only the shading information [3].

Shading is the variation in brightness from one point to another in an image [3].

Shading carries information about shape because the amount of light a surface patch reflects depends on its orientation (surface normal) relative to the incident light. So, in the absence of variability in surface reflectance properties (surface material), the variability in brightness can only be due to changes in local surface orientation and hence conveys strong information about shape [3].

The statistical technique, principal component analysis (PCA) has been used to derive a low dimensional parametrization of head shape space [3].

The ideal diffuser model or Lambertian model for surface reflectance is used under this technique [3].

Statistical Approach to Shape from Shading (A1) (cont.)

Lambertian surfaces have two basic properties: firstly, they reflect light diffusely or equally in all directions; secondly, their brightness at any point is proportional to the cosine of the angle between the surface normal at that point and the incident light ray [3].

This algorithm, although idealized, turns out to be a fairly realisticapproximation to many surfaces including human skin [3].

Area Matching Based on Belief Propagation (A2)

This method that determines correspondences between surface patches on different views of the face through a modeling of disparitymaps based on Markov Random Fields (MWFs) [8].

Under this technique, images were acquired by trinocular calibrated cameras and correspondences between the three views weredetermined [8].

To deal with the problems of occlusions and textureless regions, disparity maps were modeled with Markov Random Fields (MRFs), in order to propagate information from textured to textureless regions[8].

The Belief Propagation algorithm is applied to obtain the maximum-a-posteriori estimation of the disparity maps [8].

In order to reduce false matching due to occlusions, outliers wereeliminated by epipolar constraint check [8].

Area Matching Based on Belief Propagation (A2) (cont.)

In the above mentioned trinocular calibrated camera system, one of the cameras, taken as a reference (master) has reasonable frontal, occlusions free view, while the others (slaves) show some occlusions[8].

The proposed algorithm computes two dense disparity maps between the master and the other two slaves views, and each map is modeledby one pairwise MRF [8].

The marginal probability is estimated performing Belief Propagation(BP) iterations on the MRF [8].

At the end of the process, the MRF results are coupled in order to satisfy epipolar constraint on the triplet of images and hence toeliminate outliers [8].

Area Matching Based on Belief Propagation (A2) (cont.)

Fig. 1. Face image triplet [8]

Fig. 2. Reconstructed 3D face [8]

Analysis-by-Synthesis Technique Based on 3D Morphable Model (A3)

(cont.)

In order to solve the ill-posed problem of reconstructing an unknown shape with unknown texture from a single image, the morphable model approachuses prior knowledge about the class of solutions [10].

In case of 3D face reconstruction, this prior knowledge is represented by a parametrized manifold of face-like shapes embedded in the high-dimensionalspace of general textured surfaces of a given topology [10].

More specifically, the morphable model captures the variations observed within a dataset of 3D scans of examples by converting them to a vectorspace representation [10].

For surface reconstruction, the search is restricted to the linear span of theseexamples [10].

Under this technique, estimation of 3D shapes, texture, pose and lighting aredone simultaneously in an analysis-by-synthesis loop [10].

The main goal of the analysis is to find suitable parameters for the morphable model that make the synthetic image as similar as possible to the originalimage in terms of pixel wise image difference [10].

Analysis-by-Synthesis Technique Based on 3D Morphable Model (A3)

(cont.)

Fig. 3. The top row shows the reconstructions of 3D shape and texture. In the second row, results are rendered into the original images with pose and illumination recovered by the algorithm. The third row shows novel views [10].

Shape Estimation Based on a Set of Feature Point Locations (A4)

From a small number of 2D positions of feature points, the algorithm can recover 3D shape of human faces at high resolution, inferring both depth and the missing vertex coordinates [9].

The system is based on a morphable model that has been built from laser scans of 200 faces, using a modified optical flow algorithm to compute dense point-to-point correspondence. Each face is represented by the coordinates of 75972 vertices at a spacing of less than 1mm. 140 most relevant principal components have been used [9].

For shape reconstruction, the user clicks on feature points in the image and the corresponding points on the 3D reference model. Good results areachieved with 15 to 20 points [9].

Due to the automated 3D alignment, no estimate of pose, position and size is required. The system successfully compensates for rotation, scaling andtranslation [9].

The color values of the image are mapped as a texture on the surface, and missing color values are reflected from visible parts or filled in with theaverage texture of the morphable model [9].

Shape Estimation Based on a Set of Feature Point Locations (A4) (cont.)

Fig. 4. From an original image at unknown pose (top, left) and a frontal starting position (top, right), the algorithm estimates 3D shape and pose from 17 feature coordinates, including 7 directional constraints (second row). 140 principal components and 7 vectors for transformations were used. The third row shows the texture-mapped result. Computation time is 250ms [9,10].

Minimum Variance Estimation of 3D Face Shape (A5)

This 3D face surface reconstruction method is based on deformable mesh[11].

The developed system uses six synchronized cameras to capture face imagesfrom six different views [11].

Then, a minimum variance estimation framework for 3D face reconstruction isapplied to reconstruct a personalized 3D model of the face [11].

Next, a 3D surface reconstruction algorithm based on deformable subdivision mesh is applied to the images captured from different views to get more observations of the 3D face, especially the depth information, which could notbe obtained from a single image directly [11].

This algorithm continuously deforms a triangular mesh to minimize an energy function that measures the matching cost of input images [11].

Finally, the minimum variance estimation is again used to refine the result ofthe 3D surface reconstruction algorithm [11].

Minimum Variance Estimation of 3D Face Shape (A5) (cont.)

Fig. 5. Synchronized images captured from six views [11]

Fig. 6. (a) 2D face alignment; (b) Initial 3D shape estimated from the 2D facial feature points; (c) The deformable mesh; (d) Result of 3D surface reconstruction; (e) The 3D shape estimated from 3D points [11]

Comparative Study

Algorithm Number of

2D images used

Depends on the registered faces in the database

A1 1 YES

A2 3 NO

A3 1 YES

A4 1 YES

A5 6 YES

Comparative Performance Analyses

Algorithm Advantages Limitations Suitable areas of application

A1

Suitable for various pose generation.

Not optimized for speed or output quality.Albedo is assumed to be constant.Requires prior knowledge about the class of solutions.

Face recognition, animation, 3D transformation etc.

A2

Capable to handle occlusions and textureless regions.Generates very accurate 3D face model.Computational cost grows linearly.Belief Propagation is highly parallelizable.

The regularization parameter is determined heuristically.Uses calibrated cameras i.e. it cannot work with unknown camera parameters.

Face recognition, and applications that require 3D face reconstruction without any prior knowledge about the class of solutions.

Comparative Performance Analyses (cont.)

Algorithm Advantages Limitations Suitable areas of application

A3

Uses morphable models of 3D faces which provide a promising technique for face recognition under uncontrolled imaging conditions.Works well even on a wider ethnic variety of faces.Capable to handle occlusions.

Requires prior knowledge about the class of solutions.

Face recognition, 3D transformation etc.

A4

Performs comparatively faster 3D reconstruction from feature points. Computation time for the example presented in [9,10] is 250ms. Uses morphable models of 3D faces which provide a promising technique for face recognition under uncontrolled imaging conditions.

Estimation of illumination is not handled.Requires prior knowledge about the class of solutions.

Face recognition, and computer aided design of non-uniform 3D surfaces.

Comparative Performance Analyses (cont.)

Algorithm Advantages Limitations Suitable areas of application

A5

The overall procedure requires about half a minute of computation time.Reconstructs the 3D face very precisely.

Requires prior knowledge about the class of solutions.Uses calibrated cameras i.e. it cannot work with unknown camera parameters

Face recognition, animation, 3D transformation etc.

Findings & Conclusions

A brief survey of existing 3D face surface reconstruction techniqueshas been conducted under this study.

In addition, this study also focused on some general 3D surface reconstruction algorithms which can contribute in 3D face surfacereconstruction.

Along with the description of the methodologies of five 3D face surface reconstruction algorithms, a detailed comparative analyses of their characteristics, advantages, limitations and the areas of application has been presented under this study.

The comparative study found two broad categories of 3D face surface reconstruction techniques; one of which requires prior knowledge about the class of solutions and other works independently based onthe input 2D images.

Findings & Conclusions (cont.)

The focus of research in 3D face surface reconstruction is shiftingmore towards uncontrolled imaging conditions.

The techniques that employ 3D morphable models for faces seem tohandle the uncontrolled imaging conditions most promisingly.

3D face surface reconstruction under different pose, facial expression and illumination is still a great challenge to the researchers.

Thank You.Questions or Comments?