Articulated Soft Objects forMultiview Shape and Motion Capture

Ralf Plankers and Pascal Fua, Member, IEEE

Abstract—We develop a framework for 3D shape and motion recovery of

articulated deformable objects. We propose a formalism that incorporates the use

of implicit surfaces into earlier robotics approaches that were designed to handle

articulated structures. We demonstrate its effectiveness for human body modeling

from synchronized video sequences. Our method is both robust and generic. It

could easily be applied to other shape and motion recovery problems.

Index Terms—Body tracking, shape and motion, multiple views, implicit surfaces.

�

1 INTRODUCTION

RECENTLY, many approaches for tracking and modeling articulated3D objects have been proposed. They have been used to capturepeople’s motion in video sequences with potential applications toanimation, surveillance, medicine, and man-machine interaction.

Such systems are promising. However, they typically useoversimplified models, such as cylinders or ellipsoids attached toarticulated skeletons. These models are too crude for preciserecovery of both shape and motion. We propose a framework thatretains the articulated skeleton but replaces the simple geometricprimitives by soft objects. Each primitive defines a field functionand the skin is taken to be a level set of the sum of these fields. Thisimplicit surface formulation has the following advantages:

. Effective use of stereo and silhouette data. Definingsurfaces implicitly allows us to define a distance functionfrom data points to models that is both differentiable andcomputable without search.

. Accurate shape description from a small number of

parameters. Varying a few dimensions yields models thatcan match different body shapes and allow both shape andmotion recovery.

. Explicit modeling of 3D geometry. The model can beprojected into images to predict the expected location ofimage features and occluded areas, thereby makingsilhouette extraction more robust.

The Articulated Soft Object Model depicted by Figs. 1a, 1b, 1c, and1d is at the root of our approach. It is equally applicable to othervertebrates, such as the horse and cow of Fig. 1e and, moregenerally, to nonpolyhedral articulated objects.

The main contribution of this paper is to show that thearticulated implicit surface formalism is powerful enough to yieldqualitatively good results under very difficult conditions, even inthe absence of the sophisticated probabilistic frameworks that manyother approaches rely on. Furthermore, it is amenable to amathematically elegant and simple implementation by extendingwell-known robotics results [7].

We integrate our formalism into a complete framework fortracking and modeling and demonstrate its robustness usingtrinocular video sequences of complex 3D motions. To validate it,we focus on using stereo and silhouette data because they arecomplementary sources of information. Stereo works well on bothtextured clothes andbare skin for surfaces facing the camera but fails

where the view direction and the surface normal is close to beingorthogonal which is exactly where silhouettes provide robustinformation.

In the remainder of this paper, we first describe related

approaches to articulated shape and motion recovery. We then

describe our model in more detail and introduce our optimization

framework. Finally, we present reconstruction results on complex

human motions.

2 RELATED WORK

Modeling humans from images involves recovering both body-shape and motion. Most existing approaches can be classified asaddressing one or the other of these two problems, which is whatwe do in this section. However, in the following sections, we willargue that these two issues are intimately connected and one of theoriginal features of our approach is that it simultaneously capturesboth shape and motion.

2.1 Human Shape Reconstruction

Digital photogrammetry and structured light have long been usedto quickly acquire the stereo information needed to model livesubjects. By using large numbers of cameras and space carvingtechniques [28], [8], it is possible to go even further and to capturethe shape at frame-rate. However, these approaches mainly focuson rerendering captured scenes from new viewpoints usingdisparity maps and multiple textures to interpolate betweenobserved viewpoints. This is, in part, because the data producedin this way cannot be directly animated to create new motions:Animating the body shape requires partitioning it into body partsand attaching those parts to an articulated skeleton [2], [13], [30].

At the other end of the cost and complexity range are methodsthat accept simple photographs as input [18], [21]: They usesilhouettes to deform generic 3D models and create 3D facsimilesof individual people that are suitable to represent articulatedmovement in a virtual world. Such systems are low-cost but do notproduce a realistic body shape for a specific individual. Instead,they rely on texture mapping to hide the deficiencies of theapproximated shape.

2.2 Human Motion Capture

In recent years, there has been much interest in capturing complexmotions solely by analyzing video sequences. Single camerasolutions such as [4], [32], [29] would be ideal to process standardimage sequences. However, they are not always robust, in partbecause image data is inherently noisy and, in part, because it isinherently ambiguous [24].

In contrast, using multiple cameras leads to a considerablereduction in the size of the search space and a considerable

1182 IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 25, NO. 9, SEPTEMBER 2003

. The authors are with the Computer Vision Laboratory, EPFL, 1015Lausanne, Switzerland. E-mail: Pascal.Fua@epfl.ch.

Manuscript received 8 June 2002; revised 17 Sept. 2002; accepted 17 Sept.2002.Recommended for acceptance by D. Fleet.For information on obtaining reprints of this article, please send e-mail to:tpami@computer.org, and reference IEEECS Log Number 115658.

Fig. 1. Articulated soft objects: (a) Skeleton. (b) Volumetric primitives used tosimulate muscles and fat tissue. (c) Polygonal surface representation of the skin.(d) Shaded rendering. (e) A cow and a horse modeled using the same technique.

0162-8828/03/$17.00 � 2003 IEEE Published by the IEEE Computer Society

improvement in robustness, at the cost of having to deal with a largeamount of data, most of which is redundant. Systems such as thoseproposed by [6], [10], [14], amongmany others, handle this problemeffectively. Many of these approaches rely on sophisticatedstatistical models [12], [9], [5] to further reduce the size of the searchspace involved in modeling the whole human body. Theseapproaches, however, typically use oversimplified models, such ascylinders or ellipsoids attached to articulated skeletons. Suchmodels are too crude for precise recovery of both shape and motionand it is this shortcoming that our approach addresses. The physics-based spring system model proposed in [19], [20] is one the rareexceptions to this modeling approach: Limbs are represented byvolumetric primitives that are attached to one another by springs.Relatively loose connections between the body parts replace theusual underlying rigid structure. This allows for more flexibility inthe model. However, with such a model, it is difficult to constrainjoint angles in order to enforce anatomically correct postures and themethodhas beenmostly demonstrated onvery clean silhouette data.

For a more detailed description of existing approaches, we referthe interested reader to recent surveys of visual motion captureresearch [1], [17], [23].

3 ARTICULATED SOFT OBJECTS

The human body model depicted by Figs. 1a, 1b, 1c, and 1d is at theroot of our approach. It was originally developed solely foranimation purposes [31]. Smooth implicit surfaces, also known asmetaballs or soft objects [3], [11], are attached to an articulated skeletonand are arranged in an anatomically-based approximation.

This particular human body model includes 230 metaballs. Inorder to prevent body parts from blending into each other, the bodyis segmented into 10distinct parts:Arms, legs, and torso are split intoupper and lower parts. When computing the implicit surfaces, onlymetaballs belonging to the same segments are taken into account.Weconstrain the left and right limbs to be symmetric. The head, hands,and feet are explicit surfaces that are attached to thebody.Fordisplaypurposes, a polygonal skin surface is constructed via B-splinepatches over control points computed by ray casting [31].

Our goal is to use video-sequences to estimate the model’s shapeand derive its position in each frame. To this end, in the followingsections, we reformulate the animation model as an articulated softobject. We outline this formalism below and refer the interestedreader to earlier publications [26], [25] for additional details.

3.1 State Vector

Body shape and position are controlled by a state vector�, which is aset ofparameters defining joint locations and limb sizes.Weassign toeach body part variable length and width coefficients. Thesedimensions change from person to person but we take them to beconstant within a particular sequence. This constraint could berelaxed, for example to model muscular contraction. The motion isgiven by rotational degrees of freedom of the articulated skeleton’sjoints in all frames and by six parameters of global position andorientation.

3.2 Metaballs

The skin metaball surface S is a generalized algebraic surface thatis defined as a level set of the summation over n three-dimensionalGaussian density distributions, each called a primitive [3]. S is theimplicit surface defined by the level set F ðx; y; zÞ ¼ L:

S ¼ x; y; z½ � 2 R3 j F ðx; y; zÞ ¼ L� �

ð1Þ

F ðx; y; zÞ ¼Xni¼1

fiðx; y; zÞ ð2Þ

fiðx; y; zÞ ¼ expð�2diðx; y; zÞÞ: ð3Þ

Here, di represents the algebraic ellipsoidal distance introduced in(4) and L is taken to be 0.5. For simplicity, we omit the index i forspecific primitives wherever the context is unambiguous.

3.3 3D Quadratic Distance Function

We use ellipsoidal metaballs because they allow accurate modelingof human limbs with relatively few primitives. To express thetransformations of these implicit surfaces that are caused by theirattachment to an articulated skeleton, we write the ellipsoidaldistance functions d of (3) as follows:

dðx;�Þ ¼ xT � ST� �QT

� �Q� � S� � x; ð4Þ

where Q� and S� are 4� 4 matrices that represent the metaball’sshape and its skeleton-induced transformation, respectively, andx ¼ x; y; z; 1½ �T is a 3D point.

More specifically, Q� defines the scaling and translation alongthe metaball’s principal axes. The size and position of a metaball isrelative to the segment it is attached to. A length parameter notonly specifies the length of a skeleton segment but also the shape ofall attached metaballs in the direction of the segment. Widthparameters influence the metaballs’ shape in the other directions.S� is a 4� 4 rotation-translation matrix from the world frame tothe frame to which the metaball is attached. It is defined as a seriesof matrix multiplications where each matrix corresponds to thetransformation introduced by one joint.

We can now compute the global field function F of (2) byplugging (4) into the individual field functions and adding upthese fields for all primitives. In other words, the field functionfrom which the model surface is derived can be expressed in termsof the Q� and S� matrices and so can its derivatives [26].

4 ESTIMATION FRAMEWORK

The expected output of our system is the instantiated state vector �of Section 3.1 that describes the model’s shape and motion. This isa highly nonlinear problem because the model consists of anarticulated set of implicit surfaces. As a result, it contains rotationsin Euclidean space as well as quadratic and exponential distancefunctions. Simplifying the volumetric models, replacing theperspective transform by an orthographic one and using a differentrepresentation for rotational joints can be used to linearize parts ofthe problem [4]. Doing so, however, tends to lose in generality.Instead, we chose to solve full nonlinear problems using theLevenberg-Marquart least-squares estimator to minimize thedistance between observations and model.

4.1 Least-Squares Estimator

In practice, we use the image data to write nobs observationequations of the form

F ðxi;�Þ ¼ Lþ �i; 1 � i � nobs; ð5Þ

whereF is the global field function of (2),L is the level value of (1),xi

is a data point, and �i is treated as an independently distributedGaussian error term. We then minimize vTPv, where v ¼½�1; . . . ; �nobs

� is the vector of residuals and P is a diagonal weight

matrix associated with the observations. Our systemmust be able todeal with observations coming from different sources, here, stereo

or silhouettes, that may not be commensurate with each other. Wetherefore use the following heuristic that has proven to be veryeffective. To ensure that the minimization proceeds smoothly, we

multiply the weight ptypei of the ntype individual observations of agiven type by a global coefficient ctype computed as follows:

Gtype ¼

ffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiP1�i�nobs ;j¼type

r�F ðxtypei ;�Þ

�� ��2r

ntype

ctype ¼�type

Gtype;

ð6Þ

where �type is a user-supplied coefficient between 0 and 1 thatindicates the relative importance of the various kinds of

IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 25, NO. 9, SEPTEMBER 2003 1183

observations. This guarantees that, initially at least, the magni-tudes of the gradient terms for the various types have appropriaterelative values.

4.2 Data Constraints

In this work, we concentrate on combining stereo and silhouettedata. Fig. 2 illustrates their complementarity: In this example, weused a single stereo pair. In Fig. 2c, only stereo data in the form of acloud of 3D points derived from the disparity map, was used. Thestereo data is too noisy and shallow to sufficiently constrain themodel. As a result, the fitting algorithm tends tomove it too far awayfrom the 3D data and to compensate by inflating the arms to keepcontact with the point cloud. This behavior is very similar to thatobserved when fitting ellipses to noisy 2D points, especially in thecase of nonuniformly distributed points such as those obtained fromstereo reconstruction [15]. Using the silhouettes in addition to thestereo data, however, sufficiently constrains the fitting problem toobtain the much improved result of Fig. 2d.

Because our field-function F is both well-defined and differ-entiable, the observations and their derivatives can be computedboth simply and without search using the matrix formalism ofSection 3.3. We sketch these computations below and, again, referthe interested reader to earlier publications [26], [25] for additionaldetails.

4.2.1 3D Point Observations

Disparity maps such as those of Fig. 2 are used to compute cloudsof noisy 3D points. Each one is used to produce one observation ofthe kind described by (5). Minimizing the corresponding residualstends to force the fitted surface to be as close as possible to thesepoints. Because of the long-range effect of the exponential fieldfunction in the error function F of (2), the fitting succeeds evenwhen the model is not very close to the data. Also, during least-squares optimization, an error measure that approaches zeroinstead of becoming ever greater with growing distance has theeffect of filtering outliers.

To compute the Jacobian of the error function of (2), we mustdifferentiate the individual field functions of (3). Derivatives withrespect to parameter � 2 � can be computed as:

@

@�fðdðx;�ÞÞ ¼ @f

@d

@d

@�¼ �2

@@� dðx;�Þe2dðx;�Þ ; ð7Þ

@

@�dðx;�Þ ¼ 2 � xT � ST

� �QT� � @

@�Q� � S�

� �� x; ð8Þ

where the @@�Q�S�

� �term canbe computed simplywith a fewmatrix

multiplications [26]. In short, these derivatives consist of modulesthat either are inexpensive to compute or need only be computedonce because they are constant over all observations and constraints.

4.2.2 Silhouettes Observations

A silhouette point in the image defines a line of sight tangential tothe surface. For any � 2 �, we define the implicit surfaceSð�Þ ¼ f½x; y; z� 2 R3; F ðx; y; z; �Þ ¼ Lg. Let xð�Þ; yð�Þ; zð�Þ½ � be the

point on the line of sight where it is tangential to Sð�Þ. By definition,

xð�Þ; yð�Þ; zð�Þ½ � satisfies two constraints:

1. Thepoint ison the surface, thereforeF ðxð�Þ; yð�Þ; zð�Þ; �Þ¼L.2. The normal to Sð�Þ is perpendicular to the line of sight at

xð�Þ; yð�Þ; zð�Þ½ �.We integrate silhouette observations into our framework by

performing an initial search along the line of sight to find the

point x that is closest to the model in its current configuration. This

point is used to add one of the observations described by (5). By

construction, the point on the ray with the lowest field value

satisfies the second constraint.A change in model position or size induces a motion of x along

the line of sight so that x remains the point closest to the model.

This involves computing first and second order derivatives for the

Jacobian entries of the form

@2d

@xi@xj¼ 2 � @x

@xi

T

� ST� �QT

� �Q� � S� � @x@xj

;@2d

@xi@�¼

2 � xT � @

@�ST� �QT

�

� ��Q� � S� þ ST

� �QT� � @

@�Q� � S�

� � � @x@xi

;

where xi; xj represent translational degrees of freedom, while �

stands for one of the rotational ones [26]. Again, this involves

evaluating the same @@�Q�S�

� �terms as before and, therefore, very

little extra computation.

5 IMPLEMENTATION AND RESULTS

We initialize the model in one frame of the sequence by clicking on

the approximate locations of several key joints in two images. This

gives us a rough scaling of the skeleton and an approximate model

pose. The system then goes through the following two steps.

1. Frame-to-frame tracking. For a given time step, the systemextracts clouds of 3D points from our synchronized inputvideo sequences using a correlation-based approach [16].These points are used to create observations of the kinddescribed in Section 4.2.1. The tracking process adjusts themodel’s joint angles byminimizing the objective function ofSection 4.1 with respect to the joint angle values relative tothat frame. This modified posture serves as the initializa-tion for the next one. Optionally, the system may use themodel’s projection into the images to derive initialsilhouette estimates, optimize these using image gradients,and derive silhouette observations such as those intro-duced in Section 4.2.2.

2. Global fitting. The results from the tracking in all framesserve as initialization for global fitting to refine the posturesin all frames and to adjust the skeleton and/or metaballparameters to improve the model’s shape. To this end, itminimizes again the objective function of Section 4.1, overall frames and with respect to the full state vector.

1184 IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 25, NO. 9, SEPTEMBER 2003

Fig. 2. The importance of silhouette information for shape modeling. (a) First image of a stereo pair. (b) Corresponding disparity map. (c) and (d) Two different fitting

results. The black curves are the actual body outlines in (a). In (c), no silhouette constraints were used and the fitting puts the model too far away from the cloud. The

system compensates by incorrectly enlarging the primitives. In (d), we used the silhouette constraints provided by the black outlines.

Global fitting is required to correctly model the proportions of

the skeleton and derive the exact position of the articulations

inside the skin surface by simultaneously using as much

information as possible. To stabilize the optimization, we add to

our objective function additional observations that favor constant

angular speeds. Their weight is taken to be small so that they do

not degrade the quality of the fit but, nevertheless, help avoid local

minima in isolated frames and yield smoother and more realistic

motions. In the examples shown below, optimization is performed

on the 27 degrees of freedom that define the shape and motion of

the upper body.

5.1 Using Stereo Alone

The images in Fig. 3 are from 10-second sequences acquired bythree progressive-scan cameras arranged in an inverted “L”configuration and capturing noninterlaced images at 30 Herz withan effective resolution of 640� 400. These sequences feature highlycomplex motions of the subject’s arms and upper body. Note thefrequent self-occlusions as well as merging of body parts, such asgrasping both hands in frame 110.1

Using our correlation-based approach, we created approxi-mately 4,000 3D point observations per frame of the type describedin Section 4.2.1. In the middle row, we show a shaded version of therecovered 3D bodymodel reprojected into the images. In the bottomrow, we overlay the outline of this model on the original images.

Note that in frame 100 the system places the left elbow too far fromthe body but recovers before frame 110. In effect, the arm is “sliding”along the stereo data. In Section 5.2, we will show that usingsilhouettes during the reconstruction fixes this problem. In thisexample, neither hand nor head motion were modeled. Both areinterpreted as rigid objects rigidly attached to neighboring bodyparts. This is why, in frame 40, the left hand is not at the right place,and, in frame 110, the model seems to look in the wrong directionwhen the subject moves his shoulder with respect to the head.

Fig. 4 depicts the tracking of another very complex motion: Inaddition to the unpredictable arm movements, the torso bends andtwists away from the cameras.

In Fig. 5, we consider the case of somebody wearing a baggysweater instead of being bare-chested. Edge-based methods wouldfail on such a sequence because the various deformations of thecloth hide the person’s contour. The stereo data we obtain from theimages is discriminating enough for our estimator to find thecorrect posture. In this case, we optimized only with respect to themotion parameters, the actual body shape being unobservable.Careful examination of those results, however, reveals some of thelimitations of the shoulder model we use: A single ball-and-socketjoint is convenient for motion estimation but it cannot capturerealistic shoulder poses and deformations. A more anatomicallycorrect model [22] is needed.

5.2 Using Silhouette Information

At video rates, we can take the initial guess for the silhouettelocation to be the projected body outline in the previous frame. Wethen optimize this position using an active contour. However, inthe presence of a cluttered and unknown background, even a good

IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 25, NO. 9, SEPTEMBER 2003 1185

1. Higher resolution versions of Figs. 3, 4, 5, and 6 will be made availableon the Digital Library as mpeg movies. They can be accessed at http://computer.org/publications/dlib.

Fig. 4. Selected frames of a trinocular sequence in which the subject performs complex upper body motions coupled with abrupt arm waving. Top row: One of the three

original images at a given time. Center row: Reconstructed body model. Bottom row: Superposition of the original image and of the shaded model.

Fig. 3. Tracking results in frames 20, 40, 80, 100, and 110 of a 300-frame sequence. Top row: Disparity maps. Middle row: Tracking and fitting results. Bottom row:

Projections of the recovered model outline overlaid on the original images. The system correctly tracks until frame 80, misplaces the elbow in frame 100, and recovers by

frame 110. As shown in Fig. 6, using silhouette information will eliminate the error in the frame 100.

initial body-outline estimate does not guarantee convergence of anactive contour to the correct solution. We therefore exploit ourdisparity maps to reject those gradients that are too far from areaswhose distance from the cameras make them likely to be part of thesubject and to correspond to actual silhouettes [27].

We then rerun our fitting algorithm on the 3D stereo dataaugmented by this hypothesized silhouette outline. Even though theactive contour may occasionally miss parts of the true body outline,thanks to the implicit surface treatment of outliers and to the strongstereo information, the system is usually able to find the correctmodel pose.

In a final refinement step, the model, now in its correct pose, isagain projected into the camera frame and reoptimized. The full3Dmodel is then fitted again using this improved silhouette and thestereo information. This approach yields the results of Fig. 6. Theseresults show that this new algorithm is able to overcome the errorsthat occurred when using stereo alone as depicted by Fig. 3.

6 CONCLUSION

We have presented a flexible framework for video-based modelingusing articulated 3D soft objects. The volumetric models we use aresophisticated enough to recover shape and simple enough to trackmotion using noisy image data. This has allowed us to validate ourapproach using trinocular video-sequences featuring complex fullythree-dimensional motions without engineering the environment oraddingmarkers. Even thoughwe use a straightforward approach tooptimization instead of a sophisticated probabilistic one, our systemcan handle complicated motions that involve self-occlusions,temporarymerging of body parts, and noise introduced by clothing.

The implicit surface approach to modeling we advocate extendsearlier robotics approaches designed to handle articulated bodies. Ithas a number of advantages for our purposes. First, it allows us todefine a distance function from data points to models that is bothdifferentiable and computable without search. Second, it lets usdescribe accurately both shape and motion using a fairly smallnumber of parameters. Last, the explicit modeling of 3D geometrylets us predict the expected location of image features such assilhouettes and occluded areas, thereby increasing the reliability ofimage-based algorithms.

Our approach relies on optimization to deform the genericmodelso that it conforms to the image data. This involves computing firstand second derivatives of the distance function from model to datapoints. To this end, we have developed a mathematical formalismthat greatly simplifies these computations and allows a fast and

robust implementation. This is in many ways orthogonal to recentapproaches to human body tracking as we address the question ofhow to best represent the human body for tracking and fittingpurposes. The specific optimization scheme we use could easily bereplaced by amore sophisticated one that incorporates statistics andcan handle multiple hypotheses [12], [9], [5]. Another naturalextension of this work would be to develop better body and motionmodels: The current model constrains the shape and imposes jointangle limits. This is not quite enough under difficult circumstances:A complete model ought to also include more biomechanicalconstraints that dictate how body parts can move with respect toeachother, for example in termsof dependencies among joint angles.

In our current work, we rely on cheap and easily installed videocameras to provide data. This, we hope, will lead to practicalapplications in the fields ofmedicine, athletics, and entertainment. Itwould also be interesting to test our approach using high qualitydata coming from a new breed of image or laser-based dynamic3D scanners [28], [8]. Our technique will provide the relativeposition of the skeleton inside the data and a standard joint-angle-based description of the subject’s motion. Having high-resolutionfront and back data coverage of the subject should allow us torecover very high-quality animatable body models.

ACKNOWLEDGMENTS

This work was supported in part by the Swiss Federal Office forEducation and Science under contract 99.0230-2 and in part by theSwiss National Science Foundation.

REFERENCES

[1] J.K. Aggarwal and Q. Cai, “Human Motion Analysis: A Review,” ComputerVision and Image Understanding, vol. 73, no. 3, pp. 428-440, 1999.

[2] E. Bittar, N. Tsingos, and M.-P. Gascuel, “Automatic Reconstruction ofUnstructured 3D Data: Combining Medial Axis and Implicit Surfaces,”Proc. EUROGRAPHICS Computer Graphics Forum, H.-P. Seidel andP.J. Willis, eds., vol. 14, pp. 457-468, Sept. 1995.

[3] J.F. Blinn, “A Generalization of Algebraic Surface Drawing,” ACM Trans.Graphics, vol. 1, no. 3, pp. 235-256, 1982.

[4] C. Bregler and J. Malik, “Tracking People with Twists and ExponentialMaps,” Proc. Conf. Computer Vision and Pattern Recognition, June 1998.

[5] K. Choo and D.J. Fleet, “People Tracking Using Hybrid Monte CarloFiltering,” Proc. Int’l Conf. Computer Vision, July 2001.

[6] M. Covell, A. Rahimi, M. Harville, and T. Darrell, “Articulated-PoseEstimation Using Brightness and Depth-Constancy Constraints,” Proc. Conf.Computer Vision and Pattern Recognition, 2000.

[7] J.J. Craig, “Electrical and Computer Engineering,” Introduction to Robotics:Mechanics and Control, second ed. Addison-Wesley, chapter 5, 1989.

1186 IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 25, NO. 9, SEPTEMBER 2003

Fig. 6. Applying the model-based silhouette extraction method of Section 5.2 to the sequence of Fig. 3. The tracking errors around frame 100 have been corrected by

combining stereo and silhouette information. The snake-optimized contours are overlaid on the original images and on the reconstructed body model.

Fig. 5. A few frames from a sequence in which the subject wears a baggy sweater, thus complicating the task of the tracker. Top row: One of the three original images at a

given time. Bottom row: Tracking results.

[8] L. Davis, E. Borovikov, R. Cutler, D. Harwood, and T. Horprasert, “Multi-Perspective Analysis of Human Action,” Proc. Third Int’l WorkshopCooperative Distributed Vision, Nov. 1999.

[9] A.J. Davison, J. Deutscher, and I.D. Reid, “Markerless Motion Capture ofComplex Full-Body Movement for Character Animation,” Proc. Euro-graphics Workshop Computer Animation and Simulation, 2001.

[10] Q. Delamarre and O. Faugeras, “3D Articulated Models and MultiviewTracking with Physical Forces,” Computer Vision and Image Understanding,vol. 81, pp. 328-357, Mar. 2001.

[11] M. Desbrun and M.P. Gascuel, “Animating Soft Substances with ImplicitSurfaces,” Proc. SIGGRAPH Computer Graphics, pp. 287-290, 1995.

[12] J. Deutscher, A. Blake, and I. Reid, “Articulated Body Motion Capture byAnnealed Particle Filtering,” Proc. Conf. Computer Vision and PatternRecognition, 2000.

[13] I. Douros, L. Dekker, and B. Buxton, “An Improved Algorithm forReconstruction of the Surface of the Human Body from 3D Scanner DataUsing Local B-Spline Patches,” Proc. ICCV Workshop Modeling People, Sept.1999.

[14] T. Drummond and R. Cipolla, “Real-Time Tracking of Highly ArticulatedStructures in the Presence of Noisy Measurements,” Proc. Int’l Conf.Computer Vision, July 2001.

[15] M. Fitzgibbon, A.W. Pilu, and R.B. Fisher, “Direct Least-Squares Fitting ofEllipses,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 21, no. 5,pp. 476-480, May 1999.

[16] P. Fua, “From Multiple Stereo Views to Multiple 3D Surfaces,” Int’l J.Computer Vision, vol. 24, no. 1, pp. 19-35, Aug. 1997.

[17] D.M. Gavrila, “The Visual Analysis of Human Movement: A Survey,”Computer Vision and Image Understanding, vol. 73, no. 1, Jan. 1999.

[18] A. Hilton, D. Beresford, T. Gentils, R. Smith, and W. Sun, “Virtual People:Capturing Human Models to Populate Virtual Worlds,” Proc. ComputerAnimation, May 1999.

[19] I. Kakadiaris and D. Metaxas, “3D Human Body Model Acquisition fromMultiple Views,” Proc. Int’l Conf. Computer Vision, 1995.

[20] I.A. Kakadiaris and D. Metaxas, “Model Based Estimation of 3D HumanMotion with Occlusion Based on Active Multi-Viewpoint Selection,” Proc.Conf. Computer Vision and Pattern Recognition, June 1996.

[21] W.S. Lee, J. Gu, and N. Magnenat Thalmann, “Generating Animatable 3DVirtual Humans from Photographs,” Proc. Eurographics Computer GraphicsForum, pp. C1-C10, Aug. 2000.

[22] W. Maurel and D. Thalmann, “Human Shoulder Modeling IncludingScapulo-Thoracic Constraint and Joint Sinus Cones,” Computers andGraphics, vol. 24, pp. 203-218, 2001.

[23] T.B. Moeslund and E. Granum, “A Survey of Computer Vision-BasedHuman Motion Capture,” Proc. Computer Vision and Image Understanding,vol. 81, no. 3, Mar. 2001.

[24] D. Morris and J. Rehg, “Singularity Analysis for Articulated ObjectTracking,” Proc. Conf. Computer Vision and Pattern Recognition, pp. 289-296, 1998.

[25] R. Plankers, “Human Body Modeling from Video Sequences,” PhD thesis,EPFL, Lausanne, Switzerland, Nov. 2001.

[26] R. Plankers and P. Fua, “Articulated Soft Objects for Video-Based BodyModeling,” Proc. Int’l Conf. Computer Vision, pp. 394-401, July 2001.

[27] R. Plankers and P. Fua, “Model-Based Silhouette Extraction for AccuratePeople Tracking,” Proc. European Conf. Computer Vision, May 2002.

[28] H. Saito and T. Kanade, “Shape Reconstruction in Projective Grid Spacefrom Large Number of Images,” Proc. Conf. Computer Vision and PatternRecognition, June 1999.

[29] C. Sminchisescu and B. Triggs, “Covariance Scaled Sampling for Mono-cular 3D Body Tracking,” Proc. Conf. Computer Vision and PatternRecognition, 2001.

[30] W. Sun, A. Hilton, R. Smith, and J. Illingworth, “Building LayeredAnimation Models of Captured Data,” Proc. Eurographics WorkshopComputer Animation and Simulation, 1999.

[31] D. Thalmann, J. Shen, and E. Chauvineau, “Fast Realistic Human BodyDeformations for Animation and VR Applications,” Proc. Computer GraphicsInt’l, June 1996.

[32] S. Wachter and H.-H. Nagel, “Tracking Persons in Monocular ImageSequences,” Computer Vision and Image Understanding, vol. 73, no. 3, pp. 174-192, June 1999.

. For more information on this or any other computing topic, please visit ourDigital Library at http://computer.org/publications/dlib.

IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 25, NO. 9, SEPTEMBER 2003 1187