Eurographics/ACM SIGGRAPH Symposium on Computer Animation (2005)K. Anjyo, P. Faloutsos (Editors)

Video-Based Character Animation

J. Starck, G. Miller and A. Hilton

Centre for Vision, Speech and Signal ProcessingUniversity of Surrey, Guildford, GU2 7XH, UK

Abstract

In this paper we introduce a video-based representation for free viewpoint visualization and motion control of3D character models created from multiple view video sequences of real people. Previous approaches to video-based rendering provide no control of scene dynamics to manipulate, retarget, and create new 3D content fromcaptured scenes. Here we contribute a new approach, combining image based reconstruction and video-basedanimation to allow controlled animation of people from captured multiple view video sequences. We represent acharacter as a motion graph of free viewpoint video motions for animation control. We introduce the use of ge-ometry videos to represent reconstructed scenes of people for free viewpoint video rendering. We describe a novelspherical matching algorithm to derive global surface to surface correspondence in spherical geometry imagesfor motion blending and the construction of seamless transitions between motion sequences. Finally, we demon-strate interactive video-based character animation with real-time rendering and free viewpoint visualization. Thisapproach synthesizes highly realistic character animations with dynamic surface shape and appearance capturedfrom multiple view video of people.

Categories and Subject Descriptors (according to ACM CCS): I.3.7 [Computer Graphics]: Animation

1. Introduction

In this paper we introduce a video-based representation forfree viewpoint visualization and motion control of 3D char-acter models created from multiple view video sequences ofreal people. Our work combines image based reconstruction[VBK02, MBR∗00] and video-based animation techniques[SE02] to provide a new video-based approach to characteranimation. In the same way that motion capture technologycaptures complex human motions for convincing skeletal an-imation, video based character animation provides the ap-pearance of complex surface dynamics such as cloth motionand wrinkles captured from multiple view video for synthe-sis of highly realistic human appearance in animation.

Image based reconstruction and rendering in com-puter vision and computer graphics concentrates on static

scenes [LH96, GGSC96] and more recently dynamic scenes[ZKU∗04] for free-viewpoint visualization. Free-viewpointvideo rendering enables a change of view in a scene and spe-cial effects such as freeze frame or slow-motion fly around.However, it is limited to replaying the original captured mo-tion. Providing control of scene dynamics opens up the po-tential to manipulate, retarget, and create new 3D content,ultimately at the quality of conventional video. Previous ap-proaches to the motion control of people have used modelfitting [CTMS03, SH03], where constraints on model shapedo not accurately reproduce the geometry of loose clothingand hair, reducing the visual quality compared to model-free video-based rendering [ZKU∗04, SH05]. Video-basedanimation techniques [SSSE00, SE02] have been proposedwithout the requirement for a prior geometric model, buthave been restricted to a fixed viewpoint and non-rigid ob-

c© The Eurographics Association 2005.

J. Starck, G. Miller & A. Hilton / Video-based character animation

jects without self-occlusion. Animation control in video ren-dering of complex articulated objects such as people remainsan open and challenging problem.

Here we present a new technique for controlled anima-tion of people using free viewpoint video. We represent a 3Dcharacter as a motion graph [KGP02] of seamless charactermotions constructed from free-viewpoint video sequences ofa real person. Our approach makes the following contribu-tions:

• Reconstruction and representation of real people as se-quences of 2D spherical geometry images [PH03], a ge-ometry video [BSM∗03] providing a temporally consis-tent structure for level of detail and frame rate control infree viewpoint video.

• A new multiple resolution coarse-to-fine spherical match-ing technique to derive dense surface-to-surface corre-spondence for geometry image sequences, enabling mo-tion blending between geometry videos.

• Animation control using a motion graph of geometryvideo clips with seamless motion cycles and transitionalmotions constructed using spherical matching.

In Section 2 we review related work on image based ren-dering and video-based animation techniques. In Section 3,we outline the process of constructing a geometry video fora multiple view sequence of a person. In Section 4, we intro-duce a spherical matching algorithm for motion blending. InSection 5 we outline the construction of a motion graph forcharacter animation and our video renderer for real-time ren-dering and free viewpoint visualization. Finally in Section 6we present results for video-based character animation.

2. Background

2.1. Video-based rendering

Rendering novel views of dynamic scenes such as peo-ple from multiple view video capture has received con-siderable interest over the past decade. Kanade et al.’s[KRN97]Virtualized RealityT M system used a 5m hemi-spherical dome of 51 cameras to capture sequences of a dy-namic scene. Multiple view silhouettes [MTG97, MBR∗00]and photo-consistency between views [VBK02] have beenused to render novel views from smaller numbers of cam-eras. Temporal correspondence has also been introducedin recent work to improve surface reconstruction [VBK02,GM04]. Two problems arise with these approaches: qualityof novel view rendering due to errors in geometric surfacereconstruction; and reconstruction of an unstructured modelat each time instant which does not allow the scene dynamicsto be modified for reanimation.

High quality video-based rendering has been addressedusing stereo reconstruction techniques for a 1D array of cam-eras [ZKU∗04, SH05]. Zitnick et al. [ZKU∗04] introduce anovel stereo approach that simultaneously mattes a general

scene into two layers and reconstructs depth. Results demon-strate rendered views comparable to the captured video, buthave limited viewpoint control and no control over scene dy-namics. To allow reanimation of dynamic scenes of people,model-based reconstruction techniques which fit a generichumanoid model to observations from multiple views havebeen developed [CTMS03, SH03, PF01] . Constrained mul-tiple view optimization of model pose and shape is per-formed to minimize the model re-projection error for sil-houette [CTMS03] and stereo correspondence [SH03,PF01].Model-based approaches reconstruct a structured representa-tion with an articulated joint structure which can be manip-ulated for reanimation. Visual-quality of reconstructed mod-els is limited due to both errors in reconstruction and con-straints on model shape which do not represent geometrysuch as loose clothing or hair.

In this work we propose a temporally consistent structuredrepresentation based on spherical mapping and remeshing asa 2D geometry image. We introduce spherical matching thatallows motion blending between geometry image sequencesfor motion control. This representation allows high-qualityvideo-based rendering with motion control for reanimationwithout model-based constraints on surface shape.

2.2. Video-based animation

Video-based techniques such as video interpolation [BCS97,CG00] and video-sprites [SE02] provide an image-basedrepresentation of dynamic scenes allowing synthesis ofnovel image sequences with visual quality comparable tothe captured video. Novel video sequences are synthesizedby concatenating segments of the captured video based ona transition graph. This approach has been used to producevideo-realistic synthesis of faces [BCS97, CG00, EGP02] orsimple non-rigid subjects [SSSE00, SE02] with limited con-trol over scene dynamics. Schodl et al. [SE02] use a sim-ilarity metric between video frames to identify transitionpoints which minimize visual artifacts. Graphcut textures[KSE∗03] optimized the transition over space and time be-tween video sequences to achieve video quality compositionof dynamic scenes.

These approaches are currently limited to a fixed view-point and non-rigid objects without self-occlusion. In thiswork we combine video-based rendering with video-basedanimation to provide a representation that allows free view-point visualization of complex non rigid motions of peoplewith control of motion dynamics.

2.3. Motion graphs

Character animation from motion capture data often requiresthe modification of captured movement to satisfy constraintsor transition seamlessly between sequences. Motion graphsfor databases of human motion capture have been introduced

c© The Eurographics Association 2005.

J. Starck, G. Miller & A. Hilton / Video-based character animation

to allow reuse by recombining segments of the motion cap-ture data [AFO03, KGP02, LCR∗02, LAS02]. The problemof human motion synthesis is formulated as a graph searchwhere the nodes correspond to frames in the motion databaseand edges are weighted by the similarity of frames. Novelanimation sequences are synthesized by optimizing the paththrough the graph to satisfy external user constraints andminimize transition artifacts.

In this work we define a graph of human motions and tran-sitional motions reconstructed from multiple view video. Asimilar approach is used in computer games for real-timecharacter animation with a predefined set of transitions andcorresponding sequences of motion capture data. Here themotions and transitions are defined by free viewpoint videosequences. A seamless blend between motions and tran-sitions is achieved using spherical matching between twovideo sequences. The final motion graph enables controlledanimation of the video-based character for seamless motionsynthesis.

3. Video capture and representation

In this section we outline the process of reconstruction andrepresentation of multiple view video as a geometry video.Figure 1 illustrates the process. Multiple view video se-quences of people are first captured in a dedicated multiplecamera studio. For each time frame, a triangulated surfacemesh is then reconstructed. This surface is parameterized onthe spherical domain [PH03] and resampled onto a sphericalsubdivision surface [PH03, ZBS04]. The subdivision meshhas a predefined mapping to the 2D domain and geometryplus color is mapped to a 2D geometry image. Finally, thegeometry images created for a sequence are encoded as asingle geometry video for rendering as free-viewpoint video.

3.1. Surface reconstruction

Multiple-view video is captured in a 10 camera studio witha blue-screen backdrop for foreground segmentation. SonyDXC-9100P 3-CCD color cameras are used, providing PAL-resolution progressive scan images at 25Hz. The cameras arepositioned around a capture volume of 3m × 2m × 2m, suf-ficient for a single cycle of walk and jog motions. Intrinsicand extrinsic camera parameters are calibrated using a publicdomain calibration toolbox [Bou03].

Surface extraction is performed at each time frame us-ing volumetric reconstruction. The visual-hull is first derivedfrom segmented foreground image silhouettes, this is thenrefined to a color consistent photo-hull using the General-ized Voxel Coloring algorithm [CMS99]. The surface of thevolumetric reconstruction is then extracted using the March-ing Cubes algorithm [LC87] and smoothed to provide a con-tinuous representation for the surface normals. Surface coloris derived at each vertex by blending the sampled colors ineach visible camera view [DYB98]. Blend weights are based

Figure 1: Construction of a single frame of a sequence from mul-tiple view video, showing volumetric reconstruction, spherical pa-rameterization and mapping as an image with geometry plus color.

on the sampling density in each view using the cosine of theangle between the camera to surface viewing direction andthe surface normal.

3.2. Spherical mapping

Reconstructed surfaces are parameterized in the sphericaldomain using the coarse-to-fine technique introduced byPraun and Hoppe [PH03]. The starting point is a closed tri-angulated mesh M in <3 of zero genus. It should be notedthat this limits video capture to motions where the limbs donot connect with the body giving a non genus zero surface.

A unique 1-to-1 map is constructed for M onto the unitsphere S, (M → S). The mesh M is simplified to a tetrahe-dron while creating a progressive mesh structure [Hop96].Spherical mapping is achieved by first mapping the tetrahe-

c© The Eurographics Association 2005.

J. Starck, G. Miller & A. Hilton / Video-based character animation

dron to the unit sphere, then traversing the progressive meshsequence and inserting vertices onto the sphere. A 1-to-1map is achieved as progressive mesh construction maintainsan embedding during mesh decimation and vertices are in-serted into the kernel of their 1-neighborhood on the sphere,maintaining the embedding in the spherical domain [PH03].

For highly deformed genus-zero surfaces, producing auniform sampling on the sphere during mapping is a chal-lenging task. This represents a particular problem for thehuman body, where narrow limbs become highly distortedin the spherical domain. Praun and Hoppe [PH03] investi-gate several approaches to minimize distortion during map-ping and perform mesh optimization to minimize the stretchin the representation with respect to the original mesh. Inthis work we do not seek to minimize the stretch, instead weseek a uniform sampling in the domain such that we can ad-equately sample highly deformed genus-zero surfaces. Wepropose a cost function derived from Zhou et al. [ZBS04]and optimize the mesh to minimize the vertex sampling den-sity on the sphere during mapping.

The sampling density D(si) at a spherical vertex si is de-fined as follows, where i′ spans the 1-neighborhood of a ver-tex i, Ni is the valence of the vertex and ∆ii′ is the sphericaltriangle with the directed edge ii′ in the mesh.

D(si) =Ni

∑i′ Area(∆ii′)(1)

We minimize the distortion in the vertex density on thesphere with respect to a target density D0 using the follow-ing cost function, where ∇ii′ is the gradient on the edge ii′,∇ii′ = si′ − si.

Edensity (S) = ∑i

∑i′

D0(si′)

D(si′)‖∇ii′si‖

2 (2)

For density minimization on the unit sphere we assume auniform low resolution target density of, D0 = 1.

3.3. Image mapping

The spherical mesh is resampled onto a regular subdivisionsurface S by constructing a map (S → S) while maintainingthe sampling density of the original mesh as proposed byZhou et al. [ZBS04].

A subdivision surface S is constructed using regular 1-to-4 subdivision of a unit octahedron. The surface is updatedto minimize the density cost function in Equation 2 withthe target density derived from the surface S. Gradient de-scent optimization corresponds to iterative application of theweighted laplacian operator ∆density(si) as proposed by Zhouet al. [ZBS04].

∆density (si) =1

∑i′D0(si′ )D(si′ )

∑i′

D0(si′)

D(si′)(si − si′) (3)

Given the mapping M → S and S → S, we can resamplethe attributes of the original mesh M onto the uniform do-main of the subdivision surface S. The surface is mapped toa geometry image by unfolding the subdivided octahedrononto a 2D grid with a 1-to-1 correspondence between thevertices of the subdivision surface and image pixels [PH03].For a subdivided octahedron there will be (2n+1)×(2n+1)samples in the image domain. In this work we resample ge-ometry onto a (2n− 1)× (2n− 1) image for efficient rep-resentation as a (2n)2 image appropriate for rendering. Weensure that there is no loss of visual detail in resampling thegeometry image by deriving color from the original cameraimages for the resampled geometry. We use multiple reso-lution blending [BA83] with spherical boundary continuityto fill missing colors in the final image. This process is per-formed for sequences of captured multiple view video to ob-tain a geometry video representation for each motion. Theproblem then is to blend different motions to achieve seam-less character animation.

4. Motion blending

To construct seamless motion sequences for character ani-mation it is necessary to blend between motions at the tran-sition points, either at the start end of cyclic motions or whenswitching from one motion to another. In this section weintroduce a multiple resolution coarse-to-fine algorithm forshape matching in the spherical domain. Spherical matchingis employed to align the geometry images between overlap-ping motion sequences. Transitions between two sequencescan then be created by blending between matched geometryimages.

Surface matching for blending is formulated as a continu-ous bijective mapping between two surfaces (S → S0) in thespherical domain. Our approach poses the mapping problemas an energy minimization task. We construct a cost func-tion for joint estimation of correspondence and deformationin the spherical domain to minimize disparity in shape andappearance between two surfaces.

The matching function for minimization, Ecost(S,W), isdefined as a function of the set of vertices S for a surfacein the spherical domain and a fuzzy multi-point correspon-dence W = {wi j ∈ (0,1)} defining the match between twosurfaces S = {si} and S0 = {s j}. The function consists ofthree terms: Edisparity measuring the distance in shape andcolor of the surfaces; Ede f orm measuring the distance be-tween the surfaces in the spherical domain; and Eregularizedefining regularizing constraints on the deformation, subjectto a control parameter λ.

c© The Eurographics Association 2005.

J. Starck, G. Miller & A. Hilton / Video-based character animation

Ecost (S,W) = (1−λ)Edisparity (W) (4)

+ (1−λ)Ede f orm (S,W)

+ λEregularize (S)

In the following sections we describe each of the compo-nents of the matching functional followed by the algorithmfor optimization and spherical matching.

4.1. Surface disparity

Data fit is based on the similarity of a set of attributes definedon the surfaces S and S0. In energy minimization we seek thecorrespondence W that minimizes the disparity in shape andappearance using the position x, surface normal n and RGBcolor c of the resampled surfaces. The disparity is measuredusing a squared error metric and a separate energy functionis constructed for each attribute as follows

Edisparity (W) =1

σ2x∑

i∑

jwi j‖xi − x j‖

2 (5)

+1

σ2n∑

i∑

jwi j‖ni −n j‖

2

+1

σ2c∑

i∑

jwi j‖ci − c j‖

2

The disparity terms are normalized by the expected vari-ance σ2 for a correct match. The variance parameters pro-vide control over the expected range of disparities betweentwo surfaces for each of the attributes used in matching. Inthis work we set these as follows, σx = 5cm, σn = 0.5(30o),and σc = 50 ∈ (0,255). These values were derived usinghand selected vertex correspondences between motion se-quences.

4.2. Spherical deformation

The matching problem is posed as the deformation of thesurface S = {si} to minimize the disparity with respect tothe surface S0 = {s j} in the spherical domain, s ∈ <2. Sur-face deformation is driven by minimizing the distance on thesphere based on the multi-point correspondence W betweenthe surfaces. In practise we linearize the problem and com-pute distances in <3 while constraining vertices to remain onthe unit sphere. The deformation energy function is definedas

Ede f orm (S,W) =1

T 2 ∑i

∑j

wi j‖si − s j‖2 (6)

+ ∑i

∑j

wi j(

log(wi j)−1)

A deterministic annealing approach [CR03] is adopted foroptimization using an entropy term and temperature controlparameter T in the deformation functional. The term wi j de-fines a multi-point fuzzy correspondence between the twosurfaces and the temperature provides control of the corre-spondence during optimization. At a high temperature valuethe correspondence becomes more fuzzy and as the tempera-ture is successively reduced the correspondence can be hard-ened, providing a coarse-to-fine refinement of the surfacecorrespondence.

4.3. Regularization

Regularization is used to minimize the distortion in thespherical domain while developing the map S → S0. Verticessi are updated sequentially and constrained to remain withinthe kernel of the 1-neighborhood. This ensures that the sur-face S maintains an embedding on the sphere. Left uncon-strained the surface S will quickly reach a local minima withvertices collapsed onto the border of the 1-neighborhood.Regularization minimizes the distortion in the distributionof vertices such that the surface S can continue to deformto minimize the disparity function. This is achieved by min-imizing the vertex density in the spherical domain duringoptimization

Eregularize (S) =1

T 2 Edensity (S) (7)

A weighting parameter λ (Equation 4), is used to controlthe trade-off between regularization and data-fitting in op-timization. In this work a fixed value, λ = 0.75, is used.Spherical matching was found to be relatively insensitiveto the value of λ, provided that it was sufficient to preventcollapsed vertex neighborhoods in optimization. The regu-larization term simply maintains an even vertex distributionsuch that the surface is free to deform and minimize the datafitting cost function.

4.4. Multiple resolution coarse-to-fine optimization

Joint estimation of correspondence and deformation issolved using a strategy of alternately minimizing the match-ing cost with respect to the correspondence matrix W andthen updating the surface S to satisfy the correspondence.For a fixed surface configuration S a unique solution existsfor the correspondence as follows. The constraint ∑ j wi j = 1is imposed to enforce surface assignment S → S0.

w′

i j = exp

(

‖xi − x j‖2

σ2x

)

exp

(

‖ni −n j‖2

σ2n

)

(8)

× exp

(

‖ci − c j‖2

σ2c

)

exp

(

‖si − s j‖2

T 2

)

c© The Eurographics Association 2005.

J. Starck, G. Miller & A. Hilton / Video-based character animation

Figure 2: Geometry images for frames 0, 5, 10 of a walk cy-cle showing (top) original images, (bottom) the result of sphericalmatching applied frame to frame in the sequence.

wi j =w′

i j

∑ j w′

i j(9)

For a fixed correspondence W we use gradient descent tosolve the continuous problem of surface deformation

dEcost

dsi= (1−λ)wi j

(

si − s j

)

+λ∆density (si) (10)

The optimization algorithm proceeds by first finding theglobal rotation that minimizes the cost function. A multi-resolution strategy is then adopted for local optimization.Surface deformation is scheduled at successive resolutionsof the subdivision surface S starting with the base octahe-dron. Coarse-to-fine matching is achieved by setting the tem-perature parameter T to restrict the range of matches wi j tolie principally within the kernel of the 1-neighborhood fora vertex. This is achieved using a value of T equal to theedge length on the undeformed spherical mesh at each levelof subdivision. Optimization at each resolution proceeds un-til the maximum update for a vertex reaches a fixed valueε = 0.0001 on the unit sphere.

The optimization algorithm terminates at the finest reso-lution of the subdivision surface S and provides a continuous1-to-1 map between the surfaces S, S0. The surface S can fi-nally be resampled at the vertices of the target mesh S0 anda new geometry image constructed such that S and S0 are incorrespondence in the 2D image domain for blending. Fig-ure 2 shows the result of spherical matching applied to aligngeometry images.

5. Interactive character animation

In this section we outline the creation of motion graphs forvideo based character animation and the video renderer forinteractive character control and free-viewpoint visualiza-tion.

Figure 3: Motion graph for character animation using cyclic (idle,walk) and transitional motions (walk to idle, idle to walk) repre-sented by geometry videos.

5.1. Motion graphs

We capture multiple view video sequences for a set of cyclicmotions for a person such as idle, walk and jog. We also cap-ture transitional motions such as idle-walk, walk-idle, walk-jog and jog-walk. The motion sequences and transitions arerepresented as a geometry video as outlined in Section 3.Seamless motions are created using the spherical matchingalgorithm introduced in Section 4. To create seamless cyclicmotions the final frames of a sequence are blended with theframes preceding the start of the sequence. For transitionalmotions the start and end frames are blended with the pre-ceding and succeeding cyclic motions. Motion sequences aretemporally aligned using the heel strike position and a 5-frame linear blend is used between matched geometry im-ages. The geometry videos are finally assembled into a mo-tion graph. The graph contains a set of cyclic motion statesfor a character (idle, walk and jog) and a set feasible tran-sitions (idle-walk, walk-idle, walk-jog and jog-walk) as il-lustrated in Figure 3. The motion graph allows interactivecontrol of character state in the video renderer.

5.2. Video renderer

The motion graph for a character is defined by a set ofmotion states and transitional motions encoded as geome-try videos in AVI format, together with the correspondingtranslational motion for the video sequences. We render ge-ometry videos for free viewpoint visualization using ShaderModel 3.0 compliant graphics hardware (nVidia 6600GT).This shader model supports texture lookup in the vertexshader, allowing the extraction of vertex position in a ge-

c© The Eurographics Association 2005.

J. Starck, G. Miller & A. Hilton / Video-based character animation

Figure 4: 255 × 255 geometry image with (left) geometry quan-tized at 1cm resolution with 24 bits/pixel, (middle) non-quantizedgeometry with 96 bits/pixel and (right) non-quantized geometry withview-dependent rendering.

ometry image for real-time rendering. Our video rendereris implemented using DirectX 9 and the High Level ShaderLanguage (HLSL). The renderer loads a motion graph for acharacter and provides interactive control of character stateand the direction of character motion with free viewpointcontrol in visualization.

6. Results and Discussion

Character construction was tested using three subjects cap-tured in multiple view video sequences performing idle,walking, jogging, and kicking motions together with thetransitional movements such as walk-jog. Figure 1 illustratesa single frame of multiple view video recorded for an idlemotion and the corresponding geometry image constructedfor rendering.

We create geometry images at a resolution of 255 × 255with geometry quantized to 255 levels at 1cm resolution ina 24 bits/pixel RGB image. There is a trade-off in our rep-resentation between the visual quality that can be achieved

in rendering and the memory overhead required for the rep-resentation. At low image resolutions there will be under-sampling of both geometry and appearance in the geome-try videos. At high resolutions we can achieve greater ge-ometric detail and visual fidelity to the original multipleview video, at the cost of greater storage requirements. Inthis work we target a compact representation for our motiongraph to minimize the memory overhead for real-time ren-dering. We found that a minimum resolution of 255 × 255was required to retain the geometry at the tips of the limbswhen remeshing onto the subdivision domain S. Greater vi-sual fidelity could be achieved either by using a larger num-ber of bits per pixel to increase geometric accuracy, using alarger image size for color texture to increase the samplingfor appearance or by using view dependent rendering fromthe original video for highly realistic rendering at the costof significantly higher storage requirements for a completemotion graph. Figure 4 compares the effect of quantized andunquantized geometry as well as view-dependent renderingfor an image resolution of 255 × 255.

Animation control is achieved using a motion graph ofgeometry video sequences. Here we capture cyclic motionssuch as walking and jogging, and the transitional motionsbetween these states. A user can control character dynam-ics either through the direction of motion when the characteris in a cyclic state or by controlling the state of the charac-ter in the motion graph and transitioning between differentcyclic motions. Figure 5 shows rendered frames for the tran-sition between two cyclic states in the motion graphs createdfor the three captured subjects. Seamless cyclic motions andblends between transitional and cyclic motions are achievedusing spherical matching and motion blending between thegeometry videos in the motion graph. We are therefore ableto control and create new 3D dynamic scenes of people fromshort sections of multiple view video sequences representedas free viewpoint geometry videos.

One of the principal advantages of our video-based rep-resentation for animation is the realism that can be achievedby replaying the surface dynamics captured from the originalmultiple view video. Figures 6 and 7 show screen capturesfrom the video renderer with interactive animation controland free viewpoint rendering. Video clips for the differentanimations are provided for download from [VMR]. Thesemovies demonstrate the realism that can be achieved by re-playing the captured cloth motion, giving a video-realisticlooking result in character animation. Our video rendererwas tested on an Intel Pentium IV 3.2GHz, 1GB RAM PCwith a NVIDIA 6600GT graphics card running WindowsXP. A maximum frame rate of 88fps was achieved witha complete motion graph for a character stored in texturememory.

The geometry video representation has several advantagesas a representation for free viewpoint rendering from multi-ple view video sequences of people. Geometry images pro-

c© The Eurographics Association 2005.

J. Starck, G. Miller & A. Hilton / Video-based character animation

Figure 5: Animation control using a motion graph of geometry videos showing the transitions between motion states for three subjects, (top)walk-idle-walk, (middle) walk-jog-walk, and (bottom) stance-kick-stance.

Figure 6: Interactive character control and free viewpoint rendering for a subject performing walk and idle motions.

c© The Eurographics Association 2005.

J. Starck, G. Miller & A. Hilton / Video-based character animation

Figure 7: Interactive character control and free viewpoint rendering for a subject performing walk and jog motions.

vide a consistent temporal structure for level of detail andframe rate control [BSM∗03] in free viewpoint video. Spher-ical matching enables the correspondence to be derived be-tween spherical geometry images. This has been applied toblend motions in geometry videos. It also enables the recov-ery of temporal correspondence within a sequence. Figure2 illustrates a temporally consistent geometry video createdby matching successive frames of a walk cycle. This poten-tially provides a technique both for geometry and appearanceediting in geometry videos, as well as for compression andefficient representation of free viewpoint video, subjects forfuture research.

7. Conclusions

A video-based representation has been introduced for free-viewpoint visualization and animation control of 3D charac-ter models of real people reconstructed from multiple viewvideo. Character motions are represented as free viewpointvideo reconstructed from multiple view video sequencescaptured in a 10 camera studio. Image based reconstructionis employed to extract a 3D surface for each frame of a mo-tion sequence. Surfaces are parameterized on the sphericaldomain, then mapped to a 2D geometry image providinga geometry video representation for the video sequences.Spherical matching is used to blend motion sequences toprovide seamless motion cycles and smooth transitions be-tween different motions. A multiple resolution coarse-to-finespherical matching algorithm has been developed that si-multaneously optimizes across the entire spherical domainto provide the correspondence between spherical geometryimages. The blended video sequences are constructed as anoff-line process and represented as a motion graph for in-

teractive 3D character animation. A video renderer is usedto provide real-time free viewpoint visualization and inter-active character control, making use of Shader Model 3.0compliant graphics hardware.

The representation for video-based character animationintroduced in this paper has several significant advantages.

• Free viewpoint video captured from multiple view videosequences of people, creating highly realistic rendered an-imations with the dynamic shape and appearance capturedin the original video.

• Compact representation of multiple view video for freeviewpoint rendering using geometry videos, that providesa temporally consistent structure for level of detail andframe rate control in rendering [BSM∗03].

• Spherical matching for motion blending between geom-etry videos to create seamless transitions between freeviewpoint video sequences and provide temporal corre-spondence within a geometry video for compression.

• Character representation as a motion graph of free view-point video sequences for animation control and synthesisof new 3D content from captured motion clips with the re-alism of the captured video.

• Real-time free viewpoint visualization and interactive an-imation control of multiple view video using ShaderModel 3.0 compliant graphics hardware.

References

[AFO03] ARIKAN O., FORSYTH D. A., O’BRIEN J. F.:Motion synthesis from annotations. Proceedings ACMSIGGRAPH (2003), 402–408. 3

[BA83] BURT P., ADELSON E.: A multiresolution spline

c© The Eurographics Association 2005.

J. Starck, G. Miller & A. Hilton / Video-based character animation

with application to image mosaics. ACM Transactions onGraphics 2, 4 (1983), 217–236. 4

[BCS97] BREGLER C., COVELL M., SLANEY M.: Videorewrite: Driving visual speech with audio. ProceedingsACM SIGGRAPH (1997), 1–8. 2

[Bou03] BOUGUET J.-Y.: Camera Calibration Toolboxfor Matlab: www.vision.caltech.edu/bouguetj/calib-doc.Tech. rep., MRL-INTEL, 2003. 3

[BSM∗03] BRICEÑO H., SANDER P., MCMILLAN L.,GORTLER S., HOPPE H.: Geometry videos: A new rep-resentation for 3d animations. ACM Symposium on Com-puter Animation (2003), 136–146. 2, 9

[CG00] COSATTO E., GRAF P.: Photo-realistic talkingheads from image samples. IEEE Transaction on Mul-timedia 2, 3 (2000), 152–163. 2

[CMS99] CULBERTSON W. B., MALZBENDER T.,SLABAUGH G.: Generalized voxel coloring. Proceedingsof the International Workshop on Vision Algorithms, Lec-ture Notes in Computer Science 1883 (1999), 100–115.3

[CR03] CHUI H., RANGARAJAN A.: A new point match-ing algorithm for non-rigid registration. Computer Visionand Image Understanding 89 (2003), 114–141. 5

[CTMS03] CARRANZA J., THEOBALT C., MAGNOR M.,SEIDEL H.-P.: Free-viewpoint video of human actors.Proceedings ACM SIGGRAPH 22, 3 (2003), 569–577. 1,2

[DYB98] DEBEVEC P., YU Y., BORSHUKOV G.: Effi-cient view-dependent image-based rendering with projec-tive texture-mapping. 9th Eurographics Rendering Work-shop (1998), 105–116. 3

[EGP02] EZZAT T., GEIGER G., POGGIO T.: Trainablevideorealistic speech animation. Proceedings ACM SIG-GRAPH (2002), 388–398. 2

[GGSC96] GORTLER S., GRZESZCZUK R., SZELISKI

R., COHEN M.: The lumigraph. Proceedings ACM SIG-GRAPH 30 (1996), 43–54. 1

[GM04] GOLDLUECKE B., MARCUS M.: Space-time iso-surface evolution for temporally coherent 3d reconstruc-tion. IEEE International Conference on Computer Visionand Pattern Recognition I (2004), 350–355. 2

[Hop96] HOPPE H.: Progressive meshes. ProceedingsACM SIGGRAPH (1996), 99–108. 3

[KGP02] KOVAR L., GLEICHER M., PIGHIN F.: Motiongraphs. Proceedings ACM SIGGRAPH (2002), 473–482.2, 3

[KRN97] KANADE T., RANDER P., NARAYANAN P.: Vir-tualized reality: Constructing virtual worlds from realscenes. IEEE Multimedia 4, 1 (1997), 34–47. 2

[KSE∗03] KWATRA V., SCHÖDL A., ESSA I., TURK G.,

BOBICK A.: Graphcut textures: Image and video syn-thesis using graph cuts. Proceedings ACM SIGGRAPH(2003), 277–286. 2

[LAS02] LI Y., ANG T., SHUM H.-Y.: Motion texture: Atwo-level statistical model for character motion synthesis.Proceedings ACM SIGGRAPH (2002), 465–471. 3

[LC87] LORENSON W., CLINE H.: Marching cubes: Ahigh resolution 3d surface construction algorithm. Com-puter Graphics 21, 4 (1987), 163–169. 3

[LCR∗02] LEE J., CHAI J., REITSMA P. S., HODGINS

J. K., POLLARD N. S.: Interactive control of avatars an-imated with human motion data. Proceedings ACM SIG-GRAPH (2002), 491–500. 3

[LH96] LEVOY M., HANRAHAN P.: Light field rendering.Proceedings ACM SIGGRAPH 30 (1996), 31–42. 1

[MBR∗00] MATUSIK W., BUEHLER C., RASKAR R.,GORTLER S., MCMILLAN L.: Image-based visual hulls.Proceedings of ACM SIGGRAPH (2000), 369–374. 1, 2

[MTG97] MOEZZI S., TAI L., GERARD P.: Virtual viewgeneration for 3d digital video. IEEE Multimedia 4, 1(1997), 18–25. 2

[PF01] PLANKERS R., FUA P.: Articulated soft objectsfor video-based body modeling. IEEE International Con-ference on Computer Vision (2001), 394–401. 2

[PH03] PRAUN E., HOPPE H.: Spherical parameteri-zation and remeshing. Proceedings ACM SIGGRAPH(2003), 340–349. 2, 3, 4

[SE02] SCHÖDL A., ESSA I.: Controlled animation ofvideo sprites. ACM Symposium on Computer Animation(2002), 121–127. 1, 2

[SH03] STARCK J., HILTON A.: Model-based multipleview reconstruction of people. IEEE International Con-ference on Computer Vision (2003), 915–922. 1, 2

[SH05] STARCK J., HILTON A.: Virtual view synthesisof people from multiple view video sequences. GraphicalModels (to appear 2005). 1, 2

[SSSE00] SCHÖDL A., SZELISKI R., SALESIN D., ESSA

I.: Video textures. ACM Symposium on Computer Ani-mation (2000), 489–498. 1, 2

[VBK02] VEDULA S., BAKER S., KANADE T.: Spatio-temporal view interpolation. Eurographics Workshop onRendering (2002), 1–11. 1, 2

[VMR] www.ee.surrey.ac.uk/research/vssp/vmrg/vcphome.htm.7

[ZBS04] ZHOU K., BAO H., SHI J.: 3d surface fil-tering using spherical harmonics. Computer-Aided De-sign,Elsevier 36 (2004), 363–375. 3, 4

[ZKU∗04] ZITNICK C., KANG S. B., UYTTENDAELE

M., WINDER S. A. J., SZELISKI R.: High-quality videoview interpolation using a layered representation. Pro-ceedings ACM SIGGRAPH 23 (2004), 600–608. 1, 2

c© The Eurographics Association 2005.