Illumination Insensitive Model-Based 3D Object Trackingand Texture Refinement

Hua Yang Greg Welch Marc PollefeysComputer Science Department

University of North Carolina at Chapel HillChapel Hill, NC 27599

Abstract

A common approach to model-based tracking is to use amodel of the object to predict what will be observed, andthen to compare that with real observations. For meth-ods that make use of the object’s photometric properties(appearance) in their measurements, illumination inconsis-tencies between the modeled and actual scene can causetracking problems. In this paper we address one case:model-based tracking of Lambertian objects under direc-tional light sources. We present an iterative optimizationmethod that uses a Kalman filter to simultaneously refineestimates of the object motion, the illumination, and themodel texture. We model the illumination variance be-tween the real and predicted observation using the inten-sity ratios of corresponding surface points, which we thenuse to make model-based image predictions consistent withthe real lighting. To demonstrate the effectiveness of ourmethod we present experimental results using both synthetic(controlled) and real image sequences.

1. Introduction

Conventional model-based tracking approaches often ex-tract and matchgeometricfeatures, e.g., contours or edges.Because such features are generally invariant to illumina-tion, the methods can be consideredillumination insensi-tive. On the other hand, methods [5, 12, 11] have been pro-posed to exploit theappearanceinformation. For example,tracking or registration between the real and predicted im-ages can be performed by comparing photometric proper-ties such as intensity, color, or texture. These methods haveproven to be accurate and efficient in many object track-ing and recognition applications. Unfortunately, becausethe appearance of an object can change dramatically un-der different lighting conditions, typical appearance-basedmethods are prone to difficulties when the model and the

real object are illuminated differently, as is usually the case.In this paper, we present an illumination insensitive

model-based 3D object tracking system. The inputs of thesystem consist of a texture-mapped graphics model and im-age sequences of the target object. The system outputsare the object pose estimate over frames, a refined texturemodel and an map that models the illumination variance be-tween the model space and the real scene. We use a Kalmanfilter to recursively refine the pose (position and orienta-tion), a model for the illumination, and the texture compo-nent of the object model. During tracking, we use graphicshardware (with texture mapping) to render synthetic imagesof the graphics model at its estimated pose, from the viewpoint of each of the cameras. We use the pixel intensitiesof the real and synthetic images as the measurements. Wehandle the illumination inconsistency problem by estimat-ing an Illumination Normal Map (INM)—a mapping fromthe surface normal of a 3D point to the intensity ratio ofits projections in the real and synthetic images. The INMdefines an illumination transformation between the real andsynthetic images. Using this transformation, we can gen-erate illumination-corrected synthetic images and comparethem with the real images, as if they were taken under thesame illumination.

In the next section we discuss some related work. InSection 2 we describe the Illumination Normal Map andanalyze its properties. In Section 3 we provide a brief in-troduction to the relevant parts of the Kalman filter, and inSection 4 we describe our estimation process in the con-text of the Kalman filter framework. Finally in Section 5we shows some results from both synthetic and real imagesequences, and in Section 6 we discuss some future plans.

1.1. Previous Work

Classical model-based tracking methods typically usegeometric object models that describe theshapeof the ob-jects. For instance, both Gennery and Bray exploited a wireframe model to track polyhedral objects [7, 2]. Rehg and

Kanade tracked articulated hand motion using a generalizedcylinder model of the hand [14]. Stengeret al. proposed amethod to capture hand motion using an hand model that isbuilt from truncated quadrics [16]. Rosenhahnet al. pro-posed a silhouette-based approach for pose estimation offree-form surface models [15]. These methods avoid theillumination inconsistency problem by defining geometricfeatures that are illumination invariant. However, geometricfeatures like contours do not contain all the information use-ful for object tracking. For example, the rotation of a spherecannot be detected using its contour information alone. Inaddition, extracting and matching features can be expensivewhen the object is geometrically complex.

Alternatively, some approaches useappearancemod-els of objects—predicting the appearance of the object ineach specific pose. Photometric properties such as intensity,color or texture can then be compared between the predictedand real observations. Dellaertet al. proposed a Kalmanfilter based system that simultaneously estimates the poseand refines the texture of a planar patch [5]. Nickels andHutchinson tracked articulated objects through matchingfeature templates [12]. In [20] a novel 3D object track-ing system was developed where the model of the object isa pre-acquired light-field. Such appearance-based methodsmake more direct and complete use of available informationfrom the image observations, and have achieved outstandingperformance in applications like face tracking. However,because the real and predicted observations are registeredusing illumination dependent properties, these methods areprone to difficulties or failure in the presence of illuminationinconsistencies between the modeled and real scenes.

Illumination insensitive methods have been developedto recognize and track objects under varying lighting con-ditions. To maintain color consistency through image se-quences, Cubberet al. used a transformed color space thatis invariant to highlights, viewing direction, surface orien-tation and illumination directions [4]. Hageret al. devel-oped an efficient region tracking method that uses a three-dimensional linear subspace to model all possible surfaceappearance of the same object under varying lighting con-ditions [9]. Training images are taken under sample light-ing conditions to serve as the illumination templates. In[3], the linear subspace illumination model is improved togenerate subject-independent templates. Ramamoorthi andHanrahan [13] and Brasi and Jacobs [1] formulated therelationship between radiance and irradiance as a spheri-cal convolution whose kernel is determined by the surfaceBRDF, and proved that under the assumption of Lamber-tian surfaces and directional light sources the illuminationfunction can be recovered using first nine coefficients of aspherical harmonic basis. Freedman and Turek developeda tracking method that computes illumination-invariant op-tical flow using graph cuts [6]. They make the assumption

that intensity consistency across pixels can be propagatedover frames. Recently, Zhanget al. proposed the use of theintensity ratio between corresponding pixels to model theillumination variance between an image pair [18]. BeliefPropagation is used to simultaneously estimate per-pixel il-lumination ratio and disparity.

2. Illumination Ratio Estimation

Illumination variation between the real and syntheticscenes can be modeled as the ratio of the intensities of cor-responding pixels in two images. Consider a 3D pointp onthe surface of the target object. The projection ofp on theimage plane is (x, y). Under Lambertian surface and di-rectional lighting assumption, in the absence of shadow, thepixel intensity ofI(x, y) is given by

I(x, y) =∑

k

cka(p)Lk · N(p) (1)

where a(p) is the nonnegative absorption coefficient(albedo) of the surface at the pointp, Lk is the unit direc-tion vector of light sourcek, ck is the intensity of the lightsourcek, N is the unit inward normal vector of the surfaceatp.

The intensity ratior of corresponding pixels at (x, y) inthe real and synthetic images can be computed as

r(x, y) =I(x, y)

I(x, y)=

∑k cka(p)Lk · N(p)

∑k cka(p)Lk · N(p)

(2)

whereI(x, y) is the pixel intensity in the synthetic image,ck andLk represent the different illumination conditions inthe synthetic scene. Since the albedoa(p) is constant, itcan be moved outside the summation and canceled in thedivision. Thus Eqn. (2) can be rewritten as

r(x, y) =

∑k ckLk · N(p)

∑k ckLk · N(p)

(3)

2.1. Illumination Normal Map

One can see from Eqn. (3) that the illumination ratior ofa pointp is independent of the albedoa, and is a smoothfunction of the surface normalN . This key observationleads to the formulation of the Illumination-Normal-Map(INM), a lookup table that maps a normal to an illuminationratio. Each point in the INM corresponds to a unit normalvectorN in the 3D space. Its horizontal(θ ∈ [0, 2π)) andvertical coordinate(φ ∈ [0, π)) is defined by the projectionof N in certain fixed spherical coordinate system. Its valueis the illumination ratio valueρ for the normalN (Fig. 1).An INM defines an illumination transformation between thereal and the synthetic scene. A good property of INM is that

Ν

θ

φ ρ(θ,φ)

θ

φ

Figure 1. Spherical coordinate and INM.

ρ is a smooth function of(θ, φ). Thus we can computeρ forrelatively sparse samples of the INM and estimate the restusing interpolation.

2.2. Compute Illumination Ratio

Consider a pixel(x, y) in the synthetic image. Given thegeometry model and pose estimation of the object, the cor-responding surface pointp with normalN(p) in the worldspace can be found through back-projection. Its coordinate(θ, φ) in the INM can then be derived by projectingN(p) toa specified spherical space. Thusρ(θ, φ), which equals tor(x, y), can be computed by comparing the correspondingpixel intensities in two images Eqn. (2).

All surface points are illuminated by the same directionallight source. For a convex object (simple self-occlusion),this means that points with the same normals share the sameillumination ratio within one frame. Moreover, if the scenelighting is static, illumination ratio consistency shouldholdfor points with same surface normals across frames. Thismeans multiple observations of the same illumination ra-tio variable can be acquired with the presence of measure-ment noise. We formulate illumination estimation problemin the Kalman filter framework. The detailed approach isexplained in the Section 5.

So far, we have made the assumption that the real im-age and the synthetic image are exactly registered. In otherwords, there is no error in object pose estimation and cam-era calibration. Unfortunately, in practice such errors al-ways exist. Thus the per-pixel-based illumination compu-tation is subject to fail, for the ratio maybe computed bycomparing the imaging of two different 3D surface points.However, since most object are piece-wise smooth and il-lumination ratio is a smooth function of the surface nor-mal,r(x, y) should be spatially smooth, discontinuities onlyhappen at places whereN(p) changes abruptly. This obser-vation leads to the area-based method for illumination ratioestimation. The real and synthetic images can be blurred us-ing a box filter. The illumination ratio is then computed asthe relative scale of intensities of the corresponding pixelsin the filtered images. Notice that blurring across regions

with normal discontinuity should be prohibited.

2.3. Illumination Correction

Once the INM is computed, it can be applied to gener-ate illumination corrected synthetic image. For each pixel(x, y) in the synthetic image, its INM coordinate(θ, φ)can be computed using back-projection as described earlier.The value ofρ(θ, φ) in INM is then set tor(x, y). When(θ, φ) is not a grid point of the INM,ρ(θ, φ) can be com-puted by interpolating the ratio values of neighboring gridpoints. After acquiringr(x, y), the corrected pixel intensityI−(x, y) is computed asI(x, y) ∗ r(x, y). Applying thisprocess to all the pixels occupied by the synthetic objectwill generate an image that is considered to be illuminatedunder the lighting condition of the real scene.

3. Kalman Filtering

The Kalman filter (KF) is a popular and time-testedBayesian estimation tool [10]. While the theoretical as-sumptions about noise characteristics and models are usu-ally violated, in practice the filter has proven very popularas it offers a robust and efficient means for on-line estima-tion. In the classical KF framework, an uncontrolled linearsystem is modeled as

X−

t = AXt−1 + Wt−1 (4)

Zt = HtX−

t + Vt (5)

Eqn. (4) is known as the state-transition or process equa-tion, wheret is the time step,X is the estimate of the realsystem stateX , X− is thea priori (predicted) state,A isthe deterministic state transition between steps, andW isthe process noise. Eqn. (5) is known as the measurementequation, whereZ is the estimate of the real measurementZ, H represents the linear observation function that relatesthe state of the system to its measurements, andV is themeasurement noise.W andV are assumed to be normallydistributed with covariancesQ andR respectively (used be-low), spectrally white, and independent of each other. Inaddition to estimating the system stateX, the filter alsoestimates the error covarianceP . Similar to Eqn. (4) it isassumed thea priori error covarianceP− can be modeled(predicted) usingP from the previous step as follows.

P−

t = APt−1AT + Q (6)

A Kalman filter performs system state estimation using aprediction-correction mechanism. At each frame, the fil-ter predictsX− and Z using the deterministic portions ofEqn. (4) and Eqn. (5), andP− using Eqn. (9). It then cor-rects the state by subtracting the estimated from the real

observations, and factoring the residual back into the es-timated state. Thea posterioristate and error covariancethen serves as the basis for the next step. Themeasurementupdateequations are as follows.

Kt = P−

t HTt (HtP

−

t HTt + R)−1 (7)

Xt = X−

t + Kt(Zt − HtX−

t ) (8)

Pt = (I − KtHt)P−

t (9)

In theory, theKalman gainK as computed in Eqn. (7) pro-vides the optimal weighting of the observation residual inEqn. (8), minimizing the mean of the diagonal ofP .

While Eqn. (4)–Eqn. (9) model linear systems, the filtercan be reformulated to accommodate a nonlinear processfunctiona(X) and/or observation functionh(X) using lin-ear approximations aboutX. In thisextended Kalman filter(EKF) formulation the matricesA andH in Eqn. (4) andEqn. (5) are replaced by the Jacobian matricesA = ∂a/∂XandH = ∂h/∂X respectively. Note that for our object poseestimation we actually used aniteratedEKF. We describethis in Section 4.2. For more complete KF/EKF informa-tion the readers can refer to [17] for example.

4. Iterative Estimation

We implement our model-based tracking system usingthe Kalman filter framework. The state of the systemX ismodeled as[S, M, T ], whereS is the object pose parame-ters,M is the Illumination Normal Map andT representsthe texture of the object. Pixel intensitiesI are used asthe system measurements. The estimated measurement ofa pixel at(x, y) in the synthetic image is computed as:

I(x, y, S, M, T ) = M(HM (S, x, y)) · T (HT (S, x, y))(10)

where HM and HT are the back-projection functions.Given a pose estimateS, they define the mapping froman image pixel at(x, y) to an INM pixel at (θ, φ) anda texture pixel at(u, v). In fact, the camera calibrationand object model are also implicitly used byHM andHT .However, since they are constant throughout the image se-quences, they are not included in the equation. One can seethat Eqn. (10) can be divided into two parts. The fist partM(HM (S, x, y)) defines the illumination correction. Thesecond partT (HT (S, x, y)) represents the texture mapping.

HM defines the mapping between discrete coordinatesin the image and the INM. A pixel in the image may bemapped to a non-grid point in the INM. In this case, theillumination ratio of the pixel should be computed by inter-polating neighboring grid points, where the values are de-fined. A similar issue holds for the image-texture mappingHT . Moreover, the projection of a pixel in the texture spaceis an ellipsoid rather than a point [8]. When a relatively

high-resolution texture is provided, such an ellipsoid coversmultiple texture pixels. Thus, the value of the image pixelshould be computed as a weighted average of the coveredtexture pixels. Eqn. (10) can then be rewritten as

I(x, y, S, M, T ) =∑

θ,φ

M(θ, φ)ξM (HM (S, x, y), θ, φ)

·

∑

u,v

T (u, v)ξT (HT (S, x, y), u, v)

(11)

where ξM and ξT are the filtering kernels centered atHM (S, x, y) andHT (S, x, y).

With the above system model definition, the Kalman fil-ter can be used to estimate the object pose, IlluminationNormal Map, and texture in a recursive fashion. The pro-cessing at each frame includes four steps: prediction, poseupdate, illumination estimation and texture refinement.

4.1. Prediction

We predict the stateX− based on the previous state esti-mateX as in Eqn. (4). Recall that the system state consistsof the object poseS, textureT and Illumination NormalMap M . T is fixed for a given object.S is predicted usingcertain motion model. For instance, constant-speed motionmodel is used in our system.M can be predicted usingdifferent dynamic models. For instance, if we assume thescene lighting is static,M would be constant, and an iden-tity matrix should be used for the state-transition matrixA.However if for example the light source intensity and/or di-rection vary, a non-identity dynamic model could be used topredictM . The prediction step is important as it provides agood initialization for the remaining steps.

4.2. Pose Update

Here we assume that the INM and texture estimatesMandT are fixed. The estimated intensity measurementI ofa specific image pixel is then a nonlinear function of the ob-ject pose estimateS as in Eqn. (11). As described earlier,an Extended Kalman filter can be used to linearizeI arounds. However, sinceI is highly nonlinear this linearizationprocess can cause significant error ifS is not close enoughto the true pose. One approach to address this problem isto use an Iterated Extended Kalman filter (IEKF). The keypoint about the IEKF is that you iteratively refine the statewithin each step. The IEKF generates an estimate of thestate, which can be used to predict the measurement, whichcan be used to update the state, which can again be usedto predict the measurement, etc. A crucial point is that af-ter each iteration, the measurement function is re-linearizedaround the newly updated state estimate. A detailed de-scription of the IEKF can be found in [19].

4.3. Illumination Estimation

In this step, we estimate the values of INM pixels bycomparing the image intensities of corresponding pixels inthe synthetic and the real image. At this time, the poseSand the textureT are fixed. For a particular pixel at(x0, y0)whose normal is mapped to(θ0, φ0) = HM (S, x0, y0) withneighboring INM grid points(θk, φk), the measurementfunction and Jacobian matrix can be written as

I(M) = c(x0, y0)

n∑

k=1

M(θk, φk)ξM (θ0, φ0, θk, φk) (12)

H(M) = c(x0, y0)[ξM (θ0, φ0, θ1, φ1), ..., ξM (θ0, φ0, θn, φn)](13)

wherec(x0, y0) =∑

u,v T (u, v)ξT (HT (S, x0, y0), u, v) isthe intensity of pixel(x0, y0) computed directly from tex-ture mapping,ξM (θ0, φ0, θk, φk) is the interpolation kernelused to computeM(θ0, φ0), andn is the size of the inter-polation window. Currently, we computeM(θ0, φ0) usingbi-linear interpolation of the surroundingn = 4 pixels.

The intensity measurement in Eqn. (12) is a linear func-tion of M . This means the standard Kalman measurementupdate equations can be applied to estimateM . The in-tensity measurement for each pixel is sent to the Kalmanfilter one at a time. ThenH is ann-vector with elementswk = c(x0, y0) ξM (θ0, φ0, θk, φk), Kalman gainK is ann-vector and the noise covarianceR becomes a scalerσ2.In addition, we make the assumption that pixel values in theINM are independent of each other. In this case, the processcovarianceP is ann × n diagonal matrix. Given the defi-nition of these matrices, the measurement update equationsEqn. (7)-Eqn. (9) can be transformed into

Kk = P−

kkwk(

n∑

j=1

(P−

jjw2j ) + σ2)−1 (14)

Mk = M−

k + Kk(I(x0, y0) − I(x0, y0)) (15)

Pkk = (1 − Kkwk)P−

kk (16)

From the above equations one can see that the residual ofintensity measurement of an image pixel is used to updatethe values of the INM pixels that contribute in computingits illumination ratio.

4.4. Texture Refinement

The texture refinement is performed in a way similar tothe illumination estimation. The poseS and illuminationMare held constant. Thus for a synthetic pixel at(x0, y0) withtexture coordinate(u0, v0) = HT (S, x0, y0) we can derivethe following equations

I(T ) = l(x0, y0)n∑

k=1

T (uk, vk) ξT (u0, v0, uk, vk) (17)

H(T ) = l(x0, y0)[ξT (u0, v0, u1, v1), ..., ξT (u0, v0, un, vn)](18)

wherel(x0, y0) =∑

θ,φ M(θ, φ)ξM (HM (S, x, y), θ, φ) isthe illumination ratio at(x0, y0), and(uk, vk) are then tex-ture pixels inside the projection ellipsoid of pixel(x0, y0).We can see thatI is a linear combination of texture intensi-ties. Currently we use equal weights for all the texture pix-els, i.e.ξT (u0, v0, uk, vk) equals to1/n if (uk, vk) is insidethe projection ellipsoid around(u0, v0), or 0 otherwise.

The measurement estimateI(T ) in Eqn. (17) is alinear function of T . Therefore, we can use a linearKalman filter to estimate the textureT using Eqn. (14)-Eqn. (16). The elements in theH vector becomewk =l(x0, y0) ξT (u0, v0, uk, vk).

5. Experiment Results

We have tested our method with both synthetic and realdata. First we present results using synthetic data. Thesynthetic scene consists of one textured sphere (an earthmodel), two cameras, and two directional light sources. Oneof the light sources remains static, and the other changesits direction. Synthetic image sequences rendered from thetwo camera views are used for tracking and illumination es-timation. Fig. 2 demonstrates the illumination correctionprocess. The INMs in (c) are computed by comparing pixelintensities in image pairs (a,d) and (b,e). The INM is thenused to transform (d) to generate illumination corrected im-age (f). Although (a) and (b) are textured differently, theyare generated using the same illumination parameters. Asshown in (c), the INMs computed from (a) and (b) are verysimilar. This demonstrates that our INM estimation methodis insensitive to the surface texture. Fig. 3 shows the track-ing and INM estimation of the synthetic sphere under vary-ing lighting conditions. One can tell from the estimatedINMs that the direction of the light source changes. The res-olution of the INMs shown in Fig. 2 and Fig. 3 is72 × 36.Each pixel in the INM represents an interval of5 × 5 de-grees.

We also tested the proposed method on real data. Weused two calibrated and synchronized Point Grey Dragon-fly cameras to capture image sequences of an ambient cubeundergoing 6D free motion. The cameras were set to cap-ture at 15 frames per second. Currently camera photo-consistency is not enforced, thus an INM is estimated foreach of the two cameras. The scene lighting environmentconsists of the ambient light and two strong distant lightsources. The pose of the cube is manually initialized. Fig. 4illustrates the illumination correction. As shown in Fig. 5,without illumination correction the system loses trackingwithin a few frames. The texture refinement result is pre-sented in Fig. 6. One can see that the texture extracted fromreal image (d) is apparently darker, which shows that there

is a significant illumination difference between the modeland the real scene. However the refined texture (c) esti-mated using illumination-corrected measurements still pre-serves the luminance inherent in the initial texture input (b).

6. Future Work

The illumination ratio is a smooth function under Lam-bertian reflectance. Thus our method only estimates valuesfor a sparse set of INM grid points and recovers the entireINM through interpolation. In fact, due to the low-pass-filtering property of Lambertian reflectance, the illumina-tion can be closely approximated using nine coefficients ofa spherical harmonic basis, as proved in [13, 1]. This indi-cates that we can further reduce the state dimensionality ofthe Kalman filter through estimating the coefficients of thespherical harmonics.

At the moment we use the graphics hardware to rendera predicted imagewithout illumination correction, then wetransfer that image back to main memory and perform allof the subsequent per-pixel computations using the CPU.However we believe that we could implement the illumi-nation correction directly on the graphics card while ren-dering, and then transfer that image to texture memory, re-rendering lower resolution versions instead of blurring, etc.In addition to the base texture of the object model, therecould be a normal (vector) texture and an INM texture, bothwhich are used by Cg programs (for example) that imple-ment the per-pixel computation in parallel.

Acknowledgements

This effort is primarily supported by National Libraryof Medicine contract N01-LM-3-3514: “3D Telepresencefor Medical Consultation: Extending Medical ExpertiseThroughout, Between and Beyond Hospitals,” and in partby NSF grant EIA-0303590: “Ubiquitous Pixels: Trans-forming Collaboration & Teaching with Pervasive Wall Dis-plays.”

References

[1] R. Basri and D. W. Jacobs. Lambertian reflectance and lin-ear subspaces.IEEE Transaction on Pattern Analysis andMachine Intelligence, 25(2), February 2003.

[2] A. J. Bray. Tracking objects using image disparities.Imageand Vision Computing, 8(1):4–9, February 1990.

[3] M. L. Cascia, S. Sclaroff, and V. Athitsos. Fast, reliable headtracking under varying illumination: An apporach based onregistration of texture-mapped 3d models.IEEE Transactionon Pattern Analysis and Machine Intelligence, 21(6), June1999.

[4] G. D. Cubber, H. Sahli, H. Ping, and E. Colon. A colourconstancy approach for illumination invariant color targettracking. Inproceeding of the IARP Workshop on Robotsfor Humanitarian Demining, 2002.

[5] F. Dellaert, S. Thrun, and C. Thorpe. Jacobian images ofsuperresolved texture maps for model-based motion estima-tion and tracking. Inproceeding of the Fourth Workshop onApplications of Computer Vision, 1998.

[6] D. Freedman and M. W. Turek. Illumination-invariant track-ing via graph cuts. Inproceeding of the International Con-ference on Computer Vision and Pattern Recognition, 2005.

[7] D. B. Gennery. Visual tracking of known three-dimensionalobjects. International Journal of Computer Vision,7(3):243–270, April 1992.

[8] N. Greene and P. S. Heckbert. Creating raster omnimaximages from multiple perspective views using the ellipticalweighted average filter.IEEE Computer Graphics and Ap-plications, 6(6), June 1986.

[9] G. D. Hager and P. N. Belhumeur. Efficient region trackingwith parametric models of geometry and illumination.IEEETransaction on Pattern Analysis and Machine Intelligence,20(10), November 1998.

[10] R. E. Kalman. A new approach to linear filtering and pre-diction problems.In Transactions of the ASME Journal ofBasic Engineering, 82, 1960.

[11] E. Munoz, J. M. Buenaposada, and L. Baumela. Efficientmodel-based 3d tracking of deformable objects. Inproceed-ing of the International Conference on Computer Vision andPattern Recognition, 2005.

[12] K. Nickels and S. Hutchinson. Model-based tracking ofcomplex articulated objects.IEEE TRANSACTIONS ONROBOTICS AND AUTOMATION, 17:28–36, 2001.

[13] R. Ramamoorthi and P. Hanrahan. On the relationship be-tween radiance and irradiance: Determining the illuminationfrom images of a convex lambertian object.Journal of theOptical Society of America, 2001.

[14] J. M. Rehg and T. Kanade. Model-based tracking of self-occluding articulated objects. Inproceedings of the FifthInternational Conference on Computer Vision, 1995.

[15] B. Rosenhahn and G. Sommer. Pose estimation of free-form objects. Inproceedings of the European Conferenceon Computer Vision, 2004.

[16] B. Stenger, P. R. S. Mendonca, and R. Cipolla. Model-based3d tracking of an articulated hand. Inproceeding of theInternational Conference on Computer Vision and PatternRecognition, 2001.

[17] G. Welch and G. Bishop. An introduction to the kalmanfilter. Technical report, Department of Computer Science,University of North Carolina at Chapel Hill, 1995.

[18] J. Zhang, J. Yu, and L. McMillan. Robust tracking and stereomatching under variable illumination. In(to appear) theInternational Conference on Computer Vision and PatternRecognition, 2006.

[19] Z. Zhang. Parameter estimation techniques: A tutorialwithapplication to conic fitting.International Journal of Imageand Vision Computing, 25:59–76, 1997.

[20] M. Zobel, M. Fritz, and I. Scholz. Object tracking and poseestimation using light-field object models. Inproceeding ofVISION, MODELING, AND VISUALIZATION, 2002.

Figure 2. Illustration of illumination correction. (a) Ill uminated textured sphere. (b) Illuminatedtexture-less white sphere. (c) Upper: INM computed by compa ring (a) and (d). Lower: INM com-puted by comparing (b) and (e). (d) Texture mapped sphere. (e ) Solid white sphere. (f) Illuminationcorrected textured sphere transformed from (d) using the up per INM in (c). Notice that (a) and (b) arerendered under the same lighting condition.

Figure 3. Tracking and INM estimation for synthetic image se quences with varying illuminations. Thefirst row shows the original synthetic images at frames 3, 33 a nd 53. The second row are the INMestimated at these frames. One can see that the lighting dire ction changes over frames. The last rowrepresents the tracking results. Wire-frame sphere render ed using the estimated pose parameter aresuperimposed onto the original images.

Figure 4. Illumination correction on real data. (a) Real ima ge. (b) Synthetic image without illuminationcorrection. (c) Illumination corrected synthetic image.

Figure 5. Tracking of a cubic object under static directiona l lighting. Wire-frame cube rendered usingthe estimated pose parameter are superimposed onto the orig inal images. The first row shows thetracking results (frames 1, 20 and 110) with illumination co rrection. The second row shows thetracking results (frames 1, 10 and 20) without illumination correction.

Figure 6. Texture refinement results. (a) Original texture. (b) Initial input texture generated by blurring(a). (c) Refined texture. (d) Texture directly extracted fro m a real image.