Forward-Backward Correlation for Template-Based Tracking

Xiao Wang

ECE Dept.

Clemson University

Introduction

Object tracking: An important computer vision problem Security and surveillance Medical therapy Retail space instrumentation Video abstraction Traffic Management Video editing

Template-Based Tracking A classic technique Idea of template-based tracker

Related Work

Jepson et al. Robust Online Appearance Models for Visual Tracking, CVPR 2001

Ho et al. Visual Tracking Using Learned Subspaces, CVPR 2004

Davis et al. Tracking Rigid Motion using a Compact-Structure Constraint, ICCV 1999

Avidan et al. Ensemble Tracking, CVPR 2005

Overview of the Approach

Next Frame

Forward Correlation

Module

Backward Correlation Module

Textured Background?

Gradient Module

Update Template

YesNo

Template Selection: first frame vs. previous frame Motion Model:

Similarity transformation

Template Selection: first frame vs. previous frame Motion Model:

Similarity transformation

Template-Based Tracking

scaling displacement

Template-Based Tracking

Cross Correlation: SSD Cross Correlation: SSD

displacement reference imagesearch image

Template-Based Tracking

Similarity measure: s(Δx, Δy) Correlation Coefficient: c(Δx, Δy)

Similarity measure: s(Δx, Δy) Correlation Coefficient: c(Δx, Δy)

Mean of templateMean of image region

Forward Correlation

Forward Correlation: Reference frame: previous frame Goal: find transformation vector (dx, dy, α) Approach: cross-correlation

Forward Correlation: Reference frame: previous frame Goal: find transformation vector (dx, dy, α) Approach: cross-correlation

Template Update:Template Update:

Put into correlation coefficient framework

Forward Correlation

Drifting Problem: Forward correlation

approximates rotation with translation.

Forward correlation does not check the reliability of the template.

We need a mechanism to question the assumption of forward correlation.

Drifting Problem: Forward correlation

approximates rotation with translation.

Forward correlation does not check the reliability of the template.

We need a mechanism to question the assumption of forward correlation.

Out-of-plane rotationOut-of-plane rotation

Previous frame

Current Frame

Backward Correlation

Consider our problem as motion segmentation Goal of motion segmentation Why is motion segmentation of video sequences difficult?

Under-constrained Occlusion & Disocclusion Image noise

A two-step procedure: Determine the motion vectors associate with each pixel or feature point. Group pixels or feature points that perform common motion.

Backward Correlation

Kanade-Lucas-Tomasi (KLT) feature tracker Idea: minimize the dissimilarity of feature windows in

two images Assumption: mutual correspondence

Kanade-Lucas-Tomasi (KLT) feature tracker Idea: minimize the dissimilarity of feature windows in

two images Assumption: mutual correspondence

Backward Correlation

Now consider the dissimilarity under the template window. Decompose the template window into 2 partitions:

Now consider the dissimilarity under the template window. Decompose the template window into 2 partitions:

Rewrite dissimilarity as:Rewrite dissimilarity as:

foreground background

highlow

Backward Correlation

Background is moving at a different velocity than the foreground. Foreground pixels have similar velocity and generate low SSD error. Correlation between background pixels using foreground velocity

generates high SSD error. Goal: group foreground pixels which are moving at similar velocities

Background is moving at a different velocity than the foreground. Foreground pixels have similar velocity and generate low SSD error. Correlation between background pixels using foreground velocity

generates high SSD error. Goal: group foreground pixels which are moving at similar velocities

Reference frame I(x)Reference frame I(x)Current image J(x)Current image J(x) Difference image

D(x)=[I(x)-J(x+d)]2

Difference image

D(x)=[I(x)-J(x+d)]2

Backward Correlation

Formulations for backward correlation Formulations for backward correlation

Set of template candidates

Correlation coefficient (likelihood)

Untextured Backgrounds

Limitation of backward correlation: Fails if background has little texture. Why? --- Examine the assumption. Backward correlation has no reason to prefer the

foreground to the background which is untextured.

Limitation of backward correlation: Fails if background has little texture. Why? --- Examine the assumption. Backward correlation has no reason to prefer the

foreground to the background which is untextured.

Also low if untexturedlow

Untextured Backgrounds

Likelihood of backward correlation: textured vs. untextured Likelihood of backward correlation: textured vs. untextured

Textured backgroundTextured background Untextured backgroundUntextured background

Foreground

Template containing background pixels

Gradient Module

Motivation: Seek a module focusing on the boundary of the target being tracked.

An edge-based segmentation problem. Prior information: an ellipse model.

Gradient Module:

Motivation: Seek a module focusing on the boundary of the target being tracked.

An edge-based segmentation problem. Prior information: an ellipse model.

Gradient Module:

Unit vector normal at pixel i Intensity gradient

Combining Modules

Gradient module and backward correlation module have orthogonal failure modes.

Textured or Untextured? Use sum of the gradient magnitude of the

neighborhood region. Combination of forward correlation module and

backward correlation module is straightforward. Combination of forward correlation module and

gradient module requires the normalization of the matching scores.

Combining Modules

Normalize the matching score (likelihood): Normalize the matching score (likelihood):

Finial state is decided by:Finial state is decided by:

Adaptive Scale

Vary the scale by ± 10 percent during search process. Filter the result to avoid oversensitive scale adaptation.

Comaniciu et al. Kernel-based object tracking, TPAMI 2003

Vary the scale by ± 10 percent during search process. Filter the result to avoid oversensitive scale adaptation.

Comaniciu et al. Kernel-based object tracking, TPAMI 2003

Size of the best state given by the alg.

Size of the object in the previous frame

Experimental Results:Cluttered Background

Traditional template-based tracker slides off target: Traditional template-based tracker slides off target:

Experimental Results:Cluttered Background

Our algorithm remains locked onto target: Our algorithm remains locked onto target:

Experimental Results:Cluttered Background Tracking error plot:

Our algorithm (blue, solid) vs. traditional template-based tracker (red, dashed)

Tracking error plot: Our algorithm (blue, solid) vs. traditional template-based tracker (red, dashed)

Error in x direction Error in y direction

Experimental Results:Untextured Background

Tracking results of traditional template-based tracker: Tracking results of traditional template-based tracker:

Experimental Results:Untextured Background

Tracking results of our algorithm: Tracking results of our algorithm:

Experimental Results:Occlusion

Experimental Results:Tracking a vehicle

Conclusion

Presented an extension to template-based tracking. Achieved robustness to out-of-plane rotation. Effective tracking in both textured and untextured

environment. Remaining challenges:

Robustness when scale changes. Use motion discontinuities to improve performance. Analysis of parameter sensitivity for untextured

backgrounds.

Questions?

Thank You!