Machine Vision and Applications

Real-Time Tracking-with-Detection for Coping With Viewpoint Change

Shaul Oron · Aharon Bar-Hillel · Shai Avidan

Received: 11 May 2014 / Revised: 02 Nov 2014 / Accepted: 09 Mar 2015

Abstract We consider real-time visual tracking with targetsundergoing view-point changes. The problem is evaluatedon a new and extensive dataset of vehicles undergoing largeview-point changes. We propose an evaluation method inwhich tracking accuracy is measured under real-time com-putational complexity constraints and find that state-of-the-art agnostic trackers, as well as class detectors, are still strug-gling with this task. We study tracking schemes fusing real-time agnostic trackers with a non-real-time class detectorused for template update, with two dominating update strate-gies emerging. We rigorously analyze the template updatelatency and demonstrate such methods significantly outper-form stand-alone trackers and class detectors. Results aredemonstrated using two different trackers and a state-of-the-art classifier, and at several operating points of algo-rithm/hardware computational speed.

Keywords Tracking · Detection · Real time · Fusion

S.OronTel Aviv UniversityTel Aviv 69978, IsraelE-mail: shauloro@post.tau.ac.il

A.Bar-HillelAdvanced Technical Labs Israel,Microsoft research, IsraelE-mail: aharonb@microsoft.com

S.AvidanTel Aviv UniversityTel Aviv 69978, IsraelE-mail: avidan@eng.tau.ac.il

1 Introduction

Visual tracking is a fundamental problem in computer vi-sion and a key component in many higher level vision appli-cations such as surveillance [40], robotic vision [34], activevehicle safety systems [3] and natural user interface [37] toname a few.Performing reliable tracking is very challenging with manyobstacles to overcome. One such challenge is handling largeview-point changes or out-of-plane rotations. In such casesthe tracking problem becomes extremely difficult as it gainsan almost unsupervised nature: the target has to be detectedafter seeing one (or few) relevant examples, and its simi-larity even to these examples is limited. Coping with se-vere view-point changes requires updating the target tem-plate which is a difficult task with many pitfalls. In addition,in an application context, visual tracking is almost alwaysrequired to operate in real-time, i.e. frames must be pro-cessed as fast as they are streaming in. This constraint is anessential aspect of the problem, as complying with it usuallycomes at the expense of accuracy.

We focus on real-time tracking of targets from a knownclass, in scenarios where these targets undergo severe view-point changes. Unlike agnostic trackers, that are not awareof the target class and train classifiers on-line, we assumethe target class is known and train a class detector off-line.

We demonstrate that real time tracking under large view-point changes is difficult for state-of-the-art agnostic track-ing methods and class detectors. To address this problem wepropose several strategies for real-time tracking with detec-tor assisted template update which we name tracking-with-detection. These methods are shown to outperform both track-ers and class detectors when evaluated under run-time con-straints.

In order to systematically span a space of targets under-going severe view-point changes, we have created a chal-

2 Shaul Oron et al.

lenging new dataset, which will be made publicly available.This dataset contains 167 annotated vehicle sequences cap-tured from cameras mounted on a moving car. These se-quences were chosen from a larger corpus mainly focusingon road scenes in which the target vehicle undergoes largeview-point and scale changes (examples shown in figure 6).

The reason we propose a new dataset in this work is thatwe want to focus specifically on view-point changes in asetup where both camera and target are maneuvering. Weare not aware of any other dataset with a similar empha-sis. Moreover, as we show in the experimental section, per-formance of state-of-the-art tracking methods on this newlyproposed dataset indicate that the tracking problem is stillfar from being solved, even in this simplified setup wherethe deformation space is limited almost exclusively to viewpoint change and targets are rigid. This is true either with orwithout runtime constraints.

We propose a new evaluation methodology in which ac-curacy is measured under a computational complexity con-straint. Correct evaluation methodology is important bothfor evaluating overall algorithm performance as well as forparameter tuning. Previous work on tracking evaluation fo-cus only on algorithm accuracy. In this work we take thediscussion on tracking evaluation one step further and arguethat run-time is an integral part of tracking and as such track-ing evaluation cannot be done without considering run-timeconstraints.

We attempt to systematically introduce real-time con-straints into tracking performance evaluation and proposeda new evaluation protocol. The proposed method has the de-sired property of being generic with respect to the accuracyevaluation metric chosen, meaning any accuracy measurecan be used.

In the proposed evaluation method computation time ismeasured on the fly and an algorithm is evaluated in a strictlycausal manner. An algorithm operating slower than the framerate would obtain a delayed state hypothesis, and missedframes will be evaluated using the last known state. We demon-strate how this evaluation methodology can be generalized,emulating more efficient algorithm implementations and faster/ slower hardware configurations, allowing one to gain afairly general understanding of the accuracy / processing-speed trade-off and providing a powerful design tool.

We establish a benchmark for real-time tracking undersevere view point changes using our data. This is done byevaluating six publicly available, state-of-the-art, trackingmethods. We find that all methods struggle when evaluatedfor real-time performance, and even when unconstrained ac-curacy is considered (i.e. algorithms are assumed to performin real-time), the tested methods still leave room for im-provement. These results demonstrate the difficulty of tra-ditional tracking methods in coping with large target view-point changes, even without run-time constraints.

In light of these results we propose incorporating a de-tector into the tracking system, and demonstrate that with-out run-time constraint a detector carefully designed for thetracking task can deliver superior performance. We then ad-dress the question: how to compose a tracking system thatincorporates a detector and runs in real-time ? Althoughtracking-by-detection or incorporating trackers and detec-tors is not a new concept, most such systems run detec-tion exhaustively and are unable to deliver real-time perfor-mance. In general, some real-time detectors do exist, but stillmost state-of-the-art classifiers are computationally complexand not real-time. Therefore tracking-with-detection whichenables non-real-time detectors to be incorporated into real-time tracking systems is an important contribution.

In this context we propose several strategies for perform-ing tracking-with-detection, using the detector to validatethe tracker response as well as for template update purposes.We demonstrate the performance improvement obtained bothover individual system components as well as state-of-the-art methods. In addition, we analyze the template update la-tency associated with each such system and look into therelation between latency, tracking accuracy and template up-date accuracy.

The contributions of our work are two fold: i) Intro-ducing and analyzing several tracking-with-detection meth-ods and demonstrating their superior performance for track-ing a known class under severe view point changes, anddiscussing latency accuracy relations. ii) Providing a newdataset and benchmark for tracking under severe view-pointchanges, and a new methodology for evaluating tracking al-gorithms under run-time constraints.

2 Related work

In this work we benchmark several trackers on our proposeddataset demonstrating the difficulty of tracking under viewpoint changes and establishing that this problem is far frombeing solved. Readers interested in benchmark results forvisual tracking methods are referred to the work of Wu et al.[42]. For a through survey of the vast literature on trackingtechniques we refer the reader to Yilmaz et al. [43]. We focusour attention on real-time visual tracking research and workrelated to detector-tracker fusion and template update.

Handling drift in visual tracking systems is a difficulttask, often addressed by some form of target template up-date. Different strategies have been proposed to perform tem-plate update. The work of Matthews, Ishikawa and Baker[31] is an early attempt to address this problem directly.Their method, like many others that followed, propose atemplate update strategy regularized by the first template(i.e. at frame 1) and using some template appearance basisthat can change over time [10,30]. This work inspired meth-ods using multiple templates and sparse representations to

Real-Time Tracking-with-Detection for Coping With Viewpoint Change 3

represent target appearance [6,23,35,45]. Another widelyused strategy for template update is using on-line learningbased methods. This can be achieved either by pixel-wiseclassification [4,21], or exemplar based classification [5,44].These methods learn a classifier on-line adjusting it accord-ing to target appearance changes. One prominent work inthis category is that of Kalal et al. [24] learning an on-linetarget detector, updating it by mining positive and negativeexamples. However, this technique is agnostic to the targetclass and learns a detector on-line. It also keeps using theinitial target appearance to avoid drift, unlike our proposedmethod. An additional strategy that can be viewed as a typeof template update is updating the template not directly butrather through adjusting parameters controlling the space ofpossible matches [26,32].

Tracking-by-detection or incorporating trackers and de-tectors [2,12,27,38] is an appealing approach for limitingdrift and producing reliable tracking. This idea can be usedwhen one knows the class of objects being tracked. Unfor-tunately the above mentioned methods require running thedetector exhaustively on each frame and thus are unable torun in real-time. Fan et al.[15] propose a unified approachof tracking and recognition incorporating trackers and detec-tors for multiple object classes, however their method is alsounable to run in real-time and not evaluated under such con-straints. Some template-based detectors for specific classes(pedestrians, faces) achieve real-time detection [8,11] andenable real-time tracking-by-detection. However, state-of-the-art general-purpose classifiers usually do not operate inreal-time. Such classifiers often include multiple deformableparts [7,19], and/or multiple prototypes for handling dif-ferent viewpoints or large class variability. More generally,classification is far from being a solved problem, and inmany cases requires considerable computations. It is likelythat while performance improves, state-of-the-art classifierswill remain slow in the near future (for example because ofthe need to run on weak hardware) and the best classifierswill usually not be real-time ones. On the other hand, manyvisual tracking algorithms are able to operate at close to real-time or faster [30,31,45]. Our work hence focuses on fusingnon real-time detectors with real-time tracking algorithms.

Evaluation methodology for visual tracking was discussedin multiple papers and an elaborated review of performancemeasures and evaluation programs can be found in [9,25,28]. However we are not aware of any previous work at-tempting to systematically introduce run-time constraints intotracking evaluation.

3 Tracking-with-detection

We present several tracker-detector fusion schemes, evalu-ating them under real-time constraints. We consider trackersof various speeds integrated with a ’slow’ detector for which

typical detection takes a few frames to carry out. In thesesystems the tracker is responsible for continuously trackingthe target while the detector provides updates to the appear-ance of the target being tracked. In this context we analyzethe latency associated with updating the template, i.e. thetime or number of frames that pass between the beginningof a detection process until the template is updated.

For the latency analysis the following notations are used.Let us denote the input frame rate fin and the interval be-tween frames tin = 1

fin. We denote tdet the time required to

obtain a detection and ttrk the tracker processing time for asingle frame. Assuming this tracker operates in real-time i.e.ttrk < tin, we denote by tidle the time between consecutiveframes not used for tracking computations:

tidle = tin − ttrk (1)

This idle processing bandwidth can be used to perform ad-ditional tasks such as detection as will be discussed later.Intuitively, larger latency entails accuracy decrease since bythe time a new template is available, it is already outdated.In section 4 we show empirically that this is indeed the case.

3.1 Tracking-by-detection

We begin with the known tracking-by-detection scheme(e.g. [2,12,27]). In this scheme a detector is used for track-ing by performing detection for every frame streaming in.In this system the detector performs the tracking and thereis no additional tracker. We can compute the latency ldet0associated with the detection process:

ldet0 =tdettin

(2)

This means that every time we perform detection it takes usldet0 frames until the results is obtained and updated. Wenote that, of course, if tdet < tin resulting in ldet0 < 1 thenwe have a system working in real time without any latency.

3.2 Template update with near-real-time tracker

We consider scenarios where ttrk ≈ tin i.e. the trackeris near-real-time, but not much faster (or slower). In this sit-uation applying the tracker requires almost all the computa-tional bandwidth i.e tidle → 0. In such a scenario, for anygiven frame, we can either run tracking or perform detectionin order to update our template (but not both). If we performdetection then we suffer a latency of ldet0 frames, as in thetracking-by-detection case, and this affects accuracy. On theother hand if we do not perform template update our trackermay drift, losing the target all together. This trade off leavesus with the difficult question of “When should we update

4 Shaul Oron et al.

the template?”. A trivial solution would be to perform tem-plate update on a regular basis, using some fixed interval i.e.performing template update every K frames. However wewould like to perform tracking most of the time updating thetemplate only when necessary in order to avoid state extrap-olation which damages accuracy. A key observation in thisrespect is that classification is much faster than detection,since the latter requires, at least implicitly, classification ofmany locations and scales. A single detector query (classifi-cation) can usually be done very fast even for a complex de-tector. In light of this observation we propose the validationscheme, resembling [41]. In this scheme the tracker result,i.e. its proposed state, is validated by classifying it using thedetector. If the validation succeeds, meaning the proposedstate is a valid detection, we continue tracking. Only whenthe validation fails we resort to template update by running adetection scheme in a window around the last known targetstate, while extrapolating until a detection is obtained. Oncewe obtain the detection results (although delayed) we up-date template appearance and continue from the last knowntarget state. The described scheme answers the question of“when should we update the template?”, and is expected tooutperform the naive fixed interval template update.

3.3 Template update with a fast tracker

In this scenario ttrk � tin, which means that tidle � 0,i.e. we have substantial idle time, not used for tracking com-putations, that can be used for other purposes. We proposeusing this idle time to perform detection and update the tem-plate. The most simple way to do this would be to run thedetector in the idle time and update the template once a de-tection is obtained. Using this update scheme would resultin template update latency of ldet1 frames.

ldet1 =tdettidle

=tdet

tin − ttrk(3)

The interpretation of this scenario is that tracking is per-formed continuously with template updates occurring ev-ery ldet1 frames. In this scenario the template is updatedonce obtained although it is relevant to ldet1 frames ago. Wenote that equation (3) is only meaningful when tin > ttrk,also naturally ldet0 < ldet1 (for any feasible setup wherettrk > 0).

We consider a second option. Since any detection ob-tained is delayed, i.e. relevant to some past frame, we pro-pose tracking it to the current frame rather than instanta-neously updating the template. This means we take the newdetection, ’catch-up’ to the input stream, and only then per-form the template update. This would of course result inlonger latency but might provide more accurate appearanceupdate. We denote the latency associated with this strategy

by ldet2. This latency can be broken down into two parts:

ldet2 = ldet1 + ltrk (4)

where ldet1 is the detection latency calculated earlier andltrk the tracking latency induced by tracking the detectionto the current frame (the ’catch-up’). We can deduce ldet2from the following equation:

ltrk · tidle = (ldet1 + ltrk) · ttrk (5)

To understand this equation note that the number of framesbetween the time a detection is obtained and until we catch-up is ltrk. This means the catch-up tracker has a time intervalof ltrk · tidle (left-hand side) in which it has to track throughldet1 + ltrk frames (right hand side). Isolating ltrk we have:

ltrk =ldet1 · ttrktidle − ttrk

=ldet1 · ttrktin − 2ttrk

(6)

Substituting ltrk and ldet1 in equation (4) gives:

ldet2 = ldet1(1 +ttrk

tin − 2ttrk) =

tdettin − 2ttrk

(7)

We observe that equation (7) is only meaningful when tin >2ttrk, since this method can only work if two tracker appli-cations can be done faster than the inter-frame interval (onefor the main tracker, and one for the ’catch-up’ tracker).

From a system design perspective, in order to avoid verylong latencies, limitations on ttrk emerge. In the scenarioof instantaneous update, with latency ldet1, we should havettrk � tin. If we perform ’catch-up’, with latency ldet2,then we require ttrk � 1

2 tin.Comparing the performance of these two strategies pro-

vides interesting insights into the trade off between trackingaccuracy, template update accuracy and template update la-tency, and will be further explored in the experimental sec-tion.

4 Experiments

We evaluate tracking-with-detection using the vehicle objectclass. We present a new dataset, which will be made pub-licly available1. The data contains 167, fully annotated se-quences, ranging from 100 to 1200 frames. These sequenceswere captured from 3 cameras (two grayscale, one color),recording 640× 480p images at fin = 15fps. The cameraswere mounted on a moving car, facing backwards. These se-quences were mostly chosen to exhibit road scenes in whichcars undergo large view-point and scale changes. The datacontains 77 overtake sequences, 22 traffic circles, 42 turnand 26 tailing sequences with over 35,000 frames overall.Typical examples from this data are presented in figure 6.For our experiments the data was divided into a training set

1 http://www.eng.tau.ac.il/∼oron/TWD/TWD.html

Real-Time Tracking-with-Detection for Coping With Viewpoint Change 5

comprised of 117 sequences and a test set of 50 sequences.The sequences were divided between the train and test setssuch that both sets have examples of overtakes, traffic cir-cles, turns and tailing.

The rest of this section is organized as follows. We be-gin by introducing a new real-time evaluation methodologyfor real-time and non-real-time algorithms under run-timeconstrains. We then provide implementation details relatedto our experiments. Next, we present benchmark results ofseveral state-of-the-art tracking methods on our dataset, fol-lowed by tests demonstrating the performance of our tracking-with-detection schemes. We validate empirically our latencyanalysis equations. Finally we demonstrate how our real-time evaluation methodology can be used to emulate moreefficient algorithm implementations or different hardwareoperating points providing a powerful system design tool.

4.1 Evaluation methodology

Most applications requiring tracking also require it tooperate in real-time. Therefore evaluation of tracking algo-rithms should also be carried out considering run-time con-straints. We suggest a methodology allowing any algorithmto be evaluated under real-time constraints, inherently demon-strating the speed-accuracy tradeoff. The proposed protocolis complementary to previous works on tracking evaluationin the sense that it is accuracy measure independent mean-ing any accuracy measure can be used with our protocol.We also discuss and demonstrate how the proposed method-ology can be used to simulate acceleration of different sys-tem components as well as simulate running on platformswith different computational power. In doing so one can gainan understanding of the speed-accuracy tradeoff in a giventracking system.

Simulating real-time is done by measuring actual pro-cessing times, on the fly (not including I/O), of the algorithmused for tracking, denoted Alg. We note that Alg is a track-ing algorithm in the broad sense of the word and can be atracker, a detector (used for tracking), or a fusion of the two.Denote the input frame rate fin and the number of frames inthe video by N , indexing them 0, .., N − 1. The algorithmAlg receives a certain frame f and returns a state X andits processing time tproc. Algorithm Alg is evaluated basedon a sequence of N responses, with response i reflecting itsknowledge of stateX at time (i+1)/fin, i.e. when frame i isreplaced by its successor. The algorithm response sequenceis determined using the protocol given in algorithm 1.

The scheme described in algorithm 1 can be trivially ex-tended to handle tracking with multiple components (e.g.a tracker and a classifier). Note that for ’skipped’ frames

Algorithm 1: Evaluation protocol considering run-time constraints

Input: {fi}Ni=1 Sequence of video framesOutput: {Xi}Ni=1 State hypothesis for each input frame

1 Set t← 0 (initialize time line)2 Set f ← f13 while t < N/fin do4 Set f ← bt · finc5 if frame f has already been processed then6 Set t← dt · fine/fin7 else8 Send frame f to Alg for processing9 Upon receiving algorithm response X and tproc

10 Set t← t+ tproc11 Record state X for frame bt · finc (but not for earlier

frames)12 end13 end14 for each frame f from 0 to N − 1 do15 if frame f was not processed i.e. has no recorded response

then16 Use the last recorded response (prior to f ) as the state

at frame f17 end18 end

(frames with no recorded response) we perform zero-order-hold (ZOH) extrapolation i.e. holding the last known tar-get state. Our main justifications for using ZOH over morecomplex extrapolation techniques is the desire to make aclean comparison, that does not depend on the extrapolationmethod and its fit to various tracking schemes.

As presented above, the evaluation protocol is limitedto measuring real-time tracking performance of specific al-gorithm implementations and specific hardware. However,the protocol can easily simulate faster or slower computingspeed of all, or part, of the algorithms, simply by dividingthe measured computation time tproc by an ’accelerationfactor’ α. For example, dividing tproc by α = 2 predictsthe tracking performance using twice-as-powerful hardware.Moreover, this evaluation scheme may guide more subtle en-gineering decisions. For example, given some fused detector-tracker system one might be interested in knowing: Whatwould happen if we implement one of the components moreefficiently? Where should we invest our optimization efforts?To answer such questions one can assign two different vir-tual acceleration coefficients for both tracker αtrk and de-tector αdet.

For measuring tracking accuracy we adopt the methodpresented in a recently published tracking benchmark [42].For each frame we measure the bounding box overlap givenby overlap =

area(Bd∩Bgt)area(Bd∪Bgt)

where Bd and Bgt denotes thedetected and ground truth bounding boxes respectively. Webuild a curve showing success rate, i.e. the fraction of frameswith overlap ≥ threshold for threshold ∈ [0, 1] for eachsequence and average over all sequences in order to obtain

6 Shaul Oron et al.

a final success curve. We then measure the area under thecurve (AUC). This measure quantifies both centering accu-racy as well as scale accuracy giving a broad performanceoverview, not restricted to a single threshold value (whichmight be biased).

4.2 Implementation detail

We have implemented the different tracking with de-tector assisted template update systems presented in sec-tion 3. The detector used in our implementations is a DPM[17]-based detector, which was adapted for the tracking task.The classifier is based on the cascade implementation [17]available with the on-line code package [16]. We trained aview-point invariant vehicle classifier with resemblance tothe vehicle classifier of Leibe et al.[27]. Four models weretrained, each capturing the vehicle in a different-view point(0◦,±30◦,±60◦,±90◦). The classifiers were trained on ex-amples extracted from our training set of 117 sequences.

The basic tracker used is our own implementation of thestandard optical flow LK tracker [30], this tracker searchesfor a rigid 2D warp, with 3 degrees of freedom (x,y,scale),between the target template and the new frame starting fromthe last known target location and scale.

Turning the DPM classifier into a tracking-by-detectionalgorithm that runs at a reasonable rate was done in thefollowing manner. First, spatial, scale and view-point con-sistencies were considered, i.e. detection is performed in asmall region-of-interest (ROI), around the last known targetstate, and only in the last known and adjacent view-points(e.g. if the last view-point was 30◦ we search at 0◦, 30◦, 60◦).Multiple detections, in the ROI, are ranked based on classi-fication score as well as spatial and scale consistencies rela-tive to the last known state (modeled using normal distribu-tions around the last known location and scale). Secondly,standard DPM implementation performs a fine grain scalespace search using 6 octaves each of which is divided intomany sub-scales resulting in ∼ 90 scales. This is done sincethe detector has no prior knowledge regarding target scale.In tracking, on the other hand, one has a good estimation oftarget scale using the current state. This allows using fewerscales and in fact we limit DPM scale search to 3 octaves(0.25 − 2) containing only 25 fine grain scales. The scale0.75 is located in the middle of this scale range (i.e. at scaleindex 13) and we resize our image aiming that the target isdetected at this scale. This is done in order to ensure the tar-get is detected in the limited scale space searched and alsoin order to help maintain constant detection time. Perform-ing this is easily done as we know for each DPM detectionat which scale it occurred. In order to limit degradation inresolution we also limit image resizing factor to the range(0.25− 2).

All our experiments were conducted using a machineequipped with an Intel R© CORE

TMi7-2620M 2.7GHz pro-

cessor and 8GB of RAM. All our code is Matlab R© basedwith some mex implementations. The input frame rate wasfin = 15fps, typical runtime performance for our trackerand detector for this setup are as follows: The DPM de-tector runs at ∼ 2.5fps (= fin/6) for a search windowof 150 × 150 pixels. Our LK tracker runs at ∼ 60fps (=4 × fin), and when also performing validation we run at∼ 30fps (= 2× fin), both for a 100× 50 pixel target.

Table 1 summarizes all the tracking methods evaluatedin this work. We note that any parameter used by any algo-rithm was kept fixed in all experiments.

Table 1 Summary of tracking methods evaluated.

Method Description

ASLA Tracking via Adaptive Structural Lo-cal Sparse Appearance Model [23]

CT Compressive Tracking [44]DPM Tracking based on DPM onlyL1 Real-time Robust L1 Tracker [6]

L1-xαtrk-DPM-xαdet

L1-DPM fusion with αtrk, αdet ac-celeration factors for tracking and de-tection

LK Basic LK tracker [30]

LK-DPM-FIX LK with DPM based template updateat fixed intervals

LK-DPM-WO-CUP LK at each frame with DPM using LKidle time

LK-DPM-CUP As LK-DPM-WO-CUP only LK alsoused for “catch-up”

LK-DPM-VLD LK with validation and DPM basedtemplate update

LOT Locally orderless tracking [32]

SCM Tracking via Sparsity-based Collabo-rative Model [45]

TLD Tracking-Detection-Learning [24]

4.3 Experiment 1 - Tracking under severe view pointchanges is difficult

We benchmark several tracking methods on our new dataset.Among these methods are several state-of-the-art algorithmsproducing top tier results in a recently published trackingbenchmark [42]. The algorithms are: Tracking-Detection-Learning (TLD) [24], Locally Orderless Tracking (LOT)[32], Real-time Robust L1 Tracker (L1) [6], Tracking viaSparsity-based Collaborative Model (SCM) [45], Compres-sive Tracking (CT) [44] and Tracking via Adaptive Struc-tural Local Sparse Appearance Model (ASLA) [23].

Real-Time Tracking-with-Detection for Coping With Viewpoint Change 7

The tracking performance of the benchmarked methodsare presented in figure 1. Figure 1-(a) shows real-time track-ing results, measured using the methodology outlined in sec-tion 4 (TLD results are missing from this graph as it is onlyavailable in an executable version and could not be adjustedfor running with our evaluation methodology). It is clearthat all methods perform rather poorly in these conditions,with all methods having AUC< 0.5. Also if we look atoverlap = 0.65, which is a reasonable operating point fortracking, we observe that all methods produces a successrate < 0.4.

Since most of the provided implementations are academiccodes, one may claim that they are not optimized and canbe better implemented to meet the real-time constraint. Toestimate the potential for improvement, we evaluated thesealgorithms without run-time constraints, i.e. assuming theyrun at real-time. These results are shown in figure 1-(b). Re-moving the real-time constraint significantly improves thetracking performance, yet still at an overlap of 0.65, the bestmethods provide a success rate of ∼ 0.65 which is still notsuited for any practical application.

4.4 Experiment 2 - Tracking-with-detection

We evaluate the performance of the proposed tracking-with detection systems. To do so we implement these sys-tems using our DPM detector and LK-tracker. Results arepresented in figure 2.

Tracking-by-detection discussed in section 3 is imple-mented using the DPM detector. We first evaluate tracking-by-detection without run-time constraints, i.e. performingdetection at every frame, denoting it DPM-NOT-RT. Weobserve that, as expected, this specially designed view-pointinvariant detector is able to produce good results with AUCof 0.766 and a success rate > 0.9 for overlap threshold of0.65. These results indicate that using a detector can in-deed solve the accuracy problem. However once tracking-by-detection is evaluated under real-time constraints (de-noted DPM), we see that accuracy decreases significantly.These results, along with the benchmark results, motivate usto perform tracking-with-detection incorporating a detectorin the tracking system in an attempt to achieve more reliabletracking performance under real-time constraints.

Both template update with near-real-time tracker meth-ods, presented in section 3, are implemented. The first, in-cludes template update at fixed intervals, denoted LK-DPM-FIX. The second, where the template is update using thevalidation scheme is denoted LK-DPM-VLD. The templateupdate latency of both methods is ldet0 and the main dif-ference between the two is the update interval. In order tomake a fare comparison we measure the performance ofLK-DPM-FIX with all update interval values between 2

frames and 10 frames (K = {2, 3, .., 10}) and present thebest results (obtained for K = 5).

The two strategies of section 3 which use a fast tracker(ttrk < 1

2 tin) are implemented next. These algorithms per-form tracking at each frame then use tidle for template up-date purposes. We denote by LK-DPM-WO-CUP, the strat-egy where the template is updated as soon as a detection isobtained. This strategy has an update latency of ldet1. Thesecond option is not to update the template once it is ob-tained but rather use tidle to perform ’catch-up’. This strat-egy is denoted LK-DPM-CUP and has an update latency ofldet2.

In addition to the above methods we also present the per-formance of the LK tracker on its own as well as the ”best-tracker” curve showing the best performance achieved byany of the benchmark trackers tested with real-time con-straint at each operating point independently (i.e. for eachthreshold we choose the tracker from figure 1-(a) that per-formed best).

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

Overlap threshold

Su

cce

ss r

ate

DPM−NOT−RT (0.766)LK−DPM−CUP (0.709)LK−DPM−WO−CUP (0.697)LK−DPM−VLD (0.686)LK−DPM−FIX (0.623)DPM (0.583)BEST−TRK (0.488)LK (0.432)

Fig. 2 Experiment 2 - Tracking-with-detection: LK-DPM methodsdominate over LK and DPM alone which have low performance, theformer due to lack of information, the latter due to lack of processingspeed. Cyan curve shows performance of DPM tracker without anycomputational constraints, demonstrating potential performance. Seetext for more detail (best viewed in color).

Results presented in figure 2 show that the different tem-plate update strategies provide significant improvement of∼40% (multiplicative) in AUC over the best tracking method,and a 17%− 21% improvements over the DPM tracker. Theresults show the significance of template update and the syn-ergy obtained by tracker-detector fusion, producing a systemsuperior to its components. Moreover, the LK-DPM meth-ods, evaluated under real-time constraint, have a 3% − 6%

increase in AUC over the best tracking methods evaluatedwithout real-time constraint. For overlap = 0.65 perfor-

8 Shaul Oron et al.

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

Overlap threshold

Su

cc

es

s r

ate

L1 (0.472)ASLA (0.450)CT (0.392)LK (0.432)LOT (0.251)SCM (0.258)

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

Overlap threshold

Su

cc

es

s r

ate

ASLA (0.665)SCM (0.635)L1 (0.668)LOT (0.611)TLD (0.606)CT (0.392)LK (0.477)

(a) With real-time constraint (b) Without real-time constraint

Fig. 1 Experiment 1 - Tracking is difficult: Benchmark of several state-of-the-art tracking methods on the proposed dataset both with (a) andwithout (b) real-time constraint. Graphs show the tracking success rate for overlap threshold ∈ [0, 1]. Numbers in the legend indicate the AUCfor each method. The task is difficult due to frequent view-point changes, and this difficulty is stressed when methods are evaluated under acomputational complexity constraint (best viewed in color).

mance jumps from 0.6 (L1 not real-time) to ∼ 0.75 forthe LK-DPM methods. This performance gain can be at-tributed to the additional class prior information availablefor the LK-DPM methods.

We compared LK-DPM-VLD which uses the validationscheme to decide when to perform template update and LK-DPM-FIX which updates the template at fixed intervals. Inthe comparison, we notice that using the validation schemeoutperforms fixed interval updates. This indicates that thetemplate update rate is not uniformly distributed, which ofcourse makes sense, as we expect the template to be updatedmore frequently when appearance changes occur, e.g. turn-ing, relative to a simple tailing scenario.

4.5 Experiment 3 - Latency and accuracy

We simulate different detection and tracking operatingpoints for two tracking-with-detection methods, LK-DPM-WO-CUP with latency ldet1 and LK-DPM-CUP with la-tency ldet2. This is done by accelerating or decelerating oursystem assigning different values of ’acceleration factor’ αas explained in section 4. Introducing α into latency equa-tions (3) and (7) gives:

ldet1 =tdetα

tin − ttrkα

(8)

ldet2 =tdetα

tin − 2 ttrkα(9)

We measure the empirical average tracking and detec-tion processing intervals ttrk and tdet when the algorithmsrun without acceleration (i.e. α = 1). We plug in these val-ues along with tin into equations (8) and (9) with differentvalues of α, ranging from 0.5 to 10, and obtain a predictionfor the latency of each method at several operating points.We then run the tracking methods with these α values us-ing our evaluation methodology and compare the predictedlatency values with the latency values measured empirically.

Results are presented in figure 3 showing the predictedlatency for different values of α as obtained from equations(8) and (9) along with the latency measured experimentally.The empirical latencies are highly correlated with the pre-dicted ones, with ρ = 0.996 for LK-DPM-CUP and ρ =

0.997 for LK-DPM-WO-CUP, validating our latency anal-ysis.

After seeing our latency predictions are correct we turnour attention to the relation between tracking accuracy andlatency, plotting this data for the different α values, as pre-sented in figure 4. We first note the trends showing an in-crease in accuracy for decreasing latency, for each methodon its own. This trend is broken for LK-DPM-CUP at α >2 where the performance reaches that of the DPM detec-tor. We observe the performance margin between the meth-ods, meaning that for a given latency LK-DPM-CUP out-performs LK-DPM-WO-CUP (again this is true for α <

3) . These results indicate that more accurate template up-date, achieved by performing ’catch-up’, leads to better sys-tem accuracy compared to instantaneous appearance update.More generally we note that producing more accurate tem-plate update should be considered even at the expense of in-

Real-Time Tracking-with-Detection for Coping With Viewpoint Change 9

0 0.5 1 1.5 2 3 4 6 8 100

5

10

15

20

25

30

Acceleration factor α

La

ten

cy [

fra

me

s]

LK−DPM−WOCUP MeasuredLK−DPM−WOCUP PredictedLK−DPM−CUP MeasuredLK−DPM−CUP Predicted

Fig. 3 Experiment 3 - Latency prediction: Measured and predictedlatency value for two tracking-with-detection methods. Prediction andmeasurements are highly correlated (ρ ≤ 0.996 for both methods),validating our latency analysis (best viewed in color).

creased latency, as this might improve overall performance.

5 10 15 20 25

0.62

0.64

0.66

0.68

0.7

0.72

0.74

0.76

0.78

0.8

Latency [frames]

Acc

ura

cy [

AU

C]

x0.50

x0.75

x1.00

x1.25

x1.50x1.75

x2.00

x4.00

x10.0

x0.75

x1.00

x1.25

x1.50x1.75

x2.00x4.00

x10.0

x0.50

x0.75

x1.00

x1.25

x1.50x1.75

x2.00

x4.00

x10.0

x0.75

x1.00

x1.25

x1.50x1.75

x2.00x4.00

x10.0LK−DPM−WOCUPLK−DPM−CUP

Fig. 4 Experiment 3 - Accuracy and latency: For each of the meth-ods lower latency leads to higher accuracy. The performance marginbetween methods indicates that more accurate template update can re-sult in better overall system performance even at the expense of in-creased latency. See text for more details (best viewed in color).

4.6 Experiment 4 - Real-time evaluation for betterdesign

In the following experiment we take the L1 tracker, whichis the leading tracking algorithm under real time constraints,and combine it with our DPM detector using the validationscheme presented in section 3, we denote this system L1-DPM. We assign individual acceleration factors, αtrk, αdet,for tracking and detection respectively. Using these factorswe emulate the following scenarios i) Accelerating only thetracker by×2 (αtrk = 2, αdet = 1) ii) Accelerating only thedetector by ×2 (αtrk = 1, αdet = 2) iii) Running the sys-tem on twice as powerful hardware (αtrk = 2, αdet = 2).We denote the different accelerated systems by L1xαtrk-DPMxαdet e.g. L1x1-DPMx1 denotes the original systemwithout any accelerations.

Results are presented in figure 5, on the left side, (a),is the full scale results and on the right, (b), we zoom into the region threshold ∈ [0.5, 0.8] which is the most in-teresting region for practical applications. We first observethat L1-x1-DPM-x1 significantly outperforms both of itscomponents (L1 and DPM). In addition we observe thatthis system also outperforms LK-DPM-VLD, which can beattributed to the performance margin between L1 and LKtrackers. The results for accelerating the detector and trackerprovide interesting design insights. We observe that the per-formance gained by accelerating the detector by a factor of×2 is equivalent to the expected performance gain obtainedby accelerating the tracker by a factor of ×2. Such a re-sult is of high practical engineering interest as one can nowdecide where to direct research resources. Typically, accel-erating a tracker is easier than accelerating a detector and,according to these results, is expected to produce the sameperformance gain. Moreover we can see that increasing thecomputational power by a factor of ×2 is only marginallybetter than accelerating only the tracker, with improvementnoticed mostly at low threshold values, this again may sug-gest that for this system accelerating the tracker is the mostcost-effective coarse of action for increasing performance.

5 Conclusions

Tracking, even rigid, targets undergoing severe view-pointchanges is a challenging task, made even more difficult un-der runtime constraints. Using our newly proposed datasetwe have demonstrated that state-of-the-art tracking meth-ods still struggle with tracking rigid targets undergoing largeview point changes both with and without runtime constraints.When the target class is known, using a class detector is keyfor making a successful tracking system. Using a carefullytrained classifier can boost performance significantly, trans-ferring a system from ’almost always fail’ to ’usually suc-ceed’.

10 Shaul Oron et al.

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

Overlap threshold

Su

cce

ss r

ate

Enlarged in (b)

DPM−NOT−RT (0.766)L1x2−DPMx2 (0.740)L1x2−DPMx1 (0.736)L1x1−DPMx2 (0.734)L1x1−DPMx1 (0.725)LK−DPM−VLD (0.686)DPM (0.583)L1 (0.472)

0.45 0.5 0.55 0.6 0.65 0.7 0.750.45

0.5

0.55

0.6

0.65

0.7

0.75

0.8

0.85

0.9

0.95

Overlap threshold

Su

cce

ss r

ate

DPM−NOT−RT (0.766)L1x2−DPMx2 (0.740)L1x2−DPMx1 (0.736)L1x1−DPMx2 (0.734)L1x1−DPMx1 (0.725)LK−DPM−VLD (0.686)DPM (0.583)L1 (0.472)

(a) Full scale (b) Zoom in

Fig. 5 Experiment 4 - Real-time evaluation as system design tool: We examine L1-DPM when accelerating only the tracker, only the detectoror both. Results indicate that the performance gained by tracker acceleration is similar to detector or full system acceleration providing invaluableinsight into where optimization efforts should be focused (best viewed in color).

When working under runtime constraints wise fusion of adetector into a tracking system is advised and is expected todeliver superior performance compared to plain fusion tech-niques. Advanced fusion schemes, such as the ’validation’or ’catch-up’ schemes, proposed in this work, were demon-strated to out perform simpler fusion techniques such as sim-ply running the detector on every possible frame (’DPM’)of updating the template at fixed intervals using the detector(’LK-DPM-FIX’).Finally, when practical real-time tracking systems are de-signed one cannot evaluate tracking accuracy alone but ratherconsider accuracy subject to runtime constraints. In this con-text our newly proposed evaluation methodology will give abetter estimate of actual system performance, enable betterunderstanding of its latency and allow an insight into thesensitivity of SW/HW acceleration.

References

1. Thirteenth ieee international workshop on performance evaluationof tracking and surveillance (pets) (2010)

2. Andriluka, M., Roth, S., Schiele, B.: People-tracking-by-detectionand people-detection-by-tracking. CVPR (2008)

3. Avidan, S.: Support vector tracking. PAMI (2004)4. Avidan, S.: Ensemble tracking. CVPR (2005)5. Babenko, B., Yang, M., Belongie, S.: Visual tracking with online

multiple instance learning. CVPR (2009)6. Bao, C., Wu, Y., Ling, H., Ji, H.: Real time robust l1 tracker using

accelerated proximal gradient approach. CVPR (2012)7. Bar-Hillel, A., Levi, D., Krupka, E., Goldberg, C.: Part-based fea-

ture synthesis for human detection. ECCV (2010)8. Benenson, R., Mathias, M., Timofte, R., Van Gool, L.: Pedestrian

detection at 100 frames per second. CVPR (2012)

9. Bernardin, K., Stiefelhagen, R.: Evaluating multiple object track-ing performance: the clear mot metrics. EURASIP Journal on Im-age and Video Processing (2008)

10. Cootes, T., Edwards, G., Taylor, C.: Active appearance models.TPAMI (2001)

11. Dollr, P., Belongie, S., Perona, P.: The fastest pedestrian detectorin the west. BMVC (2010)

12. Ess, A., Leibe, B., Schindler, K., Van-Gool, L.: Robust multi-person tracking from a mobile platform. TPAMI (2009)

13. Everingham, M., Van Gool, L., Williams, C., Winn,J., Zisserman, A.: The PASCAL Visual Object ClassesChallenge 2011 (VOC2011) Results. http://www.pascal-network.org/challenges/VOC/voc2011/workshop/index.html

14. Everingham, M., Van-Gool, L., Williams, C., Winn, J., Zisserman,A.: The pascal visual object classes (voc) challenge. IJCV (2010)

15. Fan, J., Shen, X., Wu, Y.: What are we tracking: a unified approachof tracking and recognition. IEEE Transactions on Image Process-ing 22(2), 549–560 (2013)

16. Felzenszwalb, P., Girshick, R., McAllester, D.: Dis-criminatively trained deformable part models, release 4.http://people.cs.uchicago.edu/ pff/latent-release4/

17. Felzenszwalb, P., Girshick, R., McAllester, D.: Cascade object de-tection with deformable part models. CVPR (2010)

18. Felzenszwalb, P., Girshick, R., McAllester, D., Ramanan, D.: Ob-ject detection with discriminatively trained part based models.TPAMI (2010)

19. Felzenszwalb, P., Huttenlocher, D.: Pictorial structures for objectrecognition. IJCV 61(1) (2005)

20. Geiger, A., Lenz, P., Urtasun, R.: Are we ready for autonomousdriving? the kitti vision benchmark suite. In: CVPR (2012)

21. Grabner, H., Grabner, M., Bischof, H.: Real-time tracking via on-line boosting. BMVC (2006)

22. Hager, G., Belhumeur, P.: Efficient region tracking with paramet-ric models of geometry and illumination. TPAMI (1998)

23. Jia, X., Lu, H., Yang, M.: Visual tracking via adaptive structurallocal sparse appearance model. CVPR (2012)

24. Kalal, Z., Mikolajczyk, K., Matas, J.: Tracking-learning-detection.TPAMI (2010)

25. Kasturi, R., Goldgof, D., Manohar, S., Garofolo, J., Bowers, R.,Boonstra, M., Korzhova, V., Zhang, J.: Framework for perfor-

Real-Time Tracking-with-Detection for Coping With Viewpoint Change 11

Frame 1 Frame 90 Frame 182 Frame 255

Frame 1 Frame 100 Frame 137 Frame 210

Frame 1 Frame 48 Frame 116 Frame 155

Frame 1 Frame 32 Frame 110 Frame 174

Fig. 6 Sample frames for several sequences. Tracking results for LK-DPM-CUP marked in Green. (best viewed in color)

mance evaluation of face, text, and vehicle detection and trackingin video: data, metrics, and protocol. PAMI (2009)

26. Kwon, J., Lee, K.: Visual tracking decomposition. CVPR (2010)27. Leibe, B., Schindler, K., Cornelis, N., Van-Gool, L.: Coupled ob-

ject detection and tracking from static cameras and moving vehi-cles. TPAMI (2008)

28. Leichter, I., Krupka, E.: Monotonicity and error type differentia-bility in performance measures for target detection and tracking invideo. In: CVPR (2012)

29. Lucas, B., Kanade, T.: An iterative image registration techniquewith an application to stereo vision. In: Proccedings of ImageingUnderstanding Workshop (1981)

30. Matthews, I., Baker, S.: Lucas-kanade 20 years on: A unifyingframework. IJCV (2004)

31. Matthews, I., Ishikawa, T., Baker, S.: The template update prob-lem. TPAMI (2004)

32. Oron, S., Bar-Hillel, A., Levi, D., Avidan, S.: Locally orderlesstracking. CVPR (2012)

33. Panin, G., Klose, S., Knoll, A.: Real-time articulated hand detec-tion and pose estimation. Advances in Visual Computing (2009)

34. Papanikolopoulos, N., Khosla, P., Kanade, T.: Visual tracking of amoving target by a camera mounted on a robot: a combination ofcontrol and vision. IEEE transactions on robotics and automation(1993)

35. Ross, D., Lim, J., Lin, R., Yang, M.: Incremental learning for ro-bust visual tracking. IJCV (2007)

36. Santner, J., Leistner, C., Saffari, A., Pock, T., Bischof, H.:Prost:parallel robust online simple tracking. CVPR (2010)

37. Shotton, J., Fitzgibbon, A., Cook, M., Sharp, T., Finocchio, M.,Moore, R., Kipman, A., Blake, A.: Real-time human pose recog-nition in parts from a single depth image. CVPR (2011)

38. Siebel, N., Maybank, S.: Fusion of multiple tracking algorithmsfor robust people tracking. ECCV (2002)

39. Stalder, S., Grabner, H., van Gool, L.: Beyond semi-supervisedtracking: Tracking should be as simple as detection, but not sim-pler than recognition. ICCV workshops (2009)

40. Stauffer, C., Grimson, E.: Learning patterns of activity using real-time tracking. PAMI (2000)

41. Williams, O., Blake, A., Cipolla, R.: Sparse bayesian learning forefficient visual tracking. TPAMI (2005)

42. Wu, Y., Lim, J., Yang, M.: Online object tracking: A benchmark.In: CVPR (2005)

43. Yilmaz, A., Javed, O., Shah, M.: Object tracking: A survey. ACM.Comp. Survey 38(4) (2006)

44. Zhang, K., Zhang, L., Yang, M.: Real-time compressive tracking.ECCV (2012)

45. Zhong, W., Lu, H., Yang, M.: Robust object tracking via sparsity-based collaborative model. CVPR (2012)

12 Shaul Oron et al.

46. Zhu, X., Ramanan, D.: Face detection, pose estimation, and land-mark localization in the wild. CVPR (2012)