Introduction Tracking with static camera Tracking with moving camera

Tracking

Hakan Ardo

March 4, 2013

Hakan Ardo Tracking March 4, 2013 1 / 57

Introduction Tracking with static camera Tracking with moving cameraState space Sliding Window Detection

Outline

1 IntroductionState spaceSliding Window Detection

2 Tracking with static cameraGreedyKalman filterParticle filter

3 Tracking with moving cameraSelf-motion

Hakan Ardo Tracking March 4, 2013 2 / 57

Introduction Tracking with static camera Tracking with moving cameraState space Sliding Window Detection

State space

State: position X = (X ,Y ,Z , 1), velocity, v = (vx , vy , vz , 0), ...

Observation: detection in image x = (x , y , 1)

Observation model: λx = PX

Dynamic model: Xt+1 = Xt + vt and vt+1 = vt

Sthocastic dynamic model: Introduce noise, random numbers q and w

Xt+1 = Xt + vt + q

vt+1 = vt + w

Sthocastic observation model: Introduce noise, a random number r

λx = PX + r

Hakan Ardo Tracking March 4, 2013 3 / 57

Introduction Tracking with static camera Tracking with moving cameraState space Sliding Window Detection

State space

State: position X = (X ,Y ,Z , 1), velocity, v = (vx , vy , vz , 0), ...

Observation: detection in image x = (x , y , 1)

Observation model: λx = PX

Dynamic model: Xt+1 = Xt + vt and vt+1 = vt

Sthocastic dynamic model: Introduce noise, random numbers q and w

Xt+1 = Xt + vt + q

vt+1 = vt + w

Sthocastic observation model: Introduce noise, a random number r

λx = PX + r

Hakan Ardo Tracking March 4, 2013 3 / 57

Introduction Tracking with static camera Tracking with moving cameraState space Sliding Window Detection

State space

State: position X = (X ,Y ,Z , 1), velocity, v = (vx , vy , vz , 0), ...

Observation: detection in image x = (x , y , 1)

Observation model: λx = PX

Dynamic model: Xt+1 = Xt + vt and vt+1 = vt

Sthocastic dynamic model: Introduce noise, random numbers q and w

Xt+1 = Xt + vt + q

vt+1 = vt + w

Sthocastic observation model: Introduce noise, a random number r

λx = PX + r

Hakan Ardo Tracking March 4, 2013 3 / 57

Introduction Tracking with static camera Tracking with moving cameraState space Sliding Window Detection

State space

State: position X = (X ,Y ,Z , 1), velocity, v = (vx , vy , vz , 0), ...

Observation: detection in image x = (x , y , 1)

Observation model: λx = PX

Dynamic model: Xt+1 = Xt + vt and vt+1 = vt

Sthocastic dynamic model: Introduce noise, random numbers q and w

Xt+1 = Xt + vt + q

vt+1 = vt + w

Sthocastic observation model: Introduce noise, a random number r

λx = PX + r

Hakan Ardo Tracking March 4, 2013 3 / 57

Introduction Tracking with static camera Tracking with moving cameraState space Sliding Window Detection

State space

State: position X = (X ,Y ,Z , 1), velocity, v = (vx , vy , vz , 0), ...

Observation: detection in image x = (x , y , 1)

Observation model: λx = PX

Dynamic model: Xt+1 = Xt + vt and vt+1 = vt

Sthocastic dynamic model: Introduce noise, random numbers q and w

Xt+1 = Xt + vt + q

vt+1 = vt + w

Sthocastic observation model: Introduce noise, a random number r

λx = PX + r

Hakan Ardo Tracking March 4, 2013 3 / 57

Introduction Tracking with static camera Tracking with moving cameraState space Sliding Window Detection

State space

State: position X = (X ,Y ,Z , 1), velocity, v = (vx , vy , vz , 0), ...

Observation: detection in image x = (x , y , 1)

Observation model: λx = PX

Dynamic model: Xt+1 = Xt + vt and vt+1 = vt

Sthocastic dynamic model: Introduce noise, random numbers q and w

Xt+1 = Xt + vt + q

vt+1 = vt + w

Sthocastic observation model: Introduce noise, a random number r

λx = PX + r

Hakan Ardo Tracking March 4, 2013 3 / 57

Introduction Tracking with static camera Tracking with moving cameraState space Sliding Window Detection

Sliding window detectors

Hakan Ardo Tracking March 4, 2013 4 / 57

Introduction Tracking with static camera Tracking with moving cameraState space Sliding Window Detection

Detection probability

Hakan Ardo Tracking March 4, 2013 5 / 57

Introduction Tracking with static camera Tracking with moving cameraGreedy Kalman filter Particle filter

Greedy tracker

Hakan Ardo Tracking March 4, 2013 6 / 57

Introduction Tracking with static camera Tracking with moving cameraGreedy Kalman filter Particle filter

UNC Chapel Hill Computer Science Slide 1

STC Lecture Series

An Introduction to the Kalman FilterGreg Welch and Gary Bishop

University of North Carolina at Chapel Hill

Department of Computer Science

http://www.cs.unc.edu/~welch/kalmanLinks.html

Hakan Ardo Tracking March 4, 2013 7 / 57

Introduction Tracking with static camera Tracking with moving cameraGreedy Kalman filter Particle filter

Sanjay Patil1 and Ryan Irwin2

Graduate research assistant1,REU undergrad2

Human and Systems Engineering

URL: www.isip.msstate.edu/publications/seminars/msstate/2005/particle/

HUMAN AND SYSTEMS ENGINEERING:Gentle Introduction to Particle Filtering

Hakan Ardo Tracking March 4, 2013 8 / 57

Introduction Tracking with static camera Tracking with moving cameraGreedy Kalman filter Particle filter

UNC Chapel Hill Computer Science Slide 5

Some Intuition

Hakan Ardo Tracking March 4, 2013 9 / 57

Introduction Tracking with static camera Tracking with moving cameraGreedy Kalman filter Particle filter

UNC Chapel Hill Computer Science Slide 6

First Estimate

ˆ x 1 = z1

ˆ σ 21 = σ 2z1

Conditional Density Function

14121086420-2

N(z1,σz12 )

z1 σ 2z1

,

Hakan Ardo Tracking March 4, 2013 10 / 57

Introduction Tracking with static camera Tracking with moving cameraGreedy Kalman filter Particle filter

UNC Chapel Hill Computer Science Slide 7

Second Estimate

Conditional Density Function

z2 σ 2z2

,

ˆ x 2 = ...?

ˆ σ 22 = ...? 14121086420-2

N(z2,σz22 )

Hakan Ardo Tracking March 4, 2013 11 / 57

Introduction Tracking with static camera Tracking with moving cameraGreedy Kalman filter Particle filter

UNC Chapel Hill Computer Science Slide 8

Combine Estimates

= σ z2

2 σ z1

2 +σ z2

2( )[ ] z1+ σ z1

2 σ z1

2 +σ z2

2( )[ ] z2ˆ x 2= ˆ x 1 + K2 z2 − ˆ x 1[ ]

whereK2 = σ z1

2 σ z1

2 +σ z2

2( )

Hakan Ardo Tracking March 4, 2013 12 / 57

Introduction Tracking with static camera Tracking with moving cameraGreedy Kalman filter Particle filter

UNC Chapel Hill Computer Science Slide 9

Combine Variances

1 σ 2 = 1 σ z1

2( )+ 1 σ z2

2( )2

Hakan Ardo Tracking March 4, 2013 13 / 57

Introduction Tracking with static camera Tracking with moving cameraGreedy Kalman filter Particle filter

UNC Chapel Hill Computer Science Slide 10

Combined Estimate Density

ˆ x =

ˆ σ 2 = σ 22

ˆ x 2

14121086420-2

Conditional Density Function

N( σ 2)ˆ x,ˆ

Hakan Ardo Tracking March 4, 2013 14 / 57

Introduction Tracking with static camera Tracking with moving cameraGreedy Kalman filter Particle filter

UNC Chapel Hill Computer Science Slide 11

Add Dynamics

dx/dt = v + w

wherev is the nominal velocityw is a noise term (uncertainty)

Hakan Ardo Tracking March 4, 2013 15 / 57

Introduction Tracking with static camera Tracking with moving cameraGreedy Kalman filter Particle filter

Page 7 of 20Particle Filtering – Gentle Introduction and Implementation Demo

Particle filtering algorithm step-by-step (1)

Hakan Ardo Tracking March 4, 2013 16 / 57

Introduction Tracking with static camera Tracking with moving cameraGreedy Kalman filter Particle filter

Page 8 of 20Particle Filtering – Gentle Introduction and Implementation Demo

Particle filtering step-by-step (2)

Hakan Ardo Tracking March 4, 2013 17 / 57

Introduction Tracking with static camera Tracking with moving cameraGreedy Kalman filter Particle filter

Page 9 of 20Particle Filtering – Gentle Introduction and Implementation Demo

Particle filtering step-by-step (3)

Hakan Ardo Tracking March 4, 2013 18 / 57

Introduction Tracking with static camera Tracking with moving cameraGreedy Kalman filter Particle filter

Page 10 of 20Particle Filtering – Gentle Introduction and Implementation Demo

Particle filtering step-by-step (4)

Hakan Ardo Tracking March 4, 2013 19 / 57

Introduction Tracking with static camera Tracking with moving cameraGreedy Kalman filter Particle filter

Page 11 of 20Particle Filtering – Gentle Introduction and Implementation Demo

Particle filtering step-by-step (5)

Hakan Ardo Tracking March 4, 2013 20 / 57

Introduction Tracking with static camera Tracking with moving cameraGreedy Kalman filter Particle filter

Page 12 of 20Particle Filtering – Gentle Introduction and Implementation Demo

Particle filtering step-by-step (6)

Hakan Ardo Tracking March 4, 2013 21 / 57

Introduction Tracking with static camera Tracking with moving cameraGreedy Kalman filter Particle filter

UNC Chapel Hill Computer Science Slide 13

Some Details

xx AAxx wwzz HH xx........

= +=

Hakan Ardo Tracking March 4, 2013 22 / 57

Introduction Tracking with static camera Tracking with moving cameraGreedy Kalman filter Particle filter

UNC Chapel Hill Computer Science Slide 14

Discrete Kalman Filter

Maintains first two statistical moments

process state (mean)

error covariance

z

y

x

Hakan Ardo Tracking March 4, 2013 23 / 57

Introduction Tracking with static camera Tracking with moving cameraGreedy Kalman filter Particle filter

UNC Chapel Hill Computer Science Slide 16

Necessary Models

measurementmodel

dynamicmodel

previous state next state

statemeasurement

image plane( u , v )

Hakan Ardo Tracking March 4, 2013 24 / 57

Introduction Tracking with static camera Tracking with moving cameraGreedy Kalman filter Particle filter

UNC Chapel Hill Computer Science Slide 17

The Process Model

x k+1 = Axk + wk

zk = Hxk + vk

Process Dynamics

Measurement

Hakan Ardo Tracking March 4, 2013 25 / 57

Introduction Tracking with static camera Tracking with moving cameraGreedy Kalman filter Particle filter

UNC Chapel Hill Computer Science Slide 18

Process Dynamics

x k+1 = Axk + wk

xk ∈ Rn contains the states of the process

state vector

Hakan Ardo Tracking March 4, 2013 26 / 57

Introduction Tracking with static camera Tracking with moving cameraGreedy Kalman filter Particle filter

UNC Chapel Hill Computer Science Slide 19

Process Dynamics

nxn matrix A relates state at time step k to time step k+1

state transition matrix

x k+1 = Axk + wk

Hakan Ardo Tracking March 4, 2013 27 / 57

Introduction Tracking with static camera Tracking with moving cameraGreedy Kalman filter Particle filter

UNC Chapel Hill Computer Science Slide 20

Process Dynamics

process noise

wk ∈ Rn models the uncertainty of the process

wk ~ N(0, Q)

x k+1 = Axk + wk

Hakan Ardo Tracking March 4, 2013 28 / 57

Introduction Tracking with static camera Tracking with moving cameraGreedy Kalman filter Particle filter

UNC Chapel Hill Computer Science Slide 21

Measurement

zk = Hxk + vk

zk ∈ Rm is the process measurement

measurement vector

Hakan Ardo Tracking March 4, 2013 29 / 57

Introduction Tracking with static camera Tracking with moving cameraGreedy Kalman filter Particle filter

UNC Chapel Hill Computer Science Slide 22

Measurement

zk = Hxk + vk

state vector

Hakan Ardo Tracking March 4, 2013 30 / 57

Introduction Tracking with static camera Tracking with moving cameraGreedy Kalman filter Particle filter

UNC Chapel Hill Computer Science Slide 23

Measurement

mxn matrix H relates state to measurement

measurement matrix

zk = Hxk + vk

Hakan Ardo Tracking March 4, 2013 31 / 57

Introduction Tracking with static camera Tracking with moving cameraGreedy Kalman filter Particle filter

UNC Chapel Hill Computer Science Slide 24

Measurement

measurement noise

zk ∈ Rm models the noise in the measurement

vk ~ N(0, R)

zk = Hxk + vk

Hakan Ardo Tracking March 4, 2013 32 / 57

Introduction Tracking with static camera Tracking with moving cameraGreedy Kalman filter Particle filter

UNC Chapel Hill Computer Science Slide 25

State Estimates

a priori state estimate

a posteriori state estimate

ˆ x –k

ˆ xk

Hakan Ardo Tracking March 4, 2013 33 / 57

Introduction Tracking with static camera Tracking with moving cameraGreedy Kalman filter Particle filter

UNC Chapel Hill Computer Science Slide 26

Estimate Covariances

a priori estimate error covariance

a posteriori estimate error covariance

Pk– = E[(xk- xk

–)(xk - xk–)T] ˆ ˆ

Pk = E[(xk- xk)(xk - xk)T] ˆ ˆ

Hakan Ardo Tracking March 4, 2013 34 / 57

Introduction Tracking with static camera Tracking with moving cameraGreedy Kalman filter Particle filter

UNC Chapel Hill Computer Science Slide 27

Filter Operation

Time update (a priori estimates)

Measurement update (a posteriori estimates)

Project state and covariance forwardto next time step, i.e. predict

Update with a (noisy) measurementof the process, i.e. correct

Hakan Ardo Tracking March 4, 2013 35 / 57

Introduction Tracking with static camera Tracking with moving cameraGreedy Kalman filter Particle filter

UNC Chapel Hill Computer Science Slide 28

Time Update (Predict)

state

error covariance

Hakan Ardo Tracking March 4, 2013 36 / 57

Introduction Tracking with static camera Tracking with moving cameraGreedy Kalman filter Particle filter

UNC Chapel Hill Computer Science Slide 29

Measurement Update (Correct)

Hakan Ardo Tracking March 4, 2013 37 / 57

Introduction Tracking with static camera Tracking with moving cameraGreedy Kalman filter Particle filter

UNC Chapel Hill Computer Science Slide 30

Time Update (Predict)

a priori state and error covariance

ˆ xk+1 = Axk–

Pk+1 = APk A + Q–

Hakan Ardo Tracking March 4, 2013 38 / 57

Introduction Tracking with static camera Tracking with moving cameraGreedy Kalman filter Particle filter

UNC Chapel Hill Computer Science Slide 31

Measurement Update (Correct)

a posteriori state and error covariance

Kalman gain

ˆ xk = xk + Kk (zk - Hxk )ˆ ˆ – –

Pk = (I - Kk H)Pk–

Kk = Pk HT(HPk HT + R)-1 – –

Hakan Ardo Tracking March 4, 2013 39 / 57

Introduction Tracking with static camera Tracking with moving cameraGreedy Kalman filter Particle filter

UNC Chapel Hill Computer Science Slide 32

Filter Operation

Time Update(Predict)

Measurement

(Correct)Update

Hakan Ardo Tracking March 4, 2013 40 / 57

Introduction Tracking with static camera Tracking with moving cameraGreedy Kalman filter Particle filter

Kalman filter tracker

Hakan Ardo Tracking March 4, 2013 41 / 57

Introduction Tracking with static camera Tracking with moving cameraGreedy Kalman filter Particle filter

Kalman filter tracker

Hakan Ardo Tracking March 4, 2013 42 / 57

Introduction Tracking with static camera Tracking with moving cameraGreedy Kalman filter Particle filter

Kalman filter tracker

Hakan Ardo Tracking March 4, 2013 43 / 57

Introduction Tracking with static camera Tracking with moving cameraGreedy Kalman filter Particle filter

Kalman filter tracker

Hakan Ardo Tracking March 4, 2013 44 / 57

Introduction Tracking with static camera Tracking with moving cameraGreedy Kalman filter Particle filter

Particle filter

Hakan Ardo Tracking March 4, 2013 45 / 57

Introduction Tracking with static camera Tracking with moving cameraGreedy Kalman filter Particle filter

Patricle filter states

Current state {x

(1)k , x

(2)k , x

(3)k , · · · , x (n)

k

}Predicted state (a priori state extimate){

x−(1)k+1 , x

−(2)k+1 , x

−(3)k+1 , · · · , x

−(n)k+1

}Corrected state (a posteriori state extimate){

x(1)k+1, x

(2)k+1, x

(3)k+1, · · · , x

(n)k+1

}

Hakan Ardo Tracking March 4, 2013 46 / 57

Introduction Tracking with static camera Tracking with moving cameraGreedy Kalman filter Particle filter

Patricle filter states

Current state {x

(1)k , x

(2)k , x

(3)k , · · · , x (n)

k

}Predicted state (a priori state extimate){

x−(1)k+1 , x

−(2)k+1 , x

−(3)k+1 , · · · , x

−(n)k+1

}Corrected state (a posteriori state extimate){

x(1)k+1, x

(2)k+1, x

(3)k+1, · · · , x

(n)k+1

}

Hakan Ardo Tracking March 4, 2013 46 / 57

Introduction Tracking with static camera Tracking with moving cameraGreedy Kalman filter Particle filter

Patricle filter states

Current state {x

(1)k , x

(2)k , x

(3)k , · · · , x (n)

k

}Predicted state (a priori state extimate){

x−(1)k+1 , x

−(2)k+1 , x

−(3)k+1 , · · · , x

−(n)k+1

}Corrected state (a posteriori state extimate){

x(1)k+1, x

(2)k+1, x

(3)k+1, · · · , x

(n)k+1

}

Hakan Ardo Tracking March 4, 2013 46 / 57

Introduction Tracking with static camera Tracking with moving cameraGreedy Kalman filter Particle filter

Dynamic model

Any sthocastic model from which we can sample

f (xk+1 |xk )

Example: The dynamic model from the kalman filter

xk+1 = Axk + wk

f (xk+1 |xk ) = N(Axk ,Q)

Hakan Ardo Tracking March 4, 2013 47 / 57

Introduction Tracking with static camera Tracking with moving cameraGreedy Kalman filter Particle filter

Dynamic model

Any sthocastic model from which we can sample

f (xk+1 |xk )

Example: The dynamic model from the kalman filter

xk+1 = Axk + wk

f (xk+1 |xk ) = N(Axk ,Q)

Hakan Ardo Tracking March 4, 2013 47 / 57

Introduction Tracking with static camera Tracking with moving cameraGreedy Kalman filter Particle filter

Particle filter prediction step

Propagate each particle i , separately

The prediction x−(i)k+1 is choosen as a sample from

f(xk+1

∣∣∣x (i)k

)

Hakan Ardo Tracking March 4, 2013 48 / 57

Introduction Tracking with static camera Tracking with moving cameraGreedy Kalman filter Particle filter

Measurement model

Any sthocastic model from which we can calculate likelihoods

f (zk |xk )

Example: The measurement model from the kalman filter

zk = Hxk + vk

f (zk |xk ) = N(Hxk ,R)

Hakan Ardo Tracking March 4, 2013 49 / 57

Introduction Tracking with static camera Tracking with moving cameraGreedy Kalman filter Particle filter

Measurement model

Any sthocastic model from which we can calculate likelihoods

f (zk |xk )

Example: The measurement model from the kalman filter

zk = Hxk + vk

f (zk |xk ) = N(Hxk ,R)

Hakan Ardo Tracking March 4, 2013 49 / 57

Introduction Tracking with static camera Tracking with moving cameraGreedy Kalman filter Particle filter

Particle filter correction step

The measurement model gives a weight for each particle, i

w(i)k = f

(zk

∣∣∣x−(i)k

)Resample the set of particles using the weights

Each corrected particle, x(i)k , is choosen randomly

x(i)k = x

−(j)k for some random j

The probability of choosing sample j is

w(j)k∑

l w(l)k

The same particle may be choosen several times

Hakan Ardo Tracking March 4, 2013 50 / 57

Introduction Tracking with static camera Tracking with moving cameraGreedy Kalman filter Particle filter

Particle filter correction step

The measurement model gives a weight for each particle, i

w(i)k = f

(zk

∣∣∣x−(i)k

)Resample the set of particles using the weights

Each corrected particle, x(i)k , is choosen randomly

x(i)k = x

−(j)k for some random j

The probability of choosing sample j is

w(j)k∑

l w(l)k

The same particle may be choosen several times

Hakan Ardo Tracking March 4, 2013 50 / 57

Introduction Tracking with static camera Tracking with moving cameraGreedy Kalman filter Particle filter

Particle filter correction step

The measurement model gives a weight for each particle, i

w(i)k = f

(zk

∣∣∣x−(i)k

)Resample the set of particles using the weights

Each corrected particle, x(i)k , is choosen randomly

x(i)k = x

−(j)k for some random j

The probability of choosing sample j is

w(j)k∑

l w(l)k

The same particle may be choosen several times

Hakan Ardo Tracking March 4, 2013 50 / 57

Introduction Tracking with static camera Tracking with moving cameraGreedy Kalman filter Particle filter

Particle filter correction step

The measurement model gives a weight for each particle, i

w(i)k = f

(zk

∣∣∣x−(i)k

)Resample the set of particles using the weights

Each corrected particle, x(i)k , is choosen randomly

x(i)k = x

−(j)k for some random j

The probability of choosing sample j is

w(j)k∑

l w(l)k

The same particle may be choosen several times

Hakan Ardo Tracking March 4, 2013 50 / 57

Introduction Tracking with static camera Tracking with moving cameraGreedy Kalman filter Particle filter

Particle filter correction step

The measurement model gives a weight for each particle, i

w(i)k = f

(zk

∣∣∣x−(i)k

)Resample the set of particles using the weights

Each corrected particle, x(i)k , is choosen randomly

x(i)k = x

−(j)k for some random j

The probability of choosing sample j is

w(j)k∑

l w(l)k

The same particle may be choosen several times

Hakan Ardo Tracking March 4, 2013 50 / 57

Introduction Tracking with static camera Tracking with moving cameraGreedy Kalman filter Particle filter

Particle filter

Hakan Ardo Tracking March 4, 2013 51 / 57

Introduction Tracking with static camera Tracking with moving cameraGreedy Kalman filter Particle filter

Particle filter tracker

Hakan Ardo Tracking March 4, 2013 52 / 57

Introduction Tracking with static camera Tracking with moving cameraSelf-motion

Recording sport events

Hakan Ardo Tracking March 4, 2013 53 / 57

Introduction Tracking with static camera Tracking with moving cameraSelf-motion

Self-motion

Detect keypoints

Track each keypoint separately

Use RANSAC to find camera motion

Compensate for camera motion (stiching)

Hakan Ardo Tracking March 4, 2013 54 / 57

Introduction Tracking with static camera Tracking with moving cameraSelf-motion

Self-motion

Hakan Ardo Tracking March 4, 2013 55 / 57

Introduction Tracking with static camera Tracking with moving cameraSelf-motion

Self-motion

Hakan Ardo Tracking March 4, 2013 56 / 57

Introduction Tracking with static camera Tracking with moving cameraSelf-motion

Master Thesis: Automatisk inspelning av sport

Detta projekt syftar till att utveckla mjukvara som automatiskt kan filma,folja och mata tider for ett idrottsevenemang, till exempel ett 100m-lopp ifriidrott. For detta kravs dels kunskaper i matematisk bildanalys, men endel kunskaper i programmering kommer aven att behovas. Vi far hjalp avIFK Lund att spela in film och utvardera resultatet.

Ett examensarbete syftar till att utveckla algoritmer for automatiskfoljning av lopp. Fran borjan ar kameran stilla och efter starten skallden hela tiden folja loparna genom att rotera och zooma. Algortimenmaste kunna kompensera och den egna kamerarorelsen.Ytterligare ett examensarbete behandlar automatisk detektion avloparbanan. Genom kamerakalibrering och banans kanda matt kanloparnas hastighet beraknas och diagram over hur hastighetenforandrats genom loppet presenteras for publik och tranare. Dennainformation efterfragas av aktiva lopare. Automatisk detektion avmallinje ar aven viktigt for tidtagningen ovan.

Contact: Petter Strandmark <petter@maths.lth.se>Hakan Ardo Tracking March 4, 2013 57 / 57