Fusion of Multiple Cues from Color and Depth Domains using

Occlusion Aware Bayesian Tracker

Kourosh MESHGIShin-ichi MAEDAShigeyuki OBAShin ISHII

18 MAR 2014

Integrated System Biology Lab(Ishii Lab)Graduate School of InformaticsKyoto Universitymeshgi-k@sys.i.kyoto-u.ac.jp

IEICE NC Tamagawa’14

TRACKING APPLICATIONS

K O U R O S H M E S H G I – I S H I I L A B - D E C 2 0 1 3 - S L I D E 2

MAIN APPLICATIONS

Surveillance Public Entertainment

Robotics Video Indexing

Action Recog.

TRACKING CHALLENGES

K O U R O S H M E S H G I – I S H I I L A B - D E C 2 0 1 3 - S L I D E 3

MAIN CHALLENGES

Varying ScaleClutterNon-Rigid

OcclusionIlluminationAbrupt Motion

[Mihaylova et al., 07] RGB Color + Texture + Motion + Edge, Two PF

[Spinello & Arras,11] HOG on RGB + HOG on Depth, SVM Classification

[Shotton et al, 11] Skeleton from Depth, Random Forrest

[Choi et al, 11] Ensemble of Detectors (upper body, face, skin, shape from depth, motion from depth), RJ-MCMC

LITERATURE REVIW(Channel Fusion & Occlusion)

[Song et al., 13] 2.5D Shape + Motion + HOG on Color and Depth, Occlusion Indicator, SVM

K O U R O S H M E S H G I – I S H I I L A B – M A R 2 0 1 4 – S L I D E 5

OBSERVATION MODELFrame: t

Observation

, ,{ , }t rgb t d tI I I

Image Patch

K O U R O S H M E S H G I – I S H I I L A B – M A R 2 0 1 4 – S L I D E 6

STATE REPRESENTATIONFrame: t

State

{ , , , }t t t t tB x y w h{ }t tX B

( ; )t t tY g I B

w

h

(x,y)

K O U R O S H M E S H G I – I S H I I L A B – M A R 2 0 1 4 – S L I D E 7

FEATURES

Feature Set

1{ ,..., }nF f f

Color

Shape Edge

Texture

K O U R O S H M E S H G I – I S H I I L A B – M A R 2 0 1 4 – S L I D E 8

TEMPLATE INIT.Frame: 1

Template

1 1,1 ,1{ ,..., }n

f1 fj fn

1 ,1 1{ }ni i

1 1{ ( )}if Y

… …

K O U R O S H M E S H G I – I S H I I L A B – M A R 2 0 1 4 – S L I D E 9

PARTICLES INITIALIZATION

Frame: 1

Particles

, ,{ }k t k tX B1,2, ,k N

Initialized Overlapped

K O U R O S H M E S H G I – I S H I I L A B – M A R 2 0 1 4 – S L I D E 1 0

MOTION MODELFrame: t

Motion Model

, , ,k t k t k tB B

→ t + 1

K O U R O S H M E S H G I – I S H I I L A B – M A R 2 0 1 4 – S L I D E 1 1

FEATURE EXTRACTIONFrame: t + 1

Feature Vectors

, 1( )i k tf Y

f1 f2 fn

X1,t+1

X2,t+1

XN,t+1

…

…

K O U R O S H M E S H G I – I S H I I L A B - M A R 2 0 1 4 - S L I D E 1 2

FEATURE FUSIONFrame: t

Probability of Observation( | )t tp Y X ,1

( | , )n

i t t i tip Y B

( | , )t t tp Y B 1, 2, ,( | , , , , )t t t t n tp Y B 1 1, ,( | , ) ( | , )t t t n t t n tp Y B p Y B ,1( ),

n

i i i t i tip D f Y

,1exp ( ),

n

i i i t i tiD f Y

,1

exp ( ),n

i i i t i tiD f Y

Each Feature(.)if(.)iD

i Indep

enden

ce

Assum

ption

!

K O U R O S H M E S H G I – I S H I I L A B – M A R 2 0 1 4 – S L I D E 1 3

PROB. CALCULATIONFrame: t + 1

Particles

Brighter = More Probable

,k tp

K O U R O S H M E S H G I – I S H I I L A B – M A R 2 0 1 4 – S L I D E 1 4

TARGET ESTIMATIONFrame: t + 1

Expectation

1 ,ˆ

t k tB B E

K O U R O S H M E S H G I – I S H I I L A B – M A R 2 0 1 4 – S L I D E 1 5

MODEL UPDATEFrame: t + 1

New Model

Model Update

1ˆ ˆ( ; )t t tY g I B

1ˆ ˆ( )t i tf Y

, 1 , 1

,

ˆ

(1 )i t i i t

i i t

K O U R O S H M E S H G I – I S H I I L A B – M A R 2 0 1 4 – S L I D E 1 6

RESAMPLINGFrame: t + 1

Proportional to Probability

1( | )t tp X X1

2

345

6

7

K O U R O S H M E S H G I – I S H I I L A B – M A R 2 0 1 4 – S L I D E 1 7

APPEARANCE CHANGES

Same Color Objects

Background Clutter

Illumination Change

Shadows, Shades

Use Depth!

K O U R O S H M E S H G I – I S H I I L A B – M A R 2 0 1 4 – S L I D E 1 8

MODEL DRIFT PROBLEM

Templates Corrupted! t

Handle Occlusion!

PERSISTENT OCCLUSION

Particles Converge to Local Optima / Remains The Same Region

K O U R O S H M E S H G I – I S H I I L A B – M A R 2 0 1 4 – S L I D E 1 9

Advanced Motion Models

(not always feasible)

Restart Tracking

(slow occlusion recovery)

Expand Search Area!

OCCLUSIONdo not address occlusion explicitly

maintain a large set of hypotheses

computationally expensive

direct occlusion detection

robust against partial & temp occ.

persistent occ. hinder tracking

GENERATIVE MODELSDISCRIMINATIVE

MODELS

Dynamic Occlusion: Pixels of other object close to camera

Scene Occlusion: Still objects are closer to camera than the target object

Apparent Occlusion: Due to shape change, silhouette motion, shadows, self-occ

UPDATE MODEL FOR TARGET TYPE OF OCCLUSION IS IMPORTANT KEEP MEMORY VS. KEEP FOCUS ON THE TARGET

Combin

e

Them!

DYNAMICS…

* The Search is not Directed

* Neither of the Channels have Useful Information

* Particles Should Scatter Away from Last Known Position

K O U R O S H M E S H G I – I S H I I L A B – M A R 2 0 1 4 – S L I D E 2 1

Occlusion!

Occlusion Flag (for each particle)

Observation Model

No-Occlusion Particles Same as Before

Occlusion-Flagged Particles Uniform Distribution

OCCLUSION AWAREPARTICLE FILTER FRAMEWORK

( | ) ( | , , )t t t t t tp Y X p Y B Z ( | ) (1 ) ( | , 0, ) ( | , 1, )t t t t t t t t t t t tp Y X Z p Y B Z Z p Y B Z

,k tZ

( | , 1, ) 1t t t tp Y B Z

,1( | , 0, ) exp ( ),

n

t t t t i i i t i tip Y B Z D f Y

K O U R O S H M E S H G I – I S H I I L A B – M A R 2 0 1 4 – S L I D E 2 2

Probability of Occlusion for the Next Box

Modified Dynamics Model of Particle

PARTICLE FILTER DYNAMICS

1 , , , , ,1 1ˆ ( ) ( ) ( | )

N N

t t i t i t i t i t i ti iZ Z p Z Z p X Y Z

E

1 1 1 1 1( | ) ( , | , ) ( | ) ( | )t t t t t t t t t tp X X p B Z B Z p B B p Z Z

K O U R O S H M E S H G I – I S H I I L A B – M A R 2 0 1 4 – S L I D E 2 3

Model Update

Separately for each Feature

UPDATE RULE

11

1 1

ˆ( ) ,( )

ˆ ˆ( ) (1 ) ( ) ,

t t occt

t t t occ

f Zf

f Y f Z

K O U R O S H M E S H G I – I S H I I L A B – M A R 2 0 1 4 – S L I D E 2 4

OA-PF DYNAMICS

K O U R O S H M E S H G I – I S H I I L A B – M A R 2 0 1 4 – S L I D E 2 5

Occlusion!

OA-PF DYNAMICS

K O U R O S H M E S H G I – I S H I I L A B – M A R 2 0 1 4 – S L I D E 2 6

Occlusion!

GOTCHA!

OA-PF DYNAMICS

K O U R O S H M E S H G I – I S H I I L A B – M A R 2 0 1 4 – S L I D E 2 7

Quick Occlusion Recovery

Low CPE

No Template Corruption

No Attraction to other Object/ Background

CO

LO

R

(HO

C)

TE

XT

UR

E

(LB

P)

ED

GE

(L

OG

)

DE

PT

H

(HO

D)

3D S

HA

PE

(P

CL

Σ)

FEATURES

K O U R O S H M E S H G I – I S H I I L A B – M A R 2 0 1 4 – S L I D E 2 8

Princeton Tracking Dataset

DATASET( )

5 Validation Video with Ground Truth95 Evaluation Video

K O U R O S H M E S H G I – I S H I I L A B – M A R 2 0 1 4 – S L I D E 2 9

EXPERIMENT

K O U R O S H M E S H G I – I S H I I L A B – M A R 2 0 1 4 – S L I D E 3 0

A. Edge PF

B. Edge + Color PF

C. Edge + Color + Depth PF

D. Edge + Color + Depth + Texture PF

E. Edge + Color + Depth + Texture + 3D Shape PF

F. Occlusion Aware PF

DEMONSTRATION(Yellow Dashed Line is Ground Truth)

PASCAL VOC

CRITERIA I

1

1

*1

* *1 1 1

*1 1

*1 1

ˆ

ˆ ˆ, 0

ˆ1 , 1

ˆ1 ,

t

t

t

t t t

t t t

t t

B B

B B Z Z

S Z Z

Z Z

0 1ott oS t AUC

toS

ucc

ess

Overlap Threshold

0

1

1

Area Under Curve

K O U R O S H M E S H G I – I S H I I L A B – M A R 2 0 1 4 – S L I D E 3 2

RESULTS

K O U R O S H M E S H G I – I S H I I L A B - M A R 2 0 1 4 - S L I D E 3 3

Success Plot A D

B EC F

1

1Overlap Threshold

Succ

ess

Rat

e

Mean Central Point Error: Localization Success

Mean Scale Adaption Error

CRITERIA II

* 2 * 2

1ˆˆ( ) ( )

T

t t t ttw w h h

SAET

* 2 * 2

1ˆ ˆ( ) ( )

T

t t t ttx x y y

CPET

K O U R O S H M E S H G I – I S H I I L A B – M A R 2 0 1 4 – S L I D E 3 4

ˆˆ ˆ ˆ ˆ{ , , , }t t t t tB x y w h * * * * *{ , , , }t t t t t

B x y w h

Estimated Ground Truth

RESULTS

K O U R O S H M E S H G I – I S H I I L A B - M A R 2 0 1 4 - S L I D E 3 5

Center Positioning ErrorA D

B EC F

100

50Frames

CP

E (

pixe

ls)

RESULTS

K O U R O S H M E S H G I – I S H I I L A B - M A R 2 0 1 4 - S L I D E 3 6

Scale Adaptation Error

K O U R O S H M E S H G I – I S H I I L A B - M A R 2 0 1 4 - S L I D E 3 6

140

SAE

(pi

xels

)

50Frames

A D

B EC F

RESULTS

K O U R O S H M E S H G I – I S H I I L A B – M A R 2 0 1 4 – S L I D E 3 7

Tracker AUC CPE SAE

A (edg)15.7

2192.27

41.62

B (edg+hoc)29.8

893.4

746.7

1

C (edg+hoc+hod)46.7

434.6

240.8

8

D (edg+hoc+hod+tex)

48.49

30.18

46.27

E (edg+hoc+hod+tex+shp)

58.03

23.84

29.62

F (all + occlusion handling)

63.58

17.46

25.07

FUTURE WORKS

K O U R O S H M E S H G I – I S H I I L A B – M A R 2 0 1 4 – S L I D E 3 8

More Resilient Features + Scale

Adaptation

Active Occlusion Handling

Measure the Confidence of

each Data Channel

Adaptive Model Update

QUESTIONS?Thank you for your time…

Image Credit: http://www.engg.uaeu.ac.ae/