Rule-based Action Recognition using Object Trajectories

UCF VIRAT Efforts

Recognition Process Input: • Time ordered series of 2 dimensional locations

of object centroid in image coordinates (trajectory)

Output: • Action(s) pertaining to the given trajectory

Method:• Compute multiple discriminative features for

each trajectory• Classify action(s) using a comprehensive set of

rules

Trajectory Features Dynamics based

• For a curve,r (t) = {(x0,y0),…,(xt,yt)}

• Instantaneous speed, v = || dr / dt ||

• Average speed,vav = ∑n vi

/ n• Acceleration,

a = || dv / dt ||• Arc length,

s = ∫ v dt

Trajectory Features Shape based• Capture geometrical information

• Tangent vector, v = dr / dt

• Unit Tangent vector, T = v / ||v||• Curvature, k(t) = || dT / ds || = ||T’ / v||• Four point cross ratio,

Cr (p1,p2,p3,p4) = (p3-p1)(p4-p2) / (p4-p1)(p3-p2)

Rules Rules are based on quantization of feature values

A decrease followed by an increase in unit tangent vector = Right turn, or vice versa

Two consecutive increase or decrease = U-turn

Rules

Going Straight Turn Right U-turn

Trajectory

Trajectory

Trajectory

dT / dt dT / dt dT / dt

Rules Accelerate / Decelerate events are detected directly from features

Deceleration to zero speed = Stopping

Maintaining speed and direction = Maintain distance

….

Results Trajectories extracted from the UCF Aerial Actions dataset

Handle walking, running, turning left and right, and taking U-turn

Results

Walking Forward

Results

Turn Right

Results

U-turn

Results

U-turn

Results

Turn Right

Results

Turn Left

Results

Running Forward

VIRAT Events

Standing Walking Running Digging Gesturing Carry Object

Person Actions Accelerate

Decelerate Turning Stopping U-turn Maintain distance

Vehicle Actions

Ideas & Future Work Object Classification

• Discriminate between similar trajectories of different objects• Separate person, vehicle and facility detectors• Person-Vehicle and Person-Facility events

Multiple Trajectories per object

• Track multiple points on each object• Recognize stationary trajectories using object

kinematics• Gesturing, Digging• Can also help or eliminate object detector /

classifier

Ideas & Future Work Geo-Registration

• Distance, speed, and acceleration in image plane may not be useful

• Allow use of computed distances on ground• Do not have to be absolute longitude and latitude• Accelerate, Decelerate, Maintain Distance

Invariant features

• Trajectory features should be invariant to: Changes in view Changes in scale

• Projective invariant features can eliminate need for geo-registration

Ideas & Future Work Learning based framework

• Learning and clustering of discriminative features• More robust compared to rule based methods• Representative of training trajectory samples• Associated confidence for each recognition• Recognition of unseen and composite events

Quality of Input Tracks

• Availability of object trajectories is assumed• Features are highly dependant on tracking accuracy• Broken, merged and split tracks severely affect

performance

VIRAT Events

Standing Walking Running Digging Gesturing Carry Object

Person Actions Accelerate

Decelerate Turning Stopping U-turn Maintain distance

Vehicle Actions Loading Unload Open trunk Close trunk Getting into car Getting out of car Enter/exit building

Multi-agent Actions

• With provision of object recognition / classification, multiple trajectories and geo-registration

Thank You!