Computer Vision GroupUniversity of California Berkeley

Estimating Human Body Configurations using Shape Context Matching

Greg Mori and Jitendra Malik

Computer Vision GroupUniversity of California Berkeley

Problem

Computer Vision GroupUniversity of California Berkeley

Approach: Exemplar-based Matching

• Set of stored exemplars with hand-labeled keypoints

• Obtain sample points

• Deformable matching to exemplars:– Shape context matching to get correspondences– Kinematic chain deformation model

• Estimate 3D body configuration

Computer Vision GroupUniversity of California Berkeley

Comparing Pointsets

Computer Vision GroupUniversity of California Berkeley

Shape ContextCount the number of points inside each bin, e.g.:

Count = 4

Count = 10

...

Compact representation of distribution of points relative to each point

Computer Vision GroupUniversity of California Berkeley

Shape Context

Computer Vision GroupUniversity of California Berkeley

Comparing Shape ContextsCompute matching costs using Chi Squared distance:

Recover correspondences by solving linear assignment problem with costs Cij

[Jonker & Volgenant 1987]

Computer Vision GroupUniversity of California Berkeley

Deformable Matching

• Kinematic chain-based deformation model

• Use iterations of correspondence and deformation

• Keypoints on exemplars are deformed to locations on query image

Computer Vision GroupUniversity of California Berkeley

Computer Vision GroupUniversity of California Berkeley

Problem

Computer Vision GroupUniversity of California Berkeley

Estimate 3D Body Configuration [Taylor ’00]

• Known:– Relative lengths of body segments– (x,y) Image locations of keypoints– “closer endpoint” labels for each segment– Scaled orthographic camera model

• Solve for 3D locations of keypoints up to some scale factor– Scale factor can be estimated automatically

Computer Vision GroupUniversity of California Berkeley

Solving for Foreshortening

)()()( 2121212222 ZZYYXXl

2221

221

2 /))()(( suuuuldZ

Computer Vision GroupUniversity of California Berkeley

Choosing Scale

))()(( 221

221 vvuu

ls

Computer Vision GroupUniversity of California Berkeley

Results

Computer Vision GroupUniversity of California Berkeley

Computer Vision GroupUniversity of California Berkeley

Multiple Exemplars

• Parts-based approach– Use a combination of keypoints/whole limbs from different

exemplars– Reduces the number of exemplars needed

• Compute a matching cost for each limb from every exemplar

• Compute pairwise “consistency” costs for neighbouring limbs

• Use dynamic programming to find best K configurations

Computer Vision GroupUniversity of California Berkeley

Parts-based Approach

Computer Vision GroupUniversity of California Berkeley

Tracking by Repeated Finding