Detecting Actions, Poses, and Objects with Relational ...
Transcript of Detecting Actions, Poses, and Objects with Relational ...
Detecting Actions, Poses, and Objects with Relational Phraselets
by Chaitanya Desai and Deva Ramanan
Presented by: Antonia CreswellDetecting Actions, Poses, and Objects with Relational Phraselets Chaitanya Desai and Deva Ramanan
Wednesday, 5 November 14
Problem
• Humans interact with objects in a variety of ways
• Interaction with objects leads to occlusions
• May be many people in one image
Wednesday, 5 November 14
Interact in different ways:
Deva Ramanan, University of California, Irvine
Wednesday, 5 November 14
Interaction lead to occlusions
Deva Ramanan, University of California, Irvine
Wednesday, 5 November 14
Many people in one image
Deva Ramanan, University of California, Irvine
Wednesday, 5 November 14
Motivation
• Articulated Skeletons
• Visual Phrases
• Poselets
http://www.urbiforge.org/index.php/Modules/UKinect2
Poselets and Their Applications in High-Level Computer Vision
Recognition using Visual Phrases Ali Farhadi, Mohammad Amin Sadeghi
Wednesday, 5 November 14
Key Contributions/ Technical Ideas
• Identify phraselets
• Create a model as a composite of phraselets
• Apply relational constraints between phraselets
Detecting Actions, Poses, and Objects with Relational Phraselets Chaitanya Desai and Deva RamananWednesday, 5 November 14
Identify PhraseletsPosition of part
Occluded or not?
Phraselet Label
Feature for part i in image n:
Cluster these to get the phraselets labelsKey Point: Occluded and non-Occluded parts are clustered separately: They have their own
set of labels!Detecting Actions, Poses, and Objects with Relational Phraselets Chaitanya Desai and Deva Ramanan Deva Ramanan, University of California, Irvine
Wednesday, 5 November 14
Detecting Actions, Poses, and Objects with Relational Phraselets Chaitanya Desai and Deva RamananWednesday, 5 November 14
Relational Model
- E is the edge (or relation) between two parts - S is the score
encodes a prior acting as a compatibility measure
template tuned for mixture t(i)
HOG feature vector
springs that spatially constrain the parts i and j
deformation vector computed from the offset of pi&pj
Detecting Actions, Poses, and Objects with Relational Phraselets Chaitanya Desai and Deva Ramanan Deva Ramanan, University of California, Irvine
Wednesday, 5 November 14
Learning this modelpart i
from class: t(2)
part jfrom class: t(1)
Edge label: I(z(i)| z(j))- Maximise Score S
- Find Max weight spanning tree
Location and types for all parts in n
Linear model
Learn Thetas to minimise:
Detecting Actions, Poses, and Objects with Relational Phraselets Chaitanya Desai and Deva Ramanan
Wednesday, 5 November 14
Models learned with the tree structure:
Detecting Actions, Poses, and Objects with Relational Phraselets Chaitanya Desai and Deva RamananWednesday, 5 November 14
Experimental Setup & Results
• Action Detection
• Action Classification
• Pose Evaluation considering occlusion
Wednesday, 5 November 14
Action Detection
Detecting Actions, Poses, and Objects with Relational Phraselets Chaitanya Desai and Deva Ramanan
Wednesday, 5 November 14
False False Positives
Top False PositivesFalse False Positives due to bounding box errors
Detecting Actions, Poses, and Objects with Relational Phraselets Chaitanya Desai and Deva RamananWednesday, 5 November 14
Action Detection : Precision - Recall
Compares to visual phrase as a base line
Detecting Actions, Poses, and Objects with Relational Phraselets Chaitanya Desai and Deva Ramanan
Recognition using Visual Phrases Ali Farhadi, Mohammad Amin Sadeghi
Wednesday, 5 November 14
Action Classification
Detecting Actions, Poses, and Objects with Relational Phraselets Chaitanya Desai and Deva Ramanan
Compare to DPM/VP, FMP, FMP + occ
Wednesday, 5 November 14
Pose Estimation
• Should report location of all parts and any that have been occluded
• Novel scheme for evaluating models
Detecting Actions, Poses, and Objects with Relational Phraselets Chaitanya Desai and Deva RamananWednesday, 5 November 14
Pose Scores:
Detecting Actions, Poses, and Objects with Relational Phraselets Chaitanya Desai and Deva Ramanan
F1 scores:Penalise for labelling occluded points as visible
Combines pose estimation with aspect estimation
Wednesday, 5 November 14
Percentage of correct parts
• Reports on location of all parts including occlusions
• Suggests that this model predicts location of occluded parts well
Wednesday, 5 November 14
Strengths & Weaknesses
• Relation between parts
• Ability to predict the location of occluded parts
• Separating clusters for occluded and non-occluded parts
Wednesday, 5 November 14
Questions
Wednesday, 5 November 14