Leveraging orientation knowledge to enhance human pose ... · the body joint positions at the same...
Transcript of Leveraging orientation knowledge to enhance human pose ... · the body joint positions at the same...
![Page 1: Leveraging orientation knowledge to enhance human pose ... · the body joint positions at the same time. 6/16. Using an orientation information to improve the pose ... I Depth images](https://reader033.fdocuments.net/reader033/viewer/2022042807/5f79d8ebb31d860013218904/html5/thumbnails/1.jpg)
Leveraging orientation knowledge to enhancehuman pose estimation methods
S. Azrour, S. Pierard, M. Van Droogenbroeck
INTELSIG Laboratory, University of Liege, Belgium
Conference on Articulated Motion and Deformable Objects (AMDO 2016)
13-15th July 2016
1 / 16
![Page 2: Leveraging orientation knowledge to enhance human pose ... · the body joint positions at the same time. 6/16. Using an orientation information to improve the pose ... I Depth images](https://reader033.fdocuments.net/reader033/viewer/2022042807/5f79d8ebb31d860013218904/html5/thumbnails/2.jpg)
What is human pose estimation ?
Definition (Human pose estimation)
In computer vision, it is the study of algorithms and systems thatrecover the pose of a human body, which consists of joints andrigid parts.
2 / 16
![Page 3: Leveraging orientation knowledge to enhance human pose ... · the body joint positions at the same time. 6/16. Using an orientation information to improve the pose ... I Depth images](https://reader033.fdocuments.net/reader033/viewer/2022042807/5f79d8ebb31d860013218904/html5/thumbnails/3.jpg)
Application of human pose estimation: some examples
Motion analysis Medical
Entertainment Animation movies
3 / 16
![Page 4: Leveraging orientation knowledge to enhance human pose ... · the body joint positions at the same time. 6/16. Using an orientation information to improve the pose ... I Depth images](https://reader033.fdocuments.net/reader033/viewer/2022042807/5f79d8ebb31d860013218904/html5/thumbnails/4.jpg)
Types of camera-based pose estimation
The camera-based pose estimation (or motion capture) can bemarker-based or markerless:
maker-based: markers are put on the subject and the pose isrecovered by localizing these markers with a multi-camera setup.
markerless: the subject has nothing to wear and its pose isrecovered using a body model tracking method or a machinelearning technique.
4 / 16
![Page 5: Leveraging orientation knowledge to enhance human pose ... · the body joint positions at the same time. 6/16. Using an orientation information to improve the pose ... I Depth images](https://reader033.fdocuments.net/reader033/viewer/2022042807/5f79d8ebb31d860013218904/html5/thumbnails/5.jpg)
Markerless pose estimation using a machine learningtechnique
I Pose estimation algorithms developed by Microsoft for the Kinect camera(from“J. Shotton, R. Girshick et al., PAMI 2013”).
5 / 16
![Page 6: Leveraging orientation knowledge to enhance human pose ... · the body joint positions at the same time. 6/16. Using an orientation information to improve the pose ... I Depth images](https://reader033.fdocuments.net/reader033/viewer/2022042807/5f79d8ebb31d860013218904/html5/thumbnails/6.jpg)
Silhouette ambiguity
I There is an intrisic limitation when using color cameras: forone given silhouette, two di↵erent poses are possible
=∆ Depth cameras help to overcome this limitation but it stillremains hard to disambiguate the silhouette orientation and predictthe body joint positions at the same time.
6 / 16
![Page 7: Leveraging orientation knowledge to enhance human pose ... · the body joint positions at the same time. 6/16. Using an orientation information to improve the pose ... I Depth images](https://reader033.fdocuments.net/reader033/viewer/2022042807/5f79d8ebb31d860013218904/html5/thumbnails/7.jpg)
Using an orientation information to improve the poseestimation
Idea
It is preferable to rely on an additional method that is specificallydesigned for orientation estimation instead of trying to recover the jointpositions and disambiguate the silhouette orientation all at once.
How can we estimate the orientation ?
I The orientation estimation can be obtained from the image itself orthanks to any kind of sensors through a machine learning or atracking algorithm.
7 / 16
![Page 8: Leveraging orientation knowledge to enhance human pose ... · the body joint positions at the same time. 6/16. Using an orientation information to improve the pose ... I Depth images](https://reader033.fdocuments.net/reader033/viewer/2022042807/5f79d8ebb31d860013218904/html5/thumbnails/8.jpg)
Using an orientation information to improve the poseestimation
I The configuration considered in this work:
I How do we take advantage of the orientation estimation ?
=∆ We slice the full orientation range into smaller ranges and learna di↵erent model for each of these smaller ranges.
8 / 16
![Page 9: Leveraging orientation knowledge to enhance human pose ... · the body joint positions at the same time. 6/16. Using an orientation information to improve the pose ... I Depth images](https://reader033.fdocuments.net/reader033/viewer/2022042807/5f79d8ebb31d860013218904/html5/thumbnails/9.jpg)
Outline of our method
9 / 16
![Page 10: Leveraging orientation knowledge to enhance human pose ... · the body joint positions at the same time. 6/16. Using an orientation information to improve the pose ... I Depth images](https://reader033.fdocuments.net/reader033/viewer/2022042807/5f79d8ebb31d860013218904/html5/thumbnails/10.jpg)
Synthetic data generation
I The body model is created with MakeHuman.
I Depth images are rendered inside Blender.
I Poses are taken randomly from the CMU motion capture database.
10 / 16
![Page 11: Leveraging orientation knowledge to enhance human pose ... · the body joint positions at the same time. 6/16. Using an orientation information to improve the pose ... I Depth images](https://reader033.fdocuments.net/reader033/viewer/2022042807/5f79d8ebb31d860013218904/html5/thumbnails/11.jpg)
Pose estimation algorithm used in this work
We use our own implementation of the o↵set joint regression
algorithm proposed by Microsoft (R. Girshick et al., ICCV, 2011).
I The machine learning technique used is a random forest.
I Each pixel of the silhouette predicts a set of 3D o↵sets toward thebody joints.
I These predictions are then aggregated using Mean Shift.
11 / 16
![Page 12: Leveraging orientation knowledge to enhance human pose ... · the body joint positions at the same time. 6/16. Using an orientation information to improve the pose ... I Depth images](https://reader033.fdocuments.net/reader033/viewer/2022042807/5f79d8ebb31d860013218904/html5/thumbnails/12.jpg)
Experiments
I We compared the accuracy of the estimated pose when using1, 4 and 12 models.
I We considered two scenarios:
1 A constant global learning dataset size.
2 A constant learning dataset size per model.
12 / 16
![Page 13: Leveraging orientation knowledge to enhance human pose ... · the body joint positions at the same time. 6/16. Using an orientation information to improve the pose ... I Depth images](https://reader033.fdocuments.net/reader033/viewer/2022042807/5f79d8ebb31d860013218904/html5/thumbnails/13.jpg)
Results with a constant global learning dataset size
∆ Significant reduction of the error when going from 1 to 4 models.
∆ However, going from 4 to 12 models slightly worsens the performance.
13 / 16
![Page 14: Leveraging orientation knowledge to enhance human pose ... · the body joint positions at the same time. 6/16. Using an orientation information to improve the pose ... I Depth images](https://reader033.fdocuments.net/reader033/viewer/2022042807/5f79d8ebb31d860013218904/html5/thumbnails/14.jpg)
Results with a constant learning dataset size per model
∆ Systematic decrease of the error when the number of models isincreased.
∆ However, small di↵erence between 4 and 12 models suggests a plateauis reached.
14 / 16
![Page 15: Leveraging orientation knowledge to enhance human pose ... · the body joint positions at the same time. 6/16. Using an orientation information to improve the pose ... I Depth images](https://reader033.fdocuments.net/reader033/viewer/2022042807/5f79d8ebb31d860013218904/html5/thumbnails/15.jpg)
Mean error according to the orientation
5
10
15
30
210
60
240
90
270
120
300
150
330
180 0
Right shoulder
1 model (|LS|=2000)12 models (|LS|=12x2000)1 model (|LS|=8000)4 models (|LS|=4x2000)12 models (|LS|=12x666)
5
10
15
30
210
60
240
90
270
120
300
150
330
180 0
Right elbow
1 model (|LS|=2000)12 models (|LS|=12x2000)1 model (|LS|=8000)4 models (|LS|=4x2000)12 models (|LS|=12x666)
5
10
15
30
210
60
240
90
270
120
300
150
330
180 0
Right wrist
1 model (|LS|=2000)12 models (|LS|=12x2000)1 model (|LS|=8000)4 models (|LS|=4x2000)12 models (|LS|=12x666)
5
10
15
30
210
60
240
90
270
120
300
150
330
180 0
Right hip
1 model (|LS|=2000)12 models (|LS|=12x2000)1 model (|LS|=8000)4 models (|LS|=4x2000)12 models (|LS|=12x666)
5
10
15
30
210
60
240
90
270
120
300
150
330
180 0
Right knee
1 model (|LS|=2000)12 models (|LS|=12x2000)1 model (|LS|=8000)4 models (|LS|=4x2000)12 models (|LS|=12x666)
5
10
15
30
210
60
240
90
270
120
300
150
330
180 0
Right ankle
1 model (|LS|=2000)12 models (|LS|=12x2000)1 model (|LS|=8000)4 models (|LS|=4x2000)12 models (|LS|=12x666)
15 / 16
![Page 16: Leveraging orientation knowledge to enhance human pose ... · the body joint positions at the same time. 6/16. Using an orientation information to improve the pose ... I Depth images](https://reader033.fdocuments.net/reader033/viewer/2022042807/5f79d8ebb31d860013218904/html5/thumbnails/16.jpg)
Conclusion
I We can improve the accuracy of the estimated pose by taking advantage
of an orientation estimation.
I One way to take advantage of the orientation estimation is to learn
multiple models specialized for di↵erent range of orientations.
I We show that accuracy can be significantly improved when the number ofmodels increases, even while keeping a constant global learning datasetsize.
16 / 16