MSCOCO Keypoints Challenge 2018 -...
Transcript of MSCOCO Keypoints Challenge 2018 -...
![Page 1: MSCOCO Keypoints Challenge 2018 - cocodataset.orgpresentations.cocodataset.org/ECCV18/COCO18-Keypoints-Megvii.pdf · test_challenge test_dev mini_validation Ensemble model 76.4(final](https://reader034.fdocuments.net/reader034/viewer/2022042109/5e8943ed419b9b2d1e2b2d22/html5/thumbnails/1.jpg)
MSCOCO KeypointsChallenge 2018
Megvii (Face++)
![Page 2: MSCOCO Keypoints Challenge 2018 - cocodataset.orgpresentations.cocodataset.org/ECCV18/COCO18-Keypoints-Megvii.pdf · test_challenge test_dev mini_validation Ensemble model 76.4(final](https://reader034.fdocuments.net/reader034/viewer/2022042109/5e8943ed419b9b2d1e2b2d22/html5/thumbnails/2.jpg)
Megvii (Face++)
Team members:
Zhicheng Wang
Gang YuZeming Li Jian SunXiangyu Zhang
Wenbo Li Tianzi XiaoBinyi Yin Qixiang Peng Yuming Du
![Page 3: MSCOCO Keypoints Challenge 2018 - cocodataset.orgpresentations.cocodataset.org/ECCV18/COCO18-Keypoints-Megvii.pdf · test_challenge test_dev mini_validation Ensemble model 76.4(final](https://reader034.fdocuments.net/reader034/viewer/2022042109/5e8943ed419b9b2d1e2b2d22/html5/thumbnails/3.jpg)
Results on COCO 2018
Year 2016 2017 2018
mmAP 60.5 72.1 76.4(ours)
Results on COCO test challenge recent years
![Page 4: MSCOCO Keypoints Challenge 2018 - cocodataset.orgpresentations.cocodataset.org/ECCV18/COCO18-Keypoints-Megvii.pdf · test_challenge test_dev mini_validation Ensemble model 76.4(final](https://reader034.fdocuments.net/reader034/viewer/2022042109/5e8943ed419b9b2d1e2b2d22/html5/thumbnails/4.jpg)
Results on COCO 2018
test_challenge test_dev mini_validation
Ensemble model76.4(final
submission)78.1 80.0
Single model - 77.1 79.0
Year 2016 2017 2018
mmAP 60.5 72.1 76.4(ours)
Results on COCO test challenge recent years
Results of our method
![Page 5: MSCOCO Keypoints Challenge 2018 - cocodataset.orgpresentations.cocodataset.org/ECCV18/COCO18-Keypoints-Megvii.pdf · test_challenge test_dev mini_validation Ensemble model 76.4(final](https://reader034.fdocuments.net/reader034/viewer/2022042109/5e8943ed419b9b2d1e2b2d22/html5/thumbnails/5.jpg)
Overview• Pipeline
• Proposed Method
• Experiments
• Summary
![Page 6: MSCOCO Keypoints Challenge 2018 - cocodataset.orgpresentations.cocodataset.org/ECCV18/COCO18-Keypoints-Megvii.pdf · test_challenge test_dev mini_validation Ensemble model 76.4(final](https://reader034.fdocuments.net/reader034/viewer/2022042109/5e8943ed419b9b2d1e2b2d22/html5/thumbnails/6.jpg)
Overview▪ Pipeline
![Page 7: MSCOCO Keypoints Challenge 2018 - cocodataset.orgpresentations.cocodataset.org/ECCV18/COCO18-Keypoints-Megvii.pdf · test_challenge test_dev mini_validation Ensemble model 76.4(final](https://reader034.fdocuments.net/reader034/viewer/2022042109/5e8943ed419b9b2d1e2b2d22/html5/thumbnails/7.jpg)
Pipeline
Single Person
Pose
Estimation Network
MegDet[1] crop
[1] Megdet: A large mini-batch object detector:C Peng, eta. (CVPR2018)
Human AP 66.0
![Page 8: MSCOCO Keypoints Challenge 2018 - cocodataset.orgpresentations.cocodataset.org/ECCV18/COCO18-Keypoints-Megvii.pdf · test_challenge test_dev mini_validation Ensemble model 76.4(final](https://reader034.fdocuments.net/reader034/viewer/2022042109/5e8943ed419b9b2d1e2b2d22/html5/thumbnails/8.jpg)
Overview▪ Pipeline
▪ Proposed Method
![Page 9: MSCOCO Keypoints Challenge 2018 - cocodataset.orgpresentations.cocodataset.org/ECCV18/COCO18-Keypoints-Megvii.pdf · test_challenge test_dev mini_validation Ensemble model 76.4(final](https://reader034.fdocuments.net/reader034/viewer/2022042109/5e8943ed419b9b2d1e2b2d22/html5/thumbnails/9.jpg)
Proposed Methods
Cascade Pyramid Network[1]
[1] Cascaded pyramid network for multi-person pose estimation: Y Chen eta. (CVPR2018)
![Page 10: MSCOCO Keypoints Challenge 2018 - cocodataset.orgpresentations.cocodataset.org/ECCV18/COCO18-Keypoints-Megvii.pdf · test_challenge test_dev mini_validation Ensemble model 76.4(final](https://reader034.fdocuments.net/reader034/viewer/2022042109/5e8943ed419b9b2d1e2b2d22/html5/thumbnails/10.jpg)
Proposed Methods
Cascade Pyramid Network[1]
[1] Cascaded pyramid network for multi-person pose estimation: Y Chen eta. (CVPR2018)
headbackbone
![Page 11: MSCOCO Keypoints Challenge 2018 - cocodataset.orgpresentations.cocodataset.org/ECCV18/COCO18-Keypoints-Megvii.pdf · test_challenge test_dev mini_validation Ensemble model 76.4(final](https://reader034.fdocuments.net/reader034/viewer/2022042109/5e8943ed419b9b2d1e2b2d22/html5/thumbnails/11.jpg)
Proposed Methods
Cascade Pyramid Network[1]
[1] Cascaded pyramid network for multi-person pose estimation: Y Chen eta. (CVPR2018)
headbackbone
![Page 12: MSCOCO Keypoints Challenge 2018 - cocodataset.orgpresentations.cocodataset.org/ECCV18/COCO18-Keypoints-Megvii.pdf · test_challenge test_dev mini_validation Ensemble model 76.4(final](https://reader034.fdocuments.net/reader034/viewer/2022042109/5e8943ed419b9b2d1e2b2d22/html5/thumbnails/12.jpg)
Proposed Methods
BackBone of CPN
![Page 13: MSCOCO Keypoints Challenge 2018 - cocodataset.orgpresentations.cocodataset.org/ECCV18/COCO18-Keypoints-Megvii.pdf · test_challenge test_dev mini_validation Ensemble model 76.4(final](https://reader034.fdocuments.net/reader034/viewer/2022042109/5e8943ed419b9b2d1e2b2d22/html5/thumbnails/13.jpg)
Proposed Methods
Cascade Backbone Network
![Page 14: MSCOCO Keypoints Challenge 2018 - cocodataset.orgpresentations.cocodataset.org/ECCV18/COCO18-Keypoints-Megvii.pdf · test_challenge test_dev mini_validation Ensemble model 76.4(final](https://reader034.fdocuments.net/reader034/viewer/2022042109/5e8943ed419b9b2d1e2b2d22/html5/thumbnails/14.jpg)
Proposed Methods
Cascade Backbone Network with skip connection
![Page 15: MSCOCO Keypoints Challenge 2018 - cocodataset.orgpresentations.cocodataset.org/ECCV18/COCO18-Keypoints-Megvii.pdf · test_challenge test_dev mini_validation Ensemble model 76.4(final](https://reader034.fdocuments.net/reader034/viewer/2022042109/5e8943ed419b9b2d1e2b2d22/html5/thumbnails/15.jpg)
Proposed Methods
Cascade Backbone Network • Online Hard Keypoints Ming (OHKM) supervision• Coarse to fine supervision
![Page 16: MSCOCO Keypoints Challenge 2018 - cocodataset.orgpresentations.cocodataset.org/ECCV18/COCO18-Keypoints-Megvii.pdf · test_challenge test_dev mini_validation Ensemble model 76.4(final](https://reader034.fdocuments.net/reader034/viewer/2022042109/5e8943ed419b9b2d1e2b2d22/html5/thumbnails/16.jpg)
Overview• Pipeline
• Proposed Method
• Experiments
![Page 17: MSCOCO Keypoints Challenge 2018 - cocodataset.orgpresentations.cocodataset.org/ECCV18/COCO18-Keypoints-Megvii.pdf · test_challenge test_dev mini_validation Ensemble model 76.4(final](https://reader034.fdocuments.net/reader034/viewer/2022042109/5e8943ed419b9b2d1e2b2d22/html5/thumbnails/17.jpg)
Experiments
• Ablation experimentsBaseline(backbone res50) 72.3 256x192
![Page 18: MSCOCO Keypoints Challenge 2018 - cocodataset.orgpresentations.cocodataset.org/ECCV18/COCO18-Keypoints-Megvii.pdf · test_challenge test_dev mini_validation Ensemble model 76.4(final](https://reader034.fdocuments.net/reader034/viewer/2022042109/5e8943ed419b9b2d1e2b2d22/html5/thumbnails/18.jpg)
Experiments
• Ablation experimentsBaseline(backbone res50) 72.3 256x192
2res50 72.8 256x192
![Page 19: MSCOCO Keypoints Challenge 2018 - cocodataset.orgpresentations.cocodataset.org/ECCV18/COCO18-Keypoints-Megvii.pdf · test_challenge test_dev mini_validation Ensemble model 76.4(final](https://reader034.fdocuments.net/reader034/viewer/2022042109/5e8943ed419b9b2d1e2b2d22/html5/thumbnails/19.jpg)
Experiments
• Ablation experimentsBaseline(backbone res50) 72.3 256x192
2res50 72.8 256x192
2res50+up-skip 73.4 256x192
![Page 20: MSCOCO Keypoints Challenge 2018 - cocodataset.orgpresentations.cocodataset.org/ECCV18/COCO18-Keypoints-Megvii.pdf · test_challenge test_dev mini_validation Ensemble model 76.4(final](https://reader034.fdocuments.net/reader034/viewer/2022042109/5e8943ed419b9b2d1e2b2d22/html5/thumbnails/20.jpg)
Experiments
• Ablation experimentsBaseline(backbone res50) 72.3 256x192
2res50 72.8 256x192
2res50+up-skip 73.4 256x192
2res50+up-skip+down-skip 74.8 256x192
![Page 21: MSCOCO Keypoints Challenge 2018 - cocodataset.orgpresentations.cocodataset.org/ECCV18/COCO18-Keypoints-Megvii.pdf · test_challenge test_dev mini_validation Ensemble model 76.4(final](https://reader034.fdocuments.net/reader034/viewer/2022042109/5e8943ed419b9b2d1e2b2d22/html5/thumbnails/21.jpg)
Experiments
• Ablation experimentsBaseline(backbone res50) 72.3 256x192
2res50 72.8 256x192
2res50+up-skip 73.4 256x192
2res50+up-skip+down-skip 74.8 256x192
2res50+skip+coarse to fine loss 75.0 256x192
![Page 22: MSCOCO Keypoints Challenge 2018 - cocodataset.orgpresentations.cocodataset.org/ECCV18/COCO18-Keypoints-Megvii.pdf · test_challenge test_dev mini_validation Ensemble model 76.4(final](https://reader034.fdocuments.net/reader034/viewer/2022042109/5e8943ed419b9b2d1e2b2d22/html5/thumbnails/22.jpg)
Experiments
• Ablation experimentsBaseline(backbone res50) 72.3 256x192
2res50 72.8 256x192
2res50+up-skip 73.4 256x192
2res50+up-skip+down-skip 74.8 256x192
2res50+skip+coarse to fine loss 75.0 256x192
2res50+skip+coarse to fine loss 75.8 384x288
![Page 23: MSCOCO Keypoints Challenge 2018 - cocodataset.orgpresentations.cocodataset.org/ECCV18/COCO18-Keypoints-Megvii.pdf · test_challenge test_dev mini_validation Ensemble model 76.4(final](https://reader034.fdocuments.net/reader034/viewer/2022042109/5e8943ed419b9b2d1e2b2d22/html5/thumbnails/23.jpg)
Experiments
• Ablation experimentsBaseline(backbone res50) 72.3 256x192
2res50 72.8 256x192
2res50+up-skip 73.4 256x192
2res50+up-skip+down-skip 74.8 256x192
2res50+skip+coarse to fine loss 75.0 256x192
2res50+skip+coarse to fine loss 75.8 384x288
3res50+skip+coarse to fine loss 77.5 384x288
4res50+skip+coarse to fine loss 79.0 384x288
Ensemble 80.0 -
![Page 24: MSCOCO Keypoints Challenge 2018 - cocodataset.orgpresentations.cocodataset.org/ECCV18/COCO18-Keypoints-Megvii.pdf · test_challenge test_dev mini_validation Ensemble model 76.4(final](https://reader034.fdocuments.net/reader034/viewer/2022042109/5e8943ed419b9b2d1e2b2d22/html5/thumbnails/24.jpg)
Experiments
▪ Data augmentation• Half body augmentation(+0.4 AP)
30% probability if more than 8 labeled keypoints
![Page 25: MSCOCO Keypoints Challenge 2018 - cocodataset.orgpresentations.cocodataset.org/ECCV18/COCO18-Keypoints-Megvii.pdf · test_challenge test_dev mini_validation Ensemble model 76.4(final](https://reader034.fdocuments.net/reader034/viewer/2022042109/5e8943ed419b9b2d1e2b2d22/html5/thumbnails/25.jpg)
Experiments
▪ Data augmentation• Half body augmentation(+0.4 AP)
• Others(+0.4 AP)• Random scales(0.7 ~ 1.35)• Rotation (-45º ~ 45º)• Random flip
• Ensemble(+1.0 AP)
• External data(+0.7 AP)• Private data with 10W boxes
![Page 26: MSCOCO Keypoints Challenge 2018 - cocodataset.orgpresentations.cocodataset.org/ECCV18/COCO18-Keypoints-Megvii.pdf · test_challenge test_dev mini_validation Ensemble model 76.4(final](https://reader034.fdocuments.net/reader034/viewer/2022042109/5e8943ed419b9b2d1e2b2d22/html5/thumbnails/26.jpg)
Results
PredictionGround truth
![Page 27: MSCOCO Keypoints Challenge 2018 - cocodataset.orgpresentations.cocodataset.org/ECCV18/COCO18-Keypoints-Megvii.pdf · test_challenge test_dev mini_validation Ensemble model 76.4(final](https://reader034.fdocuments.net/reader034/viewer/2022042109/5e8943ed419b9b2d1e2b2d22/html5/thumbnails/27.jpg)
Results
PredictionGround truth
![Page 28: MSCOCO Keypoints Challenge 2018 - cocodataset.orgpresentations.cocodataset.org/ECCV18/COCO18-Keypoints-Megvii.pdf · test_challenge test_dev mini_validation Ensemble model 76.4(final](https://reader034.fdocuments.net/reader034/viewer/2022042109/5e8943ed419b9b2d1e2b2d22/html5/thumbnails/28.jpg)
Results
PredictionGround truth
![Page 29: MSCOCO Keypoints Challenge 2018 - cocodataset.orgpresentations.cocodataset.org/ECCV18/COCO18-Keypoints-Megvii.pdf · test_challenge test_dev mini_validation Ensemble model 76.4(final](https://reader034.fdocuments.net/reader034/viewer/2022042109/5e8943ed419b9b2d1e2b2d22/html5/thumbnails/29.jpg)
Results
PredictionGround truth
![Page 30: MSCOCO Keypoints Challenge 2018 - cocodataset.orgpresentations.cocodataset.org/ECCV18/COCO18-Keypoints-Megvii.pdf · test_challenge test_dev mini_validation Ensemble model 76.4(final](https://reader034.fdocuments.net/reader034/viewer/2022042109/5e8943ed419b9b2d1e2b2d22/html5/thumbnails/30.jpg)
Overview• Pipeline
• Proposed Method
• Experiments
• Summary
![Page 31: MSCOCO Keypoints Challenge 2018 - cocodataset.orgpresentations.cocodataset.org/ECCV18/COCO18-Keypoints-Megvii.pdf · test_challenge test_dev mini_validation Ensemble model 76.4(final](https://reader034.fdocuments.net/reader034/viewer/2022042109/5e8943ed419b9b2d1e2b2d22/html5/thumbnails/31.jpg)
Summary
• Cascade backbone• OHKM and coarse to fine supervision• Skip connections cross stages.
![Page 32: MSCOCO Keypoints Challenge 2018 - cocodataset.orgpresentations.cocodataset.org/ECCV18/COCO18-Keypoints-Megvii.pdf · test_challenge test_dev mini_validation Ensemble model 76.4(final](https://reader034.fdocuments.net/reader034/viewer/2022042109/5e8943ed419b9b2d1e2b2d22/html5/thumbnails/32.jpg)
Looking for Intern, Researcher, Research [email protected]@megvii.com