Image-based Clothes Animation for Virtual Fitting Zhenglong Zhou, Bo Shu, Shaojie Zhuo, Xiaoming...
-
Upload
gregory-turner -
Category
Documents
-
view
226 -
download
3
Transcript of Image-based Clothes Animation for Virtual Fitting Zhenglong Zhou, Bo Shu, Shaojie Zhuo, Xiaoming...
![Page 1: Image-based Clothes Animation for Virtual Fitting Zhenglong Zhou, Bo Shu, Shaojie Zhuo, Xiaoming Deng, Ping Tan, Stephen Lin * National University of.](https://reader035.fdocuments.net/reader035/viewer/2022062308/56649cf55503460f949c4895/html5/thumbnails/1.jpg)
![Page 2: Image-based Clothes Animation for Virtual Fitting Zhenglong Zhou, Bo Shu, Shaojie Zhuo, Xiaoming Deng, Ping Tan, Stephen Lin * National University of.](https://reader035.fdocuments.net/reader035/viewer/2022062308/56649cf55503460f949c4895/html5/thumbnails/2.jpg)
Image-based Clothes Animation for Virtual Fitting
Zhenglong Zhou, Bo Shu, Shaojie Zhuo, Xiaoming Deng, Ping Tan, Stephen Lin*
National University of Singapore, Microsoft Research Asia*
![Page 3: Image-based Clothes Animation for Virtual Fitting Zhenglong Zhou, Bo Shu, Shaojie Zhuo, Xiaoming Deng, Ping Tan, Stephen Lin * National University of.](https://reader035.fdocuments.net/reader035/viewer/2022062308/56649cf55503460f949c4895/html5/thumbnails/3.jpg)
Virtual Clothes Fitting
Awesaba (Aveilan)
![Page 4: Image-based Clothes Animation for Virtual Fitting Zhenglong Zhou, Bo Shu, Shaojie Zhuo, Xiaoming Deng, Ping Tan, Stephen Lin * National University of.](https://reader035.fdocuments.net/reader035/viewer/2022062308/56649cf55503460f949c4895/html5/thumbnails/4.jpg)
Lots of Systems in the Market
![Page 5: Image-based Clothes Animation for Virtual Fitting Zhenglong Zhou, Bo Shu, Shaojie Zhuo, Xiaoming Deng, Ping Tan, Stephen Lin * National University of.](https://reader035.fdocuments.net/reader035/viewer/2022062308/56649cf55503460f949c4895/html5/thumbnails/5.jpg)
2D Systems
• Overlay a still image on the user’s figure
• Limitation:– No clothes animation
Swivel (Face cake)
Virtual dressing room (Zugara )
![Page 6: Image-based Clothes Animation for Virtual Fitting Zhenglong Zhou, Bo Shu, Shaojie Zhuo, Xiaoming Deng, Ping Tan, Stephen Lin * National University of.](https://reader035.fdocuments.net/reader035/viewer/2022062308/56649cf55503460f949c4895/html5/thumbnails/6.jpg)
3D Systems
styku
Fitnect
• Render and animate 3D garment models according to the user’s motion
• Limitations:– 3D modeling is difficult– Real-time animation is
difficult– Realistic rendering is difficult
Shuang et al. 2011
![Page 7: Image-based Clothes Animation for Virtual Fitting Zhenglong Zhou, Bo Shu, Shaojie Zhuo, Xiaoming Deng, Ping Tan, Stephen Lin * National University of.](https://reader035.fdocuments.net/reader035/viewer/2022062308/56649cf55503460f949c4895/html5/thumbnails/7.jpg)
Our Data-driven Method
Database Input
Data preparation Garment transfer
OutputModel data
![Page 8: Image-based Clothes Animation for Virtual Fitting Zhenglong Zhou, Bo Shu, Shaojie Zhuo, Xiaoming Deng, Ping Tan, Stephen Lin * National University of.](https://reader035.fdocuments.net/reader035/viewer/2022062308/56649cf55503460f949c4895/html5/thumbnails/8.jpg)
Advantages of Our System
• No 3D modeling & rendering• No 3D cloth animation• “Image-based virtual fitting” in real-time
![Page 9: Image-based Clothes Animation for Virtual Fitting Zhenglong Zhou, Bo Shu, Shaojie Zhuo, Xiaoming Deng, Ping Tan, Stephen Lin * National University of.](https://reader035.fdocuments.net/reader035/viewer/2022062308/56649cf55503460f949c4895/html5/thumbnails/9.jpg)
Data Preparation
• Record approximately 5000 video frames– A blue background to facilitate segmentation in Adobe Affter Effects– Store segmented images and corresponding skeletal poses.
![Page 10: Image-based Clothes Animation for Virtual Fitting Zhenglong Zhou, Bo Shu, Shaojie Zhuo, Xiaoming Deng, Ping Tan, Stephen Lin * National University of.](https://reader035.fdocuments.net/reader035/viewer/2022062308/56649cf55503460f949c4895/html5/thumbnails/10.jpg)
Garment Transfer
• Pose estimation – from Microsoft Kinect
• Pose descriptor
• Garment database query– Input key: User’s pose vector– Return value: Segmented garment image
of similar pose
– Concatenation of joint positions
![Page 11: Image-based Clothes Animation for Virtual Fitting Zhenglong Zhou, Bo Shu, Shaojie Zhuo, Xiaoming Deng, Ping Tan, Stephen Lin * National University of.](https://reader035.fdocuments.net/reader035/viewer/2022062308/56649cf55503460f949c4895/html5/thumbnails/11.jpg)
Motion Smoothness Optimization
Input
Nearest Neighbor
#52#12 #71#55
Input video
Discontinuous animation
![Page 12: Image-based Clothes Animation for Virtual Fitting Zhenglong Zhou, Bo Shu, Shaojie Zhuo, Xiaoming Deng, Ping Tan, Stephen Lin * National University of.](https://reader035.fdocuments.net/reader035/viewer/2022062308/56649cf55503460f949c4895/html5/thumbnails/12.jpg)
Buffered frames
Motion Smoothness Optimization
#10 #11
#12
#13MultipleNearest Neighbors Smooth Motion
![Page 13: Image-based Clothes Animation for Virtual Fitting Zhenglong Zhou, Bo Shu, Shaojie Zhuo, Xiaoming Deng, Ping Tan, Stephen Lin * National University of.](https://reader035.fdocuments.net/reader035/viewer/2022062308/56649cf55503460f949c4895/html5/thumbnails/13.jpg)
Buffered frames
Motion Smoothness Optimization
i
iii
i llSilDE ,, 1
SourceTarget
Temporal motion smoothness
Pose similarity
Displaying
Shortest path
![Page 14: Image-based Clothes Animation for Virtual Fitting Zhenglong Zhou, Bo Shu, Shaojie Zhuo, Xiaoming Deng, Ping Tan, Stephen Lin * National University of.](https://reader035.fdocuments.net/reader035/viewer/2022062308/56649cf55503460f949c4895/html5/thumbnails/14.jpg)
Motion Smoothness Optimization
Buffered framesDisplaying
Target
![Page 15: Image-based Clothes Animation for Virtual Fitting Zhenglong Zhou, Bo Shu, Shaojie Zhuo, Xiaoming Deng, Ping Tan, Stephen Lin * National University of.](https://reader035.fdocuments.net/reader035/viewer/2022062308/56649cf55503460f949c4895/html5/thumbnails/15.jpg)
Buffered framesDisplaying Buffered frames
Motion Smoothness Optimization
New frame
SourceTarget
![Page 16: Image-based Clothes Animation for Virtual Fitting Zhenglong Zhou, Bo Shu, Shaojie Zhuo, Xiaoming Deng, Ping Tan, Stephen Lin * National University of.](https://reader035.fdocuments.net/reader035/viewer/2022062308/56649cf55503460f949c4895/html5/thumbnails/16.jpg)
Displaying Buffered frames
Motion Smoothness Optimization
New frame
SourceTarget
![Page 17: Image-based Clothes Animation for Virtual Fitting Zhenglong Zhou, Bo Shu, Shaojie Zhuo, Xiaoming Deng, Ping Tan, Stephen Lin * National University of.](https://reader035.fdocuments.net/reader035/viewer/2022062308/56649cf55503460f949c4895/html5/thumbnails/17.jpg)
Displaying Buffered frames
Motion Smoothness Optimization
New frame
SourceTarget
![Page 18: Image-based Clothes Animation for Virtual Fitting Zhenglong Zhou, Bo Shu, Shaojie Zhuo, Xiaoming Deng, Ping Tan, Stephen Lin * National University of.](https://reader035.fdocuments.net/reader035/viewer/2022062308/56649cf55503460f949c4895/html5/thumbnails/18.jpg)
Motion-aware Frame Query
• Clothes deformation depends on motion
• Measure motion by concatenating neigboring pose vectors– Give higher weight to the central frames
• Replace the pose similarity in optimization by motion similarity
![Page 19: Image-based Clothes Animation for Virtual Fitting Zhenglong Zhou, Bo Shu, Shaojie Zhuo, Xiaoming Deng, Ping Tan, Stephen Lin * National University of.](https://reader035.fdocuments.net/reader035/viewer/2022062308/56649cf55503460f949c4895/html5/thumbnails/19.jpg)
Image Warping
• Exact match often cannot be found
• Skeleton based warping– Apply moving least square
warping [Schaefer et al. 2006]
– Use the skeleton joints as control points.
![Page 20: Image-based Clothes Animation for Virtual Fitting Zhenglong Zhou, Bo Shu, Shaojie Zhuo, Xiaoming Deng, Ping Tan, Stephen Lin * National University of.](https://reader035.fdocuments.net/reader035/viewer/2022062308/56649cf55503460f949c4895/html5/thumbnails/20.jpg)
• Our optimization chooses locally consistent sequences• Discontinuity exists at the connection of different sequences
Frame Interpolation and Alignment
#11 #12 #13 #14 #55 #56 #57 #58
Apply optical flow based linear interpolation to transit
![Page 21: Image-based Clothes Animation for Virtual Fitting Zhenglong Zhou, Bo Shu, Shaojie Zhuo, Xiaoming Deng, Ping Tan, Stephen Lin * National University of.](https://reader035.fdocuments.net/reader035/viewer/2022062308/56649cf55503460f949c4895/html5/thumbnails/21.jpg)
Results
Please refer to the video demo on the project website.
![Page 22: Image-based Clothes Animation for Virtual Fitting Zhenglong Zhou, Bo Shu, Shaojie Zhuo, Xiaoming Deng, Ping Tan, Stephen Lin * National University of.](https://reader035.fdocuments.net/reader035/viewer/2022062308/56649cf55503460f949c4895/html5/thumbnails/22.jpg)
Conclusion
• We propose an image-based technique for clothes animation
• It provides a practical solution for virtual clothes fitting
![Page 23: Image-based Clothes Animation for Virtual Fitting Zhenglong Zhou, Bo Shu, Shaojie Zhuo, Xiaoming Deng, Ping Tan, Stephen Lin * National University of.](https://reader035.fdocuments.net/reader035/viewer/2022062308/56649cf55503460f949c4895/html5/thumbnails/23.jpg)
Future work
• Body shape estimation.
• Online system.– Send pose vector– Receive garment image– Simple image rendering.
Pose vector
Garment image
![Page 24: Image-based Clothes Animation for Virtual Fitting Zhenglong Zhou, Bo Shu, Shaojie Zhuo, Xiaoming Deng, Ping Tan, Stephen Lin * National University of.](https://reader035.fdocuments.net/reader035/viewer/2022062308/56649cf55503460f949c4895/html5/thumbnails/24.jpg)
Thank you!