MR Fitting Simulation and Clothes Modeling for Wheelchair ...

National Institute of Advanced Industrial Science and Technology

MR Fitting Simulation and Clothes Modeling for Wheelchair Users

○Ryosuke Ichikari, Masaki Onishi, and Takeshi Kurata AIST, Japan

2014/4/18-20 KJMR2014@Jeonju (KIST Jeonbuk Branch), Korea


Backgrounds • Demands for improving quality of life (QoL) of

people with disabilities • Wheelchair users’ problems for purchasing clothes

– Difficulties of trial fitting – Lack of items specially designed for the users

Clothes fabrication is normally based on standing position

⇒Research development for supporting clothes fabrication for wheelchair users is required

2 Fashion show showing clothes for the wheelchair users

Images from http://www.rehab.go.jp/rehanews/japanese/webnews/201302/news_201302_6.html


Research Project System

3

Project members consist of AIST, National Rehabilitation Center for Persons with Disabilities, and Bunka Fashion College (all in Japan)

Non-contactMeasurement by RGB-D Camera

Modeling/Capturing Clothes

MR Fitting Simulation

Contact-typeMeasurement

Pattern Makingby Apparel CAD

Design (Photo) ofthe Dedicated DummyMeasurement

of the Wheel-chair User

Virtual FittingActual ClothesFabrication

Visualizingthe measuredresults Utilizing data from

fitting results (size, personal preference, etc.)

Utilizing the measured data for the clothes fabrication

Feedback of data obtained duringclothes fabrication


MR(Mixed Reality) Fitting Simulation

4

• Needs for MR fitting simulation from the field – Virtual fitting without physically-taxing operations – Visualizing the fitting result from various viewpoints – Simple configuration at a low price

• Computer-generated clothes are superimposed onto the users body by MR technology.

• Camera position is not restricted by the real positing, it can be changed virtually in the 3D-CG space.


Related Works • MR-based virtual fitting systems in the industry

– EON Reality Eon Interective Mirror Visualizing virtual fitting results with simple cloth simulation in real time on a display which represents mirror

– DNP Fitting system demonstrated in UNIQLO. This shows color variations of the clothes by changing color of the clothes in the display.

• Researches related to virtual fitting or clothes fabrication – Stefan et.al. (ISMAR2011)：

Fitting simulation based pre-captured clothes with various poses – Igarashi et.al.(UIST2002)：

Interactive simulation of sewing the parts of clothes and fitting on CG dolls

5

No work have focused on the fitting simulation with sitting positions

http://www.eonreality.com/eon-interactive-mirror/




http://www.dnp.co.jp/news/10056353_2482.html


System Requirements and Prototype To be beneficial for both users and system owners Prototype Measuring users’ body for actual fabrications ✔ Trial fitting without forcing users to get off from the wheelchair ✔ Improve the quality of the experience of trial fitting ✔

6

Requirements for system configurations Simple configuration at a moderate price ✔

Adopting contact-less measurement for the users ✔

Requirements for MR functions Allowing to check their appearance from various view-points ✔ Composition of the clothes without visual paradox ✔ Accurate registration between CG clothes and real body ✔ Tracking the user motion and deformation of the clothes On going Practical method for preparing clothes models On going


MR System Based on 3D Reconstruction of Objects in the Real World

• Visualization of output from the non- contact measuring method

• Inputting the sequence from a fixed RGB-D camera – Library:OpenNI(RGB-D camera)

OpenGL（Rendering CG space） • MR composition in the 3D-CG space where

real objects in the world are reconstructed in – Enabling virtual camera movement

• CG Clothes are prepared by capturing real clothes and superimposed onto the image of the user. – Utilizing multi-path rendering for rigorous compositions without

visual paradox

• . 7

RGB-D camera: Asus Xtion


How to Render the MR Space • Real-Objects: Importing RGB-D sequence and rendered as

point cloud in the 3D-CG space (with OpenNI, OpenGL) • Clothes: Rendering mesh of the clothes generated by RGB-D

sequence in the same space file format of meshes: Collada(.dae)

8

CG model of the clothes Rendered RGB-D

sequence

Composition result


Rigorous Composition without Visual Paradox • Geometric consistency is important

for MR-based visualization e.g. ) If inner clothes under superimposed outer clothes is visible, it may cause visual paradox.

• Rendering Method – Keeping the silhouette of the clothes – If the silhouette of the body is unnaturally protruded from the

clothes, these area should be deleted – Necessary parts (face, hand, foot) are preserved in the “clean-

up” process – The deleted area and occluded area of RGB-D camera are filled

with the pre-captured background image or inpainting method ⇒Composition process is implemented using multipath-rendering

9

Example of bad case


Fulfilling the holes or the deleted areas • The holes and the deleted areas are fulfilled with

pre-captured background images or inpainting method*

10 *A. Telea, "An image inpainting technique based on the fast marching method," J. Graphics Tools, 2004

Input RGB-D image (rendered)

Background RGB-D Image(rendered)

After composition


Topics about Preparing Clothes Model and Deformation

11

• How to prepare CG model of clothes – Simulation based: Using output of 3D sewing simulation from

apparel CAD

– Capturing based：Scanning actual clothes by 3D scanning.

• Deformation of the clothes model according to the shape of body and movement of joint angles

• Consideration of expansion and contraction(stretching) of the material

• Measuring the shape and pose of the users and their tracking.

• How to utilize the result of the simulation for the actual fabrication – Compatibility with the apparel CAD


Pattern→3D clothes model

(Sewing Simulation)

Conceptual-Flow

Clothes model→Fitting results

(Fitting Simulation)

Pattern data

Torso data

3D clothes model

Shape data

Sitting Pose data Fitting result

Rigorous Composition

of the CG clothes

MR composition result

Contact-type measurement

Contact-less body capture

Real scene capture

Prototype

3D scanning of real clothes by RGB-D camera

Mesh Deformation of scanned mesh by

skeleton and measured data


3D Scanning with RGB-D Camera • ReconstructMe: A 3D scanning software based on the

SLAM with RGB-D camera – Outputs: Triangle mesh with vertex color

13


Pipeline of the Sawing Emulation • Preparing the CG model of the clothes

– Adding pattern information

• Captured mesh is separated into parts which imitates pattern of the clothes – No UV coordinate and texture at this time

• Example of the operating pipeline with CG software (Autodesk Maya)

– Meth separation – UV space definition by planner/cylindrical projection (Mesh Lab.) – Baking(Projecting) vertex color onto texture

14


Mesh Separation and UV Space Definition • Imitating pattern of the real clothes by UV space

definition – (e.g.) Separating into four parts (front, back, left, right) – Capability for accessing/modifying each parts

separately

15


Approximate Fitting Simulation • Deformation to deal with variation of the shape of body

– Shape deformation: Changing vertex position along normal direction

– Estimating size of real body and clothes for the deformation by ellipse approximation

• Deformation to deal with variation of the joint angles – Rigorous clothes simulation is not employed in the prototype – Employing skeleton based mesh deformation

• Pros: Calculate deformation of the clothes in real-time • Cons:

– Quality of the result depends on the mesh weight – Not physically aculeate (gravity, volume preservation, stretch

property, etc.)

16


Example of the Shape Deformation • Shape deformation is done by moving vertex position

along the normal while keeping the UV coordinate

17

Normal of a Mesh

Mask for the deformation (target: waist)


Example of Skinning and Deformation of Captured Mesh in Maya

• Definition of joints (skeleton) • Definition of mesh weight • Skeleton based mesh deformation

18


Demo

19


Conclusions

20

• Proposed MR fitting simulation system which aims for supporting wheelchair users to purchase clothes

• Utilized RGB-D camera for changing virtual viewpoints in the MR space

• Rigorous rendering pipeline keeps geometric consistency

• Proposed methods for preparing clothe model and deformation of the model – Clothe model creation using RGB-D camera based

scanning method – Mesh deformation to deal with the variety of the

shapes and the joint angles


Future Works

• Integration with contact-type measuring method for measuring occluded parts

• Capturing from back-side of the users • Automatic tracking of the skeleton from RGB-D

sequence • Measuring or representing material stretching • Feasibility study of the service with the proposed

system

21

MR Fitting Simulation and Clothes Modeling for Wheelchair ...

Documents

Transcript of MR Fitting Simulation and Clothes Modeling for Wheelchair ...