Model-free Approach to Garments UnfoldingBased on Detection of Folded Layers

Jan Stria, Vladimır Petrık, Vaclav Hlavac

Czech Institute of Informatics, Robotics and CyberneticsCzech Technical University in Prague

Czech Republic

26 September 2017

1 / 14

Project CloPeMa (Clothes Perception and Manipulation)

I Funded by European Commission in FP7, 2012–2015I Czech Technical University, University of Genoa,

CERTH, University of GlasgowI Fully autonomous pipeline for garments classification,

unfolding, flattening and folding using a dual-arm robot

2 / 14

Problem formulation

I Input: Unknown piece of garment folded once or more timeswith the top folded layers not overlapping

I Goal: Repeatedly detect one folded layer, estimate thefolding axis and unfold it with two cooperating robotic arms.

3 / 14

Motivation

I Geometric unfolding1

I The garment is grasped and partially unfolded in the air.I The remaining fold is detected after placing it on a table.

1Dimitra Triantafyllou et al. “A geometric approach to robotic unfolding ofgarments”. In: Robotics and Autonomous Systems (2016)

4 / 14

Related work

I Foldable templates1

I Input: RGB imageI Partial template fitting to

the garment contourI Folding axis generation

I Depth segmentation2

I Input: Depth mapI Watershed segmentationI Checking various

unfolding directions

2David Estevez et al. “Towards Robotic Garment Folding: A VisionApproach for Fold Detection”. In: Proc. ICARSC (2016)

5 / 14

Input

I RGBD sensor ASUS Xtion attached to the wrist is positionedabove the garment and oriented downwards.

I Acquire single image and depth map of the garment.

950

990

1030

1070

[mm]

6 / 14

Layers detection

0

10

20

[mm]

I Layers detection is formulated as pixels labeling:

p 7→ zp ∈ {T ,B}

I Combine information provided by the image and heights:I Pixels from the top (folded) layer are higher above the table

than the pixels from the bottom layer.I The boundary between layers is coincident with image edges.

7 / 14

Energy minimization 1

I Pixels labeling formulated as energy minimization problem:

Z ∗ = arg minZ∈{T ,B}|P|

∑p∈P

Up(zp) +∑{p,q}∈N

Vp,q(zp, zq)

I Unary potential Up evaluates how the pixel height H(p)corresponds to the estimated mean height of the bottomlayer µB and the top layer µT = 2µB :

Up(zp) = − logN(H(p);µzp , σ

2)

03691215

Top Bottom

8 / 14

Energy minimization 2

I Pairwise potential Vp,q depends on the spatial d(p, q) andvisual g(I , p, q) difference of the neighboring pixels:

Vp,q(zp, zq) = γ1 + γ2Jzp 6= zqKd(p, q)

exp

(−g(I , p, q)

2E [g ]

)I Solved by finding the minimum cut of a specific graph.

I The largest top layer is chosen for unfolding.

9 / 14

Folding axis estimation

I The folding axis must form an approximate segment on thegarment contour.

I The garment is unfolded virtually over each candidatefolding axis and the best one is selected.

10 / 14

Robotic manipulation

I Cooperated manipulation of two robotic arms:I One arm is holding the garment to prevent it from slipping.I The other arm grasps the top layer and unfolds it.

I Test various grasping and holding candidates.

I The holding gripper follows a triangular unfolding path.

11 / 14

Vision experimentsI Dataset containing 13 garments of various types, colors and

materials; each posed in 15 folded configurations

Garment SuccessFailure

Layers AxisJacket 14 / 15 1 0Jeans 12 / 15 3 0Shorts 14 / 15 1 0Skirt (2) 25 / 30 5 0Sweater (2) 26 / 30 2 2Sweatshirt 14 / 15 0 1Towel 14 / 15 1 0T-shirt (4) 51 / 60 9 0Total 87% 11% 2%

12 / 14

Robotic experiments

Garment SuccessReason of failure

Detection Planning ExecutionShorts 3 / 5 1 1 0Sweatshirt 4 / 5 1 0 0Towel 5 / 5 0 0 0T-shirt 5 / 5 0 0 0Total 17 / 20 2 1 0

13 / 14

Thank you for your attention.Questions, please?

http://bit.do/unfolding

Czech Institute of Informatics, Robotics and CyberneticsCzech Technical University in Prague

14 / 14