Drag-and-drop Pasting

By Chui Sung Him, Gary

Supervised by Prof. Chi-keung Tang

Outline

Background Objectives Techniques Results & extended application Demo

Background

Seamless object cloning Traditional method

– User interaction– Time– Expertise

Objectives

Reduce user-interaction Suppress unnatural look automatically Optimize boundary to achieve the above

objectives

Techniques

User provide rough region of interest (RoI)– Contiaining object of interest (OoI)– Drag-and-drop to the target

Optimization problem

Euler-Lagrange equation Poisson equation

|*| with min

2fff

fv

|*| with ,over div fff v

Ω

Ωobj

f*

Problem

Objectives

Reduce user-interaction Suppress unnatural look automatically Optimize boundary

User provides only rough RoI Assume v=∇g and let f’=f – g, reformulate opti

mization problem

Poisson equation becomes Laplace equation

Approach zero when (f*-g) = constant– find an optimal boundary to satisfy this

Techniques (Cont’d)

|)*(|' with 'min

2

'gfff

f

|*|' with ,over 0' gfff

Techniques (Cont’d)

To find the optimal boundary– Inside the RoI– Outside the OoI

Define an energy function– Total color variance–

Minimize it

Ω

Ωobj

f*

objp

kpgpfkE

\ s.t. )))()(*((, 2

Iterative minimization Initialize ∂Ω as boundary of RoI Given new ∂Ω, optimize E w.r.t. k

Given new k, optimize E with new ∂Ω– Shortest path problem

Until convergence reached

0),(

k

kE

p

pgpfk *1

objp

kpgpfkE

\ s.t. )))()(*((, 2

Shortest path problem?

Cost of each pixel = its color variance w.r.t. new k

Path to find in closed band Ω\Ωobj

– Not a usual shortest path

A shortest closed-path problem

Ω

Ωobj

f*

Shortest closed-path

Break the band with a cut– Not closed now

Shortest closed-path

Perform usual shortest path algorithm on a yellow pixel– Dijkstra O(NlogN)

Shortest closed-path

Perform on M yellow pixels– O(MNlogN)

Selecting the cut

With minimum length M

Reduce probability of twisting path– Not to pass the cut more than once

Reduce running time (MNlogN)

Results

Results

Result

Result

Extended Application

Seamless image completion A hole in an image S Another image D provided by user

– Semantically correct

Auto complete the hole

Seamless Image Completion

D and S semantically agreed– Color– Scene objects

Selecting region on D to complete the hole– Sum of Squared Difference (SSD) of color– Distance to the hole on S

Seamless Image completion Result

Seamless Image completion Result

Live Demo

Q&A

THE END