Image Warping and Morphing

Leow Wee Kheng CS4243 Computer Vision and Pattern Recognition

CS4243 Image Warping and Morphing 1

Morphing is artistic

CS4243 Image Warping and Morphing 2

Morphing is captivating!

CS4243 Image Warping and Morphing 3

Morphing is dynamic!

CS4243 Image Warping and Morphing 4

Morphing is fun!

CS4243 Image Warping and Morphing 5

Morphing is fun!

CS4243 Image Warping and Morphing 6

OK, that was not really morphing.

But, you get the point.

Two kinds of morphing Image morphing

2D

Object morphing 2D: like image morphing 3D: more complex

CS4243 Image Warping and Morphing 7

our focus

Image Warping & Morphing Involves 3 things

Spatial transformation Colour transfer Continuous transition

CS4243 Image Warping and Morphing 8

image warping

Image Warping & Morphing Involves 3 things

Spatial transformation Colour change Continuous transition

CS4243 Image Warping and Morphing 9

Spatial Transformation Simplest form: affine transformation

CS4243 Image Warping and Morphing 10

Affine Transformation

CS4243 Image Warping and Morphing 11

-3

-2

-1

0

1

2

3

-3 -2 -1 0 1 2 3

−

1005.010

8.001

Source (x, y) Target (u, v)

-3

-2

-1

0

1

2

3

-3 -2 -1 0 1 2 3

translation

Affine Transformation

CS4243 Image Warping and Morphing 12

-3

-2

-1

0

1

2

3

-3 -2 -1 0 1 2 3

10002.10008.0

Source (x, y) Target (u, v)

-3

-2

-1

0

1

2

3

-3 -2 -1 0 1 2 3

scaling

Affine Transformation

CS4243 Image Warping and Morphing 13

-3

-2

-1

0

1

2

3

-3 -2 -1 0 1 2 3

100011.002.01

Source (x, y) Target (u, v)

-3

-2

-1

0

1

2

3

-3 -2 -1 0 1 2 3

shearing

parallel lines remain parallel

Affine Transformation

CS4243 Image Warping and Morphing 14

-3

-2

-1

0

1

2

3

-3 -2 -1 0 1 2 3

−

1000cossin0sincos

Source (x, y) Target (u, v)

-3

-2

-1

0

1

2

3

-3 -2 -1 0 1 2 3

rotation

parallel lines remain parallel

Affine Transformation To solve for affine matrix:

For i = 1,…, n, arrange into two matrix equations:

Then, solve each equation using linear least square.

CS4243 Image Warping and Morphing 15

Perspective Transformation Generalisation of affine transformation:

Looks like a linear equation. But, perspective transformation is nonlinear. Will discuss further later.

CS4243 Image Warping and Morphing 16

Polynomial Transformation Described by polynomial equations

Example: 2nd-order polynomial, e.g., quadratic

CS4243 Image Warping and Morphing 17

Quadratic Transformation In matrix form

Solving the parameters is easy! Just like for the affine case (exercise).

CS4243 Image Warping and Morphing 18

If these are zero, what is the matrix?

Quadratic Transformation

CS4243 Image Warping and Morphing 19

-3

-2

-1

0

1

2

3

-3 -2 -1 0 1 2 3

Source (x, y) Target (u, v)

-3

-2

-1

0

1

2

3

-3 -2 -1 0 1 2 3

0100000011.000

tapering / perspective distortion

Quadratic Transformation

CS4243 Image Warping and Morphing 20

-3

-2

-1

0

1

2

3

-3 -2 -1 0 1 2 3

− 0101.000

001000

Source (x, y) Target (u, v)

-3

-2

-1

0

1

2

3

-3 -2 -1 0 1 2 3

tapering / perspective distortion

Quadratic Transformation

CS4243 Image Warping and Morphing 21

-3

-2

-1

0

1

2

3

-3 -2 -1 0 1 2 3

− 01001.00

001001.0

Source (x, y) Target (u, v)

-3

-2

-1

0

1

2

3

-3 -2 -1 0 1 2 3

expansion / compression

Quadratic Transformation

CS4243 Image Warping and Morphing 22

-3

-2

-1

0

1

2

3

-3 -2 -1 0 1 2 3

− 010001.0

00101.00

Source (x, y) Target (u, v)

-3

-2

-1

0

1

2

3

-3 -2 -1 0 1 2 3

warping

General Transformation General form of u and v

Possible kernel functions: Affine Polynomial Splines: B-splines, cubic splines, etc. Thin-plate spline: model physical bending energy

CS4243 Image Warping and Morphing 23

),(1

yxhau k

K

kk∑

=

=

parameters kernel function

Which kernel to use? General guidelines

Sufficient to capture transformation Not overly complex

CS4243 Image Warping and Morphing 24

Example

Fit affine transformation. Correct transformation, small error.

CS4243 Image Warping and Morphing 25

source target mapped source

Example

Fit affine transformation. Incorrect transformation, large error.

CS4243 Image Warping and Morphing 26

source target mapped source

Example

Fit quadratic transformation. Looks correct, small error; but can over fit.

CS4243 Image Warping and Morphing 27

source target mapped source

Example

Fit quadratic transformation. Correct transformation, small error.

CS4243 Image Warping and Morphing 28

source target mapped source

Image Warping & Morphing Involves 3 things

Spatial transformation Colour transfer Continuous transition

CS4243 Image Warping and Morphing 29

Colour Transfer Determine colour in transformed image. Remember to use backward mapping plus

bilinear interpolation

CS4243 Image Warping and Morphing 30

Colour Transfer Forward mapping

Copy colour of p in I to q in I’. But q is at real-valued coordinates; problem.

CS4243 Image Warping and Morphing 31

source target

Colour Transfer Backward mapping

Map q in I’ to p in I. Use bilinear interpolation to determine colour of p. Copy colour of p to q.

CS4243 Image Warping and Morphing 32

source target

Image Warping Example With affine mapping

CS4243 Image Warping and Morphing 33

Image Warping Example With affine mapping

CS4243 Image Warping and Morphing 34

Image Warping Example With affine mapping

CS4243 Image Warping and Morphing 35

Image Warping Example With affine mapping

CS4243 Image Warping and Morphing 36

Image Warping Example With quadratic mapping

CS4243 Image Warping and Morphing 37

Why so much distortion?

Image Warping Example With quadratic mapping

CS4243 Image Warping and Morphing 38

Image Warping Example With quadratic mapping

CS4243 Image Warping and Morphing 39

Image Warping Example With quadratic mapping

CS4243 Image Warping and Morphing 40

Local Transformation Divide image into regions. Warp each region by a different transform.

Ensure changes over boundaries are smooth.

Achieve finer control.

CS4243 Image Warping and Morphing 41

Summary Need to mark good corresponding points. Lower-order function may not warp enough. Higher-order function can lead to distortions. Local transformations give finer control.

CS4243 Image Warping and Morphing 42

Image Warping & Morphing Involves 3 things

Spatial transformation Colour transfer Continuous transition

CS4243 Image Warping and Morphing 43

Image Morphing Basic ideas:

Want positions to change smoothly. Want colours to change smoothly.

CS4243 Image Warping and Morphing 44

Image Morphing Simplest transition: linear

CS4243 Image Warping and Morphing 45

pi qi

ri

I J

morphing path

CS4243 Image Warping and Morphing 46

Morphing path is actually like this:

Image Morphing Set time step ∆t = 1 / number of frames. Repeat for 0 ≤ t ≤ 1

Warp I to I(t) by mapping pi to ri(t). Warp J to J(t) by mapping qi to ri(t). Blend I(t) and J(t) into M(t)

Save M(t) into a video.

CS4243 Image Warping and Morphing 47

Example

CS4243 Image Warping and Morphing 48

Anything peculiar?

CS4243 Image Warping and Morphing 49

Sample frames

Summary For seamless morphing

Match all corresponding features. Need accurate transformation.

For smooth morphing Use smaller time step.

CS4243 Image Warping and Morphing 50

Further Reading More sophisticated image warping: [Wolberg90]. More sophisticated image morphing: [Lee96]

CS4243 Image Warping and Morphing 51

References S.-Y. Lee and S. Y. Shin. Warp generation and transition control in

image morphing. In Interactive Computer Animation. Prentice Hall, 1996.

G. Wolberg. Digital Image Warping. IEEE Computer Society Press, 1990.

CS4243 Image Warping and Morphing 52