Super-Resolution

Authors: Wagner, Waagen and Cassabaum

Presented By: Mukul Apte

Introduction

Definition: Create High Resolution visual output from Low Resolution visual input.

Mathematical assistance to features viz. motion detection, face recognition, person detection. Action Packed Sports Images Astronomy Medical Imaging Surveillance

Range of resources: low (single camera, polaroid lenses) to high (super-resolution chips, androids-ASIMO)

Basic ideaGiven: A set of degraded (warped, blurred, decimated, noised) images:

Required: Fusion of the measurements into a higher resolution image:

Types

Static Super-Resolution (SSR) - The creation of a single improved image, from the finite measured sequence of images

Dynamic Super-Resolution (SSR) - Low Quality Movie In - High Quality Movie Out

Simple Example

Concepts I: Raster, CCD, Image in mathematical domain, Matlab

Concepts II: Bresenham’s line algorithm, non-uniform interpolation, frequency domain

Simple Example

D

For a given band-limited image, the Nyquist sampling theorem states that if a uniform sampling is fine enough (pixel density = twice highest static frequency), perfect reconstruction is possible.

D

Due to our limited camera resolution, we sample using an insufficient 2D grid

2D

2D

Simple Example

However, we are allowed to take a second picture and so, shifting the camera ‘slightly to the right’ we obtain

2D

2D

Simple Example

Similarly, by shifting down we get a third image

2D

2D

Simple Example

And finally, by shifting down and to the right we get the fourth image

2D

2D

Simple Example

Simple Example - Conclusion

It is trivial to see that interlacing the four images, we get that the desired resolution is obtained, and thus perfect reconstruction is guaranteed.

This is Super-Resolution in its simplest form

Uncontrolled Displacements

In the previous example we counted on exact movement of the camera by D in each direction.

What if the camera displacement is uncontrolled?

Uncontrolled Displacements

It turns out that there is a sampling theorem due to Yen (1956) and Papoulis (1977) covering this case, guaranteeing perfect reconstruction for periodic uniform sampling if the sampling density is high enough (1 sample per each D-by-D square).

Uncontrolled Rotation/Scale/Disp.

In the previous examples we restricted the camera to move horizontally/vertically parallel to the photograph object.

What if the camera rotates? Gets closer to the object (zoom)?

Uncontrolled Rotation/Scale/Disp.

There is no sampling theorem covering this case

Further Complications

Sampling is not a point operation – there is a blur

Motion may include perspective warp, local motion, etc.

Samples may be noisy – any reconstruction process must take that into account.

Static Super-Resolution

Static Super-Resolution

Algorithm

N321 Y,,Y,Y,YfX̂

1Y

X̂

2Y NY

Low Resolution Measurements

High Resolution Reconstructed Image

t

3YStatic Super-Resolution

Low Resolution Measurements

High Resolution Reconstructed Images

1tX̂,tYftX̂

ttY

ttX̂

Dynamic Super-Resolution

Algorithm

t

t

Dynamic Super-Resolution

Approach

Image Registration Motion Estimation Projection onto High-Resolution Grid

Non-Uniform Interpolation Frequency domain – alias correction

Registration Projection

High Res GridLow-res Images Registration (sub-pixel grid)

•Rotation Calculation

•Correlate 1st LR image with all LR images at all angles

OR

•Calculate energy at all angles for all LR images. Correlate energy vector to find the rotation angle

Anglei = max index(correlation(I1(θ), Ii (θ)))

Energy at angle Ii(θ)

LR image 1

Energy at angle I2(θ)

LR image 2

i = 2,3,..,N (number of LR images)

1.1 Registration (angle)

Shift Calculated using Frequency Domain Method

Δs [Δx Δy]T

u [fx fy]

Fi (uT) = ej2πuΔsF1(uT)

Δs = angle( Fi (uT) / F1(uT) )

2πu

• Used only 6% lower u (high freq could be aliased)

• Used least square to calculate Δs

1.1 Registration (shift)

Input: Down-sampled aliased images Goal I: Correct the low-freq aliased data Goal II: Predict the lost high freq values

2.1 Frequency Domain

π-πOriginal High-Res

π-πDown-sampled

π/2-π/2 π

Aliased (fix it)

Lost (find it)

Up-sampledπ-π

Desired High-Res

I (known pixel positions) = Known Values

I_fft = fft2(I)

I_fft(higher Freq) = 0

I= ifft2 (I_fft)

2.2 Projection onto High-res grid

Papoulis-Gerchberg AlgorithmProjection onto convex sets

Known pixel values Known Cut-off freq in the HR image

Algorithm:

N

1k

1kkkkkkk ,0~V ,VXY

WNFHD

X

High-Resolution

ImageH

H

Blur

1

N

F =I1

FN

Geometric Warp

D

D1

N

Decimation

V1

VN

Additive Noise

Y1

YN

Low-Resolution

Images

SSR – The Model

F[j,i]=1

X Z

Per every point in X

find a matching point in Z

N

j

1

N

j

1

z

z

z

1

1

1

x

x

x

00

Warp – Linear Operation

We assume that the images Yk and the operators Hk, Dk, Fk,& Wk are known to us, and we use them for the recovery of X.

Yk – The measured images (noisy, blurry, down-sampled ..)

Hk – The blur can be extracted from the camera characteristics

Dk – The decimation is dictated by the required resolution ratio

Fk – The warp can be estimated using motion estimation

Wk – The noise covariance can be extracted from the camera

characteristics

Model Assumptions

X

Y

Y

Y

NNN

222

111

N

2

1

FHD

FHD

FHD

Clearly, this linear system of equations should have more equations than unknowns in order to make it

possible to have a unique solution.

Special Condition - Noiseless

SSR – Handling Problems

VXY

N

1kkkk VXY F

VXY D

VXY H

YIX̂1T

WSS

YX̂ T1TT HWSSHH

YX̂ T1TT DWSSDD

N

1kk

Tk

1

TN

1kk

Tk YX̂ FWSSFF

XXXA TT WSSUsing

Single image de-noising

Single image restoration

Single image scaling

Motion compensation average

Simulated error

Weighted edges

Back projection

j

N

1k

Tjkkkkk

Tk

Tk

Tkj1j X̂X̂YX̂X̂

WSSFHDWDHF

SSR Standard Equation

All the above operations can be interpreted as iterative operations performed on images.

(Typically 15-20 iterations are performed.)

Thumb Rule on Desired Resolution

Assume that we have N images of M-by-M pixels, and we would like to produce an image X of size L-by-L. Then –

MNL

Taj Mahal – Low-res image I

Initial Setup

FFT (Initial image)

Papoulis – Gerchberg Algorithm

Image at iteration 0

FFT

I(high freq) =0

Known PixelValues

Image after 1st iteration

Papoulis – Gerchberg Algorithm

Image at iteration 1

FFT

I(high freq) =0

Known PixelValues

Image after 10 iterations

Papoulis – Gerchberg Algorithm

After 50 iterations

SR Reconstructed imageBilinear Interpolation Bicubic Interpolation

Papoulis – Gerchberg Algorithm

Original Low-res images(Courtesy: Patrick Vandewalle)

Results (Images - I)• Input: 4 snaps using a high-res digital camera• Cropped the same part of each image • SR algorithm compared with bicubic interpolation

Bicubic Interpolation

Results (Images - I)

Super-resolution

Results (Images - I)

Low-Res Image I Low-Res Image II

• SR Algorithm didn’t work as expected !!!

• Reason:

• Motion was not restricted to shifts & rotation

• Images had affine mapping

• Rule I - Need Correct Registration

Results (Images - II)

Original Bicubic SR

Why didn’t SR work???• Low-res images were created by forcing shifts at

critical velocities• Rule II - If low-res frames are at critical velocities,

can’t create good high-res frame

Results (Frames - I)

Original Bicubic SR

Why did SR work so well???• Low-res images were created by forcing shifts at

non-critical velocities• Rule III If low-res images have all the info

about high-res then HR image can be perfectly constructed

Results (Frames - II)

• Superresolution with multiple motions between frames - create high res video

• Predict the high-res frequency components using wavelet methods

Predict Predict

Predict

Future Work

Acknowledgements

Prof John Apostolopoulos Prof Susie Wee Patrick Vandewalle

THANK YOU!!!

QUESTIONS?

COMMENTS?