Bayesian Image Super-resolution, ContinuedLyndsey C. Pickup, David P. Capel, Stephen J. Roberts and Andrew Zisserman, Robotics Research Group, University of Oxford

Goal: Multi-frame super-resolution aims to produce a high-resolution image from a set of low-resolution images by recovering or inventing plausible high-frequency information. With more than one low-resolution image, high-frequency information can be recovered by making use of sub-pixel image displacements. The problem is made more difficult because the sub-pixel displacements -- as well as lighting changes and image blur parameters -- must be recovered from this noisy set of low-resolution input images.

Introduction Key Idea: Integrate over the unknown registrations

Generative model for super-resolution

The goal is to reconstruct the high-resolution image x (right) from a set of noise inputs {y(k)} generated according to the model above.

The image registrations can be viewed as “nuisance parameters” to be integrated out of the super-resolution problem in a Bayesian setting.

Maximum A Posteriori Solution

xMAP using a Huber prior, assuming perfect knowledge of image registrations.

The same problem, but with registrations perturbed by δ ~ N ( 0, 0.12 ) low-resolution pixels.

xMAP using a Gaussian image prior, giving softer edges.

Take a Taylor series expansion of the error in terms δ:

H is a block diagonal sparse matrix, so for many covariance assumptions, S is correspondingly sparse also. Typically C can be assumed diagonal, and L can be solved using gradient descent optimization methods.

Synthetic Data Example

Best Huber-MAP reconstruction. Error compared to known ground truth image.

RMSE = 15.5 grey levels.

Best registration-integrating reconstruction. Note improvement of black-white edge detail in some letters on the bottom line.

RMSE = 14.8 grey levels.

First and last of the 16 input images from the synthetic dataset.

Results on real data Inputs 1 & 10

Registration-Integrating (detail)

Huber-MAP (detail)

Tipping & Bishop(detail), extended to include photometric model.

10 input images, each with 4 registration parameters (two geometric, two photometric).

Huber-MAP: regularizes the smooth areas well, though gives slightly over-crisp edges.

Tipping & Bishop: typical Gaussian over-smoothing, though weakening the prior would risk over-fitting to the noise.

Registration-integrating: shows letters clearly but also gives good regularization on the constant white background areas.

Extra terms as an image priorThe two terms of L involving S can be viewed as an extra prior over the high-resolution image, favouring images which are not acutely sensitive to tiny perturbations in the registrations. Such a perturbation is enough to change the noise pattern on the ML estimate drastically. Two vectors δ1 and δ2 are drawn from N(0,I*0.042) and used to corrupt the registration on a synthetic dataset.

ground truth Input 1/16 Input 16/16 ML1 ML2 abs(ML1-ML2 )

This alternative: accept registration uncertainty, and integrate over the resulting distribution to find super-resolution image directly from inputs.

Most methods, including Tipping & Bishop’s Bayesian Image Super-resolution work, fix the registration first, then compute the high-resolution image.

Assume small perturbations or corrections δ in the set of registration parameters:

Now find –log(p(x|{y(k)})) by integrating over δ:

The regular Huber-MAP objective function for finding x is

where G(x) is the set of image gradients at each pixel, and is the Huber function, defined:

where Images were registered using a simple iterative intensity-based method, typically accurate to within 0.1 pixels. Outputs are at a zoom factor of 4.

Imagex.

Warp withparameters

Blur by point-spread

function.

Decimate by zoom factor.

Corrupt with additive

Gaussian noise.

Lighting changes with parameters λ

imagey(1)

imagey(K)

There are a number of advantages to integrating over the unknown registrations rather than the high-resolution image in Bayesian image super-resolution:

Integrating over registrations

1a – Can use good image priors such as the Huber prior, which give much better results than Gaussian priors.

1b – A Gaussian prior for the registration offsets is realistic.

2 – The matrix in the log-determinant term of the objective function is small (4*K).

3 – Requires little more memory than the standard Huber-MAP algorithm, so can handle large images.

Integrating over high-resolution image

1 – Must have a Gaussian prior in order for the integral to be tractable. This is not a good model for natural image statistics and tends to blur out the image edges.

2 – The matrix in the log-determinant term of the objective function is large because the matrices tend to be NxN.

3 – The full image-integrating approach can only be run on very small input images (e.g. 9x9 pixels).

Advantages of Registration-Integrating