Thomas Abraham, PhD(tabraham1@hmc.psu.edu)

• Deconvolution, also termed as “Restoration” or “Deblurring” is an image processing technique used in a wide variety of fields from 1D spectroscopy to 3D biomedical imaging.

• It has been rigorously applied in the field of astronomy, because so much expense and time are spent in acquiring astronomical images.

What is Deconvolution?

Raw Image Deconvolved

• An image is produced by an instrument. It is not the same as the object but is some “photocopy” of the object.

What makes an image?

Convolution

• The PSF describes what the instrument does to the object to produce an image.

• In mathematical terms, the image is the object convolved by the ‘point spread function’ (PSF).

Image = Object PSF

• Every lens element in the optical path alters the object (specimen) in some way.

• We call this the point spread function (PSF).

• The PSF of the system is a convolution of all of the PSF’s of all the lens elements

and apertures in the optical path.

Point Spread Function {PSF}

• The image of that macromolecular object can be considered as the superposition of large number of points convolved with the PSF.

• Likewise, a macromolecular object can be considered as a large number of points arranged in 3D space.

• For instance, an image of point source is not a point but a point spread in 3D space.

Point Spread Function {PSF}

Point Source

Image

Image(x,y,z) = ∫∫∫Object(x’,y’,z’)psf(x-x’,y-y’,z-z’)dx’dy’dz’

Convolution Theorem

In mathematical terms, for each point, take the object and multiply it by the entire PSF centered at that point in the image and sum the results of the multiplication.

Convolution Theorem

Whatever we record as the image intensity is the convoluted product of the PSF and the object intensity.

Restoring the object intensity from the smear of PSF by some mathematical operations

Image = Object PSF

Image(x,y,z) = ∫∫∫Object(x’,y’,z’)psf(x-x’,y-y’,z-z’)dx’dy’dz’

Deconvolution

The Fourier Transform

I(x) ai(coskix) ibi(sinkix)i

Amplitudes Phase

From spatial domain to Frequency Domain

•Any image can be decomposed into a series of sine and cosine wave functions.

Low frequency(Coarse structural details)

High frequency (Fine structural details)

Fourier Transform of the elastin collagen fiber bundle in lung alveoli

FT

Closely related to the diffraction pattern of the object

Spatial Domain Frequency Domain

PSD

Fourier Transform, Examples

FT

FT

Dude (Big Labowski)

Spatial Domain Frequency Domain

Bad looking Vs Good Looking

Fourier Transform, Examples

FT

FT

Spatial Domain Frequency Domain

Unidirectional Objects Vs Randomly Oriented ObjectsFrequency distribution

is orthogonal

Fourier Transform, Examples

FFT

FFT

Spatial Domain Frequency Domain

Fine Vs Fuzzy Objects

Inverse Fourier Transform of the FourierTransform Returns the original Image

F -1

A powerful mathematical tool for image processing

=

Deconvolution using Fourier Transformation

It has been proved that the Fourier transform of the image is just the Fourier transform of object times the Fourier transform

of the PSF in frequency domain

So, to obtain the object, we simply divide Fourier transform of Image by the Fourier transform of the PSF

F {Image} = F {Object PSF}

F {Object} = F {Image} / F {PSF}

This is known as Optical Transfer Function or OTF or CTF

F {Image} = F {Object} X F {PSF}

By applying an inverse Fourier transform, We can restore object

(specimen)

Transforms back to spatial domain

Object (or sample) = F {Object} -1

Step 1: Acquire the image and obtain its Fourier transform

Convoluted Image{Object

Fourier Transform of Image ⊗ PSF}

Steps in Deconvolution

Step 2: Generate PSF and obtain its Fourier transform

Steps in Deconvolution

Point spread function Fourier transform of PSF

Step 3: Divide and inverse Fourier transform

F -1

Steps in Deconvolution

Object

FT

FT

Raw Image

Deconvolved Image( Restored Object)

F {Object} = F{Image} / F{PSF}

OTF or CTF

Object (or sample) = F.T. {Object} -1

Works perfectly if there is no noise in the image. But there is always noise

Various Sources of NOISEPhoton (or shot) noise: Photon detection is a quantum mechanical event.  So, it is subjected to random fluctuations (or random noise) which is governed by Poisson statistics.

Electronic (thermal) noise: The light is detected electronically and all detectors have thermal noise.

Bit noise: Digital images have noise due to the number of bits (~8-14) used during A/D conversion.

Processing noise: The computer is operating on digital data with limited precision.

Effect of noise in data

More Noise

Relatively Noise FreeLess Noise

Each spatial frequency has an associated noise component resulting from various sources of the noise, and at high spatial frequencies (corresponds fine structural details) this can be very large.

Non-iterative Deconvolution

So simply dividing F{Image} by OTF will magnify noise components - this obscures real structural details.

Alternate approach is to modify OTF (or CTF) on the basis of local signal levels –Wiener filter.

F {Object} = F{Image} / F{PSF}

OTF or CTF

Deconvolution using Wiener filter

• Here an extra noise dependent factor (K) is added to the OTF values, so that in noisy regions the divisor is larger, and the noisy components are not boosted.

• One major drawback here is that we don’t actually know what is real signal and what is noise.

• This often results in rejecting genuinely real spatial information. Consequently the reconstructed image can be of lower resolution.

Deconvolution by Iteration1) Take a guess at the object2) Convolve guess (object) with the PSF. If PSF is unknown, guess the PSF too.3) If blurred guess = image then stop iteration, otherwise refine guess.

This is also known as maximum likelihood approach.

Richardson-Lucy algorithm (which includes Poisson Noise model) :

xy plane

xz plane

A

F

CB

D E

Abraham et al. (2010) Micron

xy plane

xz plane

A CB

Abraham et al. (2010) Micron

A B

C D

Abraham et al. (2010) Micron

• Imaging is a convolution of the object by the microscope’s PSF and an image is a convoluted product of the object and PSF.

• Deconvolution attempts to restore the various spatial frequency components to something nearer what they were in the original object before being attenuated by the optical path.

Summary

Recommended Readings:

Cannell, et al. (2006) Image Enhancement by Deconvolution, In Handbook of Biological Confocal Microscopy, Third Edition, Pawley, J. B, Ed., Springer Science + Business Media, LLC, New York

Shaw, P. (1994) Deconvolution in 3-D optical microscopy, HistochemicalJournal 26, 687-694.

Agard, D. A and Sedat, J. W. (1983) Three-dimensional architecture of a polythene nucleus. Nature 302, 676-81.

Sindelar, C.V and Grigorieff,N. (2011) An adaptation of the Wiener filter suitable for analyzing images of isolated single particles, J. Struct. Biol. 176, 60-74.

Sigworth, et al., (2010) An Introduction to Maximum-Likelihood Methods in Cryo-EM, Methods in Enzymology, Volume 482, 263-294