DIGITAL HOLOGRAPHYAND DIGITAL IMAGE

PROCESSING

University of Alicante,Valencia, Spain,

Dec. 21, 2004

Digital holography and image processing: twinsborn by the computer era

Digital holography:- computer synthesis, analysis and simulation ofwave fieldsDigital image processing:- digital image formation;- image perfection;- image enhancement for visual analysis;- image measurements and parameter estimation;- image storage & transmission;- image visualization

•Digital reconstruction of of electronicallyrecorded optical holograms

•Target location and tracking

•Multi-component image restoration

•Image re-sampling with discrete sinc-interpolation

•Nonlinear (rank) filters for image restoration,enhancement and segmentation

•3-D visualization

DIGITAL HOLOGRAPHY:numerical reconstruction of

electronically recorded opticalholograms

Laser

Collimator

Beam spatialfilter

Lens

Microscope

Object table

DigitalPhoto-graphiccamera

Computer

One of the main drawbacks ofmicroscopy: the higher is the spatialresolution, the lower is depth of focus.

This problem can be resolved byholography.Holography is capable of recording 3-Dinformation. Optical reconstruction isthen possible with visual 3-D observation.Drawbacks of optical holography:

-Intermediate step(photographic developmentof holograms) is needed.-Quantitative 3-D analysisrequires bringing inadditional facilities

Radical solution: optical holography withhologram recording by electron means(digital photographic cameras) and digitalreconstruction of holograms. This is theprinciple of digital holographicmicroscopy.

Digital Holographic/Interferometric Microscopy

HologramFourier PlaneFirst focal plane Second focal plane

Digital Reconstruction of Holograms (Equivalent optical setup)

Hologramsensor

Preprocessingof digitizedhologram

Imagereconstruction(DFT/DFrT)

Imageprocessing

Hologram

Analog-to-digital

conversion

Outputimage

Computer

Digital Holography: Digital Reconstruction of HologramsM.A. Kronrod, N.S. Merzlyakov, L.P. Yaroslavsky, Reconstruction of a Hologram with a Computer,Soviet Physics-Technical Physics, v. 17, no. 2, 1972, p. 419 - 420

PSF, resolving power and speckle phenomena inreconstruction of electronically recorded optical holograms

Problems:

•How the point spread function of the reconstructionprocess depends on parameters of the optical set-up andrecording camera

•Resolving power of the reconstruction process

•Potential accuracy in measuring phase component of theobject wavefront

•Statistical properties of the reconstruction speckle noise

Discrete representation of optical transforms:Discrete Fourier Transforms

x

a(x) Sampled signal

xΔu

f

( )fαSampled signal spectrum

fΔ

v

( ) ( )( )xukxaxaN

kreconstr_signk Δ+−= ∑

−

=

1

0ϕ

( ) ( )( )fvrffN

kreconstr_spnr Δ+−= ∑

−

=

1

0ϕαα

Continuos signal

Continuous signalspectrum

( ) ( ) ( )dxfxiexpxaf ∫∞

∞−

= πα 2 ( )( )∑−

=⎟⎠⎞

⎜⎝⎛ ++=

1

021 N

kk

v,ur N

vrukiexpaN

πα

Fourier integral Shifted DFT (canonic form)

fx/N ΔΔ= 1

Signal and spectrum sampling

( )∑−

=⎟⎠⎞

⎜⎝⎛ +

⎟⎠⎞

⎜⎝⎛=

1

0

221 N

kk

v,ur N

rukiexpNkviexpa

Nππα ( )∑

−

=⎟⎠⎞

⎜⎝⎛ +

⎟⎠⎞

⎜⎝⎛−=

1

0

221 N

kk

v,ur N

vrkiexpNruiexpa

Nππα

Direct and inverse Shifted DFTs (reduced form)

L.P. Yaroslavsky, Shifted Discrete Fourier Transforms, In: Digital Signal Processing, Ed. by V. Cappellini, and A. G.Constantinides, Avademic Press, London, 1980, p. 69- 74.

Discrete representation of optical transforms:Discrete Fresnel Transforms

( ){ }∑−

=

+−−=1

0

2, //exp1 N

kk

wr Nwrkia

Nκκπα κ

( ){ }∑−

=

+−=1

0

2, //exp1 N

kr

wk Nwrki

Na κκπα κκ

Integral Fresnel Transform:

( ) ( ) ( )∫∞

∞−⎥⎦

⎤⎢⎣

⎡ −−= dxD

fxixaf2

exp πα

Discrete Fresnel Transforms

Signal and its transform discretizationwith shift basis functions

( ) ( )( )[ ]{ }xxukxsincxk ΔΔ+−= /πφ

( ) ( )( )[ ]{ }ffvrfsincfr ΔΔ+−= /πχ

2/1 fDxN ΔΔ= ( ) 2/1/ xf ΔΔ=κ

w is the overall shift w=uκ-v/κ 0 50 100 150 200 2500

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

q=0

q=0.0001

q=0.0002

q=0.0004

q=0.0008

q=0.0016

Absolute values of function for N=256 anddifferent “focusing” parameter q

( )rqNfrinc ;;

( ) ( )∑−

=⎟⎠⎞

⎜⎝⎛−=

1

0

2 2expexp1;;N

k Nkriqki

NrqNfrinc ππ

A pictorial representation of the discrete frinc-function(DFT of the chirp-function). Amplitude is shown in green,phase in red. Horizontal coordinate – argument of thefunction; vertical coordinate – focusing parameter. Binaryrandom noise is added just for fun.

2005

Digital reconstruction of samples of the object wave front amplitude fromsamples of its hologram is treated as a process of sampling the object wave front.Signal sampling is a linear transformation that is fully specified by its pointspread function:

( ) ( )∫=X

k dxkxPSFxaa ,

Wavefront

samples

Object’swavefront

Sampling devicePoint Spread

Function

Point spread function of numerical reconstructionof electronically recorded optical holograms

According to the sampling theorem,ideal sampling PSF is sinc-function

( ) ( )[ ]( )[ ]

( )[ ]xxkxxxkx

xxkxxkPSF

ΔΔπΔΔπ

ΔΔπ

−−=

−=sin

sinc,

For hologram recording in far diffraction zone, wave propagation kernel WP(x,f) is :

)2exp(),(Z

xfifxWPλ

π−=

Assume that, for hologram reconstruction, shifted and scaled DFT is used with the reconstructionkernel: ( )

⎥⎦⎤

⎢⎣⎡ +=

Nvrki

NrkDR T

σπ2exp1),(

where vT and σ are shift and scale parameters

With this reconstruction kernel, point spread function of the reconstruction process PSFFZ(x,k) is

( ) ⎥⎦

⎤⎢⎣

⎡⎟⎠⎞

⎜⎝⎛ −

⎭⎬⎫

⎩⎨⎧

⎥⎦

⎤⎢⎣

⎡⎟⎠⎞

⎜⎝⎛ −+−⎟

⎠⎞

⎜⎝⎛ −+−= xxkxNNvkNv

xxi

ZxkxPSF Tr

dFZ ΔσΔπ

σΔπ

λΦ ,sincd

21

212exp)(,

where ∫∞

∞−

⎟⎠⎞

⎜⎝⎛ −= df

Zxfif

Zx dd )2(exp)()(

λπϕ

λΦ

fNZSZx H ΔλλΔ == ( ) ( )( )NxN

xxNsinsin,sincd =

Define hologram discretization and reconstruction device coordinate system through the objectcoordinate system by choosing vr=vT=(N-1)/2. Then

( ) ⎥⎦

⎤⎢⎣

⎡⎟⎠⎞

⎜⎝⎛ −= xxkxN

ZxkxPSF r

dFZ ΔσΔπ

λΦ ,sincd)(,

is frequency response of the hologram sampling device and

PSF of reconstruction of holograms recorded infar diffraction zone

L. Yaroslavsky, F. Zhang, I. Yamaguchi, Point spread functions of digital reconstruction of digitally recorded holograms,In: Photonics Asia conference on Information Optics and Photonics Technology, 8-12 November 2004, Beijing, China

( ) ( )[ ]xxkxNZxkxPSF r

dFZ ΔΔπλ

Φ −= ,sincd)(,

PSF of reconstruction of holograms recorded in fardiffraction zone (ctnd)

As one can see from the equation,

The point spread function is a periodical function of k:

( ) ( ) ( ) ( );1 1 kPSFNgkPSF FZNgr

FZ r −−=+ σ

(g is integer). It generates σN samples of object wavefront masked by the frequency response of thehologram recording and sampling device, the samples being taken with discretization interval

Δx/σ = λZ/ σSH =λZ/ σNΔf

within the object size So= λZ/ Δf.

The case σ =1 corresponds to a “cardinal” reconstructed object wavefront sampled with discretizationinterval Δx= λZ/ SH =λZ/ NΔf . When σ >1 , reconstructed discrete wavefront is σ -times over-sampled,or σ -times zoomed-in. One can show that in this case the reconstructed object wavefront is a discretesinc-interpolated version of the “cardinal” one.

Discrete sinc-function ( ) ( )( )xNxN

xxxxNΔ

ΔπΔπsin

sin,sincd =

Discrete sinc-function is a discrete analog of the continuous sampling sinc-function, which is a pointspread function of the ideal low-pass filter. As distinct from the sinc-function, discrete sinc-function is aperiodical function with period NΔx or 2NΔx depending on whether N is an odd or an even number andits Fourier spectrum is a sampled version of the frequency response of the ideal low pass filter

N is an oddnumber

N is an evennumber

Continuous (red dots) and discrete (blue line)sinc-functions for odd and even number ofsamples N

NΔx 2NΔx

Frequency response of the ideal low pass filter(red) and Fourier transform of the discretesinc-function (blue)

Reconstructed object areaReconstructed object area

Statistical characterization of speckle noise incoherent imagingL. Yaroslavs ky, A. Shefler, Statistical characterization of speckle noise in coherent imaging systems, in: OpticalMeasurement Systems for Industrial Inspection III, SPIE’s Int. Symposium on Optical Metrology, 23-25 June 2003,Munich, Germany, W. Osten, K. Creath, M. Kujawinska, Eds., SPIE v. 5144, pp. 175-182

2-D array that specifiesamplitude component of

the object wave front

Generating 2-D array ofpseudo-random numbers

that specify the phasecomponent of the object

wave front

Computingobject’s wave

front

Simulating wavefront propagation

(DFT, DFrT)

Introducing signaldistortions:-Array sizelimitation-Dynamic rangelimitation-Quantization

Simulating wavefront

reconstruction(IDFT, IDFrT)

Comparingreconstructed and initialwave fronts; computing

and accumulation ofnoise statistical

parameters

Continueiterations?

Yes

NoOutput data

Illustrative examples of simulated images: a) - originalimage; b) - image reconstructed in far diffraction zone from0.9 of area of the wave front; c) - image reconstructed in fardiffraction zone from 0.5 of area of the wave front; d) -image reconstructed in far diffraction zone after limitation ofthe wave front orthogonal components in the range.

Computer model

a) b)

c) d)

TARGET LOCATION ANDTRACKING

Target localization in clutter in multi-componentimagesL. Yaroslavsky, Optimal target location in color and multi component images, Asian Journal of Physics, Vol. 8, No 3(1999) 355-369

Object tracking in video sequencies: examples

Tracking fetus movements in Ultrasound movie

For details see http://www.eng.tau.ac.il/~yaro

Leonid P. Yaroslavsky, Ben-Zion Shaick Transform Oriented Image Processing Technology for Quantitative Analysisof Fetal Movements in Ultrasound Image Sequences. In: Signal Processing IX. Theories and Applications, Proceedingsof Eusipco-98, Rhodes, Greece, 8-11 Sept., 1998, ed. By S. Theodorisdis, I. Pitas, A. Stouraitis, N. Kalouptsidis,Typorama Editions, 1998, p. 1745-1748

Face detection in complex imagesBen-Zion Shaick, L. Yaroslavsky, Object Localization Using Linear Adaptive Filters, 6th Fall Workshop,Vision, Modeling And Visualization 2001 (Vmv01), November 21-23, 2001, Stuttgart, GermanyStuttgart,Germany, November 21-23, 2001, pp. 11-17

http://www.eng.tau.ac.il/~yaro/Zion/ZionPhdSeminar4Web.pdf

The developed algorithm iscapable of detecting, withhigh reliability, faces ofvarying size from minimumsize of 12 pixels width and15 pixels height to themaximum size of the inputimage.

The face detection capabilityof the developed system wasexperimentally examined ontwo test databases of imagesof high and low quality. Thedetection rates 96% and84% were achieved for thesedatabases, respectively.

Non-face

1st stage

Face

Maybe face

2-nd stage

Face-like-non-face

Non-face -like-face

Face detection: two-stage algorithm

The “non-face” detection algorithm was proved to have “non-face” rejecting rate of ~99% and false alarm rate of1.3% (faces wrongly rejected), thus leaving only 1% of the image area for subsequent thorough analysis by the “multi-template classification” algorithm. The algorithm is fast and requires approximately 200 flops per pixel in an inputimage of 640480 pixels size.

“Multi-templateclassification” algorithms usea very large set of templatesprepared for different targetshapes and varyingillumination conditions.The developed algorithmswere trained using a speciallycreated training databaseobtained by extending four“face” databases to 32,000images and one “non-face”database to one millionimages by means of scalingand rotating database images.In particular, “face”, “non-face”, “faces like clutter” and“clutter like faces” templateswere generated from thesetraining databases.

Face detection: Face-like-non-face and non-face-likeface data bases

MULTI COMPONENTIMAGE RESTORATION

Spatial/temporal adaptive linear filtersL. Yaroslavsky, Image Restoration, enhancement and target location with local adaptive filters, in:International Trends inOptics and Photonics, ICOIV, ed. by T.Asakura, Springer Verlag, 1999, pp. 111-127

3-D Local adaptive spatial-temporal filtering:denoising and deblurring thermal video (ctnd)

3-D Local adaptive spatial-temporal filtering:denoising and deblurring thermal video

Before After(5x5x5 DCT domain filtering)

L. Yaroslavsky, A. Stainman, B. Fishbain, Sh. Gepstein, Processing and Fusion of Thermal and Video Sequences forTerrestrial Long Distance Observation Systems Processing and Fusion of Thermal and Video Sequences forTerrestrial Long Distance Observation Systems, ISIF, Seventh International Conference on Information Fusion(FUSION 2004), Stockholm, Sweden, 28 June - 1 July 2004 .

IMAGE RESAMPLING:discrete sinc-interpolation

Image resampling and geometrical transformations:efficient discrete sinc-interpolation algorithms

Spline interpolationand discrete sinc-interpolation

Spectra of DFT-error (le ft) and Beta11-error (right)

Test random image

Spectrum of DFT-error (red) andBeta11 error (green)

Spectrum of DFT-error (green)and bicubic-error (red)

Discrete sinc-interpolation for image resizing

Image (a) is reduced with reduction factor 77/256 and then magnified with magnification factor256/77. b) – bilinear interpolation; c) – bicubic interpolation; d) – nearest neighbor interpolation;e) – sincd-interpolation 11x11; f) – sincd interpolation 31x31

Discrete sinc-interpolation in DCT domain

Initial signal ( )ka

DCT

DCTrα

{ }DCTrα

DFT of zero-pad

sinc-interpolation

kernel

( ){ }prη

IDST

p-shifted sinc-interpolated signal{ }pka +

IDCT

( ){ }prerη

( ){ }pimrη

+ - Zooming an image fragment (left) by sinc-interpolationin DFT domain (right upper image) and in DCT domain(right bottom image). Oscillations due to boundaryeffects that are clearly seen in DFT-interpolated imagecompletely disappear in DCT-interpolated image.

L. Yaroslavsky, Boundary effect free and adaptive discrete sinc-interpolatioon, Applied Optics, 10 July 2003, v. 42, No.20, p.4166-4175

Sliding window sinc-interpolationDCT domain: Simultaneous imageresampling and restoration/enhancement

Inputsignal

InverseDCT/DST

∑Computing slidingwindow

DCT

Outputsignal

Introducing p-shift

Modification ofthe spectrum for

restoration/enhancement

Sliding window sinc-interpolation in DCTdomain: signal resampling and denoising

Noisy image (a) and a result of the rotation and denoising withsliding window DCT sinc-interpolation and denoising (b).

a)

b)

Outputsignal

Computingsliding window

DCT

Discrete DCT domainsinc-interpolator

Nearest neighborinterpolator

Analysis of localspectrum

Inputsignal

Mixer

Sliding window sinc-interpolationDCT domain: local adaptive interpolation

Adaptive versus non adaptive signal interpolation

Comparison of nearest neighbor, linear, bicubic spline and adaptivesliding window sinc interpolation methods for zooming a digital signal(From left to right, from top to bottom: Continuous signal; initialsampled signal; nearest neighbor -interpolated signal ; linearly -interpolated signal; cubic spline -interpolated signal; sliding windowsinc-interpolated signal ).

Signal (upper plot) shift by non-adaptive (middle plot) and adaptive(bottom plot) sliding window DCT sinc-interpolation. One can noticedisappearance of oscillations at the edges of rectangle impulses wheninterpolation is adaptive.

Image rotation with adaptive and non-adaptivediscrete sinc interpolation

Direct Fourier method for inverse Radon transform:polar-to- Cartesian coordinate spectrum conversion withdiscrete sinc-interpolation

Object

Projections

Projection spectra

Interpolated 2-Dobject spectrum

Reconstructedimage

Fourier method for inverse Radon transform:polar-to- Cartesian coordinate spectrum conversion bymeans of zooming with variable zooming factor(L. P. Yaroslavsky, Y. Chernobrodov, Sinc-interpolation methods for Direct Fourier Tomographic Reconstruction, 3-d Int.Symposium, Image and Signal Processing and Analysis, Sept. 18-20, 2003, Rome, Italy)

Initial Polar Grid

Polar Grid after 2time zooming

Target Cartesian Grid

??

a

b

r1

r2

Initial Polar Grid

Target Cartesian Grid

1-D spectra of projections

radiusangle

S. Gepshtein, A. Shtainman, B. Fishbain, and L.P. Yaroslavsky, Restoration of atmospheric turbulent video containingreal motion using rank filtering and elastic image registration, Eusipco2004, Vienna, Austria, Sept. 2004

Stabilization and restoration of atmosphericturbulent video

Initial videoRestored stabilized video

with moving objectsunaffected

Moving objects

Turbulent atmosphere video

Stabilized turbulent atmosphere video

NONLINEAR (RANK)FILTERS

for image restoration,enhancement and segmentation

Image restoration and enhancement: nonlinear filtersL. Yaroslavsky, Nonlinear Filters for Image Processing in Neuromorphic Parallel Networks, Optical Memory and NeuralNetworks, v. 12, No. 1, 2003B. Hirshl, L. Yaroslavsky, FPGA implementation of sorters for non-linear filters, EUSPCO 2004, Vienna, Austria, Sept. 2004

Noisy image, stdev = 20, Pn=0.15

Iterative SCSigma-filter . Wind.5x5, Evpl=Evmn=15; 5 iterations

Initial image SIZE(Evnbh(Wnbh5x5,2,2))-filter HIST(W-nbh)-filter

Nonlinear filters: Image enhancement

Initial image

RANK(KNV(Wnbh15x15;113))

RANK(Wnbh15x15)

RANK(EV(Wnbh15x15;10,10))

Nonlinear Filters: Image EnhancementLocal histogram equalization: Wnbh, EV-nbh and KNV-nbh

Local P-histogram equalization: color images(blind calibration of CCD-camera images)

3-D VISUALIZATION:improved anaglyph method and

3-D video from 2-D video

Computer synthesis and display of stereoscopic imagesI. Ideses, L. Yaroslavsky, Efficient Compression and Synthesis of Stereoscopic Video, 2nd IASTED Internat.Conference, Visualization, Imaging and Image Processing (VIIP 2002), Sept. 9-12, 2002, Malaga, Spain, Ed. J.J.Villanueva, Acta Press, Anaheim, Calgary, Zurich, 2002, pp. 191-194 (ISBN: 0-88986-354-3; ISSN: 1482-7921)http://www.eng.tau.ac.il/~yaro/Ideses/malca1.html

Bahai garden, Haifa, Israel

Roman Querry, Tarragona, Catalonia

Computer generated stereo from 2-D videoI. Ideses, L. Yaroslavsky, A Method for Generating 3D Video from a Single Video Stream, in: G. Greiner, H.Niemann, T. Ertl, B. Girod, H.-P. Seidel (Eds.), Vision Modeling and Visualization 2002, Proceedings, Nov. 20-22,2002, Erlangen, Germany, Academishe Verlagsgesellschaft Aka GmbH, Berlin, 2002, ISBN 3-89838-034-3, ISBN 1-58603-302-6, p. 435-439

Current projects:

Multi-component image processing:•Restoration and fusion of atmospheric turbulent, thermal and visible range video•Super-resolution from video sequencies•3-D color display and artificial 3-D from video•Moving object detection and tracking

Digital holography:•Point spread functions and resolving power of digital reconstructionof near and far zone holograms

•Theory and new algorithms for fast transforms

L. Yaroslavsky,Ph.D., Dr. Sc. Phys&Math,

ProfessorDept. of Interdisciplinary Studies,Faculty of Engineering, Tel Aviv

University, Tel Aviv, Israelwww.eng.tau.ac.il/~yaro