Computational Restoration of Fluorescence Images_Noise Reduction, Deconvolution And Pattern...

8/14/2019 Computational Restoration of Fluorescence Images_Noise Reduction, Deconvolution And Pattern Recognition

1/12

CHAPTER 16

Computational Restoration of FluorescenceIm ages: Noise Reduction, Deconvolution,and Pattern Recognition

Yu-li WangDep artm en t o f P hy sio lo gyU niv ersity of M assach usetts M ed ical Scho olWo rcester, M assa ch usetts 0 16 05

I . Int roduct ionII. A daptive N oise F iltrationI II . DeconvolutionIV. Pattern R ecognition-B ased Im age R estorationV. P ro sp ec tu s

References

I . I ntr oduction

Optical m icroscopy is now perform ed extensively in conjunction with digimaging. A number of factors contribute to the limitations of the overperformance of the instrument, including the aperture of the objective lthe inclusion of light originating from out-of-focus planes, and the limisignal-to-noise ratio of the detector. Both hardware and software approachhave been developed to overcom e these problem s. The form er include im pro

objective lenses, confocal scanning optics, total internal reflection optics,high-performance cam eras. The latter, generally referred to as com putationim age restoration, include a variety of algorithm s designed to reverse the defof the optical-electronic system . The purpose of this chapter is to introduce inintuitive way the basic concept of several computational image-restoratiotechn iqu es that are particularly usefu l for pro cessing fluorescence images.

METHODS IN CELL BIOLOGY, VOL 72Copyr igh t 1998 , E l sev ie r I n c. A l l r ig h ts r e se r ved .0091-679X/03 $35.00

33 7


2/12

338 Yu-Ii Wang

II. A daptive N oise Filtration

Noise, d efined as th e uncertain ty of intensity measuremen t, o ften represeprimary factor limiting the performance of both confocal microscopevarious computational approaches. B ecause of the statistical and accumnature of photon detection, noise may be reduced m ost effectively by incthe time of image acquisition (Fig. lA, B, G, H). However, this approachimpractical for biological im aging because o f th e pro blem s of sample movpho to bleachin g, an d rad iation damage.An optimal computational noise filter should aggressively remove r

fluctuations in the image while preserving as much as possible the nonrfeatures. Filters that apply a uniform algorithm across the image, suchpass convolution and median filters, usually lead to the degradation ofresolution or amplification of artifacts (Fig. 1). For example, low-passcause both image features and noises to spread out over a defined area, wmedian filters replace every pixel w ith the m edian value of the neighborho

Better performance may be obtained using filters that adapt to thecharac te ristic s o f the image. In an iso tropic d iffu sion filter (B lacke t a t., 1 9 9Tvarusko e t a t., 1999), lo ca l in ten sity flu ctu atio ns a re redu ced th rough a simdiffusion process by diffusing from pixels of high intensity toward neighpixels of low intensity. This process, when carried out in multiple itereffectively reduces intensity fluctuations by spreading the noise around aof p ixels . In one implementa tion (Blacke t a t. , 1998), the change in in ten sity ogiven pixel during a given iteration is calculated as

A*"L.g(b.I)b./, (1 )

where A is a user-defined diffusion rate constant, b./ is the difference in infrom a neighb oring pix el, an d the sum"L .is taken o ver all eight neigh bors. Has diffusion w itho ut co nstraints would lead to h omogenization of th e entireit is c ritic al to in tro duce th e functio ng(b.I) fo r th e purpose o f p re serv ing stru ctufeatu res. In one of the algorithms, it is defined as 1 /2 [1- (b./la)2f when Ib.Ijis~aand 0 o therwise (B lacke t a t., 1998). The constanta th us defin es a th resholdintensity transitions to be preserved as structures. If the difference in intelarger thana , then diffusion is inhibited[g(b./)= 0].Although this diffusion m echanism works w ell in rem oving a large frac

the noise, the recognition of features based on b./ is not ideal, as it also proutlying noise pixels. The resulting "salt and pepper" -type noise (Fig. 2A ,

have to be rem oved subsequently with a m edian filter. In an im proved algth e diffusion is controlled not b y in tensity d ifferences between neig hboringbut by edge structures detected in the im ages. If an edge is detected across(e.g., u sing co rrelation-b ased pattern reco gnition), th en d iffu sion is allowalong the edge; otherwise, diffusion is allowed in all directions. In essen


3/12

16. Computational Restoration of Fluorescence Images 339

Fig. 1 Fluorescence images either unprocessed (A, B; G, H) or processed with a standard mefilter (C, 0) or low-pass convolution filter (E , F). M ouse fibroblasts were stained with rhodaphalloidin, and im ages collected w ith a cooled CCO camera at a short (20 m s, A , B ) o r long (200 mH) exposure. A region of the cell is m agnified to show details of noise and structures. M ediancreates a peculiar pattern of noise (C , 0), w hereas low -pass convolution filter creates fuzzy dotsslig ht d egradation in resolu tio n (E , F ).


4/12

340 Yu-li Wang

Fig. 2 N oise reduction w ith the original anisotropic diffusion filter according to Tukey (B lacke t a t. ,1 998 ;A , B ), w ith a mod ified an iso trop ic d iffu sion filter that d etects edg es (C , D ), an d w ith an adw indow edge-detection filter (E , F). Im ages in (A -D ) w ere processed by 10 iterations of difResidual noise outside the cell was easily noticeable in (A ) and (B), w hereas the noise w ascom pletely rem oved in F without a noticeable penalty in resolution. The original im age is shFig. IA and B.

algorithm treats structures in the im age as a collection of iso-intensity lineconstrain the direction of diffusion (Fig. 2C, D).An alternative ap proach is the adaptive-w indowed edge-detection filter (

and Weeks, 1993). The basic concept is to remove the noise by averaintensities w ithin local regions, the size of which is adapted according to theinformation content. In one implementation, the stanaard deviation o f intenis first calculated in a relatively large square region (e.g., I I x I I) surroundipixel. If the standard deviation is low er than a defined threshold, indicatingno structure is present w ithin the square, then the center intensity is replacthe average value w ithin the region and the calculation is m oved to the next


5/12


Otherwise, the size of the square is reduced and the process is repeated. If theof the square is eventually reduced to 3 x 3 and the standard deviation remhigh, then the central pixel is likely located at a structural border. In this casedge-detection operation is used to determine the orientation of the line, andintensity of the central pixel is replaced by the average of the pixels that lie athe detected line. This procedure is repeated at each pixel of the im age (Fig. 2EBoth anisotropic diffusion and adaptive-window edge-detection meth

can reduce the noise by up to an order of magnitude while preserving structboundaries (Fig. 2). However, neither of them can restore lost resolutcaused by high noise (cf Fig. IG and H with Fig. 2). In addition, both of thpreserve pseudofeatures form ed by stochastic distribution of noise pixels. Wanisotropic diffusion, areas of uniform signal may become populated wlump lik e artifac ts, whe re as w ith adaptiv e-w indow edge-d etec tio n filte rs, strabun dle structu res may show ajagged con to ur (Fig . 2 ). D esp ite these lim itationsreduction of noise alon e can g reatly impro ve the qu ality of noisy confocal im agethree-dimensional reconstruction or for further processing (Tvaruskoe t a t.,1999).

I I I .Deconvolu t ion

Mathem atical reversal of system ic defects of the m icroscope is known as im"deconvolution." This is because the degradation effects of the m icroscope cadescribed mathematically as the "convolution" of input signals by the pospread function of the optical system [Young, 1989; for a concise introductionconvolution, see the book by Russ (1994)]. The two-dimensional convoluoperation is defined m athem atically as

i(x,y) (9 5(X,y) = L i(x -u,y

- V)5(U, v).u,v(2)

This equation can be easily understood when one looks at the effectsco nvo lving a matrixi with a 3 x 3 matrix:

.. . . . . .i1i2i3 . . . . . . 515253. . . . . .i4i5i6 . . . . .. (9 545556. . . . . .hi8i9 . . . . . . 575859.

.

The calculation turns the elem ent (pixel)is into (il x 59) + (i2 x 58) + (i3 x 57) +(i4 x 56) + (is x5S)+ (i6 x 54) + (i7 xi3 ) + (is x i2 ) + (i9 x il). When calcula tedover the entire image, each pixel becomes "contaminated" by the surroundp ix els, to an extent sp ec ifie d by th e v alu es in th e 5ma trix (th e poin t-sp read functThus, two-dimensional convolution operation m imics im age degradation bydiffraction through a finite aperture. Three-dimensional convolution cans imila rly exp re ssed as


6/12

342 Yu-li Wang

o(x,y,z) = i( x,y,z ) Q9s(x ,y,z ), (3

where i(x,y,z) is three-dim ensional m atrix describing the signal originatingthe sample, s(x,y,z) is a m atrix describing the three-dim ensional point-sfunction, which contains the information on image degradation causedd iffraction and ou t-o f-focus sig nals, an do(x,y,z) is the stack of optical slices

d etec ted w ith th e m ic ro scope.It should be noted that such precise mathematical description ofdegradation does not mean that original signal distribution can beunambiguously from the detected im ages and the point-spread function.of the loss of high-frequency information when light passes through thaperture of the objective lens, it is im possible to obtain predegraded im aunder ideal conditions. The collected inform ation instead allow s a greatof potential "answers" to fit Eq. (3). This is because the calculation is plaa number of indeterminants as a result of "zero divided by zero." As astraightforw ard reversal of equation 3 (referred to as inverse filtering)leads to images that fit the mathematical criteria but contain unacc

artifacts an d amplified n oise.Th e challenge of computatio nal deco nvolutio n is to fin d strateg ies th at

the m ost likely answer. A num ber of deconvolution algorithm s have beefo r resto ring fluo rescen ce imag es (for a d etailed review, see Wallaceet a t., 2 001)The two m ain approaches in use are constrained iterative deconvolution1984; Agard et at., 1989; Holmes and Liu, 1992) and nearest neigdeconvolution (Castlem an, 1979; Agard, 1984; Agardet at., 1989) . Constraineiterative deconvolution uses a trial-and-error process to look fordistribution i(x,y,z), that satisfies Eq. (3). It usually starts with the assumthat i(x,y,z) equals the m easured stack of optical sections,o(x,y,z). As expec tedwhen o(x,y,z) is placed on the right-hand side of Eq. (3) in place ofi(x,y,z), igenerates a matrixo'(x,y,z) that d eviates fromo(x,y,z) on the left-hand side.decrease this deviation, adjustment is made to the initial guess, voxel bbased on the deviation ofo'(x,y,z) from o(x,y,z) and on constraints suchno nnegativity of vox el v alues. Th e modifiedo(x,y,z) is then placed back intoright-hand side of Eq. (3) to generate a new matrix,ol/(x,y,z), which sh oulre semble more c lo se lyo(x,y,z) if the adjustm ent is made properly. This procrepeated until there is no further improvement or until the calculatedm atches closely the actual im age.Various approaches have been developed to determine the adjustm

calcu la te d images b etween itera tio ns (Agarde t a t., 1989; Holm es and Liu, 19

Different algorithm s differ with respect to the sensitivity to noise andspeed to reach a stable answer, and resolution of. the final image. Undcond ition s, constrained iterativ e deco nvo lutio n can ach ieve a resolu tionbeyond what is predicted by the classical optical theory (Raleigh crand provide three-dimensional intensity distribution of photom etric a(Carrington et at., 1995). The main drawback of constrained iter


7/12


deconvolution is its computational demand, which typically takes minutehours to complete a stack. This delay m akes troubleshooting particularly tedA second disadvantage is its requirement for a complete stack of imagesdeconvolution, even if the experim ent requires only one optical slice. Iteraconstrained deconvolution is also very sensitive to the condition of the im ainclu ding optical d efects an d sign al-to -noise ratio.The nearest-neighbor algorithm is based on the decomposition of th

dimensional con volution [Equ ation (2)] into a series of two-d imen sion al co nvt ions (Agarde t a t.,1989):

o (x , y )= io (x , y )1 ;9so (x ,y ) + L I (x ,y ) 1 ; 9S - I(x ,y ) + i+ l(x , y )1 ;9S + 1(x ,y )

+ L2 (X , y ) 1 ;9L2 (X , y ) + i +2 (X ,y ) 1 ;9S+2 (X , y ) + .. . ,(4 )

where i(x,y)'s are two-dim ensional m atrices describing the signal originatifrom the plane of focus(io), and from the planes above (i+J,i+2, ...) and below(i-I, L2, . . .),s(x,y) are m atrices of tw o-dim ensional point-spread functions tdescribe how point sources on the plane offocus (so) or planes above (S+I, S+and below (LJ, S-2. ...) spread out when they reach the plane of focus.eq uatio n is further simplified b y in tro ducing three assumption s: first, that oufocus light from planes other than those adjacent to the plane of focunegligible; that is, term s containing L2, S+2,and bey ond are in sign ificant. Sethat light originating from planes immediately above or below the plane ofcan be approximated by images taken while focusing on these planes; i.e., i0-1 and i+1 ~ 0+1. Third, that point-spread functions for planes immediaabove and below the focal plane, LI and S+I, are equivalent (hereafter denas SI).

These approxim ations sim plify Eq. (4) into(5 )

Rearranging the equation and applying Fourier transformation, the equatu rns into

(6 )

where F and rl represents forward and reverse Fourier transformationrespectively. This equation can be understood in an intuitive way: It statesthe unknown s igna l d is tribution,io , can be obtained by taking the in-focus im age0, subtracting out estim ated contributions from planes above and below the pof focus, (0-1 + 0+1)1 ;9 S J ,fo llowed by th e convolu tio n w ith the matrix rl[l/F(so)] to reverse diffraction-introduced spreading of signals on the plane of foTo address artifacts introduced by the approximations, nearest-neigh

deconvolution is typically perform ed with a m odified form of Eq. (6):

(7 )


8/12

344 Yu-li Wang

where constants Cl and C2are empirical factors. Cl is used to offset errorsby oversubtraction, as described below. C2 is required to deal with proassociated w ith the calculatio n o f reciprocals,l/F(so), at the end of Eq. (6). Thconstant C2 keeps the reciprocal value from getting too large when theelem ent is small com pared toC2'In general, Cl falls in the range of 0.45 to 0.depending on the separation betw een adjacent optical slices, w hereas the vC2is in the range of 0.0001 to 0.001, depending on the level of noise in the iThe optim al values for Cl and C2should be determined through system aticThe m ain concern of nearest-neighbor deconvolution is its accuracy beca

the three app roximation s discussed abov e. In particu lar, as the neig hbo rin gsectio ns inclu de significan t contribution s of signals from the p lane of focus,of these images leads to oversubtraction and erosion of structures, particwhen the optical section is performed at too small a spacing. Howeverapproxim ations also greatly sim plify and accelerate the calculations. It rethe collection of only three im ages (the in-focus image and im ages immedabov e an d b elow ), which red uces pho to bleaching d uring image acq uisition.

constrained iterative d econv olu tion, th e compu tation is finished w ithin secoaddition, it is relatively insensitive to the quality of the lens or the point-sfunction, yielding visually satisfactory results with either actual or comcalculated point-spread functions. When the constant C l is set sm all enoughthe im age itself m ay be used as an approxim ation of the neighbors. The re"no-neighbor" deconvolution is in essence identical to "unsharp-m asking"in m any graphics program s such as Photoshop. N earest-neighbor deconvocan serve as a quick-and-dirty means for qualitative deblurring. It is idepreliminary evaluation of samples and optical sections, which may thp rocessed with constra ined ite ra tive deconvo lu tion.There are cases where confocal microscopy would be m ore suitable than

deconvo lu tion techn iqu es. Both con train ed iterative and n earest-neig hbo rvolution require im ages of relatively low noise. Although noise m ay be reb efo re and durin g th e c alcu la tio n w ith filte rs, as d iscussed earlie r, in pho ton-lcases, th is may n ot g ive accep table resu lts. In addition , laser con focal m icroscmore suitable for thick samples, where th e d ecreasin g penetration and scattelight limit the performance of deconvolution. For applications with modemand in resolution and light penetration, but high demand in speedconvenien ce, sp in nin g disk confo cal m icroscopy serves a unique purp ose. Funder ideal situations, the com bination of confocal im aging and deconvoshou ld y ie ld th e h ighe st p erfo rmance .

IV. Pattern Recognition-Based Image Restoration

Deconvolution as discussed above is based on few assumptions about theHowever, in many cases the fluorescence image co ntains sp ecific ch aracterwhich may be used as powerful preconditions for image restoration. One


9/12


example is the detection of micro tubules in immunofluorescence or Gim ages. The prior knowledge, that all the features in these images consist ofstructures, allows one to take an alternative approach based on the detectionlinear features.Pattern recognition is performed most easily with a correlation-base

algorithm. The corre la tio n ope ratio n,

i(x,y) + (x,y) = L i(x + u,y + v)s (u , v ),u,v

(8 )

is similar to that of convolution except that the corresponding (insteadtransposed) elem ents from the tw o matrices are multiplied together; that is, wa matrix i is correlated with a 3 x 3 matrix, s,

.i]i2i3""" S]S2S3

. . . . . . i4 is i 6 . . . . ..+ S4SSS6 .. . . . . . i7 is i9 . . . . . . S7SSS9

.

The element (pixel) is turns into (i] x Sj) + (i2 xS2)+ (i3 x S3)+ (i4 x S4)+ (is x ss)+ (i6 x S6)+ (i7 x i7 ) + (is x is) + (i9 xi9).It can be easily seen that this result is maximal when the two matrices con

an identical pattern of numbers, such that large numbers are multiplied by lnumbers and small numbers multiplied by small numbers. The operatithus provides a quantitative m ap indicating the degree of m atch betw een the im(i) a nd th e p atte rn(s ) at each pixel. To make the calculation more useful, Eq. (8)

mod ifie d a s

i(x,y) + (x,y) = L[i(x + u,y + v) - ia vg ][s( u, v ) - Savg] (9 )U.V

to generate both positive and negative results; negative values indicate thatmatrix is the negative image of the other (i.e., bright regions correspond toregion s, and v ice v ersa).Figure 3 shows the matricess(u,v) for detecting linear segments in eight

different orientations. The operations generate eight correlation im ages,one detecting segments of linear structures (micro tubules) along a cerorientation. Negative correlation values are then clipped to zero, and the imare added together to generate the final image (Fig. 4). This is a much faoperation than iterative deconvolution, yet the results are substantially bethan that provided by nearest neighbor deconvolution. However, despitesuperb image quality, the method is not suitable for quantitative analysisintensity distr ibution.


10/12

346 Yu-Ii Wa ng

1 .000 2 .000 4 .000 2.000 1 .0001 .000 2 .000 4 .000 2. 00 0 1 .0 001 .000 2 .000 4 .000 2.000 1 .0001.000 2 .000 4 .000 2.000 1 . 0001 .000 2 .000 4 .000 2.000 1 .000

0 .386 1 . 310 2 .468 3.684 1 . 918

0 .769 1 .693 3 .234 2.918 1 .5351 .152 2 .152 4 .000 2.152 1 .1521 .535 2 . 91 8 3 .234 1.693 0 .7691 .918 3 .6 84 2 .46 8 1. 310 0 .3 86

0 . 1 72 0 .879 1 .586 2. 58 6 4 .0 000 .879 1 .5 86 2 .58 6 4. 00 0 2 .5 861 .586 2 .586 4 .000 2.586 1 .5862. 586 4 .0 00 2 .586 1. 586 0 .8794 .000 2 .586 1 .586 0. 87 9 0 .1 72

0 .386 0 .769 1 .152 1. 535 1 . 9181 .310 1 .693 2 .152 2.918 3 .684

2.468 3 .234 4 .000 3 .234 2 .4683 .684 2 . 91 8 2 .152 1. 693 1 .3101 .918 1 .535 1 . 152 0.769 0 .386

1 .000 1 .000 1 .000 1.000 1 .0002.000 2 .000 2 .000 2.000 2 .0004.000 4 .000 4 .000 4.000 4.0002 .000 2 .000 2 .000 2.000 2 .0001 .000 1 .000 1 .000 1. 000 1 . 000

1 .918 1 .535 1 .152 0.769 0 .3863 .684 2 . 91 8 2 .152 1. 693 1 .3 102.468 3 .234 4 .000 3.234 2 .468

1 .310 1 .6 93 2 .15 2 2 . 91 8 3 .6840. 386 0 .7 69 1 .152 1. 535 1 .918

4 .000 2 .586 1 .586 0.879 0 . 17 22 .586 4 .000 2 .586 1. 586 0 .8791 .586 2 .586 4 .000 2.586 1 .5860. 879 1 .5 86 2 .586 4. 00 0 2. 5860 . 1 72 0 .879 1 .586 2.586 4.000

1. 918 3 .6 84 2 .468 1. 310 0 .3861 .535 2 . 91 8 3 .234 1.693 0 .7691 .152 2 .152 4 .000 2.152 1 .1520 .769 1 .6 93 3 .23 4 2. 91 8 1 .5 35

0.386 1 . 310 2 .468 3.684 1 .918

Fig. 3 Matrices fo r de tec t ing l inear struc tu res ly ing a long ' the direc t ion s o f no rt h -sout h ,north-northeast, n o rt h ea s t, e a st - no r th e as t , e a st -we s t, we st - no r thwes t , n o rt hwe s t, an d

north-northwest.


11/12

i. Com putational Restoration of Fluorescence Im ages 34 7

Fig. 4. Restoring of m icro tubules by feature recognition. A n optical slice of a N RK cell stainedmicro tubules is shown in (A). The image is processed to detect linear segments lying alongeast-west direction (B). A com plete image, obtained by summing structures along all the ed irectio ns, is sh own in (C ).

V. P ro spe ctu s

Image processing and restoration represent a rapidly advancing fieldinformation science. However, not all the algorithms are equally suitablefluorescence microscopic images, because of their special characteristics.optical microscopy continues to push for maximal speed and minimal irradiationovercoming the limitation in signal-to-noise ratio will likely remain a majchallenge for both the retrieval of information from unprocessed images andapplication of various restoration algorithms. Development of optimal noireduction algorithms represents one of the most urgent tasks for the yearscome. Although this chapter explores several simple methods for noise reductionew methods based on probability theories may be combined with deconvolutionto overcome the resolution limit imposed by noise. In addition, as many biologicstructures contain characteristic features, the application of feature recognitioand extraction will play an increasingly important role in both light and electrmicroscopy. The ultimate performance in microscopy will likely come frocombined hardware and software approaches, for example, by spatial/ tempormodulation of light signals coupled with software decoding and processin(Gustafsson et al., 1999; Lanni and W ilson, 2000).

~eferences

Agard, D . A . (1984).Ann. R ev. B io phys. B io en g.13, 191-219.Aga rd , D . A ., H iraoka , Y., Shaw, P., and Sedat, J. W. (1989).Methods CellBioi.30, 353-377.B lack , M . J., S apiro, G ., Marimont, D . H ., an d H eeger, D . (19 98).IEEE Trans . Image Proce ss .7,421-432.Carrington, W. A ., Lynch, R . M ., Moo re , E . D ., Is enbe rg ,G ., Fogarty, K . E ., and Fay, F. S . (1Science268, 1483-1487. J

Castleman, K. R. (1979) ."Digital Image Processing" Prentice-Hall , Englewood Cliffs , NJ.Gus tafs son, M . G ., Agard , D . A ., a nd Sedat, J . W. (1999 ).J . Micros,c .195, 10-16.Holmes, T. J ., and Liu , Y.-H. (1992) .In "Visual ization in BiomedicalMicroscopies" (A. Kriete edpp. 283-327. VCH Press, New York.

T . ~."...r"t'T "

........nnn.. T ",.T_- ~-T___~__" /n 'T--_..._~ A T---= -_ ..J A T ? '_ _ _ _ _ ..L_...I~\


12/12

348 Yu-li W ang

Myler, H. R., and Weeks, A. R. (1993). "Computer Imaging Recipes in c." Prentice-Hall, EngleC liffs, N J.

Russ, J. C. (1994). "The Image Processing Handbook." CRC Press, Boca Raton,FL .Tvarusko, W ., Bentele, M., M isteli, T., Rudolf, R., Kaether, c., Spector, D. L., Gerdes, H. H

E ils, R . (1 999 ). Proc. Natl. Acad. Sci. USA 96 , 7950-7955.Wallace, W ., Shaefer, L. H., and Swedlow, J. R. (2001).BioTechniques 31 , 1076-1097.Young, 1. T. (1989).Meth ods C ell B io i.30 , 1-45.

Computational Restoration of Fluorescence Images_Noise Reduction, Deconvolution And Pattern...

Documents

Transcript of Computational Restoration of Fluorescence Images_Noise Reduction, Deconvolution And Pattern...