Design of an Adaptive Scanning Optical Microscopefor Simultaneous Large Field of View

and High ResolutionBenjamin Potsaid, Yves Bellouard, and John T. Wen

Center for Automation TechnologiesRensselaer Polytechnic Institute

Troy, NY 12180.Emails: {potsaid,bellouard}@cat.rpi.edu, wenj@rpi.edu

Abstract— In microsystems applications from micro-assemblyto biological observation and manipulation, the optical micro-scope remains one of the most important tools. However, itsuffers from the well known trade-off between resolution and fieldof view. Traditional solutions involve moving the sample underthe microscope using a moving stage or moving the microscopeitself, and switching between low and high magnification objectivelenses. In this paper, we present a new optical microscope designthat uses a 2-dimensional high speed, high precision steeringmirror system to scan the sample. By stitching the imagestogether as a mosaic, we have the potential to achieve bothhigh resolution and large field of view. Working in coordina-tion with a deformable mirror, this arrangement offers certainadvantages over the current state of the art in demanding spatial-temporal observations. We describe the theory of operation,our design methodology, and present a preliminary simulateddesign. A reduced functionality experimental prototype has beenconstructed to demonstrate the basic efficacy of the conceptand we demonstrate with both biological and micro-assemblyexamples.

I. INTRODUCTION

The capabilities of optical imaging systems, which have tra-ditionally consisted of extremely accurate and well calibrated,but primarily static components, are rapidly expanding withthe introduction of robotic and motion control technologiesapplied to the optical elements. These adaptive and dynamicimaging systems allow the pursuit of scientific investigations,military applications, and medical diagnostics that are beyondthe theoretical capabilities of a purely static optical design.Along with the recent growth of Biotechnology and Micro-Electro-Mechanical Systems (MEMS) there is a growing needto observe, interact with, and inspect at a scale below thethreshold of the naked human eye. The optical microscopehas been, and will continue to be a critical tool as thesefields advance. While this need has inspired a resurgence ofinterest in the optical microscope, the essential optical designand operating principle has not changed significantly in thelast century and the optical microscope still suffers from awell known inherent tradeoff between the field of view andresolving power of the system. This paper seeks to showthat our new Adaptive Scanning Optical Microscope (ASOM)concept can effectively address this tradeoff, and can offer

certain advantages over the current state of the art. We dothis by integrating active optical elements, motion control, andimage processing techniques into the design.

The need for a large field of view at high resolutionoccurs in biological science, industrial manufacturing, medicaldiagnostics, etc., and there are several modern approaches toaddress the situation where a single image can not satisfy bothrequirements. These solutions include: a moving stage locatedunderneath the microscope, parfocal (equal focus) objectivelens switching, zoom lens designs, and moving microscopes.Each method has advantages and disadvantages, which aresummarized in Table I. The ASOM design that we presentin this paper will excel in applications where critical spatial-temporal observations are demanded, but will not offer thevirtually unlimited field of view associated with the movingstage do to optical limitations. Biological applications wherethe ASOM would be attractive are: observing dynamic cellularevents (mitosis, viral attachment, motility) over a large popula-tion of living cells. In industry, the ASOM will allow for visionguided micro-assembly and rapid inspection of completedparts, with the potential for higher product throughput.

The motivation for the ASOM came from the micro-assembly and packaging activities at the Center for Automa-tion Technologies at Rensselaer Polytechnic Institute and theconceptual layout was inspired by a machine designed at EcolePolytechnic Federal de Lausanne (EPFL), for laser annealingshape memory alloy [1].

The key idea is a simple one: use a low mass steeringmirror [2], [3] positioned in the middle of the optical pathto scan the images. The advantages of such an arrangementare: a large effective field of view at high resolution, nodisturbance to the sample, and high bandwidth operation.However, such a system also poses significant challenges. Incontrast to the moving stage or moving microscope designs,there is extensive off-axis imaging (i.e., images are obtainedby looking diagonally through the lens), which introducesdistortion, contrast degradation, and loss of resolution [3]. Weaddress the off-axis aberrations by: (1) explicitly incorporatingfield curvature into the design and (2) introducing an actuateddeformable mirror (DM) into the optical path for wavefront

No

spec

imen

agita

tion

duri

ngsc

anni

ng

Pres

erve

sre

solv

ing

pow

erw

hile

expa

ndin

gfie

ldof

view

Eas

ym

anuf

actu

ring

inte

grat

ion

over

conv

eyor

tran

spor

t

Scan

ning

Ban

dwid

th

Eas

ilyre

confi

gure

dfo

rdi

ffer

ent

view

ing

task

s

Multiple Parfocal Objectives X LOWZoom Lens Design X X MED XMoving Stage X MED XMoving Microscope X X X LOW XMultiple Microscopes X X X HIGHASOM X X X HIGH X

TABLE I

QUALITATIVE COMPARISON OF ASOM TO OTHER TECHNOLOGIES

correction.In this paper, we first discuss wide field of view and

high resolution imaging in Section II. Section III-A describesthe key features of the ASOM; Section III-B presents ourinitial design approach; and Section III-C presents simulatedperformance results. Finally, Section IV describes our currentexperimental proof-of-concept implementation with demon-strations in micro-assembly and biological observation.

II. WIDE FIELD OPTICS AND IMAGING

The design of wide field and high resolution microscopicimaging systems are driven by consideration of (1) an imagesampling issue and (2) an image quality issue. First, consideran imaging system with optics that are nearly perfect (i.e. theoptical aberrations are much below the diffraction limit). Sucha system will image two point sources separated by a distance,d, as two overlapping Airy patterns in the image field. As thedistance between the two points decreases, a critical distancewill be reached, r, where the two points can no longer beindividually distinguished. According to the Rayleigh criteria,this critical distance, called the resolution, is equal to the radiusof the central core in the Airy pattern and is related to thenumerical aperture, NA, of the system and the wavelength oflight, λ, by:

r =0.61λ

NA, (1)

where the NA of the system is defined by the index ofrefraction of the transmitting medium, n, and the half angle, θ,of the cone of light collected from the object: NA = nsin(θ).

A digital camera must sample with two pixels per Airy coreradius to avoid aliasing according to the Nyquist samplingcriteria. This observation provides the following maximumtheoretical object field width, Wo, for a sensor array pixelcount per edge, k, and resolution, r:

Wo =kr

2. (2)

While microscopic imaging systems are often designedwith resolutions in the 1

4µm to several µm range, the lower

practical limit on CCD camera pixel size is approximately6µm due to noise effects. Therefore, the optics must enlargethe Airy pattern to achieve proper sampling with the requiredmagnification factor, M , for a given pixel size, s is given by:M = 2s/r. The corresponding image size, Wi, is: Wi = ks.Figure 1(a) illustrates the above mentioned equations andrepresents the imaging optics as a generic black box. Theoptical design task is to specify the design of the imagingsystem, i.e., to fill in the details of the black box with specificlens or mirror geometries, glass types, and spacing.

Fig. 1. (a) Imaging with black box optics, (b) microscope objective, (c)lithography lens. The lens prescriptions for the microscope objective andlithography lens were obtained from [4] and are not shown to scale.

The first intuitive approach to designing a large field andhigh resolution imaging system might be to take an existingmicroscope layout, such as that shown in Figure 1(b), andsimply increase the pixel count of the camera while redesign-ing the optics to achieve a larger field of view. This approachmay indeed be possible, but it is not generally practical as therequirements for field size, flat field, and numerical aperturesoon approach those of lithography lenses. The 1998 Nikon

lithography lens (US Patent 5,805,344) shown in Figure 1(c)has a 0.65 NA with field sizes of 93.6mm and 23.4mm. Lithog-raphy lenses require near perfect manufacturing and extremelytight assembly tolerances (often requiring an interferometricassembly process), and can cost in the millions of dollars[4]. Also notice the presence of negatively powered elementslocated at the narrow beam regions in both the microscope andlithography lenses. This design technique is used to achieve aflat imaging field (small Petzval sum) and results in an increasein the lens count. An addition consideration is the size of theimage sensor, considering that large commercially availableCCD cameras only have approximately 9216x9216 pixels(e.g. Fairchild Imaging CCD595). Smaller CCD arrays canbe mosaiced to achieve larger pixel count with the advantageof being able to read data in parallel, but at a cost of additionalprecision assembly requirements. Consideration of these issuesmotivated the design of the ASOM.

III. ASOM CONCEPT, DESIGN, AND SIMULATION

A. Theory of operation

The ASOM operates by taking a sequence of small spatiallydisplaced images in rapid succession and then assembling alarge virtual image (mosaic) of the scene. The concept ofexpanding the field of view while preserving resolving powerthrough mosaic construction is well established and has beenapplied to biological imaging [5] as well as industrial imaging[6]. However, instead of a moving stage as is common, themechanism and scanning principle in the ASOM consists ofa high speed steering mirror working in coordination witha specially designed scanning lens assembly, a deformablemirror, and additional imaging optics. A conceptual layout ofthe ASOM is shown in Figure 2.

Fig. 2. Conceptual layout of the ASOM

Curved field scanning layout- Different than a microscopeobjective or lithography lens, the scanning lens on the ASOMis designed to exhibit significant field curvature with a rel-atively large Petzval sum. This relaxation of the flat fieldrequirement offers the advantage of a greatly simplified opticaldesign with far fewer lens elements (the advantages of curvedfield designs have been recognized for aerospace applications[7]). Additional characteristics of the ASOM scanning systeminclude:

1) The center of the field curvature, the rotation centerfor the 2-D steering mirror, the mirror surface, and anoptical pupil plane are all mutually coincident.

2) The shape of the projected image surface is nearlyspherical instead of the more typical parabolic surfaceassociated with field curvature. This is achieved throughhigher order aberration control.

Under the above mentioned conditions, as the steeringmirror angle changes, the projected curved image surfacerotates about its own center as shown in Figure 3. Stationaryimaging optics with a matching negatively curved field workwith a frame stop to sample a portion of the image surface,providing for an image scanning and selection mechanism asthe steering mirror angle changes. This layout exhibits (1)a large positively curved field associated with the scanninglens, and (2) a small negatively curved field associated withthe imaging optics, thus avoiding the significant difficultyof designing and manufacturing a large continuous flat fieldimaging system as discussed in Section II.

Fig. 3. Field curvature of scanning lens assembly

Deformable mirror wavefront correction - The scanninglayout is such that imaging is performed significantly off thecentral axis of the scanning lens assembly. Such extremeoff axis imaging generally introduces considerable opticalaberration and requires a highly compensated lens assemblywith a prohibitively large lens count. We circumvent thisproblem by designing a ”good” scanning lens with significantwavefront aberration (up to several waves of optical pathdifference) and then use a deformable mirror to compensate forthe aberrations over the specific viewing field that is selected.

Variation in the aberration is allowed between individual fieldpositions throughout the scanner’s range. However, given thatthe deformable mirror can only achieve one specific shapeat a time, the rate of change in the aberration between fieldpositions must be small enough to allow diffraction limitedimaging performance over the entire sub-field of view that isselected. This is similar to the concept of the isoplanatic patchin the atmosphere [8] that is widely recognized in the adaptiveoptics telescope community. By analogy with the ASOM, theisoplanatic patch of the scanning lens must be larger than theselected field of view.

B. Design process

Modern optical design is generally performed using the sim-ulation and optimization capabilities of commercially availablelens design software. Local minima searching algorithms canoften improve an initial design and are highly dependant on thestarting values of the design variables. However, generating anew design form requires a global search. Applied directly tothe ASOM design problem, the ZEMAX genetic algorithms(global search) proved ineffective because of the size (numberof design variables) and complexity of the system (requiresmultiple configurations to accommodate the different mirrorangles and DM shapes required for each field position).Instead of an all-in-one global optimization approach, wepartitioned the system into (1) a scanning lens assembly and(2) the imaging and wavefront correcting optics as shownin Figure 3. The scanning lens assembly was designed as afully static system (no moving elements) for a curved fieldwith up to several waves of aberration (with the assumptionthat the deformable mirror would eventually correct for thisaberration). The imaging and wavefront correcting optics werethen designed separately for diffraction limited performanceover the entire field with the deformable mirror held flat.The compatibility of the two systems at the interface wasmaintained by human intervention with much trial and error.Next, the two subsystems were joined with the steering mirrorand deformable mirror added to the system. The local op-timization algorithm and the multi-configuration capabilitiesof ZEMAX allowed for the system wide optimization of thelens geometries and spacing while simultaneously consideringmultiple field positions, steering mirror angles, and specificdeformable mirror shapes. Future work will investigate moreformal and automated design approaches and will likely drawheavily from the field of Multidisciplinary Design Optimiza-tion (MDO).

C. Specifications and predicted performance

A practical design resulting from the optimization describedin Section III-B is shown in Figure 4. The simulated resultsthat follow demonstrate that the ASOM can effectively providean expanded field of view while preserving resolution whencompared to existing microscope technologies. Table II listsperformance specifications of the specific ASOM design de-scribed in this paper, but with suitable changes to the design,the field area and numerical aperture could be tailored to the

Fig. 4. ASOM preliminary design

observation task at hand. However, in general, as the field areaincreases, the realizable NA will decrease due to physical andpractical limitations.

SpecificationEffective field of view diameter 40mmTotal Observable Field Area 1257 mm2

NA 0.21resolution 1.5 µmMagnification 15.2Camera pixel count 512 X 512Camera pixel size 10µm

TABLE II

PRELIMINARY ASOM PERFORMANCE SPECIFICATIONS

Figure 5 compares the observable field of view of theASOM to a fixed microscope with a 4096x4096 camera(considered a full field camera with standard microscopeobjectives) and with a 1024x1024 camera which is morecommon. The ASOM offers diffraction limited (Strehl ratio> 0.8) for all field positions based on high fidelity simulatio.The field sizes for the fixed microscope designs assume perfectimaging and were calculated using Equations 1 and 2 using a0.21 numerical aperture with λ = 0.510µm for the wavelengthof light (green light is relatively nondestructive and desirablefor imaging living biological cells).

Fig. 5. Field size comparison

We next demonstrate the wavefront correcting capabilitiesof the deformable mirror. The µDM100 DM from Boston Mi-cromachines with 100 electrostatic actuators, a 3.3mm roundaperture, and 2µm stroke was used in this design. Instead ofmodeling all 100 actuators, we created a user defined surface inZEMAX consisting of a set of 37 evenly spaced Gaussian basis

functions to represent the mirror surface shape. We assume thatthe surface generated with the low spatial frequency gaussianshapes can be reproduced by the deformable mirror within themaximum actuator stoke. The amplitude of each basis functionbecomes a variable in ZEMAX and can be optimized withthe other glass geometry and spacing variables as well as thesteering mirror angle.

Figure 6 shows how the DM corrects for the specificwavefront aberration associated with each field position. Overthe entire field and for all field positions, the Strehl ratiois greater than the diffraction limit of 0.8, resulting in nearperfect imaging. Figure 7 shows the worst case Modulus of theTransfer Function (image contrast vs. object spatial frequency)for the ASOM design. For comparative purposes, an opti-mization was performed with the same initial design, exceptwith no deformable mirror in the system (i.e. the DM shapewas fixed to be flat). The system with no deformable mirrorshows considerably degraded MTF imaging performance as isillustrated in the simulated images of biological cells.

Fig. 6. (a) viewing different field positions (b) the optimal deformable mirrorshape for the specific field position (c) the Strehl ratio sampled over theselected field of view

The results presented are based on idealized simulationsignoring the reality that lenses and optical housings are alwayssubject to manufacturing and assembly tolerances. We willconsider these aspects in future work.

IV. EXPERIMENTAL PROTOTYPE

To demonstrate the basic principle of scanning and imagemosaic construction, we have built a first generation prototype[9] called the Scanning Optical Mosaic Scope (SOMS), shownin Figure 8. No formal optimization of this design was per-formed, and the prototype unit was constructed using standard

Fig. 7. MTF and cell image of system with and without a deformable mirror

Fig. 8. Working Prototype of the SOMS

catalog lenses available from ThorLabs, a Sony XC-77BBCCD camera, Matrox Meteor II frame grabber, Cambridgetechnologies galvanometers and servo drivers, and a TI basedDSP board. It differs from the more advanced ASOM designproposed in this research in that: (1) the optical layout issimplified, (2) there is no deformable mirror or adaptive optics,(3) all lenses are available as standard catalog items, (4) thescanning lens is a single standard achromat doublet.

A. Micro-assembly and biological demonstrations

The micro-manipulation demonstration is based on a shapememory alloy microgripper moving between two fixed objectsin a workspace. A rudimentary correlation based image match-ing algorithm and Kalman filter are used to track the motionof the gripper tip. A 3 X 3 tile mosaic images the gripperand the scanning pattern is automatically adjusted to maintainthe tip in the center tile. The scan pattern also includes thetwo stationary objects in the workspace, demonstrating thecapability of the SOMS to observe multiple stationary andmoving objects in the workspace nearly simultaneously. Asequence of the video footage is shown in Figure 9.

Figure 10 shows a video sequence of living cells. Sev-eral events of mitosis (cell division) can be seen occurringthroughout the viewing field. The ASOM not only offers thepossibility of automatically detecting the onset of mitosis andother events, but can be easily programmed to track and record

Fig. 9. Microgripper example

multiple events at the same time. While automated quantitativecell analysis using a moving stage has recently been proposed[10], the bandwidth of the overall system is still constrainedby the response of the stage and the sensitivity of the cellspecimen to motion. The ASOM will address both of theseissues.

V. CONCLUSION

We have presented a new microscope concept that cansimultaneously achieve high resolution and large field of viewthat offers several advantages over the current state of theart for observing certain spatial-temporal events. A proof-of-concept prototype has been constructed to demonstratethe basic efficacy of this approach. The next generationof hardware will include a deformable mirror with customdesigned optics and will be based on further systematic designoptimization and consideration of manufacturing errors andassembly tolerances. The calibration and online optimizationof the active optical elements is a significant challenge andwill be a focus of future work.

ACKNOWLEDGMENT

The authors would like to thank Richard Cole and Dr.Jacques Izard at the Wadsworth Center, New York StateDepartment of Health. Their expertise and knowledge ofmicroscopy and cellular processes have contributed greatlyto the project. This research is also supported in part by theCenter for Automation Technologies under a block grant fromthe New York State Science and Technology Foundation.

REFERENCES

[1] Y. Bellouard, T. Lehnert, R. Clavel, T. Sidler, and R. Gotthardt, “Laserannealing of shape memory alloys: A versatile tool for developing smartmicro-devices iv,” J. Phys., vol. 4, no. 8, pp. 571–576, 2001.

[2] J. Montagu, “Two-axis beam steering systems, TABS,” in Proc. of SPIEVol. 1920, 1993, pp. 162–172.

[3] D. Weiner, “Design considerations for optical scanning,” in Proc. ofSPIE Vol. 3131, 1997, pp. 59–64.

[4] W. J. Smith, Modern Lens Design, Second Edition, SPIE, Ed. TheMcGraw-Hill Companies, 2005.

[5] A. Jalink, J. McAdoo, G. Halama, and H. Liu, “Ccd mosaic techniquefor large-field digital mammography,” IEEE Transactions on MedicalImaging, vol. 15, no. 3, pp. 260–267, 1996.

Fig. 10. Mosaic of Live Cell Sequence Taken with SOMS Prototype

[6] C. Guestrin, F. Cozman, and S. Godoy, “Industrial applications of imagemosaicing and stabilization,” in 1998 Second International Conferenceon Knowledge-Based Intelligent Electronic Systems, L. Jain and R. Jain,Eds., Adelaide, SA, Australia, Apr. 1998, pp. 174–183.

[7] J. M. Rodgers, “Curved focal surfaces: Design optimization throughsymmetry, not complexity,” Photonics Tech Briefs - Online, 2003,http://www.ptbmagazine.com/content/040103 ora.html.

[8] V. M. G. R. T. Fusco, J.-M. Conan and F. Assemat, “Multi-conjugateadaptive optics: Comparison of phase reconstruction approaches forlarge field of view,” in Proc. of SPIE Vol. 4167: Atmospheric Pro-pogation, Adaptive Systems, and Laser Radar Technology for RemoteSensing., 2001, pp. 168–179.

[9] B. Potsaid, Y. Bellouard, and J. Wen, “Scanning optical mosaic scope formicro-manipulation,” in Int. Workshop on Micro-Factories (IWMF02),2002.

[10] F. Ianzini, L. Bresnahan, L. Wang, K. Anderson, and M. Mackey, “Thelarge scale digital cell analysis system and its use in the quantitativeanalysis of cell populations,” in 2nd Annual Int. IEEE-EMBS SpecialTopic Conf. on Microtechnologies in Medicine & Biology, Madison, WI,2002, pp. 469–475.