Design and characterization of thin multiple aperture...

Design and characterization of thin multipleaperture infrared cameras

A. Portnoy,1 N. Pitsianis,1 X. Sun,1 D. Brady,1,* R. Gibbons,2 A. Silver,2

R. Te Kolste,3 C. Chen,4 T. Dillon,4 and D. Prather4

1Duke University, Durham, North Carolina 27708, USA2Raytheon Company, McKinney, Texas 75071, USA3Tessera, Inc., Charlotte, North Carolina 28262, USA

4University of Delaware, Newark, Delaware 19716, USA

*Corresponding author: [email protected]

Received 18 December 2008; revised 2 March 2009; accepted 10 March 2009;posted 16 March 2009 (Doc. ID 105471); published 6 April 2009

We describe a multiple-aperture long-wave infrared camera built on an uncooled microbolometer arraywith the objective of decreasing camera thickness. The 5mm thick optical system is an f =1:2 design witha 6:15mm effective focal length. An integrated image is formed from the subapertures using correlation-based registration and a least gradient reconstruction algorithm. We measure a 131mK NETD. Thesystem’s spatial frequency is analyzed with 4bar targets. With proper calibration, our multichannel in-terpolation results recover contrast for targets at frequencies beyond the aliasing limit of the individualsubimages. © 2009 Optical Society of America

OCIS codes: 110.1758, 100.6640, 110.3080.

1. Introduction

This paper describes thin cameras operating in thelong-wave infrared (LWIR) band (8–12 μm) using a3 × 3 lenslet array instead of a thicker single aper-ture optic. We refer to each of the nine individualsubimaging systems as an aperture. Our design in-tegrates optical encoding with multiple aperturesand digital decoding.The goal of this system is to use multiple shorter

focal length lenses to reduce camera thickness.Aperture size limits imaging quality in at leasttwo aspects. The aperture diameter translates to amaximum transmittable spatial bandwidth, whichlimits resolution. Also, the light collection efficiency,which affects sensitivity, is proportional to the aper-ture area. A lenslet array regains the sensitivity ofa conventional system by combining data from allapertures. The lenslet array maintains a camera’s

etendue while decreasing effective focal length.However, the spectral bandwidth is reduced.

The use of multiple apertures in imaging systemsgreatly extends design flexibility. The superior opti-cal performance of smaller aperture optics is the firstadvantage of multiple aperture design. In an earlystudy of lens scaling, Lohmann observed that f =#tends to increase as f 1=3, where f is the focal lengthin mm [1]. Accordingly, scaling a 5 cm f =1:0 opticaldesign to a 30 cm focal length system would increasethe f =# to 1.8. Of course one could counter this degra-dation by increasing the complexity of the optical sys-tem, but this would also increase system length andmass. Based on Lohmann’s scaling analysis, one ex-pects the best aberration performance and thinnestoptic using aperture sizes matching the diffractionlimit for required resolution. In conventional design,aperture sizes much greater than the diffraction-limited requirement are often used to increase lightcollection. In multiaperture design, the light collec-tion and resolution functions of a lens system maybe decoupled.

0003-6935/09/112115-12$15.00/0© 2009 Optical Society of America

10 April 2009 / Vol. 48, No. 11 / APPLIED OPTICS 2115

A second advantage arises through the use of mul-tiple apertures to implement generalized samplingstrategies. In generalized sampling, a single con-tinuously defined signal can be reconstructed fromindependently sampled data from multiple nonre-dundant channels of lower bandwidth. This advan-tage lies at the heart of Thin Observation Moduleby Bound Optics (TOMBO)-related designs.Third, multiple aperture imaging enables more

flexible sampling strategies. Multiple aperturesmay sample diverse fields of view, color, time, and po-larization projections. There is a great degree of vari-ety and flexibility in the geometry of multipleaperture design, in terms of the relationships amongthe individual fields of views and their perspectivesto the observed scene. We focus in this paper, how-ever, on multiple aperture designs where every lensobserves the same scene.The Computational Optical MONTAGE Photogra-

phy Initiative (COMP-I) Infrared Camera, CIRC,uses digital superresolution to form an integratedimage. We recognize that electronic pixels often un-dersample the optical field. For LWIR in particular,common pixel pitches exceed the size needed to sam-ple at the diffraction-limited optical resolution. InCIRC the pixel pitch is 25 μm, which is larger thanthe diffraction-limited Nyquist period of 0:5λf =#.We will show that this means image informationcan be recovered at a higher resolution with aproperly designed sampling scheme and digitalpostprocessing.In recent years, digital superresolution devices

and reconstruction techniques have been utilizedfor many kinds of imaging systems. In any situation,measurement channels must be nondegenerate ornonredundant to recover high-resolution informa-tion. An overview of digital superresolution devicesand reconstruction techniques was provided byPark et al. [2]. While numerous superresolutionapproaches gather images sequentially from a con-ventional still or video camera, the TOMBO systemby Tanida et al. [3] is distinctive in that multipleimages are captured simultaneously with multipleapertures. Another data driven approach byShekarforoush and Chellappa [4] makes use of nat-ural motion of the camera or scene. This paperaddresses digital superresolution, which should notbe confused with optical superresolution methods.While digital superresolution can break the aliasinglimit, only optical superresolution can exceed the dif-fraction limit of an optical system. The best possibleresolution that can be obtained by CIRC cannot bebetter than the diffraction limit of each of the ninesubapertures.CIRC was inspired by the TOMBO approach but

differs in the design methodology and in its spectralband. The diversity in multiple channel samplingwith lenslets is produced primarily by design withminor adjustment by calibration [5], instead of rely-ing solely on the inhomogeneity produced in fabricat-ing the lenslets. CIRC operates in the LWIR band

rather than the visible band. Previously, we havereported on the development and results of thin ima-ging systems in the visible [6] range and the LWIRband [7], respectively.

This paper describes a thin multiple apertureLWIR camera that improves on previous work in anumber of ways. We use a more sensitive, larger focalplane array, an optical design with better resolution,and a modification in subsequent image reconstruc-tion. These changes give rise to significantly higher-resolution reconstructions. Additionally this paperprovides a detailed noise analysis for these systemsby describing noise performance of the multichanneland conventional systems in the spatial frequencydomain.

In the remainder of this paper we provide addi-tional motivation for the use of multiple aperturesin imaging systems. We outline the main trade-offsconsidered in our system’s design and describe theexperimental system. An algorithm to integratethe measured subimages is presented, and weinclude sample reconstructions to compare perfor-mance against a conventional system. Finally,numerical performance metrics are investigated.We analyze both the NETD and the system’s spatialfrequency response.

2. System Transfer Function and Noise

This section describes how the architectural differ-ence between the multiaperture camera and the tra-ditional camera results in differences in modulationtransfer, aliasing, and multiplexing noise. We pre-sent a system model and transfer function for multi-aperture imaging systems. Noise arises from aliasingin systems where the passband of the transfer func-tion extends beyond the Nyquist frequency definedby the detector sampling pitch. Multiaperture ima-ging systems may suffer less from aliasing, howeverthey are subject to multiplexing noise.

Digital superresolution requires diversity in eachsubimage. CIRC offsets the optical axis of each lens-let with respect to the periodic pixel array. The lat-eral lenslet spacing is not an integer number ofpixels, meaning the pixel sampling phase is slightlydifferent in each aperture. The detected measure-ment at the ðn;mÞ pixel location for subaperture kmay be modeled as

gnmk ¼Z

∞

−∞

Z∞

−∞

Z∞

−∞

Z∞

−∞

f ðx; yÞhkðx0 − x; y0 − yÞpkðx0

− nΔ; y0 −mΔÞdxdydx0dy0

¼Z

∞

−∞

Z∞

−∞

f ðx; yÞtðx − nΔ; y −mΔÞdxdy;ð1Þ

where f ðx; yÞ represents the object’s intensity distri-bution and hkðx; yÞ and pkðx; yÞ are the optical pointspread function and the pixel sampling function forthe kth subaperture, respectively. Δ is the pixelpitch. Shankar et al. [7] discussed multiple aperture

2116 APPLIED OPTICS / Vol. 48, No. 11 / 10 April 2009

imaging systems based on coding hkðx; yÞ as a func-tion of k, and Portnoy et al. [6] discussed systemsbased on coding pkðx; yÞ. We focus here on the simplersituation in which the sampling function is indepen-dent of k and the difference between the images cap-tured by the subapertures is described by a shift inthe optical axis relative to the pixel sampling grid,such that, i.e., hkðx; yÞ ¼ hðx − δxk; y − δykÞ. In thiscase, Fourier analysis of the sampling function

tðx; yÞ ¼Z

∞

−∞

Z∞

−∞

hkðx0; y0Þpðx − x0; y − y0Þdx0dy0 ð2Þ

yields the system transfer function (STF)

j tðu; vÞj ¼ jhðu; vÞpðu; vÞj: ð3Þ

Neglecting lens scaling and performance issues,the difference between the multiaperture and con-ventional single aperture design consists simply ofthe magnification of the optical transfer functionwith scale. Figure 1 compares the STFs of a conven-tional single lens camera and a 3 × 3 multichannelsystem. The plots correspond to an f =1:0 imagingsystem with pixels that are 2.5 times larger thanthe wavelength, e.g., 25 μm pixels and a wavelength

of 10 μm. As in Eq. (1), all apertures share identicalfields of view. For this plot, pixels are modeled as uni-form sampling sensors, and their corresponding pixeltransfer function has a sinc-based functional form.The differing magnifications result in the conven-tional pixel transfer function being wider than themultiaperture case. Since the image space NA andthe pixel size are the same in both cases, the aliasinglimit remains fixed.

The conventional system aliases at a frequencyualias ¼ 1=ð2ΔÞ. The aliasing limit for multichannelsystem is determined by the shift parameters. IfΔxk ¼ Δyk ¼ kΔ=3, then both systems achieve thesame aliasing limit. The variation in samplingphases allows the multiple aperture system to matchthe aliasing limit of the single aperture system. Thedifference between the two systems is that the pixelpitch and sampling pixel size are equal to each otherin a single aperture system, but the sampling pixelsize is effectively 3 times greater than the pixel pitchfor the multiple aperture system.

Noise arises in the image estimated from gnmk fromoptical and electronic sources and from aliasing. Inthis particular example, one may argue that under-sampling of the conventional system means thataliasing is likely to be a primary noise source. A

0 0.05 0.1 0.15 0.2 0.25 0.30

0.2

0.4

0.6

0.8

1Conventional Camera

0 0 .05 0.1 0.1 5 0.2 0.2 5 0.30

0.2

0.4

0.6

0.8

13 by 3 Multiple Aperture Camera

PTFOTFSTF

PTFOTFSTF

Fig. 1. Comparison in STFs between a conventional system and a multiple aperture imager. The vertical line at 0.2 depicts the aliasinglimits for each sampling strategy.


simple model accounting for both signal noise andaliasing based onWiener filter image estimation pro-duces the means square error as a function of spatialfrequency given by

ϵðu; vÞ ¼ Sf ðu; vÞ1þ jSTFðu; vÞj2 Sf ðu;vÞ

Snðu;vÞþjSTFaðu;vÞj2Saðu;vÞ; ð4Þ

where Sf ðu; vÞ and Snðu; vÞ are the signal and noisepower spectra, respectively, and STFaðu; vÞ andSaðu; vÞ are the STFand signal spectrum for frequen-cies aliased into measured frequency ðu; vÞ, respec-tively. As demonstrated experimentally inSection 5, the multichannel and baseline systemsperform comparably for low spatial frequencies. Re-construction becomes more challenging for higherspatial frequencies as the STF falls off quicker inthe multichannel case (see Fig. 1). If aliasing noiseis not dominant, then there is a trade-off betweenform factor and noise when reconstructing highspatial frequency components. Of course, nonlinearalgorithms using image priors may substantiallyimprove over the Wiener mean square error.The ratio of the error for a multiple aperture and

single aperture system as a function of spatial fre-quency is plotted in Fig. 2. We assume a uniformSNR of 100 across the spatial spectrum. The uppercurve assumes that there is no aliasing noise, inwhich case the STF over the nonaliased range deter-mines the image estimation fidelity. In this case, bothsystems achieve comparable error levels at low fre-quencies, but the error of the multiple aperture sys-tem is substantially higher near the null in themultiple aperture STF and at higher frequencies.The middle curve assumes that the signal level inthe aliased band is 10% of the baseband signal. Inthis case, the error for the multiple aperture systemis somewhat better than the single aperture case at

low frequencies but is again worse at high frequen-cies. In the final example the alias band signal levelis comparable to the baseband. In this case, the lowertransfer function of the multiple aperture system inthe aliased range yields substantially better systemperformance at low frequencies relative to the singleaperture case.

The point of this example is to illustrate that whilethe ideal sampling system has a flat spectrum acrossthe nonaliased band and null transfer in the aliasedrange, this ideal is not obtainable in practice. Prac-tical design must balance the desire to push thespatial bandpass to the aliasing limit against the in-evitable introduction of aliasing noise. Multiple aper-ture design is a tool one can use to shape the effectiveSTF. One can imagine improving on the current ex-ample by using diverse aperture sizes or pixel sam-pling functions to reduce the impact of the basebandnull in the multiple aperture STF.

It is interesting to compare this noise analysis withan analysis of noise in multiple aperture imagingsystems developed by Haney [8]. Haney focuses onthe merit function

M ¼ ΩVSδθ2 ; ð5Þ

where Ω is the field of view, δθ is the instantaneousfield of view (ifov), V is the system volume, and S isthe frame integration time. Due to excess noise aris-ing in image estimation from multiplex measure-ments, Haney predicts that the ratio of themultiple aperture merit function to the single aper-ture covering the same total area is

MMA

MSA≅

1

n3ð1þ σ2Þ2 ; ð6Þ

where σ2 is a noise variance term and n2 is the num-ber of subapertures used. Haney’s result is based onsignal degradation due to multiplex noise and on anincrease in integration time to counter this noise. Wesuggest that only one or the other of these factorsneed be counted, meaning that under Haney’s meth-odology the degradation factor is

MMA

MSA≅

1

nð1þ σ2Þ : ð7Þ

Haney’s result suggests that the signal-to-noise ra-tio (SNR) for the multiple aperture system should bedegraded by approximately 3 times for our modelsystem rather than our prediction of comparable orsuperior low frequency performance and greaterthan 3 times SNR loss near the aliasing limit. Thisdiscrepancy is primarily due to Haney’s assumptionthat the pixel sampling function for the multipleaperture system is designed to flatten the STF, usingfor example the Hadamard coded detectors describedby Portnoy et al. [6]. Such coding strategies drama-tically improve the high-frequency response of the

0 0.05 0.1 0.15 0.210−2

10−1

100

101

102

u

εMA/εSA

Sa/S

n=0

Sa/S

n=10

Sa/S

n=100

Fig. 2. Ratio of the Wiener filter error for the multiple and singleaperture systems of Fig. 1 across the nonaliased spatial bandwidthfor various alias signal strengths.


multiple aperture system at the cost of dramaticreductions in low-frequency image fidelity. As illu-strated in Fig. 1, simple shift codes provide poorhigh-frequency response but match the single aper-ture low-frequency response.Of course, the assumption underlying this discus-

sion that mean square error is a good image metriccan be challenged on many grounds. Numerous re-cent studies of feature specific and compressive sam-pling suggest that optimal sampling system designshould focus on robust sampling of image structurerather than pixelwise sampling or STF optimization.Rather than enter into a detailed discussion of themany denoising, nonlocal, or feature analysis andnonlinear signal estimation strategies that mightbe considered here, we simply note that multipleaperture design appears to be a useful tool in balan-cing constraints in focal plane design and readout,optical system design, system form factor, and massand imager performance.

3. Optical Design and Experimental System

Instead of a conventional lens, our system subdividesthe aperture into a 3 × 3 lenslet array. Each of thesenine lenses meet the system’s required f-number butachieves a reduction in thickness by using shorter fo-cal lengths. We position the centers of the nine lensesto have unique registration with the underlying pixelarray. This creates measurement diversity thatenables high-resolution reconstruction. As was doneby Shankar et al. [7], we design each of the centers tohave a 1=3 pixel shift with respect to one another intwo dimensions.The underlying system goals motivated the design

of the lenslet array. We desire an ultrathin system

with low f-number and excellent imaging perfor-mance over a broad field of view. Each lenslet con-sists of a germanium meniscus lens and a siliconfield flattener. Both surfaces of the germanium lensare aspheric as is the top surface of the silicon ele-ment. The bottom surface of the silicon element isplanar. The f =1:2 lens combination is 5mm thickfrom the front surface of the lens to the detector pack-age. The full optical train is shown in Fig. 3. Modula-tion transfer function plots of the designed systemare shown in Fig. 4.

The germanium element was diamond turnedon both surfaces, with the front and back regis-tered to each other within a few micrometers (seeFig. 5). Each lens was turned individually and me-chanically positioned such that the decentrationerror is less than a few micrometers. The silicon

Germanium Cover Silicon

Lens

GermaniumLens

3.9 mm 1.03 mm0.57 mm

Fig. 3. (Color online) Designed optical layout of a single lenslet.

Fig. 4. (Color online) Polychromatic square wavemodulation transfer function performance of each lens in themultichannel lenslet arrayas designed.


lens was made lithographically, using a grayscalehigh-energy beam sensitive glass mask. The processexposes and develops a starting shape in thick resistthen uses reactive ion etching to transfer the shapeinto the silicon. Submicrometer accuracy wasachieved for the silicon element lenses.The optics were attached to a 12 bit, 640 × 480 un-

cooled microbolometer array with 25 μm square pix-els. Each of the 9 lenslets images onto an area ofabout 80 × 80 pixels. Our multiple aperture techni-que requires approximately one-quarter of the totaldetector pixels for image reconstruction. We usedsuch a large array primarily because of its availabil-ity, but a commercial system would likely utilize adifferent design. For example, one might use a seg-mented approach with small imaging arrays inte-grated on a larger backplane.The germanium and silicon elements were aligned

and mounted in a custom-designed aluminum holderthat is secured in front of the detector package. Tooptimize focus, mylar spacers were used to shimthe lens package appropriately from the detectorpackage in increments of 25 μm. A prototyped alumi-num enclosure protects the camera and electronicsfrom the environment while also providing mountingcapabilities.

4. Image Reconstruction

There are nine lower-resolution images produced bythe 3 × 3 lenslet array. The reconstruction processconsists of two stages, registration and integration.

A. Registration

It is critical to register subframes from the lower-resolution images. Due to parallax, the relative im-age locations on the detector are dependent on objectdistance. To register, we first choose one of the ninesubimages as a reference. Then we maximize thetwo-dimensional correlation of that image withrespect to the other eight cropped subimages. Thisresults in a coarse registration on the order of a pixel,which greatly improves efficiency of the reconstruc-tion stage. These parameters may be saved as cali-bration data because they are nominally constant

for scenes of fixed depth. Coarse registration dataare applied in a second fine registration stepdescribed below.

B. Reconstruction

We integrate the images of lower resolution into asingle one by the measurement equations,

Hkf ¼ gk; k ¼ 1; 2;…; 9;

where f is the image of the scene at the resolutionlevel targeted by the reconstruction, gk is the imageof lower resolution at the kth subregion of the detec-tor array, Hk is the discrete measurement operatorrelated to the kth aperture, mapping f to gk. Forthe CIRC system, each measurement operator by de-sign can be described in the following more specificform:

Hk ¼ ðD2;kB2;kS2;kÞ ⊗ ðD1;kB1;kS1;kÞ; ð8Þ

where Si is the displacement encoding at the aper-ture; Bi;k, i ¼ 1; 2, describes optical diffusion or blur-ring along dimension i associated with the kthsubaperture system; and Di, the downsampling atthe detector.

Iterative methods are used for the solution to thelinear system, because the number of equations is po-tentially as large as the total number of pixels on thedetector. We found that certain algorithms that seemto work in simulation fail to produce reliable resultswith empirical data, primarily due to substantial dis-crepancy between ideal assumptions and practicaldeviations. The reconstruction from measurementsat the early stage of a system development has todeal with insufficient calibration data on system-specific functions and parameters as well as noisecharacteristics. We introduce a particular approacheffective for this situation.

The underlying reconstruction model is called theleast gradient (LG) model [9]. In the LG approach,the system of measurement equations is embeddedin the following reconstruction model:

fLG ¼ arg minf∥∇f∥2 s:t: Hkf ¼ gk;

k ¼ 1; 2;…; 9;ð9Þ

where ∇ denotes the discrete gradient operator and∥ · ∥2 is the Euclidean norm. This LG model permitsunderdetermined measurement systems. Amongmultiple solutions, the LG solutions are smooth. Intheory, the LG reconstruction model (9) can be recastinto the unconstrained minimization problem asfollows:

fLG ¼ arg mind∈N ∥∇ðfp − dÞ∥2; ð10Þ

where fp is a particular solution to the system oflinear equations

Fig. 5. (Color online) Front and back (inset) surfaces of the dia-mond-turned germanium element.


Hkfp ¼ gk; k ¼ 1; 2;…; 9

and N is the null space of the linear system. Denoteby N a basis of the null space. Then the solution toEq. (10) can be expressed as follows:

fLG ¼ fp − NðNT∇T∇NÞ−1ð∇NÞT∇fp:

Based on the separability of the measurement op-erator(s) as shown in Eq. (8), we apply the LG modelto the subsystems partially and independently, i.e.,

fk;LG ¼ arg minfk∥∇fk∥2; s:t: Hkfk ¼ gk; ð11Þ

for each and every k, 1 ≤ k ≤ 9. The solution to each ofthe individual partial models can be obtained, usingfor example the explicit solution expression for thecorresponding unconstrained minimization problem.This approach is similar to the Jacobi method for

solving a large linear or nonlinear system of equa-tions. While a subsystem is decoupled from the restby partitions in the measurements and the un-knowns in the Jacobi method, a subsystem inEq. (11) is set by the natural partition in measure-ments gk, a separation in the objective function, anextension of the scalar-valued function f into the vec-tor of partial estimates ½fk�k¼1∶9, and a substitution inthe objective function. Simply stated, we get a stackof nine smooth images at the subpixel level in refer-ence to the detector pixels.There is more than one way to integrate the

stacked image estimates. Wemust specify a mappingfrom the vector-valued function to the scalar-valuedfunction. Technically, this mapping involves thealignment of the separate estimates fk, 1 ≤ k ≤ 9, atthe subpixel level. One shall notice that the relativedisplacements of the lenslets do not play a significantrole in the solution to each individual subsystem.Practically, this subpixel alignment is carried outagain by correlation at the subpixel level. The cur-rent registration method aligns the brightest regionsin the field of view to best maximize the correlation ofthe centers of the subimages. Misaligned regions dueto parallax are blurred as if out of focus. The separateimage estimates in alignment can then be integratedinto one as a weighted sum. When each of the sepa-rated estimates is normalized in intensity, theweights are equal over well-overlapped subpixelsand unequal over nonoverlapping subpixels basedon numerical quadrature rules for nonequally spacedquadrature nodes. The estimate by integration of thepartial LG solutions can be used as the initial esti-mate for an iterative method for the coupled system(9). Such initial estimates from empirical data andwithout further refinement are shown in Subsection4.C.We can use different algorithms to upsample and

integrate the data from each channel. We comparethree different approaches in Fig. 6. This imageshows the reconstruction of a 4 bar target image.The top image uses the reconstruction algorithm

we describe in this paper and shows the highestamount of contrast. The middle and bottom imagesinterpolate with a standard linear and bicubic algo-rithm, respectively. We upsample each of the 9 chan-nels individually and then use the same alignmentand combination procedure as the LG approach tointegrate the images.

The actual computation we perform for our currentimaging system consists of solving the subproblemsHk fk ¼ gk for fk as above in Eq. (11), where Hk ¼ðDBÞ ⊗ ðDBÞ with D ¼ I ⊗ ½111�=3 is a linear opera-tor that averages three contiguous pixels down to asingle pixel, B is an approximate Gaussian blurringoperator common to all lenslets, and fk ¼ðS2;k ⊗ S1;kÞf. The shifts Sk are recovered with thecorrelation alignment of the separate estimates fk.

In comparison to some other reconstruction algo-rithms, this approach via partial LG solutions is com-putationally efficient and numerically insensitive tothe boundary conditions for the CIRC system.

C. Results

Results of the reconstruction algorithm demonstratesystem performance. We include data sets acquiredat varying target ranges. Figure 7 shows the recon-struction of human subjects in a laboratory setting.Along with the processed image we show the rawdata acquired directly off the camera. For compari-son, we use cubic spline interpolation to upsamplea single lenslet image. Integration of the multiplechannels shows a clear improvement over this image.

To ensure the reconstructions do not introduce er-roneous artifacts we compare them to images takenwith a conventional single lens LWIR camera. Thecomparison system uses a 320 × 240 pixel array cor-responding to approximately the same number ofimaging pixels used by the multichannel system.Both cameras share comparable fields of viewand utilize similar 25 μm microbolometer detector

Fig. 6. Comparison of three different reconstruction approachesfrom the same raw data. The top image uses the LG approach de-tailed in this paper and shows the most contrast. The middle andbottom image were formed using traditional linear and bicubic in-terpolation methods, respectively.


technology. For the comparison system, internal elec-tronics automatically adjust the gain level and out-put data through an analog RS-170 (black and white)video stream. A computer capture card digitizesthese video frames for analysis. We use the VCE-PRO Flat PCMCIA card made by Imprex Incorpo-rated. Unfortunately, direct digital acquisition ofpixel data was not possible for the comparison cam-era. We manually focused this camera.The images in Fig. 7 also demonstrate the signifi-

cantly larger depth of field obtained by the multi-channel system. The microlens array’s effectivefocal length of 6:15mm is about 4 times shorter thanthe 25mm focal length f =1:0 optic used in the con-ventional camera. A shorter focal length translatesto a much shorter hyperfocal distance meaning closeobjects will appear more in focus.

Shown in Fig. 8, field data results compare the per-formance between the two systems for targets at adistance of 42m (i.e., long range).

5. Experimental Results

A. Noise Equivalent Temperature Difference

Thermal imaging systems are often calibrated tomeasure the equivalent blackbody temperature dis-tribution of a scene. In this application better perfor-mance means better discrimination between tworegions of different temperature. NETD [10] is ametric for characterizing a system’s effective tem-perature resolution. By definition, NETD is the tem-perature difference where the SNR ratio is unity.NETD translates pixel fluctuations resulting fromsystem noise into an absolute temperature scale.As noise statistics vary with operating temperature,the corresponding NETD fluctuates.

To experimentally calculate NETD, we image a col-limated target aperture illuminated with a black-body source. The target projector optics consist ofan all reflective off-axis Newtonian telescope witha 2:75° field of view. The blackbody source has a52mm clear aperture, and it illuminates a 37mmdiameter circular copper target. Arbitrary target de-signs aremilled in small metal discs that are selectedwith a target wheel. The copper provides a sufficientthermal mass to mask the blackbody source. Thus,the temperature of the target and the background

Fig. 7. Side by side comparison between conventional and multichannel cameras. The person is at a distance of 3 meters; the hand is atapproximately 0.7 meters. Both objects appear in focus with the CIRC as opposed to the conventional system due to the multichannelcamera’s increased depth of field. The images were taken simultaneously, so some parallax is visible.

Fig. 8. Side by side comparison between conventional and multi-channel cameras. Target distance is approximately 42m.


can be independently controlled. Figure 9 details theexperimental setup.Effective NETD calculations are performed on

large regions of constant temperature to avoid areduction in contrast by the camera’s optical system.Large targets minimize high spatial frequency com-ponents. We use a full-sized semicircle target (half-moon) for our measurements clearly segmentingtwo temperature regions. To calculate NETD weuse the following equation:

NETD ¼ ΔTSNR

¼ ðTH − TBÞmeanðdatajTHÞ−meanðdatajTBÞ

stdðdatajTBÞ: ð12Þ

TH and TB represent the “hot” and ambient tempera-ture regions created by a blackbody source, respec-tively, and ΔT ¼ TH − TB. The variables datajTHand datajTB represent arrays of pixel values corre-sponding to those temperature regions. Here, signalis defined as the difference between the mean pixelresponse in each area, and the noise is the standarddeviation of pixel values in the background region.Noise fluctuations influence NETD calculations,

especially at low SNR ratios. This is why NETD istraditionally calculated using higher-contrast tem-perature regions. However, it may become proble-matic to do this using a computational systembecause nonlinear processing will distort resultsusing Eq. (12).We imaged the semicircle target at a number of dif-

ferent temperature contrast settings, and the resultsare shown in Fig. 10. We calculate a linear regressionon the data set to interpolate the temperature atwhich the SNR is unity. Using this procedure, wecalculate that the NETDs for the conventional andmultichannel cameras are 121mK and 131mK, re-spectively. Since both cameras utilize similar un-cooled focal plane technology, this discrepancy is

likely due to the different lens transmissions of thetwo systems. Their total area, number of elements,and antireflective coatings are not identical.

B. Spatial Frequency Response

This subsection outlines an alternate interpolationmethod to combine our multichannel data. We re-move aliasing to recover contrast from bar targetswith spatial frequencies beyond the Nyquist limit de-fined by the pixel pitch. High-resolution results arerecovered by combining discrete samples from eachsubaperture with registration information. Charac-terization of the subpixel shift of each channel givesthe crucial reference needed for these reconstruc-tions. Using a collection of periodic targets, we areable to experimentally measure our system’s spatialfrequency response.

The Whittaker–Shannon interpolation formulagives us the following expansion to reconstruct acontinuous signal from discrete samples:

f ðxÞ ¼X∞n¼−∞

f

�n2B

�sincð2Bx − nÞ: ð13Þ

The reconstructed bandwidth, B, is related to thesampling interval Δ as Δ ¼ 1=2B. This strategy ofrecovering a continuous signal from discrete samplesprovides the basis for combining our multichan-nel data.

For the 3 × 3 system presented in Section 3, thelenses are positioned with 1=3 pixel offsets in two di-mensions and all share the same field of view. Thus,in the absence of parallax, the individual subimagedata could be interwoven on a grid with a samplingperiod equal to one-third the nominal pixel size.

Fig. 9. Top view of the experimental setup used for NETD andspatial frequency response measurements. A blackbody sourceilluminates a copper target that is then collimated. The cameraunder test is positioned in front of the output aperture of theprojector.

Fig. 10. (Color online) SNR comparison as a function of targettemperature difference. The circles and plus signs correspond tothe conventional and multichannel data points, respectively.


Generalizing Eq. (13) to allow for multiple chan-nels we obtain

f ðxÞ ¼XKk¼1

XN=2

n¼−N=2

mk½n�sincð2B0x − nþ δkÞ: ð14Þ

Here, mk represents the discrete samples mea-sured from channel k. The offset registrationbetween each of these sampling train is accountedby the δk parameter. Nominally, δk ¼ kΔ=K . Also re-cognize that B0 can be increased to KB because anincreased sampling rate extends system bandwidth.Any choice of B0 < KB is equivalent to simply low-pass filtering the reconstructed signal. With a 33%fill factor, one could directly interleave the multi-channel data without the need for this sinc interpo-lation strategy. However, a higher fill factor does notdirectly imply limited resolution. The pixel’s sam-pling function acts as a spatial filter, which limits re-construction contrast in the presence of noise.Our system only approximates the designed 1=3

pixel lenslet shifts. Next, we describe the character-ization procedure we use to register our channels andarrive at slightly modified δk parameters. Recon-struction on nonideally sampled data is a growing re-search area studied by Unser [11] and others [12,13].We recognize that the reconstruction approach wepresent would not be appropriate for significant mis-alignments. However, we achieve better reconstruc-tion results by tuning the registration parameters toour physical system.First, the camera is mounted on a motorized rota-

tion stage in front of the target projector discussed inSubsection 5.A. An appropriate temperature settingis chosen for good contrast. The stage rotates thecamera in regular controlled steps shifting the targeton the detector a fraction of a pixel at each step. Si-multaneously from each aperture we record data atevery camera position. The full scan should result in

the target moving by a handful of pixels on the detec-tor. This set of registered frames from each channelcontains registration information.

To create diversity in each aperture, the lensletpitch was designed to be a noninteger multiple ofthe pixel width. Each channel measures the same ob-ject, but it is recorded with a unique offset. More spe-cifically we attempted to prescribe a unique 1=3 pixelstagger in x and y for each of the 9 lenslets. Throughcorrelation or other strategies, shift information canbe extracted from the position information. Figure 11plots the responses from a pixel in each of three chan-nels to a periodic bar target as a function of cameraangle. The position-based offset between signals isdirectly related to the subpixel registration of eachchannel.

We provide interpolations to demonstrate the re-covery of higher resolution data from registereddownsampled multichannel data. In the extreme

Fig. 11. (Color online) Registered responses of a pixel in eachaperture for a rotating target scene.

Fig. 12. Data(a)andcorresponding interpolation (b) fora4bartar-get with spatial frequency of 0:120 cycles=mrad. The raw samplesare from one channel of the multiaperture camera. The bar targetfrequency is approximately equal to the critical frequency. The in-terpolation is performed by combining data from three subaper-tures to improve contrast in the horizontal dimension.


cases, each downsampled channel samples below itscorresponding “Nyquist” limit measuring aliaseddata. Reconstruction is contingent upon two majorfactors. First, the high-performance optical systemmust maintain the higher spatial frequencies (or re-solution) when imaging onto the pixel array. Second,the subpixel shifts between each channelmk must beadequately characterized.

Using Eq. (14) and characterization informationfrom Fig. 11, we generate resolution improvementin one dimension by processing data from three aper-tures on a row by row basis. Figure 12 shows a side byside comparison between the raw data and interpo-lation for a vertically placed 4bar target. The barwidth in the target aperture is 3:18mm. Using thecollimator focal length of 762mm, the calculated spa-tial frequency is 0:120 cycles=mrad. High contrastlevels are present in the subapertures as well asthe reconstruction. Next, we use the same approachon a more aggressive target. The raw data and inter-polation for a target with 1:98mm features(0:192 cycles=mrad) are shown in Fig. 13. This targetcorresponds to a period of 32:2 μm on the focal plane.As this is smaller than twice the 25 μm pixel pitch,recovery of contrast demonstrates superresolutionreconstruction. Figure 14 shows an intensity plotfor one row. The four peaks in the interpolation (solidline) correspond to the 4 bars of the target. Using onesubaperture alone (circles), it would be impossible toresolve these features.

For these reconstructions we set B0 ¼ 1:7B, whichis slightly less than the theoretical factor of 3 im-provement possible with this system. Misregistra-tion fundamentally limits the full capability. Asmentioned above, this conservative interpolation re-sult is equivalent to low-pass filtering. However, ourchoice of B0 > B allows us to demonstrate aliasremoval capabilities of the process. Note that theseresults are generated from a single row vector fromthree apertures. While the approach is extendable totwo dimensions, some experimental difficulties limitour abilities to provide them here. Subpixel align-ment and registration challenges and nonuniformityare the major limiting factors.

This work demonstrates the recovery of contrastfrom aliased data samples. To quantify the results,fringe visibilities are calculated for multiple 4bartargets with each one corresponding to a differentspatial frequency. We use the following formula toquantify the contrast:

Fig. 13. Data (a) and corresponding interpolation (b) for a 4bartarget with spatial frequency of 0:192 cycles=mrad. The raw sam-ples are from one channel of the multiaperture camera. Aliaseddata are measured because the bar target frequency exceedsthe critical frequency. The three-channel interpolation demon-strates resolution improvement in the horizontal dimension.

Fig. 14. (Color online) Cross sectional intensity plot from the fine4bar target reconstruction shown in Fig. 13. The solid line showsinterpolated result. Data points from one channel are indicated bycircles. Data from the conventional system (dotted line) show thatthe target frequency approaches the aliasing limit.

Table 1. Experimentally Calculated Contrast, V, for 4bar Targets at FiveSpatial Frequencies

Calculated TargetSpatial Frequency(cycles=mrad)

ConventionalSystemContrast

SingleLensletContrast

MultichannelReconstruction

Contrast

0.120 77% 40% 60%0.136 55% 20% 28%0.150 55% aliased 18%0.167 40% aliased 14%0.192 31% aliased 11%


V ¼�Imax −

�Imin�Imax þ �Imin

; ð15Þ

where �Imax and �Imin represent the average of inten-sities of the four local maxima and three localminima, respectively. Table 1 compares these calcu-lated values to the response of a single lenslet inthe multichannel system as well as a conventionalimaging system.

6. Conclusion

We have extended the field of multiple aperture ima-ging by describing the design and implementation ofa thin LWIR camera using a 3 × 3 lenslet array. Themultiple aperture approach provides a thickness andvolume reduction in comparison to a conventionalsingle lens approach. To form a single image, wecombine the nine subimages computationally inpostprocessing.A working prototype has been constructed, and its

data have been extensively analyzed. A LG imagereconstruction algorithm has been implemented thatshows better results than linear and spline inter-polation. Reconstruction requires system character-ization that involved determining the relativesubpixel registration of each lenslet by scanning ahigh-frequency target with subpixel precision. Bartarget data at multiple spatial frequencies werereconstructed using these registration parameters.We were able to combine aliased data from multiplesubapertures to reconstruct high-frequency bartargets. Joint postprocessing of the subimagesimproves resolution beyond the limit of a singlelenslet.

This work is supported by Microsystems Technol-ogy Office of Defense Advanced Research ProjectsAgency (DARPA) contract HR0011-04-C-0111. Theauthors wish to acknowledge the individual contribu-tions of David Fluckiger at Raytheon Company andJames Carriere at Tessera. In addition, we thank allof the COMP-I team members, including those at

Duke University, the University of North Carolinaat Charlotte, the University of Delaware, the Univer-sity of Alabama in Huntsville, the Army ResearchLaboratory, Raytheon Company, and Tessera Tech-nologies, Inc.

References1. A. W. Lohmann, “Scaling laws for lens systems,”Appl. Opt. 28,

4996–4998 (1989).2. Sung Cheol Park, Min Kyu Park, and Moon Gi Kang, “Super-

resolution image reconstruction: a technical overview,” IEEESignal Process. Mag. 20(3), 21–36 (2003).

3. J. Tanida, T. Kumagai, K. Yamada, S. Miyatake, K. Ishida,T. Morimoto, N. Kondou, D. Miyazaki, and Y. Ichioka, “Thinobservation module by bound optics (TOMBO): Concept andexperimental verification,” Appl. Opt. 40, 1806–1813(2001).

4. H. Shekarforoush and R. Chellappa, “Data-driven multichan-nel superresolution with application to video sequences,” J.Opt. Soc. Am. A 16, 481–492 (1999).

5. M. Ben-Ezra, A. Zomet, and S. K. Nayar, “Video super-resolu-tion using controlled subpixel detector shifts,” IEEE Trans.Pattern Anal. Mach. Intell. 27, 977–987 (2005).

6. A. D. Portnoy, N. P. Pitsianis, X. Sun, and D. J. Brady, “Multi-channel sampling schemes for optical imaging systems,” Appl.Opt. 47, B76–B85 (2008).

7. M. Shankar, R. Willett, N. Pitsianis, T. Schulz, R. Gibbons, R.T. Kolste, J. Carriere, C. Chen, D. Prather, and D. Brady, “Thininfrared imaging systems through multichannel sampling,”Appl. Opt. 47, B1–B10 (2008).

8. M. W. Haney, “Performance scaling in flat imagers,” Appl. Opt.45, 2901–2910 (2006).

9. D. J. Brady, M. E. Gehm, N. Pitsianis, and X. Sun, “Compres-sive sampling strategies for integrated microspectrometers,”Proc. SPIE 6232, 62320C (2006).

10. E. L. Dereniak and G. D. Boreman, Infrared Detectors andSystems (Wiley, 1996).

11. Y. C. Eldar and M. Unser, “Nonideal sampling and interpola-tion from noisy observations in shift-invariant spaces,” IEEETrans. Signal Process. 54, 2636–2651 (2006).

12. A. Aldroubi and K. Grochenig, “Nonuniform sampling andreconstruction in shift-invariant spaces,” SIAM Rev. 43,585–620 (2001).

13. A. J. Coulson, “A generalization of nonuniform bandpasssampling,” IEEE Trans. Signal Process. 43, 694–704 (1995).


Design and characterization of thin multiple aperture...

Documents

Transcript of Design and characterization of thin multiple aperture...