Lensless Imaging with A Lensless Imaging with A Controllable ApertureControllable Aperture

Assaf Zomet and Shree K. NayarColumbia University

IEEE CVPR Conference

June 2006, New York, USA

Cameras TodayCameras Today

Lens

Plane in focus

Image Detector

Aperture

AttenuatingAperture

Our Lensless CameraOur Lensless Camera

Image Detector

Volumetric

Multilayer Aperture

Layered ApertureLayered Aperture

Image Detector

Attenuating apertures: Farid&Simoncelli ‘97, Nayar&Branzoi ‘03

ImplementationImplementation

Camera body Aperture: LCD

LCDControl

Image Detector

Attenuating LayerPinhole

Controllable PinholeControllable Pinhole

Image Detector

Pinhole Attenuating Layer

Controllable Pinhole: VideoControllable Pinhole: Video

Limitations of Lensless ImagingLimitations of Lensless Imaging

fa 9.1Optimal pinhole (Raleigh):

Brightness: f-number

ff

f

a

ff

9.1/#

a2

f

a

ff

a

• Sharpness and brightness increase with FOV (image center).

Limitations of Lensless ImagingLimitations of Lensless Imaging

fa 9.1Optimal pinhole (Raleigh):

Brightness: f-number

ff

f

a

ff

9.1/#

• Sharpness and brightness increase with FOV (image center).

• f and detector larger by factor k: Resolution/brightness by .

k

ak

f

f

f

Zand, 1992

Collecting More Light: Coded AperturesCollecting More Light: Coded Apertures

Conventional View Desired View

Split Field of View: ImplementationSplit Field of View: Implementation

Attenuating Layers(2nd layer physical)

Image Detector

Pinhole

Fov 2Fov 1

Fov 3

Split Field of View: ResultsSplit Field of View: Results

Lens camera

Ourcamera

Split Field of View: VideoSplit Field of View: Video

Finding FacesFinding Faces

Wastedpixels

Finding FacesFinding Faces

Pinhole

Attenuating Layers(2nd layer physical)

Split Image: Computational CameraSplit Image: Computational Camera

Correlation pattern

Image Detector

Computations in the OpticsComputations in the Optics

Our cameraLens camera

Correlation Pattern

Normalized correlation:

Image Pattern

vu

vu

vyuxI

vuPvyuxI

yxPId

,

2

,

),(

),(),(

),,,(

Noncoherent optical processing (Rogers 1977)

A Camera as a MappingA Camera as a Mapping

vuS ,

tanu

yxI ,

CameraOptics SMI

dudvvuSvuyxMyxI ),(),,,(,

Imaging Formulation: One LayerImaging Formulation: One Layer

yxI ,

vuS ,

fvyfuxT ,

f

tanu

dudvvuSfvyfuxTyxI ,,,

Imaging Formulation: MultilayerImaging Formulation: Multilayer

vfyufxT jjj ,

yxI ,

vuS ,

fj

tanu

SMI

dudvvuSvfyufxTyxIN

jjjj

1

,,,

Varying ZoomVarying Zoom

Conventional view

Desired view

H

H

L LL

L

Varying Zoom MappingVarying Zoom Mapping

Desired view

Scene

YM TXM

MX and MX are impossible to implement with 3 layers

TXYdesired SMMI

Attenuating Layers

Image Detector

Fov 1 Fov 3

PinholePinhole

Pinhole

Fov 2

Varying Zoom: Imaging (simulation)Varying Zoom: Imaging (simulation)

Captured

Desired

Varying Zoom: ReconstructionVarying Zoom: Reconstruction

TX

TXcapturedYYcomputed MCICMI

Icomputed

TXYcomputed SMMI

Icapturted

TXYcaptured SCCI

SummarySummaryControllable Pinhole

Computational Camera Spatially Varying Zoom

Split Field of View

Programmable cameras, Nayar, Branzoi & Boult 2004