A Digital Camera to A Digital Camera to Rival the Human EyeRival the Human Eye

Dr. Dil Joseph, Assistant ProfessorDr. Dil Joseph, Assistant Professor

Electrical and Computer EngineeringElectrical and Computer Engineering

University of AlbertaUniversity of Alberta

SynopsisSynopsis

Although digital cameras have in many ways Although digital cameras have in many ways surpassed the capabilities of film cameras, the surpassed the capabilities of film cameras, the human eye remains the ultimate standard for human eye remains the ultimate standard for comparison and it vastly outperforms the best comparison and it vastly outperforms the best cameras in many respectscameras in many respects

Inspiration from biology may help us to Inspiration from biology may help us to “perfect” the technology of imaging“perfect” the technology of imaging

OutlineOutline

Film camerasFilm cameras Digital camerasDigital cameras Human eyesHuman eyes My researchMy research Your questionsYour questions

However, let us first put However, let us first put things in perspective…things in perspective…

In PerspectiveIn Perspective Society has invested over millennia in developing technology Society has invested over millennia in developing technology

to record observed scenes on an independent mediumto record observed scenes on an independent medium Artistic license aside, the aim has been to render images with a Artistic license aside, the aim has been to render images with a

maximum of perceptual accuracy using a minimum of effortmaximum of perceptual accuracy using a minimum of effort The digital camera is merely a culmination of the above but The digital camera is merely a culmination of the above but

that does not mean the technology development is overthat does not mean the technology development is over

Norbert Aujoulat © National Centre of Prehistory, Francehttp://www.culture.gouv.fr/culture/arcnat/lascaux/en/

History of Film CamerasHistory of Film Cameras

Two scientific processes Two scientific processes required development: required development: optical and chemicaloptical and chemical

During the Renaissance, During the Renaissance, the optical process was the optical process was understood well enough understood well enough for artists to begin using for artists to begin using the the camera obscuracamera obscura as as a drawing aid (see right)a drawing aid (see right)

History of Film CamerasHistory of Film Cameras

Thomas Wedgwood published his work on Thomas Wedgwood published his work on photogramsphotograms in 1802; he recorded short-lived in 1802; he recorded short-lived images using silver nitrate on leatherimages using silver nitrate on leather

John Herschel, who popularized the word John Herschel, who popularized the word “photography”, discovered in 1819 how to “photography”, discovered in 1819 how to dissolve (unexposed) silver saltsdissolve (unexposed) silver salts

Although Nicéphore Niépce produced the first Although Nicéphore Niépce produced the first photograph in 1827, Louis Daguerre was the photograph in 1827, Louis Daguerre was the most famous of the early inventorsmost famous of the early inventors

History of Film CamerasHistory of Film Cameras

The Daguerreotype was The Daguerreotype was revealed to the public in revealed to the public in 1839 to much acclaim1839 to much acclaim

Images were produced Images were produced on copper plates coated on copper plates coated with silver iodidewith silver iodide

The drawbacks of the The drawbacks of the process were the cost, process were the cost, long exposure time and long exposure time and irreproducibilityirreproducibility

History of Film CamerasHistory of Film Cameras

Fox Talbot introduced Fox Talbot introduced the Calotype in 1840, the Calotype in 1840, the first process to the first process to involve negativesinvolve negatives

Positive prints could be Positive prints could be made cheaply on papermade cheaply on paper

Others went on to create Others went on to create processes that had both processes that had both better image quality and better image quality and shorter exposure timesshorter exposure times

Film Cameras TodayFilm Cameras Today

Film cameras today have come a long way Film cameras today have come a long way since the nineteenth century, especially by since the nineteenth century, especially by improved spatial and contrast resolutionimproved spatial and contrast resolution

Thanks to the early work of Maxwell and Thanks to the early work of Maxwell and “recent” work of others, including industry, “recent” work of others, including industry, colour photography thrives todaycolour photography thrives today

However, except for low cost and large format However, except for low cost and large format photography, amateurs and professionals are photography, amateurs and professionals are replacing their film cameras with digital onesreplacing their film cameras with digital ones

History of Digital CamerasHistory of Digital Cameras

Einstein published three Einstein published three papers in 1905, one of papers in 1905, one of which explained the which explained the photoelectric effectphotoelectric effect

Photons striking a metal Photons striking a metal liberate electrons, which liberate electrons, which can carry a currentcan carry a current

Kinetic energy depends Kinetic energy depends on light frequency while on light frequency while the number of electrons the number of electrons depends on intensitydepends on intensity

History of Digital CamerasHistory of Digital Cameras

The advent of Quantum Mechanics in the early The advent of Quantum Mechanics in the early 1900s led to advances in solid-state physics1900s led to advances in solid-state physics

Russell Ohl invented the Russell Ohl invented the p-n junctionp-n junction, a , a photosensitive diode that (usually) passes photosensitive diode that (usually) passes current in only one direction, in 1940current in only one direction, in 1940

John Bardeen, Walter Brattain and William John Bardeen, Walter Brattain and William Shockley invented the transistor in 1947, a Shockley invented the transistor in 1947, a semiconductor device that can act as an semiconductor device that can act as an amplifier or a switchamplifier or a switch

History of Digital CamerasHistory of Digital Cameras

Working independently, Jack Kilby and Robert Working independently, Jack Kilby and Robert Noyce invented the integrated circuit in 1959Noyce invented the integrated circuit in 1959

Transistors, diodes, capacitors, resistors and Transistors, diodes, capacitors, resistors and wiring could be fabricated on a single crystal of wiring could be fabricated on a single crystal of semiconductor material, e.g. siliconsemiconductor material, e.g. silicon

George Smith and Willard Boyle invented the George Smith and Willard Boyle invented the CCD camera in 1969CCD camera in 1969

Digital, as opposed to analog, CCD cameras Digital, as opposed to analog, CCD cameras came in the late 1980scame in the late 1980s

Digital Cameras TodayDigital Cameras Today

A digital camera consists of many components A digital camera consists of many components (optics, housing, battery, memory etc.), of (optics, housing, battery, memory etc.), of which the image sensor is principalwhich the image sensor is principal

With market revenues of $1.7 billion in 2003, With market revenues of $1.7 billion in 2003, there is widespread research and development there is widespread research and development in a variety of image sensor designsin a variety of image sensor designs

Modern designs may be either charge coupled Modern designs may be either charge coupled device (CCD) sensors or complementary device (CCD) sensors or complementary metal-oxide-semiconductor (CMOS) sensorsmetal-oxide-semiconductor (CMOS) sensors

CCD Image SensorsCCD Image Sensors

March photo-generated March photo-generated charge systematically from charge systematically from an array of pixels to an an array of pixels to an output amplifieroutput amplifier

Established technologyEstablished technology High resolution, high High resolution, high

sensitivity, low noisesensitivity, low noise Fabrication process is Fabrication process is

optimised for imagingoptimised for imaging Market share of 93% in Market share of 93% in

1999 (49% in 2004?)1999 (49% in 2004?)

CMOS Image SensorsCMOS Image Sensors

Work like memory array Work like memory array with photosensitive pixels with photosensitive pixels instead of memory cellsinstead of memory cells

Signal processing may be Signal processing may be included on the same dieincluded on the same die

High yield of working chips High yield of working chips and good video performanceand good video performance

May be fabricated by the May be fabricated by the makers of microchipsmakers of microchips

Market share of 7% in 1999 Market share of 7% in 1999 (51% in 2004?)(51% in 2004?)

Test Your KnowledgeTest Your Knowledge

In what ways are the In what ways are the digital cameras on the digital cameras on the market better than market better than human eyes?human eyes?

And, in what ways are And, in what ways are human eyes better than human eyes better than the digital cameras on the digital cameras on the market?the market?

History of the Human EyeHistory of the Human Eye

DarwinDarwin wrote in 1859: “ wrote in 1859: “To suppose that the To suppose that the eye…could have been formed by natural eye…could have been formed by natural selection, seems, I freely confess, absurd in the selection, seems, I freely confess, absurd in the highest possible degree. Yet reason tells me…highest possible degree. Yet reason tells me…the difficulty of believing that a perfect and the difficulty of believing that a perfect and complex eye could be formed by natural complex eye could be formed by natural selection…can hardly be considered real.”selection…can hardly be considered real.”

Biologists have since filled in many gapsBiologists have since filled in many gaps

The Human Eye TodayThe Human Eye Today

There are 6–7 million cones, for bright light vision There are 6–7 million cones, for bright light vision with fine detail and colour, and 75–150 million rods, with fine detail and colour, and 75–150 million rods, for dim light vision with coarse detail and no colourfor dim light vision with coarse detail and no colour

Today, the resolution of digital pixels (5–10 Today, the resolution of digital pixels (5–10 μμm) is m) is closing on the resolution of foveal cones (2–3 closing on the resolution of foveal cones (2–3 μμm)m)

© John W. Kimballhttp://users.rcn.com/jkimball.ma.ultranet/BiologyPages/

The Human Eye TodayThe Human Eye Today

The human eye is a remarkable organ not only The human eye is a remarkable organ not only because of its ability to sense images but because of its ability to sense images but especially because of its ability to process the especially because of its ability to process the image before sending a signal to the brainimage before sending a signal to the brain

The eye encodes the abundant visual input in The eye encodes the abundant visual input in such a way that the limited neural output such a way that the limited neural output retains the most significant descriptors of the retains the most significant descriptors of the scene while the rest are discardedscene while the rest are discarded

Most digital cameras do little pre-processingMost digital cameras do little pre-processing

Dynamic RangeDynamic Range

The human eye will capture brightness and colour detail (left) The human eye will capture brightness and colour detail (left) that a typical digital camera will fail to capture (right)that a typical digital camera will fail to capture (right)

These images were computed from a series of photographs—These images were computed from a series of photographs—no human eye or digital camera was harmed in the processno human eye or digital camera was harmed in the process

Photographic data came from Paul Debevec and the visual Photographic data came from Paul Debevec and the visual model came from Gregory Larson and his colleaguesmodel came from Gregory Larson and his colleagues

Colour ConstancyColour Constancy

The human eye factors The human eye factors out the illumination to out the illumination to large measure when large measure when identifying coloursidentifying colours

A typical digital camera A typical digital camera uses a simple method to uses a simple method to balance colours that will balance colours that will fail noticeably at timesfail noticeably at times

Better methods exist but Better methods exist but they are complicatedthey are complicated

Acquired colours Perceived colours

Typical correction Better correction

© Computational Vision Lab, Simon Fraser University (annotation added)http://www.cs.sfu.ca/~colour/research/colour-constancy.html

Fixed Pattern NoiseFixed Pattern Noise

Any two photoreceptors Any two photoreceptors in a retina and any two in a retina and any two photodetectors in an photodetectors in an image sensor are never image sensor are never perfectly identicalperfectly identical

A varying response to A varying response to light stimulus causes light stimulus causes “fixed pattern noise”“fixed pattern noise”

Camera designers work Camera designers work to correct or reduce the to correct or reduce the FPN; neurons adapt (?)FPN; neurons adapt (?)

Temperature StabilityTemperature Stability

Unlike the human body, Unlike the human body, a digital camera cannot a digital camera cannot regulate temperatureregulate temperature

The response of a pixel The response of a pixel to a light stimulus will to a light stimulus will depend on temperaturedepend on temperature

When the temperature When the temperature dependence varies from dependence varies from one pixel to another, one pixel to another, FPN will appearFPN will appear

My ResearchMy Research

Pixel & readout circuits Pixel & readout circuits and image processing and image processing may be improvedmay be improved

Camera designers face Camera designers face challenges when using challenges when using CMOS processes with CMOS processes with feature sizes ≤ 180 nmfeature sizes ≤ 180 nm

Photodetectors may be Photodetectors may be fabricated in a thin film fabricated in a thin film deposited on top of the deposited on top of the CMOS microchipCMOS microchip

Linear PixelsLinear Pixels

Linear pixels (either Linear pixels (either CCD or CMOS type) CCD or CMOS type) “count” photons over a “count” photons over a discrete period of timediscrete period of time

They produce a voltage They produce a voltage directly proportional to directly proportional to the light intensitythe light intensity

Unfortunately, the Unfortunately, the response may saturate response may saturate white or black easilywhite or black easily

© IMS Chipshttp://www.ims-chips.de/

Logarithmic PixelsLogarithmic Pixels

Logarithmic pixels Logarithmic pixels (CMOS only) measure (CMOS only) measure the “rate” of photon the “rate” of photon incidence continuouslyincidence continuously

They produce a voltage They produce a voltage directly proportional to directly proportional to the logarithm of the the logarithm of the light intensitylight intensity

The response is similar The response is similar to that of human visionto that of human vision

© IMS Chipshttp://www.ims-chips.de/

My ResearchMy Research

Logarithmic pixels are Logarithmic pixels are great for high dynamic great for high dynamic range imaging but…range imaging but…

FPN is worse compared FPN is worse compared to typical linear pixelsto typical linear pixels

Colours are worse than Colours are worse than for typical linear pixelsfor typical linear pixels

Temperature stability is Temperature stability is hardly understood for hardly understood for log (and linear) pixelslog (and linear) pixels

Fixed Pattern NoiseFixed Pattern Noise

By studying the causes By studying the causes of FPN, I developed a of FPN, I developed a method to correct itmethod to correct it

Images of a uniform Images of a uniform surface are used to surface are used to define correctionsdefine corrections

My correction reduces My correction reduces the FPN to the same the FPN to the same order as the random order as the random temporal noisetemporal noise

Colour RenditionColour Rendition

I also showed how to I also showed how to render accurate colours render accurate colours with logarithmic pixelswith logarithmic pixels

Images of a reference Images of a reference chart are used to define chart are used to define a colour mappinga colour mapping

The perceptual error of The perceptual error of the rendered colours is the rendered colours is comparable to the error comparable to the error of typical camerasof typical cameras

Rendered

Rendered

Reference

Reference

Temperature StabilityTemperature Stability

The dark response of a The dark response of a pixel depends only on pixel depends only on temperaturetemperature

Thus, it may be used to Thus, it may be used to correct FPN due to correct FPN due to temperature in the light temperature in the light responseresponse

Experiments support Experiments support this conclusion but this conclusion but simulation results are simulation results are shown for clarityshown for clarity

My ResearchMy Research

A digital camera should render images with a A digital camera should render images with a maximum of perceptual accuracy (as per the maximum of perceptual accuracy (as per the human eye) using a minimum of efforthuman eye) using a minimum of effort

A logarithmic imager responds to light much A logarithmic imager responds to light much like the human eye; pre-processing and better like the human eye; pre-processing and better electronics will improve the image qualityelectronics will improve the image quality

My past work has focused on maximising the My past work has focused on maximising the perceptual accuracy; my future work will perceptual accuracy; my future work will focus on minimising the effort requiredfocus on minimising the effort required

Your QuestionsYour Questions

ResourcesResources

Robert Leggat, “A History of Photography”, Robert Leggat, “A History of Photography”, http://www.rleggat.com/photohistory/http://www.rleggat.com/photohistory/

Mary Bellis, “The History of Computers”, Mary Bellis, “The History of Computers”, http://inventors.about.com/http://inventors.about.com/

Bob Patterson, “The Evolution of Eyes”, Bob Patterson, “The Evolution of Eyes”, http://www.origins.tv/darwin/eyes.htmhttp://www.origins.tv/darwin/eyes.htm

Microsoft Office Online, Clip Art and Media, Microsoft Office Online, Clip Art and Media, http://office.microsoft.com/clipart/http://office.microsoft.com/clipart/