Image Sensors: Pi Camera!

By: Juan medina

Outline:

● Digital Cameras

○ History (Mariner) and connectivity/social impact:

● What is an image sensor?

○ Types

○ Color separation and Performance metric

● Pi Camera

○ purpose, price, datasheet, examples!

● Conclusions

Digital Cameras: History and Social Impact

On July 15 of 1965, the Mariner 4 spacecraft obtained the first flyby digital image

(Figure 1: https://en.wikipedia.org/wiki/Mariner_4) of Mars with a camera system

designed by the NASA Jet Propulsion Laboratory. Such system utilized a video camera tube

followed by a digitizer (something somehow distant from what current digital cameras are)

to record 21 images on tape that where the transmitted to Earth. By the way, such

transmission took about 20 days to complete.

In 1975, Steven Sasson, engineer at Kodak, build the first digital camera using a

newly developed CCD (will explain later) image sensor sold Fairchild Semiconductor in

1973. Such camera was 3.6 Kg, had 0.01 megapixels, and took 23 seconds to capture the

first image (Figure 2: http://giantsofscience.weebly.com/steven-sasson.html).

Nowadays, digital cameras are a fundamental component of our lives. A tremendous

amount of digital visual content is created every day (Figure 3:

http://rocketpost.com/blog/instagram-direct-how-to-use-four-tips/). From our personal

life, up to an academic and professional environment, digital cameras have changed our life

dramatically; image sensors are everywhere: phones, pc, security, robots, scanners,

printers, etc.

What’s an image sensor?

An image sensor transforms the attenuation of electromagnetic radiation as it

passess through or is reflected off objects into small currents that convey the information.

Image sensors have a wide variety of applications which include: digital cameras, medical

imaging, night vision, thermal imaging, radar, and others.

The first analogue sensors used where the video camera tubes (e.g. Mariner 4).

Wikipedia’s description of their functionality is: "The cathode ray is scanned across a target

which is illuminated by the scene to be broadcast. The current, then, is dependent on the

brightness of the image on the target." I’ll just describe the Image Orthicon (Figure 4:

https://en.wikipedia.org/wiki/Video_camera_tube) which was used in the very first

televisions. Light passes through a camera lens and falls into a photocathode

(photosensitive plate at negative potential) where is converted into an electron image. The

electrons are then accelerated and gunned against a glass where the image is created.

Video Camera Tubes were followed by semiconductor charge-coupled devices (CCD)

and active pixel sensors in complementary metal-oxide-semiconductor (CMOS). Figure 5

(http://www.dpreview.com/forums/post/52351544), which I obtained from a blog post,

presents the general functionality of a modern image sensor. The incoming infrared light is

first filtered with an IR-Blocking Filter (a). Then a physical color filter array (CFA) controls

the color light reaching each color blind sensor cell (b). The latters transforms the light

into electricity which is then digitized. Lastly, millions of such cells are arranged to

construct a megapixel image sensor (d).

CCDs (Charged Coupled Devices)

In this modern sensor (the one used by Steven Sasson), an image is projected

through a lens onto the capacitor array (the photoactive region) causing each capacitor to

accumulate an electric charge that is proportional to the light intensity at that location.

After the image is projected into onto the array a control circuit causes each capacitor to

transfer its content to the next one (Figure 6:

https://www.microscopyu.com/articles/digitalimaging/ccdintro.html). Lastly, a charge

amplifier is used to convert the currents into a sequence of voltages that can then be

sampled and digitized. This whole process is brilliantly represented by a simple example by

Nixon (Figure 7: https://www.microscopyu.com/articles/digitalimaging/ccdintro.html). A

gate is opened, rain drops fall, buckets are filled. A parallel register drops the water row by

row in a serial bucket array. Lastly, the content of each bucket is dropped in a “Calibrated

Measuring Container” and the cycle repeats.

Complementary metal–oxide–semiconductor (CMOS)

On the other side, CMOS is an active pixel sensor which consist of an integrated

circuit containing an array of pixel sensors. Each pixel sensor contains a photodetector and

an active amplifier that is constructed with CMOS transistors. Figure 7

(http://www.digitalbolex.com/global-shutter/) presents a very instructive schematic of

how CMOS and CDDs functionality differ. On the left, light is sensed by a photodiode and

generates an electric charge that is stored in an electron transfer register. As explained

before a control algorithm moves such charges across the vertical and horizontal registers

where the signal is finally amplified and digitized. Differently, on the right we see that as

soon as light arrives to the photodiodes the CMOS amplifiers maximize the signal and send

it through metal wires.

CDDs vs CMOS

Even though there is no substantial difference in image quality, CMOS can be

implemented with fewer components, use less power, and can be read faster than CCD

sensors. As such, CMOS are less expensive to manufacture and therefore more common.

There are some hybrid sensors that leverage the advantages of CMOS and CDDs. High count

pixel cameras still use CDD. Figure 8 (http://www.digitalbolex.com/global-shutter/)

present some image sensor trends. CD's are still used for high performance applications

such as Professional DSC, Motion Analysis and Medical Imaging. CMOS have a wider range

of applications: Automotive, Toys, Phones, Biometrics, etc.

Color Separation

Color separation is an important topic that I would like to cover briefly. Once you

have your image sensors cells (IR filter, color filter, signal transducer) you have to arrange

them in some way in order to obtain an RGB image. There are different types for

color-separation mechanisms such as Bayer Filter sensor, Foveon X3 sensor and 3CCD. In

this blog post I’ll just cover the Bayer filter color filter array infrastructure. Figure 9

(https://en.wikipedia.org/wiki/Bayer_filter) presents the Bayer filter CFA pattern. This

filter pattern is 50% green, 25% red and 25% blue, that’s why it’s also called RGBG. This

proportion is used to mimic the physiology of the human eyes.

Demosaicing is the process of translating the pattern into a 3 level matrix. For each pixel

there has to be a value for Red, Blue and Green. There are different techniques for

demosaicing. For instance, at a green pixel there are always 2 red neighbor pixels. The

value of this pixels can be interpolated therefore interpolated.

Performance Metrics

Image sensors are commonly compared using 3 particular metrics:

1. pixel count: total number of pixels (NxW); often measured in megapixels. Pixel

counts is an important metric, however, image sensors with the same number of

pixels but with different size can result in different quality. Larger sensors produces

images with better resolution.

2. lens quality: resolution, distortion, dispersion.

3. dynamic range: the range of luminosity that can be reproduced accurately.

Pi Camera

The Pi Camera is a Raspberry Pi 5 megapixels camera module capable to record

1080p video and still images (Figure 10:

http://www.adafruit.com/images/1200x900/1367-00.jpg). The Raspberry Pi (Rpi) is

some sort of Arduino's next level. It's like a small computer running on linux that has digital

inputs and outputs. Differently from the Arduino, the Rpi does not have analog I/O. The Pi

Camera module connects directly to the Rpi Camera Serial Interface using a ribbon cable.

The board itself is very small (25 x 20 x 9 mm) and weighs about 3 g. Differently

from more sophisticated cameras, the module has a fixed focus lens onboard (although this

focus can be carefully changed). The Rpi combined with the camera module has several

applications: from rapid prototyping of Internet of Things (IOTs) devices up to time-lapse

devices. For instance, I've been using this sensor to build an smart garbage can. The e-can is

a regular trash container equipped with the Rpi and the Pi Camera. The goal is to take

pictures of garbage and try to classify them in real-time (e.g. plastic vs. metal).

The Rpi is quite cheap: it costs about 35 USD (you can buy it here:

https://www.sparkfun.com/products/11868). Something very important to understand is

that the Pi Camera is an integrated circuit designed to be smoothly connected to the Rpi.

However, the particular image sensor used in the Pi Camera is a 1/4" color CMOS QSXGA

image sensor manufactured by OmniVision. The datasheet

(http://cdn.sparkfun.com/datasheets/Dev/RaspberryPi/ov5647_full.pdf) its a very large

pdf that describes the sensor thoroughly. In this blog post, I'll review the general aspects of

this datasheet.

First I'll start with the sensor features (Figure 11). Even though there is a long list of

features, I'll comment just on 3: 1. 1.4x1.4 um pixel. The datasheet indicates that the

sensors a OmniBSI technology for high performance (high sensitivity, low noise). 2. It has

automatic image control functions like: Automatic Exposure Control (AEC), Automatic

White Balance (AWB) and others. 3. Support for output formats: 8-10 bit RGB data. As we

learned previously in this blog, CMOS final output is a serial analog signal that can be

digitized. In this particular case, the output of the sensor is the digitized data.

Another important information presented in the datasheet are the key specifications

of the sensor (Figure 12). From this list we can learn that there are 2592x1944 active

sensors cells (5 megapixels). We see that the core of the sensor works with 1.5V and it has

an embedded voltage regulator, while the analog electronics work with 3V and digital I/O

uses 1.7-3V. Interestingly, the temperature needed for operation ranges between -30C to

70C. However, stable images are only guaranteed in the range 0C-50C. Lastly, we see that

the maximum image transfer rate varies from 12 fps for QSXGA (1592 x 1944 pixels) to 120

fps for QVGA (320x240 pixels).

Figure 13. presents a block diagram for the QV5647 image sensor. In this figure we

can evidence several concepts I have presented in this blog. For instance, after the light

hitting the CMOS image array, the electrical signals are further amplified and digitized by a

10 bit ADC. The image array is controlled by the column sample hold and row select, which

are hierarchically controlled by the timing generator and system control logic (which is the

input of the sensor). Lastly, the image is outputted by an image sensor processor and an

image output interface that are controlled by a Control Register Bank (also input).

The last topic that I'll describe from the sensor datasheet has to do with color

separation. Figure 14. shows the CFA used in the sensor. Interestingly, it uses the Bayer

Filter Array pattern we learned earlier! The dummy sections are not exposed to light and

are used internally for black level calibration and interpolation (basically, they are used as

the zero reference).

Finally and as a conclusion I would like to share a particular image (Figure 14) I

took from garbage using the Rpi and the Pi Camera. We humans are extremely good at

recognizing/classifying objects out of images. Our extraordinary ability to reconstruct the

reality, evidenced, for instance, when we dream, is used to transform bi-dimensional

images into 3D objects with texture, smell, and physical properties (e.g. weight, rigidness).

Computers clearly cannot do this. However, how can we go further with images? In this

particular example I present an advance feature engineering technique called Histogram Of

Oriented Gradients. Such technique is utilized to extract edges' information from images

and therefore help computers with classification tasks. Today's role of image sensors is

crucial, I'm really excited to see tomorrow's.

References ● Digital camera ● Image Sensor ● Video camera tube ● Mariner 4 ● CCD ● CMOS ● Active pixel sensor