Limits of Colorimetric Characterization- The Brain vs. a Digital Camera

Vincent Bivona

Duke University, NEUROSCI 355S- “Visual Perception & the Brain”, Prof. Dale Purves, May 1 2017

The goal of colorimetry is to incorporate properties of the human color vision system into the

measurement and numerical specification of visible light. This paper summarizes what is known

about the initial stages of visual coding, human color perception, and human color space, as well

as the challenges that remain in explaining color vision. Based on this understanding, the goals

of colorimetry are explored. What follows is a summary and assessment of the success that has

been achieved in using colorimetric representations to provide a foundation for scientific study

of color appearance. At the heart of this paper is an assessment of the treatment of and attempts

to apply colorimetry to color reproduction in other visual systems like the digital camera and

how the challenges and limitations faced in doing so are tied to the lack of any clear way to link

perceptual qualities of color to physical characteristics of retinal stimuli.

INTRODUCTION

The sensory quality of color must be understood as a perception, a subjective quality

generated by the brain, and is not to be thought of as a physical property or condition of any

object in the physical world. While color perceptions are initiated by radiant energy reaching the

eye, the perceptual experience that is referred to as color cannot be explained by the reflectance

properties of object surfaces or by the specific spectral composition of light stimuli (Purves et al.

2006).

There is more known about the characteristics of light that initiate color percepts and the

organization of the early visual pathway that generates signals based on different intensities of

1

light along the visible spectrum. The light that reaches the eye from a scene is characterized by

its spectral power distribution. The spectral power distribution generally specifies the radiant

power density (intensity) at each wavelength in the visible spectrum which, for humans, extends

roughly between 400 and 700 nm.

The information from which color perceptions are formed is limited by the layer of

millions of light-sensitive photoreceptors in retina (upon which an inverted image of the world is

projected by the eye’s optics). Humans have two basic types of photoreceptors- rods and cones.

Rods are the initiating elements for the achromatic visual sensations that occur in very low levels

of light and therefore have little effect on color vision. Cones on the other hand transduce

photons arriving at the eye to produce the patterns of electrical signals that lead to color

perception. Three different types of cone photoreceptors underlie human color vision. Each of

these three cone types is characterized by photopigments called opsins and vary in spectral

sensitivity, responding best to a different portion of the visible spectrum. The three types of

cones are referred to as long-, middle-, and short-wavelength- sensitive cones (L, M, and S

cones), according to the part of the visible spectrum to which they are most sensitive (Lotto,

Purves 55-56; Brainard et. al 2009).

Figure 1: The image processing chain illustrates the main stages involved in reproducing color. Light generated by an illuminant reflects from a collection of surfaces, is recorded by a color camera, and stored in digital form. The digital image must be processed by a computer and rendered on a color monitor. The rendered image, when viewed by a human observer, is intended to reproduce an image with the same color appearance at each image location as the original.

2

Knowledge of the organization of the early visual pathway and the information that

initiates color percepts are at the basis of colorimetry studies. Colorimetry tests derived from

color discrimination testing and color matching paradigms have been successful in statistically

representing humans’ ability to distinguish even small changes in the underlying qualities of

color perception (hue, saturation, and brightness) and detect how changes in these qualities vary

as a function of one another. For this reason, colorimetric characterization of light involves the

production of quantitative representations capable of predicting when two lights will appear

identical to a human observer, and assessing how these matches are related to the spectral

sensitivities of the underlying cone photoreceptors.

The development of any system that attempts to reproduce color percepts requires

particular treatments and applications of colorimetry. Take, for example, the image processing

chain of a digital camera (Figure 1) (Brainard et. al 2009). In the case of the digital camera, the

concepts and formulas of colorimetry are the primary consideration in developing transformation

methods to enable the camera to digitally record, store, process, and finally render on a color

monitor, light that is intended to reproduce, at each image location, the same color percepts in

the observer as would the actual scene. At each of these stages, overcoming the challenges to

achieving this reproduction in a digital camera requires analysis of the ways in which the human

visual system encodes the spectral properties of light. Consideration must be given to measuring

and numerically representing the spectral properties of light and the relationships that exist

between these properties. Transformations of these numerical representations must define how

the computer responds to these relationships and characterize the lights that may be produced

with a color monitor (Brainard et. al 2009). As will be explored in more depth later in the paper,

fundamental challenges exist in achieving exact reproduction. At the root of this inability is the

3

major obstacle facing our own explanation of human color vision. “The color percepts elicited by

any light stimulus are determined statistically according to past human experience, rather than by

the features of the stimulus as such” (Purves et. al 2006). In other words, the colorimetric

functions defining human color perception have been determined by humans’ past experience

with natural scenes and cannot be simply explained in terms of the physical qualities of a

stimulus. Thus, as shown in Purves et. al 2006, our understanding of the qualities that make up

color vision and our attempts to reproduce color sensations are limited by the lack of any

physical correspondence of these qualities to retinal stimuli. In the case of the digital camera, this

disconnect accounts for the challenges associated with perfectly reproducing processes of human

color vision.

HUMAN COLOR VISION/COLORIMETRY FUNDAMENTALS

Trichromacy/RGB Color Model

Figure 2: The process of color matching reveals the ranges of colors in human color vision that can be produced by the additive combination of three primary lights. This figure represents the overlapping of red, green, and blue lights that produces regions that appear cyan, purple, yellow, and white. In human color space, intermediate colors within each of these regions can be produced by varying the relative intensities of the three lights.

Individual photoreceptors are effectively color blind. Normal observers’ ability to see

color rather stems from a process of comparing the outputs of these three cone types. Our

understanding of color is based on the only information we have access to- cone receptor

responses to variations in light and our perceptual experience of color sensations. Our

understanding of color is therefore limited to the statistical mapping of subjective color

4

sensations and associated photoreceptor response patterns as they vary with changes in light.

George Palmer and Thomas Young initially defined the phenomenology of human color

sensations as an adjusted mix of three different spectra of light made possible through the three

different types of cone receptors in the retina, which is now termed trichromatic vision. Through

color-matching tests, Young postulated that human color sensations arise from the mixture or

adding together of three wavelengths of light. The responses elicited by the three types of cone

receptors with spectral sensitivities to long, middle, or short ranges of wavelength correspond to

red, green, and blue color sensations respectively (Lotto, Purves 55-56). Color-matching tests,

illustrating humans’ ability to match a test light of any spectral composition to an appropriately

adjusted mixture of just three primary lights at certain intensities, reveal color sensations to be

determined by the absorption spectra of receptors. Consequently, as represented in the Figure 2,

the human visual system is ordered in such a way that color sensations arise in response to three

variables called tristimulus values, which refer to the three primary lights with which they

match- red, green, and blue (Brainard et. al 2009).

Univariance/Human Color Space

Figure 3: Human color space illustrated the organization of the spectral qualities elicited by the spectral distribution of light. Color varies according to the functions by which lightness/brightness, hue, and saturation vary as a function of one another.

(A)

5

The three types of human cone photoreceptors are described as absorbing photons

univariantly, their outputs varying according to the rate at which photons are absorbed. Photon

absorption rates vary with photon wavelengths and the number of incident photons for a given

photoreceptor. Light within different ranges of wavelengths (long, middle, and short), visually

represented in Figure 2, is characterized in terms of its color qualities according to the varying

potential for a given photon to be absorbed at a rate by a cone receptor with a specific spectral

sensitivity. Thus, the absorption of a photon is something independent of the wavelength. Rather

than being defined by any physical property of the retinal stimulus, the qualities that define color

sensation are based in the spectral sensitivities of the cone receptors themselves (Purves et. al

2006). Colorimetry studies account for the quantitative understanding of the degree to which

variations in color qualities impact one another and relate to color sensations (Brainard et. al

2009). This understanding of subjective human color space, illustrated in Figure 3, is formed by

the three qualities of color sensation that characterize the spectral sensitivities of cone receptors:

lightness/brightness, hue, and saturation. For any light of a given intensity (lightness/brightness),

color sensation varies as a function of hue (perceptual levels of the primary colors creating a

color sensation), and saturation (overall appearance of a color with a given brightness and hue

relative to a neutral grey) (Lotto, Purves 60-64)

6

Color Contrast and Constancy

Figure 4: Color contrast and color constancy vary based on interactions of reflectance, illumination, and transmittance. A surface with a given set of reflectance properties is illuminated by variously configured spectra of the same overall intensity. The spectra generated as a product of illumination and reflectance vary systematically in properties (After Lotto and Purves, 2000).

Color contrast and constancy refer to spectral power distributions as affected by qualities

beyond classical color addition and mixing that define color vision in instances where contextual

cues influence color perception. As illustrated in Figure 4, in particular contexts created by the

spectrum of the illuminants for each surface, the reflectance properties of the surfaces, and the

transmittance of the medium through which they are seen, human color vision elicits the same

color sensation for surfaces of different spectral properties and in other cases a different color

sensation for surfaces with exactly the same spectral properties. Consistent with other strategies

of color vision, these kinds of color sensing phenomena are based in past human success arising

from making similar or disparate associations between surfaces of similar or different physical

features in a given context of light stimuli. Instances of the color qualities (brightness/lightness,

hue, and saturation) introduced in the previous section varying as a result of external changes in

the contexts in which they are viewed by an observer certainly add to the complexity of

colorimetry studies and the processes involved in statistically analyzing and quantitatively

representing the levels by which color qualities vary not only as functions of one another but

7

with external changes as well (Lotto, Purves 66-73).

OVERVIEW OF COLORIMETRIC COLOR SPACES & TRANSFORMATIONS

Figure 6: CIE 1931 xyz chromaticity diagram. The color coordinates of a light spectrum provides a rough indication of the color appearance of a stimulus at each chromaticity when viewed in a neutral context.

Figure 7: Color matching functions (CMFs) can be linearly transformed from one set of primaries defining a color space to another. CMFs for R, G, and B primaries (a), for X, Y, and Z primaries (b), and cone fundamental L, M, and S primaries (c).

Highly accurate colorimetric models such as the one representing the CIE standard

colorimetric observer in Figure 6 have come to serve as the basis for transformations throughout

almost all color spaces. Color spaces such as “CIE 1931 XYZ”, based on measurements of

human color perception and cone photoreceptor spectral sensitivities, are considered for their

success in accurately quantifying spectral relationships as they relate to the characteristics of

human color vision covered throughout this paper. Such models represent the color matching

behavior of an individual, characterized as the intensities of three independent primary lights that

are required to match a series of monochromatic spectral lights spanning the visible spectrum

8

(Brainard et. Al 2009). The colorimetric methods used in these cases reveal the possibility of

providing an efficient representation of the spectral properties of light as they affect human

vision from a set of known tristimulus values. For a given color space, two lights with identical

tristimulus values appear to be indistinguishable and may be substituted for one another while

two lights with different tristimulus values are distinguishable to an observer with normal color

vision (Brainard et. al 2009).

The quantitative relationships between spectral power distributions and tristimulus values

depends on the choice of primaries used in the color-matching experiment. Each color space

therefore has a unique set of primary colors. To allow for transformations between color spaces,

as those presented in Figure 7, tristimulus values must be well defined. The terms “color

coordinate system” and “color space” are used to refer to color representations derived with

respect to a particular choice of primaries. Computing tristimulus values depends also on the

color matching functions that fully characterize the properties of the human observer with

respect to a particular set of primaries. Color matching data also take into account the stimulus

conditions or the conditions under which tristimulus values were measured, and qualities of the

individual observer from whom they were measured. With knowledge of the monochromatic

RGB (red, green, blue) primaries and color matching functions for a color space, vector and

matrix transformations enable the computing of tristimulus values and derivation of new color

matching functions with respect to other color spaces defined by different sets of primaries.

Thus, a color space is specified by its primaries and its color matching functions (Brainard et. al

2009). In most cases, choosing or developing a color space to represent the spectral properties of

light according to human color vision involves finding a set of color matching functions that

accurately capture the color matching performance for the set of observers and viewing

9

conditions under consideration (Brainard et. al 2009).

THE DIGITAL CAMERA

Colorimetric Characterization

Figure 5: A Bayer array consists of alternating rows of red-green and green-blue filters. It includes twice as many green as red or blue sensors. Each primary color does not receive an equal fraction of the total area to model the spectral sensitivities of cone photoreceptors that overlap most between this range of wavelengths.

To characterize the colorimetry principles and functions involved in the digital camera’s

attempt to transform instances of light to a correlated point in human color space, it is reasonable

to begin with an overview of how most digital cameras generally sense light. At the moment a

camera’s shutter button is pressed, an array of millions of photosites on the camera’s sensors are

exposed and begin collecting and storing photons. The most common type of color filter array,

represented in Figure 5, is called a Bayer array. Following the moment of exposure, the camera

begins the process of measuring the number of photons absorbed by each photosite. The relative

quantity of photons in each photosite are sorted into various intensity levels (cambridgeincolour;

nikondigital).

Each photosite in the array is equipped with a different filter allowing it to respond to

only intensities of light within one of the three primary ranges of wavelength. Virtually all

current digital cameras only capture one of three primary colors at each photosite cavity, and so

they discard roughly 2/3 of the incoming light. As a result, the digital camera requires further

processing to approximate the levels of every primary color at every pixel and generate full

color. Any of the various algorithms used by different cameras achieves a particular resolution

by quantifying color levels at each pixel through what is referred to as “Bayer Demosaicing”, a

10

process that generally follows as such- Green values for pixels corresponding to red and blue

photosites are interpolated from levels recorded by adjacent green photosites, red values for

pixels corresponding to green and blue photosites are interpolated from levels recorded by

adjacent red photosites, and blue values for pixels corresponding to green and red photosites are

interpolated from levels recorded by adjacent blue photosites (cambridgeincolour; nikondigital).

The colorimetry color spaces and transformation principles described in the previous

section all apply to information specified from spectral information available to a human

observer. However, in an artificial visual system, like that of a digital camera, this information is

not available. Thus, it is the function of complex algorithms in the computing part of a digital

camera’s image processing chain to, based off color matching functions and concepts applied to

colorimetric color spaces and with knowledge of the spectral properties and sensitivities of the

photosites in the sensors, approximately assign color qualities to each pixel of light in the image

projected on the monitor that most closely match those which would lead to a specific color

sensation in a human observer of the scene (Brainard et. al 2009; Hong et al. 2000).

The advanced processing abilities of the digital camera has made it a powerful tool in

capturing images for effective use in color communication. While this is achieved through a very

technical understanding and application of colorimetric principles and functions converting

camera RGB values to XYZ values, the RGB signals generated by a digital camera are ultimately

device-dependent in that different digital cameras produce different RGB responses for the same

scene. Furthermore, the tristimulus values generated are not colorimetric. “The output RGB

signals do not directly correspond to the device-independent tristimulus values based on the CIE

standard colorimetric observer” (Hong et al. 2000). This presents a problem, evident through

instances of eye-camera metamerism- when color contrast/constancy causes two surfaces with

11

spectrally different properties to appear the same to an observer but different in the camera

image or when two surfaces with the same spectral properties appear different to a human

observer but the same in the camera image and vice versa (Hong et al. 2000).

Conclusion- Limits to Reproducing Human Color Vision

Statistical studies mapping perceptual responses to natural scenes show that the spectral

sensitivities of photoreceptors that underlie color sensations have arisen in response to the

behavioral success afforded to humans in making associations between physical objects.

Colorimetry studies have achieved much success in quantifying the spectral qualities and

relationships to which the human visual system responds. However, central to understanding

colorimetry is the major challenge facing any definition of color vision- recognition that color

qualities are not based in any physical characteristics of the retinal stimuli themselves (Purves et

al. 2006).

Such lies the limit to reproducing a visual system capable of sensing light by the exactly

the same spectral qualities as the human visual system. As seen through the limits of a digital

camera’s inability to exactly replicate perceptual color phenomena such as metameres,

application of colorimetric functions and principles, while powerful in working to find ways

around this limitation, do not completely overcome it. There is generally no evidence to suggest

that the human visual system responds to or is based in any physical properties of surfaces

themselves. Thus, the physical measurements a device can take to detect the qualities in question

are limited. For this reason, the goal of creating a visual system that replicates human color-

sensing abilities, while capable of achieving impressive proximity through application of

colorimetry, faces a major challenge for the sake of being fundamentally flawed from the start.

12

Bibliography

Brainard, D. H. & Stockman, A. (2009). Colorimetry. In the OSA Handbook of Optics (3rd

edition, M. Bass, ed). McGraw-Hill, New York, 10.1-10.56.

Digital Camera Sensors. (n.d.). http://www.cambridgeincolour.com/tutorials/camera-sensors.htm

Hong, Guowei, M. Ronnier Luo, and Peter A. Rhodes (2000). "A Study of Digital Camera

Colorimetric Characterization Based on Polynomial Modeling." Color Research &

Application: 76-84.

Long, F., Yang, Z., & Purves, D. (2006). Spectral statistics in natural scenes predict hue,

saturation, and brightness. Proceedings of the National Academy of Sciences, 103(15),

6013-6018.

Purves, D., & Lotto, R. B. (2011). “Seeing Color”. In Why We See What We Do: A

Wholly Empirical Theory of Vision. Sunderland, MA: Sinauer Associates. 51-90.

RGB: All About Digital Camera Color. (n.d.). Retrieved from

http://www.nikondigital.org/articles/rgb_digital_camera_color.htm

13