Bela Borsodics.brown.edu/courses/cs143/lectures/2020Spring_08_Camera... · 2020. 2. 12. · •...
Transcript of Bela Borsodics.brown.edu/courses/cs143/lectures/2020Spring_08_Camera... · 2020. 2. 12. · •...
![Page 1: Bela Borsodics.brown.edu/courses/cs143/lectures/2020Spring_08_Camera... · 2020. 2. 12. · • Most birds, and many other animals, have cones for ultraviolet light. • Some humans](https://reader036.fdocuments.net/reader036/viewer/2022062607/6053608a2540ce071d48b495/html5/thumbnails/1.jpg)
![Page 2: Bela Borsodics.brown.edu/courses/cs143/lectures/2020Spring_08_Camera... · 2020. 2. 12. · • Most birds, and many other animals, have cones for ultraviolet light. • Some humans](https://reader036.fdocuments.net/reader036/viewer/2022062607/6053608a2540ce071d48b495/html5/thumbnails/2.jpg)
Bela Borsodi
![Page 3: Bela Borsodics.brown.edu/courses/cs143/lectures/2020Spring_08_Camera... · 2020. 2. 12. · • Most birds, and many other animals, have cones for ultraviolet light. • Some humans](https://reader036.fdocuments.net/reader036/viewer/2022062607/6053608a2540ce071d48b495/html5/thumbnails/3.jpg)
Bela Borsodi
![Page 4: Bela Borsodics.brown.edu/courses/cs143/lectures/2020Spring_08_Camera... · 2020. 2. 12. · • Most birds, and many other animals, have cones for ultraviolet light. • Some humans](https://reader036.fdocuments.net/reader036/viewer/2022062607/6053608a2540ce071d48b495/html5/thumbnails/4.jpg)
Recap so far
• What is an image?
– Sampling and aliasing
– Thinking in frequency
• Image filtering
– Kernels and their responses
– Kernel orientation
– Kernel sizes and scale spaces
• Local features
– Detection by finding corners via peaks
– Description by hand-coding local ‘texture’
– Matching robustly
![Page 5: Bela Borsodics.brown.edu/courses/cs143/lectures/2020Spring_08_Camera... · 2020. 2. 12. · • Most birds, and many other animals, have cones for ultraviolet light. • Some humans](https://reader036.fdocuments.net/reader036/viewer/2022062607/6053608a2540ce071d48b495/html5/thumbnails/5.jpg)
COLORElephant in room:
![Page 6: Bela Borsodics.brown.edu/courses/cs143/lectures/2020Spring_08_Camera... · 2020. 2. 12. · • Most birds, and many other animals, have cones for ultraviolet light. • Some humans](https://reader036.fdocuments.net/reader036/viewer/2022062607/6053608a2540ce071d48b495/html5/thumbnails/6.jpg)
Grayscale intensity
0.92 0.93 0.94 0.97 0.62 0.37 0.85 0.97 0.93 0.92 0.99
0.95 0.89 0.82 0.89 0.56 0.31 0.75 0.92 0.81 0.95 0.91
0.89 0.72 0.51 0.55 0.51 0.42 0.57 0.41 0.49 0.91 0.92
0.96 0.95 0.88 0.94 0.56 0.46 0.91 0.87 0.90 0.97 0.95
0.71 0.81 0.81 0.87 0.57 0.37 0.80 0.88 0.89 0.79 0.85
0.49 0.62 0.60 0.58 0.50 0.60 0.58 0.50 0.61 0.45 0.33
0.86 0.84 0.74 0.58 0.51 0.39 0.73 0.92 0.91 0.49 0.74
0.96 0.67 0.54 0.85 0.48 0.37 0.88 0.90 0.94 0.82 0.93
0.69 0.49 0.56 0.66 0.43 0.42 0.77 0.73 0.71 0.90 0.99
0.79 0.73 0.90 0.67 0.33 0.61 0.69 0.79 0.73 0.93 0.97
0.91 0.94 0.89 0.49 0.41 0.78 0.78 0.77 0.89 0.99 0.93
![Page 7: Bela Borsodics.brown.edu/courses/cs143/lectures/2020Spring_08_Camera... · 2020. 2. 12. · • Most birds, and many other animals, have cones for ultraviolet light. • Some humans](https://reader036.fdocuments.net/reader036/viewer/2022062607/6053608a2540ce071d48b495/html5/thumbnails/7.jpg)
ColorR
G
B
James Hays
![Page 8: Bela Borsodics.brown.edu/courses/cs143/lectures/2020Spring_08_Camera... · 2020. 2. 12. · • Most birds, and many other animals, have cones for ultraviolet light. • Some humans](https://reader036.fdocuments.net/reader036/viewer/2022062607/6053608a2540ce071d48b495/html5/thumbnails/8.jpg)
Images in Python NumpyN x M RGB image “im”
– im[0,0,0] = top-left pixel value in R-channel
– Im[x, y, b] = x pixels to right, y pixels down in the bth channel
– Im[N-1, M-1, 3] = bottom-right pixel in B-channel
0.92 0.93 0.94 0.97 0.62 0.37 0.85 0.97 0.93 0.92 0.99
0.95 0.89 0.82 0.89 0.56 0.31 0.75 0.92 0.81 0.95 0.91
0.89 0.72 0.51 0.55 0.51 0.42 0.57 0.41 0.49 0.91 0.92
0.96 0.95 0.88 0.94 0.56 0.46 0.91 0.87 0.90 0.97 0.95
0.71 0.81 0.81 0.87 0.57 0.37 0.80 0.88 0.89 0.79 0.85
0.49 0.62 0.60 0.58 0.50 0.60 0.58 0.50 0.61 0.45 0.33
0.86 0.84 0.74 0.58 0.51 0.39 0.73 0.92 0.91 0.49 0.74
0.96 0.67 0.54 0.85 0.48 0.37 0.88 0.90 0.94 0.82 0.93
0.69 0.49 0.56 0.66 0.43 0.42 0.77 0.73 0.71 0.90 0.99
0.79 0.73 0.90 0.67 0.33 0.61 0.69 0.79 0.73 0.93 0.97
0.91 0.94 0.89 0.49 0.41 0.78 0.78 0.77 0.89 0.99 0.93
0.92 0.93 0.94 0.97 0.62 0.37 0.85 0.97 0.93 0.92 0.99
0.95 0.89 0.82 0.89 0.56 0.31 0.75 0.92 0.81 0.95 0.91
0.89 0.72 0.51 0.55 0.51 0.42 0.57 0.41 0.49 0.91 0.92
0.96 0.95 0.88 0.94 0.56 0.46 0.91 0.87 0.90 0.97 0.95
0.71 0.81 0.81 0.87 0.57 0.37 0.80 0.88 0.89 0.79 0.85
0.49 0.62 0.60 0.58 0.50 0.60 0.58 0.50 0.61 0.45 0.33
0.86 0.84 0.74 0.58 0.51 0.39 0.73 0.92 0.91 0.49 0.74
0.96 0.67 0.54 0.85 0.48 0.37 0.88 0.90 0.94 0.82 0.93
0.69 0.49 0.56 0.66 0.43 0.42 0.77 0.73 0.71 0.90 0.99
0.79 0.73 0.90 0.67 0.33 0.61 0.69 0.79 0.73 0.93 0.97
0.91 0.94 0.89 0.49 0.41 0.78 0.78 0.77 0.89 0.99 0.93
0.92 0.93 0.94 0.97 0.62 0.37 0.85 0.97 0.93 0.92 0.99
0.95 0.89 0.82 0.89 0.56 0.31 0.75 0.92 0.81 0.95 0.91
0.89 0.72 0.51 0.55 0.51 0.42 0.57 0.41 0.49 0.91 0.92
0.96 0.95 0.88 0.94 0.56 0.46 0.91 0.87 0.90 0.97 0.95
0.71 0.81 0.81 0.87 0.57 0.37 0.80 0.88 0.89 0.79 0.85
0.49 0.62 0.60 0.58 0.50 0.60 0.58 0.50 0.61 0.45 0.33
0.86 0.84 0.74 0.58 0.51 0.39 0.73 0.92 0.91 0.49 0.74
0.96 0.67 0.54 0.85 0.48 0.37 0.88 0.90 0.94 0.82 0.93
0.69 0.49 0.56 0.66 0.43 0.42 0.77 0.73 0.71 0.90 0.99
0.79 0.73 0.90 0.67 0.33 0.61 0.69 0.79 0.73 0.93 0.97
0.91 0.94 0.89 0.49 0.41 0.78 0.78 0.77 0.89 0.99 0.93
R
G
B
RowColumn
James Hays
![Page 9: Bela Borsodics.brown.edu/courses/cs143/lectures/2020Spring_08_Camera... · 2020. 2. 12. · • Most birds, and many other animals, have cones for ultraviolet light. • Some humans](https://reader036.fdocuments.net/reader036/viewer/2022062607/6053608a2540ce071d48b495/html5/thumbnails/9.jpg)
ANATOMYBut what is color?
![Page 10: Bela Borsodics.brown.edu/courses/cs143/lectures/2020Spring_08_Camera... · 2020. 2. 12. · • Most birds, and many other animals, have cones for ultraviolet light. • Some humans](https://reader036.fdocuments.net/reader036/viewer/2022062607/6053608a2540ce071d48b495/html5/thumbnails/10.jpg)
The Eye
• The human eye is a camera– Iris - colored annulus with radial muscles
– Pupil - the hole (aperture) whose size is controlled by the iris
– What’s the sensor?
– photoreceptor cells (rods and cones) in the retina
Slide by Steve Seitz
![Page 11: Bela Borsodics.brown.edu/courses/cs143/lectures/2020Spring_08_Camera... · 2020. 2. 12. · • Most birds, and many other animals, have cones for ultraviolet light. • Some humans](https://reader036.fdocuments.net/reader036/viewer/2022062607/6053608a2540ce071d48b495/html5/thumbnails/11.jpg)
The Retina
Cross-section of eye
Ganglion cell layer
Bipolar cell layer
Receptor layer
Pigmentedepithelium
Ganglion axons
Cross section of retina
Axon = nerve fibre
Ganglion cell = a neuron (nerve cell) Bipolar cell = pass ‘graded signal changes’
![Page 12: Bela Borsodics.brown.edu/courses/cs143/lectures/2020Spring_08_Camera... · 2020. 2. 12. · • Most birds, and many other animals, have cones for ultraviolet light. • Some humans](https://reader036.fdocuments.net/reader036/viewer/2022062607/6053608a2540ce071d48b495/html5/thumbnails/12.jpg)
Wait, the blood vessels are in front of the photoreceptors??
https://www.youtube.com/watch?v=L_W-IXqoxHA
![Page 13: Bela Borsodics.brown.edu/courses/cs143/lectures/2020Spring_08_Camera... · 2020. 2. 12. · • Most birds, and many other animals, have cones for ultraviolet light. • Some humans](https://reader036.fdocuments.net/reader036/viewer/2022062607/6053608a2540ce071d48b495/html5/thumbnails/13.jpg)
What humans don’t have: tapetum lucidum
Human eyes can reflect a tiny
bit and blood in the retina
makes this reflection red.
James Hays
![Page 14: Bela Borsodics.brown.edu/courses/cs143/lectures/2020Spring_08_Camera... · 2020. 2. 12. · • Most birds, and many other animals, have cones for ultraviolet light. • Some humans](https://reader036.fdocuments.net/reader036/viewer/2022062607/6053608a2540ce071d48b495/html5/thumbnails/14.jpg)
Tapetum lucidum exposed (cow eye)
![Page 15: Bela Borsodics.brown.edu/courses/cs143/lectures/2020Spring_08_Camera... · 2020. 2. 12. · • Most birds, and many other animals, have cones for ultraviolet light. • Some humans](https://reader036.fdocuments.net/reader036/viewer/2022062607/6053608a2540ce071d48b495/html5/thumbnails/15.jpg)
© Stephen E. Palmer, 2002
Cones
cone-shaped
less sensitive
operate in high light
color vision
Two types of light-sensitive receptors
Rods
rod-shaped
highly sensitive
operate at night
gray-scale vision
James Hays
![Page 16: Bela Borsodics.brown.edu/courses/cs143/lectures/2020Spring_08_Camera... · 2020. 2. 12. · • Most birds, and many other animals, have cones for ultraviolet light. • Some humans](https://reader036.fdocuments.net/reader036/viewer/2022062607/6053608a2540ce071d48b495/html5/thumbnails/16.jpg)
.
0
150,000
100,000
50,000
020 40 60 8020406080
Visual Angle (degrees from fovea)
Rods
Cones Cones
Rods
FoveaBlindSpot
# R
ecepto
rs/m
m2
© Stephen E. Palmer, 2002
Distribution of Rods and Cones
Night Sky: why are there more stars off-center?
Averted vision: http://en.wikipedia.org/wiki/Averted_visionJames Hays
![Page 17: Bela Borsodics.brown.edu/courses/cs143/lectures/2020Spring_08_Camera... · 2020. 2. 12. · • Most birds, and many other animals, have cones for ultraviolet light. • Some humans](https://reader036.fdocuments.net/reader036/viewer/2022062607/6053608a2540ce071d48b495/html5/thumbnails/17.jpg)
Rod / Cone sensitivity
![Page 18: Bela Borsodics.brown.edu/courses/cs143/lectures/2020Spring_08_Camera... · 2020. 2. 12. · • Most birds, and many other animals, have cones for ultraviolet light. • Some humans](https://reader036.fdocuments.net/reader036/viewer/2022062607/6053608a2540ce071d48b495/html5/thumbnails/18.jpg)
Does the eye alias?
Spatially, apparently not. The retina (sensor) has high resolution, but the optics (lens) of the eye cannot meet that resolution. The image is blurred optically before being sampled (removes high-frequency content!)
[Thanks to Leslie Bresnahan]
4x downsample
nearest neighbor
![Page 19: Bela Borsodics.brown.edu/courses/cs143/lectures/2020Spring_08_Camera... · 2020. 2. 12. · • Most birds, and many other animals, have cones for ultraviolet light. • Some humans](https://reader036.fdocuments.net/reader036/viewer/2022062607/6053608a2540ce071d48b495/html5/thumbnails/19.jpg)
Electromagnetic Spectrum
http://www.yorku.ca/eye/photopik.htm
Wavelength of
light and its
perceived color
![Page 20: Bela Borsodics.brown.edu/courses/cs143/lectures/2020Spring_08_Camera... · 2020. 2. 12. · • Most birds, and many other animals, have cones for ultraviolet light. • Some humans](https://reader036.fdocuments.net/reader036/viewer/2022062607/6053608a2540ce071d48b495/html5/thumbnails/20.jpg)
© Stephen E. Palmer, 2002
.
400 450 500 550 600 650
RE
LA
TIV
E A
BS
OR
BA
NC
E (
%)
WAVELENGTH (nm.)
100
50
440
S
530 560 nm.
M L
Three kinds of cones:
Physiology of Color Vision
![Page 21: Bela Borsodics.brown.edu/courses/cs143/lectures/2020Spring_08_Camera... · 2020. 2. 12. · • Most birds, and many other animals, have cones for ultraviolet light. • Some humans](https://reader036.fdocuments.net/reader036/viewer/2022062607/6053608a2540ce071d48b495/html5/thumbnails/21.jpg)
The Physics of Light
Any patch of light can be completely described
physically by its spectrum: the number of photons
(per time unit) at each wavelength 400 - 700 nm.
400 500 600 700
Wavelength (nm.)
# Photons(per ms.)
© Stephen E. Palmer, 2002
![Page 22: Bela Borsodics.brown.edu/courses/cs143/lectures/2020Spring_08_Camera... · 2020. 2. 12. · • Most birds, and many other animals, have cones for ultraviolet light. • Some humans](https://reader036.fdocuments.net/reader036/viewer/2022062607/6053608a2540ce071d48b495/html5/thumbnails/22.jpg)
The Physics of Light
.
# P
ho
tons
D. Normal Daylight
Wavelength (nm.)
B. Gallium Phosphide Crystal
400 500 600 700
# P
ho
tons
Wavelength (nm.)
A. Ruby Laser
400 500 600 700
400 500 600 700
# P
ho
tons
C. Tungsten Lightbulb
400 500 600 700
# P
ho
tons
Some examples of the spectra of light sources
© Stephen E. Palmer, 2002
![Page 23: Bela Borsodics.brown.edu/courses/cs143/lectures/2020Spring_08_Camera... · 2020. 2. 12. · • Most birds, and many other animals, have cones for ultraviolet light. • Some humans](https://reader036.fdocuments.net/reader036/viewer/2022062607/6053608a2540ce071d48b495/html5/thumbnails/23.jpg)
The Physics of Light
Some examples of the reflectance spectra of surfaces
Wavelength (nm)
% P
hoto
ns R
eflecte
d
Red
400 700
Yellow
400 700
Blue
400 700
Purple
400 700
© Stephen E. Palmer, 2002
![Page 24: Bela Borsodics.brown.edu/courses/cs143/lectures/2020Spring_08_Camera... · 2020. 2. 12. · • Most birds, and many other animals, have cones for ultraviolet light. • Some humans](https://reader036.fdocuments.net/reader036/viewer/2022062607/6053608a2540ce071d48b495/html5/thumbnails/24.jpg)
© Stephen E. Palmer, 2002
.
400 450 500 550 600 650
RE
LA
TIV
E A
BS
OR
BA
NC
E (
%)
WAVELENGTH (nm.)
100
50
440
S
530 560 nm.
M L
Three kinds of cones:
Physiology of Color Vision
• Why are M and L cones so close?
• Why are there 3?
![Page 25: Bela Borsodics.brown.edu/courses/cs143/lectures/2020Spring_08_Camera... · 2020. 2. 12. · • Most birds, and many other animals, have cones for ultraviolet light. • Some humans](https://reader036.fdocuments.net/reader036/viewer/2022062607/6053608a2540ce071d48b495/html5/thumbnails/25.jpg)
Electromagnetic Spectrum
http://www.yorku.ca/eye/photopik.htm
Human Luminance Sensitivity Function
![Page 26: Bela Borsodics.brown.edu/courses/cs143/lectures/2020Spring_08_Camera... · 2020. 2. 12. · • Most birds, and many other animals, have cones for ultraviolet light. • Some humans](https://reader036.fdocuments.net/reader036/viewer/2022062607/6053608a2540ce071d48b495/html5/thumbnails/26.jpg)
Tetrachromatism
• Most birds, and many other animals, have cones for ultraviolet light.
• Some humans seem to have four cones (12% of females).
• True tetrachromatism is _rare_; requires learning.
Bird cone
responses
James Hays
![Page 27: Bela Borsodics.brown.edu/courses/cs143/lectures/2020Spring_08_Camera... · 2020. 2. 12. · • Most birds, and many other animals, have cones for ultraviolet light. • Some humans](https://reader036.fdocuments.net/reader036/viewer/2022062607/6053608a2540ce071d48b495/html5/thumbnails/27.jpg)
Bee vision
![Page 28: Bela Borsodics.brown.edu/courses/cs143/lectures/2020Spring_08_Camera... · 2020. 2. 12. · • Most birds, and many other animals, have cones for ultraviolet light. • Some humans](https://reader036.fdocuments.net/reader036/viewer/2022062607/6053608a2540ce071d48b495/html5/thumbnails/28.jpg)
Mantis Shrimp
• 16 photoreceptor types
• ‘Mantis Shrimp’ camera: https://www.theatlantic.com/science/archive/2018/04/mantis-shrimp-eye-camera/557195/
![Page 29: Bela Borsodics.brown.edu/courses/cs143/lectures/2020Spring_08_Camera... · 2020. 2. 12. · • Most birds, and many other animals, have cones for ultraviolet light. • Some humans](https://reader036.fdocuments.net/reader036/viewer/2022062607/6053608a2540ce071d48b495/html5/thumbnails/29.jpg)
![Page 30: Bela Borsodics.brown.edu/courses/cs143/lectures/2020Spring_08_Camera... · 2020. 2. 12. · • Most birds, and many other animals, have cones for ultraviolet light. • Some humans](https://reader036.fdocuments.net/reader036/viewer/2022062607/6053608a2540ce071d48b495/html5/thumbnails/30.jpg)
Impossible Colors
Can you make the cones respond in ways that typical light spectra never would?
http://en.wikipedia.org/wiki/Impossible_colors
James Hays
![Page 31: Bela Borsodics.brown.edu/courses/cs143/lectures/2020Spring_08_Camera... · 2020. 2. 12. · • Most birds, and many other animals, have cones for ultraviolet light. • Some humans](https://reader036.fdocuments.net/reader036/viewer/2022062607/6053608a2540ce071d48b495/html5/thumbnails/31.jpg)
![Page 32: Bela Borsodics.brown.edu/courses/cs143/lectures/2020Spring_08_Camera... · 2020. 2. 12. · • Most birds, and many other animals, have cones for ultraviolet light. • Some humans](https://reader036.fdocuments.net/reader036/viewer/2022062607/6053608a2540ce071d48b495/html5/thumbnails/32.jpg)
What is color?
Why do we even care about human vision in this class?
![Page 33: Bela Borsodics.brown.edu/courses/cs143/lectures/2020Spring_08_Camera... · 2020. 2. 12. · • Most birds, and many other animals, have cones for ultraviolet light. • Some humans](https://reader036.fdocuments.net/reader036/viewer/2022062607/6053608a2540ce071d48b495/html5/thumbnails/33.jpg)
Why do we care about human vision?
• We don’t, necessarily.
• But biological vision shows that it is possible to make important judgements from images.
James Hays
![Page 34: Bela Borsodics.brown.edu/courses/cs143/lectures/2020Spring_08_Camera... · 2020. 2. 12. · • Most birds, and many other animals, have cones for ultraviolet light. • Some humans](https://reader036.fdocuments.net/reader036/viewer/2022062607/6053608a2540ce071d48b495/html5/thumbnails/34.jpg)
Why do we care about human vision?
• We don’t, necessarily.
• But biological vision shows that it is possible to make important judgements from images.
• It’s a human world -> cameras imitate the frequency response of the human eye to try to see as we see.
![Page 35: Bela Borsodics.brown.edu/courses/cs143/lectures/2020Spring_08_Camera... · 2020. 2. 12. · • Most birds, and many other animals, have cones for ultraviolet light. • Some humans](https://reader036.fdocuments.net/reader036/viewer/2022062607/6053608a2540ce071d48b495/html5/thumbnails/35.jpg)
Ornithopters
James Hays
![Page 36: Bela Borsodics.brown.edu/courses/cs143/lectures/2020Spring_08_Camera... · 2020. 2. 12. · • Most birds, and many other animals, have cones for ultraviolet light. • Some humans](https://reader036.fdocuments.net/reader036/viewer/2022062607/6053608a2540ce071d48b495/html5/thumbnails/36.jpg)
"Can machines fly like a bird?" No, because airplanes don’t flap.
"Can machines fly?" Yes, but airplanes use a different mechanism.
"Can machines perceive?" Is this question like the first, or like the second?
Adapted from Peter Norvig
![Page 37: Bela Borsodics.brown.edu/courses/cs143/lectures/2020Spring_08_Camera... · 2020. 2. 12. · • Most birds, and many other animals, have cones for ultraviolet light. • Some humans](https://reader036.fdocuments.net/reader036/viewer/2022062607/6053608a2540ce071d48b495/html5/thumbnails/37.jpg)
Cameras with Three Sensors
Objective Lens
[Edmund Optics; Adam Wilt]
![Page 38: Bela Borsodics.brown.edu/courses/cs143/lectures/2020Spring_08_Camera... · 2020. 2. 12. · • Most birds, and many other animals, have cones for ultraviolet light. • Some humans](https://reader036.fdocuments.net/reader036/viewer/2022062607/6053608a2540ce071d48b495/html5/thumbnails/38.jpg)
Color Sensing in Camera (RGB)
• 3-chip vs. 1-chip: quality vs. cost
• Why more green?
http://www.cooldictionary.com/words/Bayer-filter.wikipedia
Why 3 colors?
Slide by Steve Seitz
![Page 39: Bela Borsodics.brown.edu/courses/cs143/lectures/2020Spring_08_Camera... · 2020. 2. 12. · • Most birds, and many other animals, have cones for ultraviolet light. • Some humans](https://reader036.fdocuments.net/reader036/viewer/2022062607/6053608a2540ce071d48b495/html5/thumbnails/39.jpg)
Cheaper/More Compact Color Sensing: Bayer Grid
Estimate RGBat ‘G’ cells from neighboring values
Slide by Steve Seitz
![Page 40: Bela Borsodics.brown.edu/courses/cs143/lectures/2020Spring_08_Camera... · 2020. 2. 12. · • Most birds, and many other animals, have cones for ultraviolet light. • Some humans](https://reader036.fdocuments.net/reader036/viewer/2022062607/6053608a2540ce071d48b495/html5/thumbnails/40.jpg)
Why more green?
Approximate human spectral sensitivity
Less than
~400nm to 10nm
= ultraviolet (UV)
Greater than
~700nm to 1mm
= infrared (IR)
Human visible portion
of electromagnetic
(EM) spectrum
![Page 41: Bela Borsodics.brown.edu/courses/cs143/lectures/2020Spring_08_Camera... · 2020. 2. 12. · • Most birds, and many other animals, have cones for ultraviolet light. • Some humans](https://reader036.fdocuments.net/reader036/viewer/2022062607/6053608a2540ce071d48b495/html5/thumbnails/41.jpg)
RGB Camera Color Response
What’s going
on over here?
MaxMax.com
![Page 42: Bela Borsodics.brown.edu/courses/cs143/lectures/2020Spring_08_Camera... · 2020. 2. 12. · • Most birds, and many other animals, have cones for ultraviolet light. • Some humans](https://reader036.fdocuments.net/reader036/viewer/2022062607/6053608a2540ce071d48b495/html5/thumbnails/42.jpg)
Display Color Response
![Page 43: Bela Borsodics.brown.edu/courses/cs143/lectures/2020Spring_08_Camera... · 2020. 2. 12. · • Most birds, and many other animals, have cones for ultraviolet light. • Some humans](https://reader036.fdocuments.net/reader036/viewer/2022062607/6053608a2540ce071d48b495/html5/thumbnails/43.jpg)
Display Color Response
![Page 44: Bela Borsodics.brown.edu/courses/cs143/lectures/2020Spring_08_Camera... · 2020. 2. 12. · • Most birds, and many other animals, have cones for ultraviolet light. • Some humans](https://reader036.fdocuments.net/reader036/viewer/2022062607/6053608a2540ce071d48b495/html5/thumbnails/44.jpg)
Color spaces
How can we represent color?
http://en.wikipedia.org/wiki/File:RGB_illumination.jpg
![Page 45: Bela Borsodics.brown.edu/courses/cs143/lectures/2020Spring_08_Camera... · 2020. 2. 12. · • Most birds, and many other animals, have cones for ultraviolet light. • Some humans](https://reader036.fdocuments.net/reader036/viewer/2022062607/6053608a2540ce071d48b495/html5/thumbnails/45.jpg)
Color spaces: RGB
0,1,0
0,0,1
1,0,0
Image from: http://en.wikipedia.org/wiki/File:RGB_color_solid_cube.png
Any color = r*R + g*G + b*B• Strongly correlated channels
• Non-perceptual
Default color space
R = 1(G=0,B=0)
G = 1(R=0,B=0)
B = 1(R=0,G=0)
![Page 46: Bela Borsodics.brown.edu/courses/cs143/lectures/2020Spring_08_Camera... · 2020. 2. 12. · • Most birds, and many other animals, have cones for ultraviolet light. • Some humans](https://reader036.fdocuments.net/reader036/viewer/2022062607/6053608a2540ce071d48b495/html5/thumbnails/46.jpg)
IS COLOR A VECTOR SPACE?THINK-PAIR-SHARE
Got it. C = r*R + g*G + b*B
![Page 47: Bela Borsodics.brown.edu/courses/cs143/lectures/2020Spring_08_Camera... · 2020. 2. 12. · • Most birds, and many other animals, have cones for ultraviolet light. • Some humans](https://reader036.fdocuments.net/reader036/viewer/2022062607/6053608a2540ce071d48b495/html5/thumbnails/47.jpg)
Color spaces: HSV
Intuitive color space
![Page 48: Bela Borsodics.brown.edu/courses/cs143/lectures/2020Spring_08_Camera... · 2020. 2. 12. · • Most birds, and many other animals, have cones for ultraviolet light. • Some humans](https://reader036.fdocuments.net/reader036/viewer/2022062607/6053608a2540ce071d48b495/html5/thumbnails/48.jpg)
If you had to choose, would you rather go without:- intensity (‘value’), or- hue + saturation (‘chroma’)?
Think-Pair-Share
James Hays
![Page 49: Bela Borsodics.brown.edu/courses/cs143/lectures/2020Spring_08_Camera... · 2020. 2. 12. · • Most birds, and many other animals, have cones for ultraviolet light. • Some humans](https://reader036.fdocuments.net/reader036/viewer/2022062607/6053608a2540ce071d48b495/html5/thumbnails/49.jpg)
Most information in intensity
Only color shown – constant intensity
James Hays
![Page 50: Bela Borsodics.brown.edu/courses/cs143/lectures/2020Spring_08_Camera... · 2020. 2. 12. · • Most birds, and many other animals, have cones for ultraviolet light. • Some humans](https://reader036.fdocuments.net/reader036/viewer/2022062607/6053608a2540ce071d48b495/html5/thumbnails/50.jpg)
Most information in intensity
Only intensity shown – constant color
James Hays
![Page 51: Bela Borsodics.brown.edu/courses/cs143/lectures/2020Spring_08_Camera... · 2020. 2. 12. · • Most birds, and many other animals, have cones for ultraviolet light. • Some humans](https://reader036.fdocuments.net/reader036/viewer/2022062607/6053608a2540ce071d48b495/html5/thumbnails/51.jpg)
Most information in intensity
Original image
James Hays
![Page 52: Bela Borsodics.brown.edu/courses/cs143/lectures/2020Spring_08_Camera... · 2020. 2. 12. · • Most birds, and many other animals, have cones for ultraviolet light. • Some humans](https://reader036.fdocuments.net/reader036/viewer/2022062607/6053608a2540ce071d48b495/html5/thumbnails/52.jpg)
Color spaces: HSV
Intuitive color space
H(S=1,V=1)
S(H=1,V=1)
V(H=1,S=0)
James Hays
![Page 53: Bela Borsodics.brown.edu/courses/cs143/lectures/2020Spring_08_Camera... · 2020. 2. 12. · • Most birds, and many other animals, have cones for ultraviolet light. • Some humans](https://reader036.fdocuments.net/reader036/viewer/2022062607/6053608a2540ce071d48b495/html5/thumbnails/53.jpg)
Color spaces: YCbCr
Y(Cb=0.5,Cr=0.5)
Cb(Y=0.5,Cr=0.5)
Cr(Y=0.5,Cb=05)
Y=0 Y=0.5
Y=1Cb
Cr
Fast to compute, good for
compression, used by TV
James Hays
![Page 54: Bela Borsodics.brown.edu/courses/cs143/lectures/2020Spring_08_Camera... · 2020. 2. 12. · • Most birds, and many other animals, have cones for ultraviolet light. • Some humans](https://reader036.fdocuments.net/reader036/viewer/2022062607/6053608a2540ce071d48b495/html5/thumbnails/54.jpg)
Most JPEG images & videos subsample chroma
![Page 55: Bela Borsodics.brown.edu/courses/cs143/lectures/2020Spring_08_Camera... · 2020. 2. 12. · • Most birds, and many other animals, have cones for ultraviolet light. • Some humans](https://reader036.fdocuments.net/reader036/viewer/2022062607/6053608a2540ce071d48b495/html5/thumbnails/55.jpg)
IS COLOR PERCEPTION A VECTOR SPACE?
![Page 56: Bela Borsodics.brown.edu/courses/cs143/lectures/2020Spring_08_Camera... · 2020. 2. 12. · • Most birds, and many other animals, have cones for ultraviolet light. • Some humans](https://reader036.fdocuments.net/reader036/viewer/2022062607/6053608a2540ce071d48b495/html5/thumbnails/56.jpg)
Color spaces: L*a*b*
“Perceptually uniform”* color space
L(a=0,b=0)
a(L=65,b=0)
b(L=65,a=0)
James Hays
![Page 57: Bela Borsodics.brown.edu/courses/cs143/lectures/2020Spring_08_Camera... · 2020. 2. 12. · • Most birds, and many other animals, have cones for ultraviolet light. • Some humans](https://reader036.fdocuments.net/reader036/viewer/2022062607/6053608a2540ce071d48b495/html5/thumbnails/57.jpg)
XKCD
![Page 58: Bela Borsodics.brown.edu/courses/cs143/lectures/2020Spring_08_Camera... · 2020. 2. 12. · • Most birds, and many other animals, have cones for ultraviolet light. • Some humans](https://reader036.fdocuments.net/reader036/viewer/2022062607/6053608a2540ce071d48b495/html5/thumbnails/58.jpg)
More references
• https://www.colorsystem.com/
• A description of many different color systems developed through history.
• Navigate from the right-hand links.
• Thanks to Alex Nibley!
![Page 59: Bela Borsodics.brown.edu/courses/cs143/lectures/2020Spring_08_Camera... · 2020. 2. 12. · • Most birds, and many other animals, have cones for ultraviolet light. • Some humans](https://reader036.fdocuments.net/reader036/viewer/2022062607/6053608a2540ce071d48b495/html5/thumbnails/59.jpg)
![Page 60: Bela Borsodics.brown.edu/courses/cs143/lectures/2020Spring_08_Camera... · 2020. 2. 12. · • Most birds, and many other animals, have cones for ultraviolet light. • Some humans](https://reader036.fdocuments.net/reader036/viewer/2022062607/6053608a2540ce071d48b495/html5/thumbnails/60.jpg)
![Page 61: Bela Borsodics.brown.edu/courses/cs143/lectures/2020Spring_08_Camera... · 2020. 2. 12. · • Most birds, and many other animals, have cones for ultraviolet light. • Some humans](https://reader036.fdocuments.net/reader036/viewer/2022062607/6053608a2540ce071d48b495/html5/thumbnails/61.jpg)
Rainbow color map considered harmfulBorland and Taylor
![Page 62: Bela Borsodics.brown.edu/courses/cs143/lectures/2020Spring_08_Camera... · 2020. 2. 12. · • Most birds, and many other animals, have cones for ultraviolet light. • Some humans](https://reader036.fdocuments.net/reader036/viewer/2022062607/6053608a2540ce071d48b495/html5/thumbnails/62.jpg)
WHY DOES COLOR LOOK LIKE IT MAPS SMOOTHLY TO A CIRCLE?
Wait a minute…
“Intuitive” color space?
![Page 63: Bela Borsodics.brown.edu/courses/cs143/lectures/2020Spring_08_Camera... · 2020. 2. 12. · • Most birds, and many other animals, have cones for ultraviolet light. • Some humans](https://reader036.fdocuments.net/reader036/viewer/2022062607/6053608a2540ce071d48b495/html5/thumbnails/63.jpg)
‘Color’ != position on EM spectrum
Our cells induce color perception by interpreting spectra.
Most mammals are dichromats:
• Lack ‘L’ cone; cannot distinguish green-red
• 1% of men (protanopia color blindness)
Trichromaticity evolved.No implicit reason for effect ofextra cone to be linear.
Thanks to Cam Allen-Lloyd
.
400 450 500 550 600 650
RE
LA
TIV
E A
BS
OR
BA
NC
E (
%)
WAVELENGTH (nm.)
100
50
440
S
530 560 nm.
M L
![Page 64: Bela Borsodics.brown.edu/courses/cs143/lectures/2020Spring_08_Camera... · 2020. 2. 12. · • Most birds, and many other animals, have cones for ultraviolet light. • Some humans](https://reader036.fdocuments.net/reader036/viewer/2022062607/6053608a2540ce071d48b495/html5/thumbnails/64.jpg)
‘Color’ != position on EM spectrum
Many different ways to parameterize color.Ask Prof. Thomas Serre for a qualified answer.
Or…“When some primates started growing a third cone in their retinas, the old bipolar system remained, with the third cone adding a 2nd dimension of color encoding: red versus green. since color is now encoded in a 2d space, you find that you can draw a circle of colors in that space, which when you think about the fact that wavelength is 1d is really weird.”
- aggasalk, Reddit.
Thanks to Alexander Nibley
![Page 65: Bela Borsodics.brown.edu/courses/cs143/lectures/2020Spring_08_Camera... · 2020. 2. 12. · • Most birds, and many other animals, have cones for ultraviolet light. • Some humans](https://reader036.fdocuments.net/reader036/viewer/2022062607/6053608a2540ce071d48b495/html5/thumbnails/65.jpg)
Held and Hein (1963)
![Page 66: Bela Borsodics.brown.edu/courses/cs143/lectures/2020Spring_08_Camera... · 2020. 2. 12. · • Most birds, and many other animals, have cones for ultraviolet light. • Some humans](https://reader036.fdocuments.net/reader036/viewer/2022062607/6053608a2540ce071d48b495/html5/thumbnails/66.jpg)
A Bird’s-Eye View of Nature’s Hidden Order
Natalie Wolchover, Quanta Magazine 2016