Understanding the effect of lighting in images
description
Transcript of Understanding the effect of lighting in images
![Page 1: Understanding the effect of lighting in images](https://reader035.fdocuments.net/reader035/viewer/2022062520/56816039550346895dcf598a/html5/thumbnails/1.jpg)
Understanding the effect of lighting in images
Ronen Basri
![Page 2: Understanding the effect of lighting in images](https://reader035.fdocuments.net/reader035/viewer/2022062520/56816039550346895dcf598a/html5/thumbnails/2.jpg)
Light field
• Rays travel from sources to objects• There they are either absorbed or reflected• Energy decreases with distance and number of
bounces• Camera captures the set of rays that travel
through the focal center
![Page 3: Understanding the effect of lighting in images](https://reader035.fdocuments.net/reader035/viewer/2022062520/56816039550346895dcf598a/html5/thumbnails/3.jpg)
Specular reflectance (mirror)
• When a surface is smooth light reflects in the opposite direction of the surface normal
![Page 4: Understanding the effect of lighting in images](https://reader035.fdocuments.net/reader035/viewer/2022062520/56816039550346895dcf598a/html5/thumbnails/4.jpg)
Specular reflectance
• When a surface is slightly rough the reflected light will fall off around the specular direction
![Page 5: Understanding the effect of lighting in images](https://reader035.fdocuments.net/reader035/viewer/2022062520/56816039550346895dcf598a/html5/thumbnails/5.jpg)
Diffuse reflectance
• When the surface is very rough light may be reflected equally in all directions
![Page 6: Understanding the effect of lighting in images](https://reader035.fdocuments.net/reader035/viewer/2022062520/56816039550346895dcf598a/html5/thumbnails/6.jpg)
Diffuse reflectance
• When the surface is very rough light may be reflected equally in all directions
![Page 7: Understanding the effect of lighting in images](https://reader035.fdocuments.net/reader035/viewer/2022062520/56816039550346895dcf598a/html5/thumbnails/7.jpg)
BRDF
• Bidirectional Reflectance Distribution Function
• Specifies for a unit of incominglight in a direction how much light will be reflectedin a direction
�̂�
![Page 8: Understanding the effect of lighting in images](https://reader035.fdocuments.net/reader035/viewer/2022062520/56816039550346895dcf598a/html5/thumbnails/8.jpg)
Lambertian reflectance
![Page 9: Understanding the effect of lighting in images](https://reader035.fdocuments.net/reader035/viewer/2022062520/56816039550346895dcf598a/html5/thumbnails/9.jpg)
Lambert’s law
�̂��̂�𝜃
![Page 10: Understanding the effect of lighting in images](https://reader035.fdocuments.net/reader035/viewer/2022062520/56816039550346895dcf598a/html5/thumbnails/10.jpg)
Preliminaries
• A surface is denoted • A point on is • The tangent plane is spanned by
• The surface normal is given by
![Page 11: Understanding the effect of lighting in images](https://reader035.fdocuments.net/reader035/viewer/2022062520/56816039550346895dcf598a/html5/thumbnails/11.jpg)
Photometric stereo
• Given several images of the a lambertian object under varying lighting
• Assuming single directional source
𝑀=𝐿𝑆
![Page 12: Understanding the effect of lighting in images](https://reader035.fdocuments.net/reader035/viewer/2022062520/56816039550346895dcf598a/html5/thumbnails/12.jpg)
Photometric stereo
• We can solve for S if L is known (Woodham)• If L is unknown we can use SVD factorization (Hayakawa)
1 1 111 1
1 2
1 2
1 2
3
1 3
.... ..... ..... ....
p x y z
x x
y y
z z pf f f
f fp x y zf p f
I I l l ln nn nn n
I I l l l
𝑀=𝐿𝑆
![Page 13: Understanding the effect of lighting in images](https://reader035.fdocuments.net/reader035/viewer/2022062520/56816039550346895dcf598a/html5/thumbnails/13.jpg)
Factorization• Use SVD to find a rank 3 approximation
• Define
• Factorization is not unique, since
invertible
To reduce ambiguity we impose integrability –up to generalized bas relief transformation (Belhumeur et al.)
𝑀=𝑈 Σ𝑉 𝑇
�̂�=𝑈 √ Σ , �̂�=√Σ𝑉 𝑇 and �̂�= �̂��̂�
�̂�=( �̂� 𝐴−1 ) ( 𝐴�̂� ) , 𝐴
![Page 14: Understanding the effect of lighting in images](https://reader035.fdocuments.net/reader035/viewer/2022062520/56816039550346895dcf598a/html5/thumbnails/14.jpg)
Integrability
• Recall that
• Given , we set
• And solve for
![Page 15: Understanding the effect of lighting in images](https://reader035.fdocuments.net/reader035/viewer/2022062520/56816039550346895dcf598a/html5/thumbnails/15.jpg)
Shape from shading (SFS)
• What if we only have one image?• Assuming that lighting is known
and uniform albedo
• Every intensity determinesa circle of possible normals
• There is only one unknown ()if uniform albedo is assume
𝐼=𝐸 𝜌 cos𝜃
![Page 16: Understanding the effect of lighting in images](https://reader035.fdocuments.net/reader035/viewer/2022062520/56816039550346895dcf598a/html5/thumbnails/16.jpg)
Shape from shading
• We write
• Therefore
• We obtain
• This is a first order, non-linear PDE (Horn)
![Page 17: Understanding the effect of lighting in images](https://reader035.fdocuments.net/reader035/viewer/2022062520/56816039550346895dcf598a/html5/thumbnails/17.jpg)
Shape from shading
• Suppose , then
• This is called an Eikonal equation
![Page 18: Understanding the effect of lighting in images](https://reader035.fdocuments.net/reader035/viewer/2022062520/56816039550346895dcf598a/html5/thumbnails/18.jpg)
Distance transform
• The distance of each point to the boundary• Posed as an Eikonal equation
![Page 19: Understanding the effect of lighting in images](https://reader035.fdocuments.net/reader035/viewer/2022062520/56816039550346895dcf598a/html5/thumbnails/19.jpg)
Solution
• Right hand side in SFS
determines “speed”• Eikonal equation can be solved by a
continuous analog of “shortest path” algorithm, called “fast marching”
• The case is handled by change of variables (Kimmel & Sethian)
![Page 20: Understanding the effect of lighting in images](https://reader035.fdocuments.net/reader035/viewer/2022062520/56816039550346895dcf598a/html5/thumbnails/20.jpg)
Example
![Page 21: Understanding the effect of lighting in images](https://reader035.fdocuments.net/reader035/viewer/2022062520/56816039550346895dcf598a/html5/thumbnails/21.jpg)
Illumination cone• What is the set of images of an object under
different lighting, with any number of sources? • Due to additivity, this set forms a convex cone in
number of pixels (Belhumeur & Kriegman)
= 0.5* +0.2* +0.3*
![Page 22: Understanding the effect of lighting in images](https://reader035.fdocuments.net/reader035/viewer/2022062520/56816039550346895dcf598a/html5/thumbnails/22.jpg)
Illumination cone
• Cone characterization is generic, holds also with specularities, shadows and inter-reflections
• Unfortunately, representing the cone is complicated (infinite degrees of freedom)
• Cone is “thin” for Lambertian objects;indeed the illumination cone of many objects can be represented with few PCA vectors (Yuille et al.)
![Page 23: Understanding the effect of lighting in images](https://reader035.fdocuments.net/reader035/viewer/2022062520/56816039550346895dcf598a/html5/thumbnails/23.jpg)
Lambertian reflectance is smooth
0 1 2 30
0.5
1
0 1 2 30
0.5
1
1.5
2
lighting
reflectance
(Basri & Jacobs; Ramamoorthi & Hanrahan)
![Page 24: Understanding the effect of lighting in images](https://reader035.fdocuments.net/reader035/viewer/2022062520/56816039550346895dcf598a/html5/thumbnails/24.jpg)
Reflectance obtained with convolution
+++
![Page 25: Understanding the effect of lighting in images](https://reader035.fdocuments.net/reader035/viewer/2022062520/56816039550346895dcf598a/html5/thumbnails/25.jpg)
Reflectance obtained with convolution
+++
![Page 26: Understanding the effect of lighting in images](https://reader035.fdocuments.net/reader035/viewer/2022062520/56816039550346895dcf598a/html5/thumbnails/26.jpg)
Spherical harmonics
1
Z YX
23 1Z XZ YZ22 YX XY
2 2 2 1X Y Z+ + Positive valuesNegative values
![Page 27: Understanding the effect of lighting in images](https://reader035.fdocuments.net/reader035/viewer/2022062520/56816039550346895dcf598a/html5/thumbnails/27.jpg)
Harmonic approximation• Lighting, in terms of harmonics
• Reflectance
• Approximation accuracy, 99%(Basri & Jacobs; Ramamoorthi & Hanrahan)
![Page 28: Understanding the effect of lighting in images](https://reader035.fdocuments.net/reader035/viewer/2022062520/56816039550346895dcf598a/html5/thumbnails/28.jpg)
Harmonic transform of kernel
1.023
0.495
-0.111
0.05
-0.029
0.886
-0.5
0
0.5
1
1.5
0 1 2 3 4 5 6 7 8
𝑘(𝜃)=max ¿
![Page 29: Understanding the effect of lighting in images](https://reader035.fdocuments.net/reader035/viewer/2022062520/56816039550346895dcf598a/html5/thumbnails/29.jpg)
“Harmonic faces”
Positive valuesNegative values
( , , )x y zn n n n
ρ Albedon Surface normal
2(3 1)zn 2 2( )x yn n x yn n x zn n y zn n
zn xn yn
![Page 30: Understanding the effect of lighting in images](https://reader035.fdocuments.net/reader035/viewer/2022062520/56816039550346895dcf598a/html5/thumbnails/30.jpg)
Photometric stereo
LMS
Image n
:
Image 1
Light n
:
Light 1
SVD recovers L and S up to an ambiguity
nz
nz
ny
(3nz2-1)
(nx2-ny2)
nxny
nxnz
nynz
( r r
(Basri, Jacobs &Kemelmacher)
![Page 31: Understanding the effect of lighting in images](https://reader035.fdocuments.net/reader035/viewer/2022062520/56816039550346895dcf598a/html5/thumbnails/31.jpg)
Photometric stereo
![Page 32: Understanding the effect of lighting in images](https://reader035.fdocuments.net/reader035/viewer/2022062520/56816039550346895dcf598a/html5/thumbnails/32.jpg)
Motion + lighting
![Page 33: Understanding the effect of lighting in images](https://reader035.fdocuments.net/reader035/viewer/2022062520/56816039550346895dcf598a/html5/thumbnails/33.jpg)
Motion + lighting
• Given 2 images
• Take ratio to eliminate albedo
• If motion is small we can represent using a Taylor expansion around
(Basri & Frolova)
![Page 34: Understanding the effect of lighting in images](https://reader035.fdocuments.net/reader035/viewer/2022062520/56816039550346895dcf598a/html5/thumbnails/34.jpg)
Small motion
• We obtain a PDE that is quasi linear in
• Where
with
• Can be solved with continuation (characteristics)
![Page 35: Understanding the effect of lighting in images](https://reader035.fdocuments.net/reader035/viewer/2022062520/56816039550346895dcf598a/html5/thumbnails/35.jpg)
Reconstruction
![Page 36: Understanding the effect of lighting in images](https://reader035.fdocuments.net/reader035/viewer/2022062520/56816039550346895dcf598a/html5/thumbnails/36.jpg)
More reconstructions
![Page 37: Understanding the effect of lighting in images](https://reader035.fdocuments.net/reader035/viewer/2022062520/56816039550346895dcf598a/html5/thumbnails/37.jpg)
Conclusion
• Understanding the effect of lighting on images is challenging, but can lead to better interpretation of images
• We surveyed several problems:– Photometric stereo– Shape from shading– Modeling multiple sources with spherical harmonics– Motion and lighting
• We only looked at Lambertian objects…