Shape from Shading #1
-
Upload
august-curry -
Category
Documents
-
view
44 -
download
1
description
Transcript of Shape from Shading #1
Shape from Shading #1
Topics
irradiance and radiance basic concepts of reflection reflection map photometric stereo
Reflectance map and Photometric stereo
Brightness
irradiance amount of light falling on a surfacefalling energy measured by a unit surface area [watt/m2]
amount of light radiated from a surfaceemitting energy measured from a unit forshorted light source surface area to a unit solid angle [watt/ m2 ・ Sr]
solid angle --- steradian
radiance
2
cos
R
A R Θ
A
Reflection geometry
•irradiance at a pixel depends on•illumination•materials•geometry
•under the same illuminate condition, we observe irradiance difference on the same material surfacethere is a relationship between pixel irradiance and geometry•Reflectance geometry
L=illuminationN=normalV=viewer
VLg
VNe
LNi
cos
cos
cos
L
i=incidence anglee=emitting angleg=phase angle
NVi e
g
Gradient space
reflection functions are defined in the local coordinate system(e,i,g)
For our development, we will redefine the reflectance geometry in the gradient space viewers is always on the Z axis
1
)1,,(
,
22
qp
qpn
Y
Zq
X
Zp
q
p
Surface and body reflection Surface reflection and body reflection
surface reflection=gloss,highlights very directional(specular)body reflection =object color all direction(diffuse) plastic, paint have both metal has only surface reflection
body
air
incident light
surfacereflection
bodyreflection
internalpigment
Model for body reflection
Diffuse---scatters in all directionscommon approximation:
equal in all directions“lambertian”Lambertian’s cosine law“perfectly diffuse reflector”
reflectance=constant * geometric factorf(i,e,g) = Kb * cos i
why cos i ? angle of incidence affects “density” of illumination.(irradiance)
irradiance=light/arealight=1area=1/cos iirradiance = cos i
Calculating a reflection map (Lambertian)
for each(p,q), N=(p,q,1) light source direction, S= )1,,( ss qp
11
1
cos),,,(
2222
ss
ss
ss
qpqp
qqpp
LNiqpqpR
)0,0(),( ss qp
1
1),(
22
qpqpR
iso-brightness contour
p
q
0.5
0.80.9
Reflectance map(continue)
Lambertian
Self-shadow line
p
q
Surface reflection
metals have the only surface reflectiondielectrics(plastics,paint)have the surface reflection as well as the body reflectionsimplest approximation: perfect mirror
reflection is specular direction, S’S’ is coplanar with S,NSN = i = NS’: opposite sides
SN
S’
i i
Phong’s model
calculate angle between S’ and V ---α
f-surface(i,e,g)=Ks * cos α
typical : n = 10 to 500heuristic model
tells amount of light at each angle
n=1
Cos α
3
5
Real surfaces are rough : light scatters
n n
Reflectance map
brightdark
q
q
p
p
Better model of surface reflection
Phong’s model R=Ks cosnα not based on physics
just looks OK for graphics, not really accurate
off-specular effect
Torrance and Sparrow --- geometrical opticsBeckmann and Spizzichino --- physical optics
composite surface reflectionPhong’s model real surface
Phong’s model real surface
Torrance and Sparrow model
Geometrical optics– a collection of planar mirror-like facets– surface reflection caused only by these microfacets– their sizes are much larger than wave length
average normal direction
microfacet
facet slopes to be normally distributed
V-shaped valleys
facet normal
0
)/exp()( 22
cp
α
surface reflectance = constant for material* effect of one ray
* % not blocked by others (geometrical attenuation)
* % of all facets involved
Surface reflectance
Effect of one ray
incoming energy = A cos i
outgoing energy =(A cos i) / (A cos e) =cos i / cos e
i
e
Geometric attenuation
1) masking 2) shadowing
g(i,e)
% of all facets involved
idd cos/4'
eeigK s cos/),()/exp( 22
'd α
'dd
i
reflection distribution
facet normal distributionα
N
)/exp( 22
Beckmann and Spizzichino modelphysical optics
surface is continuous h(x,y)light is wave reflection off of surfaceroughness is amplitude and spatial frequency of variations in
h(x,y)E(x,y,z) “field” of light energysurface is assumed to be a perfect conductor(metal)--- > Maxwell’s equation exact solution is vicious integral
,!
cos'
1
4/22
222
222
22
22
m
mTm
gi xyemm
g
A
DTe
R
AEEE
where
22
y
2
)cos(coscos
cossinsincoscos1
)X)sinc(sinc(
)cos(cos2
yxxy
rii
iriri
x
rih
D
Y
g
Our model (Nayer,Ikeuchi,Kanade89)
Torrance and Sparrow + Beckmann and Spizzichino•diffuse lobe --- cosine function•specular spike --- delta function•specular lobe --- gaussian function
recall
VNe
VSa
em
aK
qpqp
qqppK
qpqpR
qqp
qpp
qp
qpS
V
qpN
qpL
s
ss
ssb
ss
ss
sss
ss
ss
ss
cos
cos
cos/exp11
1
),,,(
1,11
,11
then
)1,0,0(
)1,,(
)1,,(
2
2222
22
22
22
22
Calculating reflectance mapspecular lobe + diffuse lobe
Lambertian (diffuse lobe) contours Specular peak
p
q
Shape-from-shading
recover object shape (orientation) from image irradiance (brightness)
brightness surface orientation
E(x,y)=R(p,q) -- image irradiance equation
gives one constraint on the gradient space at each pixel--- > ill-posed problem (cannot solve !!!!!)
(p,q,1)0.8
p
q0.8
Photomotric stereo•one image irradiant equation gives only one constraint
--- > use multiple equations at each pixel.•take multiple images from the same points under different light source directions
•recall different light source directions give different reflectance map•at each pixel, multiple irradiance values
p
q
p
q
Photometric Stereo
Analytical solution
•real world gives complicated light source direction--- > look-up table method
),,(
),,(
1
1,
1,
1),,(
1
1,
1,
1),,(
3333
2222
21
21
21
21
1
21
21
11111
222222
zyx
zyx
ssss
s
ss
szyx
zyx
SSSS
SSSS
qpqp
q
qp
pSSSS
qpqp
q
qp
pnnnn
nAE
nSE
nSE
nSE
33
22
11
zyx
zyx
zyx
SSS
SSS
SSS
A
333
222
111
EAn
nAAEA
1
11
Look-up table method
Summary
Basic concepts of reflectionradiance and irradiancereflection geometrysurface reflection and body reflection
Shape-from-shading problemreflectance mapimage irradiance equationphotometric stereo
Shape-from-shading #2
Shape-from-shading #2
get a depth map from a needle map
get a needle map from a single image
Depth from surface orientation
1 dimensional caserecall
2 dimensional case
z1
)(1)( pdxdzz
dxdzP
x
0z
1zdyq
dxp
dy
dxqdypdxzz 01
Recovering depth map from a needle map(direct integration method)
Photometric stereo gives a needle map
assume a depth at the originget depth along the x of the needle map
get the depth map
pdxzz '
qdypdxzz '
Direct integration
rapid
accumulates errors
Relaxation method
observed orientation (p,q) should be same as those of the depth map (zx,zy)
reduce the total error within a boundary (the calculus of variations See Horn pp.469-474)
an iterative formula
22 )),(),(()),(),(( yxy
zyxqyx
x
zyxp
dxdyyxy
zyxqyx
x
zyxp
A
22 )),(),(()),(),((
),(),(4
1
)1,()1,(),1(),1(4
1),(1
yxy
qyx
x
p
yxzyxzyxzyxzyxz nnnnn
),(),(4
1
...),1(4
1),(1
yxy
qyx
x
p
yxzyxz nn
iterative method
needle map
brightness
depth map
Relaxation method (Example)
Jacobi’s method
yxnnnnn qpyxHyxHyxHyxHyxH
4
1)1,()1,(),1(),1(
4
1),(1
Iteratively computing the equation itself convergence is very slow
The number of iteration needed to converge is
when the error decreased to 10-p times, for the pixels whose size is J×J
2
2
1pJr
Eg. r=1200000 when J=400, p=15
Gauss-Seidel method
yxnnnnn qpyxHyxHyxHyxHyxH
4
1)1,()1,(),1(),1(
4
1),( 111
2
4
1pJr Eg. r=600000 when J=400, p=15
yxnnnnn qpyxHyxHyxHyxHyxH
4
1)1,()1,(),1(),1(
4
1),(1
Jacobi:
Gauss-Seidel:
scan-line order checkerboard pattern
Iteration number:
SOR (successive overrelaxation) method ),(1
4
1)1,()1,(),1(),1(
4
1),( 111 yxHqpyxHyxHyxHyxHyxH n
yxnnnnn
ω: overrelaxation parameter
Converge if 0<ω<2
0<ω<1 (underrelaxation): slower than Gauss-Seidel1<ω<2 (overrelaxation): faster than Gauss-Seidel
Optimal ω:J
1
2Eg. ω=1.984 when J=400
pJr3
1 Eg. r=2000 when J=400, p=15Iteration number:
Chebyshev acceleration (faster convergence): change ω properly for each iteration
same as Gauss-Seidel
ADI (alternating-direction implicit) method
yxnnnnnn qpyxHyxHyxHyxHyxHyxH )1,(),(2)1,(),1(),(2),1( 212121
yxnnnnnn qpyxHyxHyxHyxHyxHyxH ),1(),(2),1()1,(),(2)1,( 212121111
Jpr 10log3
8 Eg. r=104 when J=400, p=15Iteration number:
: changed properly for each iterationif =0 Gauss-Seidel
-1 +2 -1
……
H(1,y)
H(J,y)
……
*
…
…
*-1 +2 -1
-1 +2 -1
……
H(x,1)
H(x,J)
……
*
…
…
*-1 +2 -1
A: tridiagonal matrix (given)b: vector (given)x: solve Ax=b by using linear system solver (forward substitution) O(N)
Natural boundary condition
ds
dxds
dy
q
p
ds
dxds
dy
H
H
y
x
T
ds
dx
ds
dy
s… arc length of boundary
Natural boundary condition of "gradient-to-height problem" is
·… dot product … the normal of boundary
Algorithm:
[Truth] [Wrong result][Correct result]
Same height boundaryNatural boundary condition
Fast convergenceSlow convergence
Boundary:Height known just use it
Height unknown natural boundary condition
yxyxyx HHp ,1,1, 2
1 1,1,, 2
1 yxyxyx HHq
Calculate the height "H" at the boundary by
or for each iteration
Shape-from-shading with a single view
Photometric stereo uses multiple images.
Is there a way to recover shape from a single image?
Yes, there is a way.1. characteristics strip expansion method:obtain surface orientati
on along characteristics strips of image irradiance equation (Horn 75)
2. relaxation method:obtain surface orientation using image irradiance equation and smoothness constraint (Ikeuchi and Horn 81)
3. global method: assume a surface is a part of sphere (Pentland 83)
Characteristic strip expansion method
the steep descent direction of the reflectance map(gradient space)
the steep descent direction of the image brightness(image brightness)
p
q
x
y
x
y
SDD of RM
SDD of IB
SDD of RM
SDD of IB
p
q
move towards the SDD of the reflectance map on the image plane
move towards the SDD of the image brightness on the gradient space
Characteristic Strip
Proof1. Taylor expansion of p(x,y) and q(x,y)
2. derivative of the image irradiance equation
dyy
qdx
x
qyxqdyydxdq
dyy
pdx
x
pyxpdyydxxp
),(),(
),(),(
dyy
qdx
x
qdq
dyy
pdx
x
pdp
),( yxp
),( ydxxp
x dxx
)),(),,((),( yxqyxpRyxE
x
q
q
R
x
p
p
R
x
E
y
q
q
R
y
p
p
R
y
E
Move towards the SDD of the reflectance map on the image plane
then, what happen to (p(x,y),q(x,y)) ?
move towards the SDD of the image brightness on the gradient
space
Sp
Rx
Sq
Ry
p
R
q
R
S
),(q
R
p
R
on the image
Sx
E
Sx
q
q
R
x
p
p
RS
y
p
q
R
x
p
p
R
Sq
R
y
pS
p
R
x
py
y
px
x
pp
y
E
x
E, ),( qpon
),( qp RR
),( qp EE p
q
0.10.20.30.40.5
xEdp yEdq
x
y
0.1
0.2
0.30.4
from a known point, (you know (p,q) and E)you can determine (p,q) along a characteristic strip
x
y
Sp
Rx
Sq
Ry
character strip
reconstructed contour
q
p
Horn 75
Problem of characteristic stip method
1. Error accumulation.
2. The method starts from a singular point the start point is unreliable.
3. Determine surface orientation only along characteristic stripes.
4. Occluding contours are big evidences. We cannot use that information. (p,q) become infinite.
relaxation method with occluding contours
???
Occluding boundary
Surface orientations on occluding boundaries are known from the shape of silhouette.
These surface orientations cannot be represented by the gradient space. (p,q) becomes infinite.
We will use the stereographic plane, (f,g). On (f,g) plane, occluding boundaries lie on the unit circle.
boundary condition
Stereographic Projection
2222 11
2,
11
2
qp
qg
qp
pf
occluding boundaries lie on the unit circle
gradientspace
p
q
gaussian sphere
Relaxation method
1. Image irradiance equation on (f,g) space on the (f,g) space, we can also define a reflectance map.
2. Smoothness constraint. Neighboring points have roughly the same surface orientation.
)),(),,((),( yxgyxfRyxE
02222
y
g
x
g
y
f
x
f
Relaxation method3. Set up a minimization problem.
4. Using the calculus of variations get iterative formula.
dxdyy
g
x
g
y
f
x
f
yxgyxfRyxEE
2222
2))),(),,((),((
)),(),,((),(
)1,()1,(),1(),1(41),(1
yxgyxfRyxE
yxfyxfyxfyxfyxf
nn
nnnnn
→min
)),(),,((),(
)1,()1,(),1(),1(41),(1
yxgyxfRyxE
yxgyxgyxgyxgyxg
nn
nnnnn
...)...,1(41),(1 yxfyxf nn
...)...,1(41),(1 yxgyxg nn
n+hsolution
n+1+hsolution
brightness image
occluding boundary
needle map depth map
Ikeuchi & Horn 81
Summary
1. Gaussian sphere and reflectance map2. Get a depth map from a needle map
1. direct integration2. relaxation method
3. Get a needle map from a single image1. characteristic strip expansion method2. relaxation method regularization
4. Read Horn pp. 244-269, B&B pp. 93-101
Shape-from-shading #3
Shape-from-shading #3– Shape from shading
More advanced researches– 4-light photometric stereo– Extended light source– Photometric sampling– Shape from interreflection– Inverse polarization raytracingspecular spike interreflection
+ transmission
diffuse interreflection
diffuse
diffuse + specular
specular spike
diffuse + specular spike
4-light photometric stereo
[diffuse pixel]
4 correctanswer
smalldeviation
[specular pixel]
1 correctanswer
largedeviation
4-light photometric stereo [Coleman&Jain]
[diffuse pixel]
Averageof four
[specular pixel]
Use 3dark pixels
Deviation
Smaller than threshold Larger than threshold
4-light photometric stereo [Barsky&Petrou]
specular
diffuse
length
threshold[specular]dark 3
[non-specular]bright 3
[specular pixel] [diffuse pixel] [shadow pixel]
light
specular normal surface normal
difference
4-light photometric stereo [Solomon&Ikeuchi]
4 3
3
3
3
2 2
22
Gaussian sphere
[4 light region]
[Coleman&Jain]
[3 light region]
Use two lights
p
q
Gradient space
Two possible normals
Choose negative shading of shadow light
shadowspecular
[2 light region]Use two lights Two possible normals
diffuse
diffuse
p
q
Gradient spaceChoose negative shading of shadow light
shadowshadow
diffuse
diffuse
Planar extended light source
L
l0
X
Brightness: f
Line lamp
Specular object
Lambertian plane(Extended light source)Camera
r
Irradiance: E(f,r,X,l0,L)
Reflectance map3 linear lamps 3 reflectance maps Solve by photometric stereo
Photometric sampler
Spherical extended light sourceL()=L(;s,R,H,I,C)
C: constantI: radiant intensity of light S
: termination angle
Photometric function for Lambertian surface
)cos( nsAI Lambertian
Lambertian of extended light)cos( nkk AI
Estimate reflectance A'and surface normal n
by fitting cosine function to images
Photometric function for specular surface
)2( nsBI Specular spike
)2( nkk LBI Specular spike of extended light
Point sources are separatedby source termination angle
Only two specular peaks appear
Photometric function for specular surface)2( nkk LBI
Point sources are separatedby source termination angle
)()(
)()()(
1
1
kk
kkk LL
LLD
1-to-1 correspondence
n is calculated
)2( nk
k
L
IB
is calculated
Photometric function for hybrid surface)2()cos( nsns BAI
)2()cos( nknkk LBAI
1. Remove 2 intensities2. Calculate nl by fitting
Lambertian model fromremaining intensities
3. Calculate ns from 2 intensities
Calculate (nl,ns)for all possibilities
Choose where
Finally,
nsnl
BA
BA nsnln
Example
Interreflection
Form factor
dxdx'r
'
Interreflection kernel: 2
coscos),(),(
r
xxVxxK
Visibility function: V=1 if visible, V=0 if invisible
xdxLxxKx
xxdLm )(),()(
),(
Radiance of x Irradiance of x
Albedo Radiance of x'
Interreflection equationPKLLL s
Facet radiance vector: TmLLL ,,, 21 L Source contribution vector: Tsmsss LLL ,,, 21 L
Albedo matrix:
m
00
00
00
1 2
1
P Kernel matrix:
0
0
0
21
221
112
mm
m
m
KK
KK
KK
K
Inverse radiosity
PKLLL s sLPKIL 1
sFL s (Shading) Facet matrix: TmNNNF ,,, 21 i
ii nN
Albedo
Surface normal
Source direction vector: Tzyx sss ,,s
sFPKIL 1
sFL p
FPKIF 1pPseudo facet matrix:
(Pseudo shading) Estimate Fp by conventional photometric stereo(L: input image, s: known light source)
Estimate F iteratively from initial value Fp
pFPKIF
kP kKand is straighforwardly calculated from kF
pkkk FKPIF 1Recovery algorithm:
Example
Application to book scanning
Image scanner
Scanning a book
Observed image
Estimated shape
Restored image
Polarization
Light = wave oscillates Oscillates in certain direction polarization DOP = degree of polarization
Unpolarized(DOP 0)
Light
Perfectly polarized(DOP 1)
Polarizer
Partially polarized(DOP 0~1)
Incident Reflected
AirObject
Transmitted
Reflection and transmissionNormal
Unpolarized
Air
Object
Partially polarized
Partially polarized
Light
Dependsupon
Polarization raytracing
Calculate polarization(Mueller calculus)
Calculate reflection & transmission(Ray-tracing)
Light(4D vector)
Reflection&transmission(4x4 matrix)
Minimization
Solve by Shape-From-Shading technique
dxdyqHpHII yxRE222min
RE II Polarization raytracing equation
Input Calculated by polarization raytracing
Example
Result(10 iteration)
Glass(refractive index 1.5)
Summary
4-light photometric stereo– Photometric stereo from proper 3 lights
Extended light source– Photometric stereo from Lambertian planar light source
Photometric sampling– Shape and reflectance from spherical extended light source
Shape from interreflection– Photometric stereo for concave diffuse object
Inverse polarization raytracing– Shape of transparent object by using SFS technique
Reference Horn, B.K.P. "Robot Vision," MIT Press, 1986 Zhang, R., Tsai, P.S., Cryer, J.D., Shah, M., "Shape from Shading: A
Survey," IEEE Trans. PAMI, Vol. 21, No. 8, pp. 690-706, 1999. Horn, B.K.P “Obtaining shape from shading information,” in The Psy
chology of Computer Vision, P.H. Winston (ed.), McGraw-Hill, 1995 Ikeuchi, K. & B.K.P. Horn, “Numerical shape from shading and occlu
ding boundaries,” Artificial Intelligence, Vol. 17, 1981. Pentland, A.P., “Local shading analysis,” IEEE Trans. PAMI, Vol.6, 1
984. Klinker, G.J., S.A. Shafer & T. Kanade, “The measurement of highlig
ht in color image,” Int. J. Computer Vision, Vol.2, 1988. Nayar S.K., K. Ikeuchi, and T. Kanade "Surface reflection: physical an
d geometrical perspectives", IEEE Trans. PAMI, Vol. 13, pp.661-634, 1991.