Depth Estimate and Focus Recovery
description
Transcript of Depth Estimate and Focus Recovery
Depth Estimate and Focus Recovery
Presenter : Wen-Chih Hong Adviser: Jian-Jiun Ding
Digital Image and Signal Processing LaboratoryGraduate Institute of Communication EngineeringNational Taiwan University, Taipei, Taiwan, ROC台大電信所 數位影像與訊號處理實驗室
112/04/22 1DISP Lab @ MD531
Outlines Introduction Binocular version systems
Stereo Monocular version systems
DFF DFD
Other method Conclusions References
112/04/22 2DISP Lab @ MD531
Introduction Depth is an important information for
robot and the 3D reconstruction. Image depth recovery is a long-term
subject for other applications such as robot vision and the restorations.
Most of depth recovery methods based on simply camera focus and defocus.
Focus recovery can help users to understand more details for the original defocus images.
112/04/22 3DISP Lab @ MD531
Introduction Categories of depth estimation
Monocular
Depth from defocus (DFD)
Depth from focus (DFF)
Binocular Stereo focus
112/04/22 4DISP Lab @ MD531
Introduction Categories of depth estimation
Active : Sending a controlled energy beam Detection of reflected energy
Passive: Image-based
112/04/22 5DISP Lab @ MD531
Introduction Geometric on imaging
1 1 1
lD F
F
D/2
F
us
2R : R>0
sensor
vBiconvex
112/04/22 6DISP Lab @ MD531
Binocular version systems The flow chart to binocular depth
estimation. Depth map HVS modeling Edge detection Correspondence Vengeance control Gaze control Depth map
112/04/22 7DISP Lab @ MD531
Binocular version systems Vengeance movement :
is some kind of slow eye movement that two eyes move in different directions.
But corresponding problem Gazing point(Corresponding point)
Baseline (B)
B/2B/2
Depth (u)
2 2 22 2
2 2
sin cos cos4sin sin
L R L R
L R L R
u B
112/04/22 8DISP Lab @ MD531
Binocular version systems Complex model
RdiffLdiff
Corresponding point
Depth (u)
Right vision
Left vision Baseline (B)
(xR, yR)(xL, yL)
Figure 3.3 A more complete triangulation geometry for the binocular vision.We have to realize how much departure between the optical axis and the direction
of the
tan tan
tan tanLdiff Rdiff
Ldiff Rdiff
Bu
1tan tan / 2Rdiff Rx W
1tan tan / 2Ldiff Lx W
112/04/22 9DISP Lab @ MD531
Binocular version systems Corresponding problem But more accuracy
112/04/22 10DISP Lab @ MD531
Monocular version systems Depth from focus
Depth form defocus
112/04/22 11DISP Lab @ MD531
Depth from Focus Taking pictures at different observer distance or object
distance We need an estimator to measure degree on focus
Using Laplacian operator
Such operator point to a measurement on a single pixel influence, a sum of Laplacian operator is needed:
( , ) ( , , )* ( , )kg x y h x y t f x y
2 22
2 2( , ) ( , )
( , ) k kk
g x y g x yg x y
x y
, ,j k i k
v j k u i k
n i j ML u v
112/04/22 12DISP Lab @ MD531
Depth from Focus Gaussian interpolation
Figure 4.4 Gaussian interpolation to a measure curve, Nk≧ Nk-1, Nk≧ Nk+1
displacementdk
[SML]
NPFocus measure
Nk
Nk-1
dk-1dp
Measured curveIdeal condition
dk+1
Nk+1
112/04/22 13DISP Lab @ MD531
Depth from Focus Range from focus
using
Take pictures along the axis Find the image having highest frequency Need more than 10 images (monocular)
1 1 1f D v
112/04/22 14DISP Lab @ MD531
Depth from Focus We use Gaussian interpolation to form
a set of approximations The depth solution dp from above
Gaussian:
2 21 1
1 1 1
2 21 1
1 1 1
ln ln
2 ln ln ln ln
ln ln
2 ln ln ln ln
k k k kp
k k k k k k
k k k k
k k k k k k
N N d dd
d d N N N N
N N d d
d d N N N N
112/04/22 15DISP Lab @ MD531
Depth from Defocus Due to geometric optics, the intensity inside the blur
circle should be constant. Considering of aberration and diffraction and so on, we
easily assume a blurring function: : diffusion parameter
Diffusion parameter is related to blur radius: derived from triangularity in geometric optics
For easy computation, we assume that foreground has equal-diffusion, background has equal-diffusion and so on
However, this equal-focal assumption will be a problem
2 2
2 2
1, exp2 2
x yh x y
112/04/22 16DISP Lab @ MD531
Depth from Defocus Blurring model
Blurring radius
112/04/22 17DISP Lab @ MD531
Depth from Defocus Blurring model
1 0
0
br rv v v
2 0
0
br rv v v
0v v v
112/04/22 18DISP Lab @ MD531
Depth from Defocus Blurring model
1 0
0
br rv v v
2 0
0
br rv v v
0v v v
112/04/22 19DISP Lab @ MD531
Depth from Defocus Blurring model
1 0
0
br rv v v
2 0
0
br rv v v
0v v v
112/04/22 20DISP Lab @ MD531
Depth from Defocus Blurring model
when
blur radius is independent of the location of the point source on the object plane at depth
01 2 0 ( 1)b b
vr r r rv
0
01 2 0 ( 1)b b
vr r r rv
112/04/22 21DISP Lab @ MD531
Depth from Defocus Blurring model
Using and
We get So diffusion parameter:
1 1 1
lv F D 0
1 2 0 ( 1)b bv
r r r rv
0 00
1 1 1( )bl
r r vF v D
1 1 1( )m m mlm m
r vF v D
112/04/22 22DISP Lab @ MD531
Depth from Defocus Depth recovery
Eliminating D from m=1,2 we get
where and
1 2
1 1 1( )m m mlm m
r vF v D
1 1
2 2
rvr v
1 11 1 2 2
1 1 1 1( )l l
rvF v F v
112/04/22 23DISP Lab @ MD531
Depth from Defocus Depth recovery
From Take F.T.:
The F.T. of Gaussian is Gaussian
( , ) ( , ) ( , )k kg x y h x y f x y
( , ) ( , ) ( , )k kG w v H w v F w v
2 2
2 2
1, exp2 2
x yh x y
2 2 2 211 2
2
( , ) 1exp[ ( )( )]( , ) 2
G w v w vG w v
112/04/22 24DISP Lab @ MD531
Depth from Defocus Depth recovery
Take the log
Using the relationship between them
we get
1 2 2 2 2
2 2( 1) 2 C
2 2 11 2 2 2
2
( , )2 log( , )
G v Cv G v
2 2 2 211 2
2
( , ) 1exp[ ( )( )]( , ) 2
G w v w vG w v
112/04/22 25DISP Lab @ MD531
Depth from Defocus Depth recovery
let apha=1 we obtain the value of sigma-2
Find out the depth D
2 2 22 2( 1) 2 C
1 1
2 2
1rvr v
1 1 1( )m m mlm m
r vF v D
112/04/22 26DISP Lab @ MD531
Depth from Defocus The main sources of range errors in
DFD
Inaccurate modeling of the optical system. Windowing for local feature analysis. Low spectral content in the scene being
images. Improper calibration of camera parameters. Presence of sensor noise.
112/04/22 27DISP Lab @ MD531
Depth from Defocus Block shift-variant blur model
Consider the interaction of sub-images Define the neighborhood function
Indeed, the image we observed is
compared with
{ , 1,..., ,..., 1, }in i J i J i i J i J
( ) ( )ni
i
i h in
g m f m a d
( , ) ( , ) ( , )k kg x y h x y f x y
112/04/22 28DISP Lab @ MD531
Depth from Defocus Space-variant filtering models for
recovering depth Using complex spectrogram and P.W.D. Complex Spectrogram:
2 1( ) '( , ') ( ') 'x t h t t x t dt
*''( , ') ( ' ) '( , ) exp( ( '))2kh t t u t t H t j t t d
112/04/22 29DISP Lab @ MD531
Depth from Defocus Space-variant filtering models for
recovering depth C.S.:
g_1/g_2
where
( , ) ( , ) ( , )ig f iC t C t H t
2 1( , ) ( , ) ( , )g gC t C t H t
2
1
( , )( , )( , )
H tH tH t
112/04/22 30DISP Lab @ MD531
Depth from Defocus Space-variant filtering models for
recovering depth objective function:
Drawback: No consider the intersection of pixels there will
be interrupt in border. Regularized solution.
2 1
' 1 2 2 2 2
( ) 0
min ( ( ) ( ) exp( ( ) ( )))N
g i g is i k
C k C k k s i
2 22 1( ) ( ) ( )s i i i
112/04/22 31DISP Lab @ MD531
Depth from Defocus No corresponding problem Less accuracy
S.V. > B.S.V. Blocking Trade-off
Blocking size Too large: less accuracy Too small: noise
112/04/22 32DISP Lab @ MD531
Other method Structure from motion Shape from shading
ML Estimation of Depth and Optimal camera settings Recursive computation of depth from multiple images
112/04/22 33DISP Lab @ MD531
Other method Structure from motion
Using the relative motion between object and camera to find out surface information
Corresponding problem (binocular) Find out what motion of camera
112/04/22 34DISP Lab @ MD531
Other method Shape from shading
Need to know the reflectance Find the sliding rate and blindness
112/04/22 35DISP Lab @ MD531
Focus recovery
SML measurement
Defocused image pair
Full focused image
Maximum value searching
Depth measurement of a point
Small aperture construction
Linear canonical transform based on constructed
optical system
focal point
Using the specific depth to retrieve imaging
distance
112/04/22 36DISP Lab @ MD531
Conclusions Binocular stereo method
high accuracy Absolute depth information Complexity computation Corresponding problem
Structure form motion Nonlinear problem Corresponding problem
Shape from shading Very difficult method Active method
112/04/22 37DISP Lab @ MD531
Conclusions Range from focus :
Slowly More than 10 images
depth from defocus : Easy method Less accuracy
112/04/22 38DISP Lab @ MD531
References and future work1) Y. C. Lin, Depth Estimation and Focus Recovery, Master
thesis, National Taiwan Univ., Taipei, Taiwan, R.O.C, 20082) Subhasis Chaudhuri, A.N. Rajagopalan, ”Depth From Defocus:
A Real Aperture Imaging Approach. ” Springer-Verlag. New York, 1999.
3) M. Subbarao, “Parallel depth recovery by changing camera parameters,” Second International Conference on Computer Vision 1988, pp. 149-155, Dec. 1988.
4) M. Subbarao and T. C. Wei, “Depth from defocus and rapid autofocusing: a practical approach,” IEEE Conferences on Computer Vision and Pattern Recognition, pp. 773-776, Jun. 1992.
5) A. N. Rajagopalan and S. Chaudhuri, “A variational approach to recovering depth from defocused images,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 19, pp. 1158-1164, Oct. 1997.
112/04/22 39DISP Lab @ MD531
The end
112/04/22 DISP Lab @ MD531 40