1974 Fast Convolution Using Fermat Number Transforms With Applications to Digital Filtering
Fast Cost-volume Filtering For Visual Correspondence and Beyond
description
Transcript of Fast Cost-volume Filtering For Visual Correspondence and Beyond
![Page 1: Fast Cost-volume Filtering For Visual Correspondence and Beyond](https://reader036.fdocuments.net/reader036/viewer/2022062501/56816143550346895dd0b990/html5/thumbnails/1.jpg)
1
Fast Cost-volume Filtering For Visual Correspondence and Beyond
Asmaa Hosni, Member, IEEE,Christoph Rhemann,
Michael Bleyer, Member, IEEE,Carsten Rother, Member, IEEE,
and Margrit Gelautz,Senior Member, IEEE
IEEE Transactions on Pattern Analysis and Machine Intelligence,2013
![Page 2: Fast Cost-volume Filtering For Visual Correspondence and Beyond](https://reader036.fdocuments.net/reader036/viewer/2022062501/56816143550346895dd0b990/html5/thumbnails/2.jpg)
2
Outline• Introduction• Related Work• Stereo Matching• Optical Flow• Interactive Image Segmentation
• Method : Cost-Volume Filtering• Applications• Experimental Results• Conclusion
![Page 3: Fast Cost-volume Filtering For Visual Correspondence and Beyond](https://reader036.fdocuments.net/reader036/viewer/2022062501/56816143550346895dd0b990/html5/thumbnails/3.jpg)
3
Introduction
![Page 4: Fast Cost-volume Filtering For Visual Correspondence and Beyond](https://reader036.fdocuments.net/reader036/viewer/2022062501/56816143550346895dd0b990/html5/thumbnails/4.jpg)
4
Introduction• Many computer vision tasks can be formulated as labeling
problems.• Solution : a spatially smooth labeling where label transitions are
aligned with color edges. => very fast edge preserving filter• A generic and simple framework • (i) constructing a cost volume • (ii) fast cost volume filtering • (iii) winner-take-all label selection
![Page 5: Fast Cost-volume Filtering For Visual Correspondence and Beyond](https://reader036.fdocuments.net/reader036/viewer/2022062501/56816143550346895dd0b990/html5/thumbnails/5.jpg)
5
Related Work
![Page 6: Fast Cost-volume Filtering For Visual Correspondence and Beyond](https://reader036.fdocuments.net/reader036/viewer/2022062501/56816143550346895dd0b990/html5/thumbnails/6.jpg)
6
Related Work--Stereo Matching• Global method• Energy minimization process (GC,BP,DP,Cooperative)• Per-processing (mean-sift)• Accurate but slow
• Local method• A local support region with winner take all• Fast but inaccurate.
![Page 7: Fast Cost-volume Filtering For Visual Correspondence and Beyond](https://reader036.fdocuments.net/reader036/viewer/2022062501/56816143550346895dd0b990/html5/thumbnails/7.jpg)
7
Related Work--Optical Flow• Optical flow : the pattern of apparent motion caused by the
relative motion between an observer and the scene.• A vector field subject to Image Brightness Constancy Equation
(IBCE) • Application : motion detection, object segmentation, motion
compensated encoding, and stereo disparity measurement......
![Page 8: Fast Cost-volume Filtering For Visual Correspondence and Beyond](https://reader036.fdocuments.net/reader036/viewer/2022062501/56816143550346895dd0b990/html5/thumbnails/8.jpg)
8
Related Work--Optical Flow• Each flow vector is a label and subpixel accuracy further
increases the label space • Method : continuous optimization strategies (coarse-to-fine)• SSD• Local convolution with oriented Gaussians[13]• Local convolution with bilateral filter[14]• Adaptive support weights[15,16]
• Trade off search space (quality) against speed.[13] D. Tschumperle` and R. Deriche, “Vector-Valued Image Regularization with PDE’s: A Common Framework for Different Applications,” CVPR, 2003.[14] J. Xiao, H. Cheng, H. Sawhney, C. Rao, and M. Isnardi, “Bilateral Filtering-Based Optical Flow Estimation with Occlusion Detection,” ECCV, 2006.[15] M. Werlberger, T. Pock, and H. Bischof, “Motion Estimation with Non-Local Total Variation Regularization,” CVPR, 2010.[16] D. Sun, S. Roth, and M. Black, “Secrets of Optical Flow Estimation and Their Principles,” CVPR, 2010.
![Page 9: Fast Cost-volume Filtering For Visual Correspondence and Beyond](https://reader036.fdocuments.net/reader036/viewer/2022062501/56816143550346895dd0b990/html5/thumbnails/9.jpg)
9
Related Work--Interactive Image Segmentation• Interactive Image Segmentation is a binary labeling problem. • Goal : Separate the image into foreground and background regions
given some hints by the user.• Method :• SNAKE• Geodesic morphological
operator [8,21]
• Alpha matte [5,22]
[5] K. He, J. Sun, and X. Tang, “Guided Image Filtering,” ECCV, 2010.[8] A. Criminisi, T. Sharp, and C. Rother, “Geodesic Image and Video Editing,” ACM Graphics, 2010.[21] A. Criminisi, T. Sharp, and A. Blake, “GeoS: Geodesic Image Segmentation,”ECCV, 2008.[22] C. Rhemann, C. Rother, J. Wang, M. Gelautz, P. Kohli, and P. Rott, “A Perceptually Motivated Online Benchmark for Image Matting,” CVPR, 2009.
![Page 10: Fast Cost-volume Filtering For Visual Correspondence and Beyond](https://reader036.fdocuments.net/reader036/viewer/2022062501/56816143550346895dd0b990/html5/thumbnails/10.jpg)
10
Cost-volume Filtering
![Page 11: Fast Cost-volume Filtering For Visual Correspondence and Beyond](https://reader036.fdocuments.net/reader036/viewer/2022062501/56816143550346895dd0b990/html5/thumbnails/11.jpg)
11
Edge-preserving filtering• Edge-preserving filtering methods• Weighted Least Square [Lagendijk et al. 1988]• Anisotropic diffusion [Perona and Malik 1990]• Bilateral filter [Aurich and Weule 95], [Tomasi and Manduchi 98]• Digital TV (Total Variation) filter [Chan et al. 2001]
![Page 12: Fast Cost-volume Filtering For Visual Correspondence and Beyond](https://reader036.fdocuments.net/reader036/viewer/2022062501/56816143550346895dd0b990/html5/thumbnails/12.jpg)
12
Bilateral filter [6]
[6] K. Yoon and S. Kweon, “Adaptive Support-Weight Approach for Correspondence Search,” IEEE Trans. Pattern Analysis and Machine Intelligence, Apr. 2006.
![Page 13: Fast Cost-volume Filtering For Visual Correspondence and Beyond](https://reader036.fdocuments.net/reader036/viewer/2022062501/56816143550346895dd0b990/html5/thumbnails/13.jpg)
13
Bilateral filter• Advantages • Preserve edges in the smoothing process• Simple and intuitive• Non-iterative
• Disadvantages • Complexity O(Nr2)• Gradient distortion
Preserves edges,but not gradients
![Page 14: Fast Cost-volume Filtering For Visual Correspondence and Beyond](https://reader036.fdocuments.net/reader036/viewer/2022062501/56816143550346895dd0b990/html5/thumbnails/14.jpg)
14
Guided filter [5]
[5] K. He, J. Sun, and X. Tang, “Guided Image Filtering,” Proc. European Conf. Computer Vision, 2010.
![Page 15: Fast Cost-volume Filtering For Visual Correspondence and Beyond](https://reader036.fdocuments.net/reader036/viewer/2022062501/56816143550346895dd0b990/html5/thumbnails/15.jpg)
15
Guided filter
Bilateral filter does not have this linear model
![Page 16: Fast Cost-volume Filtering For Visual Correspondence and Beyond](https://reader036.fdocuments.net/reader036/viewer/2022062501/56816143550346895dd0b990/html5/thumbnails/16.jpg)
16
Guided filter•
![Page 17: Fast Cost-volume Filtering For Visual Correspondence and Beyond](https://reader036.fdocuments.net/reader036/viewer/2022062501/56816143550346895dd0b990/html5/thumbnails/17.jpg)
17
Bilateral filter V.S. Guided filter
![Page 18: Fast Cost-volume Filtering For Visual Correspondence and Beyond](https://reader036.fdocuments.net/reader036/viewer/2022062501/56816143550346895dd0b990/html5/thumbnails/18.jpg)
18
Guided filter• Edge-preserving filtering• Non-iterative• O(1) time, fast and non-approximate• No gradient distortion
Advantages of bilateral filter
Disadvantages of bilateral filter
![Page 19: Fast Cost-volume Filtering For Visual Correspondence and Beyond](https://reader036.fdocuments.net/reader036/viewer/2022062501/56816143550346895dd0b990/html5/thumbnails/19.jpg)
19
Cost-volume Filtering• Label l from
• • C’ : the filtered cost volume • i and j : pixel indices. • Wi,j : The filter weights depend on the guidance image I
•
• : the mean vector and covariance of I• : a smoothness parameter• U : Identity matrix
• Winner take all :k
r
r
w=(2r+1)^2
![Page 20: Fast Cost-volume Filtering For Visual Correspondence and Beyond](https://reader036.fdocuments.net/reader036/viewer/2022062501/56816143550346895dd0b990/html5/thumbnails/20.jpg)
20
Cost-volume Filtering
![Page 21: Fast Cost-volume Filtering For Visual Correspondence and Beyond](https://reader036.fdocuments.net/reader036/viewer/2022062501/56816143550346895dd0b990/html5/thumbnails/21.jpg)
21[5] K. He, J. Sun, and X. Tang, “Guided Image Filtering,” Proc. European Conf. Computer Vision, 2010.[6] K. Yoon and S. Kweon, “Adaptive Support-Weight Approach for Correspondence Search,” IEEE Trans. Pattern Analysis and Machine Intelligence, Apr. 2006.[9] F. Crow. Summed-area tables for texture mapping. SIGGRAPH, 1984.
Cost-volume Filtering
(a) Zoom of the green line in the input image. (b) Slice of the cost volume for the line(white/black/red: high/low/lowest costs)(c) The box filter [9](d) The joint bilateral filter [6](e) The guided filter [5](f) Ground-truth labeling
![Page 22: Fast Cost-volume Filtering For Visual Correspondence and Beyond](https://reader036.fdocuments.net/reader036/viewer/2022062501/56816143550346895dd0b990/html5/thumbnails/22.jpg)
22
Application
![Page 23: Fast Cost-volume Filtering For Visual Correspondence and Beyond](https://reader036.fdocuments.net/reader036/viewer/2022062501/56816143550346895dd0b990/html5/thumbnails/23.jpg)
23
Stereo Matching
![Page 24: Fast Cost-volume Filtering For Visual Correspondence and Beyond](https://reader036.fdocuments.net/reader036/viewer/2022062501/56816143550346895dd0b990/html5/thumbnails/24.jpg)
24
Stereo Matching• Cost computation
• : grayscale gradients in x direction• : balances the color and gradient terms• : truncation values
![Page 25: Fast Cost-volume Filtering For Visual Correspondence and Beyond](https://reader036.fdocuments.net/reader036/viewer/2022062501/56816143550346895dd0b990/html5/thumbnails/25.jpg)
25
Stereo Matching• Cost computation
• : grayscale gradients in x and y direction• : balances the color and gradient terms• : truncation values
• Occlusion detection :
![Page 26: Fast Cost-volume Filtering For Visual Correspondence and Beyond](https://reader036.fdocuments.net/reader036/viewer/2022062501/56816143550346895dd0b990/html5/thumbnails/26.jpg)
26
Stereo Matching• Cost computation
• : grayscale gradients in x and y direction• : balances the color and gradient terms• : truncation values
• Occlusion detection :• Postprocessing
1.Scanline filling : the lowest disparity of the spatially closest nonoccluded pixel
2.Median filter :
• : adjust the spatial and color similarity• : normalization factor
![Page 27: Fast Cost-volume Filtering For Visual Correspondence and Beyond](https://reader036.fdocuments.net/reader036/viewer/2022062501/56816143550346895dd0b990/html5/thumbnails/27.jpg)
27
Stereo Matching• Alternative—symmetric cost aggregation• The cost aggregation can be formulated symmetrically for both input
images. • Replace the 3*1 vector Ii in (3) with a 6*1 vector whose entries are given by
the RGB color channels of both Ii and I’i-l .
![Page 28: Fast Cost-volume Filtering For Visual Correspondence and Beyond](https://reader036.fdocuments.net/reader036/viewer/2022062501/56816143550346895dd0b990/html5/thumbnails/28.jpg)
28
Stereo Matching
• Effect of postprocessing
Disparity map with invalidated pixels in red
After scanline-based filling After median filtering
![Page 29: Fast Cost-volume Filtering For Visual Correspondence and Beyond](https://reader036.fdocuments.net/reader036/viewer/2022062501/56816143550346895dd0b990/html5/thumbnails/29.jpg)
29
Optical flow• Cost computation
• : grayscale gradients in x and y direction• : balances the color and gradient terms• : truncation values
• Postprocessing:Median filter and iteration
![Page 30: Fast Cost-volume Filtering For Visual Correspondence and Beyond](https://reader036.fdocuments.net/reader036/viewer/2022062501/56816143550346895dd0b990/html5/thumbnails/30.jpg)
30
Optical flow• Upscale : To find subpixel accurate flow vectors.• Color of subpixel filled with bicubic interpolation.
![Page 31: Fast Cost-volume Filtering For Visual Correspondence and Beyond](https://reader036.fdocuments.net/reader036/viewer/2022062501/56816143550346895dd0b990/html5/thumbnails/31.jpg)
31
Interactive Image Segmentation• : foreground F or background B• : background color histograms [25]• b(i) : the bin of pixel i• Cost computation :
(binary labeling) =>• Five iterations [26]
[25] S. Vicente, V. Kolmogorov, and C. Rother, “Joint Optimization of Segmentation and Appearance Models,” Proc. IEEE Int’l Conf. Computer Vision, 2009.[26] C. Rother, V. Kolmogorov, and A. Blake, “Grabcut-Interactive Foreground Extraction Using Iterated Graph Cuts,” Proc. ACM Siggraph, 2004.
![Page 32: Fast Cost-volume Filtering For Visual Correspondence and Beyond](https://reader036.fdocuments.net/reader036/viewer/2022062501/56816143550346895dd0b990/html5/thumbnails/32.jpg)
32
Experimental Results
![Page 33: Fast Cost-volume Filtering For Visual Correspondence and Beyond](https://reader036.fdocuments.net/reader036/viewer/2022062501/56816143550346895dd0b990/html5/thumbnails/33.jpg)
33
Experimental Results• Device : an Intel Core 2 Quad, 2.4 GHZ PC with GeForce
GTX480 graphics card with 1.5GB of memory from NVIDIA.• Settings parameters:
: depends on the signal-to-noise ratio of an image • 0.008 for stereo • 0.016 for optical flow
• Source : Middlebury http://vision.middlebury.edu/stereo/
![Page 34: Fast Cost-volume Filtering For Visual Correspondence and Beyond](https://reader036.fdocuments.net/reader036/viewer/2022062501/56816143550346895dd0b990/html5/thumbnails/34.jpg)
34
Stereo Matching
![Page 35: Fast Cost-volume Filtering For Visual Correspondence and Beyond](https://reader036.fdocuments.net/reader036/viewer/2022062501/56816143550346895dd0b990/html5/thumbnails/35.jpg)
35
Stereo Matching
![Page 36: Fast Cost-volume Filtering For Visual Correspondence and Beyond](https://reader036.fdocuments.net/reader036/viewer/2022062501/56816143550346895dd0b990/html5/thumbnails/36.jpg)
36
Stereo Matching
• disparity maps without occlusion filling and postprocessing (invalidated pixels are black)
![Page 37: Fast Cost-volume Filtering For Visual Correspondence and Beyond](https://reader036.fdocuments.net/reader036/viewer/2022062501/56816143550346895dd0b990/html5/thumbnails/37.jpg)
37
Stereo Matching
[27] A. Hosni, M. Bleyer, M. Gelautz, and C. Rhemann, “Local Stereo Matching Using Geodesic Support Weights,” Int’l Conf. Image Processing, 2009.[6] K. Yoon and S. Kweon, “Adaptive Support-Weight Approach for Correspondence Search,” PAMI , 2006.[7] C. Richardt, D. Orr, I. Davies, A. Criminisi, and N. Dodgson, “Real-Time Spatiotemporal Stereo Matching Using the Dual-Cross-Bilateral Grid,”ECCV,2010.
![Page 38: Fast Cost-volume Filtering For Visual Correspondence and Beyond](https://reader036.fdocuments.net/reader036/viewer/2022062501/56816143550346895dd0b990/html5/thumbnails/38.jpg)
38
Stereo Matching• Million Disparity Estimations per second (MDE/s)
• W : width • H : height• D : disparity levels• FPS : number of frames per second
• A larger MDE number means a better performing system.
![Page 39: Fast Cost-volume Filtering For Visual Correspondence and Beyond](https://reader036.fdocuments.net/reader036/viewer/2022062501/56816143550346895dd0b990/html5/thumbnails/39.jpg)
39
Stereo Matching
![Page 40: Fast Cost-volume Filtering For Visual Correspondence and Beyond](https://reader036.fdocuments.net/reader036/viewer/2022062501/56816143550346895dd0b990/html5/thumbnails/40.jpg)
40
Stereo Matching
![Page 41: Fast Cost-volume Filtering For Visual Correspondence and Beyond](https://reader036.fdocuments.net/reader036/viewer/2022062501/56816143550346895dd0b990/html5/thumbnails/41.jpg)
41
Stereo Matching
Table 2. Rankings and run times for selected local stereo methods. Run times in the table are averaged over the four Middlebury test images. *The run time in [15] was reported before left-right consistency check in the corresponding paper. Hence, for fairness, we have multiplied the reported time by a factor of 2.
![Page 42: Fast Cost-volume Filtering For Visual Correspondence and Beyond](https://reader036.fdocuments.net/reader036/viewer/2022062501/56816143550346895dd0b990/html5/thumbnails/42.jpg)
42
Stereo Matching
![Page 43: Fast Cost-volume Filtering For Visual Correspondence and Beyond](https://reader036.fdocuments.net/reader036/viewer/2022062501/56816143550346895dd0b990/html5/thumbnails/43.jpg)
43
Optical flow
![Page 44: Fast Cost-volume Filtering For Visual Correspondence and Beyond](https://reader036.fdocuments.net/reader036/viewer/2022062501/56816143550346895dd0b990/html5/thumbnails/44.jpg)
44
Optical flow
The respective AEE and AAE are given in parentheses (AEE/AAE).
![Page 45: Fast Cost-volume Filtering For Visual Correspondence and Beyond](https://reader036.fdocuments.net/reader036/viewer/2022062501/56816143550346895dd0b990/html5/thumbnails/45.jpg)
45
Optical flow
Comparison of two sequences with thin structure, where many competitors fail to preserve flow discontinuities. (We boosted the colors in the second row from the top for better visualization.)
![Page 46: Fast Cost-volume Filtering For Visual Correspondence and Beyond](https://reader036.fdocuments.net/reader036/viewer/2022062501/56816143550346895dd0b990/html5/thumbnails/46.jpg)
46
Optical flowLarge displacement flow• (b)-(e) and (l)-(o) Motion magnitude for
different methods. • (f), (p) Our flow vectors with the color
coding as in middleburry. • (h)-(k) Backward warping results using
flow of different methods—the tip of the foot is correctly recovered by our method.
• Occluded regions cannot be handled by any method.
![Page 47: Fast Cost-volume Filtering For Visual Correspondence and Beyond](https://reader036.fdocuments.net/reader036/viewer/2022062501/56816143550346895dd0b990/html5/thumbnails/47.jpg)
47
Interactive Image Segmentation
![Page 48: Fast Cost-volume Filtering For Visual Correspondence and Beyond](https://reader036.fdocuments.net/reader036/viewer/2022062501/56816143550346895dd0b990/html5/thumbnails/48.jpg)
48
Interactive Image Segmentation
![Page 49: Fast Cost-volume Filtering For Visual Correspondence and Beyond](https://reader036.fdocuments.net/reader036/viewer/2022062501/56816143550346895dd0b990/html5/thumbnails/49.jpg)
49
Interactive Image Segmentation
Our method GrabCut [26]Time 2.85 ms 300 mserror 5.3% 6.2%
[26] C. Rother, V. Kolmogorov, and A. Blake, “Grabcut-Interactive Foreground Extraction Using Iterated Graph Cuts,” Proc. ACM Siggraph, 2004.
![Page 50: Fast Cost-volume Filtering For Visual Correspondence and Beyond](https://reader036.fdocuments.net/reader036/viewer/2022062501/56816143550346895dd0b990/html5/thumbnails/50.jpg)
50
Conclusion
![Page 51: Fast Cost-volume Filtering For Visual Correspondence and Beyond](https://reader036.fdocuments.net/reader036/viewer/2022062501/56816143550346895dd0b990/html5/thumbnails/51.jpg)
51
Conclusion• Contribution• Propose a edge-preserving property • Runtime independent of the filter size • Achieved both accuracy and efficiency
• Future Work• Leverage this framework to other application areas • Process slanted surfaces [32]
[32] M. Bleyer, C. Rhemann, and C. Rother, “Patchmatch Stereo—Stereo Matching with Slanted Support Windows,” Proc. British Machine Vision Conf., 2011.