A Non-local Cost Aggregation Method for Stereo Matching

Qingxiong YangCity University of Hong Kong

2012 IEEE Conference on Computer Vision and Pattern Recognition 1

Outilne• Introduction• Related Works• Method• Experimental Results• Conclusion

2

Introduction_________________________

3

Introduction• Goal : Get fast and accurate disparity map.• Solution : Non-local cost aggregation + MST• Advantage : Better in low textures region Low complexity

4

Related Works_________________________

5

Related Works

6

1 •Matching cost computation

2 •Cost (support) aggregation

3 •Disparity computation and optimization

4 •Disparity refinement

[21] D. Scharstein and R. Szeliski. A taxonomy and evaluation of dense two-frame stereo correspondence algorithms.International Journal of Computer Vision (IJCV), 47:7–42, 2002.

Related WorksLocal methods• 1=>2=>3• A local support region

with winner take all• Implicit smoothness• Fast but inaccurate.

Global methods• 1(=>2)=>3• Energy minimization

process (GC,BP,DP,Cooperative)• Per-processing• Explicit smoothness• Accurate but slow

7

Comparison (Rank in Middleburry)Real time Non-Real Method

1 - - -

2 O (no Seg.) Cross-based Aggregation>>Scanline Optimization

3 O Mean-shift >> BP>>Self-adapting4 O Mean-shift >> Cooperative Optimization5 - - -

6 O Mean-shift>>Color-weighted>>BP7 O (no Seg.) Seed Detection>>Scanline Propagation8 O Mean-shift>>BP9 O Mean-shift(Region-based)>>B-spline10 O Up-sample>>Bilateral Filter>>BP11 - - -

12 O (No Seg.)>>Convex Relaxation>>regularization13 O Mean-shift>>Image Warping>>BP

Reference(1/2)

C. Shi, G. Wang, X. Pei, H. Bei, and X. Lin. High-accuracy stereo matching based on adaptive ground control points. Submitted to IEEE TIP 2012

X. Mei, X. Sun, M. Zhou, S. Jiao, H. Wang, and X. Zhang. On building an accurate stereo matching system on graphics hardware. GPUCV 2011.

A. Klaus, M. Sormann and K. Karner. Segment-based stereo matching using belief propagation and a self-adapting dissimilarity measure. ICPR 2006.

Z. Wang and Z. Zheng. A region based stereo matching algorithm using cooperative optimization. CVPR 2008.

Anonymous. A dense stereo matching with reliability aggregation and propagation. CVPR 2012 submission 1170.

Q. Yang, L. Wang, R. Yang, H. Stewénius, and D. Nistér. Stereo matching with color-weighted correlation, hierarchical belief propagation and occlusion handling. IEEE TPAMI 2009

9

Reference(2/2)

X. Sun, X. Mei, S. Jiao, M. Zhou, and H. Wang. Stereo matching with reliable disparity propagation. 3DIMPVT 2011.

L. Xu and J. Jia. Stereo matching: an outlier confidence approach. ECCV 2008.

M. Bleyer, C. Rother, and P. Kohli. Surface stereo with soft segmentation. CVPR 2010.

Q. Yang, R. Yang, J. Davis, and D. Nistér. Spatial-depth super resolution for range images. CVPR 2007.

Y. Mizukami, K. Okada, A. Nomura, S. Nakanishi, and K. Tadamura. Sub-pixel disparity search for binocular stereo vision. ICPR 2012 submission 1439.

S. Zhu, L. Zhang, and H. Jin. A locally linear regression model for boundary preserving regularization in stereo matching. ECCV 2012.

M. Bleyer, M. Gelautz, C. Rother, and C. Rhemann. A stereo approach that handles the matting problem via image warping. CVPR 2009.

10

Method_________________________

11

Method

12

1 •Matching cost computation － Bilateral Filter

2 •Cost aggregation － MST

3 •Disparity computation and optimization

4 •Disparity refinement　－ Median Filter

Bilateral Filter• Every sample is replaced by a weighted average of its

neighbors.• These weights reflect two forces• How close are the neighbor and the center sample• How similar are the neighbor and the center sample

• Edge-preserving and noise reducing smoothing filter

13

Bilateral Filter

14

qp

Bilateral Filter

15

Center Sample : p

Neighborhood : q

Bilateral Filter

16

Total Distance

Bilateral Filter

17

Gaussian wieght Bilateral wieghtOriginal image

• Kruskal's Algorithm• Scan all edges increasing weight order, if an edge is

safe, add it to F. 4

1

2 3

2 1

3

5

3

4

2

5 6

4

4

10

A

B C

D

E F

G

HI

J

Minimum Spanning Tree

18

PPT By Jonathan Davis

Orginal Graph

4

1

2 3

2 1

3

5

3

4

2

5 6

4

4

10

A

B C

D

E F

G

HI

J

1

4

2

5

2

5

4

3

4

4

10

1

6

3

3

2

A AB D

B B

B

C D

J C

C

E

F

D

D H

J E G

F FG I

G GI J

H J JI19

4

1

2 3

2 1

3

5

3

4

2

5 6

4

4

10

A

B C

D

E F

G

HI

J

2

5

2

5

4

3

4

10

1

6

3

3

2

B

B

D

J

C

C

E

F

D

D H

J

E G

F

F

G

I

G

G

I

J

H J

JI

1A D

4B C

4A B

Sort Edges (in reality they are placed in a priority queue - not sorted - but sorting them

makes the algorithm easier to visualize)

20

4

1

2 3

2 1

3

5

3

4

2

5 6

4

4

10

A

B C

D

E F

G

HI

J

2

5

2

5

4

3

4

10

1

6

3

3

2

B

B

D

J

C

C

E

F

D

D H

J

E G

F

F

G

I

G

G

I

J

H J

JI

1A D

4B C

4A B

Add Edge

21

4

1

2 3

2 1

3

5

3

4

2

5 6

4

4

10

A

B C

D

E F

G

HI

J

2

5

2

5

4

3

4

10

1

6

3

3

2

B

B

D

J

C

C

E

F

D

D H

J

E G

F

F

G

I

G

G

I

J

H J

JI

1A D

4B C

4A B

Add Edge

22

4

1

2 3

2 1

3

5

3

4

2

5 6

4

4

10

A

B C

D

E F

G

HI

J

2

5

2

5

4

3

4

10

1

6

3

3

2

B

B

D

J

C

C

E

F

D

D H

J

E G

F

F

G

I

G

G

I

J

H J

JI

1A D

4B C

4A B

Add Edge

23

4

1

2 3

2 1

3

5

3

4

2

5 6

4

4

10

A

B C

D

E F

G

HI

J

2

5

2

5

4

3

4

10

1

6

3

3

2

B

B

D

J

C

C

E

F

D

D H

J

E G

F

F

G

I

G

G

I

J

H J

JI

1A D

4B C

4A B

Add Edge

24

4

1

2 3

2 1

3

5

3

4

2

5 6

4

4

10

A

B C

D

E F

G

HI

J

2

5

2

5

4

3

4

10

1

6

3

3

2

B

B

D

J

C

C

E

F

D

D H

J

E G

F

F

G

I

G

G

I

J

H J

JI

1A D

4B C

4A B

Add Edge

25

4

1

2 3

2 1

3

5

3

4

2

5 6

4

4

10

A

B C

D

E F

G

HI

J

2

5

2

5

4

3

4

10

1

6

3

3

2

B

B

D

J

C

C

E

F

D

D H

J

E G

F

F

G

I

G

G

I

J

H J

JI

1A D

4B C

4A B

Cycle

Don’t Add Edge

26

4

1

2 3

2 1

3

5

3

4

2

5 6

4

4

10

A

B C

D

E F

G

HI

J

2

5

2

5

4

3

4

10

1

6

3

3

2

B

B

D

J

C

C

E

F

D

D H

J

E G

F

F

G

I

G

G

I

J

H J

JI

1A D

4B C

4A B

Add Edge

27

4

1

2 3

2 1

3

5

3

4

2

5 6

4

4

10

A

B C

D

E F

G

HI

J

2

5

2

5

4

3

4

10

1

6

3

3

2

B

B

D

J

C

C

E

F

D

D H

J

E G

F

F

G

I

G

G

I

J

H J

JI

1A D

4B C

4A B

Add Edge

28

4

1

2 3

2 1

3

5

3

4

2

5 6

4

4

10

A

B C

D

E F

G

HI

J

2

5

2

5

4

3

4

10

1

6

3

3

2

B

B

D

J

C

C

E

F

D

D H

J

E G

F

F

G

I

G

G

I

J

H J

JI

1A D

4B C

4A B

Add Edge

29

4

1

2 3

2 1

3

5

3

4

2

5 6

4

4

10

A

B C

D

E F

G

HI

J

2

5

2

5

4

3

4

10

1

6

3

3

2

B

B

D

J

C

C

E

F

D

D H

J

E G

F

F

G

I

G

G

I

J

H J

JI

1A D

4B C

4A B

Cycle

Don’t Add Edge

30

4

1

2 3

2 1

3

5

3

4

2

5 6

4

4

10

A

B C

D

E F

G

HI

J

2

5

2

5

4

3

4

10

1

6

3

3

2

B

B

D

J

C

C

E

F

D

D H

J

E G

F

F

G

I

G

G

I

J

H J

JI

1A D

4B C

4A B

Add Edge

31

4

1

2

2 1

3

32

4

A

B C

D

E F

G

HI

J

4

1

2 3

2 1

3

5

3

4

2

5 6

4

4

10

A

B C

D

E F

G

HI

J

Minimum Spanning Tree Orginal Graph

32

Cost Computation• Cd(p) : matching cost for pixel p at disparity level d• : aggregated cost

-- σS and σR : constants used to adjust the similarity.

33

• Weight between p and q • w(p, q) = | I(p)-I(q)| = image gradient

• Distance between p and q • D(p, q) = sum of weights of the connected edges

• Similarity between p and q

• • Aggregated cost•

＝＞

Cost Aggregation on a Tree Structure

34

Cost Aggregation on a MST• Claim 1. Let Tr denote a subtree of a node s and r denote

the root node of Tr, then the supports node s received from this subtree is the summation of the supports node s received from r and S(s, r) times the supports node r received from its subtrees.• Supports r = • Supports s =

35

s

Tr

r

Cost Aggregation on a MST• Aggregated cost

=>

• , if node v is a leaf node• P(vc) denote parent of nodevc

36

Cost Aggregation on a MST

37

Cost Aggregation on a MST• Aggregated cost

=>

38

Cost Aggregation on a MST• Cost aggregation process• Aggregate the original matching cost Cd from leaf

nodes towards root node using Eqn. (6)• Aggregate from root node towards leaf nodes

using Eqn. (7)

• Complexity• Each level ： 2 addition/subtraction + 3 multiplication

39

Disparity Refinement

•

• D ： the left disparity map• Unstable ： occlusion, lack of texture, specularity• Median filter overlap

40

Experimental Results_________________________

41

Experimental Results• Device ： a MacBook Air laptop computer with a 1.8 GHz

Intel Core i7 CPU and 4 GB memory• Parameter ：• σ = 0.1 (non-local cost aggregation)

• Source : Middlebury http://vision.middlebury.edu/stereo/ HHI database(book arrival) Microsofy i2i database(Ilkay)

42

Experimental Results• Time ：• Proposed average runtime ： 90 milliseconds (1.25× slower)• Unnormalized box filter average runtime ： 72 milliseconds. • Local guided image filter average runtime ： 960 milliseconds

43

[24] P. Viola and M. Jones. Robust real-time face detection.International Journal of Computer Vision, volume 57, pages 137–154, 2003.

[7] C.Rhemann, A. Hosni, M. Bleyer, C. Rother, and M. Gelautz. Fast cost-volume filtering for visual correspondence and beyond. In CVPR ,2011.

44

[7] C.Rhemann, A. Hosni, M. Bleyer, C. Rother, and M. Gelautz. Fast cost-volume filtering for visual correspondence and beyond. In CVPR ,2011.

Experimental Results

45

Experimental Results

46

Experimental Results

47

Experimental Results• Different disparity level (depth of spanning tree)

48

Max=7

49

Max=10

Max=14

50

Max=16

Max=20

51

Max=50

Max=75

Conclusion_________________________

52

Conclusion• Contributions• Outperform all local cost aggregation methods both in

speed and accuracy.• Present a near real-time stereo system with accurate

disparity results.• Future works• Apply to parallel algorithms• Refine matching cost estimation

53