INTEGRATION OF INTENSITY EDGE INFORMATION INTO

THE REACTION-DIFFUSION STEREO ALGORITHM

Atsushi Nomura1) Makoto Ichikawa2)

Koichi Okada1) Hidetoshi Miike1)

1)Yamaguchi University, Japan2)Chiba University, Japan

In this talk,

• We integrate intensity edge information into a reaction-diffusion stereo algorithm.

• Depth discontinuity => error of disparity detection.• Anisotropic diffusion.

Outline of this talk

• Stereo vision, stereo algorithm & related topics

• Reaction-diffusion equations & reaction-diffusion algorithm

• Proposed stereo algorithm integrating intensity edge information

• Experimental results• Conclusion & future work

　　ObjectObject

Binocular Stereo Vision

LI RI

),,( dyxC

Object

IR(x,y)IL(x,y)

Optical axis

Focul length

Left eye Right eye

(xL,y) (xR,y)

Depth

ObjectObject

Disparityd=xL-xR

Cross-correlation

1C

d0

dN-1 . ..

for possible disparity levels

. ..

stereo correspondence problem=> segmentation problem

Previous Stereo Algorithms

• Marr & Poggio, Proc. Roy. Soc. Lond., 1979– original cooperative algorithm– continuity & uniqueness constraints

• Zitnick & Kanade, IEEE-PAMI, 2000– modern cooperative algorithm + occlusion detection

• Sun et al., IEEE-PAMI, 2003– belief-propagation algorithm

• Deng et al., IEEE-PAMI, 2007– graph-cuts algorithm

Previous Edge Detection Algorithm

• Marr & Hildreth, Proc. Roy. Soc. Lond., 1980 – LoG filter & DOG filter

• Canny, IEEE-PAMI, 1986– Standard algorithm

• Heath et al., IEEE-PAMI, 1997– Review of edge detection algorithms– Standard images

Diffusion Equation & PDE Approach inImage Processing & Computer Vision Research

• Koenderink, Biol. Cybern., 1984– Diffusion equation = Gaussian filter

• Perona & Malik, IEEE-PAMI, 1990– Anisotropic diffusion

• Black et al., IEEE-IP, 1998– Anisotropic diffusion

• Mrázek & Navara, IJCV, 2003– Stopping time for non-linear diffusion filtering

• Galié et al., J. Math. Imaging Vis., 2008– Image compression using anisotropic diffusion

Integration of Visual Cuesin Human Vision System

• Landy et al., Vis. Res., 1995– Fusion of multiple visual cues in depth perception– Linear fusion vs. weak fusion

• Ichikawa et al., Vis. Res., 2003– Non-linear integration model of disparity & other

visual cues.

Reaction-Diffusion Algorithm

• Adamatzky et al., Reaction-Diffusion Computers, 2005– proposing novel computer architecture, by utilizing

reaction-diffusion equations.

• Suzuki et al., Trans. IEICE, 2005– image restoration + LSI implementation

• Asai et al., Int. J. Bifurcation & Chaos, 2005– Voronoi diagram + LSI implementation– Solving partial differential equations needs much computation power.

Hardware (LSI) implementation has been introduced.

Reaction-Diffusion Equations

• A set of partial differential equations having diffusion terms and reaction terms.

),(

),(2

2

vugvDv

vufuDu

vt

ut

bvuvug

vuauuvuf

),(

)1)((ε

1),(

• FitzHugh-Nagumo type reaction terms

u(x,y,t), v(x,y,t): variablesDu, Dv: diffusion coefficientsf(u,v), g(u,v): reaction terms

Constants: 0<<<1 a, b

FitzHugh, Biophysical J., 1961Nagumo et al., Proc. IRE, 1962

OperatorLaplacian :

/2

tt

Ordinary Differential Equation (ODE) System (Du=Dv=0)

• Mono-stable system & bi-stable system

-0.1

0.0

0.1

0.2

-0.5 0.0 0.5 1.0

Excited Rested

a A

B

C

(0,0)

u

v

(us,vs)

f(u,v)=0

g(u,v)=0

Mono-stable

Bi-stable

0.0

0.4

0.8

0.0001 0.01 1 10

u(t), u0=0.26

u(t), u0=0.24

u

t

0.0

0.4

0.8

0.0001 0.01 1 10

u(t), u0=0.26

u(t), u0=0.24

u

t

a: threshold value

Mono-stable

Bi-stable

a=0.25,b=1.0=1/100

a=0.25,b=10.0=1/100

Reaction-Diffusion Equations inOne-Dimensional Case

• FitzHugh-Nagumo equations: bi-stable system

0.0

0.5

1.0

0 50 100 150 200

0.0

0.5

1.0

0 50 100 150 200

0.0

0.5

1.0

0 50 100 150 200

x

u,v

x

x

u,v

u,v

u(x,t=0)

u(x,t=10)

u(x,t=12)

v(x,t=10)

v(x,t=12)

(a)

(b)

(c)

Triggered positions

Parameter setting: Du=1.0, Dv=3.0 a=0.05, b=10.0, =1/100

Propagation speed depends on the diffusion coefficients of Du and Dv.

Discrete System of Reaction-Diffusion Equations in One-Dimensional Space

• FitzHugh-Nagumo equations (discrete system)

Bi-stable system (b=10.0)Mono-stable system (b=1.0)

),(/)2(/

),(/)2(/

11

11

iiiiivi

iiiiiui

vugDvvvDdtdv

vufDuuuDdtdu

Du<<Dv (Du=1.0, Dv=5.0), D=1/4, a=0.25, =1/1000, i: index of spatial position

Initial conditions: ui(t): step function and vi(t=0)=0.0 (uniform)i i

Reaction-Diffusion Stereo Algorithm

• Nomura et al., Mach. Vis. Appl., (in press)

),(),,(

),,(μ),,(),,(2

max2

nnnvnt

nnnnunt

vugvDtyxv

dyxCuvufuDtyxu

),,(maxarg),,(}1,,1,0{

tyxutyxM nNn

nnnn

nnnnnn

bvuvug

vuuauuuvuf

),(

)1))(((ε

1),,( maxmax

: constantN: total number of possible disparity levelsC: similarity measuredn: disparity level

Disparity map:

The diffusion term Du2un drives the propagation of region of the disparity level dn.

Reaction-Diffusion Stereo Algorithm (cont.)

1max

0max tanh12

)( adu

aua

),','(max ')',','(

maxi

tyxuu nnyx

),','(maxarg 'i)',','(

tyxun nnyx

ddd

i: inhibition area

nnnnnn vuuauuuvuf )1))(((ε

1),,( maxmax

This term realizes the uniqueness constraint.Other set having umax increases this threshold level.

The uniqueness constraint is realized by,

Flow Chart of Reaction-Diffusion Stereo Algorithm

C(x,y,d0)

C(x,y,dn)

C(x,y,dN-1)

u0(x,y,t)

un(x,y,t)

uN-1(x,y,t)

...

...

...

...

v0(x,y,t)

vn(x,y,t)

vN-1(x,y,t)

...

...

M(x,y,t)

Reaction-diffusion systemsCorrelationmaps

Stereodisparity

map

Mutual inhibition through f(,,)

self-inhibition due to Du<<Dv

Left image: IL(x,y)

Right image: IR(x,y)

dn (pixel)

Stereo images

Computation of similarity measureat each disparity level dn

edge detection

resultve

edge detection

resultue

integration

Edge Detection Algorithm withReaction-Diffusion Equations

• Nomura et al., J. Phys. Soc. Jpn., 2003– Edge detection & Segmentation– Discrete version of reaction-diffusion equations– Thresholding

• Nomura et al., Patt. Recog. Image Anal., 2008– Edge detection utilizing reaction-diffusion equations– Adaptive threshold level

Integration of Edge Information into Reaction-Diffusion Stereo Algorithm

),(1

),,(μ),,(1

e

maxe

nnnvnt

nnnnunt

vugvvDv

dyxCuvufuuDu

ue(x,y), ve(x,y): spatial distributions of u(x,y,t) and v(x,y,t)at a convergence state

The diffusion coefficients (1-ue) and (1-ve) suppress propagation of region of the disparity level dn.

Experimental Results

• The Middlebury stereo vision page provides– stereo image pairs,– ground-truth data of disparity maps,– definition of areas (occlusion area & depth discontinuity),– scores of other stereo algorithms

• Example of stereo image pairs

CONES 450X375 pixels60 disparity levels

TEDDY 450X375 pixels60 disparity levels

TSUKUBA384X288 pixels

15 disparity levels VENUS 434X383 pixels30 disparity levels

Edge Detection Result withReaction-Diffusion Algorithm

ue

ve

Edges detected bya reaction-diffusion algorithm

Example: CONES

RD(Previous Reaction-Diffusion Algoriothm)

RD+IE(Proposed Reaction-Diffusion Algorithm with Intensity Edge)

Experimental Results on the Middlebury Stereo Data

RD:Nomura et al., Mach. Vis. Appl.(in press)

RD+IE:Proposed in this presentation.

ABP:Klaus et al., Proc. 18th ICPR, 2006.Adaptive back propagation algorithm+ Color segmentation

Error measure:Bad-Match-Percentage (%)threshold level = 1.0 (pixel)

Tested algorithms: Table of scores:

Image Area RD RD+IE ABP

CONES nonocc. 5.53 5.36 2.48

all 12.57 12.98 7.92

disc. 15.18 14.45 7.32

TEDDY nonocc. 14.85 14.68 4.22

all 20.86 20.91 7.06

disc. 30.15 29.06 11.8

TSUKUBA nonocc. 5.98 8.46 1.11

all 7.86 10.19 1.37

disc. 19.70 20.13 5.79

VENUS nonocc. 2.75 2.93 0.10

all 3.92 4.09 0.21

disc. 21.23 20.18 1.44

Disparity Error Maps and Intensity Edges

RD RD+IE

Definitionof areas

Intensityedges

Conclusion & Future Work

• Conclusion:– We proposed integration of intensity edge information into

the reaction-diffusion stereo algorithm.– We confirmed performance of the proposed algorithm.

• Future work:– dynamic integration– other visual cues

Acknowledgments:The present study was supported in part by the Grant-in-Aid for Scientific Research (C) (No. 20500206) from the Japan Society for the Promotion of Science.