Map Edit Distance vs. Graph Edit Distance for Matching...

Map Edit Distance vs. Graph Edit Distancefor Matching Images

Camille Combier, Guillaume Damiand and Christine Solnon

LIRIS, INSA de Lyon / CNRS

GbR 2013

1/22

Motivations

Overview of the talk:

1 Motivations

2 Recalls on Generalized Maps

3 Extension of the Map Edit Distance to Labelled Maps

4 Experimental Comparison

5 Conclusion

2/22

Motivations

Motivations

Why matching images ?

Recognize objects in images

Reconstitute 3D models from 2D images

. . .

How to match regions (or interest points) of two different images ?

Local approaches: Match each region/point with the most similar one Relationships between regions are not exploited

Global approaches (e.g. RANSAC): Consider rigid transformations Not tolerant to deformations

Graph-based approaches Able to exploit relationships while being tolerant to deformations

3/22

Motivations

From Graph Edit Distances to Graph Matchings

Definition of the Graph Edit Distance:

d(G1,G2) = minimum cost sequence of edit op. to transform G1 into G2

Edit operations = deletion/insertion/substitution of vertices and edges

Example:

Del a, fDel (a, b), (a, f ), (f , g), (h, e)Subs b → 1, c → 2, d → 3, . . .Ins 9Ins (4, 2), (8, 9), (7, 9)

9

a b c

d e

f g h i

j

1 2

3 4

5 6 7

8

Matching derived from an edit sequence:

v1 is matched with v2 iff v1 is substituted with v2 Ex.: Matching = {(b,1), (c,2), (d ,3), (e,4), (g,5), (h,6), (i ,7), (j ,8)}

4/22

Motivations

From Graph Edit Distances to Graph Matchings

Definition of the Graph Edit Distance:

d(G1,G2) = minimum cost sequence of edit op. to transform G1 into G2

Edit operations = deletion/insertion/substitution of vertices and edges

Example:

Del a, fDel (a, b), (a, f ), (f , g), (h, e)Subs b → 1, c → 2, d → 3, . . .Ins 9Ins (4, 2), (8, 9), (7, 9)

j

2

3 4

5 6 7

8 9

a b c

d e

f g h i

1

Matching derived from an edit sequence:

v1 is matched with v2 iff v1 is substituted with v2 Ex.: Matching = {(b,1), (c,2), (d ,3), (e,4), (g,5), (h,6), (i ,7), (j ,8)}

4/22

Motivations

Graphs vs Generalized Maps

Graphs model binary relations Proximity of interest points, Adjacency of regions, . . .But not the topology of the subdivision of the plane into facesGeneralized maps model the topology of the subdivision of a nD spacein cells: 0D vertices, 1D edges, 2D faces, 3D volumes, . . .

Example:

This two graphs are isomorphic. . . but not the corresponding maps

Question addressed in this work:Can we better match image regions by using maps instead of graphs?

5/22

Recalls on Generalized Maps


1 Motivations




5 Conclusion

6/22


From plane graphs to generalized mapsDefinition: nD generalized map M = (D, α0, . . . , αn)

D = finite set of darts∀i , αi is an involution on D∀i ,∀j ≥ i + 2, αi ◦ αj is an involution on D

Example:Graph:

2G-map:

Involutions:

a b c d e f g h i j k l m nα0 h c b e d g f a j i l k n m

α1 b a d c f e h g n k j m l iα2 a b c i j f g h d e k l m n

Notation:d is i-free if αi(d) = d (ex: a is 2-free)d is i-sewn with d ′ if αi(d) = d ′ 6= d (ex: a is 0-sewn with h) 7/22




Example:Graph:

2G-map:

Involutions:

a b c d e f g h i j k l m nα0 h c b e d g f a j i l k n mα1 b a d c f e h g n k j m l i

α2 a b c i j f g h d e k l m n

Notation:d is i-free if αi(d) = d (ex: a is 2-free)d is i-sewn with d ′ if αi(d) = d ′ 6= d (ex: a is 0-sewn with h) 7/22




Example: Graph: 2G-map:

Involutions:

a b c d e f g h i j k l m nα0 h c b e d g f a j i l k n mα1 b a d c f e h g n k j m l iα2 a b c i j f g h d e k l m n

Notation:d is i-free if αi(d) = d (ex: a is 2-free)d is i-sewn with d ′ if αi(d) = d ′ 6= d (ex: a is 0-sewn with h)

7/22


Orbits and cells

Cells (vertices, edges, faces, . . . ) = sets of darts corresponding to orbits

i-cell containing d : celli(d) =< {α0, . . . , αn} \ {αi} > (d)

Example:

cell0(e) =< {α1, α2} > (e) = {e, f , j , k} Vertex containing e

cell1(e) =< {α0, α2} > (e) = {d ,e, i , j} Edge containing e

cell2(e) =< {α0, α1} > (e) = {a,b, c,d ,e, f ,g,h} Face containing e

8/22


Map Edit Distance [PRL 2012]

Edit operations:Add or delete free dartsi-sew i-free darts or i-unsew darts which are i-sewn

Example:e

δ2

f

ed

δ1

a c

h

g

db

f

h

g

b

a

b

c

d

e f

g

h

δ1 = 1-unsew(e,a), 1-unsew(c,f), 1-unsew(g,h),2-unsew(a,b), 2-unsew(c,d), 0-unsew(a,c)

δ2 = del(a), del(c), 1-sew(g,e), 1-sew(h,f)

Definition of the Map Edit distance:

d(M1,M2) = minimum cost sequence of edit op. to transform M1 into M29/22

Extension of the Map Edit Distance to Labelled Maps


1 Motivations




5 Conclusion

10/22


Labelled maps

Modelling the geometry of cells:Maps model the topology but not the geometry of cells Add labels to model the geometryCells are implicitely defined by sets of darts corresponding to orbits All darts of an i-cell must have the same i-label

A labelled map is defined by M = (D, α0, . . . , αn,L, l) such that:(D, α0, . . . , αn) is an nG-map and L a set of labelsl : D×{0, ..,n} → L is s. t. ∀d ,d ′, i , celli(d)=celli(d ′)⇒ l(d , i)=l(d ′, i)

Example:

ml

k

j

i

n

d

fgh

ab c

e

l(a,2) = l(b,2) = . . . = l(h,2) = blue

l(i ,2) = l(j ,2) = . . . = l(n,2) = green

l(c,0) = l(d ,0) = l(i ,0) = l(j ,0) = X

11/22


Labelled maps

Modelling the geometry of cells:Maps model the topology but not the geometry of cells Add labels to model the geometryCells are implicitely defined by sets of darts corresponding to orbits All darts of an i-cell must have the same i-label

A labelled map is defined by M = (D, α0, . . . , αn,L, l) such that:(D, α0, . . . , αn) is an nG-map and L a set of labelsl : D×{0, ..,n} → L is s. t. ∀d ,d ′, i , celli(d)=celli(d ′)⇒ l(d , i)=l(d ′, i)

Example:

m

n

j

i k

l

c

e

d

fgh

ab

X

l(a,2) = l(b,2) = . . . = l(h,2) = blue

l(i ,2) = l(j ,2) = . . . = l(n,2) = green

l(c,0) = l(d ,0) = l(i ,0) = l(j ,0) = X

11/22


Extension of the map edit distance to labelled maps

New edit operation:Substitute dart labels

Example:i

i

j

j

i

j

c

d

e f

gdb

f

c

h

e

δ2δ1

g

h

a

h

g

b de

f

a

b

δ1 = 1-unsew(e,a), 1-unsew(c,f), 1-unsew(g,h),2-unsew(a,b), 2-unsew(c,d), 0-unsew(a,c)

δ2 = del(a), del(c),subs(2,e,red), subs(2,f,red), subs(2,i,red), subs(2,j,red),1-sew(g,e), 1-sew(h,f)

12/22

Experimental Comparison


1 Motivations




5 Conclusion

13/22


Goal of the experimentation

Match the regions of 2 different segmentations of the same image

Ground truth: Regions are well matched if their intersection is not empty

Benchmark:

6 different images from the ETHZ benchmark

For each image, generate 6= segmentations s.t. nb regions ∈ [240;460]

For each segmentation:

Build a Region Adjacency Graph (RAG)Build a 2G-map which associates a face with each region

Sizes of graphs vs sizes of maps:

RAGs 2G-maps# vertices # edges # darts # vertices # edges # facesMin Max Min Max Min Max Min Max Min Max Min Max240 460 688 1217 2792 4948 463 816 698 1237 240 460

14/22


Labels

Colour and size labels:For each region r :

color(r) = average color of the pixels of rsize(r) = number of pixels of r

Graphs: Labels are associated with the vertex corresponding to rMaps: Labels are associated with all darts of the face corresponding to r

Structural labels added to graphs:

Each edge (u, v) is labelled with the number ofadj. relations between regions associated with uand v

Each vertex u is labelled with the total nb of adj.relations of all edges incident to u

This greatly improves matching quality for graphs

Example:

Segmentation:

RAG:23

2

31

1

15/22


Matching algorithms

Computation of Graph/Map Edit Distances is NP-hard

Use polynomial-time algorithms to compute approximations

Approximation of the Graph Edit Distance:

Assignment pb solved by the Munkres algorithm [Riesen & Bunke 2009]

Time complexity = O(v3), where v = number of vertices∼ 2s for coarsest segmentations (v = 240)∼ 5s for finest segmentations (v = 460)

Approximation of the Map Edit Distance

Extension of the greedy algorithm of [GbR 2011] to labelled maps

Time complexity = O(d2 · log(d))), where d = number of darts∼ 8s for coarsest segmentations (d = 2792)∼ 25s for finest segmentations (d = 4948)

16/22


Results when color and size labels are ignored Only structural information is used for matching regions

2G−maps

Graphs

Aver

age

per

centa

ge

of

corr

ectl

y m

atch

ed d

arts

/ver

tice

s

Difference of segmentation levels between the 2 images 17/22


Results when color labels are ignored Structural information and size labels are used for matching regions

2G−maps

Graphs

Aver

age

per

centa

ge

of

corr

ectl

y m

atch

ed d

arts

/ver

tice

s



Results when size labels are ignored Structural information and colour labels are used for matching regions

2G−maps

Graphs

Av

erag

e p

erce

nta

ge

of

corr

ectl

y m

atch

ed d

arts

/ver

tice

s



Results when all labels are used

2G−maps

Graphs

Aver

age

per

centa

ge

of

corr

ectl

y m

atch

ed d

arts

/ver

tice

s

Difference of segmentation levels between the 2 images

20/22

Conclusion


1 Motivations




5 Conclusion

21/22

Conclusion

Conclusion

Summary of our contribution:

Extension of generalized maps to labelled maps Add information on cells in every dimension

Extension of the Map Edit Distance to labelled maps

Comparison with the Graph Edit Distance for matching image regions

MED significantly better than GED when colour is ignoredMED and GED not significantly different when using colour. . . but local approaches obtain similar results in this case !

Further work:

Consider other benchmarks

Consider other Graph and Map matching algorithms

Consider other similarity measures Definition of Map Kernels

22/22

Map Edit Distance vs. Graph Edit Distance for Matching...

Documents

Transcript of Map Edit Distance vs. Graph Edit Distance for Matching...