Map Edit Distance vs. Graph Edit Distancefor Matching Images
Camille Combier, Guillaume Damiand and Christine Solnon
LIRIS, INSA de Lyon / CNRS
GbR 2013
1/22
Motivations
Overview of the talk:
1 Motivations
2 Recalls on Generalized Maps
3 Extension of the Map Edit Distance to Labelled Maps
4 Experimental Comparison
5 Conclusion
2/22
Motivations
Motivations
Why matching images ?
Recognize objects in images
Reconstitute 3D models from 2D images
. . .
How to match regions (or interest points) of two different images ?
Local approaches: Match each region/point with the most similar one Relationships between regions are not exploited
Global approaches (e.g. RANSAC): Consider rigid transformations Not tolerant to deformations
Graph-based approaches Able to exploit relationships while being tolerant to deformations
3/22
Motivations
From Graph Edit Distances to Graph Matchings
Definition of the Graph Edit Distance:
d(G1,G2) = minimum cost sequence of edit op. to transform G1 into G2
Edit operations = deletion/insertion/substitution of vertices and edges
Example:
Del a, fDel (a, b), (a, f ), (f , g), (h, e)Subs b → 1, c → 2, d → 3, . . .Ins 9Ins (4, 2), (8, 9), (7, 9)
9
a b c
d e
f g h i
j
1 2
3 4
5 6 7
8
Matching derived from an edit sequence:
v1 is matched with v2 iff v1 is substituted with v2 Ex.: Matching = {(b,1), (c,2), (d ,3), (e,4), (g,5), (h,6), (i ,7), (j ,8)}
4/22
Motivations
From Graph Edit Distances to Graph Matchings
Definition of the Graph Edit Distance:
d(G1,G2) = minimum cost sequence of edit op. to transform G1 into G2
Edit operations = deletion/insertion/substitution of vertices and edges
Example:
Del a, fDel (a, b), (a, f ), (f , g), (h, e)Subs b → 1, c → 2, d → 3, . . .Ins 9Ins (4, 2), (8, 9), (7, 9)
j
2
3 4
5 6 7
8 9
a b c
d e
f g h i
1
Matching derived from an edit sequence:
v1 is matched with v2 iff v1 is substituted with v2 Ex.: Matching = {(b,1), (c,2), (d ,3), (e,4), (g,5), (h,6), (i ,7), (j ,8)}
4/22
Motivations
From Graph Edit Distances to Graph Matchings
Definition of the Graph Edit Distance:
d(G1,G2) = minimum cost sequence of edit op. to transform G1 into G2
Edit operations = deletion/insertion/substitution of vertices and edges
Example:
Del a, fDel (a, b), (a, f ), (f , g), (h, e)Subs b → 1, c → 2, d → 3, . . .Ins 9Ins (4, 2), (8, 9), (7, 9)
j
2
3 4
5 6 7
8 9
a b c
d e
f g h i
1
Matching derived from an edit sequence:
v1 is matched with v2 iff v1 is substituted with v2 Ex.: Matching = {(b,1), (c,2), (d ,3), (e,4), (g,5), (h,6), (i ,7), (j ,8)}
4/22
Motivations
Graphs vs Generalized Maps
Graphs model binary relations Proximity of interest points, Adjacency of regions, . . .But not the topology of the subdivision of the plane into facesGeneralized maps model the topology of the subdivision of a nD spacein cells: 0D vertices, 1D edges, 2D faces, 3D volumes, . . .
Example:
This two graphs are isomorphic. . . but not the corresponding maps
Question addressed in this work:Can we better match image regions by using maps instead of graphs?
5/22
Recalls on Generalized Maps
Overview of the talk:
1 Motivations
2 Recalls on Generalized Maps
3 Extension of the Map Edit Distance to Labelled Maps
4 Experimental Comparison
5 Conclusion
6/22
Recalls on Generalized Maps
From plane graphs to generalized mapsDefinition: nD generalized map M = (D, α0, . . . , αn)
D = finite set of darts∀i , αi is an involution on D∀i ,∀j ≥ i + 2, αi ◦ αj is an involution on D
Example:Graph:
2G-map:
Involutions:
a b c d e f g h i j k l m nα0 h c b e d g f a j i l k n m
α1 b a d c f e h g n k j m l iα2 a b c i j f g h d e k l m n
Notation:d is i-free if αi(d) = d (ex: a is 2-free)d is i-sewn with d ′ if αi(d) = d ′ 6= d (ex: a is 0-sewn with h) 7/22
Recalls on Generalized Maps
From plane graphs to generalized mapsDefinition: nD generalized map M = (D, α0, . . . , αn)
D = finite set of darts∀i , αi is an involution on D∀i ,∀j ≥ i + 2, αi ◦ αj is an involution on D
Example:Graph:
2G-map:
Involutions:
a b c d e f g h i j k l m nα0 h c b e d g f a j i l k n mα1 b a d c f e h g n k j m l i
α2 a b c i j f g h d e k l m n
Notation:d is i-free if αi(d) = d (ex: a is 2-free)d is i-sewn with d ′ if αi(d) = d ′ 6= d (ex: a is 0-sewn with h) 7/22
Recalls on Generalized Maps
From plane graphs to generalized mapsDefinition: nD generalized map M = (D, α0, . . . , αn)
D = finite set of darts∀i , αi is an involution on D∀i ,∀j ≥ i + 2, αi ◦ αj is an involution on D
Example: Graph: 2G-map:
Involutions:
a b c d e f g h i j k l m nα0 h c b e d g f a j i l k n mα1 b a d c f e h g n k j m l iα2 a b c i j f g h d e k l m n
Notation:d is i-free if αi(d) = d (ex: a is 2-free)d is i-sewn with d ′ if αi(d) = d ′ 6= d (ex: a is 0-sewn with h)
7/22
Recalls on Generalized Maps
From plane graphs to generalized mapsDefinition: nD generalized map M = (D, α0, . . . , αn)
D = finite set of darts∀i , αi is an involution on D∀i ,∀j ≥ i + 2, αi ◦ αj is an involution on D
Example: Graph: 2G-map:
Involutions:
a b c d e f g h i j k l m nα0 h c b e d g f a j i l k n mα1 b a d c f e h g n k j m l iα2 a b c i j f g h d e k l m n
Notation:d is i-free if αi(d) = d (ex: a is 2-free)d is i-sewn with d ′ if αi(d) = d ′ 6= d (ex: a is 0-sewn with h)
7/22
Recalls on Generalized Maps
Orbits and cells
Cells (vertices, edges, faces, . . . ) = sets of darts corresponding to orbits
i-cell containing d : celli(d) =< {α0, . . . , αn} \ {αi} > (d)
Example:
cell0(e) =< {α1, α2} > (e) = {e, f , j , k} Vertex containing e
cell1(e) =< {α0, α2} > (e) = {d ,e, i , j} Edge containing e
cell2(e) =< {α0, α1} > (e) = {a,b, c,d ,e, f ,g,h} Face containing e
8/22
Recalls on Generalized Maps
Orbits and cells
Cells (vertices, edges, faces, . . . ) = sets of darts corresponding to orbits
i-cell containing d : celli(d) =< {α0, . . . , αn} \ {αi} > (d)
Example:
cell0(e) =< {α1, α2} > (e) = {e, f , j , k} Vertex containing e
cell1(e) =< {α0, α2} > (e) = {d ,e, i , j} Edge containing e
cell2(e) =< {α0, α1} > (e) = {a,b, c,d ,e, f ,g,h} Face containing e
8/22
Recalls on Generalized Maps
Orbits and cells
Cells (vertices, edges, faces, . . . ) = sets of darts corresponding to orbits
i-cell containing d : celli(d) =< {α0, . . . , αn} \ {αi} > (d)
Example:
cell0(e) =< {α1, α2} > (e) = {e, f , j , k} Vertex containing e
cell1(e) =< {α0, α2} > (e) = {d ,e, i , j} Edge containing e
cell2(e) =< {α0, α1} > (e) = {a,b, c,d ,e, f ,g,h} Face containing e
8/22
Recalls on Generalized Maps
Orbits and cells
Cells (vertices, edges, faces, . . . ) = sets of darts corresponding to orbits
i-cell containing d : celli(d) =< {α0, . . . , αn} \ {αi} > (d)
Example:
cell0(e) =< {α1, α2} > (e) = {e, f , j , k} Vertex containing e
cell1(e) =< {α0, α2} > (e) = {d ,e, i , j} Edge containing e
cell2(e) =< {α0, α1} > (e) = {a,b, c,d ,e, f ,g,h} Face containing e
8/22
Recalls on Generalized Maps
Map Edit Distance [PRL 2012]
Edit operations:Add or delete free dartsi-sew i-free darts or i-unsew darts which are i-sewn
Example:e
δ2
f
ed
δ1
a c
h
g
db
f
h
g
b
a
b
c
d
e f
g
h
δ1 = 1-unsew(e,a), 1-unsew(c,f), 1-unsew(g,h),2-unsew(a,b), 2-unsew(c,d), 0-unsew(a,c)
δ2 = del(a), del(c), 1-sew(g,e), 1-sew(h,f)
Definition of the Map Edit distance:
d(M1,M2) = minimum cost sequence of edit op. to transform M1 into M29/22
Extension of the Map Edit Distance to Labelled Maps
Overview of the talk:
1 Motivations
2 Recalls on Generalized Maps
3 Extension of the Map Edit Distance to Labelled Maps
4 Experimental Comparison
5 Conclusion
10/22
Extension of the Map Edit Distance to Labelled Maps
Labelled maps
Modelling the geometry of cells:Maps model the topology but not the geometry of cells Add labels to model the geometryCells are implicitely defined by sets of darts corresponding to orbits All darts of an i-cell must have the same i-label
A labelled map is defined by M = (D, α0, . . . , αn,L, l) such that:(D, α0, . . . , αn) is an nG-map and L a set of labelsl : D×{0, ..,n} → L is s. t. ∀d ,d ′, i , celli(d)=celli(d ′)⇒ l(d , i)=l(d ′, i)
Example:
ml
k
j
i
n
d
fgh
ab c
e
l(a,2) = l(b,2) = . . . = l(h,2) = blue
l(i ,2) = l(j ,2) = . . . = l(n,2) = green
l(c,0) = l(d ,0) = l(i ,0) = l(j ,0) = X
11/22
Extension of the Map Edit Distance to Labelled Maps
Labelled maps
Modelling the geometry of cells:Maps model the topology but not the geometry of cells Add labels to model the geometryCells are implicitely defined by sets of darts corresponding to orbits All darts of an i-cell must have the same i-label
A labelled map is defined by M = (D, α0, . . . , αn,L, l) such that:(D, α0, . . . , αn) is an nG-map and L a set of labelsl : D×{0, ..,n} → L is s. t. ∀d ,d ′, i , celli(d)=celli(d ′)⇒ l(d , i)=l(d ′, i)
Example:
m
n
j
i k
l
c
e
d
fgh
ab
X
l(a,2) = l(b,2) = . . . = l(h,2) = blue
l(i ,2) = l(j ,2) = . . . = l(n,2) = green
l(c,0) = l(d ,0) = l(i ,0) = l(j ,0) = X
11/22
Extension of the Map Edit Distance to Labelled Maps
Extension of the map edit distance to labelled maps
New edit operation:Substitute dart labels
Example:i
i
j
j
i
j
c
d
e f
gdb
f
c
h
e
δ2δ1
g
h
a
h
g
b de
f
a
b
δ1 = 1-unsew(e,a), 1-unsew(c,f), 1-unsew(g,h),2-unsew(a,b), 2-unsew(c,d), 0-unsew(a,c)
δ2 = del(a), del(c),subs(2,e,red), subs(2,f,red), subs(2,i,red), subs(2,j,red),1-sew(g,e), 1-sew(h,f)
12/22
Experimental Comparison
Overview of the talk:
1 Motivations
2 Recalls on Generalized Maps
3 Extension of the Map Edit Distance to Labelled Maps
4 Experimental Comparison
5 Conclusion
13/22
Experimental Comparison
Goal of the experimentation
Match the regions of 2 different segmentations of the same image
Ground truth: Regions are well matched if their intersection is not empty
Benchmark:
6 different images from the ETHZ benchmark
For each image, generate 6= segmentations s.t. nb regions ∈ [240;460]
For each segmentation:
Build a Region Adjacency Graph (RAG)Build a 2G-map which associates a face with each region
Sizes of graphs vs sizes of maps:
RAGs 2G-maps# vertices # edges # darts # vertices # edges # facesMin Max Min Max Min Max Min Max Min Max Min Max240 460 688 1217 2792 4948 463 816 698 1237 240 460
14/22
Experimental Comparison
Labels
Colour and size labels:For each region r :
color(r) = average color of the pixels of rsize(r) = number of pixels of r
Graphs: Labels are associated with the vertex corresponding to rMaps: Labels are associated with all darts of the face corresponding to r
Structural labels added to graphs:
Each edge (u, v) is labelled with the number ofadj. relations between regions associated with uand v
Each vertex u is labelled with the total nb of adj.relations of all edges incident to u
This greatly improves matching quality for graphs
Example:
Segmentation:
RAG:23
2
31
1
15/22
Experimental Comparison
Matching algorithms
Computation of Graph/Map Edit Distances is NP-hard
Use polynomial-time algorithms to compute approximations
Approximation of the Graph Edit Distance:
Assignment pb solved by the Munkres algorithm [Riesen & Bunke 2009]
Time complexity = O(v3), where v = number of vertices∼ 2s for coarsest segmentations (v = 240)∼ 5s for finest segmentations (v = 460)
Approximation of the Map Edit Distance
Extension of the greedy algorithm of [GbR 2011] to labelled maps
Time complexity = O(d2 · log(d))), where d = number of darts∼ 8s for coarsest segmentations (d = 2792)∼ 25s for finest segmentations (d = 4948)
16/22
Experimental Comparison
Results when color and size labels are ignored Only structural information is used for matching regions
2G−maps
Graphs
Aver
age
per
centa
ge
of
corr
ectl
y m
atch
ed d
arts
/ver
tice
s
Difference of segmentation levels between the 2 images 17/22
Experimental Comparison
Results when color labels are ignored Structural information and size labels are used for matching regions
2G−maps
Graphs
Aver
age
per
centa
ge
of
corr
ectl
y m
atch
ed d
arts
/ver
tice
s
Difference of segmentation levels between the 2 images 18/22
Experimental Comparison
Results when size labels are ignored Structural information and colour labels are used for matching regions
2G−maps
Graphs
Av
erag
e p
erce
nta
ge
of
corr
ectl
y m
atch
ed d
arts
/ver
tice
s
Difference of segmentation levels between the 2 images 19/22
Experimental Comparison
Results when all labels are used
2G−maps
Graphs
Aver
age
per
centa
ge
of
corr
ectl
y m
atch
ed d
arts
/ver
tice
s
Difference of segmentation levels between the 2 images
20/22
Conclusion
Overview of the talk:
1 Motivations
2 Recalls on Generalized Maps
3 Extension of the Map Edit Distance to Labelled Maps
4 Experimental Comparison
5 Conclusion
21/22
Conclusion
Conclusion
Summary of our contribution:
Extension of generalized maps to labelled maps Add information on cells in every dimension
Extension of the Map Edit Distance to labelled maps
Comparison with the Graph Edit Distance for matching image regions
MED significantly better than GED when colour is ignoredMED and GED not significantly different when using colour. . . but local approaches obtain similar results in this case !
Further work:
Consider other benchmarks
Consider other Graph and Map matching algorithms
Consider other similarity measures Definition of Map Kernels
22/22
Top Related