Polyhedral Computations for the Simple Graph Partitioning Problem
From graph matching problem to Assignment problem...
Transcript of From graph matching problem to Assignment problem...
Graphs in pattern recognitionAssignment problem in graph matching
Application to CBIRConclusion
From graph matching problem to Assignmentproblem :
Application to Content-Based Image Retrieval.
Romain Raveaux
LI lab (EA 6300) – Universite de Tours
Rouen, 1rst December, 2013
1 / 38
Graphs in pattern recognitionAssignment problem in graph matching
Application to CBIRConclusion
Problem statementGraph matching problemsState of the art
Graphs are everywhere...
Graphs in Reality
Graphs model objects andtheir relationships.
Also referred to asnetworks.
All common datastructures can be modeledas graphs.
2 / 38
Graphs in pattern recognitionAssignment problem in graph matching
Application to CBIRConclusion
Problem statementGraph matching problemsState of the art
Graphs are everywhere...
Graphs in Reality
Graphs model objects andtheir relationships.
Also referred to asnetworks.
All common datastructures can be modeledas graphs.
2 / 38
Graphs in pattern recognitionAssignment problem in graph matching
Application to CBIRConclusion
Problem statementGraph matching problemsState of the art
Graphs are everywhere...
Graphs in Reality
Graphs model objects andtheir relationships.
Also referred to asnetworks.
All common datastructures can be modeledas graphs.
2 / 38
Graphs in pattern recognitionAssignment problem in graph matching
Application to CBIRConclusion
Problem statementGraph matching problemsState of the art
Definition of a graph
Definition
Let LV and LE denote the set of node and edge labels, respectively.A labeled graph G is a 4-tuple G = (V ,E , µ, ξ) , where
V is the set of nodes,
E ⊆ V × V is the set of edges
µ : V → LV is a function assigning labels to the nodes, and
ξ : E → LE is a function assigning labels to the edges.
Let G1 = (V1,E1, µ1, ξ1) be the source graph
And G2 = (V2,E2, µ2, ξ2) the target graph
With V1 = (u1, ..., un) and V2 = (v1, ..., vm) respectively
3 / 38
Graphs in pattern recognitionAssignment problem in graph matching
Application to CBIRConclusion
Problem statementGraph matching problemsState of the art
Definition and notation of a graph
Labels for both nodes and edges can be given by :
Labelling functions
a set of integers L = {1, 2, 3, . . .}a vector space L = Rn
a finite set of symbolic labels L = {x,y, z, . . .}...
Definition 1 allows us to handle arbitrarily structured graphs withunconstrained labelling functions.
4 / 38
Graphs in pattern recognitionAssignment problem in graph matching
Application to CBIRConclusion
Problem statementGraph matching problemsState of the art
Graph comparison
How similar are two graphs?
Graph similarity is the central problem for all learning taskssuch as clustering and classification on graphs.
At least two ways for comparing graphs
Graph Embedding
Graph Matching
5 / 38
Graphs in pattern recognitionAssignment problem in graph matching
Application to CBIRConclusion
Problem statementGraph matching problemsState of the art
Graph comparison
How similar are two graphs?
Graph similarity is the central problem for all learning taskssuch as clustering and classification on graphs.
At least two ways for comparing graphs
Graph Embedding
Graph Matching
5 / 38
Graphs in pattern recognitionAssignment problem in graph matching
Application to CBIRConclusion
Problem statementGraph matching problemsState of the art
Graph comparison
How similar are two graphs?
Graph similarity is the central problem for all learning taskssuch as clustering and classification on graphs.
At least two ways for comparing graphs
Graph Embedding
Graph Matching
5 / 38
Graphs in pattern recognitionAssignment problem in graph matching
Application to CBIRConclusion
Problem statementGraph matching problemsState of the art
Graph isomorphism
Graph isomorphism
Find a mapping f : V1 → V2
i.e. x , y ∈ V1 ⇒ (x , y) ∈ E1
f is an isomorphism iff (f (x), f (y)) is an edge of G2.No polynomial-time algorithm is known for graph isomorphismNeither it is known to be NP-complete
Subgraph isomorphism
Means finding a subgraph G3 of G2 such that G1 and G3 areisomorphic.Subgraph isomorphism is NP-complete
6 / 38
Graphs in pattern recognitionAssignment problem in graph matching
Application to CBIRConclusion
Problem statementGraph matching problemsState of the art
Graph isomorphism
Graph isomorphism
Find a mapping f : V1 → V2
i.e. x , y ∈ V1 ⇒ (x , y) ∈ E1
f is an isomorphism iff (f (x), f (y)) is an edge of G2.No polynomial-time algorithm is known for graph isomorphismNeither it is known to be NP-complete
Subgraph isomorphism
Means finding a subgraph G3 of G2 such that G1 and G3 areisomorphic.Subgraph isomorphism is NP-complete
6 / 38
Graphs in pattern recognitionAssignment problem in graph matching
Application to CBIRConclusion
Problem statementGraph matching problemsState of the art
Graph isomorphism
Graph isomorphism
Find a mapping f : V1 → V2
i.e. x , y ∈ V1 ⇒ (x , y) ∈ E1
f is an isomorphism iff (f (x), f (y)) is an edge of G2.No polynomial-time algorithm is known for graph isomorphismNeither it is known to be NP-complete
Subgraph isomorphism
Means finding a subgraph G3 of G2 such that G1 and G3 areisomorphic.Subgraph isomorphism is NP-complete
6 / 38
Graphs in pattern recognitionAssignment problem in graph matching
Application to CBIRConclusion
Problem statementGraph matching problemsState of the art
Graph isomorphism
Graph isomorphism
Find a mapping f : V1 → V2
i.e. x , y ∈ V1 ⇒ (x , y) ∈ E1
f is an isomorphism iff (f (x), f (y)) is an edge of G2.No polynomial-time algorithm is known for graph isomorphismNeither it is known to be NP-complete
Subgraph isomorphism
Means finding a subgraph G3 of G2 such that G1 and G3 areisomorphic.Subgraph isomorphism is NP-complete
6 / 38
Graphs in pattern recognitionAssignment problem in graph matching
Application to CBIRConclusion
Problem statementGraph matching problemsState of the art
Graph isomorphism
Graph isomorphism
Find a mapping f : V1 → V2
i.e. x , y ∈ V1 ⇒ (x , y) ∈ E1
f is an isomorphism iff (f (x), f (y)) is an edge of G2.No polynomial-time algorithm is known for graph isomorphismNeither it is known to be NP-complete
Subgraph isomorphism
Means finding a subgraph G3 of G2 such that G1 and G3 areisomorphic.Subgraph isomorphism is NP-complete
6 / 38
Graphs in pattern recognitionAssignment problem in graph matching
Application to CBIRConclusion
Problem statementGraph matching problemsState of the art
Graph isomorphism
Graph isomorphism
Find a mapping f : V1 → V2
i.e. x , y ∈ V1 ⇒ (x , y) ∈ E1
f is an isomorphism iff (f (x), f (y)) is an edge of G2.No polynomial-time algorithm is known for graph isomorphismNeither it is known to be NP-complete
Subgraph isomorphism
Means finding a subgraph G3 of G2 such that G1 and G3 areisomorphic.Subgraph isomorphism is NP-complete
6 / 38
Graphs in pattern recognitionAssignment problem in graph matching
Application to CBIRConclusion
Problem statementGraph matching problemsState of the art
Graph isomorphism
Graph isomorphism
Find a mapping f : V1 → V2
i.e. x , y ∈ V1 ⇒ (x , y) ∈ E1
f is an isomorphism iff (f (x), f (y)) is an edge of G2.No polynomial-time algorithm is known for graph isomorphismNeither it is known to be NP-complete
Subgraph isomorphism
Means finding a subgraph G3 of G2 such that G1 and G3 areisomorphic.Subgraph isomorphism is NP-complete
6 / 38
Graphs in pattern recognitionAssignment problem in graph matching
Application to CBIRConclusion
Problem statementGraph matching problemsState of the art
Graph isomorphism
Graph isomorphism
Find a mapping f : V1 → V2
i.e. x , y ∈ V1 ⇒ (x , y) ∈ E1
f is an isomorphism iff (f (x), f (y)) is an edge of G2.No polynomial-time algorithm is known for graph isomorphismNeither it is known to be NP-complete
Subgraph isomorphism
Means finding a subgraph G3 of G2 such that G1 and G3 areisomorphic.Subgraph isomorphism is NP-complete
6 / 38
Graphs in pattern recognitionAssignment problem in graph matching
Application to CBIRConclusion
Problem statementGraph matching problemsState of the art
Graph isomorphism
Graph isomorphism
Find a mapping f : V1 → V2
i.e. x , y ∈ V1 ⇒ (x , y) ∈ E1
f is an isomorphism iff (f (x), f (y)) is an edge of G2.No polynomial-time algorithm is known for graph isomorphismNeither it is known to be NP-complete
Subgraph isomorphism
Means finding a subgraph G3 of G2 such that G1 and G3 areisomorphic.Subgraph isomorphism is NP-complete
6 / 38
Graphs in pattern recognitionAssignment problem in graph matching
Application to CBIRConclusion
Problem statementGraph matching problemsState of the art
Graph isomorphism families
1 Exact Matching : Exact mapping between vertex and/oredge labels.
2 Inexact Matching : No exact mapping between vertexand/or edge labels can be found, but when the mapping canbe associated to a cost.
In pattern recognition
Vertices and edges are labeled with measures/features whichmay be affected by noise.
Direct and exact comparisons are useless.
The matching problem turns from a decision one to anoptimization one.
7 / 38
Graphs in pattern recognitionAssignment problem in graph matching
Application to CBIRConclusion
Problem statementGraph matching problemsState of the art
Inexact matching
Three sub-problems :
1 Substitution-Tolerant Subgraph Isomorphism
2 Inexact Subgraph Isomorphism
3 Inexact Graph Isomorphism
8 / 38
Graphs in pattern recognitionAssignment problem in graph matching
Application to CBIRConclusion
Problem statementGraph matching problemsState of the art
Substitution-Tolerant Subgraph Isomorphism
1 This formulation allows differences between labels.2 The mapping does not affect the topology.3 The mapping should be edge-preserving in one direction.
V’ V E E’
f(v) f-‐1(v’)
f
0.5
0.3
0.15 0.13
Figure: Substitution-tolerant subgraph isomorphism problem
9 / 38
Graphs in pattern recognitionAssignment problem in graph matching
Application to CBIRConclusion
Problem statementGraph matching problemsState of the art
Inexact Subgraph Isomorphism
1 This formulation allows differences between labels.2 Topology can be different. The mapping is not
edge-preserving.
V’ V E E’
f(e)
f
0.5
0.3
0.13
Δe’
0.16
Unmatched
Matched
Figure: Inexact Subgraph Isomorphism problem
10 / 38
Graphs in pattern recognitionAssignment problem in graph matching
Application to CBIRConclusion
Problem statementGraph matching problemsState of the art
Inexact Graph Isomorphism
1 Bijective function2 This formulation allows differences between labels.3 Topology can be different.
V’ V E E’
f
0.5
0.3
0.13
Δe’
0.16
Δe Δv
Figure: Inexact graph isomorphism problem
11 / 38
Graphs in pattern recognitionAssignment problem in graph matching
Application to CBIRConclusion
Problem statementGraph matching problemsState of the art
Graph Matching problems
Figure: Graph isomorphism constraints
12 / 38
Graphs in pattern recognitionAssignment problem in graph matching
Application to CBIRConclusion
Problem statementGraph matching problemsState of the art
Comparison between Classical Graph-Matching Methods
Name Graphmatchingproblem
Models For-mulation
Optimality Complexityclass
Deterministic Optimizationmethod
References
NodeAssignment
IGI Approximate Sub-optimal Polynomial Deterministic Hungarian al-gorithm
[Riesen 2009],[Raveaux 2010],[Jouili 2009] ,[Shokoufandeh 2006]
Neural Net-work
ISGI Approximate Sub-Optimal Polynomial Non-Deterministic
Gradient [Kuner 1988],[Suganthan 2000]
Stringbasedmethods
ISGI Approximate Sub-Optimal Polynomial Deterministic Tree Search [J. Llados 2001],[Anjan Dutta 2012]
ApproximatedGED
IGI Exact Sub-Optimal NP-Complete Deterministic Heuristic Treesearch
[Neuhaus M 2006]
Genetic al-gorithm
ISGI Exact Sub-Optimal Polynomial Non-Deterministic
Genetic Algo-rithm
[A.D.J. Cross 1997],[Ford 1991],[Jiang 1998]
ProbabilisticRelaxation
ISGI Exact Sub-Optimal Polynomial Deterministic ProbabilisticRelaxation
[W. Christmas 1995],[Finch 1998],[Gold 1996]
Similaritysearch
MIGI Exact Sub-Optimal Polynomial Non-Deterministic
Greedy search,Tabu Search
[Solnon 2005]
GED IGI Exact Optimal NP-Complete Deterministic Tree search [Bunke 1998],[Bunke 1999]
GED ILP IGI Exact Optimal NP-Complete Deterministic MathematicalSolver
[Justice D 2006]
ILP STSI Exact Optimal NP-Complete Deterministic Mathematicalsolver
[LeBodic 2012]
Indexingmethod
ISGI Exact Optimal NP-Complete Deterministic Indexed search [Messmer 1998]
SpectralTheory
STSI Exact Optimal Polynomial Deterministic Eigen Decom-position
[Ferrer 2005] ,[Umeyama 1988].
Similaritysearch
MIGI Exact Optimal NP-Complete Deterministic Branch AndBound
[Champin ]
13 / 38
Graphs in pattern recognitionAssignment problem in graph matching
Application to CBIRConclusion
Problem statementGraph matching problemsState of the art
Some highlights
Exact graph matching is useless in many computer visionapplications
Concerning graph matching under noise and distortion
The matching incorporates an error model to identify thedistortions which make one graph a distorted version of theother
Problems for real world applications
Error-tolerant
To measure the similarity of two graphs.
Runtime may grow exponentially with number of nodes
This is an enormous problem for large datasets of graphs
Wanted: Polynomial-time similarity measure for graphs
14 / 38
Graphs in pattern recognitionAssignment problem in graph matching
Application to CBIRConclusion
Problem statementGraph matching problemsState of the art
Some highlights
Exact graph matching is useless in many computer visionapplications
Concerning graph matching under noise and distortion
The matching incorporates an error model to identify thedistortions which make one graph a distorted version of theother
Problems for real world applications
Error-tolerant
To measure the similarity of two graphs.
Runtime may grow exponentially with number of nodes
This is an enormous problem for large datasets of graphs
Wanted: Polynomial-time similarity measure for graphs
14 / 38
Graphs in pattern recognitionAssignment problem in graph matching
Application to CBIRConclusion
Problem statementGraph matching problemsState of the art
Some highlights
Exact graph matching is useless in many computer visionapplications
Concerning graph matching under noise and distortion
The matching incorporates an error model to identify thedistortions which make one graph a distorted version of theother
Problems for real world applications
Error-tolerant
To measure the similarity of two graphs.
Runtime may grow exponentially with number of nodes
This is an enormous problem for large datasets of graphs
Wanted: Polynomial-time similarity measure for graphs
14 / 38
Graphs in pattern recognitionAssignment problem in graph matching
Application to CBIRConclusion
Problem statementGraph matching problemsState of the art
Some highlights
Exact graph matching is useless in many computer visionapplications
Concerning graph matching under noise and distortion
The matching incorporates an error model to identify thedistortions which make one graph a distorted version of theother
Problems for real world applications
Error-tolerant
To measure the similarity of two graphs.
Runtime may grow exponentially with number of nodes
This is an enormous problem for large datasets of graphs
Wanted: Polynomial-time similarity measure for graphs
14 / 38
Graphs in pattern recognitionAssignment problem in graph matching
Application to CBIRConclusion
Problem statementGraph matching problemsState of the art
Some highlights
Exact graph matching is useless in many computer visionapplications
Concerning graph matching under noise and distortion
The matching incorporates an error model to identify thedistortions which make one graph a distorted version of theother
Problems for real world applications
Error-tolerant
To measure the similarity of two graphs.
Runtime may grow exponentially with number of nodes
This is an enormous problem for large datasets of graphs
Wanted: Polynomial-time similarity measure for graphs
14 / 38
Graphs in pattern recognitionAssignment problem in graph matching
Application to CBIRConclusion
Problem statementGraph matching problemsState of the art
Some highlights
Exact graph matching is useless in many computer visionapplications
Concerning graph matching under noise and distortion
The matching incorporates an error model to identify thedistortions which make one graph a distorted version of theother
Problems for real world applications
Error-tolerant
To measure the similarity of two graphs.
Runtime may grow exponentially with number of nodes
This is an enormous problem for large datasets of graphs
Wanted: Polynomial-time similarity measure for graphs
14 / 38
Graphs in pattern recognitionAssignment problem in graph matching
Application to CBIRConclusion
Problem statementGraph matching problemsState of the art
Some highlights
Exact graph matching is useless in many computer visionapplications
Concerning graph matching under noise and distortion
The matching incorporates an error model to identify thedistortions which make one graph a distorted version of theother
Problems for real world applications
Error-tolerant
To measure the similarity of two graphs.
Runtime may grow exponentially with number of nodes
This is an enormous problem for large datasets of graphs
Wanted: Polynomial-time similarity measure for graphs
14 / 38
Graphs in pattern recognitionAssignment problem in graph matching
Application to CBIRConclusion
Problem statementGraph matching problemsState of the art
Some highlights
Exact graph matching is useless in many computer visionapplications
Concerning graph matching under noise and distortion
The matching incorporates an error model to identify thedistortions which make one graph a distorted version of theother
Problems for real world applications
Error-tolerant
To measure the similarity of two graphs.
Runtime may grow exponentially with number of nodes
This is an enormous problem for large datasets of graphs
Wanted: Polynomial-time similarity measure for graphs
14 / 38
Graphs in pattern recognitionAssignment problem in graph matching
Application to CBIRConclusion
Problem statementGraph matching problemsState of the art
Some highlights
Exact graph matching is useless in many computer visionapplications
Concerning graph matching under noise and distortion
The matching incorporates an error model to identify thedistortions which make one graph a distorted version of theother
Problems for real world applications
Error-tolerant
To measure the similarity of two graphs.
Runtime may grow exponentially with number of nodes
This is an enormous problem for large datasets of graphs
Wanted: Polynomial-time similarity measure for graphs
14 / 38
Graphs in pattern recognitionAssignment problem in graph matching
Application to CBIRConclusion
DefinitionAssignment-based Matching
Assignment problem
Jacobi (1890) exposes a polynomial time algorithm to solve thefollowing problem:
Let ai ,j be a square matrix nxn, find a permutation σ suchthat
∑ai ,σ(i) be maximal.
Such an algorithm was only rediscovered in 1955 par Kuhn,who called it “Hungarian method”.
It is a well-known problem in mathematical economy, underthe name of “assignment problem”.
15 / 38
Graphs in pattern recognitionAssignment problem in graph matching
Application to CBIRConclusion
DefinitionAssignment-based Matching
Assignment problem
Say you have three workers: Jim, Steve and Alan. You need tohave one of them clean the bathroom, another sweep the floorsand the third wash the windows. What’s the best (minimum-cost)way to assign the jobs? First we need a matrix of the costs of theworkers doing the jobs.
Clean bathroom Sweep floors Wash windows
Jim 1 e 2 e 3 eSteve 3 e 3 e 3 eAlan 3 e 3 e 2 e
Table: Cost matrix
Then the Hungarian algorithm, when applied to the above tablewould give us the minimum cost it can be done with: Jim cleansthe bathroom, Steve sweeps the floors and Alan washes thewindows.
16 / 38
Graphs in pattern recognitionAssignment problem in graph matching
Application to CBIRConclusion
DefinitionAssignment-based Matching
Graph comparison through assignment problem
Basic idea:
Methods are based on an optimization procedure mappinglocal substructures
Any node(un) from G1 can be assigned to any node(vm) of G2,
Incurring some cost that depends on the un-vm assignment.
It is required to map all nodes in such a way that the totalcost of the assignment is minimized.
Cost matrix representation (C ):
Cij correspond to the costs of assigning the i th node of G1 tothe j th node of G2.
17 / 38
Graphs in pattern recognitionAssignment problem in graph matching
Application to CBIRConclusion
DefinitionAssignment-based Matching
Graph comparison through assignment problem
Basic idea:
Methods are based on an optimization procedure mappinglocal substructures
Any node(un) from G1 can be assigned to any node(vm) of G2,
Incurring some cost that depends on the un-vm assignment.
It is required to map all nodes in such a way that the totalcost of the assignment is minimized.
Cost matrix representation (C ):
Cij correspond to the costs of assigning the i th node of G1 tothe j th node of G2.
17 / 38
Graphs in pattern recognitionAssignment problem in graph matching
Application to CBIRConclusion
DefinitionAssignment-based Matching
Graph comparison through assignment problem
Basic idea:
Methods are based on an optimization procedure mappinglocal substructures
Any node(un) from G1 can be assigned to any node(vm) of G2,
Incurring some cost that depends on the un-vm assignment.
It is required to map all nodes in such a way that the totalcost of the assignment is minimized.
Cost matrix representation (C ):
Cij correspond to the costs of assigning the i th node of G1 tothe j th node of G2.
17 / 38
Graphs in pattern recognitionAssignment problem in graph matching
Application to CBIRConclusion
DefinitionAssignment-based Matching
Graph comparison through assignment problem
Basic idea:
Methods are based on an optimization procedure mappinglocal substructures
Any node(un) from G1 can be assigned to any node(vm) of G2,
Incurring some cost that depends on the un-vm assignment.
It is required to map all nodes in such a way that the totalcost of the assignment is minimized.
Cost matrix representation (C ):
Cij correspond to the costs of assigning the i th node of G1 tothe j th node of G2.
17 / 38
Graphs in pattern recognitionAssignment problem in graph matching
Application to CBIRConclusion
DefinitionAssignment-based Matching
Graph comparison through assignment problem
Basic idea:
Methods are based on an optimization procedure mappinglocal substructures
Any node(un) from G1 can be assigned to any node(vm) of G2,
Incurring some cost that depends on the un-vm assignment.
It is required to map all nodes in such a way that the totalcost of the assignment is minimized.
Cost matrix representation (C ):
Cij correspond to the costs of assigning the i th node of G1 tothe j th node of G2.
17 / 38
Graphs in pattern recognitionAssignment problem in graph matching
Application to CBIRConclusion
DefinitionAssignment-based Matching
Graph comparison through assignment problem
Basic idea:
Methods are based on an optimization procedure mappinglocal substructuresAny node(un) from G1 can be assigned to any node(vm) of G2,Incurring some cost that depends on the un-vm assignment.It is required to map all nodes in such a way that the totalcost of the assignment is minimized.
Cost matrix representation (C ):
Cij correspond to the costs of assigning the i th node of G1 tothe j th node of G2.
17 / 38
Graphs in pattern recognitionAssignment problem in graph matching
Application to CBIRConclusion
DefinitionAssignment-based Matching
Graph comparison through assignment problem
Basic idea:
Methods are based on an optimization procedure mappinglocal substructuresAny node(un) from G1 can be assigned to any node(vm) of G2,Incurring some cost that depends on the un-vm assignment.It is required to map all nodes in such a way that the totalcost of the assignment is minimized.
Cost matrix representation (C ):
Cij correspond to the costs of assigning the i th node of G1 tothe j th node of G2.
17 / 38
Graphs in pattern recognitionAssignment problem in graph matching
Application to CBIRConclusion
DefinitionAssignment-based Matching
Graph comparison through assignment problem
Univalent matching
Sets of equal size
Find a bijection b : G1 → G2
Given two sets of subgraphs, G1 and G2 of equal size, the problemcan be expressed as a standard linear program with the objectivefunction Eq 1 to be minimized.∑
i∈G1
∑j∈G2
C (i , j)xij (1)
The variable xij represents the assignment of a subgraph of G1 to asubgraph of G2, taking value 1 if the assignment is done and 0otherwise.
18 / 38
Graphs in pattern recognitionAssignment problem in graph matching
Application to CBIRConclusion
DefinitionAssignment-based Matching
Graph comparison through assignment problem: state ofthe art
Node signature Distance
[Gold 1996] Node degree+Label *
[Shokoufandeh 2006] Eigen vector L2
[Riesen 2009] (1)Node+(2)Edge Edit cost
[Jouili 2009] Node degree+Weight L1
[Raveaux 2010] Sub Graph Edit Distance*: Depends on the graph attribute type.
19 / 38
Graphs in pattern recognitionAssignment problem in graph matching
Application to CBIRConclusion
DefinitionAssignment-based Matching
Graph comparison through assignment problem: anexample
A subgraph (sg):
A structure gathering theedges and theircorresponding endingvertices from a rootvertex.
Romain Raveaux et al. Agraph matching methodand a graph matchingdistance based onsubgraph assignments.Pattern RecognitionLetters (2010)
Figure: Decomposition intosubgraph world
20 / 38
Graphs in pattern recognitionAssignment problem in graph matching
Application to CBIRConclusion
DefinitionAssignment-based Matching
Graph comparison through assignment problem: anexample
A subgraph (sg):
A structure gathering theedges and theircorresponding endingvertices from a rootvertex.
Romain Raveaux et al. Agraph matching methodand a graph matchingdistance based onsubgraph assignments.Pattern RecognitionLetters (2010)
Figure: Decomposition intosubgraph world
20 / 38
Graphs in pattern recognitionAssignment problem in graph matching
Application to CBIRConclusion
DefinitionAssignment-based Matching
Graph comparison through assignment problem: anexample II
A distance between graph
Subgraph decomposition
Optimization algorithm
Figure: Subgraph matching: A bipartite graph
21 / 38
Graphs in pattern recognitionAssignment problem in graph matching
Application to CBIRConclusion
DefinitionAssignment-based Matching
Graph comparison through assignment problem: anexample II
A distance between graph
Subgraph decomposition
Optimization algorithm
Figure: Subgraph matching: A bipartite graph
21 / 38
Graphs in pattern recognitionAssignment problem in graph matching
Application to CBIRConclusion
DefinitionAssignment-based Matching
Graph comparison through assignment problem: anexample II
A distance between graph
Subgraph decomposition
Optimization algorithm
Figure: Subgraph matching: A bipartite graph
21 / 38
Graphs in pattern recognitionAssignment problem in graph matching
Application to CBIRConclusion
DefinitionAssignment-based Matching
Synthesis
Inexact graph matching problem
Univalent matching
Polynomial algorithm : O(n3)
Two main directions
Node signature based methods (vectors).Subgraph based methods. (not a dot in a feature space.)
22 / 38
Graphs in pattern recognitionAssignment problem in graph matching
Application to CBIRConclusion
CBIR problemCBIR methodExperiments
How Graph Matching can be used in Pattern RecognitionProblems
Hypothesis :
a graph database consisting of n graphs. D = g1, g2, ..., gn
a query graph q
Problem definitions :1 Full graph search. Find all graphs gi in D s.t. gi is the same
as q.2 Subgraph search. Find all graphs gi in D containing q or
contained by q.3 Similarity search. Find all graphs gi in D s.t. gi is similar to
q within a user-specified threshold based on some similaritymeasures.
4 Graph mining. Graph mining problem gathers similar graphor subgraph of D in order to find clusters or prototypes.
23 / 38
Graphs in pattern recognitionAssignment problem in graph matching
Application to CBIRConclusion
CBIR problemCBIR methodExperiments
CBIR problem
Figure: Feature clustering stage
24 / 38
Graphs in pattern recognitionAssignment problem in graph matching
Application to CBIRConclusion
CBIR problemCBIR methodExperiments
CBIR scheme
Figure: Feature clustering stage
25 / 38
Graphs in pattern recognitionAssignment problem in graph matching
Application to CBIRConclusion
CBIR problemCBIR methodExperiments
CBIR scheme
Figure: Images to graphs
26 / 38
Graphs in pattern recognitionAssignment problem in graph matching
Application to CBIRConclusion
CBIR problemCBIR methodExperiments
CBIR scheme
Figure: Similarity search 27 / 38
Graphs in pattern recognitionAssignment problem in graph matching
Application to CBIRConclusion
CBIR problemCBIR methodExperiments
(a) (b)
Figure: Regions of Interest found by the SIFT algorithm. ProcessingSIFT took 407ms, 60 features were identified and processed
28 / 38
Graphs in pattern recognitionAssignment problem in graph matching
Application to CBIRConclusion
CBIR problemCBIR methodExperiments
Figure: A segmentation result. Processing SRM took 1625ms and 26features were identified and processed
29 / 38
Graphs in pattern recognitionAssignment problem in graph matching
Application to CBIRConclusion
CBIR problemCBIR methodExperiments
Figure: Multiple representations 30 / 38
Graphs in pattern recognitionAssignment problem in graph matching
Application to CBIRConclusion
CBIR problemCBIR methodExperiments
Figure: CBIR approach categorization according to the amount of spatialinformation captured
31 / 38
Graphs in pattern recognitionAssignment problem in graph matching
Application to CBIRConclusion
CBIR problemCBIR methodExperiments
Figure: CBIR taxonomy
32 / 38
Graphs in pattern recognitionAssignment problem in graph matching
Application to CBIRConclusion
CBIR problemCBIR methodExperiments
Figure: Image Samples from the Caltech-101 Data set. The 101 objectcategories and the background clutter category.
33 / 38
Graphs in pattern recognitionAssignment problem in graph matching
Application to CBIRConclusion
CBIR problemCBIR methodExperiments
Figure: Columbia University Image Library
34 / 38
Graphs in pattern recognitionAssignment problem in graph matching
Application to CBIRConclusion
CBIR problemCBIR methodExperiments
Recognition rate
(a) Coil-100 (b) Caltech-101
Figure: Comparison between CBIR methods. Summary of resultsobtained with the best number of words for each method.
Romain Raveaux et al: Structured representations in a content based image retrieval context. J. Visual
Communication and Image Representation (2013). Elsevier.
35 / 38
Graphs in pattern recognitionAssignment problem in graph matching
Application to CBIRConclusion
CBIR problemCBIR methodExperiments
Vocabulary size impact
(a) Coil-100 (b) Caltech-101
Figure: Comparison between the number of words in used by themethods.
Romain Raveaux et al: Structured representations in a content based image retrieval context. J. Visual
Communication and Image Representation (2013). Elsevier.
36 / 38
Graphs in pattern recognitionAssignment problem in graph matching
Application to CBIRConclusion
CBIR problemCBIR methodExperiments
Time experiments
Average response time in seconds SIFTBoW IFFSBoW IFFSGBR
Coil − 100 28 37 1555
Table: Average response time of a query. A speed comparison of the fourmethods at the worst case.
37 / 38
Graphs in pattern recognitionAssignment problem in graph matching
Application to CBIRConclusion
Conclusion
To take the stock of :
SIFTBoW is better on Caltech data
IFFS is better on Coil-100
The Graph Based Approach gave similar recognition rate thanBoW methods.
A structural approach requires a fewer number of words toreach its best performance.
Future work
To avoid visual features clustering.
To enrich the graph representation.
Combination of BoW and GbR : BoW first and then to processa re-ranking stage with the top k responses based on GbR.
38 / 38
Graphs in pattern recognitionAssignment problem in graph matching
Application to CBIRConclusion
R.C. Wilson A.D.J. Cross and E.R. Hancock.Inexact Graph Matching Using Genetic Search.Pattern Recognition, pages 953–970, 1997.
Josep Llados Anjan Dutta and Umapada Pal.A symbol spotting approach in graphical documents byhashing serialized graphs.Pattern Recognition, pages 752–768, 2012.
Horst Bunke and Kim Shearer.A graph distance metric based on the maximal commonsubgraph.Pattern Recognition Letters, vol. 2, page 255–259, 1998.
Horst Bunke.Error Correcting Graph Matching: On the Influence of theUnderlying Cost Function.
38 / 38
Graphs in pattern recognitionAssignment problem in graph matching
Application to CBIRConclusion
IEEE Trans. Pattern Anal. Mach. Intell., vol. 21, no. 9, pages917–922, 1999.
Pierre-antoine Champin and Christine Solnon.Measuring the similarity of labeled graphs.
Miquel Ferrer, Francesc Serratosa and Alberto Sanfeliu.Synthesis of Median Spectral Graph.In IbPRIA (2), pages 139–146, 2005.
et al. Finch Wilson.An energy function and continuous edit process for graphmatching.Neural Computat, vol. 10, 1998.
G.P. Ford and J. Zhang.A Structural Graph Matching Approach to ImageUnderstanding.
38 / 38
Graphs in pattern recognitionAssignment problem in graph matching
Application to CBIRConclusion
Intelligent Robots and Computer Vision X: Algorithms andTechniques, pages 559–569, 1991.
S. Gold and A Rangarajan.A graduated assignment algorithm for graph matching.Workshops SSPR and SPR, pages 377 – 388, 1996.
E. Marti J. Llados and J. J. Villanueva.Symbol recognition by error-tolerant sub- graph matchingbetween region adjacency graphs.Patt. Anal. Mach., pages 1137–1143, 2001.
X. Jiang and H. Bunke.Marked subgraph isomorphism of ordered graphs.Workshops SSPR and SPR, pages 122–131, 1998.
Salim Jouili and Salvatore Tabbone.Graph Matching Based on Node Signatures.In GbRPR, pages 154–163, 2009.
38 / 38
Graphs in pattern recognitionAssignment problem in graph matching
Application to CBIRConclusion
Hero A Justice D.A binary linear programming formulation of the graph editdistance.IEEE Trans Pattern Anal Mach Intell., vol. 28, page1200–1214, 2006.
P Kuner and B Ueberreiter.Pattern Recognition by Graph Matching: Combinatorial versusContinuous Optimization.International Journal in Pattern Recognition and ArtificialIntelligence, vol. 2, page 527–542, 1988.
P. LeBodic and H. Heroux.An integer linear program for substitution-tolerant subgraphisomorphism and its use for symbol spotting in technicaldrawings.Pattern Recognition, pages 4214–4224, 2012.
38 / 38
Graphs in pattern recognitionAssignment problem in graph matching
Application to CBIRConclusion
B. Messmer and Bunke.Error-Correcting Graph Isomorphism using Decision Trees.Int. Journal of Pattern Recognition and Ar, vol. 12, page721–742, 1998.
Bunke H Neuhaus M.A convolution edit kernel for error-tolerant graph matching.Proceedings of IEEE international conference on patternrecognition., vol. 30, pages 220–223, 2006.
Burie J. Raveaux R. and Ogier.A graph matching method and a graph matching distancebased on subgraph assignments.Pattern Recognition Letters, page 394–406, 2010.
Kaspar Riesen and Horst Bunke.Approximate graph edit distance computation by means ofbipartite graph matching.
38 / 38
Graphs in pattern recognitionAssignment problem in graph matching
Application to CBIRConclusion
Image Vision Comput., vol. 27, no. 7, pages 950–959, 2009.
Bretzner L. Macrini D. Fatih Demirci M. Jonsson C.Shokoufandeh A. and S. Dickinson.The representation and matching of categorical shape.Computer Vision and Image Understanding, page 139–154,2006.
Christine Solnon.Reactive Tabu Search for Measuring Graph.pages 172–182, 2005.
P. N. Suganthan.Attributed relational graph matching by neural-gas networks.IEEE Signal Processing Society Workshop on Neural Networksfor Signal Processing X, pages 366–374, 2000.
S. Umeyama.
38 / 38
Graphs in pattern recognitionAssignment problem in graph matching
Application to CBIRConclusion
An eigendecomposition approach to weighted graph matchingproblems.IEEE Pattern Anal. Mach. Intel, page 695–703, 1988.
J. Kittler W. Christmas and M. Petrou.Structural matching in computer vision using probabilisticrelaxation.IEEE Trans. PAMI,, vol. 2, page 749–764, 1995.
38 / 38