Graphs in pattern recognitionAssignment problem in graph matching

Application to CBIRConclusion

From graph matching problem to Assignmentproblem :

Application to Content-Based Image Retrieval.

Romain Raveaux

LI lab (EA 6300) – Universite de Tours

Rouen, 1rst December, 2013

1 / 38

Graphs in pattern recognitionAssignment problem in graph matching

Application to CBIRConclusion

Problem statementGraph matching problemsState of the art

Graphs are everywhere...

Graphs in Reality

Graphs model objects andtheir relationships.

Also referred to asnetworks.

All common datastructures can be modeledas graphs.

2 / 38

Graphs in pattern recognitionAssignment problem in graph matching

Application to CBIRConclusion

Problem statementGraph matching problemsState of the art

Graphs are everywhere...

Graphs in Reality

Graphs model objects andtheir relationships.

Also referred to asnetworks.

All common datastructures can be modeledas graphs.

2 / 38

Graphs in pattern recognitionAssignment problem in graph matching

Application to CBIRConclusion

Problem statementGraph matching problemsState of the art

Graphs are everywhere...

Graphs in Reality

Graphs model objects andtheir relationships.

Also referred to asnetworks.

All common datastructures can be modeledas graphs.

2 / 38

Graphs in pattern recognitionAssignment problem in graph matching

Application to CBIRConclusion

Problem statementGraph matching problemsState of the art

Definition of a graph

Definition

Let LV and LE denote the set of node and edge labels, respectively.A labeled graph G is a 4-tuple G = (V ,E , µ, ξ) , where

V is the set of nodes,

E ⊆ V × V is the set of edges

µ : V → LV is a function assigning labels to the nodes, and

ξ : E → LE is a function assigning labels to the edges.

Let G1 = (V1,E1, µ1, ξ1) be the source graph

And G2 = (V2,E2, µ2, ξ2) the target graph

With V1 = (u1, ..., un) and V2 = (v1, ..., vm) respectively

3 / 38

Graphs in pattern recognitionAssignment problem in graph matching

Application to CBIRConclusion

Problem statementGraph matching problemsState of the art

Definition and notation of a graph

Labels for both nodes and edges can be given by :

Labelling functions

a set of integers L = {1, 2, 3, . . .}a vector space L = Rn

a finite set of symbolic labels L = {x,y, z, . . .}...

Definition 1 allows us to handle arbitrarily structured graphs withunconstrained labelling functions.

4 / 38

Graphs in pattern recognitionAssignment problem in graph matching

Application to CBIRConclusion

Problem statementGraph matching problemsState of the art

Graph comparison

How similar are two graphs?

Graph similarity is the central problem for all learning taskssuch as clustering and classification on graphs.

At least two ways for comparing graphs

Graph Embedding

Graph Matching

5 / 38

Graphs in pattern recognitionAssignment problem in graph matching

Application to CBIRConclusion

Problem statementGraph matching problemsState of the art

Graph comparison

How similar are two graphs?

Graph similarity is the central problem for all learning taskssuch as clustering and classification on graphs.

At least two ways for comparing graphs

Graph Embedding

Graph Matching

5 / 38

Graphs in pattern recognitionAssignment problem in graph matching

Application to CBIRConclusion

Problem statementGraph matching problemsState of the art

Graph comparison

How similar are two graphs?

Graph similarity is the central problem for all learning taskssuch as clustering and classification on graphs.

At least two ways for comparing graphs

Graph Embedding

Graph Matching

5 / 38

Graphs in pattern recognitionAssignment problem in graph matching

Application to CBIRConclusion

Problem statementGraph matching problemsState of the art

Graph isomorphism

Graph isomorphism

Find a mapping f : V1 → V2

i.e. x , y ∈ V1 ⇒ (x , y) ∈ E1

f is an isomorphism iff (f (x), f (y)) is an edge of G2.No polynomial-time algorithm is known for graph isomorphismNeither it is known to be NP-complete

Subgraph isomorphism

Means finding a subgraph G3 of G2 such that G1 and G3 areisomorphic.Subgraph isomorphism is NP-complete

6 / 38

Graphs in pattern recognitionAssignment problem in graph matching

Application to CBIRConclusion

Problem statementGraph matching problemsState of the art

Graph isomorphism

Graph isomorphism

Find a mapping f : V1 → V2

i.e. x , y ∈ V1 ⇒ (x , y) ∈ E1

f is an isomorphism iff (f (x), f (y)) is an edge of G2.No polynomial-time algorithm is known for graph isomorphismNeither it is known to be NP-complete

Subgraph isomorphism

Means finding a subgraph G3 of G2 such that G1 and G3 areisomorphic.Subgraph isomorphism is NP-complete

6 / 38

Graphs in pattern recognitionAssignment problem in graph matching

Application to CBIRConclusion

Problem statementGraph matching problemsState of the art

Graph isomorphism

Graph isomorphism

Find a mapping f : V1 → V2

i.e. x , y ∈ V1 ⇒ (x , y) ∈ E1

f is an isomorphism iff (f (x), f (y)) is an edge of G2.No polynomial-time algorithm is known for graph isomorphismNeither it is known to be NP-complete

Subgraph isomorphism

Means finding a subgraph G3 of G2 such that G1 and G3 areisomorphic.Subgraph isomorphism is NP-complete

6 / 38

Graphs in pattern recognitionAssignment problem in graph matching

Application to CBIRConclusion

Problem statementGraph matching problemsState of the art

Graph isomorphism

Graph isomorphism

Find a mapping f : V1 → V2

i.e. x , y ∈ V1 ⇒ (x , y) ∈ E1

f is an isomorphism iff (f (x), f (y)) is an edge of G2.No polynomial-time algorithm is known for graph isomorphismNeither it is known to be NP-complete

Subgraph isomorphism

Means finding a subgraph G3 of G2 such that G1 and G3 areisomorphic.Subgraph isomorphism is NP-complete

6 / 38

Graphs in pattern recognitionAssignment problem in graph matching

Application to CBIRConclusion

Problem statementGraph matching problemsState of the art

Graph isomorphism

Graph isomorphism

Find a mapping f : V1 → V2

i.e. x , y ∈ V1 ⇒ (x , y) ∈ E1

f is an isomorphism iff (f (x), f (y)) is an edge of G2.No polynomial-time algorithm is known for graph isomorphismNeither it is known to be NP-complete

Subgraph isomorphism

Means finding a subgraph G3 of G2 such that G1 and G3 areisomorphic.Subgraph isomorphism is NP-complete

6 / 38

Graphs in pattern recognitionAssignment problem in graph matching

Application to CBIRConclusion

Problem statementGraph matching problemsState of the art

Graph isomorphism

Graph isomorphism

Find a mapping f : V1 → V2

i.e. x , y ∈ V1 ⇒ (x , y) ∈ E1

f is an isomorphism iff (f (x), f (y)) is an edge of G2.No polynomial-time algorithm is known for graph isomorphismNeither it is known to be NP-complete

Subgraph isomorphism

Means finding a subgraph G3 of G2 such that G1 and G3 areisomorphic.Subgraph isomorphism is NP-complete

6 / 38

Graphs in pattern recognitionAssignment problem in graph matching

Application to CBIRConclusion

Problem statementGraph matching problemsState of the art

Graph isomorphism

Graph isomorphism

Find a mapping f : V1 → V2

i.e. x , y ∈ V1 ⇒ (x , y) ∈ E1

f is an isomorphism iff (f (x), f (y)) is an edge of G2.No polynomial-time algorithm is known for graph isomorphismNeither it is known to be NP-complete

Subgraph isomorphism

Means finding a subgraph G3 of G2 such that G1 and G3 areisomorphic.Subgraph isomorphism is NP-complete

6 / 38

Graphs in pattern recognitionAssignment problem in graph matching

Application to CBIRConclusion

Problem statementGraph matching problemsState of the art

Graph isomorphism

Graph isomorphism

Find a mapping f : V1 → V2

i.e. x , y ∈ V1 ⇒ (x , y) ∈ E1

f is an isomorphism iff (f (x), f (y)) is an edge of G2.No polynomial-time algorithm is known for graph isomorphismNeither it is known to be NP-complete

Subgraph isomorphism

Means finding a subgraph G3 of G2 such that G1 and G3 areisomorphic.Subgraph isomorphism is NP-complete

6 / 38

Graphs in pattern recognitionAssignment problem in graph matching

Application to CBIRConclusion

Problem statementGraph matching problemsState of the art

Graph isomorphism

Graph isomorphism

Find a mapping f : V1 → V2

i.e. x , y ∈ V1 ⇒ (x , y) ∈ E1

f is an isomorphism iff (f (x), f (y)) is an edge of G2.No polynomial-time algorithm is known for graph isomorphismNeither it is known to be NP-complete

Subgraph isomorphism

Means finding a subgraph G3 of G2 such that G1 and G3 areisomorphic.Subgraph isomorphism is NP-complete

6 / 38

Graphs in pattern recognitionAssignment problem in graph matching

Application to CBIRConclusion

Problem statementGraph matching problemsState of the art

Graph isomorphism families

1 Exact Matching : Exact mapping between vertex and/oredge labels.

2 Inexact Matching : No exact mapping between vertexand/or edge labels can be found, but when the mapping canbe associated to a cost.

In pattern recognition

Vertices and edges are labeled with measures/features whichmay be affected by noise.

Direct and exact comparisons are useless.

The matching problem turns from a decision one to anoptimization one.

7 / 38

Graphs in pattern recognitionAssignment problem in graph matching

Application to CBIRConclusion

Problem statementGraph matching problemsState of the art

Inexact matching

Three sub-problems :

1 Substitution-Tolerant Subgraph Isomorphism

2 Inexact Subgraph Isomorphism

3 Inexact Graph Isomorphism

8 / 38

Graphs in pattern recognitionAssignment problem in graph matching

Application to CBIRConclusion

Problem statementGraph matching problemsState of the art

Substitution-Tolerant Subgraph Isomorphism

1 This formulation allows differences between labels.2 The mapping does not affect the topology.3 The mapping should be edge-preserving in one direction.

V’  V  E   E’  

    f(v)  f-­‐1(v’)  

f  

0.5      

0.3  

0.15   0.13  

Figure: Substitution-tolerant subgraph isomorphism problem

9 / 38

Graphs in pattern recognitionAssignment problem in graph matching

Application to CBIRConclusion

Problem statementGraph matching problemsState of the art

Inexact Subgraph Isomorphism

1 This formulation allows differences between labels.2 Topology can be different. The mapping is not

edge-preserving.

V’  V  E     E’  

   

f(e)  

f  

0.5      

0.3  

0.13  

Δe’  

0.16  

Unmatched  

Matched  

Figure: Inexact Subgraph Isomorphism problem

10 / 38

Graphs in pattern recognitionAssignment problem in graph matching

Application to CBIRConclusion

Problem statementGraph matching problemsState of the art

Inexact Graph Isomorphism

1 Bijective function2 This formulation allows differences between labels.3 Topology can be different.

V’  V  E     E’  

   

f  

0.5      

0.3  

0.13  

Δe’  

0.16  

Δe   Δv  

Figure: Inexact graph isomorphism problem

11 / 38

Graphs in pattern recognitionAssignment problem in graph matching

Application to CBIRConclusion

Problem statementGraph matching problemsState of the art

Graph Matching problems

Figure: Graph isomorphism constraints

12 / 38

Graphs in pattern recognitionAssignment problem in graph matching

Application to CBIRConclusion

Problem statementGraph matching problemsState of the art

Comparison between Classical Graph-Matching Methods

Name Graphmatchingproblem

Models For-mulation

Optimality Complexityclass

Deterministic Optimizationmethod

References

NodeAssignment

IGI Approximate Sub-optimal Polynomial Deterministic Hungarian al-gorithm

[Riesen 2009],[Raveaux 2010],[Jouili 2009] ,[Shokoufandeh 2006]

Neural Net-work

ISGI Approximate Sub-Optimal Polynomial Non-Deterministic

Gradient [Kuner 1988],[Suganthan 2000]

Stringbasedmethods

ISGI Approximate Sub-Optimal Polynomial Deterministic Tree Search [J. Llados 2001],[Anjan Dutta 2012]

ApproximatedGED

IGI Exact Sub-Optimal NP-Complete Deterministic Heuristic Treesearch

[Neuhaus M 2006]

Genetic al-gorithm

ISGI Exact Sub-Optimal Polynomial Non-Deterministic

Genetic Algo-rithm

[A.D.J. Cross 1997],[Ford 1991],[Jiang 1998]

ProbabilisticRelaxation

ISGI Exact Sub-Optimal Polynomial Deterministic ProbabilisticRelaxation

[W. Christmas 1995],[Finch 1998],[Gold 1996]

Similaritysearch

MIGI Exact Sub-Optimal Polynomial Non-Deterministic

Greedy search,Tabu Search

[Solnon 2005]

GED IGI Exact Optimal NP-Complete Deterministic Tree search [Bunke 1998],[Bunke 1999]

GED ILP IGI Exact Optimal NP-Complete Deterministic MathematicalSolver

[Justice D 2006]

ILP STSI Exact Optimal NP-Complete Deterministic Mathematicalsolver

[LeBodic 2012]

Indexingmethod

ISGI Exact Optimal NP-Complete Deterministic Indexed search [Messmer 1998]

SpectralTheory

STSI Exact Optimal Polynomial Deterministic Eigen Decom-position

[Ferrer 2005] ,[Umeyama 1988].

Similaritysearch

MIGI Exact Optimal NP-Complete Deterministic Branch AndBound

[Champin ]

13 / 38

Graphs in pattern recognitionAssignment problem in graph matching

Application to CBIRConclusion

Problem statementGraph matching problemsState of the art

Some highlights

Exact graph matching is useless in many computer visionapplications

Concerning graph matching under noise and distortion

The matching incorporates an error model to identify thedistortions which make one graph a distorted version of theother

Problems for real world applications

Error-tolerant

To measure the similarity of two graphs.

Runtime may grow exponentially with number of nodes

This is an enormous problem for large datasets of graphs

Wanted: Polynomial-time similarity measure for graphs

14 / 38

Graphs in pattern recognitionAssignment problem in graph matching

Application to CBIRConclusion

Problem statementGraph matching problemsState of the art

Some highlights

Exact graph matching is useless in many computer visionapplications

Concerning graph matching under noise and distortion

The matching incorporates an error model to identify thedistortions which make one graph a distorted version of theother

Problems for real world applications

Error-tolerant

To measure the similarity of two graphs.

Runtime may grow exponentially with number of nodes

This is an enormous problem for large datasets of graphs

Wanted: Polynomial-time similarity measure for graphs

14 / 38

Graphs in pattern recognitionAssignment problem in graph matching

Application to CBIRConclusion

Problem statementGraph matching problemsState of the art

Some highlights

Exact graph matching is useless in many computer visionapplications

Concerning graph matching under noise and distortion

The matching incorporates an error model to identify thedistortions which make one graph a distorted version of theother

Problems for real world applications

Error-tolerant

To measure the similarity of two graphs.

Runtime may grow exponentially with number of nodes

This is an enormous problem for large datasets of graphs

Wanted: Polynomial-time similarity measure for graphs

14 / 38

Graphs in pattern recognitionAssignment problem in graph matching

Application to CBIRConclusion

Problem statementGraph matching problemsState of the art

Some highlights

Exact graph matching is useless in many computer visionapplications

Concerning graph matching under noise and distortion

The matching incorporates an error model to identify thedistortions which make one graph a distorted version of theother

Problems for real world applications

Error-tolerant

To measure the similarity of two graphs.

Runtime may grow exponentially with number of nodes

This is an enormous problem for large datasets of graphs

Wanted: Polynomial-time similarity measure for graphs

14 / 38

Graphs in pattern recognitionAssignment problem in graph matching

Application to CBIRConclusion

Problem statementGraph matching problemsState of the art

Some highlights

Exact graph matching is useless in many computer visionapplications

Concerning graph matching under noise and distortion

The matching incorporates an error model to identify thedistortions which make one graph a distorted version of theother

Problems for real world applications

Error-tolerant

To measure the similarity of two graphs.

Runtime may grow exponentially with number of nodes

This is an enormous problem for large datasets of graphs

Wanted: Polynomial-time similarity measure for graphs

14 / 38

Graphs in pattern recognitionAssignment problem in graph matching

Application to CBIRConclusion

Problem statementGraph matching problemsState of the art

Some highlights

Exact graph matching is useless in many computer visionapplications

Concerning graph matching under noise and distortion

The matching incorporates an error model to identify thedistortions which make one graph a distorted version of theother

Problems for real world applications

Error-tolerant

To measure the similarity of two graphs.

Runtime may grow exponentially with number of nodes

This is an enormous problem for large datasets of graphs

Wanted: Polynomial-time similarity measure for graphs

14 / 38

Graphs in pattern recognitionAssignment problem in graph matching

Application to CBIRConclusion

Problem statementGraph matching problemsState of the art

Some highlights

Exact graph matching is useless in many computer visionapplications

Concerning graph matching under noise and distortion

The matching incorporates an error model to identify thedistortions which make one graph a distorted version of theother

Problems for real world applications

Error-tolerant

To measure the similarity of two graphs.

Runtime may grow exponentially with number of nodes

This is an enormous problem for large datasets of graphs

Wanted: Polynomial-time similarity measure for graphs

14 / 38

Graphs in pattern recognitionAssignment problem in graph matching

Application to CBIRConclusion

Problem statementGraph matching problemsState of the art

Some highlights

Exact graph matching is useless in many computer visionapplications

Concerning graph matching under noise and distortion

The matching incorporates an error model to identify thedistortions which make one graph a distorted version of theother

Problems for real world applications

Error-tolerant

To measure the similarity of two graphs.

Runtime may grow exponentially with number of nodes

This is an enormous problem for large datasets of graphs

Wanted: Polynomial-time similarity measure for graphs

14 / 38

Graphs in pattern recognitionAssignment problem in graph matching

Application to CBIRConclusion

Problem statementGraph matching problemsState of the art

Some highlights

Exact graph matching is useless in many computer visionapplications

Concerning graph matching under noise and distortion

The matching incorporates an error model to identify thedistortions which make one graph a distorted version of theother

Problems for real world applications

Error-tolerant

To measure the similarity of two graphs.

Runtime may grow exponentially with number of nodes

This is an enormous problem for large datasets of graphs

Wanted: Polynomial-time similarity measure for graphs

14 / 38

Graphs in pattern recognitionAssignment problem in graph matching

Application to CBIRConclusion

DefinitionAssignment-based Matching

Assignment problem

Jacobi (1890) exposes a polynomial time algorithm to solve thefollowing problem:

Let ai ,j be a square matrix nxn, find a permutation σ suchthat

∑ai ,σ(i) be maximal.

Such an algorithm was only rediscovered in 1955 par Kuhn,who called it “Hungarian method”.

It is a well-known problem in mathematical economy, underthe name of “assignment problem”.

15 / 38

Graphs in pattern recognitionAssignment problem in graph matching

Application to CBIRConclusion

DefinitionAssignment-based Matching

Assignment problem

Say you have three workers: Jim, Steve and Alan. You need tohave one of them clean the bathroom, another sweep the floorsand the third wash the windows. What’s the best (minimum-cost)way to assign the jobs? First we need a matrix of the costs of theworkers doing the jobs.

Clean bathroom Sweep floors Wash windows

Jim 1 e 2 e 3 eSteve 3 e 3 e 3 eAlan 3 e 3 e 2 e

Table: Cost matrix

Then the Hungarian algorithm, when applied to the above tablewould give us the minimum cost it can be done with: Jim cleansthe bathroom, Steve sweeps the floors and Alan washes thewindows.

16 / 38

Graphs in pattern recognitionAssignment problem in graph matching

Application to CBIRConclusion

DefinitionAssignment-based Matching

Graph comparison through assignment problem

Basic idea:

Methods are based on an optimization procedure mappinglocal substructures

Any node(un) from G1 can be assigned to any node(vm) of G2,

Incurring some cost that depends on the un-vm assignment.

It is required to map all nodes in such a way that the totalcost of the assignment is minimized.

Cost matrix representation (C ):

Cij correspond to the costs of assigning the i th node of G1 tothe j th node of G2.

17 / 38

Graphs in pattern recognitionAssignment problem in graph matching

Application to CBIRConclusion

DefinitionAssignment-based Matching

Graph comparison through assignment problem

Basic idea:

Methods are based on an optimization procedure mappinglocal substructures

Any node(un) from G1 can be assigned to any node(vm) of G2,

Incurring some cost that depends on the un-vm assignment.

It is required to map all nodes in such a way that the totalcost of the assignment is minimized.

Cost matrix representation (C ):

Cij correspond to the costs of assigning the i th node of G1 tothe j th node of G2.

17 / 38

Graphs in pattern recognitionAssignment problem in graph matching

Application to CBIRConclusion

DefinitionAssignment-based Matching

Graph comparison through assignment problem

Basic idea:

Methods are based on an optimization procedure mappinglocal substructures

Any node(un) from G1 can be assigned to any node(vm) of G2,

Incurring some cost that depends on the un-vm assignment.

It is required to map all nodes in such a way that the totalcost of the assignment is minimized.

Cost matrix representation (C ):

Cij correspond to the costs of assigning the i th node of G1 tothe j th node of G2.

17 / 38

Graphs in pattern recognitionAssignment problem in graph matching

Application to CBIRConclusion

DefinitionAssignment-based Matching

Graph comparison through assignment problem

Basic idea:

Methods are based on an optimization procedure mappinglocal substructures

Any node(un) from G1 can be assigned to any node(vm) of G2,

Incurring some cost that depends on the un-vm assignment.

It is required to map all nodes in such a way that the totalcost of the assignment is minimized.

Cost matrix representation (C ):

Cij correspond to the costs of assigning the i th node of G1 tothe j th node of G2.

17 / 38

Graphs in pattern recognitionAssignment problem in graph matching

Application to CBIRConclusion

DefinitionAssignment-based Matching

Graph comparison through assignment problem

Basic idea:

Methods are based on an optimization procedure mappinglocal substructures

Any node(un) from G1 can be assigned to any node(vm) of G2,

Incurring some cost that depends on the un-vm assignment.

It is required to map all nodes in such a way that the totalcost of the assignment is minimized.

Cost matrix representation (C ):

Cij correspond to the costs of assigning the i th node of G1 tothe j th node of G2.

17 / 38

Graphs in pattern recognitionAssignment problem in graph matching

Application to CBIRConclusion

DefinitionAssignment-based Matching

Graph comparison through assignment problem

Basic idea:

Methods are based on an optimization procedure mappinglocal substructuresAny node(un) from G1 can be assigned to any node(vm) of G2,Incurring some cost that depends on the un-vm assignment.It is required to map all nodes in such a way that the totalcost of the assignment is minimized.

Cost matrix representation (C ):

Cij correspond to the costs of assigning the i th node of G1 tothe j th node of G2.

17 / 38

Graphs in pattern recognitionAssignment problem in graph matching

Application to CBIRConclusion

DefinitionAssignment-based Matching

Graph comparison through assignment problem

Basic idea:

Methods are based on an optimization procedure mappinglocal substructuresAny node(un) from G1 can be assigned to any node(vm) of G2,Incurring some cost that depends on the un-vm assignment.It is required to map all nodes in such a way that the totalcost of the assignment is minimized.

Cost matrix representation (C ):

Cij correspond to the costs of assigning the i th node of G1 tothe j th node of G2.

17 / 38

Graphs in pattern recognitionAssignment problem in graph matching

Application to CBIRConclusion

DefinitionAssignment-based Matching

Graph comparison through assignment problem

Univalent matching

Sets of equal size

Find a bijection b : G1 → G2

Given two sets of subgraphs, G1 and G2 of equal size, the problemcan be expressed as a standard linear program with the objectivefunction Eq 1 to be minimized.∑

i∈G1

∑j∈G2

C (i , j)xij (1)

The variable xij represents the assignment of a subgraph of G1 to asubgraph of G2, taking value 1 if the assignment is done and 0otherwise.

18 / 38

Graphs in pattern recognitionAssignment problem in graph matching

Application to CBIRConclusion

DefinitionAssignment-based Matching

Graph comparison through assignment problem: state ofthe art

Node signature Distance

[Gold 1996] Node degree+Label *

[Shokoufandeh 2006] Eigen vector L2

[Riesen 2009] (1)Node+(2)Edge Edit cost

[Jouili 2009] Node degree+Weight L1

[Raveaux 2010] Sub Graph Edit Distance*: Depends on the graph attribute type.

19 / 38

Graphs in pattern recognitionAssignment problem in graph matching

Application to CBIRConclusion

DefinitionAssignment-based Matching

Graph comparison through assignment problem: anexample

A subgraph (sg):

A structure gathering theedges and theircorresponding endingvertices from a rootvertex.

Romain Raveaux et al. Agraph matching methodand a graph matchingdistance based onsubgraph assignments.Pattern RecognitionLetters (2010)

Figure: Decomposition intosubgraph world

20 / 38

Graphs in pattern recognitionAssignment problem in graph matching

Application to CBIRConclusion

DefinitionAssignment-based Matching

Graph comparison through assignment problem: anexample

A subgraph (sg):

A structure gathering theedges and theircorresponding endingvertices from a rootvertex.

Romain Raveaux et al. Agraph matching methodand a graph matchingdistance based onsubgraph assignments.Pattern RecognitionLetters (2010)

Figure: Decomposition intosubgraph world

20 / 38

Graphs in pattern recognitionAssignment problem in graph matching

Application to CBIRConclusion

DefinitionAssignment-based Matching

Graph comparison through assignment problem: anexample II

A distance between graph

Subgraph decomposition

Optimization algorithm

Figure: Subgraph matching: A bipartite graph

21 / 38

Graphs in pattern recognitionAssignment problem in graph matching

Application to CBIRConclusion

DefinitionAssignment-based Matching

Graph comparison through assignment problem: anexample II

A distance between graph

Subgraph decomposition

Optimization algorithm

Figure: Subgraph matching: A bipartite graph

21 / 38

Graphs in pattern recognitionAssignment problem in graph matching

Application to CBIRConclusion

DefinitionAssignment-based Matching

Graph comparison through assignment problem: anexample II

A distance between graph

Subgraph decomposition

Optimization algorithm

Figure: Subgraph matching: A bipartite graph

21 / 38

Graphs in pattern recognitionAssignment problem in graph matching

Application to CBIRConclusion

DefinitionAssignment-based Matching

Synthesis

Inexact graph matching problem

Univalent matching

Polynomial algorithm : O(n3)

Two main directions

Node signature based methods (vectors).Subgraph based methods. (not a dot in a feature space.)

22 / 38

Graphs in pattern recognitionAssignment problem in graph matching

Application to CBIRConclusion

CBIR problemCBIR methodExperiments

How Graph Matching can be used in Pattern RecognitionProblems

Hypothesis :

a graph database consisting of n graphs. D = g1, g2, ..., gn

a query graph q

Problem definitions :1 Full graph search. Find all graphs gi in D s.t. gi is the same

as q.2 Subgraph search. Find all graphs gi in D containing q or

contained by q.3 Similarity search. Find all graphs gi in D s.t. gi is similar to

q within a user-specified threshold based on some similaritymeasures.

4 Graph mining. Graph mining problem gathers similar graphor subgraph of D in order to find clusters or prototypes.

23 / 38

Graphs in pattern recognitionAssignment problem in graph matching

Application to CBIRConclusion

CBIR problemCBIR methodExperiments

CBIR problem

Figure: Feature clustering stage

24 / 38

Graphs in pattern recognitionAssignment problem in graph matching

Application to CBIRConclusion

CBIR problemCBIR methodExperiments

CBIR scheme

Figure: Feature clustering stage

25 / 38

Graphs in pattern recognitionAssignment problem in graph matching

Application to CBIRConclusion

CBIR problemCBIR methodExperiments

CBIR scheme

Figure: Images to graphs

26 / 38

Graphs in pattern recognitionAssignment problem in graph matching

Application to CBIRConclusion

CBIR problemCBIR methodExperiments

CBIR scheme

Figure: Similarity search 27 / 38

Graphs in pattern recognitionAssignment problem in graph matching

Application to CBIRConclusion

CBIR problemCBIR methodExperiments

(a) (b)

Figure: Regions of Interest found by the SIFT algorithm. ProcessingSIFT took 407ms, 60 features were identified and processed

28 / 38

Graphs in pattern recognitionAssignment problem in graph matching

Application to CBIRConclusion

CBIR problemCBIR methodExperiments

Figure: A segmentation result. Processing SRM took 1625ms and 26features were identified and processed

29 / 38

Graphs in pattern recognitionAssignment problem in graph matching

Application to CBIRConclusion

CBIR problemCBIR methodExperiments

Figure: Multiple representations 30 / 38

Graphs in pattern recognitionAssignment problem in graph matching

Application to CBIRConclusion

CBIR problemCBIR methodExperiments

Figure: CBIR approach categorization according to the amount of spatialinformation captured

31 / 38

Graphs in pattern recognitionAssignment problem in graph matching

Application to CBIRConclusion

CBIR problemCBIR methodExperiments

Figure: CBIR taxonomy

32 / 38

Graphs in pattern recognitionAssignment problem in graph matching

Application to CBIRConclusion

CBIR problemCBIR methodExperiments

Figure: Image Samples from the Caltech-101 Data set. The 101 objectcategories and the background clutter category.

33 / 38

Graphs in pattern recognitionAssignment problem in graph matching

Application to CBIRConclusion

CBIR problemCBIR methodExperiments

Figure: Columbia University Image Library

34 / 38

Graphs in pattern recognitionAssignment problem in graph matching

Application to CBIRConclusion

CBIR problemCBIR methodExperiments

Recognition rate

(a) Coil-100 (b) Caltech-101

Figure: Comparison between CBIR methods. Summary of resultsobtained with the best number of words for each method.

Romain Raveaux et al: Structured representations in a content based image retrieval context. J. Visual

Communication and Image Representation (2013). Elsevier.

35 / 38

Graphs in pattern recognitionAssignment problem in graph matching

Application to CBIRConclusion

CBIR problemCBIR methodExperiments

Vocabulary size impact

(a) Coil-100 (b) Caltech-101

Figure: Comparison between the number of words in used by themethods.

Romain Raveaux et al: Structured representations in a content based image retrieval context. J. Visual

Communication and Image Representation (2013). Elsevier.

36 / 38

Graphs in pattern recognitionAssignment problem in graph matching

Application to CBIRConclusion

CBIR problemCBIR methodExperiments

Time experiments

Average response time in seconds SIFTBoW IFFSBoW IFFSGBR

Coil − 100 28 37 1555

Table: Average response time of a query. A speed comparison of the fourmethods at the worst case.

37 / 38

Graphs in pattern recognitionAssignment problem in graph matching

Application to CBIRConclusion

Conclusion

To take the stock of :

SIFTBoW is better on Caltech data

IFFS is better on Coil-100

The Graph Based Approach gave similar recognition rate thanBoW methods.

A structural approach requires a fewer number of words toreach its best performance.

Future work

To avoid visual features clustering.

To enrich the graph representation.

Combination of BoW and GbR : BoW first and then to processa re-ranking stage with the top k responses based on GbR.

38 / 38

Graphs in pattern recognitionAssignment problem in graph matching

Application to CBIRConclusion

R.C. Wilson A.D.J. Cross and E.R. Hancock.Inexact Graph Matching Using Genetic Search.Pattern Recognition, pages 953–970, 1997.

Josep Llados Anjan Dutta and Umapada Pal.A symbol spotting approach in graphical documents byhashing serialized graphs.Pattern Recognition, pages 752–768, 2012.

Horst Bunke and Kim Shearer.A graph distance metric based on the maximal commonsubgraph.Pattern Recognition Letters, vol. 2, page 255–259, 1998.

Horst Bunke.Error Correcting Graph Matching: On the Influence of theUnderlying Cost Function.

38 / 38

Graphs in pattern recognitionAssignment problem in graph matching

Application to CBIRConclusion

IEEE Trans. Pattern Anal. Mach. Intell., vol. 21, no. 9, pages917–922, 1999.

Pierre-antoine Champin and Christine Solnon.Measuring the similarity of labeled graphs.

Miquel Ferrer, Francesc Serratosa and Alberto Sanfeliu.Synthesis of Median Spectral Graph.In IbPRIA (2), pages 139–146, 2005.

et al. Finch Wilson.An energy function and continuous edit process for graphmatching.Neural Computat, vol. 10, 1998.

G.P. Ford and J. Zhang.A Structural Graph Matching Approach to ImageUnderstanding.

38 / 38

Graphs in pattern recognitionAssignment problem in graph matching

Application to CBIRConclusion

Intelligent Robots and Computer Vision X: Algorithms andTechniques, pages 559–569, 1991.

S. Gold and A Rangarajan.A graduated assignment algorithm for graph matching.Workshops SSPR and SPR, pages 377 – 388, 1996.

E. Marti J. Llados and J. J. Villanueva.Symbol recognition by error-tolerant sub- graph matchingbetween region adjacency graphs.Patt. Anal. Mach., pages 1137–1143, 2001.

X. Jiang and H. Bunke.Marked subgraph isomorphism of ordered graphs.Workshops SSPR and SPR, pages 122–131, 1998.

Salim Jouili and Salvatore Tabbone.Graph Matching Based on Node Signatures.In GbRPR, pages 154–163, 2009.

38 / 38

Graphs in pattern recognitionAssignment problem in graph matching

Application to CBIRConclusion

Hero A Justice D.A binary linear programming formulation of the graph editdistance.IEEE Trans Pattern Anal Mach Intell., vol. 28, page1200–1214, 2006.

P Kuner and B Ueberreiter.Pattern Recognition by Graph Matching: Combinatorial versusContinuous Optimization.International Journal in Pattern Recognition and ArtificialIntelligence, vol. 2, page 527–542, 1988.

P. LeBodic and H. Heroux.An integer linear program for substitution-tolerant subgraphisomorphism and its use for symbol spotting in technicaldrawings.Pattern Recognition, pages 4214–4224, 2012.

38 / 38

Graphs in pattern recognitionAssignment problem in graph matching

Application to CBIRConclusion

B. Messmer and Bunke.Error-Correcting Graph Isomorphism using Decision Trees.Int. Journal of Pattern Recognition and Ar, vol. 12, page721–742, 1998.

Bunke H Neuhaus M.A convolution edit kernel for error-tolerant graph matching.Proceedings of IEEE international conference on patternrecognition., vol. 30, pages 220–223, 2006.

Burie J. Raveaux R. and Ogier.A graph matching method and a graph matching distancebased on subgraph assignments.Pattern Recognition Letters, page 394–406, 2010.

Kaspar Riesen and Horst Bunke.Approximate graph edit distance computation by means ofbipartite graph matching.

38 / 38

Graphs in pattern recognitionAssignment problem in graph matching

Application to CBIRConclusion

Image Vision Comput., vol. 27, no. 7, pages 950–959, 2009.

Bretzner L. Macrini D. Fatih Demirci M. Jonsson C.Shokoufandeh A. and S. Dickinson.The representation and matching of categorical shape.Computer Vision and Image Understanding, page 139–154,2006.

Christine Solnon.Reactive Tabu Search for Measuring Graph.pages 172–182, 2005.

P. N. Suganthan.Attributed relational graph matching by neural-gas networks.IEEE Signal Processing Society Workshop on Neural Networksfor Signal Processing X, pages 366–374, 2000.

S. Umeyama.

38 / 38

Graphs in pattern recognitionAssignment problem in graph matching

Application to CBIRConclusion

An eigendecomposition approach to weighted graph matchingproblems.IEEE Pattern Anal. Mach. Intel, page 695–703, 1988.

J. Kittler W. Christmas and M. Petrou.Structural matching in computer vision using probabilisticrelaxation.IEEE Trans. PAMI,, vol. 2, page 749–764, 1995.

38 / 38