Algorithms and graphsDoctoral course “Optimization on graphs” - Lecture 1.2

Giovanni Righini

January 14th , 2013

Outline

• Part I: Algorithms

• Part II: Graphs: terminology

• Part III: Graph search algorithms

Part I: Algorithms

Algorithms

Algorithms are mathematical entities, just like numbers, sets,functions, geometrical figures, etc.

We can define algorithms and we can prove theorems on them, aswe do with any other mathematical entity.

There is a main difference: algorithms “work”, they evolve, theytransform some input in some output; they are dynamic mathematicalentities.

Algorithms are designed by us to compute something, to solve someproblem of interest.

Theorems about algorithms mainly concern their correctness andtheir complexity.

Optimization problems and algorithms

Algorithms have been studied since many thousands years ago.

The term “algorithm” itself comes from Muhammad ibn Musaal-Khwarizmi (Corasmia 780, Baghdad 850 a.d.), who wrote “al-Kitabal-mukhtasar fi hisab al-jabr wa al-muqabala” (820 a.d.) on how tosolve equations of order 1 and 2, for which he is considered one ofthe fathers of algebra.

However, we are especially interested in algorithms to solveoptimization problems (such as linear programs).

The systematic study and design of optimization algorithms is muchmore recent: Mathematical Programming is a branch of OperationsResearch.

Correctness

A proof of correctness for an algorithm states that for any giveninstance in input, the algorithm will produce an output satisfying givenproperties, that can be formally described in mathematical terms.

For optimization algorithms, a proof of correctness means that for anyparticular instance of a given optimization problem, the algorithm willterminate in a finite number of steps and it will output an optimalsolution of the problem, if it exists.

If no optimal solution exists, the algorithm must terminate anyway!

Complexity

The computational complexity of an algorithm tells how much itrequires to perform its computation in terms of computationalresources, i.e. time (elementary operations to be executed) andmemory space (amount of information that needs to be stored atsome point in time for later use).

The evaluation is done asymptotically, i.e. we do not ask “How muchit consumes for solving this instance”, but rather “How the amount ofnecessary resources scales with the size of the instance?”

The complexity can be studied in the worst case or in the averagecase.

• Average-case complexity requires the use of probabilitydistributions on the input data; it is sometimes used for studyingand designing non-deterministic algorithms.

• Worst-case complexity is more common to study, design andcompare deterministic algorithms.

Problem complexity

Worst-case complexity is represented with the O( ) notation, as afunction of the size of the instance.

In case of graph optimization problems, the size of the instance isgiven by the number of nodes (n) and arcs (m) in the graph.

The complexity of a problem is at most the complexity of the bestalgorithm we know for that problem.

Computational complexity theory classifies problems in many classesand sub-classes. The most important separation, however, isbetween problems for which we know a polynomial-time algorithmand those for which we do not.

The “same” algorithm may have different complexity according to thedifferent data-structures that are used to implement its basicoperations.

Part II: Graphs

Definitions - 1

A graph, indicated as G = (V , E), is defined by• a set of vertices V ;• a set of edges E .

The vertex set V is an elementary set, i.e. its elements are justatomic items.The edge set E is a complex set, i.e. its elements are sets: each edgeis a pair of vertices in V .

E ⊆ {[i, j] : i ∈ V , j ∈ V , i 6= j}.

A digraph (directed graph), indicated as D = (N ,A), is defined by• a set of nodes N ;• a set of arcs A.

Each arc is an ordered pair of nodes in N .

A ⊆ {(i, j) : i ∈ N , j ∈ N , i 6= j}.

We exclude self-loops, i.e. edges (arcs) whose endpoints coincide.

Definitions - 2

A subgraph G′ induced by a subset V ′ of vertices is G′ = (V ′, E ′) with

E ′ = {[i, j] ∈ E : i ∈ V ′, j ∈ V ′.}

An analogous definition applies to di-graphs.

We will not consider multi-(di-)graphs and hyper-graphs.

A multi-(di-)graph is a (di-)graph whose edge or arc set is a multi-set,i.e. it may contain multiple copies of the same element.

An hyper-graph is a graph whose edges are subsets of vertices, notnecessarily pairs.

Definitions - 3

If [i, j] ∈ E , then:

• vertices i and j are adjacent,

• edge [i, j] is incident to vertex i and to vertex j.

If (u, v) ∈ A, then:

• arc (u, v) is incident to node u and to node v ,

• arc (u, v) leaves node u,

• arc (u, v) enters node v ,

• node u is a predecessor of node v ,

• node v is a successor of node u.

The degree of a vertex i ∈ V is the n. of edges e ∈ E incident to it.The in-degree of a node i ∈ N is the n. of arcs a ∈ A entering it.The out-degree of a node i ∈ N is the n. of arcs a ∈ A leaving it.

Weights and objectives

A graph is weighted when there is a function associating a weightwith each edge, i.e. c : E 7→ ℜ, or vertex, i.e. c : V 7→ ℜ.

The same definition applies to di-graphs as well.

Weights often represent costs and the objective to be optimized(minimized) is the overall cost of a subset of edges or arcs,representing the solution: so we search for minimum cost paths,minimum cost trees, minimum cost flows, etc.

Combinatorial structures

When we solve combinatorial optimization problems on graphs, it isusually because we want to find the best among solutions with aparticular structure:

• paths, representing origin to destination routes on street graphs;

• trees, representing links between geographically dispersed sitesin telecommunication networks;

• flows, representing amounts of freight, passengers, goods,money... moving from one site to another;

• matchings, representing pairings in graphs of relations betweenpeople, activities, attributes,...;

• and many others...

The solutions correspond to subsets of edges or arcs.

Complexity - 1

The problem of finding the optimal edge (arc) subset with a givenstructure is combinatorial when the number of solutions iscombinatorial in the number of edges or arcs, i.e. there as manysolutions as the possible combinations of edges and arcs.

Due to the combinatorial explosion in the number of solutions, it isimpractical to enumerate all of them explicitly: hence we needsuitable (efficient) graph optimization algorithms.

We are interested in optimization algorithms that scale well with thesize of the graph, i.e. algorithms that compute an optimal solutiontaking polynomial time and space in the size of the instance.

Complexity - 2

In the case of graph optimization problems the size of the instance isthe size of the graph.

We indicate with n the number of vertices or nodes.We indicate with m the number of edges or arcs.The computational complexity of graph optimization algorithms isusually given as a function of n and m.

We do not know polynomial complexity algorithms for allcombinatorial optimization problems on graphs, but for some of themwe do.

It is very important to know these well-solved cases, because theyoften occur as sub-problems within larger and more complicatedoptimization problems.

Complexity - 3

The maximum number of edges a graph can have is

mmax =n(n − 1)

2.

The maximum number of arcs a digraph can have is

mmax = n(n − 1).

A (di-)graph is complete if and only if it contains mmax edges or arcs.A (di-)graph is dense (sparse) when m

mmax is large (small).

The density/sparsity of a graph can affect the computing time ofgraph optimization algorithms, according to the data-structures usedto represent the graph.

Data-structures

There are many possibilities to store the information corresponding toa (di-)graph in a computer memory, i.e. in a data-structure.

The choice of the most suitable data-structure depends:

• on the efficiency of the operations we need to execute on it;

• on the density/sparsity of the graph.

The most used data-structures to represent graphs are:

• adjacency matrix,

• incidence matrix,

• edge (arc) list,

• (in/out-)stars.

Adjacency matrix

An adjacency matrix M is a square n × n matrix, whose rows andcolumns correspond to vertices/nodes. Each entry M[i, j] contains thepiece of information associated with edge [i, j] or arc (i, j).

For instance:M[i, j] = 0 when [i, j] 6∈ E and M[i, j] = 1 when [i, j] ∈ E ;M[i, j] = ∞ when [i, j] 6∈ E and M[i, j] = cij when [i, j] ∈ E .

An arc set requires the whole matrix; an edge set requires half of it.

1 2

34

22

3

15 207

6

8

18

Figure: A digraph.

∞ 22 3 ∞∞ ∞ 15 7∞ 20 ∞ 68 ∞ 18 ∞

Table: Its adjacency matrix.

Incidence matrix

An incidence matrix M is an n × m matrix, whose rows correspond tovertices/nodes and whose columns correspond to edges/arcs.

Each entry M[i, e] contains a significant piece of information if andonly if edge e is incident in vertex i.The same applies to arcs and nodes, with the sign indicating thedirection.

1 2

34

22

3

15 207

6

8

18

Figure: A digraph.

(1, 2) (1, 3) (2, 3) (2, 4) (3, 2) (3, 4) (4, 1) (4, 3)1 -22 -3 82 22 -15 -7 203 3 15 -20 -6 184 7 6 -8 -18

Table: Its incidence matrix.

Edge (arc) list

An edge (arc) list L is a list of all the edges (arcs) in the (di-)graph.

Each element in the list is a record with all relevant information aboutthe edge (arc).

1 2

34

22

3

15 207

6

8

18

Figure: A digraph.

Node Node Cost1 2 221 3 32 3 153 2 202 4 73 4 64 1 84 3 18

Table: Its arc list.

(In/Out-)stars

A star S is a list of all edges incident in a vertex.An in-star I is a list of all arcs entering a node.An out-star O is a list of all arcs leaving a node.

Each element in the list is a record with all relevant information aboutthe edge (arc).

The whole (di-)graph is represented by a list of all (in/out-)stars, for allits vertices (or nodes).

1 2

34

22

3

15 207

6

8

18

Figure: A digraph.

Node Successors and weight1 2,22 3, 32 3,15 4, 73 2,20 4, 64 1, 8 3,18

Table: Its out-stars.

Part III: Graph search algorithms

Breadth-first search

Given:

• a graph G = (V , E)

• a vertex s ∈ V ,

we indicate with Vk the set of all vertices that

• are reachable from s along a path made of k edges;

• and not reachable from s along any path with less than k edges.

Recursive definition:

• V0 = {s}

• Vk+1 = {v ∈ V\⋃k

i=0 Vi : ∃u ∈ Vk ∧ ∃[u, v ] ∈ E}.

Analogous definitions hold for digraphs.

Breadth-first search

To compute Vk+1 it is enough to scan the set of all edges (arcs)incident to (leaving) the vertices (nodes) in Vk e to insert thesevertices (nodes) into Vk+1, if they have not been reached before. Abinary flag associated with each vertex (node) is enough to checkthis.

The complexity of this algorithm is O(m), because each edge (arc) isscanned at most twice (once).

This BFS algorithm determines the shortest path from s to any othervertex (node) of the (di-)graph in the special case of unit weightedges (arcs).

Pseudo-code

Breadth-First Search (Berge 1958, Moore 1959):

beginfor v :=1 to n do flag[v ]:=0; flag[s]:=1;k := 0; Vk := {s};while Vk 6= ∅ do

Vk+1 := ∅;for u ∈ Vk do

for [u, v ] ∈ δ(u) doif (flag[v ]=0) then

Vk+1 := Vk+1 ∪ {v};flag[v ] := 1;

k := k + 1;end.

The vertices (nodes) non reached when the algorithm terminates donot belong to the same connected component of s.

Depth-First Search (Tarry 1895)

Given:

• a digraph D = (N ,A)

• a node s ∈ N ,

we define Scan(s) the following recursive procedure:

for (s, v) ∈ δ+(s) dofor (u, v) ∈ δ−(v) : u 6= s do

delete (u, v);Scan(v);

If all nodes in N are reachable from s, the arcs not deleted byScan(s) form an arborescence rooted in s and spanning them.

(Pre-)topological order

The nodes of a digraph are sorted in topological order if

i < j ∀(vi , vj) ∈ A.

Hence a subset N ′ of nodes can be sorted in topological order only ifthe induced subgraph (N ′,A(N ′)) is acyclic (i.e. it does not containcircuits).

The nodes of a digraph are sorted in pre-topological order if thefollowing condition holds:

vi ≺ vj ⇒ i < j

dove vi ≺ vj means that j is reachable from i but i is not reachablefrom j.

If the digraph is acyclic, then any pre-topological order is alsotopological.

Pre-topological order

Theorem. Given a di-graph D = (N ,A) and a node s ∈ N , the nodesin N reachable from s can be sorted in pre-topological order inO(m′), where m′ is the number of arcs reachable from s.

Proof. In the execution of Scan(s) all nodes reachable from s arescanned. The order in which their Scan() procedure terminates is thereverse of their pre-topological order. For each pair of nodes u and vreachable from s, if there is a path from u to v but not from v to u,then Scan(v ) terminates before Scan(u).

Corollary 1. The nodes of a digraph D(N ,A) can be sorted inpre-topological order in linear time.

Proof. Insert a dummy node s into the digraph together with arcs(s, v) ∀v ∈ N and then apply the previous theorem.

Corollary 2. The nodes of an acyclic digraph D(N ,A) can be sortedin topological order in linear time.

Example

A D C

G

E

L M

H

F

B

Scan End Order 10 9 8 7 6 5 4 3 2 1

Example

A D C

G

E

L M

H

F

B

Scan A End Order 10 9 8 7 6 5 4 3 2 1

Example

A D C

G

E

L M

H

F

B

Scan A B End Order 10 9 8 7 6 5 4 3 2 1

Example

A D C

G

E

L M

H

F

B

Scan A B F End Order 10 9 8 7 6 5 4 3 2 1

Example

A D C

G

E

L M

H

F

B

Scan A B F H End Order 10 9 8 7 6 5 4 3 2 1

Example

A D C

G

E

L M

H

F

B

Scan A B F H G End Order 10 9 8 7 6 5 4 3 2 1

Example

A D C

G

E

L M

H

F

B

Scan A B F H G E End Order 10 9 8 7 6 5 4 3 2 1

Example

A D C

G

E

L M

H

F

B

Scan A B F H G E End E Order 10 9 8 7 6 5 4 3 2 1

Example

A D C

G

E

L M

H

F

B

Scan A B F H G E End E G Order 10 9 8 7 6 5 4 3 2 1

Example

A D C

G

E

L M

H

F

B

Scan A B F H G E End E G H Order 10 9 8 7 6 5 4 3 2 1

Example

A D C

G

E

L M

H

F

B

Scan A B F H G E End E G H F Order 10 9 8 7 6 5 4 3 2 1

Example

A D C

G

E

L M

H

F

B

Scan A B F H G E M End E G H F Order 10 9 8 7 6 5 4 3 2 1

Example

A D C

G

E

L M

H

F

B

Scan A B F H G E M L End E G H F Order 10 9 8 7 6 5 4 3 2 1

Example

A D C

G

E

L M

H

F

B

Scan A B F H G E M L End E G H F L Order 10 9 8 7 6 5 4 3 2 1

Example

A D C

G

E

L M

H

F

B

Scan A B F H G E M L End E G H F L M Order 10 9 8 7 6 5 4 3 2 1

Example

A D C

G

E

L M

H

F

B

Scan A B F H G E M L End E G H F L M B Order 10 9 8 7 6 5 4 3 2 1

Example

A D C

G

E

L M

H

F

B

Scan A B F H G E M L End E G H F L M B A Order 10 9 8 7 6 5 4 3 2 1

Example

A D C

G

E

L M

H

F

B

Scan A B F H G E M L C End E G H F L M B A Order 10 9 8 7 6 5 4 3 2 1

Example

A D C

G

E

L M

H

F

B

Scan A B F H G E M L C D End E G H F L M B A Order 10 9 8 7 6 5 4 3 2 1

Example

A D C

G

E

L M

H

F

B

Scan A B F H G E M L C D End E G H F L M B A D Order 10 9 8 7 6 5 4 3 2 1

Example

A D C

G

E

L M

H

F

B

Scan A B F H G E M L C D End E G H F L M B A D C Order 10 9 8 7 6 5 4 3 2 1

Example

3 2 1

9

10

6 5

8

7

4

Scan A B F H G E M L C D End E G H F L M B A D C Order 10 9 8 7 6 5 4 3 2 1

Strongly connected components

Theorem (Kosaraju e Sharir, 1981). Given a digraph D = (N ,A) itsstrongly connected components can be computed in linear time.

Proof. Sort the nodes in pre-topological order: v1, v2, . . . , vn. Let N1

be the set of nodes from which v1 is reachable. Then N1 is thestrongly connected component v1 belongs to: each vj ∈ N1 isreachable from v1 for the pre-topological order properties.For the previous theorem N1 can be computed in O(|A1|) time, whereA1 is the set of arcs with their head in N1.Deleting all nodes in N1 and the arcs in A1 another digraph isobtained whose nodes are sorted in pre-topological order in the samesequence as before. Therefore, by repeatedly applying theprocedure, all strongly connected components (s.c.c.) are obtained.

Corollary (Shirey, 1969). The connected components of G = (V , E)can be computed in linear time.