Chapter 11Chapter 11

Limitations of Limitations of Algorithm PowerAlgorithm Power

Lower BoundsLower Bounds

Lower boundLower bound: an estimate on a minimum amount of work : an estimate on a minimum amount of work needed to solve a given problemneeded to solve a given problem

Examples:Examples: number of comparisons needed to find the largest element number of comparisons needed to find the largest element

in a set of in a set of nn numbers numbers number of comparisons needed to sort an array of size number of comparisons needed to sort an array of size nn number of comparisons necessary for searching in a sorted number of comparisons necessary for searching in a sorted

arrayarray number of multiplications needed to multiply two number of multiplications needed to multiply two nn-by--by-n n

matrices matrices

Lower Bounds (cont.)Lower Bounds (cont.)

Lower bound can beLower bound can be• an exact count an exact count • an efficiency class (an efficiency class ())

TightTight lower bound: there exists an algorithm with the same lower bound: there exists an algorithm with the same efficiency as the lower boundefficiency as the lower bound

ProblemProblem Lower bound Lower bound TightnessTightness sortingsorting ((nnloglog n n) yes) yes searching in a sorted arraysearching in a sorted array (log(log n n) yes) yes element uniqueness element uniqueness ((nnloglog n n) yes) yes nn-digit integer multiplication -digit integer multiplication ((nn) unknown) unknown multiplication of multiplication of nn-by--by-n n matrices matrices ((nn22) unknown) unknown

Methods for Establishing Lower BoundsMethods for Establishing Lower Bounds

trivial lower boundstrivial lower bounds

information-theoretic arguments (decision trees)information-theoretic arguments (decision trees)

adversary argumentsadversary arguments

problem reductionproblem reduction

Trivial Lower BoundsTrivial Lower Bounds

Trivial lower boundsTrivial lower bounds: based on counting the number of items that : based on counting the number of items that must be processed in input and generated as outputmust be processed in input and generated as output

ExamplesExamples finding max elementfinding max element

polynomial evaluationpolynomial evaluation

sortingsorting

element uniquenesselement uniqueness

Hamiltonian circuit existenceHamiltonian circuit existence

Conclusions Conclusions may and may not be usefulmay and may not be useful

be careful in deciding how many elements be careful in deciding how many elements mustmust be processed be processed

Decision TreesDecision Trees

Decision treeDecision tree — a convenient model of algorithms involving — a convenient model of algorithms involving comparisons in which:comparisons in which: internal nodes represent comparisonsinternal nodes represent comparisons leaves represent outcomes leaves represent outcomes

Decision tree for 3-element insertion sortDecision tree for 3-element insertion sort

a < b

b < c a < cyes

yes no

noyesno

a < c b < c

a < b < c

c < a < b

b < a < c

b < c < a

no yes

abc

abc bac

bcaacb

yes

a < c < b c < b < a

no

Decision Trees and Sorting AlgorithmsDecision Trees and Sorting Algorithms

Any comparison-based sorting algorithm can be represented Any comparison-based sorting algorithm can be represented by a decision treeby a decision tree

Number of leaves (outcomes) Number of leaves (outcomes) nn!!

Height of binary tree with Height of binary tree with nn! leaves ! leaves loglog22nn!!

Minimum number of comparisons in the worst case Minimum number of comparisons in the worst case loglog22nn!! for any comparison-based sorting algorithmfor any comparison-based sorting algorithm

loglog22nn!! n n loglog22nn

This lower bound is tight (mergesort)This lower bound is tight (mergesort)

Adversary ArgumentsAdversary Arguments

Adversary argumentAdversary argument: a method of proving a lower bound by : a method of proving a lower bound by playing role of adversary that makes algorithm work the hardestplaying role of adversary that makes algorithm work the hardestby adjusting input by adjusting input

Example 1: “Guessing” a number between 1 and Example 1: “Guessing” a number between 1 and n n with yes/nowith yes/no questionsquestionsAdversary: Puts the number in a larger of the two subsetsAdversary: Puts the number in a larger of the two subsets

generated by last question generated by last question

Example 2: Merging two sorted lists of size Example 2: Merging two sorted lists of size nn

aa11 < < aa22 < … < < … < aann and and bb11 < < bb22 < … < < … < bbnn

Adversary: Adversary: aai i < b< bjj iff iff ii < < jj

Output Output bb11 < < aa11 < < bb22 < < aa22 < … < < … < bbnn < < aann requires 2 requires 2nn-1 comparisons-1 comparisons

of adjacent elementsof adjacent elements

Lower Bounds by Problem ReductionLower Bounds by Problem Reduction

Idea: If problem Idea: If problem P P is at least as hard as problem is at least as hard as problem QQ, then a lower, then a lower

bound for bound for Q Q is also a lower bound for is also a lower bound for P. P.

Hence, find problem Hence, find problem Q Q with a known lower boundwith a known lower bound that canthat can be reduced to problem be reduced to problem PP in question. in question.

Example: Example: PP is finding MST for is finding MST for nn points in Cartesian plane points in Cartesian plane Q Q is element uniqueness problem (known to be in is element uniqueness problem (known to be in ((nnloglognn))))

11.3 P, NP, and NP-complete Problems11.3 P, NP, and NP-complete Problems

An algorithm solves a problem in polynomial time if its An algorithm solves a problem in polynomial time if its worst-case time efficiency belongs to O(p(n)) worst-case time efficiency belongs to O(p(n)) • Where p(n) is a polynomial of the problem’s input size n Where p(n) is a polynomial of the problem’s input size n

Problems that can be solved in polynomial time are called Problems that can be solved in polynomial time are called tractabletractable

Problems that cannot be solved in polynomial time are Problems that cannot be solved in polynomial time are called called intractableintractable..• Cannot solve intractable problems in a reasonable length of timeCannot solve intractable problems in a reasonable length of time

25 city Traveling Salesperson Problem25 city Traveling Salesperson Problem

There are 25! different possible paths to be considered. There are 25! different possible paths to be considered. • That is approximately 1.5 x 10That is approximately 1.5 x 102525 different paths. different paths.

Suppose the computer can analyze 10,000,000, or 10Suppose the computer can analyze 10,000,000, or 1077, paths , paths per second. per second. • The number of seconds required to check all possible paths is about The number of seconds required to check all possible paths is about

1.5 x 101.5 x 102525/10/1077, or about 1.5 x 10, or about 1.5 x 101818 seconds. seconds.

• That’s roughly 10That’s roughly 101212 years: about a trillion years. years: about a trillion years.

This would not be a feasible algorithm.This would not be a feasible algorithm.

The Halting ProblemThe Halting Problem

Turing – 1936Turing – 1936 Given a computer program and an input to it, determine whether the program will halt on Given a computer program and an input to it, determine whether the program will halt on

that input or continue working indefinitely on it.that input or continue working indefinitely on it.

Assume that A is an algorithm that solves the halting problem: A(P, I) = 1 if P halts on Assume that A is an algorithm that solves the halting problem: A(P, I) = 1 if P halts on input I, 0 otherwiseinput I, 0 otherwise

Consider P as an input to itself and use the output of A for pair (P,P) to construct a Consider P as an input to itself and use the output of A for pair (P,P) to construct a program Q as follows:program Q as follows:

Q(P) halts if A(P,P) = 0 (if program P does not halt on input P)Q(P) halts if A(P,P) = 0 (if program P does not halt on input P) Q(P) does not halt if A(P, P) = 1Q(P) does not halt if A(P, P) = 1

Then substituting Q for P we obtainThen substituting Q for P we obtain Q(Q) halts if A(Q, Q) = 0 i.e. if program Q does not halt on input QQ(Q) halts if A(Q, Q) = 0 i.e. if program Q does not halt on input Q Q(Q) does not halt if A(Q, Q) = 1 i.e. if program Q halts on input QQ(Q) does not halt if A(Q, Q) = 1 i.e. if program Q halts on input Q

This is a contradiction because neither of the two outcomes for program Q is possible. This is a contradiction because neither of the two outcomes for program Q is possible. QEDQED

Classifying Problem ComplexityClassifying Problem Complexity

Is the problemIs the problem tractabletractable,, i.e., is there a polynomial-time (O(i.e., is there a polynomial-time (O(pp((nn)) )) algorithm that solves it?algorithm that solves it?

Possible answers:Possible answers: yes (give examples)yes (give examples)

nono

• because it’s been proved that no algorithm exists at all because it’s been proved that no algorithm exists at all (e.g., Turing’s (e.g., Turing’s halting problemhalting problem))

• because it’s been be proved that any algorithm takes because it’s been be proved that any algorithm takes exponential timeexponential time

unknownunknown

Problem Types: Optimization and DecisionProblem Types: Optimization and Decision

Optimization problemOptimization problem: find a solution that maximizes or : find a solution that maximizes or minimizes some objective functionminimizes some objective function

Decision problemDecision problem:: answer yes/no to a question answer yes/no to a question

Many problems have decision and optimization versions.Many problems have decision and optimization versions.

E.g.: traveling salesman problemE.g.: traveling salesman problem optimizationoptimization: find Hamiltonian cycle of minimum length: find Hamiltonian cycle of minimum length decisiondecision: find Hamiltonian cycle of length : find Hamiltonian cycle of length mm

Decision problems are more convenient for formal investigation of Decision problems are more convenient for formal investigation of their complexity.their complexity.

Class Class PP

PP: the class of decision problems that are solvable in O(: the class of decision problems that are solvable in O(pp((nn)) time, )) time, where where pp((nn) is a polynomial of problem’s input size ) is a polynomial of problem’s input size nn

Examples:Examples: searchingsearching

element uniquenesselement uniqueness

graph connectivity graph connectivity

graph acyclicitygraph acyclicity

primality testing (finally proved in 2002)primality testing (finally proved in 2002)

Class Class NPNP

NPNP ( (nondeterministic polynomialnondeterministic polynomial): class of decision problems whose ): class of decision problems whose proposed solutions can be verified in polynomial time = solvable proposed solutions can be verified in polynomial time = solvable by a by a nondeterministicnondeterministic polynomial algorithmpolynomial algorithm

A A nondeterministic polynomial algorithmnondeterministic polynomial algorithm is an abstract two-stage is an abstract two-stage procedure that:procedure that:

generates a random string purported to solve the problemgenerates a random string purported to solve the problem checks whether this solution is correct in polynomial timechecks whether this solution is correct in polynomial timeBy definition, it solves the problem if it’s capable of generating and By definition, it solves the problem if it’s capable of generating and

verifying a solution on one of its tries verifying a solution on one of its tries

Why this definition?Why this definition? led to development of the rich theory called “computational led to development of the rich theory called “computational

complexity”complexity”

Example: CNF satisfiabilityExample: CNF satisfiability

Problem: Is a boolean expression in its conjunctive normal Problem: Is a boolean expression in its conjunctive normal form (CNF) satisfiable, i.e., are there values of its form (CNF) satisfiable, i.e., are there values of its variables that makes it true?variables that makes it true?

This problem is in This problem is in NPNP. Nondeterministic algorithm:. Nondeterministic algorithm: Guess truth assignmentGuess truth assignment Substitute the values into the CNF formula to see if it Substitute the values into the CNF formula to see if it

evaluates to trueevaluates to true

Example: Example: ((A | A | ¬B ¬B | | ¬C¬C)) & & ((AA | | BB)) & & ((¬¬B | B | ¬D |¬D | EE)) && ((¬D | ¬E¬D | ¬E))Truth assignments:Truth assignments:

A B C D EA B C D E0 0 0 0 00 0 0 0 0

. . .. . .1 1 1 1 11 1 1 1 1

Checking phase: Checking phase: O(O(nn))

What problems are in What problems are in NPNP??

Hamiltonian circuit existence Hamiltonian circuit existence Partition problem: Is it possible to partition a set of Partition problem: Is it possible to partition a set of nn

integers into two disjoint subsets with the same sum?integers into two disjoint subsets with the same sum? Decision versions of TSP, knapsack problem, graph Decision versions of TSP, knapsack problem, graph

coloring, and many other combinatorial optimization coloring, and many other combinatorial optimization problems. (Few exceptions include: MST, shortest paths)problems. (Few exceptions include: MST, shortest paths)

All the problems in All the problems in PP can also be solved in this manner (but can also be solved in this manner (but no guessing is necessary), so we have: no guessing is necessary), so we have:

PP NPNP

Big question: Big question: PP = = NP NP ??

NPNP-Complete Problems-Complete Problems

A decision problem A decision problem DD is is NPNP-complete-complete if it’s as hard as any if it’s as hard as any

problem in problem in NPNP, i.e.,, i.e., D D is in is in NPNP every problem in every problem in NPNP is polynomial-time reducible to is polynomial-time reducible to DD

Cook’s theorem (1971): CNF-sat is Cook’s theorem (1971): CNF-sat is NPNP-complete-complete

NP-completeproblem

NP problems

NPNP-Complete Problems (cont.)-Complete Problems (cont.)

Other Other NPNP-complete problems obtained through polynomial--complete problems obtained through polynomial-

time reductions from a known time reductions from a known NPNP-complete problem-complete problem

Examples: TSP, knapsack, partition, graph-coloring andExamples: TSP, knapsack, partition, graph-coloring and hundreds of other problems of combinatorial nature hundreds of other problems of combinatorial nature

knownNP-complete

problem

NP problems

candidate for NP -

completeness

P P = = NP NP ?? Dilemma RevisitedDilemma Revisited

P P = = NP NP would imply that every problem in would imply that every problem in NP, NP, including all including all NP-NP-complete problems,complete problems, could be solved in polynomial timecould be solved in polynomial time

If a polynomial-time algorithm for just one If a polynomial-time algorithm for just one NP-NP-complete complete problem is discovered, then every problem in problem is discovered, then every problem in NP NP can be solved can be solved in polynomial time, i.e., in polynomial time, i.e., P P = = NPNP

Most but not all researchers believe that Most but not all researchers believe that P P NP NP , i.e. , i.e. P P is a is a proper subset of proper subset of NPNP

NP-completeproblem

NP problems