CSE 3813Introduction to Formal Languages and Automata

Chapter 14An Introduction to Computational

Complexity

These class notes are based on material from our textbook, An Introduction to Formal Languages and Automata, 4th ed., by Peter Linz, published by Jones and Bartlett Publishers, Inc., Sudbury, MA, 2006. They are intended for classroom use only and are not a substitute for reading the textbook.

Additional notes on complexity

(Acknowledgement: These notes are derived from material developed by Dr. Susan Bridges and Dr. Lois Boggess, and from the course textbook for CS 8833.)

Algorithmics and Computational Complexity

Algorithmics is the study of the performance characteristics of specific algorithms. Computational Complexity (also called Computability Theory) looks at the characteristics of problems, rather than algorithms.

Solvable and unsolvable

We make a distinction between solvable and unsolvable problems. Within the solvable problems, we make a three-way distinction. So the subdivision looks like this:

Solvable and unsolvableUnsolvable problems

Example: The halting problem

Solvable problems

Provably intractable problems (also calledinfeasible problems)

Probably intractable problems

Tractable problems (feasible problems)

Solvable and unsolvableExample of a provably intractable (infeasible) problem:

Generalized chess - Chess is played on a board of size 8 x 8; generalized chess is played on a board of size N x N, with the set of pieces and the set of moves adjusted as needed. It has been proven to have an exponential lower bound.

Does P = NP?

There is a question that has been unanswered in spite of a lot of work in computational theory: Does P = NP?

Some related questions:What are the meanings of the terms, P, NP, NP-

complete, and NP-hard?Why do we care whether P = NP?What is the best guess as to whether P = NP?What are the consequences, if that best guess is

right?

Polynomial time algorithms

What is a polynomial time algorithm?

Traditional answer: A polynomial time algorithm is one that can be executed by a standard (deterministic) Turing machine in polynomial time, proportional to the size of the input.

Easier to understand answer: A polynomial time algorithm is one in which the worst case running time is O(nk) where n is the input size and k is a constant.

Polynomial time algorithms

Can all problems be solved in polynomial time, if we just make k large enough?

Answer: No.

It can be proved that some problems that we can describe can’t be solved at all.

Some problems can be proven to require at least exponential time.

There is an important class of problems that “probably” require exponential time.

Polynomial time algorithms

Are algorithms that are polynomial time considered efficient?

Answer: Algorithms are considered efficient if there is a polynomial p(n) such that the algorithm can solve any instance of the problem of size n in O(p(n)) time. Of course, in reality, the efficiency of a polynomial-time algorithm also depends on:

Polynomial time algorithms

• the constant of proportionality that is being ignored in the big-O formalism. If very large, the algorithm still isn’t efficient, for any practical purposes

• the size of the exponent of the polynomial. An algorithm whose exponent is 50 wouldn’t be efficient in practice

• the size of n itself. Very, very large inputs make even n2 algorithms unattractive.

What’s so great about polynomial time?

What’s so great about polynomial time, then?

A problem that is (n50) wouldn’t really be efficient, but polynomial time problems with that high an exponent are rare. Usually the exponents are small.

What’s so great about polynomial time?

For most reasonable models of computation, a problem that can be solved in polynomial time in one model can be solved in polynomial time in another.

So, for example, a problem that is polynomial in time on a serial random access machine is typically also polynomial on a Turing machine as well as on a parallel computing system, and vice versa.

What’s so great about polynomial time?

The class of polynomial time problems has some nice closure properties: polynomial-time algorithms are closed under addition (f(n) + g(n)), multiplication (f(n)*g(n)), and composition (g(f(n)).

Decision problems

What are decision problems?In the theory of NP-Completeness, we concentrate on one

particular type of problem – decision problems. We think of each instance of a given problem as mapping onto a solution space that consists of {yes, no} or {0,1}.

This may seem like a severe restriction on the type of problems that we will consider, but it is not. For example, optimization problems are fairly easy to restate as decision problems. And there exists a proof that if the decision problem is easy, then the optimization problem is easy.

Optimization problems

Example optimization problem - A shortest path problem:

Given a graph G = <V, E>, two vertices, u, v V, and a non-negative integer k, does a path exist in G between u and v whose length is k or less?

Let i = (G, u, v, k) be an instance of the problem. Then the shortest path “question” with a yes/no answer becomes Path(i) = 1 or Path(i) = 0, where 1 = yes and 0 = no.

Optimization problems

To change an optimization problem into a decision problem, we can usually just impose a bound on the value to be optimized.

Examples from path problems include minimization problems (Is the length of the shortest path at most k?) and maximization problems (Is the length of the longest path at least k?).

Optimization problems

Usually, if the optimization problem can be solved quickly, the decision problem can also be solved quickly.

Conversely, evidence that the decision problem is hard normally is evidence that the optimization problem is also hard.

Abstract problems

What are abstract problems?

An abstract problem Q is a binary relation on a set I of problem instances and a set S of problem solutions.

Encoding

What is an encoding?

An encoding of a set S of abstract objects is a mapping e from S to the set of binary strings.

Concrete problems

What is a concrete problem?

A concrete problem is a problem whose instance is the set of binary strings.

What is P?

The complexity class P is the set of concrete decision problems that are solvable in polynomial time.

Alternately, P is the class of languages that are accepted in polynomial time by a standard Turing machine.

What is P?

(The limitation to concrete, rather than abstract, decision problems is not a big limitation, because abstract decision problems are mapped into concrete problems by an encoding. Care must be taken for this step, however, because a bad encoding can make a problem much more difficult. We will ignore the difference between abstract and concrete, and assume that any encodings required were implemented reasonably well.)

The class NP

What is a verification algorithm?

A verification algorithm is a two-argument algorithm A, in which one argument is an input string x and the other is a binary string y called a certificate. The verification algorithm A verifies an input string x if there exists a certificate y such that A(x, y) = 1. The algorithm A verifies a language L if, for any string x L, there is a certificate y that A can use to prove that x L. For example, in the Hamiltonian cycle problem, the certificate is a list of vertices that comprise the Hamiltonian cycle.

The class NP

The complexity class NP is the class of languages that can be verified by a polynomial time algorithm.

Alternatively, a language L belongs to NP iff there exists a two-input polynomial-time algorithm A and a constant c such that L is a set of binary strings for which there exists a certificate y with |y| = O(|x|c) such that A(x, y) = 1.

The class NP

What does the term NP mean?

NP stands for Nondeterministic Polynomial. That is, a nondeterministic Turing machine can solve it in polynomial time, but no known deterministic Turing machine can.

The class NP

When does a problem belong to the class NP?

A problem belongs to the class NP under the following conditions:

Given that an oracle can guess an answer to the problem (provide a certificate),

then we can find an algorithm that can verify that the answer is correct in polynomial time..

The class NP

Note that the problem is NP. It is the verifying algorithm that is P.

Note also that the class of problems solvable in polynomial time (P) is a subset of NP by definition.

Example of an NP problem.

Consider the question of whether a given graph has a Hamiltonian cycle (a simple cycle that visits every vertex in the graph). This question is NP as we show below.

Given a certificate (a list of vertices that purports to be a Hamiltonian cycle in the graph):

check to see if the first and last vertex on the list are the same

check that every vertex in the graph is on the listcheck that every vertex in the list is connected by an edge of

the graph to the next vertex on the list.check to see that no vertex other than the first/last vertex is

repeated.

NP-Complete problems

What is the class of NP-Complete problems?

In some sense, the NP-complete problems are the “hardest” problems in the class NP.

No polynomial time algorithm is known to solve any of them.

Every NP-Complete problem is reducible to every other NP-complete problem in polynomial time.

NP-Complete problems

What do we mean by reducible, in the previous sentence?

A decision problem Q is polynomial-time reducible to decision problem Q' if a polynomial time algorithm can be developed which changes each instance of problem Q into an instance of problem Q' such that if the answer for an instance of Q' is yes, the answer to the corresponding instance of Q is yes. Then we write: Q

p Q'

NP-Complete problems

Lemma 1:

If Q1 and Q2 are problems such that

then Q2 P implies Q1 P.

Proof:

Suppose A1 is a polynomial time algorithm that reduces instances of Q1 to instances of Q2 and that A2 is a polynomial time algorithm for problem Q2. Then the composition of A1 and A2 is a polynomial time algorithm for Q1.

Q1 p Q2

NP-Complete problems

Definition of NP-Complete:

A problem Q is NP-Complete if :

• Q is an element of the class NP

• for every Q' in NP

Q' p Q

NP-Hard problems

Definition of NP-Hard:

• A problem Q is NP-Hard if :• for every Q' in NPQ'

p Q

NP-Complete problems

Theorem:

If any NP-complete problem is polynomial-time solvable, then P = NP. If any problem in NP is not polynomial-time solvable, then all NP-complete problems are not polynomial-time solvable

P = NP???

Proof of the first statement:

Let Q be NP-complete and polynomial. Then for all Q' in NP,

, and by a previous Lemma it follows that Q' is in P.

Q' p Q

P = NP???

Proof of the second statement:

Let Q be NP but not polynomial. By definition of NP-complete, for all Q' in the set of NP-complete problems. It follows that all Q' P. (Suppose not. Then there exists some Q' in P, but all elements of NPC, including Q', so Q is in P by a previous Lemma, which contradicts our initial assumption that Q was not in P.)

Q p Q'

Q' p

Conclusion

It can be proved that some problems that we can describe can’t be solved at all. These are Unsolvable problems. Example: The halting problem.

Conclusion

Of the problems that can be solved:some problems can be proven to require at least exponential time. These are infeasible or provably intractable problems.

There is an important class of problems that “probably” require exponential time. These are probably intractable or NP problems.

The problems that we prefer to work with require only polynomial time. These are feasible, tractable, or P problems.