Lecture 2 Computational Complexity Time Complexity Space Complexity.
Computational Complexity - Lecture 01 - Introduction - Tom Kelsey
Transcript of Computational Complexity - Lecture 01 - Introduction - Tom Kelsey
Computational ComplexityLecture 01 - Introduction and Revision
Tom Kelsey & Susmit Sarkar
School of Computer ScienceUniversity of St Andrews
http://www.cs.st-andrews.ac.uk/~tom/[email protected]
Tom Kelsey and Susmit Sarkar CS3052-CC 2013-31-01 1
Administrivia
Lectures every Thursday, 2 - 4:30, Physics CNo "standard" tutorials - one revision tutorial at the endThree practicals – check exact times & dates on MMS
1 P1 is an essay - you tell me about fundamental concepts –Thu Week 3
2 P2 – You write code that converts one problem into another,then solves it – Fri Week 6
3 P3 – I give you algorithm code, you tell me everything thereis to know about its complexity – Thu Week 9
4 P2 & P3 may change! I need to consult with Susmit Sarkar
2 hour exam, answer any 3 from 4 questions
Tom Kelsey and Susmit Sarkar CS3052-CC 2013-31-01 2
Books & Sources
Computational Complexity: A Modern Approach bySanjeev Arora and Boaz BarakIntroduction to Automata Theory, Languages, andComputation, 2nd Ed. by John E. Hopcroft, Rajeev Motwaniand Jeffrey D. UllmanIntroduction to Automata Theory, Languages, andComputation by John E. Hopcroft and Jeffrey D. UllmanMatrix Computations by Gene H. Golub and Charles F. vanVan LoanIntroduction to the Theory of Computation by MichaelSipserA Practical Introduction to Data Structures and AlgorithmAnalysis by Clifford A. ShafferAlgorithm Design by Jon Kleinberg and Eva Tardos
Tom Kelsey and Susmit Sarkar CS3052-CC 2013-31-01 3
Notes
Notes
Notes
Books & Sources
This course is a mashup between:
My notes for Machines & Computation (based on MikeAtkinson’s notes based on Sipser)Steve’s notes for CS3109 Computability & ComplexitySteve’s notes for Graphs & AlgorithmsKevin Wayne’s notes for COS 423 Theory of Algorithms(Princeton)Extra examples/results from the books listed on theprevious slideInteresting stuff from my research
Tom Kelsey and Susmit Sarkar CS3052-CC 2013-31-01 4
Why are we all here?
If you design, modify or deploy an algorithm, someone willask you about its complexity
If you clearly don’t understand the question, we’ve failed asa schoolIf you answer the question incorrectly, you look like a halfwitTo give correct answers, you need a good understanding oftheoretical limits and practical assessment
If someone designs, modifies or deploys an algorithm, youwill want to know about its complexity
If you can’t correctly formulate the question, we’ve failed asa schoolIf you can’t spot incorrect answers, you look like a halfwitIf you can prevent inefficiency – or drastically improveefficiency – you look like a geniusTo assess the correctness of answers, you need a goodunderstanding of theoretical limits and practical assessment
Tom Kelsey and Susmit Sarkar CS3052-CC 2013-31-01 5
Structure of the course
Theoretical Computer ScienceAbstract models of computationTuring machinesComplexity classes, translations
Practical Computer ScienceFlops, master theorem, complexity classesWorst and average case analysesComplexity of important classes of algorithms
Tom Kelsey and Susmit Sarkar CS3052-CC 2013-31-01 6
Notes
Notes
Notes
Computational complexity explained in one slide
Nearly all algorithms that allow inputs of any length willrun forever, returning no solutionMany of those that can be solved, can be solved in constanttime for inputs of a fixed sizeThe honest answer to most questions about complexityclasses is "we have no idea"Much of the work done in complexity is irrelevantThere is a huge mismatch between theory and practiceNevertheless, ignoring complexity is not a good idea
Tom Kelsey and Susmit Sarkar CS3052-CC 2013-31-01 7
This lecture
Go over three topics you covered in 2nd year
1 Finite state machines2 Context free grammars (maybe lecture 2)3 Big O notation
I’ll be using slightly different notation & definitions – go overyour 2nd year notes to make sure you understand everything
Tom Kelsey and Susmit Sarkar CS3052-CC 2013-31-01 8
Automata
For now, we study various abstract machines - automata -which focus on different aspects of computing powerFor every such abstract machine we define the set ofproblems it can solve - the language of the machineOr, given a set of problems, define the automata that solvesthis set
and no other
Computer ≡ AutomataSyntactically correct program ≡ solvable problemAn automata solves a problem in a precise number ofmachine translations
this number measures the time taken
Tom Kelsey and Susmit Sarkar CS3052-CC 2013-31-01 9
Notes
Notes
Notes
Automata Theory
Definitions & properties of mathematical models ofcomputationThe finite automaton
Text processingCompilersHardware design
The context-free grammar or pushdown automatonProgramming languagesArtificial intelligence
The Turing machineAny algorithmic process
Tom Kelsey and Susmit Sarkar CS3052-CC 2013-31-01 10
Computability Theory
We can’t solve all problems using computersWhich problems can we solve?Which problems can’t we solve?
why not?
What if we use a different computer?No solution known or no solution possible?Where are the boundaries between possible and impossible?
Tom Kelsey and Susmit Sarkar CS3052-CC 2013-31-01 11
Complexity Theory
Why are some problems hard?Why are others easy?Can we classify families of problems?Some problems are easy most of the timeWhat if new methods were devised?Time vs Space complexityWhat if better computers were invented?
Tom Kelsey and Susmit Sarkar CS3052-CC 2013-31-01 12
Notes
Notes
Notes
This Course is Important
Many people canDevelop computer graphicsAnalyse operating systemsUtilize component technologywhatever
A Computer Scientist understands the practical & possiblelimits of computation
Tom Kelsey and Susmit Sarkar CS3052-CC 2013-31-01 13
Mathematics
Basic mathematical toolsChapter 0 of Sipser gives a short review of these topicsSetsSequencesFunctions & relationsStrings & languagesBoolean logic – see your 2nd year notes
Proofs
Tom Kelsey and Susmit Sarkar CS3052-CC 2013-31-01 14
Sequences
A sequence is an ordered list of objects( 7, 21, 57 )( apples, oranges, pears )( 0, 1, 1, 2, 3, 5, 8, 13, . . . )
Duplicates are allowedOrder matters
( jump, count, pull ripcord, land )
A tuple is a finite sequenceQuite often, the next term in a sequence depends on itspredecessors
Tom Kelsey and Susmit Sarkar CS3052-CC 2013-31-01 15
Notes
Notes
Notes
Cartesian products
Suppose A = {1, 2, 3} and B = {x, y}A× B is the set of ordered pairs from A and B:
{(1, x), (1, y), (2, x), . . . , (3, y)}
|A× B| = |A|.|B|
We write A×A as A2
A×A× · · · ×A︸ ︷︷ ︸k times
= Ak
Tom Kelsey and Susmit Sarkar CS3052-CC 2013-31-01 16
Functions
N is the set of natural numbers,Z is the set of integers,Q is the set of rational numbers andR is the set of real numbers.
A function takes input from a domain D, which is often aCartesian product
Output is in its range, which is a setµ : N 7→N
µ(n) =
1 if n = 10 if a2 | n for some a > 1(−1)r if n has r distinct prime factors
Tom Kelsey and Susmit Sarkar CS3052-CC 2013-31-01 17
Predicates
A predicate is function with domain D and range {true, false}
pass? : {0, 1, . . . , 20} × {0, 1, 2, . . . , 60} 7→ {true, false}
se : N×N 7→ {true, false}
se(73, 6) = false
se(888888, 222220) = true
Tom Kelsey and Susmit Sarkar CS3052-CC 2013-31-01 18
Notes
Notes
Notes
Relations
A relation is predicate with domain Ak
Often k = 2 and we use infix notation:
a ∼R b means R(a, b) = true
Properties:
Reflexive: a ∼ a ∀a ∈ ASymmetric: a ∼ b iff b ∼ a ∀a, b ∈ AAntisymmetric: a ∼ b and b ∼ a⇒ a = bTransitive: a ∼ b and b ∼ c⇒ a ∼ c
ReST & RoTA
Tom Kelsey and Susmit Sarkar CS3052-CC 2013-31-01 19
Graph of a relation
Tom Kelsey and Susmit Sarkar CS3052-CC 2013-31-01 20
Strings & languages
An alphabet is a set of symbols
Σ = {0, 1}
Γ = {a, b, c, . . . , z}
A string over Σ is a tuple of symbols from Σ
w = 010101110|w| = 9w2 = 1
x = penguin|x| = 8w2 = e
Tom Kelsey and Susmit Sarkar CS3052-CC 2013-31-01 21
Notes
Notes
Notes
Strings & languages
Empty string is denoted by ε
|ε| = 0
Reverse of a string: wR = wnwn−1 . . . w1
z is a substring of w if z appears consecutively within w
Concatenation:
xy = x1x2 . . . xny1y2 . . . ym
xk = xx · · · x︸ ︷︷ ︸k times
Tom Kelsey and Susmit Sarkar CS3052-CC 2013-31-01 22
Strings & languages
Lexicographic ordering of strings is as you’d expect,except thatlength matters
{0, 1} → (ε, 0, 1, 00, 01, 10, 11, 000, 001, . . .)
A language is a set of stringsEnglishEspañolJavaC++{ε, 0, 1, 00, 01, 10, 11, 000, 001, . . .}
Tom Kelsey and Susmit Sarkar CS3052-CC 2013-31-01 23
Induction
Suppose we have a series of dominoes, each 5cm apart.If we roll a marble at the first domino, will they all fall overin sequence?Two questionsDoes the marble knock over domino 1?
can we get to the first step?is the 5cm gap small enough?
can we get from any step to the next?
Tom Kelsey and Susmit Sarkar CS3052-CC 2013-31-01 24
Notes
Notes
Notes
Structural Induction
Works whenever we build incrementally from a small baseNatural numbers
base is 0 or 1, increment by adding 1Sets
base is ∅, increment by inserting membersStrings
base is ε, increment by appending symbols
Lists, Graphs, etc.
Tom Kelsey and Susmit Sarkar CS3052-CC 2013-31-01 25
Structural Induction
The power set P(X) is the set of all subsets of XIf |S| = n, show that |P(S)| = 2n
Base case|∅| = 0, ∅ has one subset, and 20 = 1
Induction stepsuppose that if |S| = n, then |P(S)| = 2n
insert a new element e into Sfor each p ∈ P(S) we can form p∪ {e}giving another 2n members of P(S∪ {e})2n + 2n = 2n+1 and the proof is complete
Tom Kelsey and Susmit Sarkar CS3052-CC 2013-31-01 26
More on Power Sets
Let P be a power set.Show that ⊆ defines an order relation on PLet X, Y and Z be any members of P.
1 X ⊆ X, so ⊆ is reflexive.2 If X ⊆Y and Y⊆ X, then X = Y, so ⊆ is antisymmetric.3 If X ⊆ Y and Y ⊆ Z, then X ⊆ Z, so ⊆ is transitive
Tom Kelsey and Susmit Sarkar CS3052-CC 2013-31-01 27
Notes
Notes
Notes
Mathematical method
1 Read the problem carefully2 Draw a picture3 Look for patterns and counterexamples4 Choose a proof method5 Check each possible case6 Write in formal notation7 Look for ways to apply the result
Tom Kelsey and Susmit Sarkar CS3052-CC 2013-31-01 28
"Proof" by diagrams
Scientifically, diagrams (not geometric constructions) areonly for (counter-)examplesReal and complete proofs might take too much time for anexam or even a practicalFor examination purposes it is sometimes sufficient to showthat you have understood the problem, found the solution,etc., e.g. by:
drawing diagramsgiving a proof sketch (in English)
Get a feeling from the lectures what is requiredRead the exam questions carefully!
Tom Kelsey and Susmit Sarkar CS3052-CC 2013-31-01 29
Computation
Old way of thinkingOnly tangentially related to
operating systemsnetworksgraphicscommunicationsmediaetc
Tom Kelsey and Susmit Sarkar CS3052-CC 2013-31-01 30
Notes
Notes
Notes
Computation
Solve a useful problem, quicklydomains: especially military, financialapproximate solutions often acceptedis an exact solution possible, if so how quickly?
Architecture not important for the principlemechanical "calculators" like Charles Babbage enginehowever: lack of flexibility can restrict the computationalpowerwhich is what this part of the course is about
We are interested in algorithmic problems: applying afunction f to an input x, what is f (x)?
Tom Kelsey and Susmit Sarkar CS3052-CC 2013-31-01 31
Computation
Canonical useful problem from early 20th centuryDreadnought class battleships developed, having samecalibre gunsHow do you ensure a salvo hits a target?
Analog gun laying computers were developedx1 = range, x2 = barrel temp., x3 = propellant temp., x4 =latitude, x5 = own course & speed, x6 = target course &speed, etc.
For input x, what is f (x)?f returns direction & elevation for gunsapproximate values, since slow firing is sub-optimalaccurate results, since missing the target is pointless
Tom Kelsey and Susmit Sarkar CS3052-CC 2013-31-01 32
Jacquard Loom (1801)
programmable: used punch cards to control a sequence ofoperationsno calculations, but important step in the history ofcomputing hardware
Tom Kelsey and Susmit Sarkar CS3052-CC 2013-31-01 33
Notes
Notes
Notes
Zuse Z3 (1941)
programmablemathematical computationsfully automaticconditional branchingcomputer
Tom Kelsey and Susmit Sarkar CS3052-CC 2013-31-01 34
First 5 "Computers"
Tom Kelsey and Susmit Sarkar CS3052-CC 2013-31-01 35
Motivating example 1
Suppose we’re writing C/C++ codeWe send our code to a C pre-processor prior to compilationOne of its tasks is to strip out comments
Lines starting with // in C++Text within /* */ in C
How can we model this?
Tom Kelsey and Susmit Sarkar CS3052-CC 2013-31-01 36
Notes
Notes
Notes
Motivating example 1
ASCII symbols are read oneby oneInitial state unless and untila / is seenIf the next symbol is
/ go to the C++ state;wait for an EOL* go to the C state; waitfor a "*/" pairanything else; return tothe initial state
Tom Kelsey and Susmit Sarkar CS3052-CC 2013-31-01 37
Motivating example 1
Note that:always in exactly one of a finite number of statesthe next state depends on the next symbolthere is a rule for every symbol at each statewe don’t have to visit every statewe could think of the initial state as an accepting state
Tom Kelsey and Susmit Sarkar CS3052-CC 2013-31-01 38
Motivating example 1
We can formalise this as:Q ={none, slash, star, C, C++}Σ = { ASCII chars} ∪{ ASCII device chars}δ : Q× Σ 7→ Q is thetransition functionnone ∈ Q is the startstate{none} ⊆ Q is the set ofaccept states
These 5 define a finiteautomaton M
accepts correctlycommented C, C++rejects incorrectlycommented C, C++we call correctlycommented C, C++ thelanguage of M, and writeL(M)
All other finite automataare defined in the same way
Tom Kelsey and Susmit Sarkar CS3052-CC 2013-31-01 39
Notes
Notes
Notes
Motivating example 2 – grep nano
We have an informal diagram, and we can posit a ΣFrom this we can write down the formal definitionThe start and accept states are usually easyThe transition function is given by a table
a row for each statea column for each symbol
We can convert to and from diagrams and 5-part definitionsTom Kelsey and Susmit Sarkar CS3052-CC 2013-31-01 40
DFAs
Informally:A deterministic finite automaton (DFA) over a finitealphabet Σ is a finite directed graph with the property thateach node emits one labeled edge for each distinct elementof Σ. There is a unique start node, and a non-empty set ofaccept nodes.
Tom Kelsey and Susmit Sarkar CS3052-CC 2013-31-01 41
DFAs
Formally:A deterministic finite automaton (DFA), M, over a finitealphabet Σ is a 5-tuple
M = (Q, Σ, δ, q0, F)
If M is a DFA, and A = {all strings accepted by M}, thenL(M) = AM recognises AM accepts strings from A
Tom Kelsey and Susmit Sarkar CS3052-CC 2013-31-01 42
Notes
Notes
Notes
DFAs
We now have a simple model of computation"compute" means "decide if this string is in a language"For any input string, a DFA will either accept or reject itAlphabets aren’t important
simple pre-process can check that an input string is OKwe can always convert to binary stringsin our examples, we could use the ASCII bitstringrepresentationswe’d need more states and arrows, but the transition tableonly has 2 columns
Everything is deterministic – no gaps in the transition table
Tom Kelsey and Susmit Sarkar CS3052-CC 2013-31-01 43
The computation of a DFA
Let s1s2s3 . . . sn ∈ Σ∗ be an input stringΣ∗ = { all finite strings over Σ}each sj ∈ Σ∗
Consider the sequence of states (q0, q1, q2, . . . , qn)q0 is the start state and, for j = 1 . . . n :qj = δ(qj−1, sj) is the sequence of states the input causes themachine to go through
Tom Kelsey and Susmit Sarkar CS3052-CC 2013-31-01 44
Formal Acceptance
Let M = (Q, Σ, δ, q0, F) be a DFA and w = s1s2s3 . . . sn ∈ Σ∗
be an input stringM accepts w if a sequence of states r0, r1, r2, . . . , rn exists in Qsuch that:
1 r0 = q02 ri+1 = δ(ri, xi+1) for i = 1 . . . , n3 rn is a member of F
Tom Kelsey and Susmit Sarkar CS3052-CC 2013-31-01 45
Notes
Notes
Notes
Asymptotic Order of Growth
K. Wayne – Princeton
Tom Kelsey and Susmit Sarkar CS3052-CC 2013-31-01 46
Asymptotic Order of Growth
K. Wayne – Princeton
Tom Kelsey and Susmit Sarkar CS3052-CC 2013-31-01 47
Asymptotic Order of Growth
K. Wayne – Princeton
Tom Kelsey and Susmit Sarkar CS3052-CC 2013-31-01 48
Notes
Notes
Notes
Asymptotic Order of Growth
K. Wayne – Princeton
Tom Kelsey and Susmit Sarkar CS3052-CC 2013-31-01 49
Summary
This course builds on some concepts considered in pre-honoursAsymptotic complexityFinite state machinesBoolean propositions, truth tables, tautology, validity, ....Regular expressions
The idea is to assess what is and isn’t possible. If it is possible,how long is it likely to take?
Tom Kelsey and Susmit Sarkar CS3052-CC 2013-31-01 50
Notes
Notes
Notes