Post on 18-Dec-2015
AlgorithmsCS4HS at Marquette University
Which is a problem?
a) Find the sum of 4 and 7b) Sort a list of words in alphabetical
orderc) State the most beautiful phrase in
the English languaged) Name the smallest even number
greater than 2 that cannot be written as the sum of two primes
Computational problems
A problem is a relationship between input data and output data
For every set of input values, there is only one correct set of output
A specific set of input and output is called an instance
Adding 4 and 7 is an instance of the general problem of adding integers
What is an algorithm?
An algorithm is a solution to a problem
It gives a finite series of steps that you can follow from any legal input data to reach the correct output data
Can you give an example?
How good is an algorithm?
How do you measure how long an algorithm takes?
We can describe the number of steps the algorithm takes as a function of the size of input data (usually abbreviated as n)
How does the number of steps needed to solve an instance grow as the input grows?
Big-Oh notation
Let's say an algorithm takes 34n3 – 5n2 + 102n + 1471 steps for an input of size n Ugh.
Since we care about growth when n gets big, the biggest term is what matters
Since different computers are better at different operations, we ignore constant coefficients
We say 34n3 – 5n2 + 102n + 1471 is O(n3) Cubic growth is what matters
Addition by hand
How long does it take to do addition by hand?
123+ 456 579
Let’s assume the numbers have n digitsn additions + n carries Running time: O(n)
Multiplication by hand
How long does it take to do multiplication by hand?
123 x 456
738 615
_492__ 56088
Let’s assume the numbers have n digits (n multiplications + n carries) x n digits + (n + 1 digits) x
n additions Running time: O(n2)
Searching and Sorting
Find my number
I've got a list of n numbers You want to look at my list to see if
your favorite number (42) is on it How many numbers do you have to
look at to see if my list has your number?
How long does that take in Big-Oh notation?
Can we do better?
Is there any way to go smaller than O(n)?
What if the numbers were processed in some way beforehand?
What if you knew that the numbers were sorted in ascending order?
Binary search
Repeatedly divide the search space in half
We play a high-low game Again, let's say we're looking for 42
5423 31
Check the middle(Too high)
Check the middle(Too low)
Check the middle(Too low)
Check the middle(Found it!)
42
How long does it take?
We cut the search space in half every time At worst, we keep cutting n in half until we
get 1 In that case, we know we didn't find our
number Let’s say x is the number of times we look:
The running time is O(log2 n)
Interview question
This is a classic interview question asked by Microsoft, Amazon, and similar companies
Imagine that you have 9 red balls One of them is just slightly heavier
than the others, but so slightly that you can’t feel it
You have a very accurate two pan balance you can use to compare balls
Find the heaviest ball in the smallest number of weighings
What’s the smallest possible number?
It’s got to be 8 or fewer We could easily test one ball against
every other ball
There must be some cleverer way to divide them up
Something that is related somehow to binary search
That’s it!
We can divide the balls in half each time
If those all balance, it must be the one we left out to begin with
Nope, we can do better
How? They key is that you can actually cut the number of balls
into three parts each time
We weigh 3 against 3, if they balance, then we know the 3 left out have the heavy ball
When it’s down to 3, weigh 1 against 1, again knowing that it’s the one left out that’s heavy if they balance
Thinking outside the box, er, ball
The cool thing is… Yes, this is “cool” in the CS sense, not in
the real sense Anyway, the cool thing is that we are
trisecting the search space each time
This means that it takes log3 n weighings to find the heaviest ball
We could do 27 balls in 3 weighings, 81 balls in 4 weighings, etc.
Bubble sort
CS people have written hundreds of papers just about how to sort things better
Bubble sort is a very simple sorting algorithm It is not the fastest algorithm
Make a number of passes In each pass, swap each contiguous pair
of items if they're out of order Keep making passes until no swaps are
necessary
Single pass example
Run through the whole list, swapping any entries that are out of order
7
45
0
54
37
108
51
No swapSwap
45
0
No swapSwap
54
37
No swapSwap10
8
51
Activity
Everyone stand up We're going to use the bubble sort
algorithm to sort us all by birth month and day (you can keep the year to yourself)
In each pass, we sequentially swap pairs that are out of order
We keep making passes until no one needs swapping
We are guaranteed that the last person is put in the right place after each pass
How many passes do we need?
How bad could it be? What if the array was in
reverse-sorted order? One pass would only move the
largest number to the bottom We would need n – 1 passes to
sort the whole array What's the total time for
bubble sort?
7
6
5
4
3
2
1
6
7
5
4
3
2
1
6
5
7
4
3
2
1
6
5
4
7
3
2
1
6
5
4
3
7
2
1
6
5
4
3
2
7
1
6
5
4
3
2
1
7
Paths
Google Maps
How does Google Maps find the shortest route from Philadelphia to Milwaukee?
It simplifies all the streets into a graph
A graph has nodes and edges Nodes represent
locations Edges are (parts of)
streets
Shortest paths
We can write a number next to each edge which gives its weight
Weight can represent time, distance, cost: anything, really
The shortest path (lowest total weight) is not always obvious
5
3 4
16
A
B
D
C
E
2
8
6
A
B
D
C
E
D
E
C
B
A
What’s the shortest path?
Take a moment and try to find the shortest path from A to E.
The shortest path has length 14
5
3 4
16
2
8
6
How can we always find the shortest path?
On a graph of that size, it isn’t hard to find the shortest path
A Google Maps graph has millions and millions of nodes
How can we come up with an algorithm that will always find the shortest path from one node to another?
Dijkstra's algorithm adds the closest node we can find, expanding our set of stuff we know the best way to get to
Dijkstra’s algorithm notation
Notation Meaning
s Starting node
d(v)The best distance from s to v found so far
d(u, v)The direct distance between nodes u and v
SA set which contains the nodes for which we know the shortest path from s
V
A set which contains the nodes for which we do not yet know the shortest path from s
Dijkstra’s Algorithm
1. Start with two sets, S and V: S has the starting node in it V has everything else
2. Set the distance to all nodes in V to ∞3. Find the node u in V that is closest to a
node in S4. For every neighbor v of u in V
If d(v) > d(u) + d(u,v) Set d(v) = d(u) + d(u,v)
5. Move u from V to S6. If V is not empty, go back to Step 2
Sets
S V
A B, C, D, E
Sets
S V
A, B C, D, E
Sets
S V
A, B, C D, E
Sets
S V
A, B, C, D E
Sets
S V
A, B, C, D, E
Example for Dijkstra
Node d(u)
A 0
B 5
C 8
D ∞
E 16
5
3 4
16
A
B
D
C
E
2
8
6
Node d(u)
A 0
B 5
C 7
D 11
E 16
Node d(u)
A 0
B 5
C 7
D 10
E 16
Node d(u)
A 0
B 5
C 7
D 10
E 16
Node d(u)
A 0
B 5
C 7
D 10
E 14
Finding the shortest distance
from A to all other nodes
A
B
D
C
E
A
B
C
D
E
Traveling salesman
The traveling salesman problem (TSP) also wants to find the shortest path, but it adds two requirements You have to return to the point where you start You have to visit all the cities exactly once
Like a UPS guy who wants to make all his dropoffs and return to the shipping center
5
3 4
16
2
8
6
What algorithm should we use?
TSP seems easy! There must be some way we can
adapt Dijkstra's algorithm to work Suggestions?
The greedy approach always adds the nearest neighbor
A brute force approach tries all possible tours and sees which is shortest
Greedy doesn’t work
We are tempted to always take the closest neighbor, but there are pitfalls Greedy Optimal
Brute force is brutal
In a completely connected graph, we can try any sequence of nodes
If there are n nodes, there are (n – 1)! tours
For 30 locations, 29! = 8841761993739701954543616000000
If we can check 1,000,000,000 tours per second, it will only take about 20,000 times the age of the universe to check them all
We will (eventually) get the best answer!
A mathematical interlude Many of my students have a poor intuitive
grasp of mathematics The following series often comes up:
Anyone know the closed form of this sum? Students often see this and say, "Isn't that
just n factorial?" does look similar
However, O(n2) is reasonable while O(n!) is a horrifying demon that algorithm designers have nightmares about
NP-complete problems
TSP is just one NP-complete problem These problems are pretty useful (scheduling,
optimizing resource usage, packing boxes) No running-times for NP-complete problems are
known that are better than exponential O(2n) All these problems can easily be converted into
each other If you could solve one efficiently, you could solve
them all efficiently The Clay Mathematics Institute has a $1 million
prize for such an algorithm You get the same prize if you could prove that no such
algorithm is possible
Computability
Turing machine
A Turing machine is a model of computation It consists of a head, an infinitely long tape,
a set of possible states, and an alphabet of characters that can be written on the tape
A finite list of rules says what it should write and whether it should move left or right given the current symbol and state
1 0 1 1 1 1 0 0 0 0
A
Computational power
Computational power means the ability to solve a problem The ability to run an algorithm
Speed is not related to computational power
Turing machines have as much computational power as any known computer Any algorithm that can be run on any
computer can be run on a Turing machine
Halting problem
There are some computational problems that you cannot design an algorithm for
No computer can ever solve the general problem (though sometimes you can solve instances of the problem)
Examples It's impossible to write a program that
can tell if any program will run to completion
Turntables
Douglas Hofstadter uses the metaphor of turntables
Imagine that evil people design records that will shake turntables apart when they're played
Maybe turntable A can play record A and turntable B can play record B
However, if turntable A plays record B, it will shatter
Stuff you have to buy for this proof
Turing machines can perform all possible computations
It's possible to encode the way a Turing machine works such that another Turing machine can read it
It's easy to make a slight change to a Turing machine so that it gives back the opposite answer (or goes into an infinite loop)
Proof by contradiction
You've got a Turing machine M with encoding m
You want to see if M will halt on input x
Assume there is a machine H that can take encoding m and input x H(m,x) is YES if it halts H(m,x) is NO if it loops forever
We create (evil) machine E that takes description m and runs H(m,m) If H(m,m) is YES, E loops
forever If H(m,m) is NO, E returns
YES
Hm
xYES
NO
H YES
NO
Em Loop
forever
YES
A mind-bending proof
Let's say that e is the description of E What happens if you feed description e into E?
E(e) says what E will do with itself as input If it returns YES, that means that E on input e loops
forever But it can't, because it just returned YES
If it loops forever, then E on input e would return YES But it can't, because it's looping forever!
Our assumption that machine H exists must be wrong