Algorithm Design and Analysis

Miles Jones

mej016@eng.ucsd.edu

Office 4208 CSE Building

Lecture 22: Dynamic Programming Examples (Edit Distance/Knapsack)

CSE 101

EDIT DISTANCE

Given two words (strings), how can we define a notion of “closeness”

For example:

Is PELICAN closer to PENGUIN or POLITICIAN?

EDIT DISTANCE (DEFINITION)

We can keep track of how many “changes” we need to change one

word into another.

The changes can be

▪ insertion,

▪ deletion, or

▪ substitution.

For example, if we line up the words PELICAN and OSTRICH

P E L I C A N

O S T R I C H

s s s s s s s

EDIT DISTANCE (DEFINITION)

Is 7 the cheapest cost?

P E L I C A N

O S T R I C H

s s s s s s s

EDIT DISTANCE (DEFINITION)

Is 7 the cheapest cost?

P E L I C A N

O S T R I C H

s s s s s s s

P E L - I C A N

O S T R I C - H

s s s i d s

EDIT DISTANCE (DEFINITION)

Is 4 the cheapest?

P E L I C A N

P E N G U I N

s s s s

EDIT DISTANCE (DEFINITION)

Is 7 the cheapest?

P E L I C A N - - -

P O L I T I C I A N

s s s s i i i

EDIT DISTANCE (BRUTE FORCE)

Brute force: try all possible combinations and find the minimum cost

of all of them.

What is the lower bound of the number of possible combinations if the

size of the words are n,m, n≤m?

EDIT DISTANCE (BRUTE FORCE)

Brute force: try all possible combinations and find the minimum cost

of all of them.

Lower bound of the number of possible combinations if the size of the

words are n,m, n≤m.

Each column could be one of three things (at least for the first n

columns.) So there are at least 3𝑛 different combinations. And really

there are much more than that!!!!!

EDIT DISTANCE (DYNAMIC PROGRAMMING)

Find the minimum cost of two words x[1…n] and y[1…m]

Step 1: Define subproblems:

Step 2: base case

Step 3: express recursively

Step 4: order the subproblems

EDIT DISTANCE (DYNAMIC PROGRAMMING)

Find the minimum cost of two words x[1…n] and y[1…m]

Step 1: Define subproblems:

▪ Let E(i,j) be the minimum cost of the two words x[1… i] and y[1…j].

Step 2: base case

Step 3: express recursively

Step 4: order the subproblems

EDIT DISTANCE (DYNAMIC PROGRAMMING)

What are the base cases?

EDIT DISTANCE (DYNAMIC PROGRAMMING)

What are the base cases?

When the first word is empty then the edit distance is the length of the second

word and when the second word is empty the edit distance is the length of the

first word.

E(i,0)=i

E(0,j)=j

EDIT DISTANCE (DYNAMIC PROGRAMMING)

Find the minimum cost of two words x[1…n] and y[1…m]

Step 1: Define subproblems:

▪ Let E(i,j) be the minimum cost of the two words x[1… i] and y[1…j].

Step 2: base case

▪ E(i,0) = i

▪ E(0,j) = j

Step 3: express recursively

Step 4: order the subproblems

EDIT DISTANCE (DYNAMIC PROGRAMMING)

Step 3: express recursively:

▪ Split into cases depending on the last column of the alignment of x[1… i] and y[1…j].

▪ Case 1: the last column looks like

▪ Case 2: the last column looks like

▪ Case 3: the last column looks like

x[i]

-

-

y[j]

x[i]

y[j]

EDIT DISTANCE (DYNAMIC PROGRAMMING)

Step 3: express recursively:

▪ Case 1: the last column looks like

▪ This is a deletion with a cost of 1 so if the minimum cost of x[1… i] and y[1…j] has

this in the last column then…..

x[i]

-

EDIT DISTANCE (DYNAMIC PROGRAMMING)

Step 3: express recursively:

▪ Case 1: the last column looks like

▪ This is a deletion with a cost of 1 so if the minimum cost of x[1… i] and y[1…j] has

this in the last column then

▪ E(i,j)=1+E(i-1,j)

x[i]

-

EDIT DISTANCE (DYNAMIC PROGRAMMING)

Step 3: express recursively:

▪ Case 2: the last column looks like

▪ This is an insertion with a cost of 1 so if the minimum cost of x[1… i] and y[1…j] has

this in the last column then….

-

y[j]

EDIT DISTANCE (DYNAMIC PROGRAMMING)

Step 3: express recursively:

▪ Case 2: the last column looks like

▪ This is an insertion with a cost of 1 so if the minimum cost of x[1… i] and y[1…j] has

this in the last column then

▪ E(i,j)=1+E(i,j-1)

-

y[j]

EDIT DISTANCE (DYNAMIC PROGRAMMING)

Step 3: express recursively:

▪ Case 3: the last column looks like x[i]

y[j]

EDIT DISTANCE (DYNAMIC PROGRAMMING)

Step 3: express recursively:

▪ Case 3: the last column looks like

▪ Case 3.1: x[i]=y[j] (no cost)

▪ E(i,j)=E(i-1,j-1)

▪ Case 3.2: x[i]≠y[j] (substitution cost of 1)

▪ E(i,j)=1+E(i-1,j-1)

x[i]

y[j]

EDIT DISTANCE (DYNAMIC PROGRAMMING)

Step 3: express recursively:

▪ So take the minimum of all three cases

▪ E(i,j)=min( 1 + E(i-1,j), 1 + E(i,j-1), (1-𝛿𝑥 𝑖 ,𝑦[𝑗])+E(i-1,j-1))

▪ (delta function 𝛿𝑎,𝑏 = ቊ0 𝑖𝑓 𝑎 ≠ 𝑏1 𝑖𝑓 𝑎 = 𝑏

)

EDIT DISTANCE (DYNAMIC PROGRAMMING)

Step 4: ordering…..

▪ To calculate E(i,j), we need to know E(i-1,j), E(i,j-1) and E(i-1,j-1)

▪ Think of a 2-d array and where are the indices in relation to ( i,j)?

▪ So, order them in such a way to visit all the necessary entries before you visit (i,j).

▪ One way to do this is left to right through rows going from top to bottom.

(i-1,j-1) (i-1,j)

(i,j-1) (i,j)

EDIT DISTANCE (DYNAMIC PROGRAMMING)

EditDist(x[1…n],y[1…m])

Initialize for i from 1 to n, E(i,0)=i and for j from 1 to m, E(0,j)=j

for i from 1 to n

for j from 1 to m

E(i,j)=min( 1 + E(i-1,j), 1 + E(i,j-1), (1-𝛿𝑥 𝑖 ,𝑦[𝑗])+E(i-1,j-1))

EDIT DISTANCE (EXAMPLE)∅ P E L I C A N

∅

P

O

L

I

T

I

C

I

A

N

TABULATION/MEMOIZATION

THE KNAPSACK PROBLEM

Suppose you are a burglar who breaks into a store and

you want to leave with the maximum value of items.

Your knapsack can only hold 13 lbs and the items in the store are:

Value 4 9 12 15 19 21

Weight 2 4 5 7 8 9

THE KNAPSACK PROBLEM

What is the maximum value you can have from a list

of items a[1],…,a[n] where each item has a value v[ i] and a weight w[i]

given that you cannot have more weight than W.

Step 1: subproblems:

THE KNAPSACK PROBLEM

What is the maximum value you can have from a list

of items a[1],…,a[n] where each item has a value v[ i] and a weight w[i]

given that you cannot have more weight than W.

Step 1: subproblems:

Let K(w) be the maximum value you can have in a w-capacity

knapsack.

THE KNAPSACK PROBLEM

What is the maximum value you can have from a list

of items a[1],…,a[n] where each item has a value v[ i] and a weight w[i]

given that you cannot have more weight than W.

Step 2: base cases:

Let K(w) be the maximum value you can have in a w-capacity

knapsack.

THE KNAPSACK PROBLEM

What is the maximum value you can have from a list

of items a[1],…,a[n] where each item has a value v[ i] and a weight w[i]

given that you cannot have more weight than W.

Let K(w) be the maximum value you can have in a w-capacity

knapsack.

Step 3: express recursively

THE KNAPSACK PROBLEM

What is the maximum value you can have from a list

of items a[1],…,a[n] where each item has a value v[ i] and a weight w[i]

given that you cannot have more weight than W.

Let K(w) be the maximum value you can have in a w-capacity knapsack.

Step 3: express recursively

What is K(w)? Take away the weight of each item and see what value it

is if you add that item.

𝐾 𝑤 = max𝑖 :𝑤[𝑖]≤𝑤

𝐾 𝑤 − 𝑤 𝑖 + 𝑣 𝑖

THE KNAPSACK PROBLEM

What is the maximum value you can have from a list

of items a[1],…,a[n] where each item has a value v[ i] and a weight w[i]

given that you cannot have more weight than W.

Let K(w) be the maximum value you can have in a w-capacity knapsack.

𝐾 𝑤 = max𝑖 :𝑤[𝑖]≤𝑤

𝐾 𝑤 − 𝑤 𝑖 + 𝑣 𝑖

Step 4: order.

Then order the subproblems from 1 to W.

THE KNAPSACK PROBLEM

pseudocode:

Knapsack(v[1…n],w[1…n],W)

K(0):=0

prev(0):=nil

for w from 1 to W

K(w):=0

for j from 0 to n

if K(w)<K(w-w[j])+v[j] then

K(w):= K(w-w[j])+v[j]

prev(w):=j

return K(W)

Runtime???

THE KNAPSACK PROBLEMValue 4 9 12 15 19 21

Weight 2 4 5 7 8 9

v[i] w[i] 0 1 2 3 4 5 6 7 8 9 10 11 12 13

4 2

9 4

12 5

15 7

19 8

21 9

THE KNAPSACK PROBLEM (NO REPEATS)

Suppose you are a burglar who breaks into somebody’s

house where there is only one item of each.

You want to leave with the maximum value of items but you can’t take

more than one of each thing.

Your knapsack can only hold 13 lbs and the items in the house are:

Value 4 9 12 15 19 21

Weight 2 4 5 7 8 9

THE KNAPSACK PROBLEM (NO REPEATS)

What is the maximum value you can have from a list

of items a[1],…,a[n] where each item has a value v[ i] and a weight w[i]

given that you cannot have more weight than W.

Step 1: subproblems:

THE KNAPSACK PROBLEM (NO REPEATS)

What is the maximum value you can have from a list

of items a[1],…,a[n] where each item has a value v[ i] and a weight w[i]

given that you cannot have more weight than W.

Step 1: subproblems:

Let K(w,j) be the maximum value you can have in a w-capacity

knapsack using only the items a[1],…a[j]

THE KNAPSACK PROBLEM

What is the maximum value you can have from a list

of items a[1],…,a[n] where each item has a value v[ i] and a weight

w[i]

given that you cannot have more weight than W.

Let K(w,j) be the maximum value you can have in a w-capacity

knapsack using only the items a[1],…a[j]

Step 2: express recursively

THE KNAPSACK PROBLEM

Let K(w) be the maximum value you can have in a w-capacity knapsack.

Step 2: express recursively

What is K(w,j)?

Case 1: taking a[j] is better value:

add the item a[j] to a knapsack with max cap w-w[j] and add the value of j.

𝐾 𝑤, 𝑗 = 𝐾 𝑤 − 𝑤 𝑗 , 𝑗 − 1 + 𝑣[𝑗]

Case 2: taking a[j] is not better value.

𝐾 𝑤, 𝑗 = 𝐾 𝑤, 𝑗 − 1

So take the maximum of these two scenarios.

𝐾 𝑤, 𝑗 = max {𝐾 𝑤 − 𝑤 𝑗 , 𝑗 − 1 + 𝑣 𝑗 , 𝐾 𝑤, 𝑗 − 1 }

THE KNAPSACK PROBLEM

𝐾 𝑤, 𝑗 = max{𝐾 𝑤 − 𝑤 𝑗 , 𝑗 − 1 + 𝑣 𝑗 , 𝐾 𝑤, 𝑗 − 1 }

Step 3: order and base cases.

Base cases:

K(0,j)=0 for all j

K(w,0)=0 for all w

We need to know (𝑤 − 𝑤 𝑗 , 𝑗 − 1) and 𝐾 𝑤, 𝑗 − 1 before computing

K(w,j)

Order the problems left to right top to bottom.

(𝒘, 𝒋 − 𝟏)

(𝑤 − 𝑤[𝑗], 𝑗 − 1) (𝑤, 𝑗)

THE KNAPSACK PROBLEMValue 4 9 12 15 19 21

Weight 2 4 5 7 8 9

v[i] w[i] 0 1 2 3 4 5 6 7 8 9 10 11 12 13

4 2

9 4

12 5

15 7

19 8

21 9

TABULATION/MEMOIZATION

THE KNAPSACK PROBLEMValue 4 9 12 15 19 21

Weight 2 4 5 7 8 9

v[i] w[i] 0 1 2 3 4 5 6 7 8 9 10 11 12 13

4 2

9 4

12 5

15 7

19 8

21 9

THE KNAPSACK PROBLEM (NO REPEATS)

pseudocode:

Knapsack(v[1…n],w[1…n],W)

K(0,j):=0 for all j

K(w,0):=0 for all w

prev(0):=nil

for w from 1 to W

for j from 0 to n

𝐾 𝑤, 𝑗 = max{𝐾 𝑤 − 𝑤 𝑗 , 𝑗 − 1 + 𝑣 𝑗 , 𝐾 𝑤, 𝑗 − 1 }

return K(w,n)

Runtime???