UMass Lowell Computer Science 91.503Analysis of Algorithms
UMass Lowell Computer Science 91.503Analysis of AlgorithmsAnalysis of Algorithms
Prof. Karen DanielsSpring 2010
Analysis of AlgorithmsProf. Karen Daniels
Spring 2010Spring, 2010Spring, 2010
Design PatternsDesign Patterns forforDesign Patterns Design Patterns for for Optimization ProblemsOptimization Problems
D namicD namic ProgrammingProgrammingDynamic Dynamic ProgrammingProgrammingMatrix ParenthesizingMatrix Parenthesizing
Longest Common SubsequenceLongest Common SubsequenceLongest Common SubsequenceLongest Common SubsequenceActivity SelectionActivity Selection
Algorithmic Paradigm ContextAlgorithmic Paradigm Context
Divide & Conquer
Dynamic Programming
Greedy AlgorithmConquer Programming Algorithm
View problem as collection of subproblems
“Recursive” nature Independent subproblems overlapping typically
sequential qdependence
Number of subproblems depends on partitioning
factors
typically small
Preprocessing typically sort Characteristic running time typically log depends on number often dominatedCharacteristic running time typically log
function of n depends on number and difficulty of subproblems
often dominated by nlogn sort
Primarily for optimization problems (find an optimal solution)
solution) Optimal substructure: optimal solution to problem contains within it optimal solutions to subproblems
Greedy choice property: y p p ylocally optimal produces globally optimal Heuristic version useful for bounding optimal value
Subproblem solution orderSubproblem solution orderMake choice, then solve Make choice, then solve subproblem(s)subproblem(s)
Solve subproblem(s), Solve subproblem(s), then make choicethen make choice
Dynamic Programming Approach to Optimization ProblemsDynamic Programming Approach to Optimization Problemsto Optimization Problemsto Optimization Problems
1.1. Characterize structure of an optimal Characterize structure of an optimal solution.solution.
2.2. Recursively define value of an optimal Recursively define value of an optimal solution.solution.
3.3. Compute value of an optimal solution in Compute value of an optimal solution in bottombottom--up fashion.up fashion.
4.4. Construct an optimal solution from Construct an optimal solution from computed information.computed information.
source: 91.503 textbook Cormen, et al.source: 91.503 textbook Cormen, et al.
Example: Matrix Parenthesization D fi i i
Example: Matrix Parenthesization D fi i iDefinitionsDefinitions
Given “chain” of n matrices: <AGiven “chain” of n matrices: <A AA AA >>Given chain of n matrices: <AGiven chain of n matrices: <A11, A, A22, … A, … Ann, >, >
Compute product ACompute product A11AA22… A… Ann efficientlyefficiently
Minimize “cost” = number of scalar multiplicationsMinimize “cost” = number of scalar multiplications
Multiplication order matters!Multiplication order matters!u p c o o de e s!u p c o o de e s!
source: 91.503 textbook Cormen, et al.source: 91.503 textbook Cormen, et al.
Example: Matrix Parenthesization Step 1: Characterizing an Optimal SolutionExample: Matrix Parenthesization Step 1: Characterizing an Optimal SolutionStep 1: Characterizing an Optimal SolutionStep 1: Characterizing an Optimal Solution
Observation: Observation: Any parenthesization of Any parenthesization of AAiiAAi+1i+1… A… Ajj must split it between must split it between AAk k andand AAk+1 k+1 for some k.for some k.jj
THM: Optimal Matrix Parenthesization:THM: Optimal Matrix Parenthesization:If an optimal parenthesization of If an optimal parenthesization of AAiiAAi+1i+1… A… Ajj splits at k, then splits at k, then parenthesi ation of prefiparenthesi ation of prefi AA AA AA m st be an optimal parenthesi ationm st be an optimal parenthesi ationparenthesization of prefix parenthesization of prefix AAiiAAi+1i+1… A… Akk must be an optimal parenthesization.must be an optimal parenthesization.
Why?Why?If existed less costly way to parenthesize prefix, then substituting thatIf existed less costly way to parenthesize prefix, then substituting thatIf existed less costly way to parenthesize prefix, then substituting that If existed less costly way to parenthesize prefix, then substituting that
parenthesization would yield less costly way to parenthesize parenthesization would yield less costly way to parenthesize AAiiAAi+1i+1… A… Ajj , , contradicting optimality of that parenthesization. contradicting optimality of that parenthesization.
source: 91.503 textbook Cormen, et al.source: 91.503 textbook Cormen, et al.
common DP proof technique: common DP proof technique: “cut“cut--andand--paste” proof by contradictionpaste” proof by contradiction
Example: Matrix Parenthesization Step 2: A Recursive SolutionExample: Matrix Parenthesization Step 2: A Recursive SolutionStep 2: A Recursive SolutionStep 2: A Recursive Solution
Recursive definition of minimum parenthesization Recursive definition of minimum parenthesization ppcost:cost:
0 if i = jm[i,j]= min{m[i,k] + m[k+1,j] + pi-1pkpj} if i < j
0 if i j
i <= k < j
How many distinct subproblems?How many distinct subproblems?
each matrix Ai has dimensions pi-1 x pi
source: 91.503 textbook Cormen, et al.source: 91.503 textbook Cormen, et al.
Example: Matrix Parenthesization Step 3: Computing Optimal CostsExample: Matrix Parenthesization Step 3: Computing Optimal CostsStep 3: Computing Optimal CostsStep 3: Computing Optimal Costs
2 6252 625
2,5002,500
1 0001 00000
2,6252,625 1,0001,000
s: value of k that achieves optimal s: value of k that achieves optimal cost in computing m[i, j]cost in computing m[i, j]
source: 91.503 textbook Cormen, et al.source: 91.503 textbook Cormen, et al.
Example: Matrix Parenthesization Step 4: Constructing an Optimal SolutionExample: Matrix Parenthesization Step 4: Constructing an Optimal SolutionStep 4: Constructing an Optimal SolutionStep 4: Constructing an Optimal Solution
PRINTPRINT--OPTIMALOPTIMAL--PARENS(s, i, j)PARENS(s, i, j)if i = jif i = j
then print “A”then print “A”iielse print “(“else print “(“
PRINTPRINT--OPTIMALOPTIMAL--PARENS(s, i, s[i, j])PARENS(s, i, s[i, j])PRINTPRINT--OPTIMALOPTIMAL--PARENS(s, s[i, j]+1, j)PARENS(s, s[i, j]+1, j)print “)“print “)“
source: 91.503 textbook Cormen, et al.source: 91.503 textbook Cormen, et al.
Example: Matrix Parenthesization Memoization
Example: Matrix Parenthesization MemoizationMemoizationMemoization
Provide Dynamic Programming Provide Dynamic Programming source: 91.503 textbook Cormen, et al.source: 91.503 textbook Cormen, et al.y g gy g gefficiencyefficiencyButBut with with toptop--downdown strategystrategy
Use recursionUse recursion LOOKUPLOOKUP--CHAIN(p,i,j) CHAIN(p,i,j) Use recursionUse recursionFill in table “on demand”Fill in table “on demand”
Example: Example:
(p j)(p j)1 1 ifif m[i,j] < m[i,j] < 2 2 then return then return m[i,j]m[i,j]3 3 ifif i=j i=j
∞
RECURSIVERECURSIVE--MATRIXMATRIX--CHAIN:CHAIN:
MEMOIZEDMEMOIZED--MATRIXMATRIX--CHAIN(p)CHAIN(p)1 l h[ ]1 l h[ ] 11
jj44 then then m[i,j] 0m[i,j] 05 5 elseelse forfor k i to jk i to j--1166 dodo q LOOKUPq LOOKUP--CHAIN(p i k)CHAIN(p i k)1 n length[p] 1 n length[p] -- 11
2 2 forfor i 1 i 1 toto nn3 3 dodo for j i for j i toto nn
6 6 dodo q LOOKUPq LOOKUP--CHAIN(p,i,k)CHAIN(p,i,k)+ LOOKUP+ LOOKUP--CHAIN(p,k+1,j)CHAIN(p,k+1,j)+ p+ pii--1 1 ppk k ppjj
77 ifif [i j][i j]44 dodo m[i,j] m[i,j] 5 return LOOKUP5 return LOOKUP--CHAIN(p,1,n)CHAIN(p,1,n)
∞ 77 ifif q < m[i,j] q < m[i,j] 88 thenthen m[i,j] qm[i,j] q9 9 returnreturn m[i,j] m[i,j]
Example: Longest Common Subsequence (LCS): MotivationExample: Longest Common Subsequence (LCS): MotivationSubsequence (LCS): MotivationSubsequence (LCS): Motivation
Strand of DNA: string over finite set {A,C,G,T}Strand of DNA: string over finite set {A,C,G,T}each element of set is a base: each element of set is a base: adenine, guanine, cytosine or thymineadenine, guanine, cytosine or thymine
Compare DNA similaritiesCompare DNA similaritiesSS1 1 = = ACCACCGGGGTCGTCGAGAGTTGGCGCGCCGGGGAAGCCGGCCGAAAAGCCGGCCGAA
SS2 2 = = GTCGTGTCGTTTCGGAACGGAATTGCCGGCCGTTTTGCGCTTCCTTGGTTAAAAAAOne measure of similarity:One measure of similarity:yy
find the longest string Sfind the longest string S33 containing bases that also appear (not containing bases that also appear (not necessarily necessarily consecutivelyconsecutively) in S) in S11 and Sand S22
SS3 3 = = GTCGTCGGAAGCCGGCCGAAGTCGTCGGAAGCCGGCCGAA
source: 91.503 textbook Cormen, et al.source: 91.503 textbook Cormen, et al.
Example: LCS Definitions
Example: LCS DefinitionsDefinitionsDefinitions
SequenceSequence is a is a subsequencesubsequence of of source: 91.503 textbook Cormen, et al.source: 91.503 textbook Cormen, et al.
qq qqif if (strictly increasing (strictly increasing
indices of X)indices of X) such thatsuch thatexample: example: is subsequence of is subsequence of
with index sequencewith index sequence
Z i b f X d Y if Z iZ i b f X d Y if Z iZ is common subsequence of X and Y if Z is Z is common subsequence of X and Y if Z is subsequence of both X and Ysubsequence of both X and Y
example:example:example:example:common subsequence but not common subsequence but not longestlongestcommon subsequence. Longest?common subsequence. Longest?
Longest Common Subsequence ProblemLongest Common Subsequence Problem: Given 2 sequences : Given 2 sequences X, Y, find maximumX, Y, find maximum--length common subsequence Z.length common subsequence Z.
Example: LCS Step 1: Characterize an LCS
Example: LCS Step 1: Characterize an LCSStep 1: Characterize an LCSStep 1: Characterize an LCS
THM 15.1: Optimal LCS Substructure THM 15.1: Optimal LCS Substructure 5 Opt a CS Subst uctu e5 Opt a CS Subst uctu eGiven sequences:Given sequences:For any LCSFor any LCS of X and Y: of X and Y:
1 1 if if thenthen and Zand Zkk--11 is an LCS of Xis an LCS of Xmm--11 and Yand Ynn--11
2 2 if if thenthen Z is an LCS of XZ is an LCS of Xmm--11 and Y and Y 3 if3 if thenthen Z is an LCS of X and YZ is an LCS of X and Ynn 113 if 3 if thenthen Z is an LCS of X and YZ is an LCS of X and Ynn--11
PROOF: based on producing PROOF: based on producing contradictionscontradictions
1 a) 1 a) Suppose . Appending to Z contradicts Suppose . Appending to Z contradicts longest longest nature of Z.nature of Z.b) To establish b) To establish longest longest nature of Znature of Zkk--11, suppose common subsequence W of , suppose common subsequence W of XXmm--11 and and YYnn--11
has length > khas length > k--1. Appending to W yields common subsequence of length > k = contradiction.1. Appending to W yields common subsequence of length > k = contradiction.
22 C b W fC b W f XX dd YY f l th k ld l b bf l th k ld l b b2 2 Common subsequence W of Common subsequence W of XXmm--11 and and Y Y of length > k would also be common subsequence of length > k would also be common subsequence of of XXmm, Y, contradicting , Y, contradicting longest longest nature of Z.nature of Z.
3 3 Similar to proof of (2)Similar to proof of (2)source: 91.503 textbook Cormen, et al.source: 91.503 textbook Cormen, et al.
Example: LCS Step 2: A Recursive Solution
Example: LCS Step 2: A Recursive SolutionStep 2: A Recursive SolutionStep 2: A Recursive Solution
Implications of Theorem 15 1:Implications of Theorem 15 1:Implications of Theorem 15.1:Implications of Theorem 15.1:
?? nono??yesyes nono
Fi d LCS(X Y )Find LCS(Xm-1, Yn-1) Find LCS(Xm-1, Y) Find LCS(X, Yn-1)
LCS1(X, Y) = LCS(Xm-1, Yn-1) + xm LCS2(X, Y) = max(LCS(Xm-1, Y), LCS(X, Yn-1))
Example: LCS Step 2: A Recursive Solution (continued)
Example: LCS Step 2: A Recursive Solution (continued)Step 2: A Recursive Solution (continued)Step 2: A Recursive Solution (continued)
Overlapping subproblem structure:Overlapping subproblem structure:source: 91.503 textbook Cormen, et asource: 91.503 textbook Cormen, et app g ppp g p),(),( 111 YXLCSYXLCS mnm −−− ⊆
)()( YXLCSYXLCS),( YXLCS⊆
),(),( 111 −−− ⊆ nnm YXLCSYXLCSΘΘ(mn) distinct (mn) distinct subproblemssubproblems
Recurrence for length of optimal solution:Recurrence for length of optimal solution:0 if i=0 or j=0
c[i,j]= c[i-1,j-1]+1 if i,j > 0 and xi=yj
max(c[i,j-1], c[i-1,j]) if i,j > 0 and xi=yj
0 if i=0 or j=0
Conditions of problem can exclude some subproblems!
max(c[i,j 1], c[i 1,j]) if i,j > 0 and xi yj
Example: LCS Step 3: Compute Length of an LCSExample: LCS Step 3: Compute Length of an LCSStep 3: Compute Length of an LCSStep 3: Compute Length of an LCS
BB CC BB AA
What is the What is the asymptotic worstasymptotic worst--
case time case time complexity?complexity?
BB
CC
0
12
complexity?complexity?
CC
BB
AA
2
3
c tablec table
4
source: 91.503 textbook Cormen, et al.source: 91.503 textbook Cormen, et al.
(represent b table)(represent b table)
Example: LCS Step 4: Construct an LCSExample: LCS Step 4: Construct an LCSStep 4: Construct an LCSStep 4: Construct an LCS
source: 91.503 textbook Cormen, et al.source: 91.503 textbook Cormen, et al.
Dynamic ProgrammingDynamic Programmingl di t G d Al ithl di t G d Al ith…leading to a Greedy Algorithm……leading to a Greedy Algorithm…
Activity SelectionActivity Selection
Activity SelectionOptimization ProblemActivity SelectionOptimization ProblemOptimization Problem Optimization Problem
Problem Instance:Problem Instance:Problem Instance:Problem Instance:Set Set SS = {1,2,...,= {1,2,...,nn} of } of nn activitiesactivitiesEach activityEach activity ii has:has:Each activity Each activity ii has:has:
start time: start time: ssii
finish time:finish time: ff ii fs ≤finish time: finish time: ffii
Activities Activities i, ji, j are compatible iff nonare compatible iff non--overlapping: overlapping: )[ fs ≤ )[ fs ≤
Objective:Objective:select aselect a maximummaximum--sizedsized set of mutually compatible activitiesset of mutually compatible activities
)[ ii fs ≤ )[ jj fs ≤
select a select a maximummaximum--sizedsized set of mutually compatible activities set of mutually compatible activities
source: 91.404 textbook Cormen, et al.source: 91.404 textbook Cormen, et al.
Activity SelectionActivity Selection
Activity Time DurationActivity Time Duration11 22 33 44 55 66 77 88 99 1010 12121111 1313 1414 1515 1616
11
Activity Time DurationActivity Time Duration
Activity Activity NumberNumber
22
33
4444
66
55
88
77
Activity SelectionActivity Selection
}{ ffSS ≤<≤ }:{ jkkikij sfsfSaS ≤<≤∈=Solution to Solution to SSijij including including aakk produces 2 subproblems:produces 2 subproblems:1)1) SS (start after(start after aa finishes; finish beforefinishes; finish before aa starts)starts)1) 1) SSikik (start after (start after aaii finishes; finish before finishes; finish before aakk starts)starts)2) 2) SSkjkj (start after (start after aakk finishes; finish before finishes; finish before aajj starts) starts)
cc[[i ji j] size of maximum] size of maximum size subset ofsize subset of
⎫⎧ /if
cc[[i,ji,j]=size of maximum]=size of maximum--size subset of size subset of mutually compatible activities in mutually compatible activities in SSijij..
⎪⎭
⎪⎬⎫
⎪⎩
⎪⎨⎧
/≠++/=
=∈<< 0 if}1],[],[{max
0if0],[
; ijSajki
ij
SjkckicS
jicijk
source: 91.404 textbook Cormen, et al.source: 91.404 textbook Cormen, et al.
Top Related