Algorithm Design and Analysisudel.edu/~amotong/teaching/algorithm design... · (Design) Algorithm...

Algorithm Design and Analysis

Version: Summer 2018 Amo G. Tong 1

• Course ID: CISC621010

• Time: Tues. and Thur. 8:00am-9:15am

• Location: GOR116

• Instructor: Dr. Tong• Email: [email protected] (temporary)

• Office hours: Tues 9:30am-11:00am

• Office: 412 Smith Hall

• TA: Fanchao Meng• Email: [email protected]

• Office hours: Thurs 5:00pm-7:00pm

• Office: 201 Smith Hall

Course Information


• Text book:• Cormen, Leiserson, Rivest and Stein, Introduction to

Algorithms (3rd edition) McGraw-Hill & MIT Press.

• Other materials:• As announced.

• Prerequisite: • Undergraduate algorithm course• Discreate math• Or equivalent.

Course Information


• Organization

• Five assignments. (35%)

• Midterm one. (25%)

• Midterm two. (Homework demo) (10%)

• Final exam. (25%)

• Pop quiz in class. (5%)

• Extra credits. (5% more or less)

Course Information


• Pop quiz

• Pop quiz is used to encourage class participation.

• The ultimate goal is to learn well.

• A student will automatically get full credit for pop quiz if

they get no less than 92 out of 100 points in the final exam.

Course Information


• Final Grading Letter

• The score in each category is less important than the score

relative to the class average. There is no fixed curve. If

everyone performs well in the class then everyone can get

top grades.

Course Information


• Canvas

• Online systems will be available soon.

Course Information


Lecture 1Introduction and Sorting• Introduction

• O-notation

• Sorting


Introduction


Algorithm This Course

The way we solve problems. (Design) Algorithm design

Implementation (Design) Pseudocode

Introduction





Effectiveness (Analysis and Proofs) Correctness and optimality

Efficiency (Analysis) Time and space complexity

Hardness of problems (Analysis and Proofs) Computing theory and Np-completeness

Introduction





Effectiveness (Analysis and Proofs) Correctness and optimality

Efficiency (Analysis) Time and space complexity

Hardness of problems (Analysis and Proofs) Computing theory and Np-completeness

Typically, given a problem, we design an algorithm, provide pseudocode, prove the correctness, and analyze the running time.

• Algorithm Design:

• Self-reducibility• Solve the problem by solving sub-problems

• Merge sort

• Incremental Method• Update the current solution until a desired one is found.

• Insertion sort.

Introduction


• Time Complexity:• 𝑇(𝑛): running time (# of operations) with respect to the

input size 𝑛.• Worst-case

• Average (assume some distribution of the input)

Introduction





• Asymptotic analysis:• Ignore constants and look at 𝑇 𝑛 when 𝑛 approaches to +∞

• 𝑂-notation, Ω-notation, Θ-notation.

Introduction





• Asymptotic analysis:• Ignore constants and look at 𝑇 𝑛 when 𝑛 approaches to +∞


• Find time complexity:• Compute directly

• Solve recurrence

Introduction


• Hardness of the problem:

• Is this problem solvable? (easy)

• Can we solve the problem in 𝑂(𝑓(𝑛)) time? (hard)

• Can we solve the problem in polynomial time? (hard)

• What is fastest possible algorithm? (hard)

Introduction



Big O notation


(Def) 𝑂-notation:

𝑂(𝑔(𝑛)) = {𝑓(𝑛): there exist positive constants 𝑐 and 𝑛0 such that 0 ≤ 𝑓 𝑛 ≤ 𝑐 𝑔(𝑛) for all 𝑛 ≥ 𝑛0}.


• We write 𝑓(𝑛) = 𝑂(𝑔(𝑛)) if 𝑓 𝑛 ∈ 𝑂(𝑔(𝑛)).

• Asymptotic upper bound

• 𝑐 is independent of 𝑛.

Big O notation





Big O notation




Example:𝑛 = 𝑂(𝑛2) ? yes ! 𝑛 ≤ 𝑛2 when 𝑛 ≥ 1.


Big O notation




Example:𝑛 = 𝑂(𝑛2) ? yes ! 𝑛 ≤ 𝑛2 when 𝑛 ≥ 1.𝑛 = 𝑂(𝑛2 − 10000) ? Yes ! 𝑛 ≤ 𝑛2 when 𝑛 ≥ 1000.𝑛 = 𝑂(𝑛2 − 1000000000000) ? Yes !


Big O notation




Example:𝑛 = 𝑂(𝑛2) ? yes ! 𝑛 ≤ 𝑛2 when 𝑛 ≥ 1.𝑛 = 𝑂(𝑛2 − 10000) ? Yes ! 𝑛 ≤ 𝑛2 when 𝑛 ≥ 1000.𝑛 = 𝑂(𝑛2 − 1000000000000) ? Yes ! 𝒏 = 𝑶(𝐥𝐨𝐠𝒏) ? No !

Proof: for any constant 𝒄, there exists some 𝒏𝟎depending on 𝒄, such that 𝒏 > 𝒄𝐥𝐨𝐠𝒏 when 𝒏 > 𝒏𝟎


Big O notation




(Def) 𝛀-notation:

𝑶(𝒈(𝒏)) = {𝒇(𝒏): there exist positive constants 𝒄 and 𝒏𝟎 such that 𝟎 ≤ 𝒄𝒈 𝒏 ≤ 𝒇(𝒏) for all 𝒏 ≥ 𝒏𝟎}.

Asymptotic lower bound


Big O notation




(Def) Ω-notation:

𝑂(𝑔(𝑛)) = {𝑓(𝑛): there exist positive constants 𝑐 and 𝑛0 such that 0 ≤ 𝑐𝑔 𝑛 ≤ 𝑓(𝑛) for all 𝑛 ≥ 𝑛0}.

Asymptotic tight bound

(Def) 𝚯-notation:

𝑶(𝒈(𝒏)) = {𝒇(𝒏): there exist positive constants 𝒄𝟏, 𝒄𝟐 and 𝒏𝟎 such that 𝟎 ≤ 𝒄𝟏𝒈 𝒏 ≤ 𝒇(𝒏) ≤ 𝒄𝟐𝒈 𝒏 for all 𝒏 ≥ 𝒏𝟎}.


Big O notation


Practice:For any two functions 𝑓(𝑛) and 𝑔(𝑛), we have 𝑓(𝑛) = Θ(𝑔(𝑛)) if and only if 𝑓(𝑛) = 𝑂(𝑔(𝑛)) and 𝑓(𝑛) = Ω(𝑔(𝑛)).

Practice:

𝑓 𝑛 = 𝑂 𝑔 𝑛 ⇔ 𝑔(𝑛) = Ω(𝑓(𝑛))

• Logarithms

Big O notation


log 𝑛 = log2 𝑛lg 𝑛 = log10 𝑛ln 𝑛 = log𝑒 𝑛

This is different from the notations used in the textbook. (ref. Chapter 3.2)

Practice: 𝑓 𝑛 = 𝑂 log 𝑛 ⇔ 𝑓 𝑛 = 𝑂 lg 𝑛


Big O notation


Practice:Find the asymptotical order for the following terms.

𝑓1(𝑛) = 𝑎𝑘𝑛𝑘 + 𝑎𝑘−1𝑛

𝑘−1 +⋯+ 𝑎1𝑛 + 𝑎0 (𝑎𝑖 > 0)𝑓2(𝑛) = 𝑏𝑛 (𝑏 > 1)𝑓3(𝑛) = log𝑐 𝑛 (𝑐 > 1)𝑓4(𝑛) = 𝑛!𝑓5(𝑛) = 𝑛𝑛

𝑓3 < 𝑓1 < 𝑓2 < 𝑓4 < 𝑓5

(ref. Chapter 3.2)

• Sorting Problem

Sorting


Input: a sequence 𝑎1, … , 𝑎𝑛 of numbers

Output: a permutation 𝑎1′ , … , 𝑎𝑛

′ such that 𝑎1′≤ 𝑎2

′ ≤ ⋯ ,≤ 𝑎𝑛′

Example:Input: 0, 9, 2, 1,Output: 0, 1, 2, 9

• Insertion Sort• Algorithm design and Implementation

• Consider the elements one by one.

• Insert the considered element into an appropriate position.

Sorting



• Consider the elements one by one.

• Insert the considered element into an appropriate position.

Sorting


Example:Input: 0, 9, 2, 1Step 1: 0

Step 2: 0, 9Step 3: 0, 2, 9Step 4: 0, 1, 2, 9

Output: 0, 1, 2, 9


Sorting


Pseudocode:

INSERTION-SORT(𝐴, 𝑛)1 for 𝑖 = 2 to 𝑛2 𝑘𝑒𝑦 = 𝐴[𝑖]3 𝑗 = 𝑖 − 14 while 𝑗 > 0 and 𝐴[𝑗] > 𝑘𝑒𝑦5 𝐴[𝑗 + 1] = 𝐴[𝑗]6 𝑗 = 𝑗 − 17 end8 𝐴[𝑗 + 1] = 𝑘𝑒𝑦9 end

• Insertion Sort• Correctness

• Sometimes the correctness is obvious but hard to prove.

Sorting




Sorting


𝑎1, 𝑎2, … , 𝑎𝑘 , 𝑎𝑘+1, 𝑎𝑘+2, … , 𝑎𝑛

𝑎1′ , 𝑎2

′ , … , 𝑎𝑘′ , 𝑎𝑘+1, 𝑎𝑘+2, … , 𝑎𝑛

′



Sorting


𝑎1, 𝑎2, … , 𝑎𝑘 , 𝑎𝑘+1, 𝑎𝑘+2, … , 𝑎𝑛

𝑎1′ , 𝑎2

′ , … , 𝑎𝑘′ , 𝑎𝑘+1, 𝑎𝑘+2, … , 𝑎𝑛

′

𝑎1′′, 𝑎2

′′, … , 𝑎𝑘′′ , 𝑎𝑘+1

′′ , 𝑎𝑘+2… , 𝑎𝑛′

If 𝑎1′ , 𝑎2

′ , … , 𝑎𝑘′ are correctly

sorted, after inserting 𝑎𝑘+1, 𝑎1′′, 𝑎2

′′, … , 𝑎𝑘+1′′ are also

correctly sorted.

The correctness of insertion sort follows inductively.

• Insertion Sort• Time Complexity: 𝑶(𝒏𝟐)

Sorting


Pseudocode:

INSERTION-SORT(𝐴, 𝑛)1 for 𝑖 = 2 to 𝑛2 𝑘𝑒𝑦 = 𝐴[𝑖] 𝑂(1)3 𝑗 = 𝑖 − 1 𝑂(1)4 while 𝑗 > 0 and 𝐴[𝑗] > 𝑘𝑒𝑦 𝑂(1)5 𝐴[𝑗 + 1] = 𝐴[𝑗] 𝑂(1)6 𝑗 = 𝑗 − 1 𝑂(1)7 end8 𝐴[𝑗 + 1] = 𝑘𝑒𝑦 𝑂(1)9 end

𝑂(𝑛)Θ(𝑛)

• Merge Sort

Sorting


• Merge Sort• Algorithm design and Implementation:

• Divide the input into two subarrays.

• Sort the subarrays recursively.

• Combine the subarrays.

Sorting


Example:Input: 0, 9, 2, 1

0, 9 2, 10 9 1 20, 9 1, 2

0, 1, 2, 9Output: 0, 1, 2, 9

• Merge Sort• Algorithm design and Implementation

Sorting


Pseudocode:

MERGE-SORT(𝐴, 𝑝, 𝑟)1 if 𝑝 < 𝑟2 𝑞 = ⌊(𝑝 + 𝑟)/2⌋3 MERGE-SORT(𝐴, 𝑝, 𝑞)4 MERGE-SORT(𝐴, 𝑞 + 1, 𝑟)5 MERGE(𝐴, 𝑝, 𝑞, 𝑟)

For two sorted arrays, MERGE can be done in Θ(𝑛)

• Merge Sort• Correctness

Sorting


Pseudocode:

MERGE-SORT(𝐴, 𝑝, 𝑟)1 if 𝑝 < 𝑟2 𝑞 = ⌊(𝑝 + 𝑟)/2⌋3 MERGE-SORT(𝐴, 𝑝, 𝑞)4 MERGE-SORT(𝐴, 𝑞 + 1, 𝑟)5 MERGE(𝐴, 𝑝, 𝑞, 𝑟)

Invariant: If MERGE-SORT works on subarrays, then it works on the whole array.

• Merge Sort• Time Complexity

Sorting


Pseudocode:

MERGE-SORT(𝐴, 𝑝, 𝑟) 𝑇(𝑛)1 if 𝑝 < 𝑟 𝑂(1)2 𝑞 = ⌊(𝑝 + 𝑟)/2⌋ 𝑂(1)3 MERGE-SORT(𝐴, 𝑝, 𝑞) 𝑇(𝑛/2)4 MERGE-SORT(𝐴, 𝑞 + 1, 𝑟) 𝑇(𝑛/2)5 MERGE(𝐴, 𝑝, 𝑞, 𝑟) Θ(𝑛)

For two sorted arrays, MERGE can be done in Θ(𝑛)

𝑇(𝑛) = 2𝑇(𝑛/2) + Θ(𝑛)

• Solve Recurrence• 𝑇(𝑛) = 2𝑇(𝑛/2) + Θ(𝑛)

• Assume 𝑛 = 2𝑘 and 𝑇(1) = Θ(1)

• Consider 𝑇(𝑛) = 2𝑇(𝑛/2) + 𝑐𝑛, where 𝑐 > 0 is a constant.

Sorting


• Solve recurrence by recursion tree𝑇(𝑛) = 2𝑇(𝑛/2) + 𝑐𝑛

Sorting


𝑐𝑛

𝑇(𝑛

2) 𝑇(

𝑛

2)


Sorting


𝑐𝑛

𝑐𝑛/2 𝑐𝑛/2

𝑇(𝑛

4) 𝑇(

𝑛

4) 𝑇(

𝑛

4) 𝑇(

𝑛

4)


Sorting


𝑐𝑛 𝑐𝑛

𝑐𝑛/2 𝑐𝑛/2 𝑐𝑛

𝑐𝑛/4 𝑐𝑛/4 𝑐𝑛/4 𝑐𝑛/4 𝑐𝑛

Θ(1) 𝑛Θ(1)


Sorting


𝑐𝑛 𝑐𝑛

𝑐𝑛/2 𝑐𝑛/2 𝑐𝑛

𝑐𝑛/4 𝑐𝑛/4 𝑐𝑛/4 𝑐𝑛/4 𝑐𝑛

Θ(1) 𝑛Θ(1)

𝑘 levels

(𝑛 = 2𝑘)

Totally, Θ 𝑛𝑐𝑘 = Θ(𝑛 log 𝑛)

log 𝑛 = log2𝑛

lg 𝑛 = log10𝑛

ln 𝑛 = loge𝑛

Algorithm Design and Analysisudel.edu/~amotong/teaching/algorithm design... · (Design) Algorithm...

Documents

Transcript of Algorithm Design and Analysisudel.edu/~amotong/teaching/algorithm design... · (Design) Algorithm...