PowerPoint Presentation

Chapter 3The FundamentalsChapter 3.1Algorithms

What is an algorithm?An algorithm is a finite set of precise instructions for performing a computation or for solving a problemPseudocode (T: Szde Kod)Describing an algorithm by using a specific computer language: Complex instructions and difficult to understand. Intemadiate step between Natural Language & Programming Language

3Algorithm 1: Maximum elementAlgorithm for finding the maximum element in a list:procedure max (a1, a2, , an: integers)max := a1for i := 2 to nif max < ai then max := ai{max is the largest element}

How long does this take?If the list has n elements, worst case scenario is that it takes n steps4Algorithm 2: Linear searchGiven a list, find a specific element in the listList does NOT have to be sorted!

procedure linear_search (x: integer; a1, a2, , an: integers)i := 1while ( i n and x ai )i := i + 1if i n then location := ielse location := 0

{location is the subscript of the term that equals x, or it is 0 if x is not found}

5Algorithm 3: Binary searchGiven a list, find a specific element in the listList MUST be sorted!Each time it iterates through, it cuts the list in half

procedure binary_search (x: integer; a1, a2, , an: increasing integers)i := 1{ i is left endpoint of search interval }j := n{ j is right endpoint of search interval }while i < jbeginm := (i+j)/2{ m is the point in the middle }if x > am then i := m+1else j := mendif x = ai then location := ielse location := 0

{location is the subscript that equals x, or it is 0 if x is not found}6Binary search running timeHow long does this take (worst case)?

If the list has 8 elementsIt takes 3 stepsIf the list has 16 elementsIt takes 4 stepsIf the list has 64 elementsIt takes 6 steps

If the list has n elementsIt takes log2 n steps7

Sorting AlgorithmsSort a sequence A for a given order criteriaBubble Sort, Insertion Sort, ..

Chapter 3.2 3.3

Growth of FunctionsComplexity of Algorithms9How does one measure algorithmsWe can measure time how long it takes a computerWhat if the computer is doing other things?And what happens if you get a faster computer?A 3 Ghz Windows machine chip will run an algorithm at a different speed than a 3 Ghz MacintoshWe can measure how many machine instructions an algorithm takesDifferent CPUs will require different amount of machine instructions for the same algorithm

We can loosely define a step as a single computer operationA comparison, an assignment, etc.Regardless of how many machine instructions it translates intoAn efficient algorithm on a slow computer will always beat an inefficient algorithm on a fast computer10Binary Search running timeThe binary search takes log2n steps

Lets say the binary search takes the following number of steps on specific CPUs:Intel Pentium IV CPU: 58* log2n /2Motorola CPU: 84.4*(log2n + 1)/2Intel Pentium V CPU: 44*(log2n)/2

Notice that each has an log2n termAs n increases, the other terms will drop out

As processors change, the constants will always changeThe exponent on n will not11Big-Oh notationIf f(x) and g(x) are two functions of a single variable, the statement f(x)O(g(x)) means thatkR, cR, xR, xk 0f(x)|cg(x)|.

Informallyc g(x) is greater than f(x) for sufficiently large x.f(x) grows no faster than g(x), as x gets large.How proof goesNeed to find k and c to show f(x)O(g(x)).Conventionally people use f(x)O(g(x)).12Big-Oh proofs13Big-Oh proofs

14A variant of the last questionShow that f(x) = 3x+7 is O(x2)In other words, show that 3x+7 c*x2

15What that meansIf a function is O(x)Then it is also O(x2)And it is also O(x3)

Meaning a O(x) function will grow at a slower or equal to the rate x, x2, x3, etc.16Function growth ratesFor input size n = 1000

O(1)1O(log n)10O(n)103O(n log n)104O(n2)106O(n3)109O(n4)1012O(nc)103*cc is a consant2n10301n!102568nn103000

Many interesting problems fall into this category17Properties of Big-OIf f is O(g) and g is O(f) then O(f) = O(g)Big O, as a relation, is transitive: f is O(g) and g is O(h), then f is O(h)The set O(g) is closed under addition:If f is O(g) and h is O(g) then f + h is O(g)The set O(g) is closed under multiplication by a scalar a (real number):If f is O(g) then af is O(g)

Theorem If f1 is O(g1) and f2 is O(g2) theni) f1f2 is O(g1g2)ii) f1 + f2 is O(max{ g1, g2})

Proof of iiThere is a k1 and C1 such that1. f1(n) < C1g1(n) when n > k1.There is a k2 and C2 such that2. f2(n) < C2g2(n) when n > k2.We must find a k3 and C3 such that3. f1(n)f2(n) < C3g1(n)g2(n) when n > k3

Proof (cont.)We use the inequalityif 0 < a < b and 0 < c < d then ac < bdto conclude thatf1(n)f2(n) < C1C2g1(n)g2(n)as long as k > max{k1, k2} so that both inequalities 1 and 2 hold at the same time.Therefore, chooseC3 = C1C2 andk3 = max{k1, k2}.

Big and notationIf f(x) and g(x) are two functions of a single variable, the statement f(x)(g(x)) means thatkR, cR, xR, xk f(x)| c g(x)|.Informally, f(x) is greater than c g(x) for sufficiently large x.f(x) = 8x3 + 5x2 + 7 is (x3).since 8x3 + 5x2 + 7 > 8x3 for all x > 0.f(x)(g(x)) is equivalent to g(x)O(f(x)).

f(x) = (g(x)) if f(x) = O(g(x)) and f(x) = (g(x)).We can find c1 and c2 such that c1 |g(x)| |f(x)| c2 |g(x)|.3x2+ 8xlog x = (x2).3x2+8xlog x < 3x2+8x2 = 11x2 for x > 1 3x2+8xlog x =O(x2).3x2+8xlog x > x2 for x > 1. 3x2+8xlog x =(x2).Little bit of ComplexityEfficiency of your algorithm? Given input size,Time complexity: Time used by your computer to solve the problemWe use number of operationsMemory (space) complexity: Memory used by your computer to solve the problemWe use number of variables allocatedExampleFind the maximal element in a setmax = a1, i = 2While (i n)if (max < ai) max = aii++Time complexity: 2 (n 1)+1 comparisons (one to determine at the end of the list has not been reached, another to determine whether to update max)= (n)Memory complexity: 2 = (1)

Example 2: Linear Searchprocedure linear search (x: integer, a1, a2, , an: distinct integers)i := 1t1while (i n and x ai)t2 i := i + 1t3 if i n then location := it4 else location := 0t5 return locationt6

Linear search analysisWorst case time complexity order:

Best case:

Average case, if item is present:

Understanding the complexity of AlgorithmsTractable ProblemsPolynomial worst-case complexity (P Class)(nc), c 1 constant [Eg.Bubble Sort Alg. is (n2)] Intractable Problems Unsolvable ProblemsNo algorithms exists for solving themHalting Problem Class NP: Problems which a solution can be checked in polynomial timeClass NP-complete: any of these problems can be solved in polynomial time, then problems in class NP can be solved

Chapter 3.4

Integers and Division27DivisionDef: a (a0) divides b if c such that b = ac.a | b: a divides b3 | 7? 3 | 12?3 | 0? 0 | 3?

Theorem: Let a, b, c be integers. Thena | b, a | c a | (b + c).a | b a | bc integer c.a | b, b | c a | c.DivisionLet a, b, c be integers. Thena | b, a | c a | (b + c).DivisionLet a, b, c be integers. Thena | b a | bc integer c.

Proof Suppose a|b. By definition, there is a number n such that b = an. Multiply both sides by c to get bc = anc = a (nc ). Consequently, bc has been expressed as a times the integer nc so by definition of |, a|bc DivisionLet a, b, c be integers. Thena | b, b | c a | c.Divisiona | b and b | c a | (mb + nc) for all integer m, n.

Chapter 3.5

Primes and Greatest Common Divisors33Showing a number is primeShow that 113 is prime

SolutionThe only prime factors less than 113 = 10.63 are 2, 3, 5, and 7Neither of these divide 113 evenlyThus, by the fundamental theorem of arithmetic, 113 must be prime34Primes are infiniteTheorem (Euclid): There are infinitely many prime numbers(Proof) Proof by contradictionAssume there are a finite number of primesList them as follows: p1, p2 , pn.Consider the number q = p1p2 pn + 1

Since we have only finite number of primes and q is not one of them, pi should divide q for some i.Therefore, pi | (q - p1p2 pn) = 1. 35The prime number theoremThe radio of the number of primes not exceeding x and x/ln(x) approaches 1 as x grows without boundRephrased: the number of prime numbers less than x is approximately x/ln(x)

When x = 2512, # of primes = 2512/512 2503

36Greatest common divisorThe greatest common divisor of two integers a and b is the largest integer d such that : d | a and d | bDenoted by gcd(a,b)

Examplesgcd (24, 36) = 12gcd (17, 22) = 1gcd (100, 17) = 137Relative primesTwo numbers are relatively prime if they dont have any common factors (other than 1)Rephrased: a and b are relatively prime if gcd (a,b) = 1

gcd (25, 39) = 1, so 25 and 39 are relatively prime38Pairwise relative primeA set of integers a1, a2, an are pairwise relatively prime if, for all pairs of numbers, they are relatively primeFormally: The integers a1, a2, an are pairwise relatively prime if gcd(ai, aj) = 1 whenever 1 i < j n.

Example: are 10, 17, and 21 pairwise relatively prime?gcd(10,17) = 1, gcd (17, 21) = 1, and gcd (21, 10) = 1Thus, they are pairwise relatively primeExample: are 10, 19, and 24 pairwise relatively prime?Since gcd(10,24) 1, they are not39More on gcdsGiven two numbers a and b, rewrite them as:

Example: gcd (120, 500)120 = 23*3*5 = 23*31*51500 = 22*53 = 22*30*53

Then compute the gcd by the following formula:Example: gcd(120,500) = 2min(3,2)3min(1,0)5min(1,3) = 223051 = 20

40Least common multipleThe least common multiple of the positive integers a and b is the smallest positive integer that is divisible by both a and b.Denoted by lcm (a, b)

Example: lcm(10, 25) = 50What is lcm (95256, 432)?95256 = 233572, 432=2433lcm (233572, 2433) = 2max(3,4)3max(5,3)7max(2,0) = 243572 = 190512

41lcm and gcd theoremLet a and b be positive integers. Then a*b = gcd(a,b) * lcm (a, b)

Example: gcd (10,25) = 5, lcm (10,25) = 5010*25 = 5*50

Example: gcd (95256, 432) = 216, lcm (95256, 432) = 19051295256*432 = 216*19051242

Chapter 3.6

Integers and Algorithms43Eucledean AlgorithmFor all integers a, b,gcd(a, b) = gcd((a mod b), b).

Sort a,b so that a>b, and then (given b>1) (a mod b) < a, so problem is simplified.

Example: gcd (287, 91)=?287 mod 91 = 14 (287=91*3+14) gcd (91, 14) 91 mod 14 = 7 (91=14*6+7) gcd (14, 7) gcd (287, 91)= gcd (14, 7) = 7

44Eucledean AlgorithmLet a = b q + r, where a, b, q, r be integers. Then gcd (a, b) = gcd (b, r).Proof :Suppose d divides both a and bThen d divides a-bq = rHence, any common divisor a and b is also a common divisor of b and rLikewise, suppose d divides both b and rThen d divides bq + r = aHence, any common divisor b and r is also a common divisor of b and aConsequently, gcd (a,b) = gcd (b,r)

45Eucledian Algorithmprocedure gcd (a, b: positive integer)x := a, y := bwhile y 0r := x mod yx := yy := rgcd (a, b) = x