Data structures and algorithm analysis (3rd ed) by mark allen weiss (p2)

Bismillah hir Rehman nir Raheem ------------^^^^^^^^^^----------Assalat o Wasalam o Allika Ya RasoolALLAH

To Read Online & Download: WWW.ISSUU.COM/SHEIKHUHASSAN

Data Structures and Algorithm

Analysis in C++

(3rd Ed)

Mark Allen Weiss

Ch. #7 to Ch. #12 + Appendix

Published By: Muhammad Hassan Riaz Yousufi

Third Edition

Data

++

Mark Allen Weiss Florida International University

BOSj-OlI San Francisco New York London Toronto Sydney 'j()kyo SingJ.pore fvl:!drid

Mexico City Munich P:lris C:lpC Ti.)wll Hong Kong Montreal

Puhlisher Greg ]()bin Sellior ;\cquhiliolls Fe/itm lVlichacl llirsch Prodllctioll SHptTVisOl I\'tarilyn Lloyd Ldilorial Assis/m1/ ljnclscy Triche! Cover Di'si,~11 Mll1agcl Joyce (j)scntino 'vVclls Cover Design Night & Day Design Cover Ima)!.e (.0 Pele (~ardnLT/Digital Vision fdarl1eling Jvlal1(/)!.Cf Michelle Brown Mar]u>lillg i\SSlsttllll Dana! .opreato Project Mal1ag,emclll. vVindfal1 Software Composition \Vindfall Software, using Zill:.X Tcc/miwlllluslralioll Ccorge Nichols J>r()(~rrc(/(lfl tVIaryEllen N. Oliver 1m/excr Ted LUllX PrCfm:ss and Mmudtic/liring, Caroline j;cll Prinler Courier Stoughton, Ine.

Access the latest information about Addison-\Veslcy liLIes from Our \Vorld \Vide Vv'ch sitt: http://www.aw-hc.com/computing

rvlany of the designations used by manufacturers and sellers to distinguish their products arc claimed as trademarks. \Vhcrc those designalions appear in this book, and Addison-\Vcslcy was aware of a trademark claim, the designations have bcen printed in initial caps or all caps.

J.ibrary of Congress Cataloging-in-Pu\)liGllion l)ala \\leiss, Mark Allen.

Data structures and algorithm analysis ill C++ I ivlark Allen \Vdss.~Jrd cd. p. Clll.

Includes bibliographical rderences and index. ISBN 0-32 I A4146-X (aIle paper)

1. C++ (Computer program language) 2. Data structures (Computer sciencr) 3. Computer algorithms. l. Title.

QA76.Tl.USlW46200') 00').1 fl-den

Copyright

Preface xv

Chapter 1 Introduction 1.1 What's the Book About' 1,2 Mathematics Review 2

1.2.1 Exponents >3 12.2 Logarithms >3 1.2.3 Series 4 1.2.4 Modular Arithmetic ') 1.2.5 The P Word 6

1,3 A Brief Introduction to Recursion 7 I .4 C++ Classes II

1.4.1 Basic class Syntax 12 1.4.2 Extra Constructor Syntax and Accessors 12 1.4.3 Separation of Interf~Kc and Implementation 15 1.4.4 vector and str; ng 17

1.5 C++ Details 19 1.5.1 Pointers 19 1.5.2 Paramerer Passing 21 1.5.3 Return Passing 22 1.5.4 Reference Variables 23 1.'5."5 The Big Three: Destructor, Copy Constructor, operator'" 23 1.5.6 C-stylc Arrays and Strings 26

1.6 Templates 29 1.6.1 function Icmplates 29 1.6.2 Class Templates 30 1.6.3 Object, Comparable, and an Example 32 1.6.4 function Objects 34 1.0.5 Separate Compilation of Class Templates 35

1.7 Using Matrices 37 1.7.1 The Data Members, Constructor, and Basic Accessors 37 1.7.2 operator[] 37 1.7.3 Destructor, Copy Assignment, Copy Constructor 39

1

vii

Bismillah hiir Rehman nir Raheem-----------------------------Assalat o Wasalam o Alika Ya RasoolALLAH


viii Contents

Summary 39 Exercises '_39 References 4 ]

Chapter 2 Algorithm Analysis 2.1 Mathematical Background 43 2.2 Model 46 2.3 What to Analyze 46 2.4 Running Time Calculations

2.4.1 A Simple Example 2.4.2 Cencral Rules 50

49 49

2.4.3 Solutions for the Maximum Subsequence Sum Prohlem 52 2.4.4 Logarithms in the Running Time 58 2.4.5 Checking Your Analysis 62 2.4.6 A C;rain 01 Salt 63 Sumrn

Chapter 4 Trees 4,1 Preliminaries 11'3

4. L I Implementation of]h'es 114 4.1.2 Tree Traversals \-\lith an Applicalion 11')

4.2 Binary'lin's 119 4.2. I Implementation 120 4.2.2 An Example: Expression Trees 121

4.3 The Search '1\'ee ADT-Binary Search Trees 124 43.1 contains 125 4.3.2 findMinandfindMax 125 4.3.3 insert 129 4.3.4 remove 130 4.3.5 Destructor and Copy Assignment Operator 132 4.3.6 Averag('~Casc Analysis 133

4.4 AVL Trees J 36 4.4.1 Single Rotation 139 4.4.2 Double Rotation 142

4.5 Splay'hees 149 4,5,1 A Simple Idea (That Docs Not Work) 150

4.6 4.7 4.R

4'5.2 Splaying 1'52 Tree 'haversals (I

x Contents

5.6 Hash 1ilbles in the Standard Library 204 5.7 Extendible Hashing 204

Summary 207 Exercises 20S References 211

Chapter 6 Priority Queues (Heaps) 0.1 Model 213 6.2 Simple Implementations 214 6.3 Binary Hear 215

6.4

6.5 6.6

6.7

6.3.1 Structure Property 21 'j 6.3.2 Heap-Order Property 216 6.3.3 Ilasie Heap Operations 217 6.3.4 Other Heap Operations 220 Applications of Priority Queues 6.4.1 The Selection Prohlem 6.4.2 Event Simulation 227 d-Heaps 228 Leftist Heaps 229 6.6.1 Leftist Heap Property 6.6.2 Leftist Heap Operations Skew Heaps 235

225 226

229 230

6.S Binomial Queues 239 6.8.1 Binomial Queue Structure 240 6.8.2 Binomial Queue Operations 241 6.S.3 Implementation of Binomial Queues 244

6.9 Priority Queues in the Standard Library 251 Summary 251 Exercises 251 References 257

Chapter 7 Sorting 7.1 Preliminaries 261 7.2 Insertion Son 262

7.2.1 The Algorithm 262 7.2.2 STI. Implementation of Insertion Sort 263 7.2.3 Analysis of Insertion Sort 264

7.3 A Lower Bound for Simple SOIling AlgOrithms 265

213

261



Contents

7'j ShcllsorL 266 7.4.1 Worst-Case Analysis of Shcllsort 268

7.5 1 lcapson 270 7.5.1 ;\nalysis of Ilcapsort 272

7.6 TV!crgcsort 274 7.6.1 Analysis ofiv1crgcsort 276

7.7 Quicksort 279 7.7.1 Picking the Pivot 280 7.7.2 Partitioning Strategy 2B2 7.7.3 Small Arrays 2B4 7.7.4 J\clLlai Quicksort P.outincs 2B4 7.7. 'j AnalysIs of Quicksort 287 7.7.6 A Lillcar-FxpcC1.cd-Timc Algorithm for Selection 2YO

7.B Indirect Sorting 292 7.R.l vector Docs Not \Vork 295 7.8.2 Smart Pointer Class 20'5 7.8.3 Overloading operator< 29'3 7.8.4 Dcrcferellcing a Pointer \Vilh * 29'5 7.8. '5 ()verloading the -TYpe Conversion Operator 29.'5 7.8.6 Implicit rypc Conversions Arc Evcl")!\vhcrc 296 7.B.7 Dual-Direction Implicit Conversiulls em Cause Ambiguities 296 7.8.8 Pointer Subtraction Is Legal 207

7.1) A Ceneral Lower Bound for Sorting 297 7.9.1 Decision'ih:cs 297

7.10 Gucket SOrL 299 7.1 1 External Sorting JOO

7.11. I \Vhy \Vc Need New Algorit hms JOO 7.11.2 ivlodel for External SOrLinglOO 7.1 Ll The Simple Algorithm lOI 7.11.4 Multiway Mergc )02 7.11.5 Polyphase Merge 303 7.1l.6 Replacement Selection 304 SUlllrnary 30'5 Exercises 306 References 31 I

Chapter 8 The Disjoint Set Class B. I Equivalence Febt iOlls ~31 '5 B.2 The Dynamic Equi"alence Prohkrn :, t 6 R.J Basic Data Structure ")17

315

xi Bismillah hiir Rehman nir Raheem-----------------------------Assalat o Wasalam o Alika Ya RasoolALLAH


xii Contents

8.4 Smart Union Algorithms 321 8. '5 Path Compression 324 8.6 Worst Case for Union-by-Rank and Path Compression 32'5

8.6. I Analysis 01 the Union/Find Algorithml26 8.7 An Application 331

Summary 334 Exercises 33'5 References 336

Chapter 9 Graph Algorithms 9. [ Definitions 339

9.1.1 Representation of Graphs 340 9.2 Topological Sort 342 9.3 Shortest-Path Algorithms 345

9.3.1 Unweighted Shortest Paths'l47 9.3.2 Dijkstras Algorithm 351 9.3.3 Graphs with Negative Edge Costs 360 'U.4 Acyclic Graphs 360 9.3.5 All-Pairs Shortest Path ')64 9. ').6 Shortest Path Example16'j

9.4 Network Flow Problems 367 9.4.1 A Simple Maximum-Flow Algorithm 367

9_5 Minimum Spanning Tree 372 9.5. I Prilll' Algorithm 373 9.5.2 Kruskals Algorithm 376

9.6 Applications o{ Depth-First Search 378 9.6.1 Undirected Graphs 379 9.6.2 I3iconncctivity 381 9.6.3 Euler Circuits 385 9.6.4 Directed Craphs 388 9.6.5 f'inding Strong Componenls 390

9.7 Introduction to NP-Completcness 392 9.7.1 Easy vs. Hard 392 9.7.2 The Class NP 393 9.7.3 NP-Complete Problems 394 Summary 396 Exercises ]96 References 404

339



Chapter 10 Algorithm Design Techniques 10.1 C;reedy Algorithms 409

10. LI A Simple Scheduling Problem 410 10.1.2 IlulTman Codes 411 ! 0.1.3 Approximate Bin Packing 4 J 9

10.2 Divide and Conquer 427 10.2.1 Running Time of Divide and Conquer Algorithms 428 10.2.2 Closest-Points Problem 430 10.2.3 The Selection Problem 435 10.2.4 Theoretical Improvements for Arithmetic Problems 438

[0.:3 Dynamic Pmgramming 442 H1.'3..\ Using a ~1~lblc Instead of Recursion 442 lO.l.2 Ordering Matrix Multiplications 444 10.1.1 Optimal Binary Searehliec 447 10.3.4 Ail-PaIrS Shortest Path 451

10.4 Randomized Algorithms 4')4 ! O.4.l Random Numher CeneraLors 4'5'5 10.4.2 Skip Lists 459 10.4.1 Primality Testing 461

10. 'j Backt racking Algorithms 464 10.5.1 The Turnpike ReCOilS! ruction Problem 465 10.5.2 Cames 469 Summary 475 Exercises 475 References 485

Chapter 11 Amortized Analysis Il.l 11.2 11.3 11.4

1 j')

An Unrelated Puzzle 492 Binomial Queues 492 Skew Heaps 497 Fibonacci lleaps 499 11.4.1 Cultmg Nodes in Leftist Heaps 500 11.4.2 Lazy Merging for Binomial Queues 502 11.4.3 The Fibonacci 1 !cap Operations 506 1 1.4.4 Proof of the. Time Bound 506 Splay lilTS 509 SUlnmary 513 Exercises SU ReiCrences 5t')

Contents XIII

409

491



XIV Lomems

Chapter 12 Advanced Data Structures and Implementation

12.1 l(lpDoWIl Splay 11\:c5 'j 17 122 RcdBlack Iiccs 525

J 2.2.1 Bottom-Up Insertion '526 122.2 TopDown ReelG\ack 11ce5 '527 12.2. 3 Top~f)o\Vn Deletion "531

12.3 Deterministic Skip Lists .'535 12.4 ;\i\;liTes '340 1 2. '5 /ITap5'54 7 12.6 hel Trecs 549 12.7 Fairing lIeaps '553

Summary '558 Exercises 558 References '56:)

Appendix A Separate Compilation of Class Templates AI Everything in the Header 568 1\.2 Explicit instantiation ')(jt) A.3 The cxport Directive ')70

Index 571

517

567



PT E R 7

Sorting

In this chapter we discuss the problem of sorting an array of dements. To simplify matters, we will assume in our examples that the array contains only integers, although our code will once again aHow more general objects. for most of this chapter, we will also assume that the entire sort can be done in main memory, so that the number of elements is relatively small (less than a few million). Sorts that cannot be performed in main memory and must be done on disk or tape are also quill' important. This type of sorting, known as external sorting, will be discussed at the end of the chapter.

(Jur investigation of internal soning will show that There arc several casy algorithms to sort in O(N2), such as insertion sort. There is an algorithm, ShelisorL, that is vcry simple to code, runs in o(N2), and is efficient in practice. There arc slightly more complicated O(N log N) sorting algorithms. Any general-purpose sorting algorithm requires n (N log N) comparisons. The rest of this chapter will describe and analyze the various sorting algorithms.

These algorithms contain interesting and important ideas for code optimization as well as algorithm design. Sorting is also an example where the analysis can be precisely performed. Be forewarne.d that where appropriate., we will do as much analysis as possihle.

7.1 Preliminaries The algorithms we descrihe will all he interchangeable. Each will be passed an array containing the clements; we assume all array positions contain data to be sorted. We will assume that N is the number of clements passed to our sorting routines.

We will also assume the existence of the" )) operators, which can be used to place a consistent ordering on the input. Besides the aSSignment operator, these are the only operations allowed on the input. elata. Sorting under these conditions is known as comparison-based SOIling.

This interface is not the same as in the STL sorting algorithms. In the STI.., sorting is accornpiishcd by usc of the function template sort. The parameters to sort represent the start and endmarkcr of a (range in a) container, and an optional comparator:

void sort( Iterator begin, Iterator end ); void sort( Iterator begin, Iterator end, Comparator cmp ); 261



262 Chapter 7 Sorting

The itcrators must suppon random access. Th~ sort algorithm docs not guarantee that equal items retain their original order (if thai is important, use stabl e_ sort instead of sort).

As an example, in

sort( v.begin( ), v.end( ) ); sort( v.begin( ). v.end( ), greater( ) ); sort( v.begin( ), v.begin( ) + ( v.end( ) - v.begin( ) ) 12 );

the first call sorts the entire container, v, in nondecreasing order. The second call sorts the entire container in nonincreasing order. The third call sorts the first half of the container in nondccrcasing order.

"l'he sorting algorithm llsed is generally quicksort, which we describe in SeClion 7.7. In Sectioll 7.2, we implement the simplest sorting algorithm using both our style of passing the array of comparable items which yields the most straightforward code, and the interface supported by the STL, which requires more code.

7.2 Insertion Sort 7.2.1 The Algorithm One of the simplest soning algorithms is the insertion sort. 1nscnion sort consists of N - 1 passes. For pass p = I through N - 1, insertion sort ensures that the elements in positions 0 through p afe in sorted order. Insertion sort makes usc of the fact that elements in positions o through p - I arc already known to he in sorted order. hgure 7.1 shows a sample array after each pass of insertion sort.

Figure 7.1 shows the general strategy. In pass p, we move the element in position p left until its correct place is found among the flrst p + 1 elements. The code in Figure 7.2 implements this strategy. Lines 11 through 14 implement that data movement without the explicit usc of swaps. The clement in pOSition p is saved in tmp, and aJllargn elements (prior 10 position p) afC moved one spot to the right. Then tmp is placed in the correCI spot. This is the same technique that was used in the implementation of binary heaps.

()rigina1 34 8 64 51 32 21 Positions Moved

After fJ = 1 8 34 64 51 32 21 1 After p = 2 8 34 64 51 32 21 0 Arter p = 3 8 34 51 64 32 21 After p = 4 H 12 34 51 64 21 3 Aftcrp=5 8 21 12 34 51 64 4

Figure 7.1 lnsenion sort after each pass



1 /** 2 * Simple insertion sort. 3 *j ,1 temp 1 ate 5 void insertionSort( vector & a ) (, 7 i nt j; 8 9 fore int p '" 1; P < a.size( ); pH)

10 ( /1 Comparable tmp '" a[ p ]; 12 U 1'1 15 16

fore j p; j > 0 && tll1P < d[ j - 1 ]; j-- ) d[j]oa[j-l];

d[j]otIl1P;

Figure 7.2 lnscrtion sort routine

7.2.2 STL Implementation of Insertion Sort

7.2 Insertion Sort

In the STL, instead of having the sort routines take an array of com parablc items as a single parametcr, the sort routincs receive a pair of itcralors dwt represent the stan and endmarker of a range. A {wo-pararnetcr sort routine uses jusl that pair of itcrators, and presumes th


I template 2 void insertionSort{ const Iterator & begin, canst Iterator & end)

" '/ if( begin !'" end) 5 insertionSortHelp( begin, end, *begin ): 6 7 H 9 template

10 void insertionSortHelp( canst Iterator & begin. canst Iterator & end, 11 canst Object & obj ) 12 1.1 insertionSort{ begin, end. less( ) ); H

Figure 7.3 Two-parameter sort invokes three-parameter sort via a helper routine that establishes Object as a generic type

adding an additional generic type. This is shown in Figure 7.4, along with code that replaces array indexing with use of the iLcralor, and code that replaces calls to operator< with calls 10 the 1 essThan function ohject.

Observe that once we actually code the insertionSort algorithm, every statement in the original code is replaced with a corresponding statement in the new code that makes straightforward use of itcrators and the function object. The original code is arguably much simpler to read, which is why we use our simpler interface rather than the STL interface when coding our sorting algorithms.

7.2.3 Analysis of Insertion Sort TIecause of the nested loops, each of which can take N iterations, insertion sort is O(N2). Furthermore, this bound is tight, because input in reverse order can achieve this bound. A precise calculation shows that the number of clement comparisons in the inner loop in Figure 7.2 is at nlOS\ P + 1 fcn each value of p. Summing over all p gives a total of

N

I> = 2 + J + 4 + .. + N = ("leN') i=2

On the other hand, if the input is presorted, the running time is O(N), because the test in the inner for loop always fails immediately Indeed, if the input is almost sorted (this term will be more rigorously defined in the nexl section), insertion sort will run quickly Because of this wide variation, it is worth analYZing the average-case behavior of this algorithm. It turns out that the average case is C"')(N 2) for insertion sort, as well as for a variety of other sOrling algorithms, as the next section shows.



7.3 A lower Bound for Simple Sorting Algorithms

template 2 void insertionSort( canst Iterator & begin, canst Iterator & end, J Comparator lessThan ) 4 -'5 if( begin != end) 6 insertionSort( begin, end, lessThan. *begin ); 7 8 9 template

10 void insertionSort( canst Iterator & begin, canst Iterator & end, 11 Comparator lessThan, canst Object & obj ) 12 13 Iterator j; 1'1 15 fore Iterator p = begin+l; p !'" end; ++p ) 16 ( 17 18 19 20 21 22

Object tmp '" *p; fore j " p; j !" begin && lessThan( tmp, *( j-l ) ); --j )

*j " *(j-l); *j '" tmp;

Figure 7.4 Thrcc~parametcr sort invokes four~paramcter helper routine that establishes Object as it generic type

7.3 A Lower Bound for Simple Sorting Algorithms

An inversion in an array or numbers is any ordered pair (i,j) having the property that i < j but ali] > aUI. In the example of the last seetion, Ihe inplll lisl 34, 8, 64, 51, 32,21 had nine inversions, namely 04, 8), (34,32),04,21), (64, 51), (64, 32), (64, 21), (51, 32), ('51,21), and (32,21). Notice that this is exactly the number of swaps that needed to he (implicitly) performed by insertion sorl. This is always the case, hecause swapping two adjacent elements that are out of place removes exactly one inversion, and a sorted array has no inversions. Since there is O(N) other work involved in the algorithm, the running time of insertion sort is O(I + N), where 1 is the number of inversions in the original array. Thus, insertion sort runs in linear time if the numher of inversions is O(N).

We can compute precise bounds on the average running time of insertion sort by computing the average number of inversions in a permutation. As usual, defining (./vera~e is a difficult proposition. Vve \vill assume thai there are no duplicate elements (if we allow duplicates, it is not even clear what the avcruge number of duplicates is). Using this assumption, we can assume thaI the input is some permutation of the firsl N integers (since

265 Bismillah hiir Rehman nir Raheem-----------------------------Assalat o Wasalam o Alika Ya RasoolALLAH



only relaliv(' urdering is iln [){)r\ant) an(\l hal all afC equally likely. Under these assuillpl ions, we have the following theorem:

Theorem 7.1. The (]'Vl'Tagc number of inversions in an array of 1\1 distinct c1CI1WlltS is N(N - 1)/4. Proof. ror any list, L, of dements, consider LJ , the list in revcrse order. The rc\'crse list of the example is 21, :12, 51, 64, 8, 34. Consider any pair of two dements in the list (x,y), with y > x. Clearly, in exactly one of Land Lr this ordered pair represents an inversion. The total number of these pairs in a list L and its reverse I.) is N(N - 1)/2. Thus, an average list has half this amount, or N(N -- J)/4 inversions. This theorem implies that insertion son is quadratic 011 average. lL also provides a very

strong lower bound aboul any algorithm lhal only exchanges adjacent clements. Theorem 7.2. Any algorithm lhal sorts by exchanging adjacent elemenlS requires Q(Nl) timc on

aV(~rage.

Proof. Thc average' number of in vcrs ions is inil iall)' N(N - 1)/4 = Q (N L). Each swap removes only onc inversion, so Q (N 2) swaps arc required. This is an example of a lower-bound proof. It is valid nol only for inserlion son, \vhich

performs adjaccl1l exchanges impliciLly, but also for other simple algorithms such as bubble sort and selection son, which we will not ckscribc here. In fan, it is valid over an entire class of sort ing algorithms, including those undiscovered, that perform only adj,l(cnt exchanges. Because of this, this proof cannot he confirmed empirically. Although this lown~bouncl proof is rather simple, in general proving lower bounds is much more complicated tll for e\'cry i, we have ali!.:S ali + hill (where this makes sense); all clements spaced 111/ apart arc sorted. The file is



Original 81 94 II 96 12 35 17 95

After 5-sort 35 17 II 28 12 41 75 IS After 3~SOrL 28 12 II 35 15 41 58 17 After 1 ~sort II 12 I') 17 28 35 41 58

Figure 7.5 Shellsort after each pass

/** 2 * Shell sort, using Shell's (poor) increments. :3 *1 4 template 5 void shellsort{ vector & a 6 7 for( int gap = a.size( ) I 2; gap> 0; gap 1= 2 ) 8 for( int i '" gap; i < a.size( ); i++ ) 9

Comparable tmp = a[ i ]; i nt j '" i;

28

96 94 75

10 11 12 13 14 15 16 17

for( ; j >= gap && tmp < a[ j - gap]; j -= gap) a [ j ] = a [ j - gap ];

a[j]=tmp;

7.4 Shellsort

58 41 75

58 81 94 75 81 96 81 94 95

Figure 7.6 Shellsort routine using Shell's increments (better increments are possible)

then said to be hk-sortcd. For example, Figure 7.5 shows an array after several phases of Shellson. An important property of Shellsort (which we state without prooO is that an hh-sorted fIle that is then hk_l~sorted remains hvsorted. If this were not the case, the algorithm would likely be of little value, since work done by early phases would be undone by later phases.

The general strategy to hvsort is for each pOSition, i, in hI


The program in Figure 7.6 avoids the explicit usc of swaps in the sarllC manner as our implementation of insertion son.

7.4.1 Worst-Case Analysis of Shellsort Although Shells-on is simple to code, the analysis of its running lime is quite another story The running tirnc of Shellson depends on the choice of increment sequence, and the proofs can be rather involved. The average-case analysis of Shells-on is a long-standing open problem, except for the most trivial increment sequences. 'vVc will pnwc ligh! worst-case hounds for two particular increment sequences.

Theorem 7.3. The worst-case running tillle of Shellson, U;;illg Shell's increments, is H(N1). Proof. The proof requires showin,g not only an uPI)('}" hound on the worst-C(\Sl' rtll1!lillg !irne hut also showing that there exislS some input that actually takes [2(1\1 2 ) time to run. V/e prove the lower bound first, by constructing a had casco First, wc choose 1\1 to he a powcr of 2. This makes

7.4 Shellsort

fhe problell1 with SheWs increlllcnts is that pairs of increments arc not necessarily rcl-'-\lively prime, ,-\!lel thus the sl1l


The a\'cragr~case running time of Shcllsort, using Hibbard's increments, is thought to he O(N'")/4), based on simulations, but nobody has been able to prove this. Pratt has shown that the (':')(1\(3/2) hound applies to a wide range of increment sequences.

Sedgewick has proposed several Illcrement sequences thal give an O(N't/1) wo1'51-case running time (also achievahle). The average running time is conjectured to he OU..r7/6) for these increment sequences. FmpiricaJ studies show that these sequences per-form significantly hetter in practice than IIibbard's. The best of these is the sequence n, 5, "19,41, 109, . , .}, in which the terms afC either of the form 9 41 ~ 9 . 2J + I or 4i - 3 . i + 1. This is most easily implemented by placing these values in an array. This increment sequence is the best known in practice, although there is a lingering possibility that some increment sequence might exist that could give a significant improvement in the running time of ShellsorL

There aTC several other results on SheHsort that (generally) require difficult theorems from number theory and combinatorics and arc mainly of theoretical interest. Shellson is a finc example of a very simple algorithm with an extremely complex analysis.

The performance of She!!sort is quite acceptable in practice, even for N in the tens of thousands. The simplicity of the code makes it the algorithm of choice for sorting up to moderately large input.

7.5 Heapsort As mentioned in Chapter 0, priority queues can he Hsed to sort in O(N log N) time. The algorithm based on this idea is knmvll as hcapsort and gives the hes\ Big-Oh running time we have seen so far.

Recall, from Chapter 6, that the basic strategy is to build a binary heap of N elements. This stage takes O(N) time. \'Ve then perform N delete~lin operations. The elements leave the heap smallest first, in sorted order. By recording these elements in a second array anel then copying the array back, we sort. N clements. Since each del eteMi n takes O(log N) time, the towl running time is O(N log N).

The main problem with this algorithm is that it uses an extra array. Thus, the memory requirement is doubled. This could be a prohlem in somc instances. Notice that the extra time spent copying the second alTay back to the first is only O(N), so that this is not likely to ;:lITect the running time significant Iy. The problcm is space.

J\ clever way to avoid using a second array makes use of the fact that after each deleteMi n, the heap shrinks by 1. Thus the cell that VI.'(lS last. in the heap can he used to store the element that was just deleted. As an. example, suppose we have a heap with six clements. The first deleteMin produces (ll. Now the heap has only five elements, so we can place Cil in. position 6. The next deleteMin produces (/2. Since the heap will now only have four dements, \ve can place (/2 in position 5.

Using this strategy, after the last deleteMin the mray will contain the elements in decreasing sorted order. If \ve want the clements in the more typical inueasing sorted order, we can change the ordering property so that tht' parent has a larger element than the child. Thus we have a (nwx)heap.



I

IS Heapsort

In our implementation, we will use a (max-)JH'ap, hut avoid the actual ADT for the purposes of speed, As usual, everything is done ill an array. The first step builds the heap in linear time. 'vVe then perform N - i deleter~axcs by swapping the last clement in the heap with the flrst, decrementing the heap size, and percolating down. \Vhcn the algorithm terminates, the array contains the clements in sorted ordeL For instance, consider the input sequence J 1,41) 59, 26, 53, 58,97. The resulting heap is shown in Figure 7.8.

Figure 7.9 shows the heap Ibal resuits after the first deleteMax. As the figures imply, the last clement in the heap is 31; 97 has been placed in a pan of the heap array that is technically no longer pan of the heap. Arter:) more deleteMax opcnllions, the heap will

97

53 59

0 4J 58 0 197153 1591261~~ 3 J 1 1 1

0 2 3 4 5 6 7 8 9 JO

Figure 7.8 (Max) heap after bui 1 dHeap phase

59

53 58

26 4J 31 97

1 ~J o 2 3 4 5 6 7 8 9 JO

Figure 7.9 HealJ after tirs! de 1 eteMax




actually have only one element, but the elements left in the heap array will be in sorted order.

The code to perform heapson is given in Figure 7.10. The slight complication is that, unlike the binary heap, where the data begin at array index 1, the array for hearson contains data in position O. Thus the code is a linle different fromlhe hinary heap code. The changes are minor.

7.5.1 Analysis of Heapsort As we saw in Chapter 6, the first phase, which constitutes the building of the heap, uses at most 2N comparisons. In the second phase, the ilh de 1 eteMax uses at most 2 Llog iJ compar~ ison5, for a total of al most 2N log N ~ O(N) comparisons (assuming N ~ 2). Consequently, in the worst case, at most 2N log N - O(N) comparisons are used by hcapsort. Exercise 7.] 3 asks you to show that it is possible for all of the de 1 eteMax operations to achieve their worst case simultaneously

Experiments have shown that the performance of heapsort is extremely consistent: On average it uses only slightly fewer comparisons than the worst~case bound suggests. For many years nobody had been able to show nontrivial bounds on heapsort's average running time. The problem, it seems, is that successive deleteMax operations destroy the heap:S randomness, making the probability arguments very complex. Eventually another approach proved successful.

Theorem 7.5. The average number of comparisons used to hearsort a random permutalion of N distinct items is 2N log N - O(N log log N). Proof. The heap construction phase uses ("')(N) comparisons on average, and so we only need to prove the bound for the second phase. Vic assume a permutation of {I, 2, ... ,N}.

Suppose the ith deleteMax pushes the root element down di levels. Then it uses 2di comparisons. For hearson on any input, there is a cost sequence D : d I, d 2, ... , dN that defines the cost of phase 2. That cost is given by M[) = L[:} di; the number of comparisons used is thus 2M[).

Letf(N) be the number of heaps of N items. One can show (Exercise 7.53) that feN) > (N/(4e))N (where c = 2.71828 .. .) We will show that only an exponentially small fraction of these heaps (in particular (N / 16)N) have a cost smaller than M = N(log N - log log N - 4). When this is shown, it follows that the average value of MlJ is at least M minus a term that is 0(1), and thus the average number of comparisons is at least 2M. Consequently, our basic goal js to show that there are very few heaps that have small cost sequences.

Because level di has at most 2d; nodes, there are 2di possible places that the root element can go for any di. Consequently, for any sequence D, the numher of distinct corresponding de 1 ete~lax sequences is at most

Sf) = Xl1 2d2 .. 2r1N

A simple algebraic manipulation shows t hat .If)/" a given sequence D Sn = 2Mp



/'* 2 * Standard heapsort. 3 */ 4 template 5 void heapsort( vector & a 6 ( 7 forI int i ~ a.size( ) /2; i >~ 0; i--) /' buildHeap */ 8 percDown{ d, i, a.size( ) ); 9 for( int j '" a.size() 1; j > 0; j-- )

10 ( II 12 1:3 14 15 16 /**

swap( a[ 0], a[ j 1 ); percDown( d, 0, j );

17 * Internal method for heapsort. IS * i is the index of an item in the heap. 19 * Returns the index of the left child. 20 */ 21 inline int leftChild( int i ) 22 23 return2*;+1; 24 25 26 /**

1* del eteMax * /

27 * Internal method for heapsort that is used in deleteMax and bu;ldHeap. 28 * ; is the position from which to percolate down. 29 * n is the logical size of the binary heap. 30 */ .11 template 32 void percOm1n( vector & a. int i. int n ) 33 ( 34 int child; 35 Comparable tmp; .16 37 fort tmp ~ a[ i ]; leftChild( i ) < n; i child) 38 ( 39 40 41 42 43

child ~ leftChild( i ); if( child !~ n - 1 && a[ child 1 < a[ child + 1 1 )

chi 1 d++; iff tmp < a[ child 1 )

ali 1 ~a[childl; 44 else 45 break; 4(, 47 a[ i 1 ~ tmp; 48

Figure 7.10 Heapsorl




Because each Ii call assume any value bel ween ! and Liog N J, there me al !)lost (log N),\t possible sequences D. It follows thM the numher of distinct del eteMax se-quences that require cost CXiKtly M is at most the numher oj" cost sequences of total COSl M limes the number of deleteMax sequences ror each of these cosl sequences. A bound of (log N)N2l>.1 follows immediately.

The tola] number of heaps with cost sequence less than M is at most

,'\'1--1 L(log N)''\i < (log t-,I)N2M i=l

If we choose M = N{log N - log log N - 4), thell the number of heaps that have cost sequence less than M is almost (N / \6)0.', and 1 he theorem follmvs from our earlier comments.

Using a more cornplex argul1lent, it can be shown that heapsort always uses at least N log N - O(N) comparisons, and that there arc inputs that can achieve this bound. It secms that the average case should also he 2N log N ~ O(N) comparisons (rather than the nonlinear scconcllcrm in Theorem 7.5); whether this is provable (or even true) is open.

7.6 Mergesort Vve now turn our attention to mcrgesort. lVlcrgcsorl runs in O(N log N) \\'()rst~case running time, and the number of comparisons used is nearly optimal. It is a tinc example of a recursive algorithm.

The fundamental opcnHion in this algorithm is merging two sorted lists. Because the lists arc sorted, this can he done in one pass through the input, if the output is put in a third lisl. The basic merging algorithm takes two input arrays A and B, an output array C, and three counters, Aclr, Bell', and Cell", which are initiaHy sct to the beginning of their respective arrays. The smaller of AIActrJ and B[Bctrl is copied to the next entry in C, and the appropriate counters afC advanced. vVhen either input list is exhausted, the remainder of the other list is copied to C. An example of how the merge routine works is provided for the following input.

LiJi318i6J t

Actl"

1 2 115 1 27 1 3g 1 t

IIelr

'-Tt~ .--"---'---_L---L-----"--I ] Cell"

If the array A contains 1, 13,24, 26, and B contains 2, 1'5,27, :18, then the algorithm proceeds as follows: First, a comparison is done bel ween 1 and 2. 1 is added to C, and then 13 and 2 arc compared.

[I 113124 1 26 1 r

Act/"

1 2 1 15 1 271 3H 1 t

lIelr

[ITT 1 t

Cdr



2 is added to C,


1 /** 2 * Mergesort algorithm (driver). 3 '/ ,1 template -'5 void mergeSort( vector & a ) (,

7 vector tmpArray( a.size( ) ); 8 9 mergeSort{ d, tmpArray. D, a.size( ) - 1 );

10 11 12 /** 13 * Internal method that makes recursive calls. 1"1 * a is an array of Comparable items. 15 * tmpArray is an array to place the merged result. 16 * left is the left-most index of the subarray. 17 '!.. ri ght is the ri ght-most index of the subarray. 18 * / 19 template 20 void mergeSort( vector & a, 21 vector & tmpArray, int left, int right) 22 2l if ( 1 eft < rl ght 2'1 ( 25 int center:= left + right) I 2; 26 mergeSort( a, tmpArray, left, center ); 27 mergeSort( a. tmpArray. center + 1. right ); 18 merge( a. tmpArray. left. center + 1, right); 29 30

Figure 7.11 Mergesort routines

The merge routine is subtle. ! f a 1empor

/** 2. * Internal method that merges two sorted halves of a subarray. 3 * a is an array of Comparable items. 4 * tmpArray is an array to place the merged result . .5 * leftPos is the left-most index of the subarray. 6 * rightPos is the index of the start of the second half. 7 * rightEnd is the right-most index of the subarray, 8 */ 9 template

]0 void merge{ vector & a, vector & tmpArray, II int leftPos, int rightpos. int rightEnd ) 12 13 int leftEnd '" rightPos - 1; Hint tmpPos '" 1 eftPos; 15 i nt numEl ements = ri ghtEnd - 1 eftPos + 1; 16 17 // ~Iain loop 18 while( leftPos

1.10 I...napIer I :::Jorrlng

This is a standard reCurrence rdation, which can be solved several ways. We will show two methods. The first idea is to divide the recurrence relation through by N. The reason for doing this will hecome apparent soon. This yields

T(N) T(N II) --=--+1

N Nil This equation is valid for any N that is a power of 2, so we may also write

T(N/2) T(N/4) --=--+1

N/l N/4 and

T(N/4) T(N/8) --=--+1

N/4 N/8

T(2) T(I) -=-+1

2 1 Now add up all the equations. This means that we add all of the terms on the left -hand side and set the result equal to the sum of all of the terms on the right-hand side. Observe that the term T(N/2)/(N/2) appears on both sides and thus cancels. In fact, virtually all the terms appear on both sides and cancel. This is called telescoping a sum. After everything is added, the final result is

T(N) 1'(1) -- = ~- + log N

N I because all of the 01 her terms cancel and there are log N equations, and so all the Is at the end of these equations add up to Jog N, Multiplying through by N gives the final answer.

T(N) = N log N + N = O(N log N) Notice that if we did not divide through by N at the start of the solutions, the sum

would not. t.elescope. This is why it was necessary t.o divide through by N. An alternative method is to substitute the recurrence relation continually on the right-

hand sick. vVe have

T(N) = 2T(N/2) + N Since we can substitute N /2 into the main equation,

2T(N/I) = 2(I(T(N/4)) + N/I) = 4T(N/4) + N we have

T(N) = 4T(N/4) + 2N Again, by substituting N /4 into the main equation, we sec that

4T(N/4) = 4(IT(N/8) + N/4) = 8T(N/S) + N So we have

T(N) = BI(N/8) + 3N



7.7 Quicksort

Continuing in this manTler, we obtain

Using h = log N, we obtain

T(N) = NT(!) + N log N = N log N + N The choice of which lllcthod to usc is a matter of taste. The first method tends to

produce scrap work that fits better on a standard, 8 1/2 x 11 sheet of paper, leading to fewer mathematical errors, but it requires a certain amount of experience to apply The second method is more of a hrute-force approach.

Recall that we have assumed N = il. The analysis can be refilled to handle cases when N is not a power of 2. The answer turns out to be almost identical (this is usually tbe case).

Although mcrgcsort's running time is O(N log N), it has the significant problem that merging two snrtcd lists uses lincar extra memory. The additional work involved in copying 10 the temporary array and hack, throughout the algorithm, slows the sort considerably This copying can be avoided byjuciiciously switching the roles or a and tmpArray at alternate kvcls of the recursion. A variant of mergcsorl can also be implemented nonreeursivcly (f'xcrcisc 7.16).

The running time of mcrgesort, when compared with other O(N log N) alternatives, depends heavily on the relative costs of comparing clements and moving clements in the array (and the temporary array)_ These costs arc language dependent.

For instance, in other lallguagcs, such as Java, when sorting general ohjects, an dement comparison is expensive, but moving dements is chcap. Mcrgesor! uses the lowest number of comparisons of all the popular sOrLing algorithms, and thus is a good candidate for general purpose sorting in Java. In bet, it is the algorithm used in the standard Jav(l Iihrary for generic sort ing.

On the other hand, in C++, in a generic sort, copying objects can he expensIvc if thc objects are large, while comparing objects often is relatively cheap because of the ahility of the compiler to aggressively perform inJine optimization in conjunction with tht expansion of function templates. In this scenario, it might he reasonable to have ,Ul algorithm llSC a kw more cornparisons, if wc can also use significantly fewcr data movements. Quic/?SOrl, which w(' discuss in the next section, achieves this tradcoff, and is thc sorting routine commonly used in C++ lihraries.

7.7 Quicksort As its name implics for C++, quicksort is the fastest known gencric sorting algorithm in practice. Its average running lime is O(N log N). It is vcry f~lst, mainly due to a very tight and highly optimized inner loop. It has O(N2) worst-case performance, but this can he made exponentially unlikely with a little dfort. By combining quicksort with heal-/sort, we can achieve quicksort's fast running time on almost all inputs, with heapsorts O(N log N) worst-case running time. Exercise 7.27 describes this approach.

The quicksort algorithm is simple (0 understand and prove correct, although for many years it had the reputation of being an algorithm that could in theory be highly optimizc-ci



ZIlU lnapter I ~ol1lng

but in practice was impossihle to code correctly. Like mcrgcsort, quicksort is a djvidc~anclconquer recursive algorithm. The hasic algorithm to sort an array 5 consists of the following four easy steps:

L If the number of elements in S is 0 or 1, then return. 2. Pick any clement v in S. This is called the pivot. 3. Partition 5 ~ {v} (the remaining elements in 5) into t.wo disjoint groups:

5, = (x E 5 - (v) I x :" vI, and 5, = (x E 5 - (v) I x::: v). 4. Relurn (quickson(S I) !"llowcd by v followed by quicksorl(S,)}.

Since the partition step amhiguously describes what to do with elements equal to the pivot, this becomes a design decision. Part of a good implementation is handling this case as effICiently as possible. Intuitively, we would hope thaI about half the elements that are equal to the pivot go into S I and the other half into 52, much as we like binary search trees to he balanced.

Figure 7.13 shows the aclion of quicksort on a set of numbers. The pivot is chosen (by chance) to be 65. The remaining elernents in the set are partitioned into two smaller sels. Recursively solting the. sel 01 smaller numbers yields 0, 13,26,31,43,57 (by rule. 3 of recursion). The set of large numbers is similarly sorted. The sorted arrangement of the entire set is then trivially ohtained.

It should be clear that this algorithm works, but it is not clear why it is any faster than mergesort. Like mergesort, it recursively solves two subproblems and requires linear additional work (step 3), but, unlike mergesort, the subproblems are not guaranteed to be of equal size, which is potentially bad. The reason that quicksort is faster is that the partitioning step can actually be performed in place and very dlicicntly. This effiCiency more than makes up for the lack of equal-sized recursive calls.

The algorithm as described so far lacks quite a few details, which we. now fill in. There are many ways to implement steps 2 and 3; the method presented here is the result of extensive analysis and empirical study and represe.nts a vcry efficient way to implement quicksort. Even the slightest deviations from this method can cause surprisingly had results.

7.7.1 Picking the Pivot Although the algorithm as described works no matter which element is chosen as pivot, sornc choices are obviously better than others.

A Wrong Way The popular, uninformed choice is to usc the first clement as the pivot. This is acceptable if the input is random, but if the input is presorted or in reverse order, then the pivot provides a poor partition, because either all the elements go into 51 or they go into 52' Worse, this happens consistently throughout the recursive GIlls. The practical c.lTcct is that jf the first element is used as the pivot and the input is presorted, then quicksort will take quadratic time to do essentially nothing at all, which is quite embarrassing. tvioreover, presorted input (or input with a large presorted section) is quite frequent, so using the first clement as pivot is an absolutely horrible idea and should be discarded immediately. An alternative is choosing



7.7 QUICksort

81 31 57 75 o 43 26 13 92 65

select pivot t

81 31 57 75 o 43 26 13 92

partition t

o 13 43

26

31 @U5 81 92

57

quicksort sma!1 t

quicksort Jarge t

8 G326 31 "

/'

~13 26 31 43 57 65 75 81 90 Figure 7.11 The steps of quicksort illustrated by example

the larger of the first two distinct elements as pivot, but this has the same bad properties as merely choosing the first element. Do not use that pivoting strategy either.

A Safe Maneuver A safe course is merely to choose the pivot randomly This strategy is generally perfectly safe, unless the random number generator has a flaw (which is not as uncommon as you might think), since it is very unlikely that a random pivot would consistently provide a poor partition. On the other hand, random number generation is generally an expensive commodity and does not reduce the average running time of the rest of the algorithm at all.

281



2 4 t

2 4

2 4

2 4

After First Swap

9 0 ) 'j

Before Second S\vap

9 0 3 '5 t t

j

After Second S\vap

'5 0 3 9 t t

Before Third Swap

o 3 t

9 t

7.7 l!U1Cksort

e 7 6 t j

H 7 6

8 7 6

8 7 6

At this stage, i and j have crossed, so no swap is performed. "fhc final part of the partitioning is to swap the pivot element with the element pointed to by i.

2 4

After Swap with Pivot

5 o 3 6 t

8 7 ')

t pivot

When the pivot is swapped with i in the last step, we know that every clement in a position p < i must he small. This is because either position p contained a small clement to sUIii with, or the large element originally in position p was replaced during a swap. A similar argument shows lhat clements in positions p > i must be large.

One important detail we must consider is hO\v to handle elements that arc equal to the pivot. The questions arc whether or not i should stop when it sees an element equal to the pivot and whether or not j should stop when it sees an element equal to the pivot. inluitively, i and j ought 10 do the same thing, since otherwise the panitioning step is biased. For instance, if i stops and j does llot, then at! elcmenlS that arc equal to the pivot will wino up in 52.

To get an idea of what might be good, we consider the case where all the clements ill the array are identical. If hoth i and j stop, there will be many swaps between iderHical elements. Ail hough this seems useless, the positive effect is that i and j will cross in the



...... Lnapler I )Onmg

middle, so when Ihc pivot is rcphlccd, the partition creates 1wo nearly equal suharrJ),s. The mergeson analysis tells us thaL til

I" 2 * Quicksort algorithm (dl'iver). '; *j -I template 5 void quicksort( vector & a (, ;- quicksort( a, D, a.size( ) - 1 ); H

Figure 7.14 Driver fur quicksort

t k * 2 * Return medi an of "l eft. center, and ri ght. 3 * Order these and hide the pivot. f *1 'j template

7.7 Quicksort

() canst Comparable & median3( vector & a, int left, int right) {

8 int center = ( left + right) I 2; 'J if( a[ center] < a[ left] )

10 swap{ a[ left ], a[ center] ); II if( a[ right ] < a [ 1 eft J ) 12 s.ap( a[ left J. o[ right J ); Il if( a[ right] < a[ center] ) I" swap( a[ center], a[ right] ); I,) 16 II Place pivot at position right - 1 17 swap( a[ center ], a[ right - 1 ] ); 18 return a[ right - 1 ]; I'i

Figure 7.15 (~ndc 10 perform lllcdiall-Or-lhrcc parLilioning

inefficient to compute 1 he pivot by a method that does not act ually son a [1 eft] , a [center] , and a[right], but, surprisingly, this produces bad results (sec Exercise 7.46).

The real hean of the quicksort routine is in figure 7. [6. It includes the partitioning and recursive calls. There arc seyeral things \\lortb noting in this implementation. Line 16 initializes i and j to [ past their COITeet valucs, so that there are no special cases to consider. This initialization depends on the fact that l11edian~of-thrce partitioning has some side crfects; this program will not work if you try to usc it without change with a simple pivoting strategy, hecause i and j st(ut in the wrong place and tbere is Ill) longer a sentinel for j.

The swapping action at lille 22 is somcllmcs written explicitly, for speed purposes. For the algorithm to be bst, it is necessary to force the compiler 10 compile Ihis rode in-lillC'.



.... " Lnapler I )orung

1** 2 * Internal quicksort method that makes recursive calls, 3 * Uses median-of-three partitioning and a cutoff of 10. 4 * a is an array of Comparable items. 5 * left is the left-most index of the subarray. 6 * right is the right-most index of the subarray. 7 *1 8 template 9 void quicksort( vector & a. 1nt left, int right}

10 { 11 if ( 1 eft + 10

16 17 18 /') 20 21 22 2.1 24 25

i nt i = 1 eft + 1. j ri ght - 2; fort ; ; ) (

while( a[ i ] < pivot i++; while( pivot < a[ j] j--; if(i


Best-Case Analysis In the best case, the pivot is in the middle. To simplify the math, we assume that the two suharrays are each exactly half the size of the original, and although this gives a slight overestimate, this is acceptable because we arc only interested in a Big-Oh answer.

T(N) = 2T(N 12) + eN (77)

Divide bot h sides of Equation (7.7) by N.

TeN) T(NI2) ~~=~~~+c

N NI2 (78)

'We will telescope using this equation.

T(NI2) l(N/4) ~~~=~~~+e

NI2 N/4 (79)

T(N/4) T(Nj8) ~~~=~~~+c

N/4 NI8 (710)

T(2) T(1) ~~ = ~- +e

2 I (711)

We acid all the equations from (7.8) to (7.11) and 1101e that there arc log N of them:

T(N) TO) ~- = ~- +clogN N I

(712)

which yields

T(N) = eN log N + N = O(N log N) (713)

Nolice that this is the exact same analysis as merge-sort, hence we get the same answer.

Average-Case Analysis This is the most difficult part. for the average case, we assume that each of the sizes for S1 is equally likely, and hence has probability liN. This assumption is actually valid for OUI' pivoting and partitioning strategy, but it is not valid for some others. Panitioning strategies that clo not preserve the randomness of the subarrays cannot use this analysis. Interestingly, these strategies seem to result in programs that take longer to run in practice.

vVith this assumption, the average value of TO), and hence T(N - i-I), is (lIN) L:;";,,' TV). Equalion (7.1) lhen becomes

T(N) = ~ [~T(j)] + eN (714)



7.7 Quicksort

If Equation (7.14) is multiplied by N, it becomes

NT(N) = 2 ['I: TO)J + eN' )=0

(715)

'vVe need to remove the summation sign to simplify matters. Vle note that we call telescope with one more equation.

(N - l)T(N - J) = 2 [~ T(j)J + ((N - J)' )=0

(716)

If we subtract Equation (7.l6) from Equation (7.J 5), we obtain NT(N) - (N - l)T(N - J) = 2T(N - 1) + 2eN - ( (7l?)

We rearrange terms and drop the insignificant -( on the right, obtaining NT(N) = (N + I)T(N - J) + 2eN (718)

VVe now have a formula for T(N) in terms of T(N - 1) only. Again the idea is to telescope, but Equation (7.18) is in the wrong form. Divide Equation (7.18) by N(N + 1):

leN) T(N - 1) 2e --= +--N+I N N+l

Now we can telescope.

T(N - 1) l(N - 2) 2c = +-

N N -1 N _T(:....l\_' ---'.2) = T (N - 1) + _2e_ N-I N-2 N-l

T(2) TO) 2e -=-+-

3 2 3 Adding Equations (7.19) through (7.22) yields

T(N) T(l) NIl 1 --=-+2(1.:.> N+I 2 id i

(719)

(720)

(721)

(722)

(72"3)

The sum is about logf(N + 1) + y - i, where y ~ 0.577 is known as ruler's constant, so

And so

T(N) -- = O(log N) N+I

T(N) = (l(N log N)

(7.24)

(725) Although this analysis seems complicated, it really is not-the steps are natura! once

you have seen some recurrence relations. The analysis can actually he taken further. The highly optimized version that was described above has also been analyzed, and t his result




geLs exl rCllle1y dirficult, invoking complic

/*' 2 * Internal selection method that makes recursive calls. 3 * Uses median-of-three partitioning and a cutoff of 10. 4 "Places the kth smallest item in a[k-I] . .'5 * a is an array of Comparable items. 6 * left is the left-most index of the subarray. 7 "right;s the right-most index of the subarray. 8 * k is the desired rank (1 is minimum) in the entire array. 9 'I

10 template

7.7 Quicksort

11 void quickSelect( vector & a, int left, int right. int k ) 12 { 13 if ( 1 eft + 10

292 Chapter 7 Sortmg

7.8 Indirect Sorting QuickSOrl is quicksort, and shcllsort is shcllsort. }10\Vcvcr, directly implementing function templates based on these algorithms could occasionally he inefficient if the Comparable object;.:; heing sorted arc large. The problem is that when we rearrange the Comparables, we pay the overhead of repeatedly copying Comparable objects (by calling their operator'" function). This can he expensive if the Comparable objects are large and difficult to copy

In principle, the solution to our problem is simple: Vic create all array of pointers to Comparable and rearrange the pointers. Once we know where the dements should go, we can place them there, withoullhc overhead of the intermediate copies. Doing this elegantly requires an algorithm known as in-situ permutation. Doing it in C++ requires a hunch of ncw syntax.

The first step of the algorithm creates an array of poinlcrs. Let a be the alTay to son and p he the array of pointcrs. Initially, p[i] will poinlto the ohject stored in a[;). Next, wc sort p[;], using the value of the object heing pointed at to determine the ordering. The objects in array a elo not move at all, but the pointers in array p arc rearranged. Figure 7.19 shows how the array of pointers looks after the sOrling stage.

V-lc must still rearrange the array a. The simplest way to do this is 10 declare a second array of Comparable, which we call copy. VI/c can then write the correct SOrLcd order into copy, and then write from copy hack into a. The COSI of doing this is an extra array, and a total of 2N Comparab 1 e copks.

The algorithm has a potentially imporlant pro\.Jiem. By using copy, \VC have doubled the space requirement. 'vVe can clssume that N is large (otherwise, \VC would usc insertion son) and thai the sizc of a Comparable ohject is large (otherwise, Wf would not bother using a point

7.8 Indirect Sorting

\\lill we save SP,KC, but wc will also save Lime, Before \VC step through the code, leI us get a general idea of what needs to he done. Surprisingly, we have already clone it before.

To get an idea ofwh


template 2 class Pointer 3 4 public: .5 POinter( Comparable *rhs 0: NULL) : pointee( rhs ) { } 6 7 bool operator canst Pointer & rhs canst 8 { return *pointee < *rhs.pointee; 9

10 operator Comparable * ( ) canst II { return poi ntee; } 12 private: 13 Comparable *pointee; 1"1 ); Figure 7.20 Class that stores a pointer to a Comparable

15 template 16 void largeObjectSort( vector & a 17 ( 18 vector p( a.size( ) ); 19 int i, j, nextj; 20 21 for( ; '" 0; ; < a,size( ); ;++ ) 22 p[i]"&a[i]; 23 24 quicksort( p ); 25 26 / / Shuffl e ; terns ; n place 27 fore ; '" 0; ; < a.size( ); i++ 28 if{p[i]!"&a[i]) 29 ( 30 Comparable tmp '" a[ ; ]; 31 for(j"i;p[j] !"&a[i];j"nextj) 32 { 33 nextj " p [ j ] - &a [ 0 ]; 34 a[j]"*p[j]; 35 p[j]"&a[j]; 36 .17 38 39 40

a [ j ] p [ j ]

tmp; &a[ j];

Figure 7.21 Algorithm to sOrllarge objects



7.8 Indirect Sorting

7.8.1 vector Does Not Work The basic idea would he to declare the array of pointers, p, as a vector, and then call quicksort(p) \0 rearrange the pnitllCrs. BUI this docs not \vork The problem is 11wl the template argument for quicksort would he Comparab'le*, and thus we need a < uperator that can compare two Comparabl e* types. Such an operator ducs exist for pointer I ypcs (recall from Section 1.5.1), but the result of this operator has nOlhing 10 do with the values stored in the pointed-m Comparables. Furthermore, this behavior cannot he ovcrridcn.

7.8.2 Smart Pointer Class The sohllioJl to our problem is to define ,1 new class template, POinter,l The Pointer will sture,


7.8.6 Implicit Type Conversions Are Everywhere C++ is a strongly typed language. In our codc, we have the following types:

a [ ; ] &a[ ; 1 p [ ; 1

Comparable Campa rab 1 e* Poi nter

Therefore, we expect that &a[i] and p[i] are not type-compatible. Yet we have numerous examples of incompatibilities in the code, including the following four distinct cases (three others arc simply identical to lines 22 and 28):

22 28 33 34

p[ ; ] 0 &a[ ; 1; if( p[ ; ] !o &a[ ; 1 )

nextjop[j] &a[O]; a [ j 1 0 'p [ j ];

So what happened to strong typing? We disabled it by providing the type conversion operator at line 10, and also by not using exp 1 i ci t for the Poi nter constructor. Lel us look at the specifics.

Line 22 works because the Pointer constructor is not explicit. Since p[i] is a Pointer, the right-hand side should be one, too. Even though it is nOl, a temporary can be constructed using a Poi nter constmctor, and this implicit, behind-the-scenes construction is allowed because expl lei t is omitted.

Line 28 uses operator!= to compare a Pointer with a Comparable*. This operator does notexist.lIowever, an implicit conversion (using the type conversion operator at line 10) can be used to create a t.emporary Comparab 1 e*. As a result, operator!"" succeeds.

Line 33 is another piece of C++ trickery \Ve'll talk about that later. Here, the type conversion operator at line 10 creates a temporary Comparable* from p[j]. Line 34 attempts to apply the derderencing operator to a Poi nter. However, this operator is not defined !()f that type (it could be overloaded, hut we haven't done that). But the lype conversion operator at line 10 saves the day by creating a temporary Compa rab 1 e* from p [j] .

7.8.7 Dual-Direction Implicit Conversions Can Cause Ambiguities

These type conversions are great whl'n t hey work, but they can cause unexpected problems. Suppose, for instance, that in addition to operator

7.9 A General lower Bound for Sorting

7.8.8 Pointer Subtraction Is Legal The final mystcry is at line 33. If pI and p2 point at two dements in the same arra)~ then pl-p2 is their separation distance, as an into Thus prj] -&a[O] is the index of the object that p[j] points at.

7.9 A General Lower Bound for Sorting Although we have O(N log N) algorithms for sorting, il is not clear that this is as good as we can do. 1n this section, we prove that any algorithm for sorting that uses only comparisons requires Q(N log N) comparisons (and hence time) in the worst case, so that rncrgesort and hcapsort arc optimal to within a const;:mt f~l(:tor. The proof can be extended to show tbat Q (N log N) comparisons afC required, even on average, for any sorting algorithm that uses only comparisons, which means that quicksort is optimal on average to within a constant factor.

Specifically, we. will prove the followjng result: Any sorting algorithm that uses only comparisons requires rIog(N!)l comparisons in the worst case and log(N!) comparisons on average. We will assume that all N clements arc distinct, since any sorting algorithm must work for this case.

7.9.1 Decision Trees A decision tree is an abstraction used to prove lower bounds. In our context, a decision tree is a binary tree. Each node represents a set of possible orderings, consistent with comparisons that have been made, among the elements. The results of the comparisons are the tree edges.

The decision tree in figure 7.22 represents an algorithm that sorts the three demolls a, h, and c. The initial state of the algorithm is at the root. (\Ve will use the terms stale and node interchangeably.) No comparisons have been done, so all orderings are legal. The first comparison that this particular algorithm performs compares a and }yo The two results lead to twO possible states. If a < h, then only three possibilities rCllwin. Uthe algorithm reaches node 2, then it will compare a and c. Other algorithms might do different things; a different algorithm would have a different decision tree, [f a> c, the algorithm enters state '5. Since there is only one ordering that is consistent, the algorithm can terminate and report that it has completed the sort. If a < c, the algorithm cannot do this, because there arc two possible orderings and it cannot possibly he sure which is correct. In this case, the algorithm will require one more comparison.

Every algorithm that sorts hy llsing only comparisons can be represented by a decision Iree. Of course, it is only feasible to draw the tree for ext rcmely small input sizes. The number of comparisons used by the sOrLing algorithm is equal to the depth of the deepest leaf. In our case, this algorithm uses three comparisons in the vmrst case. The average number 01 comparisons used is equat to the average depth of tne leaves. Since a decision tree is large, it follows that there must be some long paths. To prove the !ower bounds, all thaI needs Lo be shown are some basic tree properties.




a

7.10 Bucket Sort

Proof. A decision tree to sort N elements must have N! leaves. The theorem foIlmvs from the preceding lemma.

Theorem 7.7. Any sorting algorithm that uses only comparisons between elements requires Q (N log N) comparisons.

Proof. From the previous theorem, log (NO comparisons are required.

log(NI) = log(N(N - I)(N - 2) . (2)(1 = log N + log(N - I) + log(N - 2) + ... + log 2 + log I :c:: log N + log(N - I) + log(N - 2) + ... + log(N /2)

N N > -Iog-- 2 2

N N :c:: -logN --

2 2 = Q(NlogN)

This type of lower-bound argument, when used to prove a worst~case result, is some-times known as an information-theoretic lower bound. The general theorem says that if there arc P different possible cases to distinguish, and the questions afC of the form YES/NO, then rIog Pl questions are always required in some case by any algorithm to solve the problem. It is possible to prove a similar result for the average-case mnning time of any comparison-based sorting algorithm. This result is implied by the following lemma, which is left as an exercise: Any binary tree with L leaves has an average depth of at least log L

7.10 Bucket Sort Although we proved in the previous section that any general sorting algorithm that uses only comparisons requires Q (N log N) time in the worst case, recall that it is still possible LO sort in linear time in some special cases,

A simple example is bucket sort. For bucket sort to work, extra information must be available. The input AI, A2 ,. ., AN must consist of only positive integers smaller than M. (Obviously extensions to this are possible.) If this is the case, then the algorithm is simple: Keep an array called count, of size M, which is initialized to all Os. Thus, count has M cells, or huckets, which are initially empty. When Ai is read, increment count [Ai] by 1. After all the input is read, scan the count array, printing out a representation of the sorted list. This algorithm takes OeM + N); the proof is left as an exercise. If Mis O(N), then the tOlal is O(N).

Although this algorithm seems to violate the lower bound, it turns out that it does not hecause it uses a more powerful operation than simple comparisons. By incrementing the appropriate bucket, the algorithm essentially performs an M-way comparison in unit time.



lOO Chapter 7 Sorting

This is similar to the strategy u~cd in extendihle hashing (Section '5.7). This is clearly not in the model for which the lower hound was proVC'Il.

This algorithm docs, however, question the Y;:didity of the model used in proving the lower hound. The model actually is

7.11 External Sorting

7.11.3 The Simple Algorithm rhc hasic external soninf', algorithm uses the rnnging algorithm from mcrgeSOrl. Suppose Wt' have four tapes, TdJ , 7~j2' "J-hl , TilL' which alT two input and two output tapes. Depending on the point ill the algorithm, the (/ ;Jnd /) [apes arc either input tapes or output \,11)(:,5. Suppose the data arc initially on lell- Suppose fUrl her that the il1lcrnal memory can hold (and sort) M records at a time. A natural first step is to read IV! records at a time from the input tape, sort the records internally, and then \vritc the sorted records alternately to Til] and h2. \Vc will call each set of soned records a rUll. \Vhcn this is done, \ve fnvincl all 1 he tapes. Suppose we have the same input as uur example for Shcllsort.

HI 04 11 96 12 17 28 '58 7'5 1'5

If M = 3, then


TaJ

Ta2

Till 11 12 17 28 'j'i 41 58 75 81 04 136 99 rll2 15

T(II 11 12 15 17 28 15 41 '58 75 81 94 96 99 Ta2

Tlil Tl!l

7.11.4 Multiway Merge [I' we have exlw tapes, then we can expect to reduce the number of passes required to sort our input. We do this by extending the basic (tw(vway) merge to a h-way merge,

Merging two runs is done by winding each input tape to the beginning of each run. Then the smaller clement is found, placed on an output tape, anclthc appropriate input tape is advanced. If there are h input tapes, this strategy works the same way, the only difference being that it is slightly more complicated to find the smallest of the /? clements. We can fInd the smallest of these elements by using a priority queue. TiJ obtain the next element to write on the output tape, we perform a deleteMin operafion. The appropriate input tape is advanced, and if the run on the input tape is not yet completed, we insert the new element into the priority queue. Using the same example as before, we distribute the input onto the three tapes.

T(ll Ta2

Ta ) TiJl 1] HI 94 41 58 75 Tl!2 12 lS 96 IS rb1 j 7 28 C)9

We then need two lllore passes of three-way merging to complete the son.

~l~ll II 12 17 28 )') HI 9'1 96 99 T(12 15 41 ')8 75 Ttd

Thl

Th2

h3



7.J1 External Sorting

Tal To] T(/)

Till II 12 IS 17 28 :3'"5 41 81 94 96 09 T/J2

Tid

After the initial run construction phase, the number of passes required using h~way merging is flogh(NjM)l > because the runs gC'l!? times as large in each pass. For the example ahove, the formula is vcrHied, since f1og_i{ 13/3)1 = 2. If we have 10 l,-lJ1CS, then h = '5, and our large example from the previous section would require rlog,) 3201 = 4 passes_

7.11.5 Polyphase Merge The k-way merging Slnltegy dew' loped in the last section requires the lise of 2h tapes. This could be prohibitive for some applications. It is possible to gel by with only h + I tapes. As an example, we will show how to perform tWO-\v and L" and an input file on Tl lIut will produce .34 runs. One option is to put 17 runs 011 each of Tl and T). Vvc could then merge this result onto TIl obtaining one tapc> with 17 runs. The problem is that since allihe runs are on one tape, we must now put some of these runs on l2 to perform another merge. The logical way to do this is to copy the first eight runs from T) onto T2 and then perform the merge. This has the dfect of adding an extra half pass ror evcry pass we do.

An alternative method is to split the original 34 runs unevenly. Suppose we put 2l rUlls on T2 and 13 runs on T). Vie would then merge 13 rullS onto II before T~ was empty. At this point, we could rewind II and T~, and merge Tl , with"] J runs, and I 2, which bas 8 runs, onto lJ. \Ve could then merge B nms until T2 was empty, which would leave 5 nms left on I] and 8 runs on 1"1' 'vVe could then mc-rge II and T3, and so on. The following table shows the number of runs on each tape after each pass.

Run After Aner After After AfLer After After COllst. T,+ 12 ',+ 12 I, +13 T, + Tl T1 +[2 11 + Tl 12 + T,

T, 0 13 0 0 3 0 I, 21 H 0 S 2 0 I 0 Tl 13 0 8 3 0 2 0

The original distribution of runs makes a great deal of differencc. For instance, if 22 runs arc placed on T2, with 12 on TI, then after the first merge, wc obtain I2 runs on T) and .10 runs on T2. After another merge, there are 10 runs on TI and 2 runs on T 3. At this point the going gets slow, because we can only merge two sets of runs before T} is exhausted. Then T] has 8 runs and T2 has 2 runs. Again, we can only merge two sets of runs, obtaining T] with 6 runs and T" with 2 runs. /\ftcr three more passes, T2 has two runs



l04 Chapter 7 Sorting

and the other tapes afC empty. 'vVc, must copy one run to another tape, and then we can finish the merge.

It turns out that the first distribution we gave is optimaL If the number of runs is a fibonacci number f N , then the best way to distribute them is to split them into two Fibonacci numbers FN _ 1 and FN- 2, Otherwise, it is necessary to pad the tape with dummy nms in order to gel the number of runs up to a fibonacci number. Vve lc(\ve the details of how to place the initial set of runs on the tapes as an exercise.

We can extend this 10 a k-way merge, in which case we need I?th order fibonacci numbers for the distribution, where the Inh order Fibonacci number is defined as F(h)(N) = F(I

Summary

.3 Elements in Heap Array Output Next EiClll.C111 Read h [1) h [2) h [3)

Run 1 II 94 81 II 96 81 94 96 81 12' 94 96 12' 94 35' 96 31~' 12' 96 11' IT' TY 12' End of RUll Rebuild Heap

RUIl 2 12 35 17 12 99 17 35 99 17 28 28 99 35 28 58 .1'5 99 58 15 41 41 99 58 41 1'5 ' '58 99 IS' '58 End of 'ElpC 99 IS' 99

IS' End of Hun Rebuild Heap

Run 3 15 J 5

Figure 7.23 Example of run construction

Summary Using C++ for most general internal sorting applications, either insertion sort, Shellsort, or Ljuicksort will be the method of choice, and the decision of \vhich to use \vill depend mostly on the size of the input. figure 7.24 shows the running time obtained for each algorithm on various input sizes (on a slow computer)

The lima were chosen to be random permutations of N integers, and the times given include only the actual time to sort. The code given in Figure 7.2 was used for insertion sort. Shcllsort used the code in Section 7.4 modified to run with Scdgewicks increments. Based on literally millions or sorts, nmging in size from 100 to 2'5 million, the expected running time of Shell sort with these increments is conjectured to he O(N 1/ 6). The hcapsort routine is the same as in Section 7.5. "\\\10 versions of quicksort are given. The first uses a simple pivoting strategy and does not do a cutoff. Fortunately, the input was random. The second uses median-or-three partitioning and a cutoff of ten. further oplimizalions were possible. 'vVe could have coded the median-or-three routine in-line instead o/" using a function, and we could have written quicksort nonrccursively. There are some other optimizations to the code that are fairly tricky to implement, and of course \ve could have used an assembly language. We have made an honest attempt to code all routines efficiently, but of course the performance CJn vary somewhat from machine to machine.

The highly optimized version of quicksort is faster than Shellson even for very small input sizes. The improved version of quicksort still has an O(N2) worst case (one exercise



306 Chapter 7 )ortmg

InscrLioJ) Son Shcllsort l-lcapson QUicksort Quicksort (opt.) N O(NJ) O(N 7/")(,) O(N log N) c!(N log N) (l(N log N)

III 0000001 ll.l100002 0.00000.1 0.000002 (l.(100002 lOll 0.000106 0000019 0.0000J2 ooooon 0.000023 lOOO 0.0 I t240 0.00067R 0.000750 0.000365 0.000316 10000 1.047 0.009782 (J.()] 0215 0.004612 0.004129 100000 110.492 0.t:H38 0.139542 0.058481 0.052790 1000000 NA 1.6777 I. 796 7 0.6842 0.61 J4

Figure 7.24 Comparison of different soning algorithms (all times arc in seconds)

asks you to construct a small example), but the chances of this worst case appearing arc so negligible as to nol be a factor. If YOll need to sort large amounts of data, quicksort is the method of choice. But never, ('vcr, take the casy way out and USc the first element as piVOL It is just not safe to assume that the input \vill he random. If you do not want to worry about this, use Shcllsorl. ShdlSOrL will give a small performance penalty but could also be acceptable, especially if simplicity is required. Its worst case is only O(N'Ij)); the chance 01 that worst case occurring is likewise negligible.

I lcapson, although an O(N log N) algorithm with an apparently light inner loop, is slower than Shellsort. ;\ close examination of the algorithm reveals that in order to move data, hcapsort docs two comparisons. An impl\)\,cl1lent suggested by Floyd moves data with esselltially only one comparison, hut implementing this improvement makes the code somewhat longer. Vle leave it to the reader to decide whether the extra codingcfforL is worth the increased speed (Exercise 7.5]).

Insertion son is useful only for small or vcry nearly sorted inputs. We have llot included mergesort, because its performancc in C++ for sorting primitive types is llot as good as quicksort and it is not any simpler 10 code. Vic have seen, however, that merging is the central idea of external sorts.

Exercises 7.1 Son tbe sequence _), 1,4, 1,5,9,2,6, '5 using insertion sort. 7.2 vVhat is the running lime of insertion sort if ,,11 elements arc- equal? 7.3 Suppose \\ie exchange clemenls a[ i] and a[ i +kJ, which were originally out of order.

Prove thai at least I and at most 21? - I inversions arc removed. 7.4 Show the result or running Shellsort on the input 9, B, 7, 6) '), 4, 3, 2, I using the

increments {I, J, 7}. 7."5 ,L vVhat is the running lime of Shellsort using the two-increment sequence {l, 2}?

h. Show that for any N, there exists a thrcc-incrcmenl sequence sLlch thaI She1lsort runs in O(N'5ri) timc.



Exercises

c. Show that for any N, there exists a six-increment sequence such that Shellsor! runs in O(N-1/ 2) time.

7.6 -/I a. Prove that the running time of Shellsort is Q(j\J2) using increments of the form 1, (, (2, ... , (1 for any integer c.

** b. Prove that for these increments, the average running lime is ("')(N 3/ 2). * 7.7 Prove that if a h-sortec1 file is then It-sorted, it remains h-soned.

-/1-/1 7.8 Prove thai I he running time of Shellsort, using the increment sequence suggested by I-Hbbard, is Q(NJ/l) in the worst case. Hint: You can prove the bound by considering the special case of what Shellsort does when all elements arc either 0 or 1. SCi (IIi] = t if i is expressihle as a linear comhination of hi' hi_I,' ., h[l/2J + I and 0 otherwise.

7.9 Determine the running time of Shellson for a. sorted input

-It b. reverse-ordered input 7.10 Do either of the follOWing modifications to the Shellson routine coded in Figure 7.6

affect the vvorst-case running time? a. Before line 8, subtract one from gap ir it is even. b. Before line 8, add one to gap if it is even,

7.11 Show how heapsort processes the input 142,543, 123,65,453,879,572,4:34, 111,242,811,102.

7.12 What is the running time of heapsort for presorted input? * 7.13 Show that there are inputs that force every percolateDown in heapson to go all the

way to a leaf. (Hint: \-\fork backward.) 7.14 Rewrite heapsort so that it sorts only items that arc ill the range low 10 high, which

are passed as additional parameters. 7.15 Son 3, 1,4,1, '5, 9, 2, 6 using mcrgeson. 7.16 How would you implement mergesort without using recursion? 7.17 Determine the running time of mergesort for

a. sorted input b. reverse-ordered input c. random input

7.18 In the analysis of mcrgcsort, constants have heen disregarded. Prove thatlhe number of comparisons used in the worst case by merge-sort is Nflog Nl - 2fiug N ] + 1-

7.19 Sort _), 1, 4, I, '5,


c. a random element * d. the a\'LTHgc or all clements in the set

7.22 a. For the quicksort implementation in this chapter, what is the running lime when all keys arc equal?

h. Suppose we change the partitioningslratcgy so that neither i nor j stops when an clement with the salTle key as the pivot is found. \Nhat fixes need to he made in the code to guarantee that quicksort works, and what is the running lime, when all keys arc equal?

c. Suppose Wf change the partitioning strategy so that i SLOpS at an clement \Vilh the same key ;15 the pivot, but j docs not SLOp in a similar case. \,VhaL fixes need to be rnadc in the cock 10 guarantee that quicksort works, and when all keys arc equal, what is the running time of quicksort?

7.23 Suppose we chouse the clement in the middle position of the array as pivot. Docs this make it unlikely that quicksort will require quadratic time?

7.24 Construct a permutation of 20 elements that is as bad ,15 possihle for quicksort using median-of-three partitioning and a cutoff of ).

7.25 The qUickson in the text uses two recursivc calls. Remove one ofthe calls as rollcnvs: a. Rewrite the code so that the second recursive call is unconditionally the last

line in quicksort. Do this hy !Tvcrsing the i flel se and returning after lhe call to insert i onSort.

h. Rcrnovc the tail recursion hy writing a. while loop alld alLcring left.

7.26 Continuing {"rom Exercise 7.2'5, arter part (a),

Exercises

way partitioning algorithm. (f /ill!: j\s i and j move toward each other, maintain fivc groups of c1elllrllls as showll helow):

EQUAL SMALL UNKNOHN LARGE EQUAL j

h. Proyc that using the algorithm alxwe, sorting an N-ckmcnt anay that contains only d differenl valucs, takes O(dN) lime.

7.29 \Vrite a program 10 implement the selection algorithm.

7.30 SnhT tile following recurrence T(N) = (I/N)IL:::o' TO)I + eN. 1(0) = O. 731 A sorting algorithm is stahle if clements with equal clcments ,liT left in the same

order as they occur in the input. \Vhich oflhe soning algorithms in this chapter arc stahle and which arc 1101? Why?

732 Suppose you are given a sorted list ofN clements followed byIU\,I) randomly ordered clements. flow would you sort the eillire list if a. [(N)=CJ(I), b. 1(1'1) = O(log 1'1)' c. [(1'1) = O(-IN)'

* d. How large canf(N) be for the entire list still 10 be sortable in O(N) lime? 7.33 Pro\'(:: lh


7.4]

References

larger (in thc case of a (l1wx)he


QuickSOrl is from Hoare [7J. This paper analyzes the basic algorithm, descrihes 1110st of the improvements, and includes the selection algorithm, A detailed analysis and empirical study was the subject ofSedgcwi_ck's dissertation [22]. tvlany of the importanl results appear in the three papers !l91, [20], and [21]. [I] provides a detailed C implementation wiLh some additional improvements, and points oUl that many implementations of the UNIX C{sort library routine afC easily driven to quadratic behavior. Exercise 7.27 is from [14].

Decision trees Hnd sorting optimality arc discussed in Ford and Johnson [3]. This paper also provides an algorithm that almost meets 1 he 100ver bound in terms of number of comparisons (but not other operations). This algorithm was eventually shown lo be slightly suboptimal by Manachcr [13[

External sorting is covered in detail in r 121. Stable sorting, described in Exercise 7.31, has been addressed by Horvath [8].

I. J. L. Bentley and M. D. iv1cElroy, "Engineering a Son Function,1I Softwan:-Praclice and Experience, 2j (199j), 1249-1265.

2. R. W Floyd, "Algorifhm 245: Trecson .)," Comnltll1icaLions (1 the ACM, 7 (1964), 70 I. T L. R. Ford and S. M. Johnson, "A T(lUrnamcnt Problem," Americalllv1at/1(:matics MOil/illy, 66

(1959), 387,,389. 4. M. Golin and R. Sedgnvick, "Fxact Analysis of Mcrgesort," FourtiJ SlAM C(H~feren(' Oil

Discrete IvlatiJematics, 1988. '5. C;. H. Gonnet and R. Baeza-Yates, lIandlJOoh of Algurithms and Data Structures, 2nd ('el.,

Addison-\Veslcy, Reading, lvIass., 1991. 6. T. II. Hibbard, "An Empirical Sludy of Minimal SLOragc Sorting," Communications oj lht

JlCM,6 (1963), 206-21J. 7. C. A. R. Hoare, "()uicksort," ComputcrJmH"lw/, '1 (1962), 10-1 '5. 8. E. C. llorvath, "Slable Sorting in /\symptotically Optimal Timc ,lilt! Extra Spacc," .Journal of

theJlCM, 25 (1978),177-199. 9. B. I luang and M. Langston, ''Practical In-placc Merging," Communications of the ACM, )1

(J 988),j48-)52 10 . .J. Incl..'Tpi and R. Scdgcwick, "lmproved Upper Bounds on ShelisorL," .Joumul (~r Computer

and System SticHas, jJ. (.198'')), 210-224. It. S. Janson and D. L Knuth, "Shcllsort with Three Inerements," nml(lom Strw:tlncs (md

Algorithms, 10 (1997),12'3-142. 12. D. E. Knuth, The Art (!f Computer Programming. Volume 3: Sort.ing. and Searchil1,i!., 2d cd.,

Addison-\Vcsley, Reading, tvIass., 1998. 13. G. K. Manachcr, "The Ford-Johnson SOriingAlgorithm Is Nol Optimal,"Jounla/ (!f the ACM,

26 (J 979),441-456. 14. D. R. Musser, "Introspective Sorting and Selection Algorilhrns," Suftware-Practice and

[;xpcriellce,27 (J 997),983-993. 1 '5. A. A. Papernov and G. V Stascvich, "1\ Method of Information Sorting in Computer

Memories," Problems oj lnformation {j(lnsl1lissioll, 1 (196'5), 63-75. 16. C. G. Plaxtoll, B. Pootletl, and T. Sucl, "Improved Lower Bounds for Shdlsort," Proceedings

of the ThirtY-lhird Annual Symposium on the hmndalions of Computer Science (1092), 226-2J5.



References

17. V R. Pran, Shdlsorl and Sorting, NClworhs, Ciarland Publishing, New York, 1979. (Originally presented as the author's Ph.D. thcsis, Stanford University, 1971.)

18. R. Schaffcr ,md R. Sedgcwick, "The Analysis of JIcapsorL," Journal (:f Algoritiuns, 14 (1993), 76-100.

19. R. Sedgewick, "Quicksort with Equal Keys," SIAM}ounw! 011 Computing, 6 (] 977), 240~267. 20. R. Sedgewick, "The Analysis of Quicksort Programs," Ada Infonnalica, 7 (1977), 327~355. 21. R. Sedgcwick, "Implementing Quicksol"l Programs," COHl/mmica/ions oIlhe ACM, 21 (1978),

847-857. 22. R. Sedgcwick, Quichsorl, Garland Publishing, New York, J 978. (Originally presented [IS the

author's Ph.D. thesis, Stanford University, .1975.) 23. R. Sedgewick, "A New Upper Bound for Shel!sort,"Jounwl (~rAlgOlitlllns, 7 (.1986), 1 ,)9~173. 24. D. L. Shell, "A I figh~Speed Soning Procedure," Communications of/he ACM, 2 (19'59), 30-32. 25. M. A Weiss, "Empirical Resulls on lhe Running Time of Shellsorl," Compuler Journal, 34

(1991), 88-9L 26. M. A. \Neiss and R. Sedgewick, "More on Shcllsort Increment Sequences," Information

Processing Leiters, ."34 (1990), 267-270. 27_ M. A. Weiss and R. Sedgcwick, 'Tight Lower Bounds for Shcllsort," Jounwl (~f Algorithms, 1.1

(1990),242-251. 28. J W J Vv'illiams, "Algorithm 2.12: llcapsort ," C0/11Immiwtions

Data structures and algorithm analysis (3rd ed) by mark allen weiss (p2)

Documents

Transcript of Data structures and algorithm analysis (3rd ed) by mark allen weiss (p2)