An efficient external sorting with minimal space requirement

16
Inwrnational Journal of Computer and Information Sciences, Vol. I1, No. 6, I982 An Efficient External Sorting with Minimal Space Requirement 1 Dalia Motzkin 2 and Christina L. Hansen 2 Received July 1982; revised January 1983 An efficient external sorting algorithm with minimal space requirement is presented in this article. The average number of passes over the data is approx- imately 1 +Ln(N+ 1)/4B, where N is the number of records in the file to be sorted, and B is the buffer size. The external storage requirement is only the file itself, no additional disk space is required. The internal storage requirement is four buffers: two for input, and two for output. The buffer size can be adjusted to the available memory space. A stack of size log 2 N is also required. KEY WORDS: Sorting; quicksort; external sorting; efficient sorting. 1. INTRODUCTION The sorting technique described below is a partition sort, partially based on quicksort. (~-2) The quicksort algorithm was first introduced by C. A. R. Hoare. (1-2) The algorithm with the best average behavior ~3) was widely accepted. Many modifications and improvements have been developed. Its performance has been thoroughly analyzed. Some of the interesting variants are described in Refs. 3, 4, and 10-14. General descriptions and analyses can be found in Refs. 3 and 4, the history and full survey of the many variants is given in Refs. 4-12. Sedgewick ~11,12) and Loeser ~5'6) have provided performance evaluation and analysis for some of the important versions of Quicksort. However, the curent variety of quicksort versions are all oriented toward internal sorting--within the memory of the computer. The constant interchanging of keys which are positioned at noncontiguous locations make J This work was partially supported by a fellowship and grant from Western Michigan University. 2 Department of Computer Science, Western Michigan University, Kalamazoo, Michigan. 381 0091-7036/82/1200-0381503.00/0 1982 Plenum Publishing Corporation 828/i 1/6 I

Transcript of An efficient external sorting with minimal space requirement

Page 1: An efficient external sorting with minimal space requirement

Inwrnational Journal of Computer and Information Sciences, Vol. I1, No. 6, I982

An Efficient External Sorting

with Minimal Space Requirement 1

Dalia Motzkin 2 and Christina L. Hansen 2

Received July 1982; revised January 1983

An efficient external sorting algorithm with minimal space requirement is presented in this article. The average number of passes over the data is approx- imately 1 +Ln(N+ 1)/4B, where N is the number of records in the file to be sorted, and B is the buffer size. The external storage requirement is only the file itself, no additional disk space is required. The internal storage requirement is four buffers: two for input, and two for output. The buffer size can be adjusted to the available memory space. A stack of size log 2 N is also required.

KEY WORDS: Sorting; quicksort; external sorting; efficient sorting.

1. INTRODUCTION

The sorting technique described below is a partition sort, partially based on quicksort. (~-2)

The quicksort algorithm was first introduced by C. A. R. Hoare. (1-2) The algorithm with the best average behavior ~3) was widely accepted. Many modifications and improvements have been developed. Its performance has been thoroughly analyzed. Some of the interesting variants are described in Refs. 3, 4, and 10-14. General descriptions and analyses can be found in Refs. 3 and 4, the history and full survey of the many variants is given in Refs. 4-12. Sedgewick ~11,12) and Loeser ~5'6) have provided performance evaluation and analysis for some of the important versions of Quicksort.

However, the curent variety of quicksort versions are all oriented toward internal sorting--within the memory of the computer. The constant interchanging of keys which are positioned at noncontiguous locations make

J This work was partially supported by a fellowship and grant from Western Michigan University.

2 Department of Computer Science, Western Michigan University, Kalamazoo, Michigan.

381

0091-7036/82/1200-0381503.00/0 �9 1982 Plenum Publishing Corporation

828/i 1/6 I

Page 2: An efficient external sorting with minimal space requirement

382 Motzkin and Hansen

quicksort ineffective for external sorting. The linked list approach of Stable Quicksort suggested by Motzkin ~7~ is also appropriate for internal sorting only. Implementing any of the existing versions of quicksort for external sorting involves a disk access for almost each comparison, thus rendering quicksort much less efficient than the merging techniques currently used.

Merging techniques are mostly used for external sorting. Extensive descriptions of the variety of merging techniques with analysis of space and time requirements can be found in Ref. 3, 4, and 8.

A merge sorting technique which sorts the file in place, without using any additional external space, is presented in Ref. 8. However, this merge algorithm is slower than the standard merging techniques and slower than the algorithm presented here. The merge sort in Ref. 8 has Log 2 N passes over the data only in the best cases. An efficient merge algorithm is described by Pardo, ~7~ but Pardo's algorithm does not have the property of sorting the file in place.

Basically, the merging (3'4~ techniques consist of essentially two phases. First, segments of the input file are sorted using a good internal sort. These sorted segments, known as runs, are written out on the external storage as they are generated. Second, the runs generated in phase one are merged together. Thus, the merge sort requires two copies of the file on disk while merging: an input copy containing the most recent set of runs and an output copy where the merged runs are being written. Given a file of N records, m initial runs, and buffers of size B; the two-way merge would require 0[log2m ] passes over the file with N/B accesses per pass, totally O[N/Blog2m ] disk accesses. Traditional Quicksort requires O[KNlogeN ] accesses, for some constant K, and only one copy of the file.

For example, if a run consists of 100 records and the buffer size is 50 records, the approximate number of disk accesses required for a two-way merge sort is O[N/50)Log2(N/IO0) while the approximate number o f accesses required for quicksort would be O[KNlogeN ]. Thus quicksort, which has the fastest average behavior for internal sorting, is quite ineffective for external sorting.

The idea of partitioning is the essential feature of quicksort which contributes to speed and efficiency. The fact that keys are compared and interchanged from opposite ends of each subfile contributes to minimizing the number of interchanges. This feature is not essential for external sorting, since each record read in for comparison has to be written out anyway. On the other hand, this feature is the one which renders quicksort inefficient for external sorting.

A sorting method is suggested here which is based on the idea of partitioning, but does not use the quicksort method of interchanges. This method provides for two input buffers and two output buffers. The

Page 3: An efficient external sorting with minimal space requirement

An Efficient External Sorting with Minimal Space Requirement 383

comparisons and interchanges will always be made from within the buffers, thus alleviating the need for successive reading of records which are not in the same physical disk block.

The important feature of this algorithm is the fact that the file is sorted in place without any significant time penalty as compared to a two-way merge.

In we call

the following sections we will describe this method of sorting which shuffiesort, and show that it possesses the following properties:

1. It has O(2Ln(N+ 1)/4B) passes over the file, where N is the number of records in the file, B- is the buffer size. In other words, on the average it has approximately the same number of passes over the data as a two-way merge.

2. It requires only one copy of the file on the disk at all times. That is, no additional disk space is required other than that of the original file.

3. It does not behave poorly on files that are already ordered or in reverse order, as traditional Quicksort does.

Therefore, shuffiesort would be suitable for situations where there is a limited amount of disk space, i.e., when keeping two copies of the file (which are needed in Mergesort) is either impossible or expensive.

2. DESCRIPTION OF THE A L G O R I T H M

The sorting method presented here partions the file as does standard Quicksort, but, as was mentioned above, it discards the Quicksort method of interchanges. The shufflesort uses four buffers: two for input and two for output. Comparisons are made with each item of a buffer, alleviating any need to read successive records from differing disk blocks.

The sorting scheme is as follows: Given a file of N records and four memory buffers of size B each, we conceive the file as a string of M = N/B segments. Denote the keys of the file by K(1, 1), K(1, 2),..., K(1, B); K(2, I), K(2 ,2) ..... K(2, B);...; K(M, 1), K(M, 2) ..... K(M,B) and the records by R(1,1), R(1 ,2) ..... R(1,B); R(2,1), R(2 ,2) ..... R(2, B);...; R(M, 1), R(M, 2) ..... R(M, B). Thus R(i,j) is the j th record in the ith segment, and K(i,j) is the key of the record R(i,j). When a key is moved the corresponding record is moved with it (although it is not always explicitly mentioned in the description that follows). A key, say K(1, I), is selected as the key controlling the sort. At each pass the key, K(1, 1), is compared to all the keys in the file in the following order: K(I , 2) ..... K(1,B); K(2, 1) ..... K(2, B);...; K(M, i) ..... K(M, B). The segments are read into the buffers from

Page 4: An efficient external sorting with minimal space requirement

384 Motzkin and Hansen

both front and back of the file. The buffers are written in the same manner. When a buffer is written out, the next buffer is read from that end of the file.

If a key, say K(i,j), satisfies K(i,j) < K(I, 1) then R( i , j ) is stored in a buffer assigned to the records with keys smaller than K(1, 1), say, buffer X. if the key K(1, 1) < K(i,j) then R(i,j) is stored in a buffer assigned to records whose keys are larger than K(1, 1), say, buffer Y. When buffer X is full, it will be written in the beginning of the space allocated for the file on the disk. Each time buffer Y gets full it is written at the end of the space allocated to the file on the disk. Thus, the X buffers will be written successively starting at the beginning of the file area, while Y buffers will be written in reverse order starting at the end of the tile area on the disk.

When the entire tile has been compared, the key will be inserted either at the end of the last X buffer or the beginning of the last Y buffer, depending on which has space for it, and written out. They key divides the file into two subfiles for the next pass.

Each subtile is processed in a similar manner. When a subtile size is small enough to be contained completely in the memory, it is read, sorted internally, written out, and not processed any more. Obviously, a subtile of size B or less need not be partitioned, but, in practice if the internal sorting method can sort a subtile in place (such as Quicksort or Insertion sort), then a subtile of size 4B or less can be sorted internally, utilizing the two input buffers and the two output buffers. In order to use all four buffers for internal sorting, it is convenient, though not necessary, that the four memory buffers be contiguous.

In the following sections several examples are given to demonstrate the algorithm, the average behavior of the sort, and some special cases. Since the files in the examples are relatively short, buffers of size 2 were used. The partitioning process was continued until the subfiles became size 2 or less. This was done in order to be able to demonstrate several passes without going through long tedious examples.

The following example will demonstrate the first two passes. Let the keys of the unsroted tile be:

26, 5, 37, 1, 61, 11, 59, 15, 48, 19

With a buffer size of two the first buffers read in will be:

26, 5, 48, 19

The key controlling the partitioning process will be 26 and the keys will be assigned to buffers as follows:

X Y

5 19 48

Page 5: An efficient external sorting with minimal space requirement

An Efficient External Sorting with Minimal Space Requirement 385

Buffer X is filled and will be output creating the following file:

5, 19, 37, 1, 61, 11, 59, 15, 48, 19

The next four keys are read into the two input buffers:

37, 1, 59, 15

Buffer Y will be filled with the 37 and will be output giving the following file:

5, 19, 37, 1, 61, 11, 59, 15, 37, 48

The remaining keys are assigned giving:

X Y

1 15 59

Buffer X is written, yielding:

5, 19, 1, 15, 61, 11, 59, 15, 37, 48

The remaining keys are read, 61, 11, and assigned

X Y

11 61 59

The key is placed in the empty space in buffer X and the full buffers are written:

X X X Key Y Y

5 19 1, 15 11 26 61, 59 37, 48

thus creating two subfiles X and Y. The subtile X need not include the key in the next pass since it has already been placed in its proper position.

The same procedure is applied to each of the subfiles as a second pass. A more efficient implementation will sort the smaller subtile first.

The first input will be the first two records (buffer) and the last two records (buffer) of subtile X:

5, 19, 15, 11

The key will be 5 and the remaining keys will be assigned:

X X X Y

15 19

Page 6: An efficient external sorting with minimal space requirement

386 Motzkin and Hansen

Buffer X Y is output and subtile X is now:

5, 19, 1, 15, 19

assignment continues with the last input:

X X X Y

1 I1

The key, 5, is added to buffer X X and it, along with the buffer fragment XY, is output

X X Key X Y X Y Key Y Y

1 5 11 15, 19 26 61, 59 37, 48

Sub-file Y is sorted in this manner:

X X Key X Y X Y Key YX YX Key

1 5 11 15, 19 36 59, 37 48 61

At this point subtile X X and YX are less than or equal to one buffer in size and are sorted internally. Subfiles X Y X Y and Y X Y X must be externally sorted yet, if the partitioning process is to continue until a subtile is of size 2 or less.

As was mentioned above, in a good implementation, a subtile that is less than or equal to four buffers in size may be sorted internally and not accessed again. This is due to the fact that there is memory space of four buffers already available, and additional space is not needed if a sorting technique such as internal Quicksort or insertion sort is used.

3. SHUFFLESORT

Notation: algorithm:

Variables

INPUT

SUBFILE

P

KEY(/)

The follosing variables and procedures are used in the

- - A n input area assigned to two contiguous buffers. Each buffer is half the size of INPUT.

- - T h e subtile that is being partitioned.

- - T h e key controlling the partitioning process.

- - T h e key of the record in locat ion/ .

Page 7: An efficient external sorting with minimal space requirement

An Efficient External Sorting with Minimal Space Requirement 387

OUTPUT1 - - T h e output buffer where the records whose keys are smaller than P are stored.

OUTPUT2 - - T h e output buffer where the records whose keys are larger than P are stored.

I,J,K - - R unn ing indices pointing to records or keys in INPUT, OUTPUT1, and OUTPUT2, respectively.

STACK - - A stack containing the parameters of subfiles to be partioned (the parameters include the location of the first record of the subtile and the number of records in the subtile).

R E C O R D P - The record whose key is P.

Functions

GET-STACK - -Ob ta in s the parameters of the entry at the top of the stack, and pops up the stack.

ADD-STACK(SUBFILE n a m e ) - - A d d s to the top of the stack the location and size of a subtile.

SIZE(buffer of SUBFILE name) - - Returns the number of records in a

READ(RIGHT)

READ(LEFT)

READ(INPUT)

WRITE(RIGHT)

WRITE(LEFT)

buffer of a subtile.

- - R e a d s the right most segment of a SUBFILE which has not yet been read (during the process of partitioning a subtile).

- - R e a d s the left most segment of a SUBFILE which has not yet been read (during the process of partitioning a subtile).

- - Reads an entire subtile of SIZE(INPUT) or less.

- - W r i t e s onto the area from where the last segment of SUBFILE on the right was read.

- - W r i t e s onto the area form where the last segment of SUBFILE on the left was read.

In the following algorithm subtiles are partitioned until their size is ~< SIZE(INPUT).

Page 8: An efficient external sorting with minimal space requirement

388 Motzkin and Hansen

A L G O R I T H M

ADD-STACK(FILE to be stored) REPEAT

GET-STACK IF SIZE(SUBFILE) > SIZE(INPUT) THEN READ(LEFT) into left half of INPUT

READ(RIGHT) into half of INPUT P ~- KEY(INPT(1)) RECORDP ~ INPUT(l) I*-2, J ~ 1, K ~ SIZE(OUTPUT2) WHILE there are still segments of SUBFILE that have not been read DO

IF n > KEY(INPUT(l)) THEN OUTPUT(J) ~ INPUT(/)

J ~ J + 1 IF J > SIZE(last segment read from left side of SUBFILE) THEN WRITE(LEFT)

READ(LEFT) into already used part of INPUT J ~ l

END if ELSE OUTPUT2(K) ~ INPUT(/)

K ~ K - 1 IF (OUTPUT2 -K) > SIZE(last segment read from right side of SUBFILE) THEN WRITE(RIGHT)

READ(RIGHT) into already used part of INPUT K ~ SIZE(OUTPUT2)

END if END if I ~ I + 1 IF I > SIZE(INPUT) THEN I ~ 1 END if

END WHILE IF J < SIZE(OUTPUT l) THEN OUTPUT 1 (J) ~- RECORDP

WRITE(LEFT) IF K < SIZE(OUTPUT2) THEN WRITE(RIGHT) END if

ELSE OUTPUT2(K) ~- RECORDP WRITE(RIGHT)

END if ADD-STACK (MAX(LEFT(SUBFILE), RIGHT(SUBFILE)) ADD-STACK (MIN(LEFT(SUBFILE), RIGHT(SUBFILE))

Page 9: An efficient external sorting with minimal space requirement

An Efficient External Sorting with Minimal Space Requirement 389

ELSE READ(INPUT) SORT(INPUT) INTERNALLY

WRITE(LEFT) WRITE(RIGHT)

END if UNTIL stack is empty END of ALGORITHM

4. PERFORMANCE ANALYSIS

4.1. Theoretical Evaluation of the Average Numbers of Comparisons, Passes, and Disk Accesses

When sorting using the shufflesort, except for the first pass, there are no complete passes in the sense that the whole tile is read and written completely from beginning to end. Therefore, in order to have an estimate for the number of passes required to sort a file using shuffiesort we will estimate the number of time N records are read and written, this will be defined as a pass here.

It has been shown (Ref. 4, p. 121) that the average number of comparisons in the internal Quicksort is 2 ( N + 1 ) L n ( ( N + 1)/(M + 2)) (the notation is taken from Ref. 4). This average is computed for a version suggested by Sedgewick. (11) In this version M is a parameter specifying the "threshold" between partitioning and final sorting. We observe that the number of comparisons in the above version of Quicksort is approximately the same as the number of comparisons in shufflesort, since the techniques of partitioning and subtile size threshold lead to the calculated value 2 ( N + 1)Ln((N+ 1) / (M+ 2)). Therefore, the methods for calculating the values used in the above reference apply here and will not be repeated. We conclude that the average number of comparisons is ~ 2 ( N + 1)Ln((N+ 1) / (M+ 2)) where M~< 4B. The above formula will be used to evaluate the average number of passes here.

Note that when a segment of a subtile is read into a buffer, each of the keys in the segment is compared once to the key controlling the partioning process of the subtile. Thus, for each time a record is read, there corresponds one key comparison, a record whose key becomes the key controlling the partitioning, is written into its final position and not processed any more. Thus the number of comparisons is equal to the number of times a record is read and written.

As was said above, when a subtile is of size 4B or less it can be read, sorted internally, and not processed again, thus we can use 4B as a threshold value. It follows that the number of records read and written in all subfiles of

Page 10: An efficient external sorting with minimal space requirement

390 Motzkin and Hanse~

size greater than 4B is ~2(N + 1)Ln((N + 1)/4B). The average number of passes (not including reading and writing of subfiles whose size is 4B or less) is ( I / N ) 2 ( N + 1)Ln((N+ 1)/4B)~ 2Ln((N + 1)/4B). It takes at most one pass to process all subfiles of size 4B or less. Thus the average number ot passes is approximately 1 + 2Ln((N + 1)/4B).

To obtain the average number of disk accesses, we simply multiply the average number of passes by 2N/B, N/B is the average number of segments, and each segment read and written involves 2 disk accesses.

4.2. Some Special Cases

4.2.1. The Ordered File and the File in Reverse Order

The technique of reading and writing segments from both ends provides an advantage in the case of an ordered or reversed, ordered file. During this process the records are shuffled so that after the first or second pass over the data there is an unordered file. This will alleviate some of the inefficiency of quicksort in the case of ordered and reversely ordered files.

The following example traces an ordered file through two passes. Let the keys be:

1 ,5 ,11 ,15 ,19 ,26 ,37 ,48 ,59 ,61 with a buffer size of two

The first buffers read in will be:

1 , 5 , 5 9 , 6 1

The key is 1 and buffer Y is filled with 59 and 5. It is output at the end of the file, and the next buffer read in (37, 48) giving:

1, 5, 11, 15, 19, 26, 37, 48, 59, 5

Buffer Y is again filled with 37 and 61 and output. The next buffer 19, 26 is read. The file is now:

1, 5, 11, 15, 19, 26, 37, 61, 59, 5

Buffer Y is filled, 19, 48. The next buffer read is 11, 15 and the file is :

1, 5, 11, 15, 19, 48, 37, 61, 59, 5

Buffer Y is filled with 11, 26, and output and again with 1, 15 (the remaining data item and the key) giving this file:

Key Y Y Y Y U 1 15 11, 26 19, 48 37, 61 59, 5

Page 11: An efficient external sorting with minimal space requirement

An Efficient External Sorting with Minimal Space Requirement 391

The next pass begins by reading the first and last buffers 15, 11, 59, 5 making 15 the key. They are assigned to buffers:

YX YY

11, 5 59

Buffer YX is output making the subtile

11, 5, 26, 19, 48,37,61,59,5

The next buffer is read, 26, 19, and 26 is assigned to buffer YY, filling it. It is output.

11, 5, 26, 19, 48, 37, 61, 26, 59

The buffer read is 37, 61 and the assignment continues

YX YY

37, 19

Buffer YY is output:

11, 5, 26, 19, 48, 37, 19, 26, 59

The last key, 48, is read and the assignments are finished

YX YY

15 48, 61

giving the file

Key YX Key YY YY YY

1 11, 5 15 48, 61 37, 19 26, 59

Subtile YX may be sorted internally and Y Y may have to continue to be partitioned depending on the value of the threshold selected. This ordered set will yield 3.2 passes in order to be "sorted," when the subfiles are partitioned until a subtile is less than or equal to buffer size.

The following example traces the frs t two passes of a reverse ordered file, using a buffer size of two, here again we assume that the partitioning process will continue until the subtile size is less than or equal to a buffer size.

61, 59, 48, 37, 26, 19, 15, 11, 5, 1

The first two buffers are

61, 59, 5, 1

Page 12: An efficient external sorting with minimal space requirement

392 Motzkin and Hansen

Selecting the key 61. Buffer X is filled with 59, 5, and output. The next buffer read is 48, 37. Buffer X is filled 1, 48, and output. The next input buffer is 26, 19. Again buffer X is filled, 37, 26, and output. The last buffer, 15, 11, is input. X is filled, 19, 15, and output and the remaining assignments are made, with the key inserted.

X

11,61

giving the following file

X X X X x Key

59, 5 1, 48 37, 26 I9, 15 11 61

The second pass starts by inputting the first buffers,

59, 5, 15, 11, with 59 as the key

The remaining keys in the subfille are all less than the key 59, so the following file results:

X X X X X X X X Key Key

5, 15 11, 1 48, 37 26, 19 59 61

This gives one large subtile remaining to be sorted, but it is now unordered. This file will be completely sorted after 4.1 passes, again assuming that the partitioning processes continues for each subtile until its size is two or less.

4.2.2. The Worst Case and the Best Case

Shufflesort does have a worst case situation, though it is not the ordered or reversed ordered sets. The following is a worst case example, using a buffer size of two:

1, 19, 59, 5, 37, 11, 48, 26, 61, 15

The first two buffers are:

1, 19, 61, 15

with 1 as the key. Each of the other keys is larger than one, so they are read in and

assigned to buffer B and printed at the end of the file with the following result:

Key Y Y Y Y Y 1 5 59, 11 37, 26 48, 15 61, 19

Page 13: An efficient external sorting with minimal space requirement

An Efficient External Sorting with Minimal Space Requirement 393

On the second pass, the key, 5, is again the smallest key in the subtile so the records are read in, reordered somewhat, and output giving this file:

Key Key Y YY YY YY

1 5 11,26 37, 15 48, 19 61, 59

The third pass has 11, the smallest key in the subtile, as a key and proceeds as the previous subfiles, resulting in the following file:

Key Key Key YYY YYY YYY YYY

1 5 11 15 37, 19 48, 59 61, 26

Each of the following passes is the same, with the key being the smallest key in the subfiles. There are eight external passes and one internal sort over the two keys.

The worst case yields 0(N) passes over the data. It could be improved to N/4B passes where B is the buffer size. The algorithm, which improves the worst case to N/4B passes where B is the buffer size, is not given here.

A good example might have the following keys:

26, 11, 5, 59, 19, 1, 15, 61, 37, 48

With a buffer size of two, the first two buffers would be:

26, 1l, 37, 48

The key is 26 and the partitioning starts:

X Y

11 48, 37

Buffer Y is output and the next buffer, 15 and 61, is read in. The next buffers are partitioned:

X Y 11, 15 61

X is output and the next buffer read, 5, 59 giving:

X Y 5 59, 61

Page 14: An efficient external sorting with minimal space requirement

394 Motzkin and Hansen

Y if full and output. The last input buffer is: 19, 1. X is filled with 5, 19, and output and filled again with 1 and the key 26 giving this file:

X X X Key Y Y

11, 15 5, 19 1 36 59,61 48, 37

Subtile X is sorted, starting with 11, 15, 19, 1. The key is 11. Buffer Y is tilled with 15, 19, and output. The last key, 5, is read. Buffer X is filled with 1, 5, and output, as is the key.

Subtile Y is also sorted and the resulting tile is:

X X Key X Y Key YX Key YY

1, 5 11 19, 15 26 48, 37 59 61

Each of the sufiles is less than or equal to one buffer in size and may be sorted internally, there are two external passes and three internal sorts, of buffers of size two.

4.3. The Key Controlling the Partioning

The selection of the key for partitioning plays an important role in the sorting. It is best when the key happens to be the median. In the shufflesort algorithm described here, the key of the first record of each subtile is used.

In standard quicksort such a selection i.e., the first key is a serious drawback, since the ordered file, as well as the file in reverse order is then sorted with 0(n 2) comparisons and 0(n) passes.

Here, since shufflesort shuffles the data, the ordered tile becomes "unor- dered" after the first pass, and has average behavior, same is true for the file in reverse order. (see Section 4.2.1).

However, as shown in Section 4.2.2, the worst case still exists, only here it occurs in some unusual situations where for each subtile constructed by shuffling the first key happens to be the smallest. The probability of this extreme situation is remote.

Thus, the selection of the first key as the key controlling the partitioning in shufflesort does not lead to serious problems. The performance can be improved if a value closer to the median were selected. Singleton's procedure t13~ of selecting the median of the three values: first, last, and middle keys, can be easily incorporated into shufflesort. It will require one additional disk access (i.e., to read the middle record) for each subtile being processed. It will cut down the number of comparisons, and therefore, the number of disk accesses, for large subfiles. However, an evaluation of the smallest subtile for which Singleton's procedure still improves the perfor- mance of shufflesort is needed.

Page 15: An efficient external sorting with minimal space requirement

An Efficient External Sorting with Minimal Space Requirement

Table I. Average Results of Experimental Runs

395

Sample size Buffer s i z e Mergesort passes Shuffiesort passes

200 4 6.880 8.192 200 10 5.600 6.565 200 50 3.000 3.641

4.4. The Stack Depth

If the processing is done so that smaller subfiles are processed first then the size of the stack will be at most log 2 N. Usually this number is small enough and the stack may be kept in memory during the sorting.

Experimental Results

The experiemental results support the theoretical evaluation. We compared shuffiesort with a two-way merge sort.

Table I shows the average results obtained from executing the merge sort and the quicksort using ten random samples for each buffer size. A pass is the total number of records read throughout the sorting process divided by the sample size.

4.6. Space Requirement

While shuffiesort is somewhat less efficient in terms of the number of disk access, it is more efficient in space. The technique of starting from both ends of a subtile makes it possible to immediately output full buffers to the same file. In other words, shuffiesort requires only a single copy of the data file. The amount of space needed in memoiy is four buffers, two for input and two for output; and log /N stack locations. Thus the space requirement is minimal.

5. CONCLUDING REMARKS

We have shown that shufflesort has an efficient average behavior yielding:

approximately 1 + 2 Ln ( - ~ - ) passes over the data and

N + I approximately 2(N + 1) Ln ~ comparisons (M ~< 4B)

Page 16: An efficient external sorting with minimal space requirement

396 Motzkin and Hansen

We have also shown that shuffle sort has minimal space requirement i.e., no additional space except the file itself. The minimal space requirement can be achieved only if the data can be read/written from both ends of the file, and on the same file. Therefore, it requires a random access storage device such as a disk and is not appropriate for a sequential device such as tape.

Thus, it is suitable for the situations when there is random access external storage where space requirement is important and where one can afford a sort, which on the average, is slightly slower than a two-way merge. It is particularly suitable for small computers with limited external random storage.

R E F E R E N C E S

1. C. A. R. Hoare, "Partition (Algorithm63), Quicksort (Algorithm64), and Find (Algorithm 65)," Comm. ACM 4(7):321-322, (July 1961).

2. C. A. R. Hoare, "Quicksort," Computer J. 5(4):10-15, (April 1962). 3. E. Horowitz and S. Sahni, Fundamentals of Data Structures (Computer Science Press,

Polomac, Maryland, 1976), pp. 347-350. 4. D. E. Knuth, Sorting and Searching, The Art of Computer Programming3 (Addison-

Wesley, Reading, Mass., 1972). 5. R. Loeser, "Some performance tests of "quicksort" and descendants," Comm. ACM 17,

3:142-152 (March 1974). 6. R. Loeser, "Survey on Algorithms 34, 422, and Quicksort," ACM Translations on

Mathematical Software, 2(3):290-299 (September 1976). 7. D. Motzkin, "A Stable Quicksort," Software-Practice and Experience, 11:607-611

(1981). 8. L. T. Pardo, "Stable Sorting and Merging with Optimal Space and Time Bounds," Siam

J. Comput., 6(2):351-371 (June 1977). 9. P. O. Rotes, "An Algorithm for Merging Disk Files in Place," Questions Informatieae,

1:41-43 (September 1979). 10. R. D. Scowen, "Quickersort (Algorithm 271)," Comm, ACM 8(11):669-670 (November

1965). 11. R. Sedgewick, "Quicksort with equal keys," Siam J. Comput., 6(2):240-267 (June 1977). 12. R. Sedgewick, "The analysis of Quicksort programs," Acta Informatiea 7:327-355

(1977). 13. R. C. Singleton, "An efficient algorithm for sorting with minimal storage

(Algorithm 347)," Comm. ACM 12(3):185-187 (March 1969). 14. M. N. vanEmden, "Increasing the efficiency of quicksort (Algorithm 402)," Comm. ACM

13, 11:693-694 (November 1970).