Slide 1

1 ALGORITMOS DE ORDENACIN EN PARALELO notas extradas fundamentalmente de Ananth Grama, Anshul Gupta, George Karypis, and Vipin Kumar como complemento al libro Introduction to Parallel Computing, Addison Wesley, 2003. Slide 2 2 Topic Overview Issues in Sorting on Parallel Computers Bubble Sort and its Variants Quicksort Bucket and Sample Sort Sorting Networks Other Sorting Algorithms Slide 3 3 Sorting: Overview One of the most commonly used and well-studied kernels. Sorting can be comparison-based or noncomparison-based. The fundamental operation of comparison-based sorting is compare-exchange. The lower bound on any comparison-based sort of n numbers is (nlog n). We focus here on comparison-based sorting algorithms. Slide 4 4 Sorting: Basics What is a parallel sorted sequence? Where are the input and output lists stored? We assume that the input and output lists are distributed. The sorted list is partitioned with the property that each partitioned list is sorted and each element in processor P i 's list is less than that in P j 's list if i < j. Slide 5 5 Sorting: Basics What is the parallel counterpart to a sequential comparator? If each processor has one element, the compare exchange operation stores the smaller element at the processor with smaller id. This can be done in t s + t w time. Slide 6 6 Sorting: Parallel Compare Exchange Operation A parallel compare-exchange operation. Processes P i and P j send their elements to each other. Process P i keeps min{a i,a j }, and P j keeps max{a i, a j }. Slide 7 7 Message-Passing Compare and Exchange Version 1 P 1 sends A to P 2, which compares A and B and sends back B to P 1 if A is larger than B (otherwise it sends back A to P 1 ): Slide 8 8 Alternative Message Passing Method Version 2 For P 1 to send A to P 2 and P 2 to send B to P 1. Then both processes perform compare operations. P 1 keeps the larger of A and B and P 2 keeps the smaller of A and B: Slide 9 9 Sorting: Basics What is the parallel counterpart to a sequential comparator? If we have more than one element per processor, we call this operation a compare split. Assume each of two processors have n/p elements. After the compare-split operation, the smaller n/p elements are at processor P i and the larger n/p elements at P j, where i < j. The time for a compare-split operation is (t s + t w n/p), assuming that the two partial lists were initially sorted. Slide 10 10 Sorting: Parallel Compare Split Operation A compare-split operation. Each process sends its block of size n/p to the other process. Each process merges the received block with its own block and retains only the appropriate half of the merged block. In this example, process P i retains the smaller elements and process P i retains the larger elements. Slide 11 11 Bubble Sort and its Variants The sequential bubble sort algorithm compares and exchanges adjacent elements in the sequence to be sorted: Sequential bubble sort algorithm. Slide 12 12 Slide 13 13 Bubble Sort and its Variants The complexity of bubble sort is (n 2 ). Bubble sort is difficult to parallelize since the algorithm has no concurrency. A simple variant, though, uncovers the concurrency. Slide 14 14 Odd-Even Transposition Sequential odd-even transposition sort algorithm. Slide 15 15 Odd-Even Transposition Sorting n = 8 elements, using the odd-even transposition sort algorithm. During each phase, n = 8 elements are compared. Slide 16 16 Odd-Even Transposition After n phases of odd-even exchanges, the sequence is sorted. Each phase of the algorithm (either odd or even) requires (n) comparisons. Serial complexity is (n 2 ). Slide 17 17 Parallel Odd-Even Transposition Consider the one item per processor case. There are n iterations, in each iteration, each processor does one compare-exchange. The parallel run time of this formulation is (n). This is cost optimal with respect to the base serial algorithm but not the optimal one. Slide 18 18 Parallel Odd-Even Transposition Parallel formulation of odd-even transposition. Slide 19 19 Parallel Odd-Even Transposition Consider a block of n/p elements per processor. The first step is a local sort. In each subsequent step, the compare exchange operation is replaced by the compare split operation. The parallel run time of the formulation is The parallel formulation is cost-optimal for p = O(log n). Slide 20 20 Quicksort Quicksort is one of the most common sorting algorithms for sequential computers because of its simplicity, low overhead, and optimal average complexity. Quicksort selects one of the entries in the sequence to be the pivot and divides the sequence into two - one with all elements less than the pivot and other greater. The process is recursively applied to each of the sublists. Slide 21 21 Quicksort A sequential quicksort algorithm. Slide 22 22 Quicksort Example of the quicksort algorithm sorting a sequence of size n = 8. Slide 23 23 Quicksort The performance of quicksort depends critically on the quality of the pivot. In the best case, the pivot divides the list in such a way that the larger of the two lists does not have more than n elements (for some constant ). In this case, the complexity of quicksort is O(nlog n). Slide 24 24 Parallelizing Quicksort Lets start with recursive decomposition - the list is partitioned serially and each of the subproblems is handled by a different processor. The time for this algorithm is lower-bounded by (n)! Can we parallelize the partitioning step - in particular, if we can use n processors to partition a list of length n around a pivot in O(1) time, we have a winner. This is difficult to do on real machines, though. Slide 25 25 Parallelizing Quicksort: Message Passing Formulation A simple message passing formulation is based on the recursive halving of the machine. Assume that each processor in the lower half of a p processor ensemble is paired with a corresponding processor in the upper half. A designated processor selects and broadcasts the pivot. Each processor splits its local list into two lists, one less (L i ), and other greater (U i ) than the pivot. A processor in the low half of the machine sends its list U i to the paired processor in the other half. The paired processor sends its list L i. It is easy to see that after this step, all elements less than the pivot are in the low half of the machine and all elements greater than the pivot are in the high half. Slide 26 26 Parallelizing Quicksort: Message Passing Formulation The above process is recursed until each processor has its own local list, which is sorted locally. The time for a single reorganization is (log p) for broadcasting the pivot element, (n/p) for splitting the locally assigned portion of the array, (n/p) for exchange and local reorganization. We note that this time is identical to that of the corresponding shared address space formulation. It is important to remember that the reorganization of elements is a bandwidth sensitive operation. Slide 27 27 Slide 28 28 Slide 29 29 Paralelizando Quicksort : alternativas para la seleccin de pivotes El tamao diferente de las sublistas que se van generando en el quicksort no afecta a la eficiencia del algoritmo secuencial, pero s a la del algoritmo paralelo por el posible desequilibrio en el reparto de la carga Para dividir una lista en dos sublistas de igual tamao, el valor ideal para el pivote sera la mediana, pero obtenerla supondra un coste considerable Para la seleccin de pivotes en el quicksort paralelo podemos: 1.Elegir (aleatoriamente o de forma predeterminada) pivote en cada iteracin y para cada grupo de procesadores involucrado 2.Elegir pivote en cada iteracin y para cada grupo de procesadores realizando un muestreo previo entre los elementos de cada grupo de procesadores involucrado 3.Realizar un muestreo previo con m elementos de cada uno de los p procesadores, y seleccionar las posiciones que separaran el vector muestra en p partes como los p-1 pivotes que se utilizarn en los diferentes pasos Slide 30 30 Bucket and Sample Sort In Bucket sort, the range [a,b] of input numbers is divided into m equal sized intervals, called buckets. Each element is placed in its appropriate bucket. If the numbers are uniformly divided in the range, the buckets can be expected to have roughly identical number of elements. Elements in the buckets are locally sorted. The run time of this algorithm is (nlog(n/m)). Slide 31 31 Parallel Bucket Sort Parallelizing bucket sort is relatively simple. We can select m = p. In this case, each processor has a range of values it is responsible for. Each processor runs through its local list and assigns each of its elements to the appropriate processor. The elements are sent to the destination processors using a single all-to-all personalized communication. Each processor sorts all the elements it receives. Slide 32 32 Parallel Bucket and Sample Sort The critical aspect of the above algorithm is one of assigning ranges to processors. This is done by suitable splitter selection. The splitter selection method divides the n elements into m blocks of size n/m each, and sorts each block by using quicksort. From each sorted block it chooses m 1 evenly spaced elements. The m(m 1) elements selected from all the blocks represent the sample used to determine the buckets. This scheme guarantees that the number of elements ending up in each bucket is less than 2n/m. Slide 33 33 Parallel Bucket and Sample Sort An example of the execution of sample sort on an array with 24 elements on three processes. Slide 34 34 Parallel Bucket and Sample Sort The splitter selection scheme can itself be parallelized. Each processor generates the p 1 local splitters in parallel. All processors share their splitters using a single all-to-all broadcast operation. Each processor sorts the p(p 1) elements it receives and selects p 1 uniformly spaces splitters from them. Slide 35 35 Parallel Bucket and Sample Sort: Analysis The internal sort of n/p elements requires time ((n/p)log(n/p)), and the selection of p 1 sample elements requires time (p). The time for an all-to-all broadcast is (p 2 ), the time to internally sort the p(p 1) sample elements is (p 2 log p), and selecting p 1 evenly spaced splitters takes time (p). Each process can insert these p 1splitters in its local sorted block of size n/p by performing p 1 binary searches in time (plog(n/p)). The time for reorganization of the elements is O(n/p). Slide 36 36 Parallel Bucket and Sample Sort: Analysis The total time is given by: The isoefficiency of the formulation is (p 3 log p). Slide 37 37 Sorting Networks Networks of comparators designed specifically for sorting. A comparator is a device with two inputs x and y and two outputs x' and y'. For an increasing comparator, x' = min{x,y} and y' = min{x,y}; and vice-versa. We denote an increasing comparator by and a decreasing comparator by . The speed of the network is proportional to its depth. Slide 38 38 Sorting Networks: Comparators A schematic representation of comparators: (a) an increasing comparator, and (b) a decreasing comparator. Slide 39 39 Sorting Networks A typical sorting network. Every sorting network is made up of a series of columns, and each column contains a number of comparators connected in parallel. Slide 40 40 Sorting Networks: Bitonic Sort A bitonic sorting network sorts n elements in (log 2 n) time. A bitonic sequence has two tones - increasing and decreasing, or vice versa. Any cyclic rotation of such networks is also considered bitonic. 1,2,4,7,6,0 is a bitonic sequence, because it first increases and then decreases. 8,9,2,1,0,4 is another bitonic sequence, because it is a cyclic shift of 0,4,8,9,2,1 . The kernel of the network is the rearrangement of a bitonic sequence into a sorted sequence. Slide 41 41 Bitonic Sequences Slide 42 42 Sorting Networks: Bitonic Sort Let s = a 0,a 1,,a n-1 be a bitonic sequence such that a 0 a 1 a n/2-1 and a n/2 a n/2+1 a n-1. Consider the following subsequences of s: s 1 = min{a 0,a n/2 },min{a 1,a n/2+1 },,min{a n/2-1,a n-1 } s 2 = max{a 0,a n/2 },max{a 1,a n/2+1 },,max{a n/2-1,a n-1 } (1) Note that s 1 and s 2 are both bitonic and each element of s 1 is less than every element in s 2. We can apply the procedure recursively on s 1 and s 2 to get the sorted sequence. Slide 43 43 Creating two bitonic sequences from one bitonic sequence Starting with the bitonic sequence 3, 5, 8, 9, 7, 4, 2, 1 we get: Slide 44 44 Sorting a bitonic sequence Compare-and-exchange moves smaller numbers of each pair to left and larger numbers of pair to right. Given a bitonic sequence, recursively performing operations will sort the list. Slide 45 45 Sorting Networks: Bitonic Sort Merging a 16-element bitonic sequence through a series of log 16 bitonic splits. Slide 46 46 Sorting Networks: Bitonic Sort We can easily build a sorting network to implement this bitonic merge algorithm. Such a network is called a bitonic merging network. The network contains log n columns. Each column contains n/2 comparators and performs one step of the bitonic merge. We denote a bitonic merging network with n inputs by BM [n]. Replacing the comparators by comparators results in a decreasing output sequence; such a network is denoted by BM [n]. Slide 47 47 Sorting Networks: Bitonic Sort A bitonic merging network for n = 16. The input wires are numbered 0,1,, n - 1, and the binary representation of these numbers is shown. Each column of comparators is drawn separately; the entire figure represents a BM[ 16 ] bitonic merging network. The network takes a bitonic sequence and outputs it in sorted order. Slide 48 48 Sorting Networks: Bitonic Sort How do we sort an unsorted sequence using a bitonic merge? We must first build a single bitonic sequence from the given sequence. A sequence of length 2 is a bitonic sequence. A bitonic sequence of length 4 can be built by sorting the first two elements using BM[ 2 ] and next two, using BM[ 2 ]. This process can be repeated to generate larger bitonic sequences. Slide 49 49 Bitonic Mergesort Slide 50 50 Slide 51 51 Sorting Networks: Bitonic Sort A schematic representation of a network that converts an input sequence into a bitonic sequence. In this example, BM[ k ] and BM[ k ] denote bitonic merging networks of input size k that use and comparators, respectively. The last merging network ( BM[ 16 ] ) sorts the input. In this example, n = 16. Slide 52 52 Sorting Networks: Bitonic Sort The comparator network that transforms an input sequence of 16 unordered numbers into a bitonic sequence. Slide 53 53 Sorting Networks: Bitonic Sort The depth of the network is (log 2 n). Each stage of the network contains n/2 comparators. A serial implementation of the network would have complexity (nlog 2 n). Slide 54 54 Mapping Bitonic Sort to Hypercubes Consider the case of one item per processor. The question becomes one of how the wires in the bitonic network should be mapped to the hypercube interconnect. Note from our earlier examples that the compare-exchange operation is performed between two wires only if their labels differ in exactly one bit! This implies a direct mapping of wires to processors. All communication is nearest neighbor! Slide 55 55 Mapping Bitonic Sort to Hypercubes Communication during the last stage of bitonic sort. Each wire is mapped to a hypercube process; each connection represents a compare-exchange between processes. Slide 56 56 Mapping Bitonic Sort to Hypercubes Communication characteristics of bitonic sort on a hypercube. During each stage of the algorithm, processes communicate along the dimensions shown. Slide 57 57 Mapping Bitonic Sort to Hypercubes Parallel formulation of bitonic sort on a hypercube with n = 2 d processes. Slide 58 58 Mapping Bitonic Sort to Hypercubes During each step of the algorithm, every process performs a compare-exchange operation (single nearest neighbor communication of one word). Since each step takes (1) time, the parallel time is T p = (log 2 n) (2) This algorithm is cost optimal w.r.t. its serial counterpart, but not w.r.t. the best sorting algorithm. Slide 59 59 Mapping Bitonic Sort to Meshes The connectivity of a mesh is lower than that of a hypercube, so we must expect some overhead in this mapping. Consider the row-major shuffled mapping of wires to processors. Slide 60 60 Mapping Bitonic Sort to Meshes Different ways of mapping the input wires of the bitonic sorting network to a mesh of processes: (a) row-major mapping, (b) row-major snakelike mapping, and (c) row- major shuffled mapping. Slide 61 61 Mapping Bitonic Sort to Meshes The last stage of the bitonic sort algorithm for n = 16 on a mesh, using the row-major shuffled mapping. During each step, process pairs compare-exchange their elements. Arrows indicate the pairs of processes that perform compare-exchange operations. Slide 62 62 Mapping Bitonic Sort to Meshes In the row-major shuffled mapping, wires that differ at the i th least-significant bit are mapped onto mesh processes that are 2 (i-1)/2 communication links away. The total amount of communication performed by each process is. The total computation performed by each process is (log 2 n). The parallel runtime is: This is not cost optimal. Slide 63 63 Block of Elements Per Processor Each process is assigned a block of n/p elements. The first step is a local sort of the local block. Each subsequent compare-exchange operation is replaced by a compare-split operation. We can effectively view the bitonic network as having (1 + log p)(log p)/2 steps. Slide 64 64 Block of Elements Per Processor: Hypercube Initially the processes sort their n/p elements (using merge sort) in time ((n/p)log(n/p)) and then perform (log 2 p) compare-split steps. The parallel run time of this formulation is Comparing to an optimal sort, the algorithm can efficiently use up to processes. The isoefficiency function due to both communication and extra work is (p log p log 2 p). Slide 65 65 Block of Elements Per Processor: Mesh The parallel runtime in this case is given by: This formulation can efficiently use up to p = (log 2 n) processes. The isoefficiency function is Slide 66 66 Performance of Parallel Bitonic Sort The performance of parallel formulations of bitonic sort for n elements on p processes. Slide 67 67 Other Sorting Algorithms We began by giving the lower bound for the time complexity of a sequential sorting algorithm based upon comparisons as O(nlogn). Consequently, the time complexity of a parallel sorting algorithm based upon comparisons is O((logn)/p) with p processors or O(logn) with n processors. There are sorting algorithms that can achieve better than O(nlogn) sequential time complexity and are very attractive candidates for parallelization but they often assume special properties of the numbers being sorted. Slide 68 68 Radix Sort Assumes numbers to sort are represented in a positional digit representation such as binary and decimal numbers. The digits represent values and position of each digit indicates their relative weighting. Radix sort starts at the least significant digit and sorts the numbers according to their least significant digits. The sequence is then sorted according to the next least significant digit and so on until the most significant digit, after which the sequence is sorted. For this to work, it is necessary that the order of numbers with the same digit is maintained, that is, one must use a stable sorting algorithm. Slide 69 69 Radix sort using decimal digits Slide 70 70 Radix sort using binary digits Slide 71 71 Parallelizing radix sort Radix sort can be parallelized by using a parallel sorting algorithm in each phase of sorting on bits or groups of bits. Already mentioned parallelized counting sort using prefix sum calculation, which leads to O(logn) time with n - 1 processors and constant b and r. Slide 72 72 Example of parallelizing radix sort sorting on binary digits Can use prefix-sum calculation for positioning each number at each stage. When prefix sum calculation applied to a column of bits, it gives number of 1s up to each digit position because all digits can only be 0 or 1 and prefix calculation will simply add number of 1s. A second prefix calculation can also give the number of 0s up to each digit position by performing the prefix calculation on the digits inverted (diminished prefix sum). When digit considered being a 0, diminished prefix sum calculation provides new position for number. When digit considered being a 1, result of normal prefix sum calculation plus largest diminished prefix calculation gives final position for number. Slide 73 73 Slide 74 74 Ordenacin en paralelo en memoria compartida Slide 75 75 Parallelizing Quicksort: Shared Address Space Formulation Consider a list of size n equally divided across p processors. A pivot is selected by one of the processors and made known to all processors. Each processor partitions its list into two, say L i and U i, based on the selected pivot. All of the L i lists are merged and all of the U i lists are merged separately. The set of processors is partitioned into two (in proportion of the size of lists L and U). The process is recursively applied to each of the lists. Slide 76 76 Shared Address Space Formulation Slide 77 77 Parallelizing Quicksort: Shared Address Space Formulation The only thing we have not described is the global reorganization (merging) of local lists to form L and U. The problem is one of determining the right location for each element in the merged list. Each processor computes the number of elements locally less than and greater than pivot. It computes two sum-scans to determine the starting location for its elements in the merged L and U lists. Once it knows the starting locations, it can write its elements safely. Slide 78 78 Parallelizing Quicksort: Shared Address Space Formulation Efficient global rearrangement of the array. Slide 79 79 Parallelizing Quicksort: Shared Address Space Formulation The parallel time depends on the split and merge time, and the quality of the pivot. The latter is an issue independent of parallelism, so we focus on the first aspect, assuming ideal pivot selection. The algorithm executes in four steps: (i) determine and broadcast the pivot; (ii) locally rearrange the array assigned to each process; (iii) determine the locations in the globally rearranged array that the local elements will go to; and (iv) perform the global rearrangement. The first step takes time (log p), the second, (n/p), the third, (log p), and the fourth, (n/p). The overall complexity of splitting an n-element array is (n/p) + (log p). Slide 80 80 Parallelizing Quicksort: Shared Address Space Formulation The process recurses until there are p lists, at which point, the lists are sorted locally. Therefore, the total parallel time is: The corresponding isoefficiency is (plog 2 p) due to broadcast and scan operations.