Uniprocessor Optimizations and Matrix Multiplication
description
Transcript of Uniprocessor Optimizations and Matrix Multiplication
![Page 1: Uniprocessor Optimizations and Matrix Multiplication](https://reader033.fdocuments.net/reader033/viewer/2022042514/56815fb2550346895dceae74/html5/thumbnails/1.jpg)
1
Uniprocessor Optimizations and
Matrix Multiplication
![Page 2: Uniprocessor Optimizations and Matrix Multiplication](https://reader033.fdocuments.net/reader033/viewer/2022042514/56815fb2550346895dceae74/html5/thumbnails/2.jpg)
2
Outline• Parallelism in Modern Processors
• Memory Hierarchies
• Matrix Multiply Cache Optimizations
![Page 3: Uniprocessor Optimizations and Matrix Multiplication](https://reader033.fdocuments.net/reader033/viewer/2022042514/56815fb2550346895dceae74/html5/thumbnails/3.jpg)
3
Idealized Uniprocessor Model• Processor names bytes, words, etc. in its address space
• These represent integers, floats, pointers, arrays, etc.• Exist in the program stack, static region, or heap
• Operations include• Read and write (given an address/pointer)• Arithmetic and other logical operations
• Order specified by program• Read returns the most recently written data• Compiler and architecture translate high level expressions into
“obvious” lower level instructions• Hardware executes instructions in order specified by compiler
• Cost• Each operations has roughly the same cost
(read, write, add, multiply, etc.)
![Page 4: Uniprocessor Optimizations and Matrix Multiplication](https://reader033.fdocuments.net/reader033/viewer/2022042514/56815fb2550346895dceae74/html5/thumbnails/4.jpg)
4
Uniprocessors in the Real World• Real processors have
• registers and caches• small amounts of fast memory• store values of recently used or nearby data• different memory ops can have very different costs
• parallelism• multiple “functional units” that can run in parallel• different orders, instruction mixes have different costs
• pipelining• a form of parallelism, like an assembly line in a factory
• Why is this your problem?In theory, compilers understand all of this and can optimize
your program; in practice they don’t.
![Page 5: Uniprocessor Optimizations and Matrix Multiplication](https://reader033.fdocuments.net/reader033/viewer/2022042514/56815fb2550346895dceae74/html5/thumbnails/5.jpg)
5
What is Pipelining?
• In this example:• Sequential execution takes
4 * 90min = 6 hours• Pipelined execution takes
30+4*40+20 = 3.3 hours
• Pipelining helps throughput, but not latency
• Pipeline rate limited by slowest pipeline stage
• Potential speedup = Number pipe stages
• Time to “fill” pipeline and time to “drain” it reduces speedup
A
B
C
D
6 PM 7 8 9
Task
Order
Time
30 40 40 40 40 20
Dave Patterson’s Laundry example: 4 people doing laundry
wash (30 min) + dry (40 min) + fold (20 min)
![Page 6: Uniprocessor Optimizations and Matrix Multiplication](https://reader033.fdocuments.net/reader033/viewer/2022042514/56815fb2550346895dceae74/html5/thumbnails/6.jpg)
6
Limits to Instruction Level Parallelism (ILP)
• Limits to pipelining: Hazards prevent next instruction from executing during its designated clock cycle
• Structural hazards: HW cannot support this combination of instructions (single person to fold and put clothes away)
• Data hazards: Instruction depends on result of prior instruction still in the pipeline (missing sock)
• Control hazards: Caused by delay between the fetching of instructions and decisions about changes in control flow (branches and jumps).
• The hardware and compiler will try to reduce these:• Reordering instructions, multiple issue, dynamic branch prediction,
speculative execution…
• You can also enable parallelism by careful coding
![Page 7: Uniprocessor Optimizations and Matrix Multiplication](https://reader033.fdocuments.net/reader033/viewer/2022042514/56815fb2550346895dceae74/html5/thumbnails/7.jpg)
7
Outline• Parallelism in Modern Processors
• Memory Hierarchies
• Matrix Multiply Cache Optimizations
![Page 8: Uniprocessor Optimizations and Matrix Multiplication](https://reader033.fdocuments.net/reader033/viewer/2022042514/56815fb2550346895dceae74/html5/thumbnails/8.jpg)
8
Memory Hierarchy• Most programs have a high degree of locality in their accesses
• spatial locality: accessing things nearby previous accesses• temporal locality: reusing an item that was previously accessed
• Memory hierarchy tries to exploit locality
on-chip cacheregisters
datapath
control
processor
Second level
cache (SRAM)
Main memory
(DRAM)
Secondary storage (Disk)
Tertiary storage
(Disk/Tape)
Speed (ns): 1 10 100 10 ms 10 sec
Size (bytes): 100s Ks Ms Gs Ts
![Page 9: Uniprocessor Optimizations and Matrix Multiplication](https://reader033.fdocuments.net/reader033/viewer/2022042514/56815fb2550346895dceae74/html5/thumbnails/9.jpg)
9
Processor-DRAM Gap (latency)
µProc60%/yr.
DRAM7%/yr.
1
10
100
100019
8019
81
1983
1984
1985
1986
1987
1988
1989
1990
1991
1992
1993
1994
1995
1996
1997
1998
1999
2000
DRAM
CPU19
82
Processor-MemoryPerformance Gap:(grows 50% / year)
Perf
orm
ance
Time
“Moore’s Law”
• Memory hierarchies are getting deeper• Processors get faster more quickly than memory
![Page 10: Uniprocessor Optimizations and Matrix Multiplication](https://reader033.fdocuments.net/reader033/viewer/2022042514/56815fb2550346895dceae74/html5/thumbnails/10.jpg)
10
Experimental Study of Memory• Microbenchmark for memory system performance
• time the following program for each size(A) and stride s
(repeat to obtain confidence and mitigate timer resolution)
for array A of size from 4KB to 8MB by 2x
for stride s from 8 Bytes (1 word) to size(A)/2 by 2x
for i from 0 to size by s
load A[i] from memory (8 Bytes)
![Page 11: Uniprocessor Optimizations and Matrix Multiplication](https://reader033.fdocuments.net/reader033/viewer/2022042514/56815fb2550346895dceae74/html5/thumbnails/11.jpg)
11
Memory Hierarchy on a Sun Ultra-IIi
L2: 2 MB, 36 ns
(12 cycles)
L1: 16K, 6 ns
(2 cycle)
Mem: 396 ns
(132 cycles)
Sun Ultra-IIi, 333 MHz
L2: 64 byte line 8 K pages
See www.cs.berkeley.edu/~yelick/arvindk/t3d-isca95.ps for details
L1: 16 byte line
Array size
![Page 12: Uniprocessor Optimizations and Matrix Multiplication](https://reader033.fdocuments.net/reader033/viewer/2022042514/56815fb2550346895dceae74/html5/thumbnails/12.jpg)
12
Memory Hierarchy on a Pentium III
L1: 32 byte line ?
L2: 512 KB 60 ns
L1: 64K5 ns, 4-way?
Katmai processor on Millennium, 550 MHz Array size
![Page 13: Uniprocessor Optimizations and Matrix Multiplication](https://reader033.fdocuments.net/reader033/viewer/2022042514/56815fb2550346895dceae74/html5/thumbnails/13.jpg)
13
Lessons• Actual performance of a simple program can be a
complicated function of the architecture• Slight changes in the architecture or program change the performance
significantly• To write fast programs, need to consider architecture• We would like simple models to help us design efficient algorithms• Is this possible?
• We will illustrate with a common technique for improving cache performance, called blocking or tiling
• Idea: used divide-and-conquer to define a problem that fits in register/L1-cache/L2-cache
![Page 14: Uniprocessor Optimizations and Matrix Multiplication](https://reader033.fdocuments.net/reader033/viewer/2022042514/56815fb2550346895dceae74/html5/thumbnails/14.jpg)
14
Outline• Parallelism in Modern Processors
• Memory Hierarchies
• Matrix Multiply Cache Optimizations
![Page 15: Uniprocessor Optimizations and Matrix Multiplication](https://reader033.fdocuments.net/reader033/viewer/2022042514/56815fb2550346895dceae74/html5/thumbnails/15.jpg)
15
Note on Matrix Storage• A matrix is a 2-D array of elements, but memory
addresses are “1-D”
• Conventions for matrix layout• by column, or “column major” (Fortran default)• by row, or “row major” (C default)
0
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
0
4
8
12
16
1
5
9
13
17
2
6
10
14
18
3
7
11
15
19
Column major Row major
![Page 16: Uniprocessor Optimizations and Matrix Multiplication](https://reader033.fdocuments.net/reader033/viewer/2022042514/56815fb2550346895dceae74/html5/thumbnails/16.jpg)
16
Optimizing Matrix Addition for Caches• Dimension A(n,n), B(n,n), C(n,n)
• A, B, C stored by column (as in Fortran)
• Algorithm 1:• for i=1:n, for j=1:n, A(i,j) = B(i,j) + C(i,j)
• Algorithm 2:• for j=1:n, for i=1:n, A(i,j) = B(i,j) + C(i,j)
• What is “memory access pattern” for Algs 1 and 2?
• Which is faster?
• What if A, B, C stored by row (as in C)?
![Page 17: Uniprocessor Optimizations and Matrix Multiplication](https://reader033.fdocuments.net/reader033/viewer/2022042514/56815fb2550346895dceae74/html5/thumbnails/17.jpg)
17
• Assume just 2 levels in the hierarchy, fast and slow
• All data initially in slow memory• m = number of memory elements (words) moved between fast and slow
memory • tm = time per slow memory operation• f = number of arithmetic operations• tf = time per arithmetic operation << tm
• q = f / m average number of flops per slow element access
• Minimum possible time = f* tf when all data in fast memory
• Actual time
• Larger q means Time closer to minimum f * tf
Using a Simple Model of Memory to Optimize
Key to algorithm efficiency
f * tf + m * tm = f * tf * (1 + tm/tf * 1/q) Key to machine efficiency
![Page 18: Uniprocessor Optimizations and Matrix Multiplication](https://reader033.fdocuments.net/reader033/viewer/2022042514/56815fb2550346895dceae74/html5/thumbnails/18.jpg)
18
Simple example using memory model
s = 0
for i = 1, n
s = s + h(X[i])
• Assume tf=1 Mflop/s on fast memory
• Assume moving data is tm = 10
• Assume h takes q flops
• Assume array X is in slow memory
• To see results of changing q, consider simple computation
• So m = n and f = q*n
• Time = read X + compute = 10*n + q*n
• Mflop/s = f/t = q/(10 + q)
• As q increases, this approaches the “peak” speed of 1 Mflop/s
![Page 19: Uniprocessor Optimizations and Matrix Multiplication](https://reader033.fdocuments.net/reader033/viewer/2022042514/56815fb2550346895dceae74/html5/thumbnails/19.jpg)
19
Warm up: Matrix-vector multiplication{implements y = y + A*x}
for i = 1:n
for j = 1:n
y(i) = y(i) + A(i,j)*x(j)
= + *
y(i) y(i)
A(i,:)
x(:)
![Page 20: Uniprocessor Optimizations and Matrix Multiplication](https://reader033.fdocuments.net/reader033/viewer/2022042514/56815fb2550346895dceae74/html5/thumbnails/20.jpg)
20
Warm up: Matrix-vector multiplication{read x(1:n) into fast memory}
{read y(1:n) into fast memory}
for i = 1:n
{read row i of A into fast memory}
for j = 1:n
y(i) = y(i) + A(i,j)*x(j)
{write y(1:n) back to slow memory}
• m = number of slow memory refs = 3n + n2
• f = number of arithmetic operations = 2n2
• q = f / m ~= 2
• Matrix-vector multiplication limited by slow memory speed
![Page 21: Uniprocessor Optimizations and Matrix Multiplication](https://reader033.fdocuments.net/reader033/viewer/2022042514/56815fb2550346895dceae74/html5/thumbnails/21.jpg)
21
“Naïve” Matrix Multiply{implements C = C + A*B}
for i = 1 to n
for j = 1 to n
for k = 1 to n
C(i,j) = C(i,j) + A(i,k) * B(k,j)
= + *C(i,j) C(i,j) A(i,:)
B(:,j)
![Page 22: Uniprocessor Optimizations and Matrix Multiplication](https://reader033.fdocuments.net/reader033/viewer/2022042514/56815fb2550346895dceae74/html5/thumbnails/22.jpg)
22
“Naïve” Matrix Multiply{implements C = C + A*B}for i = 1 to n
{read row i of A into fast memory}
for j = 1 to n
{read C(i,j) into fast memory}
{read column j of B into fast memory}
for k = 1 to n
C(i,j) = C(i,j) + A(i,k) * B(k,j)
{write C(i,j) back to slow memory}
= + *C(i,j) A(i,:)
B(:,j)C(i,j)
![Page 23: Uniprocessor Optimizations and Matrix Multiplication](https://reader033.fdocuments.net/reader033/viewer/2022042514/56815fb2550346895dceae74/html5/thumbnails/23.jpg)
23
“Naïve” Matrix MultiplyNumber of slow memory references on unblocked matrix multiply
m = n3 read each column of B n times
+ n2 read each row of A once
+ 2n2 read and write each element of C once
= n3 + 3n2
So q = f / m = 2n3 / (n3 + 3n2)
~= 2 for large n, no improvement over matrix-vector multiply
= + *C(i,j) C(i,j) A(i,:)
B(:,j)
![Page 24: Uniprocessor Optimizations and Matrix Multiplication](https://reader033.fdocuments.net/reader033/viewer/2022042514/56815fb2550346895dceae74/html5/thumbnails/24.jpg)
24
Blocked (Tiled) Matrix MultiplyConsider A,B,C to be N by N matrices of b by b subblocks where b=n / N is
called the block size
for i = 1 to N
for j = 1 to N
{read block C(i,j) into fast memory}
for k = 1 to N
{read block A(i,k) into fast memory}
{read block B(k,j) into fast memory}
C(i,j) = C(i,j) + A(i,k) * B(k,j) {do a matrix multiply on blocks}
{write block C(i,j) back to slow memory}
= + *C(i,j) C(i,j) A(i,k)
B(k,j)
![Page 25: Uniprocessor Optimizations and Matrix Multiplication](https://reader033.fdocuments.net/reader033/viewer/2022042514/56815fb2550346895dceae74/html5/thumbnails/25.jpg)
25
Blocked (Tiled) Matrix MultiplyRecall: m is amount memory traffic between slow and fast memory matrix has nxn elements, and NxN blocks each of size bxb f is number of floating point operations, 2n3 for this problem q = f / m is our measure of algorithm efficiency in the memory system
So:m = N*n2 read each block of B N3 times (N3 * n/N * n/N)
+ N*n2 read each block of A N3 times + 2n2 read and write each block of C once = (2N + 2) * n2
So q = f / m = 2n3 / ((2N + 2) * n2) ~= n / N = b for large n
So we can improve performance by increasing the blocksize b Can be much faster than matrix-vector multiply (q=2)
![Page 26: Uniprocessor Optimizations and Matrix Multiplication](https://reader033.fdocuments.net/reader033/viewer/2022042514/56815fb2550346895dceae74/html5/thumbnails/26.jpg)
26
Limits to Optimizing Matrix Multiply
The blocked algorithm has ratio q ~= b• The large the block size, the more efficient our algorithm will be• Limit: All three blocks from A,B,C must fit in fast memory (cache),
so we cannot make these blocks arbitrarily large: 3b2 <= M, so q ~= b <= sqrt(M/3)
There is a lower bound result that says we cannot do any better than this (using only algebraic associativity)
Theorem (Hong & Kung, 1981): Any reorganization of this algorithm (that uses only algebraic associativity) is limited to q = O(sqrt(M))
![Page 27: Uniprocessor Optimizations and Matrix Multiplication](https://reader033.fdocuments.net/reader033/viewer/2022042514/56815fb2550346895dceae74/html5/thumbnails/27.jpg)
27
Basic Linear Algebra Subroutines• Industry standard interface (evolving)
• Vendors, others supply optimized implementations
• History• BLAS1 (1970s):
• vector operations: dot product, saxpy (y=*x+y), etc• m=2*n, f=2*n, q ~1 or less
• BLAS2 (mid 1980s)• matrix-vector operations: matrix vector multiply, etc• m=n^2, f=2*n^2, q~2, less overhead • somewhat faster than BLAS1
• BLAS3 (late 1980s)• matrix-matrix operations: matrix matrix multiply, etc• m >= 4n^2, f=O(n^3), so q can possibly be as large as n, so BLAS3 is
potentially much faster than BLAS2
• Good algorithms used BLAS3 when possible (LAPACK)
• See www.netlib.org/blas, www.netlib.org/lapack
![Page 28: Uniprocessor Optimizations and Matrix Multiplication](https://reader033.fdocuments.net/reader033/viewer/2022042514/56815fb2550346895dceae74/html5/thumbnails/28.jpg)
28
BLAS speeds on an IBM RS6000/590
BLAS 3
BLAS 2BLAS 1
BLAS 3 (n-by-n matrix matrix multiply) vs BLAS 2 (n-by-n matrix vector multiply) vs BLAS 1 (saxpy of n vectors)
Peak speed = 266 Mflops
Peak
![Page 29: Uniprocessor Optimizations and Matrix Multiplication](https://reader033.fdocuments.net/reader033/viewer/2022042514/56815fb2550346895dceae74/html5/thumbnails/29.jpg)
29
Locality in Other Algorithms• The performance of any algorithm is limited by q
• In matrix multiply, we increase q by changing computation order
• increased temporal locality
• For other algorithms and data structures, even hand-transformations are still an open problem
• sparse matrices (reordering, blocking)• trees (B-Trees are for the disk level of the hierarchy)• linked lists (some work done here)
![Page 30: Uniprocessor Optimizations and Matrix Multiplication](https://reader033.fdocuments.net/reader033/viewer/2022042514/56815fb2550346895dceae74/html5/thumbnails/30.jpg)
30
Optimizing in Practice• Tiling for registers
• loop unrolling, use of named “register” variables
• Tiling for multiple levels of cache
• Exploiting fine-grained parallelism in processor• superscalar; pipelining
• Complicated compiler interactions
• Automatic optimization an active research area• BeBOP: www.cs.berkeley.edu/~richie/bebop • PHiPAC: www.icsi.berkeley.edu/~bilmes/phipac in particular tr-98-035.ps.gz• ATLAS: www.netlib.org/atlas
![Page 31: Uniprocessor Optimizations and Matrix Multiplication](https://reader033.fdocuments.net/reader033/viewer/2022042514/56815fb2550346895dceae74/html5/thumbnails/31.jpg)
31
Summary• Performance programming on uniprocessors requires
• understanding of fine-grained parallelism in processor • produce good instruction mix
• understanding of memory system• levels, costs, sizes• improve locality
• Blocking (tiling) is a basic approach • Techniques apply generally, but the details (e.g., block size) are
architecture dependent• Similar techniques are possible on other data structures and
algorithms