1CPS4150

Chapter 4Assessing and Understanding Performance

2CPS4150

• Measure, Report, and Summarize

• Make intelligent choices

• See through the marketing hype

• Key to understanding underlying organizational motivation

Why is some hardware better than others for different programs?

What factors of system performance are hardware related?(e.g., Do we need a new machine, or a new operating system?)

How does the machine's instruction set affect performance?

4.1 Performance

3CPS4150

• For example, to improve the performance of a software system, we may need to understand what factors in the hardware contribute to the overall system performance and the relative importance of these factors.

• The factors may include how well the program uses the instructions of the computer, how well the underlying hardware implements the instructions, and how well the memory and I/O system performance.

• Understanding how to determine the performance impact of these factors is crucial to understanding the motivation behind the design of particular aspects of the computer.

Performance

4CPS4150

Which of these airplanes has the best performance?

Airplane Passengers Range (mi) Speed (mph)

Boeing 737-100 375 4630 610Boeing 747 470 4150 610BAC/Sud Concorde 132 4000 1350Douglas DC-8-50 146 8720 544

•How much faster is the Concorde compared to the 747?

•How much bigger is the 747 than the Douglas DC-8?

•If we define performance in term of speed. There are two possibility:

•The fastest one as the one with the highest cruising speed/one

passenger

•The one transporting 450 passengers from place to another place

5CPS4150

• Response Time/execution Time (latency)

(The total time required for the computer to complete a task.)

— How long does it take for my job to run?

— How long does it take to execute a job?

— How long must I wait for the database query?

— Individual computer user will be interested in this value

(how fast the computer can get my job done)

• Throughput

— How many jobs can the machine run at once?

— What is the average execution rate?

— How much work is getting done in a given time?

— Data center managers are often interested in increasing this

value

• If we upgrade a machine with a new processor what do we increase?

• If we add a new machine to the lab what do we increase?

• Example p.242

Computer Performance: TIME, TIME, TIME

6CPS4150

• For some program running on machine X,

PerformanceX = 1 / Execution timeX

• "X is n times faster than Y"

PerformanceX / PerformanceY = nOR

Excution timey / Excution timex = n

• Example:– machine A runs a program in 20 seconds– machine B runs the same program in 25 secondsHow much faster is A than B?

• Do Example: Page 243

Book's Definition of Performance

7CPS4150

• Elapsed Time (or response time, system performance)– The total time to complete a task– counts everything (disk and memory accesses, I/O , operating

system overhead, etc.)– computer is often shared and a processor may work on several

programs simultaneously optimize throughput rather than minimize the elapsed time for one program

– a useful number, but often not good for comparison purposes• CPU time (CPU performance)

– doesn't count I/O or time spent running other programs– can be broken up into system CPU time (operating system

performing tasks on behalf of the program), and user CPU time(CPU time spent in the program)

• Our focus: user CPU time – time spent executing the lines of code that are "in" our program

Execution Time

8CPS4150

Clock Cycles

– A measure that relates to how fast the hardware can perform basic functions

– A clock that runs at a constant rate and determines when events tale place in the hardware.

– These discrete time intervals are called clock cycle.

– The length of a clock period both as the time for a complete clock cycle and as the clock rate which is the inverse of the clock period.

– Example:

clock cycle: 0.25 nanoseconds (ns)

clock rate: 4 gigahertz (GHz)

9CPS4150

4.2 Clock Cycles

• Instead of reporting execution time in seconds, we often use cycles

• Our goal reduce either the length of the clock cycle or the number of clock cycles

• BUT, a trade-ff between the number of clock cycles needed for a program and the length of each cycle.

cycle

seconds

program afor

cyclesCPU_Clock______ programafortimeexecutionCPU

RateClock

Program afor cyclesCPU_Clock______ programafortimeexecutionCPU

10CPS4150

4.2 Clock Cycles

• Instead of reporting execution time in seconds, we often use cycles

• Clock “ticks” indicate when to start activities (one abstraction):

• cycle time = time between ticks = seconds per cycle

• clock rate (frequency) = cycles per second (1 Hz. = 1 cycle/sec)

A 4 Ghz. clock has a cycle time

time

cycle

seconds

program

cycles

program

seconds

(ps) spicosecond 2501210 9104

1

11CPS4150

So, to improve performance (everything else being equal) you can either

(increase or decrease?)

________ the # of required cycles for a program, or

________ the clock cycle time or, said another way,

________ the clock rate.

How to Improve Performance

seconds

program

cycles

program

seconds

cycle

12CPS4150

• Could assume that number of cycles equals number of instructions

This assumption is incorrect,

different instructions take different amounts of time on different machines.

Why? hint: remember that these are machine instructions, not lines of C code

time

1st

inst

ruct

ion

2nd

inst

ruct

ion

3rd

inst

ruct

ion

4th

5th

6th ...

How many cycles are required for a program?

13CPS4150

• Multiplication takes more time than addition

• Floating point operations take longer than integer ones

• Accessing memory takes more time than accessing registers

• Important point: changing the cycle time often changes the number of cycles required for various instructions (more later)

time

Different numbers of cycles for different instructions

14CPS4150

• Our favorite program runs in 10 seconds on computer A, which has a 4 GHz. clock. We are trying to help a computer designer build a new machine B, that will run this program in 6 seconds. The designer can use new (or perhaps more expensive) technology to substantially increase the clock rate, but has informed us that this increase will affect the rest of the CPU design, causing machine B to require 1.2 times as many clock cycles as machine A for the same program. What clock rate should we tell the designer to target?"

• Don't Panic, can easily work this out from basic principles

Example

15CPS4150

4.2 Clock Cycles

CUP timeA = (CPU clock cyclesA ) / (clock rate A)

10 seconds = (CPU clock cyclesA ) /(4 * 109 cycles/seconds)

CPU clock cyclesA = 10 * 4 * 109 cycles = 40 * 109 cycles

CUP clock cyclesA = 40 * 109cycles

CUP timeB = 6 seconds = (1.2 * CPU clock cyclesA ) / (clock rateB)

clock rateB = (1.2* 40 * 109cycles)/ 6 seconds =8 GHz

16CPS4150

• A given program will require

– some number of instructions (machine instructions)

– some number of cycles

– some number of seconds

• We have a vocabulary that relates these quantities:

– cycle time (seconds per cycle)

– clock rate (cycles per second)

– CPI (cycles per instruction) (average number of clock cycle each

instruction takes to execution)

a floating point intensive application might have a higher CPI

– MIPS (millions of instructions per second)

this would be higher for a program using simple instructions

Now that we understand cycles

17CPS4150

Performance

• Performance is determined by execution time

• Do any of the other variables equal performance?

– # of cycles to execute program?

– # of instructions in program?

– # of cycles per second?

– average # of cycles per instruction?

– average # of instructions per second?

• Common pitfall: thinking one of the variables is indicative of performance when it really isn’t.

18CPS4150

CPI Example

• Suppose we have two implementations of the same instruction set architecture (ISA).

For some program,

Machine A has a clock cycle time of 250 ps and a CPI of 2.0 Machine B has a clock cycle time of 500 ps and a CPI of 1.2

What machine is faster for this program, and by how much?

See page 248

• If two machines have the same ISA which of our quantities (e.g., clock rate, CPI, execution time, # of instructions, MIPS) will always be identical?

rateClock

CPI Count n InstructioTime CPU

19CPS4150

Example:

A compiler designer is trying to decide between two code sequences for a particular machine. Based on the hardware implementation, there are three different classes of instructions: Class A, Class B, and Class C, and they require one, two, and three cycles (respectively).

The first code sequence has 5 instructions: 2 of A, 1 of B, and 2 of CThe second sequence has 6 instructions: 4 of A, 1 of B, and 1 of C.

Which sequence will be faster? How much?

(CUP clock cycle1 = (2X1) + (1X2) +(2X3) = 2+2+6 = 10 cycles)What is the CPI for each sequence?

(CPU clock cycle2 = (4X1) +(1X2) +(1X3) = 4+2+3 = 9 cycles)

# of Instructions Example

20CPS4150

• Workload: A set of programs run on a computer that is either the actual collection of applications run by a user or is constructed from real programs to approximate such as mix.

• Benchmark: Programs specially chosen to measure performance.• Small benchmarks

– nice for architects and designers– easy to standardize– can be abused (specific benchmark for special purpose and is not for

general case), for example engineering and developing machine– May be misleading– Different classes and applications of computers will require different

types of benchmarks.• SPEC (System Performance Evaluation Cooperative)

– companies have agreed on a set of real program and inputs– valuable indicator of performance (and compiler technology)– can still be abused

4.3 Benchmarks

21CPS4150

Benchmarks

• Reproducibility- see page 256, Figure 4-3.– Input data will affect the memory system

• Total execution Time– Directly proportional to execution time - our final measure of

performance.– The average of the execution times that is directly proportional to

total execution time is the arithmetic mean (AM):

where Timei is the execution time for the ith program of a total of n in the workload. A weight wi can be assigned to each program to indicate the frequency of the program in that workload. For example 20% of program1 and 80% of program2. By summing the products of weighting factors and execution times, we can obtain clear picture of the performance of the workload. weighted arithmetic mean.

n

iin 1

Time1

AM

22CPS4150

Check your self

– See page 258. Check your self

23CPS4150

Benchmark Games

• An embarrassed Intel Corp. acknowledged Friday that a bug in a software program known as a compiler had led the company to overstate the speed of its microprocessor chips on an industry benchmark by 10 percent. However, industry analysts said the coding error…was a sad commentary on a common industry practice of “cheating” on standardized performance tests…The error was pointed out to Intel two days ago by a competitor, Motorola …came in a test known as SPECint92…Intel acknowledged that it had “optimized” its compiler to improve its test scores. The company had also said that it did not like the practice but felt to compelled to make the optimizations because its competitors were doing the same thing…At the heart of Intel’s problem is the practice of “tuning” compiler programs to recognize certain computing problems in the test and then substituting special handwritten pieces of code…

Saturday, January 6, 1996 New York Times

24CPS4150

SPEC ‘89

• Compiler “enhancements” and performance

0

100

200

300

400

500

600

700

800

tomcatvfppppmatrix300eqntottlinasa7doducspiceespressogcc

BenchmarkCompiler

Enhanced compiler

SP

EC

pe

rfo

rman

ce r

atio

25CPS4150

SPEC CPU2000

26CPS4150

SPEC CPU2000

• The execution time measurement are first normalized by dividing the execution time on Sum Ultra 5_10 with 300 MHz processor by the execution time on the measured computer. SPEC ratio.

• The bigger the faster.

• For a given instruction set architecture, increase in CPU performance can come from three sources:

– Increase in clock rate

– Improvements in processor organization that lower the CPI.\

– Compiler enhancements that lower the instruction count or generate instructions with a lower average CPI. (e.g. By using simpler instruction sets).

27CPS4150

SPEC 2000

Does doubling the clock rate double the performance?

Can a machine with a slower clock rate have better performance?

Clock rate in MHz

500 1000 1500 30002000 2500 35000

200

400

600

800

1000

1200

1400

Pentium III CINT2000

Pentium 4 CINT2000

Pentium III CFP2000

Pentium 4 CFP2000

0.0

0.2

0.4

0.6

0.8

1.0

1.2

1.4

1.6

SPECINT2000 SPECFP2000 SPECINT2000 SPECFP2000 SPECINT2000 SPECFP2000

Always on/maximum clock Laptop mode/adaptiveclock

Minimum power/minimumclock

Benchmark and power mode

Pentium M @ 1.6/0.6 GHz

Pentium 4-M @ 2.4/1.2 GHz

Pentium III-M @ 1.2/0.8 GHz