Computer Systems Performance Analysisjain/iucee/ftp/k_01int.pdf · Computer Systems Performance...
-
Upload
duongxuyen -
Category
Documents
-
view
264 -
download
0
Transcript of Computer Systems Performance Analysisjain/iucee/ftp/k_01int.pdf · Computer Systems Performance...
![Page 1: Computer Systems Performance Analysisjain/iucee/ftp/k_01int.pdf · Computer Systems Performance Analysis Raj Jain ... ©2010 Raj Jain The Art of Performance Evaluation ... ©2010](https://reader031.fdocuments.net/reader031/viewer/2022012318/5b68ab4e7f8b9a20388cdaa8/html5/thumbnails/1.jpg)
1-1©2010 Raj Jain www.rajjain.com
Computer Systems Computer Systems Performance AnalysisPerformance Analysis
Raj JainWashington University in Saint Louis
[email protected] Collaboration on Engineering Education
Leadership Institute 2010
![Page 2: Computer Systems Performance Analysisjain/iucee/ftp/k_01int.pdf · Computer Systems Performance Analysis Raj Jain ... ©2010 Raj Jain The Art of Performance Evaluation ... ©2010](https://reader031.fdocuments.net/reader031/viewer/2022012318/5b68ab4e7f8b9a20388cdaa8/html5/thumbnails/2.jpg)
1-2©2010 Raj Jain www.rajjain.com
OverviewOverview
Goal of this CourseContents of the courseTentative ScheduleProjects
![Page 3: Computer Systems Performance Analysisjain/iucee/ftp/k_01int.pdf · Computer Systems Performance Analysis Raj Jain ... ©2010 Raj Jain The Art of Performance Evaluation ... ©2010](https://reader031.fdocuments.net/reader031/viewer/2022012318/5b68ab4e7f8b9a20388cdaa8/html5/thumbnails/3.jpg)
1-3©2010 Raj Jain www.rajjain.com
Goal of This CourseGoal of This CourseComprehensive course on performance analysisIncludes measurement, statistical modeling, experimental design, simulation, and queuing theoryHow to avoid common mistakes in performance analysisCan be taught as:
Senior Undergraduate Courses on Performance measurementStatistical Performance AnalysisDesign of ExperimentsSimulationQueueing Theory
Graduate course: ⇒ Add a Project/Survey paper (Research techniques)
![Page 4: Computer Systems Performance Analysisjain/iucee/ftp/k_01int.pdf · Computer Systems Performance Analysis Raj Jain ... ©2010 Raj Jain The Art of Performance Evaluation ... ©2010](https://reader031.fdocuments.net/reader031/viewer/2022012318/5b68ab4e7f8b9a20388cdaa8/html5/thumbnails/4.jpg)
1-4©2010 Raj Jain www.rajjain.com
Text BookText Book
R. Jain, “Art of Computer Systems Performance Analysis,” Wiley, 1991, ISBN:0471503363(Winner of the “1992 Best Computer Systems Book”Award from Computer Press Association”)
![Page 5: Computer Systems Performance Analysisjain/iucee/ftp/k_01int.pdf · Computer Systems Performance Analysis Raj Jain ... ©2010 Raj Jain The Art of Performance Evaluation ... ©2010](https://reader031.fdocuments.net/reader031/viewer/2022012318/5b68ab4e7f8b9a20388cdaa8/html5/thumbnails/5.jpg)
1-5©2010 Raj Jain www.rajjain.com
Objectives: What Students Will LearnObjectives: What Students Will LearnSpecifying performance requirementsEvaluating design alternativesComparing two or more systemsDetermining the optimal value of a parameter (system tuning)Finding the performance bottleneck (bottleneck identification)Characterizing the load on the system (workload characterization)Determining the number and sizes of components (capacity planning)Predicting the performance at future loads (forecasting).
![Page 6: Computer Systems Performance Analysisjain/iucee/ftp/k_01int.pdf · Computer Systems Performance Analysis Raj Jain ... ©2010 Raj Jain The Art of Performance Evaluation ... ©2010](https://reader031.fdocuments.net/reader031/viewer/2022012318/5b68ab4e7f8b9a20388cdaa8/html5/thumbnails/6.jpg)
1-6©2010 Raj Jain www.rajjain.com
Basic TermsBasic Terms
System: Any collection of hardware, software, and firmwareMetrics: Criteria used to evaluate the performance of the system. components.Workloads: The requests made by the users of the system.
![Page 7: Computer Systems Performance Analysisjain/iucee/ftp/k_01int.pdf · Computer Systems Performance Analysis Raj Jain ... ©2010 Raj Jain The Art of Performance Evaluation ... ©2010](https://reader031.fdocuments.net/reader031/viewer/2022012318/5b68ab4e7f8b9a20388cdaa8/html5/thumbnails/7.jpg)
1-7©2010 Raj Jain www.rajjain.com
Main Parts of the CourseMain Parts of the Course
Part I: An Overview of Performance EvaluationPart II: Measurement Techniques and ToolsPart III: Probability Theory and StatisticsPart IV: Experimental Design and AnalysisPart V: SimulationPart VI: Queueing Theory
![Page 8: Computer Systems Performance Analysisjain/iucee/ftp/k_01int.pdf · Computer Systems Performance Analysis Raj Jain ... ©2010 Raj Jain The Art of Performance Evaluation ... ©2010](https://reader031.fdocuments.net/reader031/viewer/2022012318/5b68ab4e7f8b9a20388cdaa8/html5/thumbnails/8.jpg)
1-8©2010 Raj Jain www.rajjain.com
Part I: An Overview of Performance Part I: An Overview of Performance EvaluationEvaluation
IntroductionCommon Mistakes and How To Avoid ThemSelection of Techniques and Metrics
![Page 9: Computer Systems Performance Analysisjain/iucee/ftp/k_01int.pdf · Computer Systems Performance Analysis Raj Jain ... ©2010 Raj Jain The Art of Performance Evaluation ... ©2010](https://reader031.fdocuments.net/reader031/viewer/2022012318/5b68ab4e7f8b9a20388cdaa8/html5/thumbnails/9.jpg)
1-9©2010 Raj Jain www.rajjain.com
Example IExample I
What performance metrics should be used to compare the performance of the following systems:
Two disk drives?Two transaction-processing systems?Two packet-retransmission algorithms?
![Page 10: Computer Systems Performance Analysisjain/iucee/ftp/k_01int.pdf · Computer Systems Performance Analysis Raj Jain ... ©2010 Raj Jain The Art of Performance Evaluation ... ©2010](https://reader031.fdocuments.net/reader031/viewer/2022012318/5b68ab4e7f8b9a20388cdaa8/html5/thumbnails/10.jpg)
1-10©2010 Raj Jain www.rajjain.com
Part II: Measurement Techniques and ToolsPart II: Measurement Techniques and ToolsTypes of WorkloadsPopular BenchmarksThe Art of Workload SelectionWorkload Characterization TechniquesMonitorsAccounting LogsMonitoring Distributed SystemsLoad DriversCapacity PlanningThe Art of Data PresentationRatio Games
![Page 11: Computer Systems Performance Analysisjain/iucee/ftp/k_01int.pdf · Computer Systems Performance Analysis Raj Jain ... ©2010 Raj Jain The Art of Performance Evaluation ... ©2010](https://reader031.fdocuments.net/reader031/viewer/2022012318/5b68ab4e7f8b9a20388cdaa8/html5/thumbnails/11.jpg)
1-11©2010 Raj Jain www.rajjain.com
Example IIExample II
Which type of monitor (software or hardware) would be more suitable for measuring each of the following quantities:
Number of Instructions executed by a processor?Degree of multiprogramming on a timesharing system?Response time of packets on a network?
![Page 12: Computer Systems Performance Analysisjain/iucee/ftp/k_01int.pdf · Computer Systems Performance Analysis Raj Jain ... ©2010 Raj Jain The Art of Performance Evaluation ... ©2010](https://reader031.fdocuments.net/reader031/viewer/2022012318/5b68ab4e7f8b9a20388cdaa8/html5/thumbnails/12.jpg)
1-12©2010 Raj Jain www.rajjain.com
Part III: Probability Theory and StatisticsPart III: Probability Theory and StatisticsProbability and Statistics ConceptsFour Important DistributionsSummarizing Measured Data By a Single NumberSummarizing The Variability Of Measured DataGraphical Methods to Determine Distributions of Measured DataSample StatisticsConfidence IntervalComparing Two AlternativesMeasures of RelationshipSimple Linear Regression ModelsMultiple Linear Regression ModelsOther Regression Models
![Page 13: Computer Systems Performance Analysisjain/iucee/ftp/k_01int.pdf · Computer Systems Performance Analysis Raj Jain ... ©2010 Raj Jain The Art of Performance Evaluation ... ©2010](https://reader031.fdocuments.net/reader031/viewer/2022012318/5b68ab4e7f8b9a20388cdaa8/html5/thumbnails/13.jpg)
1-13©2010 Raj Jain www.rajjain.com
Example IIIExample III
The number of packets lost on two links was measured for four file sizes as shown below:
Which link is better?
![Page 14: Computer Systems Performance Analysisjain/iucee/ftp/k_01int.pdf · Computer Systems Performance Analysis Raj Jain ... ©2010 Raj Jain The Art of Performance Evaluation ... ©2010](https://reader031.fdocuments.net/reader031/viewer/2022012318/5b68ab4e7f8b9a20388cdaa8/html5/thumbnails/14.jpg)
1-14©2010 Raj Jain www.rajjain.com
Part IV: Experimental Design and AnalysisPart IV: Experimental Design and Analysis
Introduction to Experimental Design2k Factorial Designs2kr Factorial Designs with Replications2k-p Fractional Factorial DesignsOne Factor ExperimentsTwo Factors Full Factorial Design without ReplicationsTwo Factors Full Factorial Design with ReplicationsGeneral Full Factorial Designs With k Factors
![Page 15: Computer Systems Performance Analysisjain/iucee/ftp/k_01int.pdf · Computer Systems Performance Analysis Raj Jain ... ©2010 Raj Jain The Art of Performance Evaluation ... ©2010](https://reader031.fdocuments.net/reader031/viewer/2022012318/5b68ab4e7f8b9a20388cdaa8/html5/thumbnails/15.jpg)
1-15©2010 Raj Jain www.rajjain.com
Example IVExample IV
The performance of a system depends on the following three factors:
Garbage collection technique used: G1, G2, or none.Type of workload: editing, computing, or AI.Type of CPU: C1, C2, or C3.
How many experiments are needed? How does one estimate the performance impact of each factor?
![Page 16: Computer Systems Performance Analysisjain/iucee/ftp/k_01int.pdf · Computer Systems Performance Analysis Raj Jain ... ©2010 Raj Jain The Art of Performance Evaluation ... ©2010](https://reader031.fdocuments.net/reader031/viewer/2022012318/5b68ab4e7f8b9a20388cdaa8/html5/thumbnails/16.jpg)
1-16©2010 Raj Jain www.rajjain.com
Part V: SimulationPart V: Simulation
Introduction to SimulationTypes of SimulationsModel Verification and ValidationAnalysis of Simulation ResultsRandom-Number GenerationTesting Random-Number GeneratorsRandom-Variate GenerationCommonly Used Distributions
![Page 17: Computer Systems Performance Analysisjain/iucee/ftp/k_01int.pdf · Computer Systems Performance Analysis Raj Jain ... ©2010 Raj Jain The Art of Performance Evaluation ... ©2010](https://reader031.fdocuments.net/reader031/viewer/2022012318/5b68ab4e7f8b9a20388cdaa8/html5/thumbnails/17.jpg)
1-17©2010 Raj Jain www.rajjain.com
Example VExample V
In order to compare the performance of two cache replacement algorithms:
What type of simulation model should be used?How long should the simulation be run?What can be done to get the same accuracy with a shorter run?How can one decide if the random-number generator in the simulation is a good generator?
![Page 18: Computer Systems Performance Analysisjain/iucee/ftp/k_01int.pdf · Computer Systems Performance Analysis Raj Jain ... ©2010 Raj Jain The Art of Performance Evaluation ... ©2010](https://reader031.fdocuments.net/reader031/viewer/2022012318/5b68ab4e7f8b9a20388cdaa8/html5/thumbnails/18.jpg)
1-18©2010 Raj Jain www.rajjain.com
Part VI: Queueing TheoryPart VI: Queueing Theory
Introduction to Queueing TheoryAnalysis of A Single QueueQueueing NetworksOperational LawsMean Value Analysis and Related TechniquesConvolution AlgorithmAdvanced Techniques
![Page 19: Computer Systems Performance Analysisjain/iucee/ftp/k_01int.pdf · Computer Systems Performance Analysis Raj Jain ... ©2010 Raj Jain The Art of Performance Evaluation ... ©2010](https://reader031.fdocuments.net/reader031/viewer/2022012318/5b68ab4e7f8b9a20388cdaa8/html5/thumbnails/19.jpg)
1-19©2010 Raj Jain www.rajjain.com
Example VIExample VIThe average response time of a database system is three seconds. During a one-minute observation interval, the idle time on the system was ten seconds.
Using a queueing model for the system, determine the following:System utilizationAverage service time per queryNumber of queries completed during the observation intervalAverage number of jobs in the systemProbability of number of jobs in the system being greater than 1090-percentile response time90-percentile waiting time
![Page 20: Computer Systems Performance Analysisjain/iucee/ftp/k_01int.pdf · Computer Systems Performance Analysis Raj Jain ... ©2010 Raj Jain The Art of Performance Evaluation ... ©2010](https://reader031.fdocuments.net/reader031/viewer/2022012318/5b68ab4e7f8b9a20388cdaa8/html5/thumbnails/20.jpg)
1-20©2010 Raj Jain www.rajjain.com
The Art of Performance EvaluationThe Art of Performance Evaluation
Given the same data, two analysts may interpret them differently.
Example:The throughputs of two systems A and B in transactions per second is as follows:
![Page 21: Computer Systems Performance Analysisjain/iucee/ftp/k_01int.pdf · Computer Systems Performance Analysis Raj Jain ... ©2010 Raj Jain The Art of Performance Evaluation ... ©2010](https://reader031.fdocuments.net/reader031/viewer/2022012318/5b68ab4e7f8b9a20388cdaa8/html5/thumbnails/21.jpg)
1-21©2010 Raj Jain www.rajjain.com
Possible SolutionsPossible Solutions
Compare the average:
Conclusion: The two systems are equally good.
Compare the ratio with system B as the base
Conclusion: System A is better than B.
![Page 22: Computer Systems Performance Analysisjain/iucee/ftp/k_01int.pdf · Computer Systems Performance Analysis Raj Jain ... ©2010 Raj Jain The Art of Performance Evaluation ... ©2010](https://reader031.fdocuments.net/reader031/viewer/2022012318/5b68ab4e7f8b9a20388cdaa8/html5/thumbnails/22.jpg)
1-22©2010 Raj Jain www.rajjain.com
Solutions (Cont)Solutions (Cont)
Compare the ratio with system A as the base
Conclusion: System B is better than A.Similar games in: Selection of workload, Measuring the systems, Presenting the results.Common mistakes will also be discussed.
![Page 23: Computer Systems Performance Analysisjain/iucee/ftp/k_01int.pdf · Computer Systems Performance Analysis Raj Jain ... ©2010 Raj Jain The Art of Performance Evaluation ... ©2010](https://reader031.fdocuments.net/reader031/viewer/2022012318/5b68ab4e7f8b9a20388cdaa8/html5/thumbnails/23.jpg)
1-23©2010 Raj Jain www.rajjain.com
PrerequisitePrerequisite
Statistics: Mean, varianceNormal distributionDensity function, Distribution function Coefficient of variationCorrelation coefficientMedian, mode, Quantile
C Programming, UNIX
![Page 24: Computer Systems Performance Analysisjain/iucee/ftp/k_01int.pdf · Computer Systems Performance Analysis Raj Jain ... ©2010 Raj Jain The Art of Performance Evaluation ... ©2010](https://reader031.fdocuments.net/reader031/viewer/2022012318/5b68ab4e7f8b9a20388cdaa8/html5/thumbnails/24.jpg)
1-24©2010 Raj Jain www.rajjain.com
Schedule for a 14Schedule for a 14--Week SemesterWeek SemesterClass Topic Chapters
1 Introduction 2 Common Mistakes 2 3 Introduction to Simulation 24 4 Single Queues 30 5 Selection of Techniques and Metrics 2,3 6 Workload Selection and Characterization 5, 6 7 Data Presentation, Ratio Games 10, 11 8 Summarizing Measured Data 12 9 Exam 1
10 Comparing Systems Using Random Data 13 11 Simple Linear Regression Models 14 12 Other Regression Models 15
![Page 25: Computer Systems Performance Analysisjain/iucee/ftp/k_01int.pdf · Computer Systems Performance Analysis Raj Jain ... ©2010 Raj Jain The Art of Performance Evaluation ... ©2010](https://reader031.fdocuments.net/reader031/viewer/2022012318/5b68ab4e7f8b9a20388cdaa8/html5/thumbnails/25.jpg)
1-25©2010 Raj Jain www.rajjain.com
Tentative Schedule (Cont)Tentative Schedule (Cont)
Date Topic Chapters13 Analysis of Simulation Results 25 14 Random Number Generation 26 15 Test Random Numbers 27 16 Experimental Designs 16, 17 17 Factorial Designs with Replication 18 18 Fractional Factorial Designs 19 19 Exam 2 20 One Factor Experiments 20 21 Two Factor Full Factorial Designs 21, 22 22 General Full Factorial Designs 23 23 Analysis of Single Queue 31
![Page 26: Computer Systems Performance Analysisjain/iucee/ftp/k_01int.pdf · Computer Systems Performance Analysis Raj Jain ... ©2010 Raj Jain The Art of Performance Evaluation ... ©2010](https://reader031.fdocuments.net/reader031/viewer/2022012318/5b68ab4e7f8b9a20388cdaa8/html5/thumbnails/26.jpg)
1-26©2010 Raj Jain www.rajjain.com
Tentative Schedule (Cont)Tentative Schedule (Cont)Date Topic Chapters
24 Queueing Networks 32 25 Operational Laws 33 26 Mean-Value Analysis 34 27 Convolution Algorithm 35 28 Final Exam
![Page 27: Computer Systems Performance Analysisjain/iucee/ftp/k_01int.pdf · Computer Systems Performance Analysis Raj Jain ... ©2010 Raj Jain The Art of Performance Evaluation ... ©2010](https://reader031.fdocuments.net/reader031/viewer/2022012318/5b68ab4e7f8b9a20388cdaa8/html5/thumbnails/27.jpg)
1-27©2010 Raj Jain www.rajjain.com
Graduate Student ProjectsGraduate Student ProjectsA survey paper on a performance topic
Workloads/Metrics/Analysis: Databases, Networks, Computer Systems, Web Servers, Graphics, Sensors, Distributed SystemsComparison of Measurement, Modeling, Simulation, Analysis Tools: NS2 Comprehensive Survey: Technical Papers, Industry Standards, Products
A real case study on performance of a system student is already working onRecent Developments: Last 5 to 10 years ⇒ Not in booksBetter ones may be submitted to magazines or journals
![Page 28: Computer Systems Performance Analysisjain/iucee/ftp/k_01int.pdf · Computer Systems Performance Analysis Raj Jain ... ©2010 Raj Jain The Art of Performance Evaluation ... ©2010](https://reader031.fdocuments.net/reader031/viewer/2022012318/5b68ab4e7f8b9a20388cdaa8/html5/thumbnails/28.jpg)
1-28©2010 Raj Jain www.rajjain.com
Example of Previous Case StudiesExample of Previous Case StudiesMeasure the performance of a remote procedure call mechanism used in a distributed system.Measure and compare the performance of window systems of two artificial intelligence systems.Simulate and compare the performance of two processor interconnection networks.Measure and analyze the performance of two microprocessors.Characterize the workload of a campus timesharing system.Compute the effects of various factors and their interactions onthe performance of two text-formatting programs.Measure and analyze the performance of a distributed information system.
![Page 29: Computer Systems Performance Analysisjain/iucee/ftp/k_01int.pdf · Computer Systems Performance Analysis Raj Jain ... ©2010 Raj Jain The Art of Performance Evaluation ... ©2010](https://reader031.fdocuments.net/reader031/viewer/2022012318/5b68ab4e7f8b9a20388cdaa8/html5/thumbnails/29.jpg)
1-29©2010 Raj Jain www.rajjain.com
Case Studies (Cont)Case Studies (Cont)Simulate the communications controllers for an intelligent terminal system.Measure and analyze the performance of a computer-aided design tool.Measure and identify the factors that affect the performance of an experimental garbage collection algorithm.Measure and compare the performance of remote procedure calls and remote pipe calls.Analyze the effect of factors that impact the performance of two RISC processor architectures.Analyze the performance of a parallel compiler running on a multiprocessor system.
![Page 30: Computer Systems Performance Analysisjain/iucee/ftp/k_01int.pdf · Computer Systems Performance Analysis Raj Jain ... ©2010 Raj Jain The Art of Performance Evaluation ... ©2010](https://reader031.fdocuments.net/reader031/viewer/2022012318/5b68ab4e7f8b9a20388cdaa8/html5/thumbnails/30.jpg)
1-30©2010 Raj Jain www.rajjain.com
Projects (Cont)Projects (Cont)Develop a software monitor to observe the performance of a large multiprocessor system.Analyze the performance of a distributed game program running on a network of artificial intelligence systems.Compare the performance of several robot control algorithms.
Goal: Provide an insight (or information) not obvious before the project.
Real Problems: Thesis work, or job
Homeworks: Apply techniques learnt to your system.
![Page 31: Computer Systems Performance Analysisjain/iucee/ftp/k_01int.pdf · Computer Systems Performance Analysis Raj Jain ... ©2010 Raj Jain The Art of Performance Evaluation ... ©2010](https://reader031.fdocuments.net/reader031/viewer/2022012318/5b68ab4e7f8b9a20388cdaa8/html5/thumbnails/31.jpg)
1-31©2010 Raj Jain www.rajjain.com
Project ScheduleProject Schedule
Class 11 Topic SelectionClass 15 References Due Class 17 Outline Due Class 21 First Draft Due Class 23 Reviews ReturnedClass 25 Final Report Due
![Page 32: Computer Systems Performance Analysisjain/iucee/ftp/k_01int.pdf · Computer Systems Performance Analysis Raj Jain ... ©2010 Raj Jain The Art of Performance Evaluation ... ©2010](https://reader031.fdocuments.net/reader031/viewer/2022012318/5b68ab4e7f8b9a20388cdaa8/html5/thumbnails/32.jpg)
1-32©2010 Raj Jain www.rajjain.com
SummarySummary
Goal: To prepare students for correct analysis and modeling of any systemUse measurement, modeling, simulation techniquesDesign experiments for the maximum information with the least effort
![Page 33: Computer Systems Performance Analysisjain/iucee/ftp/k_01int.pdf · Computer Systems Performance Analysis Raj Jain ... ©2010 Raj Jain The Art of Performance Evaluation ... ©2010](https://reader031.fdocuments.net/reader031/viewer/2022012318/5b68ab4e7f8b9a20388cdaa8/html5/thumbnails/33.jpg)
1-33©2010 Raj Jain www.rajjain.com
Quiz 0: PrerequisitesQuiz 0: PrerequisitesTrue or False?T F
The sum of two normal variates is normal. The sum of two normal variates with means 4 and 3 has a mean of 12. The probability of a fair coin coming up head once and tail once in two
throws is 1. The density function f(x) approaches 1 as x approaches ∞. The probability of a fair coin coming up heads twice in a row is 1/4. The difference of two normal variates with means 4 and 3 has a mean of 4/3. The cumulative distribution function F(x) approaches 1 as x approaches ∞. High coefficient of variation implies a low variance and vice versa.
Marks = Correct Answers _____ - Incorrect Answers _____ = ______