Topics to be discussed
Introduction
Performance Factors
Methodology
Test Process
Tools
Conclusion
Basics of Performance TestingAbu Bakr Siddiq
Introduction
Why Performance Test?
What is Performance Test?
How Performance Test?
When to Performance Test?
Performance Vocabulary
Why, What, How & When Performance Test Why Performance test the application?
To improve the software speed, scalability, stability and confidence of the system under different loads for desired performance.
Ex: USA Shopping in Dec . Indian Railway Reservation.
What is Performance testing?Testing for performance factors against the acceptable or suggestible configuration is performance testing.
How Performance test the application?Using a Tool which follows Performance Methodology & Performance Process
When Performance test the application?Available Performance Requirements, Stable Build, Resource Availability, Defined Performance Plan and methodology, Set entry and Exit Criteria.
Performance Testing VocabularyThe different types of performance testing
Capacity Testing: Determining the server’s failure point is called Capacity Testing.
Component Testing: Testing the architectural component of the application like, servers, databases, networks, firewalls, and storage devices.
Endurance Testing: Testing for performance characteristics with work load models anticipated during production.
Load Testing: Subjecting the system to a statistically representatives(load) is called Load Testing.
Smoke Testing: A initial run to see the system performance under normal Load.
Spike Testing: Testing for performance characteristics with work load models during production which repeatedly increase beyond anticipated.
Stress Testing: Evaluating the application beyond peak load conditions.
Validation Testing: Testing against the expectations that have been presumed for the system
Volume Testing: Subjecting the system to variable amount of data and testing for performance is Volume Testing.
User Load
Saturation point
Load Testing
Stress Testing
Th
rou
gh
p
ut
Performance Factors
Throughput
Response Time
Latency
Tuning
Benchmarking
Capacity Planning
Throughput
Response Time
Latency
Tuning
Benchmarking
Capacity Planning
ClientWeb
ServerDB
ServerInternet
N1
A1
N4
A3 N3
N2 A2
Network Latency = N1 + N2 + N3 + N4Product Latency = A1 + A2 + A3
Actual Response Time = Network Latency + Product LatencyLatency = Actual Response Time + O1
Throughput, Response Time & Latency
O1
Throughput : The number of requests/business transactions processed by the system in a specified time duration.
Tuning : Procedure by which the system performance is enhanced by setting different values to the parameters (variables) of the product, OS and other components.
Ex: “Search”
Benchmarking: Comparing the throughput and response time of the system with that of competitive products .
Ex: Comparing Open office with that of MS- Office.
Capacity Planning: The exercise to find out what resources and configurations are needed.
Ex: suggesting the ideal software, hardware and other components for the system to the customer.
Tuning, Benchmarking & Capacity Planning
Methodology
Collecting Requirements
Writing Test Cases
Automating Test Cases
Executing Test Cases
Analyzing Results
Tuning
Benchmarking
Capacity Planning
•Collecting Requirements•Authoring Test Cases
•Automating Test Cases•Executing Test Cases
•Analyzing Results•Tuning
•Benchmarking•Capacity Planning
Collecting Performance Requirements Performance Requirements types: Generic Specific
Performance Requirements Characteristics: Testable Clear Specific
Sources for gathering Requirements: Performance compared to previous release of the same product Performance compared to the competitive product(s) Performance compared to absolute numbers derived from actual
need Performance numbers derived from architecture and design
Graceful Performance Degradation
Authoring Test CasesPerformance Test Cases should have the following details:
List of Operations or business transactions to be tested
Steps for executing those operations/transactions
List of Product, OS parameters that impact the Performance
Testing
and their values
Loading Pattern
Resource and their configuration (network, hardware, software
configurations
The expected results
The product version/competitive products to be compared with
the related information such as their corresponding fields
Automating Performance Test CasesPerformance Testing naturally lends itself to automation, reasons :
Performance testing is repetitive
Performance test cases cannot be effective without automation
Results of Performance testing need to be accurate, manually
calculating may introduce inaccuracy
Performance Testing takes into account too many permutations
and combinations hence if done manually will be difficult
Extensive Analysis of performance results and failures needs to
be taken into account, is very difficult to do things manually
Executing Performance Test CasesThe following aspects need to be considered while executing Performance Tests:
Start and End time of Execution
Log and trace/audit files of the product and OS. (for future debugging and repeatability)
Utilization of resources ( CPU, memory, disk, network utilization and so on) on periodic basis
Configuration of all Environmental factors (Hardware, software and other components)
The performance factors listed in the test case documentation at regular intervals
Analyzing Performance Test resultsPerformance Test results are concluded as follows:
Whether performance of the product is consistent when tests are
executed multiple times
What performance can be expected for what type of configuration
What parameters impact and how they can be derived for better
performance
What is the effect of scenarios involving a mix of performance
factors
What is the effect of product technologies such as caching
What Loads of performance numbers are acceptable
What is the optimum throughput and response time of the product
for set of factors
What performance requirements are met how the performance
looks compared to previous version or expectations
Performance TuningTwo ways to get optimum mileage
Tuning the Product parameters Tuning the Operating system parameters
Product parameters:
Repeat the Performance tests for different values of each parameters that impact
A Particular Parameter change values is changed, it needs changes in other
Repeat the tests for default values of all parameters( factory settings tests) Repeat the performance tests for low and high values of each
parameter and combinations
Operating system parameters:
Files system related parameters Disk Management parameters Memory Management parameters Processor Management parameters Network management parameters
Performance Benchmarking & Capacity Planning
The steps involved in Performance Benchmarking are the as follows:
Identify the transactions/ scenarios & the test configuration Comparing the performance of different product
Tuning the parameters of the products bring compared fairly to deliver the best performance
Publishing the results of performance benchmarking
Capacity Planning is identifying the right configuration, which is of 3 types:
Minimum required configuration Typical configuration Special configuration
Test Process
Resource Requirements
Test Lab Setup
Responsibilities
Setting up Traces & Audits
Entry & Exit Criteria
Resource Requirement
Test Lab Setup
Setting up Traces &
Audits
Responsibilities
Entry & Exit Criteria
Performance Test ProcessObtain Measurable,
testable requirements
Create a Performance
Test Plan
Design Test Cases
Automate Test Cases
Evaluate Entry Criteria
Perform and Analyze performance test
cases
Evaluate Exit Criteria
Resource Requirements:All the resources are specifically needed , hence shall be planned and obtained.Resources are to be exclusively dedicated to the current system without interchanging the roles and responsibilities often.
Test – Lab Setup:Test Lab with all required equipment is to be setup prior to execution. Test lab has to be configured cautiously as a single mistake can lead to running the tests again.
Responsibilities:Performance defects may cause changes to architecture, design and code. Team facing customer communicates requirements for performance. Multiple teams are involved in Performance testing the system. Hence a matrix which describes the responsibilities of the team is part of the test plan.
Setting up Product traces, Audits:Performance test results need to be associated with traces and audit trails to analyze the results. Audits and traces are to be planned in advance or else may start impacting the performance results.
Entry and Exit Criteria:Performance tests require a stable product due to its complexity and accuracy that is needed. Changes to the product mean tests are to be repeated. Hence Performance tests starts after the product meets a set criteria.
Tools for Performance Testing Commercial Tools
Load Runner – HP QA Partner - Compuware Silk Test - Segue MS Stress Tool – Microsoft
Open Source Tools Web Load JMeter Open STA
Challenges
Conclusion
Commercial
Load Runner
Silk Test
MS Stress
Thank you
Top Related