Automated Detection of Performance Regressions Using Statistical Process Control Techniques
26
Automated Detection of Performance Regressions Using Statistical Process Control Techniques Thanh Nguyen , Bram Adams, ZhenMing Jiang, Ahmed E. Hassan Queen’s University, Kingston, Canada Mohamed Nasser, Parminder Flora Research in Motion, Waterloo, Canada 1
Transcript of Automated Detection of Performance Regressions Using Statistical Process Control Techniques
1
Automated Detection of Performance Regressions Using Statistical Process Control
Techniques
Thanh Nguyen, Bram Adams, ZhenMing Jiang, Ahmed E. HassanQueen’s University, Kingston, Canada
Mohamed Nasser, Parminder FloraResearch in Motion, Waterloo, Canada
2
Performance Regression
3
What is a performance regression?
Version 1 Version 1.1
Baseline Target
4
How to detect performance regression?
Applying load
Version 1.1
Version 1 CPU %, Memory usage
CPU %, Memory usage
Detect regression
Challenge in Performance Regression Testing
5
Layer 1 – Agent 1
Layer 1 – Agent 2
Layer 2 – Agent 1
Layer 2 – Agent 2
Layer 2 – Agent 3
Layer 2 – Agent 4
Layer 3 – Agent 1
Layer 4 – Agent 1
56 counters x 8 agents = 448 counters
56 counters x 2 agents = 112 counters
Layer 1 Layer 2
Lots of data
6
Data mining
7
Data mining -> Reduce and Relate
Reduce Relate
8
Proposed approach to use control charts to find performance regression
BaselinePerformance counters
TargetPerformance counters
Determine the LCL, CL, UCL
0 5 10 15 20 25710
720
730
740
750
760
770
780
Performance counter
9
Using control charts to verify load test results
BaselinePerformance counters
TargetPerformance counters
Determine the LCL, CL, UCL
0 5 10 15 20 25710
720
730
740
750
760
770
780
Performance counter
Violation ratio
Reduce
10
BaselinePerformance counters
TargetPerformance counters
TargetPerformance counters
0 5 10 15 20 25710720730740750760770780
Performance counter
BaselinePerformance counters
0 5 10 15 20 25680
700
720
740
760
780
Performance counter
Low violation ratio
High violation ratio
We can use violation ratio to detect regression Relate
11
Is there performance regression?
12
Obstacles #1: Inputs are unstable
1 2 3 4 5 60
5
10
15
20
25
30
35
40
45
Version 1.0Version 1.1
Time
CPU
%
Is there a performance regression?
13
It is very difficult to maintain stable input across test runs
Applying load
Version 1.1
Version 1 CPU %, Memory usage
CPU %, Memory usage
Detect regression
Randomization CacheWarm up
Background tasks
14
Solution #1: Scale the counter according to the input
• Step 1: Determine α and β
• Step 2:CPU% Request/s
15
Solution #1: Example of the effectiveness of scaling
16
Obstacles #2: Multiple inputs
10 20 30 40 50 60 70 80 90 1000
5
10
15
20
25
30
35
Density plot of two test runs
CPU Usage
Dens
ity %
IF … THEN…ELSE…
17
10 20 30 40 50 60 70 80 90 1000
5
10
15
20
25
30
35
Density plot of two test runs
CPU Usage
Dens
ity %
Solution #2: Isolating the counters
Local minima
18
Scale and filter
Applying load
Version 1.1
Version 1 CPU %, Memory usage
CPU %, Memory usage
Detect regression
Scale
Scale
Filter
Filter
19
Case study 1
20
Experiment set up
BaselinePerformance counters
TargetPerformance counters
TargetPerformance counters
Average violation ratio should be low
21
BaselinePerformance counters
TargetPerformance counters
TargetPerformance counters
Average violation ratio should be high
Experiment set up
22
Bad Query
Limit
Extra p
rint
Extra c
onnection
Missing k
ey index
Missing t
ext index
0.00%10.00%20.00%30.00%40.00%50.00%60.00%70.00%80.00%90.00%
100.00%
NormalProblem
Series3
Average violation ratio
23
Case study 2
24
Experiment set up
V.S.
Precision is highRecall should be high
25
0
10
20
30
40
50
60
70
80
90
100
PrecisionRecallF
Threshold
%
26
