Diversity Maximization Speedup for Fault Localization
-
Upload
liang-gong -
Category
Technology
-
view
570 -
download
3
Transcript of Diversity Maximization Speedup for Fault Localization
Diversity Maximization Speedup for Fault Localization
Liang Gong1, David Lo2, Lingxiao Jiang2, and Hongyu Zhang1
A u t h o r e d b y
1 Tsinghua University 2 Singapore Management University
• Diversity Maximization Speedup • Technical Motivation • Detailed Approach
• Motivation • Fault Localization • Test Case Prioritization
• Experiments • Settings & Results
Diversity Maximization Speedup for Fault Localization
Outline Introduction &
Framework
• Conclusion & Future work
• Software errors cost the US economy 59.5 billion dollars (0.6% of 2002's GDP) [1] • Testing and debugging activities are labor-intensive (30% to 90% of a Project) [2]
Diversity Maximization Speedup for Fault Localization
Debugging Problem
P r e s e n t e d b y
Liang Gong School of Software Tsinghua University
[1] National Institute of Standards and Technology (NIST). Software Errors Cost U.S. Economy $59.5 Billion Annually, June 28, 2002.
[2] B. Beizer. Software Testing Techniques. International Thomson Computer Press, Boston, 2nd edition, 1990.
Spectrum-based Fault Localization(abbr. SBFL) • Automatically recommend a list of suspicious program elements for inspection.
• Program Spectra consists of coverage information and execution labels.
Program Spectra
Coverage information of one element (si) in all executions
Profile of an execution trace
Correct or incorrect?
Diversity Maximization Speedup for Fault Localization
SBFL Introduction
Ochiai
Tarantula
Jaccard
Diversity Maximization Speedup for Fault Localization
Approaches Fault Localization
P r e s e n t e d b y
Liang Gong School of Software Tsinghua University
No. of failed traces covering S
No. of failed traces No. of traces covering S
For a given statement S The formula calculates the suspiciousness of S.
Intuition: If S is covered more in failed traces and less in passed traces, it is more likely to contain faults.
Correct program Test script
Correct Output (Oracle)
program (contains
faults)
Test Program Output
Label for the execution trace
(pass or fail) Execution
trace
Program Spectra
Fault localization
Fault list
Collect by tools
Compare
Diversity Maximization Speedup for Fault Localization
Process In Experiments
P r e s e n t e d b y
Liang Gong School of Software Tsinghua University
developer
Diversity Maximization Speedup for Fault Localization
Process In Practice
P r e s e n t e d b y
Liang Gong School of Software Tsinghua University
Test script program
(contains faults)
Test Program Output
Label for the execution trace
(pass or fail) Execution
trace
Program Spectra
Fault localization
Fault list
Collect by tools
Manually Label
developer
Diversity Maximization Speedup for Fault Localization
Process In Practice
P r e s e n t e d b y
Liang Gong School of Software Tsinghua University
Test script program
(contains faults)
Test Program Output
Label for the execution trace
(pass or fail) Execution
trace
Program Spectra
Fault localization
Fault list
Collect by tools
Manually Label
Research Goal: • Minimize No. of executions to label
• Preserve fault localization effectiveness
Diversity Maximization Speedup for Fault Localization
Test Case Prioritization
Introduction
P r e s e n t e d b y
Liang Gong School of Software Tsinghua University
In [3], Rothermel et al. define the problem of test case prioritization as follows: Definition 2.1 (Test Case Prioritization). Given (1) T, a set of test cases, (2) PT, the set of permutations of T (3) f, a function mapping PT to real numbers, the problem is to find a permutation p ∈ PT such that: for all p’ ∈ PT: f (p) ≥ f(p’).
permutation
{ }1 2 3 4 5t ,t ,t ,t ,t
…
{ }arg max ( )p PT
f p∈
p1
pi
t1 t2 t3 t4 t5
t3 t5 t2 t1 t4
p2 t1 t2 t3 t5 t4
[3] G. Rothermel, R. H. Untch, C. Chu, and M. J. Harrold. Prioritizing test cases for regression testing. In IEEE TSE, pages 929-948, 2001.
Diversity Maximization Speedup for Fault Localization
Test Case Prioritization
Introduction
P r e s e n t e d b y
Liang Gong School of Software Tsinghua University
In [3], Rothermel et al. define the problem of test case prioritization as follows: Definition 2.1 (Test Case Prioritization). Given (1) T, a set of test cases, (2) PT, the set of permutations of T (3) f, a function mapping PT to real numbers, the problem is to find a permutation p ∈ PT such that: for all p’ ∈ PT: f (p) ≥ f(p’).
permutation
{ }1 2 3 4 5t ,t ,t ,t ,t
…
{ }arg max ( )p PT
f p∈
p1
pi
t1 t2 t3 t4 t5
t3 t5 t2 t1 t4
p2 t1 t2 t3 t5 t4
f(p) is larger, when permutation p allows the faulty program elements to be ranked higher meanwhile a shorter prefix are considered.
[3] G. Rothermel, R. H. Untch, C. Chu, and M. J. Harrold. Prioritizing test cases for regression testing. In IEEE TSE, pages 929-948, 2001.
Diversity Maximization Speedup for Fault Localization
DMS Introduction
P r e s e n t e d b y
Liang Gong School of Software Tsinghua University
• Greedy algorithm
• Use diversity of suspiciousness as the selecting criterion.
• Speedup suspiciousness rank changing process of promising program elements to further save labeling effort.
Diversity Maximization Speedup (abbr. DMS)
• t0 is the initial failed trace that reveals the fault. • t1 and t2 are candidates to be selected for labeling.
Diversity Maximization Speedup for Fault Localization
Diversity As Criterion
P r e s e n t e d b y
Liang Gong School of Software Tsinghua University
… s1 s2 s3 s4 s5 s6 … p/f
… … fail
… s1 s2 s3 s4 s5 s6 … p/f
… … ?
… s1 s2 s3 s4 s5 s6 … p/f
… … ?
… s1 s2 s3 s4 s5 s6 … Su
spic
ious
ness
S1, S2, S3
S4, S5, S6
Initial Diagnostic Distribution Initial Inspection List
t0
t1
t2
Diversity Maximization Speedup (abbr. DMS)
Diversity Maximization Speedup for Fault Localization
Test Case Prioritization
introduction
P r e s e n t e d b y
Liang Gong School of Software Tsinghua University
… s1 s2 s3 s4 s5 s6 … p/f
… … fail
… s1 s2 s3 s4 s5 s6 … p/f
… … ?
… s1 s2 s3 s4 s5 s6 … p/f
… … ?
… s1 s2 s3 s4 s5 s6 … p/f
… … fail
… s1 s2 s3 s4 s5 s6 …
Susp
icio
usne
ss
… s1 s2 s3 s4 s5 s6 …
Susp
icio
usne
ss
… s1 s2 s3 s4 s5 s6 …
Susp
icio
usne
ss
… s1 s2 s3 s4 s5 s6 …
Susp
icio
usne
ss ? ← pass
? ← fail
S1, S2, S3
S4, S5, S6
S1, S2, S3, S4, S5, S6
S2, S3
S1, S4, S6
S5
S1, S2, S3
S4, S5, S6
t0
t1
t0
t2
• t1 is preferred by approaches aiming to detect faults
• t2 is selected according to Diversity Maximization Criterion
More Diverse, Better Result
Nothing Changed
? ← pass
? ← fail
No Suggestion
Diversity Maximization Speedup for Fault Localization
Test Case Prioritization
introduction
P r e s e n t e d b y
Liang Gong School of Software Tsinghua University
Diversity Maximization Speedup (abbr. DMS)
Diversity Maximization Speedup for Fault Localization
Evolution Trend Opportunities
P r e s e n t e d b y
Liang Gong School of Software Tsinghua University
Looking for test cases that could offer more changing opportunities to "promising" elements like s7 (with clear trend) instead of s9
Evolution Trend of Statements’ Suspiciousness.
t2 t8 t6 t4 t7 t9 …
Two questions prompt:
How can we know which statements are “promising”?
With “promising” statements, how can we speed up their suspiciousness changing process?
Diversity Maximization Speedup for Fault Localization
Speed up How to ?
P r e s e n t e d b y
Liang Gong School of Software Tsinghua University
Diversity Maximization Speedup for Fault Localization
Promising How to identify ?
P r e s e n t e d b y
Liang Gong School of Software Tsinghua University
Representative Time Series • When the rank of the program element decreases, its time series increases by 1.
• When the rank of the program element increases, its time series decreases by 1.
• If the element's rank stays the same, its time series stays the same.
Evolution trend and time series(yi) of S8
Diversity Maximization Speedup for Fault Localization
Promising How to evaluate ?
P r e s e n t e d b y
Liang Gong School of Software Tsinghua University
Changing Stability Changing Rate
Evolution trend of S8
Example trends and their potentials
• Linear Regression Analysis:
• Change-potential Score:
Two questions prompt:
How can we know which statements are “promising”?
With “promising” statements, how can we speed up their suspiciousness changing process?
Diversity Maximization Speedup for Fault Localization
Speed up How to ?
P r e s e n t e d b y
Liang Gong School of Software Tsinghua University
Newly Added Execution Trace
Diversity Maximization Speedup for Fault Localization
Speed up By Competing
P r e s e n t e d b y
Liang Gong School of Software Tsinghua University
• Speed up the suspiciousness ranking changing process by competing in Suspicious Group.
No matter what the label is, ties are broken anyway. Speedup happens anyway. It could keep “promising” or become less “promising”
Trend
Diversity Maximization Speedup for Fault Localization
Speed up Our Method
P r e s e n t e d b y
Liang Gong School of Software Tsinghua University
Change-potential Score of program element d
• Change-potential Score of Suspicious Group:
• Sums of Squares of Change-potential Score of all Groups(G)
• To choose the next trace t to label, we use the following formula:
The Sum of Squares of change-potential of all groups
The Sum of Squares of change-potential of all groups
when t is added
Intuition: When t breaks ties in more promising or larger Suspicious Groups, it is more likely to be selected.
Diversity Maximization Speedup for Fault Localization
Speed up Our Method
P r e s e n t e d b y
Liang Gong School of Software Tsinghua University
Change-potential Score of program element d
• Change-potential Score of Suspicious Group:
• Sums of Squares of Change-potential Score of all Groups(G)
• To choose the next trace t to label, we use the following formula:
• Coverage Based Prioritization • STMT-TOTAL, STMT-ADDTL, and ART.
• Fault-Exposing Potential • FEP-TOTAL and FEP-ADDTL.
• Diagnostic Prioritization • SEQUIA and RAPTOR.
Diversity Maximization Speedup for Fault Localization
Test Case Prioritization
Existing Methods
P r e s e n t e d b y
Liang Gong School of Software Tsinghua University
Diversity Maximization Speedup for Fault Localization
Experiments Dataset &
Evaluation Metric
P r e s e n t e d b y
Liang Gong School of Software Tsinghua University
1 Siemens Suite 2 UNIX Programs
1
2
• Evaluation Metric for Fault Localization:
• Benchmarks for Fault Localization
Diversity Maximization Speedup for Fault Localization
Experiment Settings & Design
P r e s e n t e d b y
Liang Gong School of Software Tsinghua University
• Effectiveness on reducing the number of test cases(i.e., labeling effort) needed for a target cost
• Effectiveness on reducing cost for a given number of labeled test cases
Experiments comparing with existing methods:
• Defining target cost cx :
Average fault localization cost when labeling all test cases
Labeling Effort Needed When Setting c101 as Target Cost
Diversity Maximization Speedup for Fault Localization
Experiment Settings & Design
P r e s e n t e d b y
Liang Gong School of Software Tsinghua University
• Effectiveness on reducing the number of test cases(i.e., labeling effort) needed for a target cost
• Effectiveness on reducing cost for a given number of labeled test cases
Experiments comparing with existing methods:
Average Cost of DMS when Selecting Different Number of Test Cases
30
Diversity Maximization Speedup for Fault Localization
Test Case Prioritization
introduction
P r e s e n t e d b y
Liang Gong School of Software Tsinghua University
• Effectiveness on Reducing Cost for a Given Number of Labeled Test Cases
-35%
-15%
5%
25%
45%
65%
85%
0 20 40 60 80 100 120 140 160 180 200
Impr
ovem
ent
No. of Versions
DMS vs STMT-TOTAL
-40%
-20%
0%
20%
40%
60%
80%
0 20 40 60 80 100 120 140 160
Impr
ovem
ent
No. of Versions
-10%0%
10%20%30%40%50%60%70%
0 10 20 30 40 50 60 70 80 90 100
110
120
130
140
150
160
170
180
Impr
ovem
ent
No. of Versions
DMS vs SEQUOIA DMS vs STMT-ADDTL
DMS vs FEP-ADDTL DMS vs ART-MIN DMS vs RAPTOR
Pair-wised T-test shows the improvements are statistically significant at 95% interval.
Diversity Maximization Speedup for Fault Localization
Conclusion & Future Work
P r e s e n t e d b y
Liang Gong School of Software Tsinghua University
• Future Work • Evaluate on more subject programs.
• We will also explore the possibility of adopting more sophisticated trend analysis methods.
• Conclusions • We propose a new technique aiming to minimize the
amount of effort in manual oracle construction, while still permitting effective fault localization.
Given a target fault localization accuracy, our approach can significantly reduce the number of test cases needed to achieve it.
Given a maximum number of test cases that a programmer can manually label, DMS can improve the accuracy of fault localization and thus helps reduce the debugging cost.
Diversity Maximization Speedup for Fault Localization
Conclusion & Future Work
P r e s e n t e d b y
Liang Gong School of Software Tsinghua University
• Future Work • Evaluate on more subject programs.
• We will also explore the possibility of adopting more sophisticated trend analysis methods.
• Conclusions • We propose a new technique aiming to minimize the
amount of effort in manual oracle construction, while still permitting effective fault localization.
Given a target fault localization accuracy, our approach can significantly reduce the number of test cases needed to achieve it.
Given a maximum number of test cases that a programmer can manually label, DMS can improve the accuracy of fault localization and thus helps reduce the debugging cost.
Thank you!
• Any questions?
. . .