Towards Building a Universal Defect Prediction Model
22
Towards Building a Universal Defect Prediction Model Feng Zhang Audris Mockus Iman Keivanloo Ying Zou
-
Upload
feng-zhang -
Category
Software
-
view
254 -
download
1
description
To predict files with defects, a suitable prediction model must be built for a software project from either itself (withinproject) or other projects (cross-project). A universal defect prediction model that is built from the entire set of diverse projects would relieve the need for building models for an individual project. A universal model could also be interpreted as a basic relationship between software metrics and defects. However, the variations in the distribution of predictors pose a formidable obstacle to build a universal model. Such variations exist among projects with different context factors (e.g., size and programming language). To overcome this challenge, we propose context-aware rank transformations for predictors. We cluster projects based on the similarity of the distribution of 26 predictors, and derive the rank transformations using quantiles of predictors for a cluster. We then fit the universal model on the transformed data of 1,398 open source projects hosted on SourceForge and GoogleCode. Adding context factors to the universal model improves the predictive power. The universal model obtains prediction performance comparable to the within-project models and yields similar results when applied on five external projects (one Apache and four Eclipse projects). These results suggest that a universal defect prediction model may be an achievable goal.
Transcript of Towards Building a Universal Defect Prediction Model
Towards Building a Universal Defect Prediction Model
Feng Zhang
Audris Mockus
Iman Keivanloo
Ying Zou
2
ONE ring that rules the other rings of power.
3
A universal model that predicts defects for all the projects.
4
Most successful prediction models are within-project models
5
How about cross-project models?
6
Deriving a universal model with cross-project models?
7
Select the training set of projects like this?
8
Or select the training set of projects like this?
9
Is it still possible to build a universal model? If so, then how?
10
What context factors to consider ?
11
C++
S
C++
L
Java
S
Java
L
Steps towards building a universal model 1. Partition
C++ Java Small size
Large size
Programming languages System Size
12
C++
S
C++
L
Java
S
Java
L
Steps towards building a universal model 1. Partition
C++
S
C++
L
Java
2. Cluster
R1(x)
R1(x)
R3(x)
3. Obtain Ranking Functions
4. Rank
Using quantiles of metric values (- ∞, 10%] => level 1 (10%, 20%] => level 2
… [90%, +∞) => level 10
Java
S
Java
L
Java
13
C++
S
C++
L
Java
S
Java
L
Build a universal model 1. Partition
C++
S
C++
L
Java
2. Cluster
R1(x)
R1(x)
R3(x)
3. Obtain Ranking Functions
4. Rank
Build a universal defect prediction model using rank-transformed values.
14
Case study setup
937
461
0 200 400 600 800
Version Control System
0
200
400
600
800
1000
Using Not Using
Issue Tracking System
0
200
400
600
800
Programming languages
15
Research Questions
16
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
Precision Recall AUC
Rank Transformation
Log Transformation
0.48 0.48 0.57
0.58 0.62
0.61
RQ1. Is our rank transformation good ?
17
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
Precision Recall AUC
Universal Model
Within-project Model
0.45 0.48
0.58 0.63 0.64
0.62
RQ2. How good is the universal model ?
18
RQ3. Does the universal model work for external projects ?
Predict
19
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
Eclipse Equinox PDE Mylyn Lucene
Universal Model
Within-project Model 0.31
0.47
0.63 0.66
0.21
0.13
Precision
0.23 0.28
0.23 0.28
RQ3. Precision comparison
20
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
Eclipse Equinox PDE Mylyn Lucene
Universal Model
Within-project Model
0.57
0.79
0.54 0.61 0.61
0.34
Recall
0.47
0.72
0.42
0.60
RQ3. Recall comparison
21
0.6 0.62 0.64 0.66 0.68
0.7 0.72 0.74 0.76 0.78
0.8
Eclipse Equinox PDE Mylyn Lucene
Universal Model
Within-project Model
0.76 0.77 0.78
0.79
0.69 0.67
AUC
0.70 0.70 0.68
0.69
RQ3. AUC comparison
22
Summary
