Gerardo  Canfora  

Andrea  De  Lucia  

Massimiliano  Di  Penta  

Rocco  Oliveto  

Annibale Panichella

Sebas<ano  Panichella  

Multi-Objective Cross-Project Defect Prediction

Bugs are everywhere…

Software Testing

Practical Constraints

Sofwtare Quality

Money Time

Defect Prediction

Spent more resources on components most

likely to fail

Indicators of defects

Cached history information Kim  at  al.    ICSE  2007  

Change Metrics Moset  at  al.    ICSE  2008.  

A metrics suite for object oriented design Chidamber  at  al.  TSE      1994  

Defect Prediction Methodology

Predic<ng  Model  

 Project    

Test  Set  

Training  Set  Defect Prone

Class1 YES

Class2 YES

Class3 NO

… YES

ClassN …

Defect Prediction Methodology

Predic<ng  Model  

 Project    

Test  Set  

Training  Set  Defect Prone

Class1 YES

Class2 YES

Class3 NO

… YES

ClassN …

Within Project

Defect Prediction Methodology

Predic<ng  Model  

 Project    

Test  Set  

Training  Set  Defect Prone

Class1 YES

Class2 YES

Class3 NO

… YES

ClassN …

Within Project Issue: Size of the

Training Set

Defect Prediction Methodology

Predic<ng  Model  

 Project    

Test  Set  

Training  Set  Defect Prone

Class1 YES

Class2 YES

Class3 NO

… YES

ClassN …

Predic<ng  Model  

Test  Set  

Training  Set  Defect Prone

Class1 YES

Class2 YES

Class3 NO

… YES

ClassN …

Within Project Issue: Size of the

Training Set

 Past  Projects  

 New  Project  

 Project  B  

 Project  A  

Defect Prediction Methodology

Predic<ng  Model  

 Project    

Test  Set  

Training  Set  Defect Prone

Class1 YES

Class2 YES

Class3 NO

… YES

ClassN …

Predic<ng  Model  

Test  Set  

Training  Set  Defect Prone

Class1 YES

Class2 YES

Class3 NO

… YES

ClassN …

Within Project

Cross-Project

Issue: Size of the Training Set

 Project  B  

 Project  A  

Defect Prediction Methodology

Predic<ng  Model  

 Project    

Test  Set  

Training  Set  Defect Prone

Class1 YES

Class2 YES

Class3 NO

… YES

ClassN …

Predic<ng  Model  

Test  Set  

Training  Set  Defect Prone

Class1 YES

Class2 YES

Class3 NO

… YES

ClassN …

Within Project

Cross-Project

Issue: Size of the Training Set

Issue: The predicting

accuracy can be lower

Cost Effectiveness

1)  Cross-project does not necessarily works worse than within-project

2)  Better precision (accuracy)

does not mirror less inspection cost

3)  Traditional predicting model: logistic regression

Recaling the “imprecision” of Cross-project Defect Prediction, Rahman   at   al.  FSE  2012  

Cost Effectiveness: example

Class  A   Class  B   Class  C   Class  D  

Cost Effectiveness: example

Predicting model 1

Class  A   Class  B   Class  A   Class  C   Class  D  

100 LOC

10,000 LOC

100 LOC

100 LOC

100 LOC

Predicting model 2

Class  A   Class  B   Class  C   Class  D  

Cost Effectiveness: example

Predicting model 1

Class  A   Class  B   Class  A   Class  C   Class  D  

BUG   BUG  

100 LOC

10,000 LOC

100 LOC

100 LOC

100 LOC

Predicting model 2

Class  A   Class  B   Class  C   Class  D  

Cost Effectiveness: example

Predicting model 1

Class  A   Class  B   Class  A   Class  C   Class  D  

BUG   BUG  

100 LOC

10,000 LOC

100 LOC

100 LOC

100 LOC

Precision  =  50  %  

Cost  =10,100  LOC  

Predicting model 2

Class  A   Class  B   Class  C   Class  D  

Cost Effectiveness: an example

Predicting model 1

Class  A   Class  B   Class  A   Class  C   Class  D  

BUG   BUG  

100 LOC

10,000 LOC

100 LOC

100 LOC

100 LOC

Precision  =  50  %  

Cost  =10,100  LOC  

Predicting model 2

Precision  =  33  %  

Cost  =  300  LOC  

Class  A   Class  B   Class  C   Class  D  

Class  A   Class  B   Class  C   Class  D  

Cost Effectiveness: an example

Predicting model 1

Class  A   Class  B   Class  A   Class  C   Class  D  

BUG   BUG  

100 LOC

10,000 LOC

100 LOC

100 LOC

100 LOC

Predicting model 2

Precision does not mirror the inspection cost

All the existing predicting models work

on precision and not on cost

We need COST oriented models

Mul+-­‐objec+ve    Logis+c  Regression  

Building Predicting Model on Training Set

Training  Set  

P1 P2 …

Class1 m11 m12 …

Class2 m21 m22 …

Class3 m31 m32 …

Class4 … … …

… … … …

Logis<c  Regression  

Pred.

C1 1

C2 1

C3 0

C4 1

… 0

Building Predicting Model on Training Set

Training  Set  

Logis<c  Regression  

Pred.

C1 1

C2 1

C3 0

C4 1

… 0

Actual Val

C1 1

C2 0

C3 1

C4 1

… 0

P1 P2 …

Class1 m11 m12 …

Class2 m21 m22 …

Class3 m31 m32 …

Class4 … … …

… … … …

Building Predicting Model on Training Set

Training  Set  

Logis<c  Regression  

Pred.

C1 1

C2 1

C3 0

C4 1

… 0

Actual Val

C1 1

C2 0

C3 1

C4 1

… 0

Comparison

P1 P2 …

Class1 m11 m12 …

Class2 m21 m22 …

Class3 m31 m32 …

Class4 … … …

… … … …

Building Predicting Model on Training Set

Training  Set  

Logis<c  Regression  

Pred.

C1 1

C2 1

C3 0

C4 1

… 0

Actual Val

C1 1

C2 0

C3 1

C4 1

… 0

Comparison

P1 P2 …

Class1 m11 m12 …

Class2 m21 m22 …

Class3 m31 m32 …

Class4 … … …

… … … …

GOAL: minimazing the predicting error (PRECISION)

Building Predicting Model on Training Set

Training  Set  

Logis<c  Regression  

Pred.

C1 1

C2 1

C3 0

C4 1

… 0

Actual Val

C1 1

C2 0

C3 1

C4 1

… 0

Comparison

P1 P2 …

Class1 m11 m12 …

Class2 m21 m22 …

Class3 m31 m32 …

Class4 … … …

… … … …

GOAL: minimazing the predicting error (PRECISION)

Multi-objective Logistic Regression

Pred.

1

0

…

1

0

LOC

100

95

…

110

10

*   =  

Cost

100

0

…

110

0

Ispection Cost = 210 LOC

Multi-objective Logistic Regression

Pred.

1

0

…

1

0

LOC

100

95

…

110

10

*   =  

Cost

100

0

…

110

0

Ispection Cost = 210 LOC

Pred.

1

0

…

1

0

Actual Values

1

1

…

1

0

*   =  

#Bug

1

0

…

1

0

Effectiveness = 2 defects

Multi-objective Logistic Regression

⎪⎩

⎪⎨

⎧

⋅=

⋅=

∑∑

iii

ii

i

ActualPredessEffectiven

CostPredCostIspection min

max

Pred.

1

0

…

1

0

LOC

100

95

…

110

10

*   =  

Cost

100

0

…

110

0

Ispection Cost = 210 LOC

Pred.

1

0

…

1

0

Actual Values

1

1

…

1

0

*   =  

#Bug

1

0

…

1

0

Effectiveness = 2 defects

Multi-objective Logistic Regression

⎪⎩

⎪⎨

⎧

⋅=

⋅=

∑∑

iii

ii

i

ActualedessEffectiven

CostPredCostIspection

Pr

min

max

Pred.

1

0

…

1

0

LOC

100

95

…

110

10

*   =  

Cost

100

0

…

110

0

Ispection Cost = 210 LOC

Pred.

1

0

…

1

0

Actual Values

1

1

…

1

0

*   =  

#Bug

1

0

…

1

0

Effectiveness = 2 defects

a + b mi1 + c mi2 + …

Multi-objective Genetic Algorithm

⎪⎩

⎪⎨

⎧

⋅=

⋅=

∑∑

iii

ii

i

ActualedessEffectiven

CostPredCostIspection

Pr

min

max

. 1

eePred+

=a + b mi1 + c mi2 + …

Chromosome        (a, b,c , …)

Fitness Function

Multiple objectives are optimized using Pareto efficient approaches

Multi-objective Genetic Algorithm

Pareto Optimality: all solutions that are not dominated by any other solutions form the Pareto optimal set.

Multiple otpimal solutions (models) can be found Cost

Effe

ctiv

enes

s

The frontier allows to make a

well-informed decision that

b a l a n c e s t h e t r a d e - o f f s

between the two objectives

Empirical Evaluation

Research Questions

RQ1: How does the multi-objective (MO)prediction perform,

compared to single-objective (SO) prediction

Research Questions

RQ1: How does the multi-objective (MO)prediction perform,

compared to single-objective (SO) prediction

Cross-project MO vs. cross-project SO vs. within project SO

Research Questions

RQ2: How does the proposed approach perform, compared to the local prediction approach by Menzie et al. ?

RQ1: How does the multi-objective (MO)prediction perform,

compared to single-objective (SO) prediction

Cross-project MO vs. cross-project SO vs. within project SO

Research Questions

RQ2: How does the proposed approach perform, compared to the local prediction approach by Menzie et al. ?

RQ1: How does the multi-objective (MO)prediction perform,

compared to single-objective (SO) prediction

Cross-project MO vs. cross-project SO vs. within project SO

Cross-project MO vs. Local Prediction

Experiment outline •  10 java projects from PROMISE dataset ü   different  sizes  

ü   different  context  applica<on  

•  10 java projects from PROMISE dataset ü   different  sizes  ü   different  context  applica<on  

Experiment outline

•  Cross-projects defect prediction: ü  Training  model  on  nine  projects  and  test  on  the  remaining  one    (10  <mes)  

RQ1  

•  10 java projects from PROMISE dataset ü   different  sizes  ü   different  context  applica<on  

Experiment outline

•  Cross-projects defect prediction: ü  Training  model  on  nine  projects  and  test  on  the  remaining  one    (10  <mes)  

•  Within project defect prediction: ü   10  cross-­‐folder  valida<on  

RQ1  

RQ1  

•  10 java projects from PROMISE dataset ü   different  sizes  ü   different  context  applica<on  

Experiment outline

•  Cross-projects defect prediction: ü  Training  model  on  nine  projects  and  test  on  the  remaining  one    (10  <mes)  

•  Within project defect prediction: ü   10  cross-­‐folder  valida<on  

•  Local prediction: ü     K-­‐means  clustering  algorithm  ü   Silhoue]e  Coefficient  

RQ1  

RQ1  

RQ2  

Results

Results

Log4j jEdit

Cross-project MO vs. Cross-project SO

0  

50  

100  

150  

200  

250  

300  

KLOC  

Cross-­‐project  SO   Cross  project  MO  

Cross-project MO vs. Cross-project SO

0  

50  

100  

150  

200  

250  

300  

KLOC  

Cross-­‐project  SO   Cross  project  MO  

The proposed multi-objective model Outperform the single-objective one

Cross-project MO vs. Within-project SO

0  

50  

100  

150  

200  

250  

300  

350  

KLOC  

Within  project  SO   Cross  project  MO  

Cross-project MO vs. Within-project SO

0  10  20  30  40  50  60  70  80  90  

100  

Precision  

Within  project  SO   Cross  project  MO  

Cross-project MO vs. Within-project SO

0  10  20  30  40  50  60  70  80  90  

100  

Precision  

Within  project  SO   Cross  project  MO  

Cross-project prediction is worse than within-project

prediction in terms of PRECISION

Cross-project MO vs. Within-project SO

0  10  20  30  40  50  60  70  80  90  

100  

Precision  

Within  project  SO   Cross  project  MO  

Cross-project prediction is worse than within-project

prediction in terms of PRECISION

But it is better than within-project predictors in term

of COST-EFFECTIVENESS

0  

50  

100  

150  

200  

250  

300  

KLOC  

Local  Predic<on   Cross  project  MO  

Cross-project MO vs. Local Prediction

0  

50  

100  

150  

200  

250  

300  

KLOC  

Local  Predic<on   Cross  project  MO  

Cross-project MO vs. Local Prediction

The multi-objective predictor outperforms the local

predictor.

Conclusions

Conclusions

Conclusions

Conclusions