Automatic Identification of Bug-Introducing Changes

Automatic Identification of Bug-Introducing Changes

Sunghun KimKai Pan E. James Whitehead, Jr.

University of California, Santa Cruz, USATom Zimmermann

Saarland University, Saarbrücken, Germany

Bugs

Motivation

Bugs

Motivation

Statistics

Bugs

Motivation

DetectionStatistics

Bugs

Motivation

DetectionStatistics

Prediction

Bugs

Motivation

DetectionStatistics

Analysis

Prediction

Bugs

Motivation

DetectionStatistics

Observation

Analysis

Prediction

So far: Focus on fixes

fixes issues mentioned in bug 45635: [hovering] rollover hovers- mouse exit detection is safer and should not allow for loopholes any more, except for shell deactiviation- hovers behave like normal ones: - tooltips pop up below the control - they move with subjectArea - once a popup is showing, they will show up instantly

teicher 2003-10-29 16:11:01

So far: Focus on fixes


teicher 2003-10-29 16:11:01

Fixes give only the location of a defect,not when it was introduced.

Bug-introducing changes

...if (foo==null) { foo.bar();...

BUG-INTRODUCING

if (foo==null) {


...if (foo!=null) { foo.bar();...

FIX

if (foo!=null) {...if (foo==null) { foo.bar();...

BUG-INTRODUCING

if (foo==null) { later fixed


Bug-introducing changes are changes that lead to problems as indicated by later fixes.

...if (foo!=null) { foo.bar();...

FIX

if (foo!=null) {...if (foo==null) { foo.bar();...

BUG-INTRODUCING

if (foo==null) { later fixed

Life-cycle of a “bug”


BUG-INTRODUCINGCHANGE



BUG REPORT




BUG REPORT

FIXCHANGE


The SZZ algorithm

1.18

FIXED BUG42233

The SZZ algorithm

1.18

FIXED BUG42233

$ cvs annotate -r 1.17 Foo.java ...20: 1.11 (john 12-Feb-03): return i/0; ...40: 1.14 (kate 23-May-03): return 42; ...60: 1.16 (mary 10-Jun-03): int i=0;

1.14 1.161.11

The SZZ algorithm

1.18

FIXED BUG42233


1.14 1.161.111.11

The SZZ algorithm

1.18

FIXED BUG42233

BUGINTRO


1.14 1.161.111.11 1.14

The SZZ algorithm

1.18

FIXED BUG42233

BUGINTRO

BUGINTRO


1.14 1.161.111.11 1.14 1.16

The SZZ algorithm

1.18

FIXED BUG42233

BUGINTRO

BUGINTRO

BUGINTRO


1.14 1.16

The SZZ algorithm

1.14 1.16 1.18

FIXED BUG42233

BUGINTRO

BUGINTRO

BUGINTRO

1.11 1.14 1.16

BUGINTRO

BUGINTRO


BUG REPORT

closedsubmitted

1.14 1.16

The SZZ algorithm

1.14 1.16 1.18

FIXED BUG42233

BUGINTRO

BUGINTRO

BUGINTRO

1.11 1.14 1.16

BUGINTRO

BUGINTRO


BUG REPORT

closedsubmitted

1.14 1.16

The SZZ algorithm

1.14 1.16 1.18

FIXED BUG42233

BUGINTRO

BUGINTRO

BUGINTRO

1.11

REMOVE FALSE POSITIVES

Drawbacks of SZZ

Annotation by SCMs is insufficient.(line number in bug-introducing revision is missing)

Drawbacks of SZZ

Not all modifications are fixes.(blank lines, comments, etc.)

Annotation by SCMs is insufficient.(line number in bug-introducing revision is missing)

False negatives and positives

BUGSBUG-INTRODUCING

False negatives

False positives

Correct

An example

void bar() { if (val==null) { println(val); }...

BUG-INTRODUCING

Revision 7: tom- introduces the defects

An example


BUG-INTRODUCING


void foo() { // print val if (val==null) { println(val); }...Revision 23: jim- inserts a comment- reformats if statement

void foo() { // print value if (val!=null) { println(val); }...

BUG FIX

Revision 42: kim- changes comment- corrects defect

An example


BUG-INTRODUCING




BUG FIX


// print val if (val!=null)

An example


BUG-INTRODUCING




BUG FIX


// print val if (val==null) {


An example


BUG-INTRODUCING




BUG FIX


// print val if (val==null) {


The originial SZZ algorithm has too many false positives (rev 23) and false negatives (rev 7).

Our study

COLUMBA

Software type

Number of revisions

Investigated period

Email client

500

11/2002-06/2003

Project

Number of fixes

Average LOC

ECLIPSE (jdt.core)

143 (29%)

48,135

IDE

1000

06/2001-03/2002

158 (16%)

111,059

Annotation graphs

Annotation graphs

fixed

revisions

foo()

Annotation graphs

fixed

revisions

foo()

Annotation graphs

fixed

revisions

bar() renamed to foo()

foo()

bar()

Annotation graphs

fixed

revisions


foo()

bar()

Annotation graphs

SZZ reports a bug-introducing change for foo (false positive) but not for bar (false negative).

fixed

revisions


foo()

bar()

Annotation graphs

fixed

revisions


Using annotation graphs we can remove 2% as false positives and identifies further 1%~4%.

Comments & blank lines

public void notifySourceElementRequestor() {- + if (reportReferenceInfo) {+ notifyAllUnknownReferences();+ } // collect the top level ast nodes int length = 0;

Comments & blank lines

public void notifySourceElementRequestor() {- + if (reportReferenceInfo) {+ notifyAllUnknownReferences();+ } // collect the top level ast nodes int length = 0;

Ignoring comments and blank linesremoves 14%~20% as false positives.

Format changes

if ( a==true ) return;BUG-INTRODUCING

Format changes


if (a==true) return;

if (a==false) return;

BUG FIX

Format changes




BUG FIX

FALSE POSITIVE

Format changes




BUG FIX

FALSE POSITIVE

FALSE NEGATIVE

Format changes




BUG FIX

FALSE POSITIVE

FALSE NEGATIVE

Ignoring format changes removes 18%~25% as false positives and identifies further 13%~14%.

Fixes that affect many files

- public boolean visit(TypeDeclaration - typeDeclaration, BlockScope scope){+ public boolean visit(LocalTypeDeclaration+ typeDeclaration, BlockScope scope){

Most large fixes are refactoring

Fixes that affect many files

- public boolean visit(TypeDeclaration - typeDeclaration, BlockScope scope){+ public boolean visit(LocalTypeDeclaration+ typeDeclaration, BlockScope scope){

Most large fixes are refactoring

Ignoring fixes that affect many files (=more than five times the median) removes 7%~16% as false positives

Manual inspection of fixes

deleteResources(actualNonJavaResources,fForce);- IResource[] remaingFiles; + IResource[] remainingFiles; try { - remaingFiles=((IFolder)res).members(); + remainingFiles=((IFolder)res).members(); }

Two judges check whether a fix is actually a fix.

Manual inspection of fixes

deleteResources(actualNonJavaResources,fForce);- IResource[] remaingFiles; + IResource[] remainingFiles; try { - remaingFiles=((IFolder)res).members(); + remainingFiles=((IFolder)res).members(); }

Manual inspection of fixes removes only 4%~5% as false positives.

Two judges check whether a fix is actually a fix.

Putting it together

BUGS

SZZ

ASE

Putting it together

BUGS

SZZ

False negativesof SZZ: 14%

ASE

Putting it together

BUGS

SZZ

False positives of SZZ: 38%~51%


ASE

Putting it together

BUGS

SZZ

False positives of SZZ: 38%~51%


ASE

Automatically we can remove 36%~48% as false positive.

Don’t program on Fridays ;-)

Sunday

Saturday

Friday

Thursday

Wednesday

Tuesday

Monday

0% 2.5% 5.0% 7.5% 10.0% 12.5% 15.0%

11.3%

10.4%

11.1%

12.1%

12.2%

11.7%

11.6%

Percentage of bug-introducing changes

Defect-prone authors

Amy

Bender

Leela

Fry

0% 10% 20% 30% 40% 50% 60% 70% 80%

Bug-Introducing changesFix changes

Authors(anonymized)

Frequency

Risk awareness

“Safe” Location(green)

Risky Location(dark red)

Change classification


X X X X

bug-introducing (“bad”)


X X X X

BUILD A CLASSIFIER



X X X X

BUILD A CLASSIFIER

new change



X X X X

BUILD A CLASSIFIER

PREDICT QUALITY

new change


Conclusions

Conclusions

• Bug-introducing changes tell when a defect was introduced, not only its location.

Conclusions


• We can automatically identify – 36%~48% of SZZ as false positives and – further 14% of missed bug-introductions.

Conclusions



• Bug-introducing changes are useful for defect prediction and software evolution.

Conclusions



• Bug-introducing changes are useful for defect prediction and software evolution.

More mining @ ASE:Friday morning.

Automatic Identification of Bug-Introducing Changes

Technology

Transcript of Automatic Identification of Bug-Introducing Changes