Automated Fitness Guided Fault LocalizationAutomated Fitness Guided Fault LocalizationJosh Wilkerson, Ph.D. candidateJosh Wilkerson, Ph.D. candidate

Natural Computation LaboratoryNatural Computation Laboratory

Page 2Technical Background

Software testing

– Essential phase in software design

– Subject software to test cases to expose errors

– Locate errors in code

• Fault localization

• Most expensive component of software debugging [1]

– Tools and techniques available to assist

• Automation of this process is a very active research area

Page 3Technical Background

Fitness Function (FF)

– Quantifies program performance

• Sensitive to all objectives of the program

• Graduated

– Can be generated from:

• Formal/informal specifications

• Oracle (i.e., software developer)

– Correct execution: high/maximum fitness

– Incorrect execution: quantify how close to being correct the execution was

Page 4The FGFL System

Fitness Guided Fault Localization (FGFL) system

– Novel application of FFs to fault localization

– Ensemble of techniques

• Three techniques currently implemented

– Modular test case generation component

• Currently random with promotion for full fitness range coverage

Page 5FGFL Technique: Trace Comparison

Trace Comparison Technique

– Enhanced version of execution slice comparison fault localization technique [2]

– Execution trace: list of lines executed in a given run of a program

– Positive test case: results in correct program execution (indicated by FF)

– Negative test case: results in incorrect program execution (indicated by FF)

– Basic concept:

• Lines unique to negative test case traces highly suspicious

• Lines shared by positive/negative test case traces moderately suspicious

• Lines unique to positive test case traces not suspicious

Page 6FGFL Technique: Trace Comparison

Error on line 6

Bold lines indicate execution

Highly suspicious lines

Moderately suspicious lines

Positive Test Case:

1: Loop(i = 1…Size)

2: Loop(j = 1…Size-1)

3: If(data[j] > data[j+1])

4: temp = data[j]

5: data[j] = data[j+1]

6: data[j+1] = data[j]

Negative Test Case:

1: Loop(i = 1…Size)

2: Loop(j = 1…Size-1)

3: If(data[j] > data[j+1])

4: temp = data[j]

5: data[j] = data[j+1]

6: data[j+1] = data[j]

Result:

1: Loop(i = 1…Size)

2: Loop(j = 1…Size-1)

3: If(data[j] > data[j+1])

4: temp = data[j]

5: data[j] = data[j+1]

6: data[j+1] = data[j]

Page 7FGFL Technique: TBLS

Trend Based Line Suspicion (TBLS) Technique

– Based on the algorithm used by the Tarantula fault localization technique [3,4,5]

– Basic concept: lines containing an error are going to be executed more by negative test cases

– Each line has an associated suspicion level

– For each execution trace

• Suspicion Adjustment Amount (SAA) calculated based on the fitness of the execution

– High fitness: negative SAA

– Low fitness: positive SAA

• SAA added to suspicion for all lines in the trace

Page 8FGFL Technique: TBLS

Example:

– Five test cases used

• One positive

• Four negative (varying performance)

– Results:

Suspicion Levels:

-1 1: Loop(i = 1…Size)

-1 2: Loop(j = 1…Size-1)

-1 3: If(data[j] > data[j+1])

5 4: temp = data[j]

5 5: data[j] = data[j+1]

5 6: data[j+1] = data[j]

Page 9FGFL Technique: Fitness Monitor

Run-time fitness monitor technique

– Novel technique

– Basic concept: lines that cause a decrease in fitness consistently are likely to contain an error

– Program is instrumented to enable calculation fitness after every line execution

– Fitness fluctuation lines are found in the trace

– Fitness regions are generated around fluctuation lines

• Start with just fluctuation line

• Expanded out until fitness becomes stable

– If execution of the region results in an overall drop in fitness the fluctuation line becomes more suspicious

– No change in suspicion otherwise

Page 10FGFL Technique: Fitness Monitor

Example:

– Fitness plots for five test cases

1: Loop(i = 1…Size)

2: Loop(j = 1…Size-1)

3: If(data[j] > data[j+1])

4: temp = data[j]

5: data[j] = data[j+1]

6: data[j+1] = data[j]

Page 11FGFL: Result Combination

Technique results combined using a voting system

Each technique is given an equal number of votes

– Number of votes equal to the number of lines in the program

– Techniques do not have to use all votes

Suspicion values adjusted to reflect confidence in result

– Suspicion values scaled relevant to maximum suspicion possible for the technique

Votes are applied proportional to the confidence of the result

Method chosen to help reduce misleading results

Page 12Experimental Setup

Statistical analysis program (46-50 lines)

– Seven versions

– Each version has error(s) seeded

– Variety of error types

– 10 sets of test cases generated, each consisting of 75 test cases

Goal:

– Proof of FGFL concept

– Expose strengths and weaknesses of techniques

– Determine synergies between techniques and strength of ensemble approach

Page 13Results

Program lines ordered from most suspicious to least

Average rank of the line(s) containing the error in ordered list:

Program Version

Trace Comp.

TBLSFitness Mon.

Trace Comp. &

TBLS

TBLS & Fitness Mon.

Trace Comp. & Fitness

Mon.All

1 16 10.9 4 4.9 4 3 32 16 4.9 2 4.9 1 2 13 17 45.9 1 22.4 1 1 14 17 45 6 20.9 12.1 5.3 10.6

5 20 9.9 46 4.5 9.9 20 4.56 49 49 49 49 49 49 497 20 45.8 8.9 25.8 15.9 8.3 15.9

Best performance: full ensemble and trace comparison paired with fitness monitor

Program version 6: incorrect branch predicate, making branch unreachable

Need a technique based on statement reachability

Page 14Ongoing Work

Techniques are still under active development

Investigating the enhancement of other state-of-the-art fault localization techniques through a FF

Development of new techniques exploiting the FF

Use of multi-objective FF with FGFL

Testing FGFL on well known, benchmark problems

Page 15References

[1] I. Vessey, “Expertise in Debugging Computer Programs: An Analysis of the Content of Verbal Protocols,” IEEE Transactions on Systems, Man and Cybernetics, vol. 16, no. 5, pp. 621–637, September 1986.

[2] H. Agrawal, J. R. Horgan, S. London, and W. E. Wong, “Fault localization using execution slices and dataflow sets,” in Proceedings of the 6th IEEE International Symposium on Software Reliability Engineering, 1995, pp. 143–151.

[3] J. A. Jones, M. J. Harrold, and J. Stasko, “Visualization of test information to assist fault localization,” in Proceedings of the 24th International Conference on Software Engineering. New York, NY, USA: ACM, 2002, pp. 467–477.

[4] J. A. Jones and M. J. Harrold, “Empirical evaluation of the tarantula automatic fault-localization technique,” in Proceedings of the 20th IEEE/ACM international Conference on Automated software engineering. New York, NY, USA: ACM, 2005, pp. 273–282.

[5] J. A. Jones, “Semi-automatic fault localization,” Ph.D. dissertation, Georgia Institute of Technology, 2008.

Page 16

Questions?