Comparing Metaheuristic Algorithms for Error Detection in Java Programs

1 / 25 SSBSE 2011, Szeged, Hungary, September 10-12

Comparing Metaheuristic Algorithms for Error Detection in Java Programs

Francisco Chicano, Marco Ferreira and Enrique Alba

Introduction Background Algorithms Experiments Conclusions

& Future Work


•  Concurrent software is difficult to test ...

•  ... and it is in the heart of a lot of critical systems

•  Techniques for proving the correctness of concurrent software are required •  Model checking → fully automatic •  Traditional techniques for this purpose have problems with large models •  We compare several metaheuristics and classical algorithms for model checking

Motivation Motivation


& Future Work


Intersection Büchi automaton

•  Objective: Prove that model M satisfies the property :

•  In the general case, f is a temporal logic formula (LTL, CTL, etc.)

s0

s4 s7

s6

s2

s1

s8 s9

s5

s3

s0

s1 s2

s3

s4

s5

Model M

s1

s0 s2

q p∧!q

!p∨q

LTL formula ¬ f (never claim)

∩

Explicit State Model Checking Explicit State MC Properties State Explosion Heuristic MC


& Future Work





s0

s4 s7

s6

s2

s1

s8 s9

s5

s3

s0

s1 s2

s3

s4

s5

Model M

s1

s0 s2

q p∧!q

!p∨q


∩ =



& Future Work





s0

s4 s7

s6

s2

s1

s8 s9

s5

s3

s0

s1 s2

s3

s4

s5

Model M

s1

s0 s2

q p∧!q

!p∨q


∩

Using Nested-DFS

=



& Future Work


•  An error trail is an execution path ending in an error state •  The search for errors is transformed in a graph exploration problem (DFS, BFS)

s0

s4 s7

s6

s2

s1

s8 s9

s5

s3

Safety properties

Properties in JPF

•  Exceptions

•  Deadlocks


& Future Work

Explicit State MC Properties State Explosion Heuristic MC


s0

s4 s7

s6

s2

s1

s8 s9

s5

s3

•  Number of states very large even for small models

•  Example: Dining philosophers with n philosophers → 3n states

•  For each state we need to store the heap and the stacks of the different threads

•  Solutions: collapse compression, minimized automaton representation, bitstate

hashing, partial order reduction, symmetry reduction

•  Large models cannot be verified but errors can be found

Memory

State Explosion Problem


& Future Work



•  The search for errors can be directed by using heuristic information

•  Different kinds of heuristic functions have been proposed in the past: •  Formula-based heuristics •  Structural heuristics

s0

s4 s7

s6

s2

s1

s8 s9

s5

s3 2

0

3

5

1

2

4 0

7 6

Heuristic value

Heuristic Model Checking

•  Deadlock-detection heuristics

•  State-dependent heuristics


& Future Work



Classification of Algorithms


& Future Work

Algorithms GA GAMO PSO ACO SA

Guided

Non-guided

Complete Non-complete

Deterministic

Stochastic

Completeness

Det

erm

inis

m

DFS, BFS

A*

GA, GAMO, PSO, SA, ACO

RS

BS

RDFS

EDA

?

(working on)


Genetic Algorithm

0.5 0.1 0.9 0.3 0.5 0.9

Solution encoding

(floating point values)

Crossover

0.5 0.1 0.9 0.3 0.5 0.9 → 0.5 0.1 0.6 0.3 0.5 0.9 Mutation



& Future Work


Genetic Algorithm with Memory Operator

0.5 0.1 0.9 0.3 0.5 0.9

Solution encoding

(floating point values)

0 1 2 3 … …

Index in a table of states


& Future Work



Inertia Personal best

Global best

→ Position (solution)

→ Velocity

0.2 -1.4 -3.5

1.0 10.3 7.2

Particles


& Future Work


Particle Swarm Optimization


•  The ant selects stochastically its next node

•  The probability of selecting one node depends on the pheromone trail and the heuristic value (optional) of the edge/node •  The ant stops when a complete solution is built

i

j l

m

k

Ni

τij

ηij

k Trail

Heuristic

Ant Colony Optimization


& Future Work



Simulated Annealing

0.5 0.1 0.9 0.3 0.5 0.9 → 0.5 0.1 0.6 0.3 0.5 0.9 Neighbor


& Future Work



•  We used 3 scalable and 2 non-scalable models for the experiments

•  Maximum number of expanded states: 200 000 •  Fitness function:

•  100 independent executions of stochastic algorithms

Program LoC Classes Processes dinj 63 1 j+1 phij 176 3 j+1 marj 186 4 j+1 giop 746 13 7 garp 458 7 7

Parameterization Parameterization Hit Rate Length of Error Trails


& Future Work


•  We used 3 scalable and 2 non-scalable models for the experiments

•  Maximum number of expanded states: 200 000 •  Fitness function:

•  100 independent executions of stochastic algorithms

Program LoC Classes Processes dinj 63 1 j+1 phij 176 3 j+1 marj 186 4 j+1 giop 746 13 7 garp 458 7 7

Parameterization

j=4 to 20

j=4 to 36

j=2 to 10

Parameterization Hit Rate Length of Error Trails


& Future Work


Hit rate


& Future Work



Hit rate


& Future Work

phi



Hit rate


& Future Work

din



Hit rate


& Future Work



Length of Error Trails


& Future Work

BS: 753




& Future Work

BS: 172

BS: 260 DFS: 245

DFS: 378





& Future Work




•  Metaheuristics are more effective than classical algorithms in finding errors

•  Beam Search has advantages over complete search algorithms

•  An even distribution of the search in depth levels tends to raise hit rate

•  Stochastic algorithms obtain short error trails

Conclusions

Future Work

•  Design a stochastic complete guided algorithm to find errors and verify

•  Design of hybrid algorithms to more efficiently explore the search space

•  Explore the design of parallel metaheuristics for this problem

Conclusions & Future Work


& Future Work

Conclusions & Future Work


Thanks for your attention !!!

Comparing Metaheuristic Algorithms for Error Detection in Java Programs

Comparing Metaheuristic Algorithms for Error Detection in Java Programs

Technology

Transcript of Comparing Metaheuristic Algorithms for Error Detection in Java Programs