A Tool to Support Testing Activities

Gilson Doi Junior Adilson Luiz Bonifacio

November 7th, 2011

Gilson Doi Junior, Adilson Luiz Bonifacio () A Tool to Support Testing Activities November 7th, 2011 1 / 15

Summary

1 Concepts

2 Motivation

3 ToolTest case generation methodsFSM generation

4 Similar Tools

5 Experiment

6 Results

7 Future works

Gilson Doi Junior, Adilson Luiz Bonifacio () A Tool to Support Testing Activities November 7th, 2011 2 / 15

Concepts

Concepts

Testing

Check whether an implementation behaves according to thespecificationGuarantee reliable systemsFind faults (avoiding financial damage or human losses)

Model-based testing

Provide a rigorous testing frameworkBlack-box approachReduce ambiguities with formalismsSupport automated testing activities

Finite State Machine (FSM)

Specify input and outputsIt can be used to specify systems’ behavior

Gilson Doi Junior, Adilson Luiz Bonifacio () A Tool to Support Testing Activities November 7th, 2011 3 / 15

Motivation

Motivation

Few model-based testing tools support FSM models

Lack of a pool of models to support practical experiments

Lack of an standard to describe FSM

Tools that only implements one test case generation method

Compare some test case generation methods

Gilson Doi Junior, Adilson Luiz Bonifacio () A Tool to Support Testing Activities November 7th, 2011 4 / 15

Tool

Tool

Specification

Implementation

FSM generator

Test suite

generator

Oracle

execution Verdict

Pool of

models

Pool

size

Figure: Tool’s architecture

Gilson Doi Junior, Adilson Luiz Bonifacio () A Tool to Support Testing Activities November 7th, 2011 5 / 15

Tool

Tool(2)

Specification and IUT

Describe FSM models

Test suite generator

Need the Specification modelApply one of test case generation methods

Oracle execution

Apply the test suite to the specification and implementation modelCompare both behaviors and gives a verdict

FSM generator

Generate FSM models to provide a pool to the testing framework

Gilson Doi Junior, Adilson Luiz Bonifacio () A Tool to Support Testing Activities November 7th, 2011 6 / 15

Tool Test case generation methods

Test case generation methods

We implemented three methods to FSM test case generation closelyrelated, but distinct in the following aspects:

W-method

Uses the Characterization sets that distinguish every pair of states

Wp-method

Uses Identification sets that distinguish every single state

G-method

Uses R sets that distinguish families of equivalent states

Gilson Doi Junior, Adilson Luiz Bonifacio () A Tool to Support Testing Activities November 7th, 2011 7 / 15

Tool FSM generation

FSM generation(1)

Obtain the pool of FSM

Define a standard grammar to describe FSM

Similar approaches

Sidhu and Leung (1989) apply changes on a model to derive distinctIUTSimao and Petrenko (2010) present a similar algorithm that obtain onlyrestrict models

Gilson Doi Junior, Adilson Luiz Bonifacio () A Tool to Support Testing Activities November 7th, 2011 8 / 15

Tool FSM generation

FSM generation(2)

Specifications

Generates FSM according to test case generation methods

Implementation candidates (IUT)

Are unknown to the testerGenerates FSM with few restrictions: upper bound on the number ofstates and input and output alphabetDistinct transitions can be obtained with distinct combinations of inputand output symbol for each pair of statesDistinct FSM are derived by creating distinct transitions on the samepair of states

Gilson Doi Junior, Adilson Luiz Bonifacio () A Tool to Support Testing Activities November 7th, 2011 9 / 15

Similar Tools

Similar tools

Similar Tools:

JTorXSpec ExplorerTGV

None of them:

Deals with FSM modelsGenerates formal modelsImplements the W-method, Wp-method or G-method

Gilson Doi Junior, Adilson Luiz Bonifacio () A Tool to Support Testing Activities November 7th, 2011 10 / 15

Experiment

Practical Experiments(1)

A pool of FSM models obtained through the FSM generator

The test suite were obtained by the Test Suite Generator

Each FSM of the pool is tested as a IUT candidate

The results are checked by the oracle

Gilson Doi Junior, Adilson Luiz Bonifacio () A Tool to Support Testing Activities November 7th, 2011 11 / 15

Experiment

Practical Experiments(2)

s0 s2 s3a/0

b/0

a/0

b/0 b/0 a/1

b/1

W-method Wp-method G-method

ba, aba, bba,baba, bbba,bbaba, bbbba

ba, aba, baba,bbba

a, aaa, aba, baa, bba,baaa, baba, bbba, bbaaa,bbaba, bbbaa, bbbba

Gilson Doi Junior, Adilson Luiz Bonifacio () A Tool to Support Testing Activities November 7th, 2011 12 / 15

Results

Results

[1] [2] [3] [4] [5]

5 Method W 91 24 0

5 Method Wp 57 17 0

5 Method G 432 72 0

10 Method W 88 22 0

10 Method Wp 86 21 0

10 Method G 2112 224 0

1 Number of states

2 Method

3 Test Suite Length

4 Number of Test Sequences

5 Implementations in conformance

Gilson Doi Junior, Adilson Luiz Bonifacio () A Tool to Support Testing Activities November 7th, 2011 13 / 15

Results

Future works

Replace QT data structures by C++ STL to improve the performance

A graphical interface to aid the task of specifying formal modelsbased on the proposed grammar

Integrate test case generation methods for real-time systems andsystems with data flow transformations

Gilson Doi Junior, Adilson Luiz Bonifacio () A Tool to Support Testing Activities November 7th, 2011 14 / 15

Results

Thanks

Questions?

Gilson Doi Junior, Adilson Luiz Bonifacio () A Tool to Support Testing Activities November 7th, 2011 15 / 15