LSR Test purposes:adapting the notion of specification to testing

Yves Ledru,

L. du Bousquet, P. Bontron,

O. Maury, C. Oriat, M.-L. PotetLSR/IMAG

Grenoble, France

ASE2001, Nov 27-29 2001, San Diego

LSR

Conformance testing• Given a black box

• And its specification

• Test cases compare the black box to its specification

X = sqrt(X)*sqrt(X)andsqrt(X) >= 0

?sqrt(4)

!2

accept

sqrt

LSR

Complex test cases

Unfortunately, complex applications

often result in complex test cases!

?Money_Transfer(1000000€)

!OK

accept

e-Bank

Money_transfer

LSR

Specifying test cases• It then makes sense

to specify these complex test cases

• « I want a test case that will execute the Money_transfer procedure successfully! »

• It is the notion of test purpose!

?Money_Transfer(1000000€)

!OK

accept

LSR

The COTE project• COTE:

– conformance testing of components vs UML specifications

– Softeam, France Telecom, Gemplus, IRISA, LSR/IMAG

– Languages and tools to express and exploit test purposes and test cases

– (compilers from test cases to target technologies)

• This work comes from a brainstorming activity around the notion of test purpose within the COTE project

LSR

The notion of test purposeTest purpose :

Description of a precise goal of thetest case, in terms of exercising a particular execution path or verifying the compliance with a specific requirement

[ISO IEC 9646-1 …conformance testing methodology and framework…]

• This notion is used:– As a structuring element in industrial test suites– In some tools based on formal specifications

(SAMSTAG, TGV, TorX)

LSR

Test purposes in TGV• TGV is a tool developed jointly by IRISA

(Rennes) and Vérimag (Grenoble)

• Test synthesis tool

• Given– A specification of the system under test– A test purpose

It generates one corresponding test case.

LSR

Example : coffee machine

?coin(1)

?coin(1)

?coin(2)

Pay 1 or 2 coins

?sugar

Ask for sugar (optional)

?tea?tea

?coffee

?coffeeAsk for tea or coffee

!coffee

!coffee!tea

!tea

!sugar

!sugar

Get the beverage

Loop

LSR

TGVTGV

?coin(1)

?coin(1)

?coin(2)

?sugar

?tea?tea

?coffee

?coffee

!coffee

!coffee!tea

!tea

!sugar

!sugar

Dynamic specification

?coin(1)

?coin(1)

?coin(2)

?sugar

?tea?tea

?coffee

?coffee

!coffee

!coffee!tea

!tea

!sugar

!sugar

A sample synthesis

!coffee

accept

Test purpose

?coin(2)

?coffee

!coffee

Test case

LSR

Test purposes as specification

• A test purpose (P) is an abstraction of test cases

• In TGV, a partial sequence of the test cases

• A test purpose specifies a set of test cases

!coffee

accept?coin(2)

?coffee

!coffee

?coin(1)

?coin(1)

?sugar

?tea?coffee

!coffee

?coin(1)

?coin(1)

?coin(2)

?sugar

?tea

?coffee

!coffee

!tea

!sugar

LSR

Benefits of abstraction

• Capture the essence of the test(here test the delivery of coffee)

• Shorter to state than the full test case(clerical completion left to the tool)

• Robust to evolutions of the dynamic specification(test cases are affected more often than test purposes, e.g. pay 3 instead of 2)

!coffee

accept

LSR

Relations between test purposes (P),

test cases (C) and specifications (S)

LSR

Weakly_refines(C,P)• A test case « refines » the test

purpose(if it includes the partial sequence)

• But several irrelevant test cases refine the test purpose also. Weak refinement!

!coffee

accept

?coin(2)

?coffee

!coffee

test case not conforming to the specification

!coffee

LSR

refines(C,P,S)• The refinement takes place in the

context of the specification

!coffee

accept?coin(2)

?coffee

!coffee

?coin(1)

?coin(1)

?coin(2)

?sugar

?tea?tea

?coffee

?coffee

!coffee

!coffee!tea

!tea

!sugar

!sugar

refines

LSR

weakly_refines vs refines• Refines is a ternary relation

• Weakly_refines is defined asweakly_refines(C,P) S refines(C,P,S)

refines

weakly_refinesTest case(C)

Specification(S)

Test Purpose(P)

LSR

Semantics for refines(C,P,S)

• Depending on the languages used for C, P, and S, a semantics can be defined for refines(C,P,S)

• Semantics of refines in TGV: C is one of the paths of the synchronous product of P and S

• See also, in the paper:semantics of a test refinement relation in the context of the B method

LSR

Semantics of weakly_refines

• As a consequence:weakly_refines(C,P) S refines(C,P,S)

gives a semantics to weakly_refines on top of the semantics of refines

LSR

conforms(C,S)

conforms(C,S) P refines(C,P,S)

• Conforms checks that the test case corresponds to the specification

refines

Test case(C)

Specification(S)

Test Purpose(P)

conforms

LSR

consistent(P,S)

consistent(P,S) C refines(C,P,S)

• Consistent checks if the test purpose is compatible with the specification

refines

cons

isten

t

Test case(C)

Specification(S)

Test Purpose(P)

LSR

Summary

• If refines(C,P,S) has a semantics,it gives a semantics to the binary relations!

• In some cases, it is possible to systematically construct the third element of the relation

refines

weakly_refines

cons

isten

t

Test case(C)

Specification(S)

Test Purpose(P)

conforms

LSR

Test specificationvs

software specification

LSR

refinesSWProgram

(M)

SoftwareSpecification

(SS)

refines

Test case(C)

Specification(S)

Test Purpose(P)

LSR

Automated Software Engineering

• Formal specs as a basis for software tools:– Verification of specification consistency – Program synthesis– Program verification

LSR

Specification consistencyoften demonstrated by proving:

M refinesSW(M,SS)(there exists a model which refines the specification)

corresponds toconsistent(P,S) C refines(C,P,S)

(a test purpose makes sense if there exists a test case)

Tool

Detection of inconsistentpurposes(e.g. !sugar comes after !coffee)avoids useless searches

!coffee

accept

reject

?coin(*)

!sugar

LSR

SynthesisMajor theme at ASE

synthesisSW(SS) M

corresponds totest synthesis

e.g., TGV can be seen as a function

synthesisTGV(P,S) C

Good news for tools

The search domain is restricted by S(more decidable?, less complex?)

LSR

Verificationchecks refinesSW(M,SS)

(program M is a valid refinement of specification SS)

corresponds to3 possible test case verifications– refines(C,P,S)

C is a refinement of P in the context of S– weakly_refines(C,P)

C is a refinement of P in some context– conforms(C,S)

C is a test case for S

LSR

Tools for test verification• Synthesis involves searching a huge

space of solutions

• If verification is cheaper than synthesis,it provides a natural complement.

• Two examples:– Evolution of specifications– Test case retrieval

LSR

Evolution of specifications• Evolution of S into S’

– Given a set of (Ci,Pi) for S– Verify refines(Ci,Pi,S’) for each (Ci,Pi)– When it fails, verify conforms(Ci,S’)– A second failure reveals regression of S’

• Example:No need to recompute existing test cases if

the coffee machinealso serves hot chocolate

?tea?coffee

?choc

LSR

Test case retrievalGiven a large set of test cases

– Retrieve a test that weakly_refines a new test purpose

– Minimize the set by identifying test cases that weakly_refine several test purposes

!coffee

accept

?coin(2)

?coffee

!coffeeweakly_refines

weakly_refines

?coin(2)

accept

LSR

Related workand

conclusion

LSR

Test selection criteria

• 1st category: domain independent– Coverage criteria– Fault models

(mutation testing)– Analysis rules,

strategies– Random testing

• 2nd category: exploits domain knowledge– Test purposes

(based on reqts)– Operational profiles

• Must be consistent with the reference

Synthesis tool

Test selection criterion(aka test requirement)

Test case(s)Reference

(specification, program)

LSR

Conclusion• Test purposes are a specification of test

cases!

• 4 relations between C, P, and S– Not specific to a given tool or language– Not specific to an application domain

(TGV: reactive systems, B: transf. systems)

• Tools: from ASE to Automated Test Eng.– Based on these 4 relations– Test verification complements test synthesis– Applications to specification evolution, test

case retrieval,…

LSR

Interactive Coffee Crafting