Programming Language Support for Automated

TestingRoy Patrick Tan

Virginia Tech

Software tools help us write bug-free programs

• Compiler– Syntax errors– Type checking

• Formal Verification– Need to write formal specifications– Proofs usually need human intervention

• Unit testing– Need to write unit tests – However, running unit tests can be automatic

We can use formal specs to generate unit tests

• Design-by-contract style specs (i.e. “executable” specifications)

• Generate test-case candidate using sequences of method calls

• Use postconditions as test-oracle

• Use preconditions to filter invalid tests

Sample Spec: Stackconcept Stack( Item ) {

model method sequence(): concept Sequence( Item ); method push( element: Item ) ensures sequence().equals( \old(sequence().insertLast( element.mathObject() ));

method pop( element: Item ) requires sequence().notEmpty() ensures \old(sequence()).equals( sequence().insertLast( element.mathObject() ));

method length(): Int ensures { \old(sequence()).equals(sequence()); length.equals(sequence().length()); };}

Model object lifetime as walks through a graph

• Every walk from init to finalize is an object lifetime -- can use as a test-case

• Some walks may be infeasible

• Generating all possible walks may be impossible

• Reseach problem: what set of walks will be good test-cases?

Automated-testing: what PL features do we need?

• Design-by-contract formal specs– Preconditions, postconditions, invariants should be

executable

• Enough features to be realistic, but not too much to be overwhelming– Encourage language features that make it easier

to write specs (separate specs and implementation)

– Eliminate language features that make it difficult to write specs (aliasing, null values)

Sulu• Main influences: RESOLVE, JML• Main differences from RESOLVE:

– object-oriented notation– spec language geared toward DBC

• Separation between specification (concepts) and implementation (realizations)

• Every object has a default initial state (no null values)

• Swapping ‘:=:’ as the main data movement operator

Digression: What’s wrong with assignment?

• Consider this java snippet:int a, b;a = 1;b = a;b = b + 1;

• After these operations what is the value of a?

• What is the value of b?

Object assignment is different from scalar assignment

• Now consider this java snippetjava.util.Stack a, b;

a = new java.util.Stack();

b = new java.util.Stack();

b = a;

b.push(“Hello”);

• What does b contain after these statements?• What about a?

Aliasing is evil• Scalar assignment in Java creates a

copy of the value• Object assignment creates an alias to

the same object, not a copy.• Aliasing is a big source of bugs!• Simple example, squaring a matrix:

matrix_multiply(a,a);

• Aliasing breaks modularity of reasoning

Is there a different way?• Copying preserves modularity of

reasoning but can be inefficient• Aliasing is efficient, but breaks

modularity of reasoning• Swapping!

– a :=: b– The pre-value of a becomes the post-value

of b, and vice versa

Swapping as an alternative to assignment?

• Swapping is efficient– Internally, compiler can make every swap

constant time.

• Swapping does not break modularity of reasoning– No aliases

Sulu Example:Using the Stack componentclass StringStack extends concept Stack(String) realization LinkedList();

x.push("Hello");x.push("World");x.push("!");

var c: Console;

x.pop(z);c.println(z);x.pop(z);c.println(z);x.pop(z);c.println(z);

Output:

!

World

Hello

Stack implementationrealization LinkedList() implements Stack( Item ) {

/* A linked-list node is a pair containing the item to store and a pointer to another node*/

class Node extends concept Pair( Item, concept ChainPointer(Node) realization Builtin() ) realization Obvious();

var top: concept ChainPointer(Node) realization Builtin(); var count: Int;

method pop( element: Item ) { //”dereference” the top pointer var topNode: Node; top.swapEntry( topNode );

//get the value at the top of //the stack and put in element topNode.swapFirst( element );

//set the value of top to the //next-pointer topNode.swapSecond( top );

count := count.minus( 1 );}

Stack implementationrealization LinkedList() implements Stack( Item ) {

/* A linked-list node is a pair containing the item to store and a pointer to another node*/

class Node extends concept Pair( Item, concept ChainPointer(Node) realization Builtin() ) realization Obvious();

var top: concept ChainPointer(Node) realization Builtin(); var count: Int;

method pop( element: Item ) { //”dereference” the top pointer var topNode: Node; top.swapEntry( topNode );

//get the value at the top of //the stack and put in element topNode.swapFirst( element );

//set the value of top to the //next-pointer topNode.swapSecond( top );

count := count.minus( 1 );}

Pointers are just another component,No special language constructs for pointers

Stack implementationrealization LinkedList() implements Stack( Item ) {

/* A linked-list node is a pair containing the item to store and a pointer to another node*/

class Node extends concept Pair( Item, concept ChainPointer(Node) realization Builtin() ) realization Obvious();

var top: concept ChainPointer(Node) realization Builtin(); var count: Int;

method pop( element: Item ) { //”dereference” the top pointer var topNode: Node; top.swapEntry( topNode );

//get the value at the top of //the stack and put in element topNode.swapFirst( element );

//set the value of top to the //next-pointer topNode.swapSecond( top );

count := count.minus( 1 );}

Swapping is the main data-movement operator

Stack implementationrealization LinkedList() implements Stack( Item ) {

/* A linked-list node is a pair containing the item to store and a pointer to another node*/

class Node extends concept Pair( Item, concept ChainPointer(Node) realization Builtin() ) realization Obvious();

var top: concept ChainPointer(Node) realization Builtin(); var count: Int;

method pop( element: Item ) { //”dereference” the top pointer var topNode: Node; top.swapEntry( topNode );

//get the value at the top of //the stack and put in element topNode.swapFirst( element );

//set the value of top to the //next-pointer topNode.swapSecond( top );

count := count.minus( 1 );}

Assignment is only allowed with methodreturn values.

What’s next?

• Fully implement specification language

• Implement test-case generation algorithms (all-pairs, all-triples, others?)

• Run experiments

• Graduate