Software Testing

Sudipto GhoshCS 406 Fall 99November 9, 1999

11/09/99 CS 406 Testing 2

Learning objectives

• Functional testing• Equivalence class partitioning• Boundary value analysis

• Structural testing• Control flow-based

11/09/99 CS 406 Testing 3

Testing for correctness

• Identify the input domain of P.• Execute P against each element of the

input domain.• For each execution of P, check if P

generates the correct output as per its specification S.

11/09/99 CS 406 Testing 4

Input domain

• The set of all valid inputs that a program P can expect is the input domain of P.

• The size of an input domain is the number of elements in it.

• An input domain could be finite or infinite.• Finite input domains may be large!

11/09/99 CS 406 Testing 5

Identifying input domains

• For the sort program:• N: size of the sequence, N < 3• Sequence of N integers, such that for any

integer K, 0 K (e –1), e = 3• K: each element in the sequence• Some sequences are:

• [ ] : An empty sequence (N = 0)• [0] : A sequence of size 1 (N = 1)• [2 1] : A sequence of size 2 (N = 2)

11/09/99 CS 406 Testing 6

Size of input domain

• Suppose that 0 N 106

• The size of the input domain is the number of all sequences of size 0, 1, 2…

• The size is computed as:

• To test for correctness, P needs to be tested on all inputs.

• How many years would that take?

610

0i

ie

11/09/99 CS 406 Testing 7

Exhaustive testing

• Previous example• P is executed on all elements in the input

domain.• For most programs, exhaustive testing is

not feasible• What is the alternative?

11/09/99 CS 406 Testing 8

Partition testing

• Input domain is partitioned into a finite number of sub-domains.

• P is then executed on a few elements in each sub-domain.

• Four sub-domains for our

example:

[], [2], [2 0], [2 1 0]

Reduced from 85 to 4 !

N=01

N=12

N=34

N=23

11/09/99 CS 406 Testing 9

Confidence in your program

• Confidence is a measure of one’s belief in the correctness of the program

• Correctness is not measured in binary terms: a correct or incorrect program

• Measured as a probability of correct operation of a program when used in various scenarios• Reliability• Test completeness: Extent to which a program has

been tested and errors have been found and removed

11/09/99 CS 406 Testing 10

Types of module testing

• “Informal testing” performed by programmer while developing module

• Methodical testing performed by SQA group• Nonexecution-based testing• Execution-based testing

• Haphazard data as input• Systematically constructed test cases

• Testing to specifications

• Testing to code

11/09/99 CS 406 Testing 11

Testing to specifications

• Black-box testing• Data-driven testing• Input/output-driven testing• Functional testing• Code is ignored, i.e. the internal structure

of the code is not important• The specification document is the only

information used for creating test cases

11/09/99 CS 406 Testing 12

Testing to code

• White-box testing• Glass-box testing• Logic-driven testing• Path-oriented testing• Specifications are not used to generate

the test cases• Test “to code,” i.e. the internal structure of

the code is important

11/09/99 CS 406 Testing 13

What is functional testing?

• Functional testing tests how well a program meets the functionality requirements.

• When test inputs are generated using program specifications, we say that we are doing functional testing.

11/09/99 CS 406 Testing 14

Feasibility of functional testing

• Problem with exhaustive testing• Very large (possibly infinite) number of test

cases.

• Need to devise:• Small, manageable set of test cases.• Maximize the chances of detecting faults.• Minimize the chances of wasting test cases by

having more than one test case detecting the same fault.

11/09/99 CS 406 Testing 15

Equivalence class partitioning

• Divide the domain of all inputs into a set of equivalence classes.

• If any test in an equivalence class succeeds, then every test in that class will succeed.

• How would you get ideal equivalence classes?• Difficult without looking at the internal structure• Difficult even with the internal structure

11/09/99 CS 406 Testing 16

Equivalence class partitioning

Set of all inputs Specifications

Put those inputs together for which the behaviorpattern of the module is specified to be different

into similar groups

Equivalence classes

11/09/99 CS 406 Testing 17

Rationale behind equivalence class partitioning

We assume that if the specifications require exactly the same behavior from each element in a class of values, then the program is likely to be constructed so that it either succeeds or fails for each of the values in that class.

11/09/99 CS 406 Testing 18

Guidelines for partitioning

• For robust software, we must test for incorrect inputs too.

• For each equivalence class of valid inputs, we have equivalence classes of invalid inputs.• Input condition specifies a range a X b.

• a X b (Valid case)• X < a, and X > b (invalid cases)

• Input specifies a value• create one for the valid value• create two for incorrect (one above, one below)

11/09/99 CS 406 Testing 19

Guidelines for partitioning

• Input specifies a value (contd.)• For boolean, only one value

• Input condition specifies a member of a set• Create one for the valid value• Create one for invalid value (not in the set)

• Example:• Factorial (n), where n 0• Valid classes {0}, {x | x 1}• Invalid class {x | x < 0}

11/09/99 CS 406 Testing 20

Non-overlapping partitions

• In the previous example, the equivalence classes were non-overlapping, i.e., the sub-domains were disjoint.

• It is sufficient to pick one test from each equivalence class to test the program.

• An equivalence class is considered covered if at least one test has been selected from it.

• Our goal is to cover all equivalence classes.

11/09/99 CS 406 Testing 21

Overlapping partitions

• Suppose a program P takes three integers, X, Y and Z. It is known that:• X < Y• Z > Y

Z > Y

X < YZ > Y

X < Y

X YZ Y

11/09/99 CS 406 Testing 22

Overlapping partition:test selection

• Select 4 test sets as• X=4, Y=7, Z=1 (satisfies X < Y)• X=4, Y=2, Z=1 (satisfies X Y)• X=1, Y=7, Z=9 (satisfies Z > Y)• X=1, Y=7, Z=2 (satisfies Z Y)

• Can also reduce the number of test cases to 2• X=4, Y=7, Z=1 (satisfies X < Y and Z Y)• X=4, Y=2, Z=3 (satisfies X Y and Z > Y)

11/09/99 CS 406 Testing 23

Boundary Value Analysis (BVA)

• Observation:• Programs that work correctly for a set of

values in an equivalence class, fail on some special values.

• These values lie on the boundary of the equivalence class.

• Choose an input from a test case from an equivalence class such that the input lies at the edge of the equivalence class.

• These test cases are “extreme cases”

11/09/99 CS 406 Testing 24

BVA examples

• Suppose the range is 0.0 X 1.0.• 0.0 (valid input)• 1.0 (valid input)• -0.1 (invalid input)• 1.1 (invalid input)

• For a list• first and last element of the list

• A program takes a string S and integer X as input such that • a X b, and length(S) 100. Derive tests.

11/09/99 CS 406 Testing 25

BVA analysis

• A program takes two integers X and Y, such that a X b, c Y d.

• How many test cases do we get?

a b

c

d

11/09/99 CS 406 Testing 26

Output variables

• Equivalence class partitioning can be applied to output variables

• BVA also can be applied to output data

11/09/99 CS 406 Testing 27

Testing functions

• Identify each function implemented in a module

• Devise test data to test each function separately

• Module may contain a hierarchy of lower level functions connected by program control structures• Perform functional testing recursively• Higher level: black-box• Lower level: glass-box

11/09/99 CS 406 Testing 28

Problems with previous approach

• Usually higher level functions are not as structured• Lower level functions are intertwined

• Functionality does not coincide with module boundaries• Distinction between module testing and

integration testing is blurred• Testing one module is not possible without

simultaneously testing other modules whose functionality is used

11/09/99 CS 406 Testing 29

Structural testing

• Intent is to exercise the different programming structures and data structures used in the program

• Intent is not to exercise all the different input and output conditions• though this may be a by-product

• Achieve test cases that force the desired “coverage” of different structures

• Criteria are formal and precise

11/09/99 CS 406 Testing 30

Control-flow based criteria

• Statement coverage• Run a series of test cases and ensure that

every statement is executed at least once• Simplest form of glass box testing• What is the weakness? Consider the example:

int abs (int x) {

if (x >= 0)

x = 0 - x;

return x;

}

Error Test inputs:

x = 5: What is coverage?x = 0: What is coverage?

11/09/99 CS 406 Testing 31

Statement coverage

• Not very strong, may leave errors undetected.

• Examples:• if statement without else part• conjunctions of predicates

• In all these cases, all branches were probably not exercised.

• Can you think of a better criterion based on the above observation?

11/09/99 CS 406 Testing 32

Branch coverage

• Require that every decision is evaluated to true and false values at least once during testing

• Branch coverage implies statement coverage• Each statement is part of some branch

11/09/99 CS 406 Testing 33

Control flow graph of a program

• Let G be the graph of a program P.• Node:

• Represents a block of statements that are always executed together

• Edge (i, j) from node i to node j:• Represents a possible transfer of control after

executing the last statement of the block represented by node i to the first statement in the block represented by node j.

11/09/99 CS 406 Testing 34

Control flow graph of a program

• Start node:• Node corresponding to a block whose first

statement is the start statement of P.

• Exit node:• Node corresponding to a block whose last

statement is an exit statement of P.

• Path:• Finite sequence of nodes (n1, n2, …, nk), k>1

such that there is an edge (ni, ni+1) for all nodes ni in the sequence, except the last node nk.

11/09/99 CS 406 Testing 35

Control flow graph of a program

• Complete Path:• Path whose first node is a start node and the

last node is an exit node.

• All-nodes criterion (statement coverage)• All-edges criterion (branch coverage)

11/09/99 CS 406 Testing 36

An example (xy)

1. scanf(x, y); if(y < 0)

2. pow = 0 – y;

3. else pow = y;

4. z = 1.0;

5. while(pow != 0)

6. { z = z * x; pow = pow – 1; }

7. if ( y < 0 )

8. z = 1.0/z;

9. printf(z);

11/09/99 CS 406 Testing 37

Control Flow Graph of example

1

2 3

6

4

5

7

8 9

11/09/99 CS 406 Testing 38

Problems with branch coverage

• What if a decision has many conditions (using and, or)

• Decision may evaluate to true or false without actually exercising all the conditionsint check (int x) {

if ((x >= 5) && (x <= 200))

return TRUE;

return FALSE;

}

Error (should be 100)

Test inputs:x = 5:x = -5:

11/09/99 CS 406 Testing 39

Solution?

• Require all individual conditions to evaluate to true and false

• Problem:• Even if individual conditions evaluate to true

and false, the decision may not get both true and false values

• Solution:• Require both decision / condition coverage!!

• Still there is a problem.

11/09/99 CS 406 Testing 40

Path testing

• Some errors are related to some combinations of branches.

• Presence revealed by an execution of a path that includes those branches.

• Solution:• Require all possible paths in the CFG to be

executed during testing.• Path-coverage criterion, or all-paths criterion• Path coverage implies branch coverage