Structural Testing: When Quality Really Matters

Structural TestingWhen Quality Really Matters

Jamie L Mitchell

Jamie Mitchell Consulting Inc.

Copyright (c) Jamie Mitchell Consulting Inc.

2

When Software Really Needs to Work


3

Why Structure-based Testing?Experience-based techniques are not thorough

enough:◦ Error-guessing◦ Exploratory testing◦ Defect- and taxonomy-based

Black-box testing depends on how good your test basis model is:◦ Requirements◦ Use cases◦ Functional specifications

How well have you tested?

Is testing less than half of your code sufficient?


4

White-Box Test BasisDesign documents and models

◦Architecture◦Functional design◦Low-level design

Object models Class interface documents

Code


5

Today’s Discussion TargetsControl-flow analysis: selecting

certain paths through the software to test, based on desired coverage

Data-flow analysis: a control-flow technique that scrutinizes the life-cycle of data variables


6

Control-flow TestingSelect certain paths through the software ◦Based on desired level of coverage Based on statements Based on decisions Based on loops Based on paths


7

Statement CoverageExecute every line of codeTools available for measurement

◦Run tool while executing tests◦Evaluate coverage◦Create more tests to reach areas

missedAlso called instruction or code

coverage


8

Decision (Branch)CoverageStatement coverage may miss bugs

◦ One test needed for statement coverage (a = 5, b = 4) Passes

z = 0;if (a > b) then

z = 12;x = 72 / z

Select inputs to force each decision to execute both possible ways (T/F)◦ Now two test cases are needed for coverage

(a = 5, b = 4) Passes (a = 4, b = 5) Finds bug


9

Condition CoverageDecisions may be arbitrarily complex

if (a > b) then… if (A || B) && (C == D) || (!E) && (F) then…

Condition testing requires that each atomic condition must be evaluated both true and false

Note that condition coverage – by itself – may not be as strong as decision coverage

if (a AND b) then

(a = F, b = T F) (a = T, b = F F)


10

Decision / Condition CoverageA combination of Condition and

Decision1. Each atomic condition must be evaluated

both ways AND2. Decision coverage must be satisfied

if (a AND b) then

(a = F, b = T F) (a = T, b = F F) (a = T, b = T T)

Is that enough coverage?


11

Modified Condition / DecisionUsually called MC/DC

◦For each condition in the predicate, there is at least one test where the outcome is TRUE only because of that condition being TRUE

◦For each condition in the predicate, there is at least one test where the outcome is FALSE only because of that condition being FALSE

Usually results in N+1 test cases where N is number of conditions in predicate

Required by some safety standards to ensure that enough testing has been done

Ex: FAA DO/178B Level A (catastrophic) software


12

MC/DC Coverage (2)

if ((a || b) && c) then …

Decision/Condition can be achieved with

(a = T, b = T, c = T) T (a = F, b = F, c = F) F

Note that (b) never independently affects the outcome of the condition


13

MC/DC Coverage (3) if ((a || b) && c) then …

Following test set will achieve MC/DC coverage

1. (a = T, b = F, c = T) T 2. (a = F, b = F, c = T) F3. (a = F, b = T, c = T) T4. (a = T, b = F, c = F) F


14

Coverage Based on LoopsLoops make testing …

interesting…The number of possible paths

may approach infinite depending on how many loops are executed

Basic coverage criteria◦Test when no loops are made◦Test when a single loop is made◦Test when n loops are made (where n

is either the expected max or a large number)


15

Loops (2)This loop can calculate a factorial (n!)

for (i = 1; i <= n; i++) f *= i;

Use (n = 0) for zero loopsUse (n = 1) for one time throughDetermining the max requires

thought◦In this case, (f) gets very large◦Use (n = 12); 13 would cause an overflow


16

Nested LoopsNested loops are really hard to testBoris Beizer* had some advice

1. Start at the innermost loop, setting all outer loops to their minimum iteration setting

2. Test boundaries for innermost loop while keeping outer loops at their minimum

3. If you have done outermost loop goto 5. Else move one loop outward and test as in step 2 with all inner loops set to their typical values

4. Continue outward until all loops done5. Test the boundaries for all loops simultaneously

* Software Testing Techniques, 2nd Ed. by Boris Beizer


17

Path TestingFull path testing in a non-trivial system is

impossible◦The number of paths approaches infinite◦This is due to loops as seen previously

We can identify independent, non-looping paths which cover all the edges and nodes of a control-flow graph◦These are called basis paths◦Testing the basis paths can guarantee decision

(and statement) coverage◦McCabe cyclomatic complexity uses this method◦Tools are available for calculating this value


18

Cyclomatic Complexity ExampleComplexity = 3Tests needed

◦ 1-2-6-7-8◦ 1-3-4-7-8◦ 1-3-5-8

Example from “A Complexity Measure” by Thomas McCabe, IEEE Transactions on SW Engineering, Vol. SE-2, No 4, December 1976


19

Data-flow Analysis (1)Programs create, set, read, evaluate and

destroy dataThe life cycle of data can be analyzed to find

defects that control-flow techniques missSome defect examples

◦ Incorrect assignment of a value◦ Incorrect input statement ◦ Failure to define a variable before use◦ Incorrect path taken due to incorrect value in a

control predicate ◦ Use of a variable after it is destroyed or out of scope◦ Redefinition of a variable before it can be used.

Look for consecutive touches to each variable


20

Data-flow Analysis (2)Dependent on the language usedWe look for anomalies in data life-

cycle ◦Not all anomalies are defects◦While this is nominally a static technique,

it will not find all data defects (often data is manipulated or assigned dynamically)

Sometimes called DUK testing◦D Define◦U Use◦K Kill


21

Data-flow Testing Combinations (1)

Anomaly Explanation1. ~d first

defineAllowed

2. du define-use

Allowed, normal case

3. dk define-kill Potential bug; data was never used

4. ~u first use Potential bug; data was used without definition. It may be a global variable, set outside the routine

5. ud use-define

Allowed: data used and then re-defined

6. uk use-kill Allowed

7. ~k first kill Potential bug; data is killed before definition

8. ku kill-use Serious defect; data is used after being killed


22

Data-flow Testing Combinations (2)

Anomaly Explanation9. kd kill-define Usually allowed. Data is killed and

then re-defined. Some theorists believe this should be disallowed

10.

dd define-define

Potential bug; double definition

11.

uu use-use Allowed; normal case. Some do not bother testing this pair since no redefinition occurred

12.

kk kill-kill Potential bug

13.

d~ define last Potential bug; dead variable? May be a global variable which is used in other context

14.

u~ use last Allowed. Variable was used in this routine but not killed off

15.

k~ kill last Allowed; normal case


23

Function to Test1. public static double calculateBill (int Usage) {

2. double Bill = 0;

3. if(Usage > 0) {

4. Bill = 40;

5.

6. if(Usage > 100) {

7. if(Usage <= 200) {

8. Bill = Bill + (Usage - 100) * 0.5;

9. }

10. else {

11. Bill = Bill + 50 + (Usage - 200) * 0.1;

12. if(Bill >= 100) {

13. Bill = Bill * 0.9;

14. }

15. }

16. }

17. }

18. return Bill;

19. }


24

Variable DUK Patterns

Pattern Explanation

1.

~d (1) normal case

2.

du (1-3) normal case

3.

uu (3-6)(6-7)(7-8)(7-11)

normal case

4.

uk (6-19)(8-19)(11-19)

normal case

5.

k~ (19) normal case

Pattern Explanation

1.

~d (2) normal case

2.

dd (2-4) suspicious

3.

du (2-18)(4-8)(4-11)(11-12)

normal case

4.

ud (8-8)(11-11)(13-13)

acceptable

5.

uu (12-13)(12-18) normal case

6.

uk (18-19) normal case

7.

k~ (19) normal case

Usage data-flow information

Bill data-flow information


25

SummaryMany testers simply use black-

box and experience-based techniques

Some systems require far deeper testing

This presentation barely touches the many advanced techniques that can be used in testing

Learning these techniques can give you an advantage in your career

Structural Testing: When Quality Really Matters

Technology

Transcript of Structural Testing: When Quality Really Matters