White-box Testing: When Quality Really Matters

15
T14 Test Techniques 5/2/2013 1:30:00 PM White-box Testing: When Quality Really Matters Presented by: Jamie Mitchell Jamie Mitchell Consulting, Inc. Brought to you by: 340 Corporate Way, Suite 300, Orange Park, FL 32073 888-268-8770 ∙ 904-278-0524 ∙ [email protected] www.sqe.com

description

Jamie Mitchell explores perhaps the most underused test technique in our arsenal—white-box testing. Also known as structural testing, white-box requires some programming expertise, access to the code, and an analysis tool. If you only employ black-box testing, you could easily ship a system having tested only 50 percent or less of the code base. Not good! Although you might believe that the developers have performed sufficient unit and integration testing, how do you know that they have achieved the level of coverage your project requires? Jamie describes the levels of code coverage that the business and your customers may need—from statement coverage to modified condition/decision coverage. Leading you through examples of pseudocode, Jamie explains when you should strive to achieve different code coverage target levels. Even if you have no personal programming experience, understanding structural testing will make you a better tester. So, join Jamie for some code-diving!

Transcript of White-box Testing: When Quality Really Matters

Page 1: White-box Testing: When Quality Really Matters

T14 Test Techniques

5/2/2013 1:30:00 PM

White-box Testing: When Quality Really

Matters

Presented by:

Jamie Mitchell

Jamie Mitchell Consulting, Inc.

Brought to you by:

340 Corporate Way, Suite 300, Orange Park, FL 32073

888-268-8770 ∙ 904-278-0524 ∙ [email protected] ∙ www.sqe.com

Page 2: White-box Testing: When Quality Really Matters

Jamie Mitchell

Jamie Mitchell has more than thirty years of experience developing and testing both hardware and software. In 1991, Jamie moved from hardware to the dark side - software. He was a pioneer in the test automation field, working with a variety of vendor, open source, and custom-built tools since Windows 3.0. Jamie's specialty is increasing the productivity of automation and test teams through innovative ideas and custom tool extensions. In addition, Jamie provides training, mentoring, and expert technical support in all aspects of testing and automation. Jamie is the coauthor (with Rex Black) of Advanced Software Testing, Volume 3: Guide to the ISTQB Advanced Certification as an Advanced Technical Test Analyst.

Page 3: White-box Testing: When Quality Really Matters

4/19/2013

1

Structural TestingStructural TestingWhen Quality Really Matters

Jamie L Mitchell

Jamie Mitchell Consulting Inc.

When Software Really Needs to WorkWhen Software Really Needs to Work

Copyright (c) Jamie Mitchell Consulting Inc. 2

Page 4: White-box Testing: When Quality Really Matters

4/19/2013

2

Why StructureWhy Structure--based Testing?based Testing?

� Experience-based techniques are not thorough enough:◦ Error-guessing◦ Exploratory testing◦ Defect- and taxonomy-based

� Black-box testing depends on how good your test basis model is:◦ Requirements◦ Use cases◦ Functional specifications

� How well have you tested?

Is testing less than half of your code sufficient?

3Copyright (c) Jamie Mitchell Consulting Inc.

WhiteWhite--Box Test BasisBox Test Basis

� Design documents and models

◦ Architecture

◦ Functional design

◦ Low-level design

� Object models

� Class interface documents

� Code

Copyright (c) Jamie Mitchell Consulting Inc. 4

Page 5: White-box Testing: When Quality Really Matters

4/19/2013

3

Today’s Discussion TargetsToday’s Discussion Targets

� Control-flow analysis: selecting certain paths through the software to test, based on desired coverage

� Data-flow analysis: a control-flow technique that scrutinizes the life-cycle of data variables

Copyright (c) Jamie Mitchell Consulting Inc. 5

ControlControl--flow Testingflow Testing

� Select certain paths through the software

◦ Based on desired level of coverage

� Based on statements

� Based on decisions

� Based on loops

� Based on paths

Copyright (c) Jamie Mitchell Consulting Inc. 6

Page 6: White-box Testing: When Quality Really Matters

4/19/2013

4

Statement CoverageStatement Coverage

� Execute every line of code

� Tools available for measurement

◦ Run tool while executing tests

◦ Evaluate coverage

◦ Create more tests to reach areas missed

� Also called instruction or code coverage

Copyright (c) Jamie Mitchell Consulting Inc. 7

Decision CoverageDecision Coverage

� Statement coverage may miss bugs◦ One test needed for statement coverage

(a = 5, b = 4) � Passes

z = 0;

if (a > b) then

z = 12;

x = 72 / z

� Select inputs to force each decision to execute both possible ways (T/F)◦ Now two test cases are needed for coverage

(a = 5, b = 4) � Passes

(a = 4, b = 5) � Finds bug

Copyright (c) Jamie Mitchell Consulting Inc. 8

Page 7: White-box Testing: When Quality Really Matters

4/19/2013

5

Condition CoverageCondition Coverage

� Decisions may be arbitrarily complex

if (a > b) then…

if (A || B) && (C == D) || (!E) && (F) then…

� Condition testing requires that each atomic condition must be evaluated both true and false

� Note that condition coverage – by itself – may not be as strong as decision coverage

if (a AND b) then

(a = F, b = T � F)

(a = T, b = F � F)

Copyright (c) Jamie Mitchell Consulting Inc. 9

Decision / Condition CoverageDecision / Condition Coverage

� A combination of Condition and Decision1. Each atomic condition must be evaluated

both ways AND

2. Decision coverage must be satisfied

if (a AND b) then

(a = F, b = T � F)

(a = T, b = F � F)

(a = T, b = T �T)

� Is that enough coverage?

Copyright (c) Jamie Mitchell Consulting Inc. 10

Page 8: White-box Testing: When Quality Really Matters

4/19/2013

6

Modified Condition / DecisionModified Condition / Decision

� Usually called MC/DC◦ For each condition in the predicate, there is at least one test where the outcome is TRUE only because of that condition being TRUE◦ For each condition in the predicate, there is at least one test where the outcome is FALSE only because of that condition being FALSE

� Usually results in N+1 test cases where N is number of conditions in predicate

� Required by some safety standards to ensure that enough testing has been done

Ex: FAA DO/178B Level A (catastrophic) software

Copyright (c) Jamie Mitchell Consulting Inc. 11

MC/DC Coverage (2)MC/DC Coverage (2)

if ((a || b) && c) then …

� Decision/Condition can be achieved with

(a = T, b = T, c = T) �T

(a = F, b = F, c = F) � F

� Note that (b) never independently affects the outcome of the condition

Copyright (c) Jamie Mitchell Consulting Inc. 12

Page 9: White-box Testing: When Quality Really Matters

4/19/2013

7

MC/DC Coverage (3)MC/DC Coverage (3)

if ((a || b) && c) then …

� Following test set will achieve MC/DC coverage

1. (a = T, b = F, c = T) �T

2. (a = F, b = F, c = T) � F

3. (a = F, b = T, c = T) �T

4. (a = T, b = F, c = F) � F

Copyright (c) Jamie Mitchell Consulting Inc. 13

Coverage Based on LoopsCoverage Based on Loops

� Loops make testing …interesting…

� The number of possible paths may approach infinite depending on how many loops are executed

� Basic coverage criteria

◦ Test when no loops are made

◦ Test when a single loop is made

◦ Test when n loops are made (where n is either the expected max or a large number)

Copyright (c) Jamie Mitchell Consulting Inc. 14

Page 10: White-box Testing: When Quality Really Matters

4/19/2013

8

Loops (2)Loops (2)

� This loop can calculate a factorial (n!)

for (i = 1; i <= n; i++)

f *= i;

� Use (n = 0) for zero loops

� Use (n = 1) for one time through

� Determining the max requires thought

◦ In this case, (f) gets very large

◦ Use (n = 12); 13 would cause an overflow

Copyright (c) Jamie Mitchell Consulting Inc. 15

Nested LoopsNested Loops

� Nested loops are really hard to test

� Boris Beizer* had some advice1. Start at the innermost loop, setting all outer loops to

their minimum iteration setting

2. Test boundaries for innermost loop while keeping outer loops at their minimum

3. If you have done outermost loop goto 5. Else move one loop outward and test as in step 2 with all inner loops set to their typical values

4. Continue outward until all loops done

5. Test the boundaries for all loops simultaneously

* Software Testing Techniques, 2nd Ed. by Boris Beizer

Copyright (c) Jamie Mitchell Consulting Inc. 16

Page 11: White-box Testing: When Quality Really Matters

4/19/2013

9

Path TestingPath Testing

� Full path testing in a non-trivial system is impossible◦ The number of paths approaches infinite◦ This is due to loops as seen previously

� We can identify independent, non-looping paths which cover all the edges and nodes of a control-flow graph◦ These are called basis paths◦ Testing the basis paths can guarantee decision (and statement) coverage◦ McCabe cyclomatic complexity uses this method◦ Tools are available for calculating this value

Copyright (c) Jamie Mitchell Consulting Inc. 17

Cyclomatic Complexity ExampleCyclomatic Complexity Example

� Complexity = 3

� Tests needed

◦ 1-2-6-7-8

◦ 1-3-4-7-8

◦ 1-3-5-8

Example from “A Complexity Measure” by Thomas McCabe, IEEE Transactions on SW Engineering, Vol. SE-2, No 4, December 1976

Copyright (c) Jamie Mitchell Consulting Inc. 18

Page 12: White-box Testing: When Quality Really Matters

4/19/2013

10

DataData--flow Analysis (1)flow Analysis (1)

� Programs create, set, read, evaluate and destroy data

� The life cycle of data can be analyzed to find defects that control-flow techniques miss

� Some defect examples◦ Incorrect assignment of a value◦ Incorrect input statement ◦ Failure to define a variable before use◦ Incorrect path taken due to incorrect value in a control predicate ◦ Use of a variable after it is destroyed or out of scope◦ Redefinition of a variable before it can be used.

� Look for consecutive touches to each variable

Copyright (c) Jamie Mitchell Consulting Inc. 19

DataData--flow Analysis (2)flow Analysis (2)

� Dependent on the language used

� We look for anomalies in data life-cycle ◦ Not all anomalies are defects

◦ While this is nominally a static technique, it will not find all data defects (often data is manipulated or assigned dynamically)

� Sometimes called DUK testing◦ D � Define

◦ U � Use

◦ K � Kill

Copyright (c) Jamie Mitchell Consulting Inc. 20

Page 13: White-box Testing: When Quality Really Matters

4/19/2013

11

DataData--flow Testing Combinations (1)flow Testing Combinations (1)

Copyright (c) Jamie Mitchell Consulting Inc. 21

Anomaly Explanation

1. ~d first define Allowed

2. du define-use Allowed, normal case

3. dk define-kill Potential bug; data was never used

4. ~u first use Potential bug; data was used without definition. It may be a global variable, set outside the routine

5. ud use-define Allowed: data used and then re-defined

6. uk use-kill Allowed

7. ~k first kill Potential bug; data is killed before definition

8. ku kill-use Serious defect; data is used after being killed

DataData--flow Testing Combinations (2)flow Testing Combinations (2)

Copyright (c) Jamie Mitchell Consulting Inc. 22

Anomaly Explanation9. kd kill-define Usually allowed. Data is killed and then re-

defined. Some theorists believe this should be disallowed

10. dd define-define Potential bug; double definition

11. uu use-use Allowed; normal case. Some do not bother testing this pair since no redefinition occurred

12. kk kill-kill Potential bug

13. d~ define last Potential bug; dead variable? May be a global variable which is used in other context

14. u~ use last Allowed. Variable was used in this routine but not killed off

15. k~ kill last Allowed; normal case

Page 14: White-box Testing: When Quality Really Matters

4/19/2013

12

Function to TestFunction to Test1. public static double calculateBill (int Usage) {

2. double Bill = 0;

3. if(Usage > 0) {

4. Bill = 40;

5.

6. if(Usage > 100) {

7. if(Usage <= 200) {

8. Bill = Bill + (Usage - 100) * 0.5;

9. }

10. else {

11. Bill = Bill + 50 + (Usage - 200) * 0.1;

12. if(Bill >= 100) {

13. Bill = Bill * 0.9;

14. }

15. }

16. }

17. }

18. return Bill;

19. } Copyright (c) Jamie Mitchell Consulting Inc. 23

Variable DUK PatternsVariable DUK Patterns

Copyright (c) Jamie Mitchell Consulting Inc. 24

Pattern Explanation

1. ~d (1) normal case

2. du (1-3) normal case

3. uu (3-6)(6-7)(7-8)(7-11)

normal case

4. uk (6-19)(8-19)(11-19)

normal case

5. k~ (19) normal case

Pattern Explanation

1. ~d (2) normal case

2. dd (2-4) suspicious

3. du (2-18)(4-8)(4-11)(11-12)

normal case

4. ud (8-8)(11-11)(13-13) acceptable

5. uu (12-13)(12-18) normal case

6. uk (18-19) normal case

7. k~ (19) normal case

� Usage data-flow information

� Bill data-flow information

Page 15: White-box Testing: When Quality Really Matters

4/19/2013

13

SummarySummary

� Many testers simply use black-box and experience-based techniques

� Some systems require far deeper testing

� This presentation barely touches the many advanced techniques that can be used in testing

� Learning these techniques can give you an advantage in your career

Copyright (c) Jamie Mitchell Consulting Inc. 25