Prescriptive or Exploratory Testing: Which Is More

Effective?

Tafline Murnane,

K.J. Ross & Associates, Australia

Europe’s Premier Software Testing Event

Stockholmsmässan, Sweden

WWW.EUROSTARCONFERENCES.COM

“Testing For Real, Testing For Now”

Prescriptive or Exploratory Testing:

Which is More Effective?

Tafline MurnaneK. J. Ross & Associates Pty. Ltd.

PO Box 131, West Burleigh, 4219, QLD, Australia

M: +61 416 117 931 F: 1300 854 064

taflinem@kjross.com.au

http://www.kjross.com.au

Boeing Australia

3

“It can be frustrating, as a tester, to watch someone

who has no experience in testing spend five

minutes using a piece of software and crash it.”

– Ron Patton, Software Testing (2006)

Boeing Australia

4 4

About me…

• Tafline Murnane has worked for the past 3 years as a senior consultant

for K. J. Ross & Associates in Australia (a specialist testing consultancy)

• Previously worked as a developer (before moving to the dark side…)

and as a lecturer and tutor in software testing

• Co-editor of the new ISO/IEC 29119 Software Testing standard

• Completing a PhD in software testing

• Presenter at software testing and engineering conferences both

nationally and internationally

• And is rather passionate about testing…

Melbourne

Boeing Australia

5

Q. How do professional testers perform

black-box testing?

• Prescriptively, using specification-based techniques? e.g.

– Equivalence Partitioning (EP)

– Boundary Value Analysis (BVA)

– Syntax Testing (ST)

• Non-prescriptively, using „experienced-based‟ approaches? e.g.

– Error Guessing

– Exploratory Testing

Boeing Australia

6

Experiment aim

• Compare effectiveness of industry approaches to black-box

testing to „Atomic Rules‟ representation of EP, BVA and ST

– Atomic Rules approach is a very prescriptive representation of EP,

BVA and ST (see following slide…)

• Effectiveness analysed in terms of:

– Usability

• Test technique coverage

• Understanding of test techniques

– Defect-detection percentage

• Percentage of „known‟ defects found

Boeing Australia

7

Defining EP, BVA and ST more prescriptively

through Atomic Rules

• Traditional black-box testing technique descriptions

can be ambiguous

• e.g., the following is an extract from Glenford Myers‟,

The Art of Software Testing (1979):

“If an input condition specifies a „must be‟ situation „first

character of the identifier must be a letter‟, identify one

valid equivalence class (it is a letter) and one invalid class

(it is not a letter)”„is a letter‟ is easy

A | B | C…What constitutes

„not a letter‟???

Boeing Australia

8 8

The Atomic Rules Approach

Equivalence Partitioning

• “Atomic Rules” definition for Equivalence Partitioning:

– EP1: less than lower boundary selection

– EP2: greater than upper boundary selection

– EP3: lower to upper boundary selection

– EP4: integer replacement

– EP5: real number replacement

– EP6: single alpha replacement

– EP7: multiple alpha replacement

– EP8: multiple alphanumeric replacement

– EP9: single non-alphanumeric replacement

– EP10: multiple non-alphanumeric replacement

– EP11: missing item replacement

– EP12: valid list selection

“is a letter”

tested by list

selector rule

“not a letter”

tested by invalid

datatype rules

Tests ranges

Boeing Australia

9 9

The Atomic Rules Approach

Boundary Value Analysis & Syntax TestingBoundary Value Analysis

• BVA1: Lower Boundary - Selection

• BVA2: Lower Boundary Selection

• BVA3: Lower Boundary + Selection

• BVA4: Upper Boundary - Selection

• BVA5: Upper Boundary Selection

• BVA6: Upper Boundary + Selection

• BVA7: First List Item Selection

• BVA8: Last List Item Selection

• BVA9: Missing Item Replacement

Syntax Testing

• ST1: Remove last character

• ST2: Replace last character

• ST3: Add extra character to end

• ST4: Remove first character

• ST5: Replace first character

• ST6: Add extra character to start

• ST7: Uppercase a lowercase letter

• ST8: Lowercase an uppercase letter

• ST9: Null all input

• ST10: Duplicate field

• ST11: Add a field

• ST12: Select each list alternative

• ST13: Select all list alternatives

• ST14: Select all list alternatives in reverse order

The lists of „Atomic Rules‟ act as a

checklist (i.e. Quick Reference

Guide) during black-box testing

BVA1 to BVA6 select values just on, just above and just below the boundaries of equivalence classes (see notes)

Boeing Australia

10 10

Attribute Atomic Rule

Test Method Equivalence Partitioning

Identifier EP1

Name Less than Lower Boundary Selection

DescriptionSelects an equivalence class containing values that lie

below the lower boundary of a field

Source [Myers79]

Rule Type Data-Set Selection Rule (DSSR)

Set Type Range

Valid or Invalid Invalid

Original Datatype Integer, Real, Alpha, Non-alphanumeric

Test Datatype Same as original

Test Data Length Same as original

# Fields Populated 1

# Tests Derived 0

Examples …

The Atomic Rules Schema

• Each Atomic Rule is defined in a characterisation schema

Boeing Australia

11 11

Two Previous University Experiments

Myers‟

representation

of EP & BVA

Atomic Rules

representation

of EP & BVA

Compare learnability & usability

• Limitations:

- involved only novice testers, not industry practitioners

- test cases designed for fictional specifications, thus defect-

detection effectiveness could not be assessed

• Nonetheless, these university experiments provided good

preparation for the industry experiment…

Boeing Australia

12

The Industry Experiment

“Practitioner”

approach to

black-box

testing

Compare usability &

defect-detection effectiveness

Atomic Rules

representation

of EP, BVA &

ST

Boeing Australia

13 13

Two Day Experiment

11 participants5 test leads

5 testers

1 manager new to testing

Average 2.5 years experience in testing

Programs tested Address Parser

Batch Processor

Data collectedTest cases

Defect reports

Feedback on participant experiences

Materials providedSlides

Quick reference guide

Boeing Australia

14 14

Group 2

Atomic Rules Testing of

Address Parser

Experiment Plan

Introductory Presentation

Group 1

Practitioner Testing of

Address Parser

Group 2

Practitioner Testing of

Batch Processor

Atomic Rules Presentation

Group 1

Atomic Rules Testing of

Batch Processor

Wrap-Up Presentation

Each participant

tested a different

program during both

phases of testing

Boeing Australia

15 15

Two Programs Tested

The Address Parser

100 Main Street Melbourne 3000.

FLAT 002 , 100 High St Sydney 2000.

UNIT 014 / 220 Main St North Ryde 2113.

C/o High Street North North Sydney 2000.

C/- St Helena Road Greensborough 3088.

The Batch Processor

sbatch 11BBB

A11A 10

ebatch 11BBB

sbatch 22ABC

B22B -10

C99E 20, A11A -40

ebatch 22ABC

lbatch

Expected the Address Parser to

be a familiar conceptExpected the Batch Processor to

be an unfamiliar concept

Boeing Australia

16 16

Research Questions

1. How did the practitioners perform black-box testing?

2. How many defects did they detect?

3. How many defects were detectable by Atomic Rules?

4. Were the results affected by domain knowledge?

5. Were the results affected by tester experience?

6. Were the results affected by current job?

7. What was their understanding of EP, BVA & ST?

• Standard tests for significance were used– result is statistically significant if it is unlikely

to have occurred by chance

– using t-tests, chi-square tests, etc.

• Threats to validity were thoroughly explored

Boeing Australia

17 17

1. How did the practitioners perform

black-box testing?Q. How would you explain your approach to test case design on day 1?

1

1

1

1

2

3

3

4

7

0 1 2 3 4 5 6 7 8

Random

Duplication / addition

Syntax

Output

Boundary

Experience-based / ad hoc

Valid

Invalid

Specification-based

Self-e

xpla

nation

Number of Testers

One participant also developed

a random number generator in

MS Excel!

Despite these responses

from participants, test

case design patterns

indicated that all

participants carried out

Exploratory Testing

Boeing Australia

18 18

1. How did the practitioners perform

black-box testing? (continued…)

Technique Approach

Technique

Coverage

Std

Dev

Significant

Difference?

EPPractitioner 13% 3.6

NoAtomic Rules 14% 8.6

BVAPractitioner 3% 1.7

YesAtomic Rules 9% 6.8

STPractitioner 2% 0.8

YesAtomic Rules 9% 10

Out of 501 tests derived during

„practitioner‟ testing, only eight (1.6%)

could not be derived by Atomic Rules

from EP, BVA and ST

Low % coverage, as large quantity of

test cases needed to be designed to

achieve 100% coverage (i.e. required

more than 3.5 hours for test design)

Q. How did test technique coverage compare between day 1 and 2?(number tests derived / total number tests derivable by the technique)

Boeing Australia

19 19

2. How many defects did they detect?

• Practitioner approach: 25%

• Atomic Rules approach: 35%

(Statistically significant difference was found)

The „prescriptive‟

Atomic Rules

approach was

more effective

=Defect Detection

Percentage (DDP)number faults detected

total number known faults

Given more time, higher

percentages may have been

achieved for both prescriptive

and non-prescriptive testing

Boeing Australia

20 20

• Address Parser

– EP 92%

– BVA 67%

– ST 67%

• Batch Processor

– EP 54%

– BVA 58%

– ST 67%

3. How many defects were detectable by

the Atomic Rules approach?• Percentage of defects detectable was identified by applying all Atomic Rules

from EP, BVA and ST to all applicable input fields

• When various test data values could be selected (e.g. selecting a value from

an equivalence class during EP), „mid-point‟ values were chosen

What program characteristics

make a black-box testing

technique more (or less)

effective?

High DDP for EP! Was it due to

Atomic Rules representation of EP

being more thorough than Myers‟?

Or was it caused by the types of

defects present?

Boeing Australia

21 21

4. Were the results affected by domain

knowledge?

Approach Program Defects Detected

Std

Dev

Significant

Difference?

PractitionerAddress Parser 34% 4

YesBatch Processor 18% 6

Atomic RulesAddress Parser 30% 8

NoBatch Processor 40% 15

Results suggest that testers either need

domain knowledge OR a prescriptive

testing technique to be effective

(i.e. Atomic Rules fills the gap)

• After the experiment, the testers explained they were more familiar with

the Address Parser, as they regularly tested addresses as part of their jobs

and since it was based on real-world concepts

• Did this have an impact on percentage of defects detected?

Significant difference

found during Practitioner

testing, but not during

Atomic Rules testing

Boeing Australia

22 22

5. Were results affected by years of

experience in testing?

• Compared percentage defects detected and test technique

coverage by years of experience

Approach n Experience

Technique

Coverage

Std

Dev

Sig

Diff?

Defects

Detected

Std

Dev

Sig

Diff?

Practitioner6 < 2 years 7% 1.5

No22% 10

No5 2 – 10 years 6% 1.7 28% 10

Atomic

Rules

6 < 2 years 9% 5.8No

35% 8No

5 2 – 10 years 13% 6.3 35% 18

Number of years of testing experience did

not affect defect-detection effectiveness or

test technique coverage

No significant differences

were found

No significant differences

were found

Boeing Australia

23 23

6. Were results affected by the tester‟s

current job?• Compared percentage defects detected and test technique

coverage by current job

Approachn Current Job

Technique

Coverage

Std

Dev

Sig

Diff?

Defects

Detected

Std

Dev

Sig

Diff?

Practitioner5 Test Lead 5% 1.8

No19% 0.1

Yes5 Tester 7% 1.4 30% 0.1

Atomic

Rules

5 Test Lead 10% 6.4No

42% 0.2No

5 Tester 11% 7.2 30% .1

Did the Atomic Rules approach fill

the gap in recent test practice for

the Test Leads?

No significant differences

were foundOnly significant difference found

was Defects Detected during

Practitioner testing

Boeing Australia

24 24

7. What was their understanding of the

techniques?• Did their understanding of EP, BVA and ST improve after learning the Atomic

Rules representation of these techniques?

Understanding of Black-Box Testing Techniques

9

8

9

3

2

3

2

2

2

5

4

5

0

1

0

3

5

3

0

0

0

0

0

0

0 1 2 3 4 5 6 7 8 9 10 11

EP Initial

BVA Initial

ST Initial

EP Final

BVA Final

ST Final

Number of Testers

Novice Intermediate Advanced Expert

Results indicate that

their understanding

significantly improved

Boeing Australia

25 25

Other Findings

• Three participants (27%) felt the Quick Reference Guide

was useful for checking test coverage

• Five participants (46%) felt matrices (mapping input fields

to Atomic Rules) would have supported test design during

Atomic Rules testing (for planning & coverage)

• Eight participants (73%) felt participation would impact on

how they perform black-box testing in future

• All enjoyed the challenge of learning a new approach to

black-box testing and/or found the

Atomic Rules approach „interesting‟

Boeing Australia

26

Recommendations for Industry

• Exploratory testing is useful and can be effective!

– Useful for gaining familiarity with a system

– Can be as effective as prescriptive testing, particularly when testers

have domain knowledge in the system

– Training in Exploratory Testing could significantly improve abilities

• Prescriptive black-box testing techniques have their place!

– Can be more effective than Exploratory Testing, particularly when

testers do not have domain knowledge in a system

– Important to have precise definitions of techniques like EP, BVA

and ST, to ensure testing is thorough and auditable

– Testers in industry do enjoy learning and using

prescriptive approaches to black-box testing

Boeing Australia

27

Future Work

• Since the experiment involved a limited number of testers, it would be ideal to repeat the experiment with more practitioners & industry-developed programs

• Additional aspects to assess:

– Why are particular black-box testing techniques more effective against certain types of programs?

– How important is domain knowledge in developing complete and effective test sets?

– Do practitioners retain knowledge they gain through black-box testing training?

– Is automation of test data generation useful for industry?

• e.g. populate parameters in automated test suites that are derived through other means, such as model-based testing or keyword-driven frameworks

Boeing Australia

28

Thank you for your interest in this presentation!

Questions?

Many thanks to the participants of this experiment and to K. J. Ross & Associates & La Trobe University

for their support of this research

Tafline MurnaneK. J. Ross & Associates Pty. Ltd.

PO Box 131, West Burleigh, QLD, 4219

Australia

P: 1300 854 063 F: 1300 854 064

M: +61 416 117 931

taflinem@kjross.com.au

http://www.kjross.com.au