Mattias Ratert - Incremental Scenario Testing

Incremental Scenario Testing:Beyond Exploratory Testing

ww

w.t

ele

ca.c

om

Matthias Ratert

Incremental Scenario Testing: Beyond Exploratory Testing 2

Background

Teleca is a world-leading supplier of software services and solutions to the mobile industry

Global capability to deliver packaged software solutions,systems design, integration and testing

Teleca was developing a complex internet service fora top 5 mobile OEM

500kLOC, large number of handsets, operators, target markets complex multi-backend with e-commerce, operator integration

High functional compliancy reached ... but due to the complexity the quality was poor


SW functionality to

be tested

Software complexitySW function range

Cross-functionalit

yHW / SW

configuration

Memory situation

Data in- & outputEnvironme

nt variations

Network

User interactio

nHW

variations

SW customizati

on

SW re-usage

Distributed developme

nt

Distributed systems


Test cases

Errors

Regression TestingSW function range

Error fixes

Existing errors...to be exposed

Regression Tests need complementation


Be part of the overall test strategy

Find as many errors as possible

Generate test cases within the test session

Challenge the tester Work without detailed test case descriptions Allow creativity, force own ideas and motivate to be pro-

active Own judgment of the test result (no expected behavior)

Utilize the feedback from previous test sessions

Exploratory Testing was not successful for us:Difficult to come up with new and creative test ideasToo many areas remained untested

Exploratory Testing


Incremental Scenario Testing (IST)

IST controls the complexity

IST guides & inspires the tester with Scenarios Encourage the tester to explore new functionality

IST automates the test planning Optimize the utilization of the available testers and test time Indentify SW areas not tested ever before

IST focuses the testing on new, modified & weak SW areas Provide an interface for collaboration between Test and SW

Team

IST increases the visibility of all activities

IST provides a web-based tool to offer the features above The Incremental Scenario Testing Tool (ISTT)


Test Scenario

High level test case description built out ofthree test items:

1. Precondition

2. State

3. Event

How to reach each test itemis up to the tester This assures additional variation

for usage, input data, timing etc.

To be understood as a guideline The tester is free to experiment and

to do additional things

State

Precondition

Event


Test items forming a Scenario

Precondition: Everything influencing the SW to be tested Dependent or independent of the functionality to be

tested The amount of preconditions is flexible from 0 – 5

State: All states and functionalities to be tested May be linked to requirements or specifications

Event: An action with internal or external influence Dependent or independent of the functionality to be

tested

Execu

tion fl

ow

Test items are grouped by test categories

NOTE: Testing can only be as complete as its test data


Test environment and setup for one tester

Hardware, Operating System, Configuration, Language, ...

Test Scope

The Test Scope won‘t be changed within the test session

All testers might get the same Test Scope

...or each tester might get a different Test Scope


SW complexity example: Voice calls

Outgoing calls (MO)

Incoming calls (MT)

States

Scopes, Preconditions & Events

User interactio

n

New MO / MT

callsOS /

Phone / Language

Battery / Headset

Call settings / Contacts /Call listsRAM / C:

status

Network coverage

Operator2G / 3G


Voice call complexity in numbers

How to ensure that the important Scenarios are selected?

Scopes cannot be

counted easily,

e.g. number of

devices supported.

Preconditions >50

States >40

Events >50


Test Scenario generation

Test itemhistory

Failed ShowstopperFailed Critical

Failed MajorFailed MinorNot tested(Passed)

Scenariohistory

Test itemfocus

Selectedto be

tested?

Scenario calculation based the Test Session Configuration

Test itemweight

OftenRegularlySporadicRarely

(Unusual)

Prefer the newor changed functionality

Test Scenario to be tested

Prefer the common

functionality (items)

Prefer the riskyand not tested functionality

(items)

Prefer the riskyand not tested

Scenarios


Scope: Nokia N96, OS 3.1,

English, T-Mobile

Scope: iPhone, OS 3.0, German, Vodafone

Voice call Scenario examples

P1: Headset connectedP2: Sceensaver = ON

P3: Profile = Silent

S: MT call ringing

E: Lose GSM network

P: Battery almost empty

S: MO call via address

book

E: Plug in charger

P1: RAM almost fullP2: UMTS active

S: MO call via speed dial

E: MT call ringing

P1: Sending of caller ID = ON

P2: Summary after call = ON

S: MO call via SMS is

ringing

E: Calendar alarm expires


Test sessions

Test sessions are individual Every tester can start at any time independent of each other Every tester can pause or stop at any time Everybody is able to see the progress and results at any time

Test sessions are flexible Test as much as possible within a given time box Test as long as testers are available (without a deadline)

Test sessions can be distributed globally

Test sessions are stopped by the Test Manager Updating the SW will stop the test session and start a new

one The testers are notified after finishing the current Scenario


Further add-ons

Ranking system (Junior Tester, Tester, Senior Tester)The rank depends on the experience within the IST projectShall motivate the tester and gives the Test Manager confidence when analyzing the test results

Re-Testing: Momentous test results, especially „Failed“ & „Impossible“, need evidence in the same sessionPreferable done by a tester with a higher rankSupports the Test Manager within the test result analysis

Developer Role: Everyone can influence the testingAdding of new test itemsMarking test items as „To be tested“


Possible scenario (here 40)

Possible scenario

Test focus: „To be tested“

Test Scenario spectrum 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40

Test session kick-off

Test session

Possible scenario


Scenario to be tested

First scenarios

are selected by

test focus

and weights


Test Scenario spectrum 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40

Test Scenario results

Test session ?

?

PASS

Failed

Impossible

Skipped

Not clear

Possible scenario


Scenario (to be) tested

Scenario (to be) re-tested

Impossible scenario


Test session evolutionTest Scenario spectrum

Test session 1

(Increment 1)

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40

?

Test session 2

(Increment 2)

?

41 42 43 44 45

PASS

Failed

Impossible

Skipped

Not clear?

46

Test session 6

(Increment 6)

47 48 49 50

Test session 5

(Increment 5)

?

Test session 4

(Increment 4)

?

Test session 3

(Increment 3)


Case study

Project period Mar/06 – Mar/09

IST usage period Nov/07 – Mar/09 (17 months)

Performed test sessions 27

Duration of test sessions in days1 – 38 (average: 4,8 days)

Possible Test Scenarios (1 precondition) 1.274.400Possible Test Scenarios (2 preconditions)

50.976.000

Executed Test Scenarios 10.219

New severe error reports 476


Our main achievement

Time

Internal errors found

Err

ors

Release to

customer

Internal + customer errors found

Internal + customererrors found

Release to

customer

Time

Internal errors found

plus IST

Err

ors


The football trainer

Scopes:TeamLeague /TournamentRules

Preconditions:Position/rankTeam conditionOpponentVenueWeather

Events:Goal InjuryYellow cardRed card

States:ScoreTime left

IST is suitable for any complex system

Matthias RatertTeleca GermanyRensingstr. 1544807 [email protected]

Thank you for your attention!

Please visit us at our stand no. 52

for a demo session & further discussions

Mattias Ratert - Incremental Scenario Testing

Software

Transcript of Mattias Ratert - Incremental Scenario Testing