SAST 2002-09-19 Managing ET

8/3/2019 SAST 2002-09-19 Managing ET

1/55

SAST 2002-09-19Managing ET & Case 2002 Amland Consulting 1

Managing Exploratory Testing

Parts of this presentation have been excerpted from:Rapid Software Testing, by James Bach,[email protected]

Black Box Software Testing, by Cem Kaner, [email protected]

High Speed WEB Testing, by Jarle Vga,[email protected]

Stle Amland, Amland Consulting,

Hulda Garborgsv. 2, 4020 STAVANGER, NORWAY

Mob:+47 905 28 930, Phone: +47 51580587, Fax: +4751585524

e-mail: [email protected] WEB: www.amland.no

SAST meeting September 19, 2002, Stockholm, Sweden
mailto:[email protected]:[email protected]:[email protected]:[email protected]://www.amland.no/http://www.amland.no/mailto:[email protected]:[email protected]:[email protected]:[email protected]

8/3/2019 SAST 2002-09-19 Managing ET

2/55


Software Testing:

There is artistry. There is craft. There isscience. It is all threeas with all

technologies.

If you understand the science and havemastered the craft, then it is meaningful toinvestigate, espouse, and perfect the art.

Boris Beizer

8/3/2019 SAST 2002-09-19 Managing ET

3/55


Software Testing:

Testing is the process of executing a programwith the intent of finding errors.

Testing is an extremely creative andintellectually challenging task.

Glenford Myers inThe Art of Software Testing

8/3/2019 SAST 2002-09-19 Managing ET

4/55


Presentation Outline

What is Exploratory Testing?

How to do Exploratory Testing?

How to Manage Exploratory Testing?

High Speed WEB Testing Case Study

8/3/2019 SAST 2002-09-19 Managing ET

5/55


Different testing approachesSkeptical approaches

Analytical Approaches

Information-driven approaches

Time-honored but less effective approaches

Experiential and intuitive approaches:Lets think blue-sky, speculate and follow our intuition.

We have good hunches about where the bugs are lurking.

Lets jump in an explore the systems behavior hands-on, sowe can decide how to test it.

Lets find the important bugs fast, and worry about the testpaperwork later.

Ross Collard (2002)

8/3/2019 SAST 2002-09-19 Managing ET

6/55


What is Exploratory Testing?"Exploratory testing involves simultaneously learning,

planning, running tests, and reporting / troubleshootingresults."

Dr. Cem Kaner (2001)

"Exploratory testing is an interactive process of concurrentproduct exploration, test design and test execution.

To the extent that the next test we do is influenced bythe result of the last test we did, we are doing

exploratory testing.

James Bach, Satisfice (2001)
http://www.kaner.com/

8/3/2019 SAST 2002-09-19 Managing ET

7/55


Said about eXtreme Programming

Agile software development is not conventional software developmentdone more quickly or done on tippie-toe. Agile software development issoftware done differently.

Ron Jeffries, (e-mail on agile-testing list, April 24, 2002)

proven (no single technique is new)application oriented

planned and disciplined

controllable and reliable

risk minimizingTwo sides of extreme programming:

for the developer: freedom, flexibility, fun

for the manager: controllability, reliability, high qualityMartin Lippert (University of Hamburg), ICSTEST 2002

8/3/2019 SAST 2002-09-19 Managing ET

8/55


The eXtreme Programmingand Exploratory Testing Analogy:

Agile software testing is not conventional (scripted) software testingdone more quickly or done on tippie-toe.

Exploratory Testing:proven (no single technique is new)

application oriented

planned and disciplined

controllable and reliable

risk minimizingTwo sides of Exploratory Testing:

for the tester: freedom, flexibility, fun

for the manager: controllability, reliability, high quality

8/3/2019 SAST 2002-09-19 Managing ET

9/55


ET vs. Scripted Testing

Fully

Scripted

Testing

Ad-hoc

Testing

AutomatedTests

Bug

Hunting

Exploratory Testing

Jarle Vga (2002)

8/3/2019 SAST 2002-09-19 Managing ET

10/55


What is Scripted Testing?Small (but realistic) example:

How to script and test this login?(Functional tests only not security!)

8/3/2019 SAST 2002-09-19 Managing ET

11/55


When to use Exploratory Testing? (1)There is little or no specifications and /or requirements

You have little or no domain knowledgeYou dont have time to specify, scriptand test

Uncertainty and Time Pressure!

8/3/2019 SAST 2002-09-19 Managing ET

12/55


When to use Exploratory Testing? (2)A common goal of exploration is to probeforweakareasof the program.

Test teams resource consumption per week:25% of the groups time developing new tests

50% executing old tests (including bugregression)

25% on exploratory testing

Cem Kaner (2001a)

8/3/2019 SAST 2002-09-19 Managing ET

13/55


Testing is done in Context1. The value of any practice depends on its context.2. There are good practices in context, but there

are no best practices.

3. People, working together, are the most importantpart of any project's context.

4. Projects unfold over time in ways that are often notpredictable.

5. The product is a solution. If the problem isn't solved, theproduct doesn't work.

6. Good software testing is a challenging intellectual process.7. Only through judgment and skill, exercised

cooperatively throughout the entire project,are we able to do the right things at the righttimes to effectively test our products.

http://www.context-driven-testing.com/

8/3/2019 SAST 2002-09-19 Managing ET

14/55







8/3/2019 SAST 2002-09-19 Managing ET

15/55


Doing Exploratory TestingKeep your mission clearly in mind.

Keep notes that help you report what you did,

why you did it, and support your assessmentof product quality.

Keep track of questions and issues raised inyour exploration.

To supercharge your testing, pair upwith another tester and test the samething on the same computer at thesame time.

From Rapid Software Testing, copyright 1996-2002 James Bach

Charter/Mission

Notes, Risks, Issues,Questions and Errors

Execution:Pairs in Sessions

8/3/2019 SAST 2002-09-19 Managing ET

16/55


Charter: A clear mission for the sessionA charter may suggest what should be tested, how itshould be tested, and what problems to look for.

A charter is not meant to be a detailed plan.

General charters may be necessary at first:Analyze the Insert Picture function

Specific charters provide better focus, but take moreeffort to design:

Test clip art insertion. Focus on stress and flow techniques,and make sure to insert into a variety of documents. Wereconcerned about resource leaks or anything else that mightdegrade performance over time.


Charter/Mission



8/3/2019 SAST 2002-09-19 Managing ET

17/55


Taking NotesTest Coverage Outline/Matrix

Evaluation Notes

Risk/Strategy ListTest Execution Log

Issues, Questions & AnomaliesIt would be easier to test if you changed/added

How does work?Is this important to test? How should I test it?

I saw something strange


Charter/Mission



8/3/2019 SAST 2002-09-19 Managing ET

18/55


ET DocumentationPlanning:

Charter overview of what to test (plan)

Migh be a flip chart on the wall

Mission What are we looking for?

ExecutionNotes what happened during testing?

What did I do? Why did I do it?

Used to assess product quality after test.Data files input data used for testing

Bug reports enough details to recreate the test / bug

Track questions and Issues

Charter/Mission



8/3/2019 SAST 2002-09-19 Managing ET

19/55


Approaches / Styles of ETAt the heart of all ET styles:

Questionsand Questioning Skills

Characterize the styles with respect to each other:Do they focus on:

Method of questioning?

Method of describing or analysing the product?

The details of the product?

The patterns of use of the product?The environment in which the product is run?

To what extent would this style benefit from groupinteraction?

From Black Box Software Testing, copyright 19962002 Cem Kaner

8/3/2019 SAST 2002-09-19 Managing ET

20/55

8/3/2019 SAST 2002-09-19 Managing ET

21/55


Exploratory Testing SkillsGeneral systems modelling and dimensional analysis

Inferencing out loud

Experiment designTechnical story telling

Use of mnemonics and heuristics

De-biasing (individual or team)

Exploratory investigationRisk analysis

Bug advocacy

8/3/2019 SAST 2002-09-19 Managing ET

22/55


I thought skill is the ability to something,more or less.

Skill varies from person to person.Its distinct from talent and knowledge.

A technique, by contrast, is a way of doingsomething; a sort of recipe.

Skill belongs to a person, technique isuniversal.

James Bach, Satisfice Inc.

[email protected]

8/3/2019 SAST 2002-09-19 Managing ET

23/55


Heuristics (and rules) and Skills we relate to heuristics as a tool to apply; something that might

help us do the right thing in a given situation,

whereas we relate to a rule as something to

comply with; something that defines right behavior.

Using heuristics properly requires that you exercise

discretion and judgment, on some level; whereas judgment may get

in the way of rules. It's helpful to have contradictory heuristics,because that's like having a variety of advice available before

making a decision; whereas contradictory rules make compliance impossible.

[email protected]

8/3/2019 SAST 2002-09-19 Managing ET

24/55


Heuristic Risk Based TestingThe purpose of finding errors

Make a prioritised list of risks.

Perform testing that exploreseach risk

As risks evaporate and newones emerge, adjust your testeffort to stay focused on thecurrent crop. Ref.: James Bach

8/3/2019 SAST 2002-09-19 Managing ET

25/55


Summarizing ETThe object of your mission is to explore theMissouri river, & such principal streams of it,

as, by its course and communication with thewaters of the Pacific ocean...may offer themost direct & practicable water

communication across this continent for the

purposes of commerce.

- Thomas Jefferson's letter to Meriwether Lewis, June 1803

Robinson, H., Microsoft, Exploratory Modeling

8/3/2019 SAST 2002-09-19 Managing ET

26/55







E i

8/3/2019 SAST 2002-09-19 Managing ET

27/55


The Test Session

1) Charter

2) Time Box3) Reviewable Result

4) Debriefing vs.


Charter/Mission



Debriefing

E ti

8/3/2019 SAST 2002-09-19 Managing ET

28/55


ET Execution - SessionsSession

Basic testing work unit

Uninterruptede-mail, meetings, telephone calls etc.

ReviewableA report should be produced

CharteredMission associated with this session; What arewe testing? What are we looking for?

Charter/Mission



Debriefing

8/3/2019 SAST 2002-09-19 Managing ET

29/55


Charter:

A clear mission for the sessionA charter may suggest what should be tested, how itshould be tested, and what problems to look for.

A charter is not meant to be a detailed plan.

General charters may be necessary at first:Analyze the Insert Picture function

Specific charters provide better focus, but take moreeffort to design:

Test clip art insertion. Focus on stress and flow techniques,and make sure to insert into a variety of documents. Wereconcerned about resource leaks or anything else that mightdegrade performance over time.


8/3/2019 SAST 2002-09-19 Managing ET

30/55


Time Box:

Focused test effort of fixed duration

Brief enough for accurate reporting.

Brief enough to allow flexible scheduling.

Brief enough to allow course correction.

Long enough to get solid testing done.Long enough for efficient debriefings.

Beware of overly precise timing.

Short: 60 minutes (+-15)Normal: 90 minutes (+-15)

Long: 120 minutes (+-15)


8/3/2019 SAST 2002-09-19 Managing ET

31/55


Reviewable Result: The session sheetCharter

#AREAS

Start Time

Tester Name(s)Breakdown

#DURATION

#TEST DESIGN AND EXECUTION

#BUG INVESTIGATION AND REPORTING#SESSION SETUP

#CHARTER/OPPORTUNITY

Data Files

Test Notes

Bugs#BUG

Issues#ISSUE

CHARTER-----------------------------------------------

Analyze MapMakers View menu functionality and

report on areas of potential risk.#AREAS

OS | Windows 2000Menu | ViewStrategy | Function TestingStrategy | Functional AnalysisSTART-----------------------------------------------5/30/00 03:20 pm

TESTER-----------------------------------------------Jonathan BachTASK BREAKDOWN-----------------------------------------------#DURATIONshort#TEST DESIGN AND EXECUTION65#BUG INVESTIGATION AND REPORTING 25#SESSION SETUP20


8/3/2019 SAST 2002-09-19 Managing ET

32/55


Debriefing:

Measurement begins with observationThe manager reviews session sheet to assurethat he understands it and that it follows theprotocol.

The tester answers any questions.

Session metrics are checked.

Charter may be adjusted.

Session may be extended.New sessions may be chartered.

Coaching happens.


Ch / i k Execution:

8/3/2019 SAST 2002-09-19 Managing ET

33/55


SummaryExploratory Test Management

Requires skills and experience practice

Be prepared! Improve your skills!Charter Define your mission

Take notes

Test in SessionsTrack Session Metrics

Charter/Mission



Debriefing

8/3/2019 SAST 2002-09-19 Managing ET

34/55







8/3/2019 SAST 2002-09-19 Managing ET

35/55


High Speed WEB Testing*

A Case Study

With permission from Jarle Vga, XIT AS,[email protected],

Oslo, Norway* contribution to the book Software Quality and Testing in Internet Times, see references

8/3/2019 SAST 2002-09-19 Managing ET

36/55


Presentation OutlineAbout the WEB application

Preparation

Environment and people

Test Planning

Test execution

The web testing experience

Its on the Web. Go find the bugs before the client does!

8/3/2019 SAST 2002-09-19 Managing ET

37/55


Application: Basic functionalityWEB publishing tool:

Content Creator creates pages

Multi dimension / organisational viewsMulti language

Topic Manager have to approve eachtopic before publication

One company-One message

8/3/2019 SAST 2002-09-19 Managing ET

38/55


Preparation...DocumentationSystem documentation not available onlyhigh level standard product description

Changes on the test environmentConstant changes during planning period

Browser difficulties

Not possible to downgradeMicrosoft software, or toinstall two versions of InternetExplorer on the same PC

8/3/2019 SAST 2002-09-19 Managing ET

39/55


What to doFeatures to be tested:Functional test

Search facilities

Browser testsInternet Explorer 4.0 and up

Netscape 4.0 and up

Operating Systems:

Windows NTWindows 98

Windows 2000

would this be enough? And how to do it??

8/3/2019 SAST 2002-09-19 Managing ET

40/55


Who?12 testers

Different business areas

Different countriesMarketing and publishing people, a few ITliterates

2 test coordinatorsTest Environment manager

8/3/2019 SAST 2002-09-19 Managing ET

41/55


Test Environment20 PCs

5 browsers (versions)

3 Operating systems12 people

2 user groups simultaneously:Publishers / Content CreatorContent Approvers / Topic Manager

8/3/2019 SAST 2002-09-19 Managing ET

42/55


How to...

Test scripts - abandonedUnstable system

Unavailable documentationPair testingTo motiovate the tesers andovercome any problems related to

inexperienced peopleExploratory testing

Will depend on the testers ability toexplore the system to find bugs

8/3/2019 SAST 2002-09-19 Managing ET

43/55


The Test Plan (Chart)

Features to be testedImportant to be precise, this helps focusing the pair testing

Features not to be testedJust as important, used to limit the test

ApproachPreparations and requirements

No specific test script (but rather guidelines)

Use of role-play and test matrixes

Item pass/ fail criteriain order to judge the outcome of the test.

8/3/2019 SAST 2002-09-19 Managing ET

44/55


Test ExecutionMajor difficulties with the set-up of the test environment

Browsers not installed

Win2000 pcs not available

Functionality test according to guidelinesCreation of pages in the portal using role-play with pairs (ContentCreator/Topic Manager)

Using a matrix, where roles, OS and browsers were integrated.

Browser and op.sys. testMatrix with 5 browsers and 3 op.sys.

Yellow pages (search functionality)re-use of pages published

Exploratory testwhat happens if...

Th t i i di t h

8/3/2019 SAST 2002-09-19 Managing ET

45/55


ExecutionThe Matrix

GroupsUser group

1 2 3 4 5 6

Content Creator

Topic Manager

Group # Operating System

W-NT W98 W2000

Netscape 4 6 1

Netscape 6 3 4/6 5

Iexplore 4 1

Iexplore 5 4/5 2/3Browser

Iexplore 5.5 2 5 6

Day 1 10:00-12:00 13:00-15:00 15:00-16:30

Time

Day 2 09:00-17:00 09:00-17:00 13:00-16:00

The upper matrix indicate howthe groups worked on differentcombinations of browsers andOS. The lower how the groups

were divided into different usergroups at the start of the test, foreach change of operatingsystem they were alternated.

I.e. Group 3 started as ContentCreator working on NT with

Netscape 6, and moved to W98with Explorer 5 after lunch,working as Topic Manager.Different test guidelines was

developed for each combination,which gave us

OS*Browser = 3*5 = 15 differentguidelines.

8/3/2019 SAST 2002-09-19 Managing ET

46/55


Execution Exploratory pair testing

Plan increase coverage by using thetesters creativity

which is what happenedIt can be difficult to keep testers ontrack

but easy to motivate, and many bugsfound.

8/3/2019 SAST 2002-09-19 Managing ET

47/55


Execution Debriefing and Notes

Continous guiding / coordination required

Taking Notes did not seem to be a problem

(due to testers working in pairs)High number of duplicate error messages dueto no clear mission set for each pair

Debriefing takes time! Need to be planned

8/3/2019 SAST 2002-09-19 Managing ET

48/55


Acceptance Test SummaryApprox. 150+ bugs reported (incl. duplicates)

Mainly due to lack of limitations in the system, youcould do anything.

Developer accepted only approx. 60Corrupted pages: Web portal made inaccessible

Menu disappeared when usingsigns like (#*\/@ etc ) in

page titleCustomer: Severe safety riskDeveloper: Tester have misusedthe system

TestManager

8/3/2019 SAST 2002-09-19 Managing ET

49/55


Case Summary

The portal was NOT recommended forpilot production after the test.

Using pair- and exploratory methods

proved successful

Has happened since then:The two test coordinators scripted tests for the 60 bugs

fixed and participated in a 2 days re-test of the applicationApplication went live after this total of 4 days testing andhas been in production without significant problems for morethen a year

8/3/2019 SAST 2002-09-19 Managing ET

50/55


Lessons learned...(1)

Pair testingSpeeded up reporting errors

Excellent to promote creativity

Encouraged discussionDemanded a strong voice to keep everyonefocused!

Enables effective testing of short-lived, intense

projects...but has also proved very useful on many testprojects, also larger (i.e. at Microsoft, ref. J.Bach)

8/3/2019 SAST 2002-09-19 Managing ET

51/55


Lessons Learned...(2)

Exploratory testingRequires skilled testers

Requires some pre-set restraints, and thorough planning

Requires a system for briefing and reporting/ documenting(ref. J.Bach -Session based test management)

Requires guidance from test manager

..but when the requirements are fulfilled, it proved to bevery successful.

Easy to motivate

MANY deviations found

and it gave testers a greater feeling of participation

8/3/2019 SAST 2002-09-19 Managing ET

52/55


Learn moreExploratory Testing:

Rapid Software Testing, by James Bach,[email protected]

Black Box Software Testing, by Cem Kaner,

[email protected]

Amland Consulting; London, Copenhagen and Stockholm:Stockholm, October 30 31, 2002 (www.enea.se)

http://www.amland.no/training_schedule.htm

People Issues, test techniques, inspections:Grove Consultants, www.grove.co.uk

+ many, many more
mailto:[email protected]:[email protected]://www.grove.co.uk/http://www.grove.co.uk/mailto:[email protected]:[email protected]

8/3/2019 SAST 2002-09-19 Managing ET

53/55


References/Bibliography (1)Amland, S., 1999. Risk Based Testing and Metrics, EuroSTAR '99Preceding, Barcelona, Spain, http://www.amland.no/articles

Argus, C. & Johnson, B., Ad Hoc Software Testing; Exploring theControversy of Unstructured Testing,

http://www.testingcraft.com/ad_hoc_testing.pdfBach, James, 1999a. Risk-Based Testing. How to conduct heuristic riskanalysis, Software Testing & Quality Engineering Magazine,November/December 1999, vol. 1, issue 6,http://www.stqemagazine.com. See also www.satisfice.com

Bach, James , 1999b. What is Exploratory Testing? Satisfice.com

Bach, Jonathan, 2000, Session Based Test Management, STQEMagazine, 11/00, www.satisfice.com

Beizer, Boris, Software Testing Techniques, Van Nostrand Reinhold.NY. 1990.

Collard, Ross, 2002, to be published, [email protected]
http://www.amland.no/articleshttp://www.testingcraft.com/ad_hoc_testing.pdfhttp://www.satisfice.com/http://www.satisfice.com/mailto:[email protected]:[email protected]://www.satisfice.com/http://www.satisfice.com/http://www.testingcraft.com/ad_hoc_testing.pdfhttp://www.amland.no/articles

8/3/2019 SAST 2002-09-19 Managing ET

54/55


References/Bibliography (2)Kaner, C., Falk, Nguyen, 1999, Testing Computer Software, John Wiley & Sons

Kaner, C., 2000. Rethinking Software Metrics, Evaluating measurement schemes,Software Testing & Quality Engineering Magazine, March/April 2000, vol. 2,issue 2.

Kaner, C., 2001a, Black-Box Software Testing, training at Satisfice July 2001.

Kaner, C., Bach, J., Pettichord, B., 2001b, Lessons Learned in Software Testing,John Wiley & Sons; http://www.testinglessons.com/, ISBN: 0471081124

Kaner, C., Bach, J. 2001c, Exploratory Testing in Pairs, presentation atSTAREast, Orlando, FL, www.kaner.com

Keith, Geordie, 2002, All Hands on Deck: How we used our whole company totest, STQE Magazine, vol. 4, issue 4, July / August 2002.

Lyndsay, James and Eeden, Niel van, 2002, Adventures in Session-BasedTesting, paper at StickyMinds.com, will also be presented at EuroSTAR 2002

Vga, J., Amland, S., 2002. High Speed Web Testing, contribution to the book Software Quality and Testing in Internet Times, Editor: Dirk Meyerhoff et al,Springer-Verlag, 2002, ISBN: 3-540-42632-9
http://www.testinglessons.com/http://www.kaner.com/http://www.kaner.com/http://www.testinglessons.com/

8/3/2019 SAST 2002-09-19 Managing ET

55/55

Stle Amland

Amland Consulting

Hulda Garborgsv. 2

N-4020 STAVANGERNorway

Phone: +47 905 28 930

Fax: +47 51 58 55 24

e-mail: [email protected]

WEB: www.amland.no

Contact Details
mailto:[email protected]://www.amland.no/http://www.amland.no/mailto:[email protected]

SAST 2002-09-19 Managing ET

Documents

Transcript of SAST 2002-09-19 Managing ET