Exploratory Testing

Anutthara BharadwajMicrosoft

http://blogs.msdn.com/anutthara

GSJGD

if there was an Olympics for FAILURE

Or a World Cup for that which fails the most

surelySOFTWARE

wouldreign

supreme

How do we find the *%&^ bugs?!!

GSJGD

Manual v. automated

• Automation is cool, it gets respect … but …– Automation is code• like any code it will take time, have bugs, cause false

positives and require maintenance• it will have an imperfect oracle

• Manual testing is tedious … but …– It finds over 90% of the bugs in the system

Manual testing

• Scripted manual testing– From rigid (click this button, enter this value) to

flexible (achieve this result)– Requires preparatory work– Scripts act as their own documentation

• Exploratory testing– Minimizes preparatory work– Documentation of testing is extra work

Manual testing

• We need to get better at it … A LOT better at it• What is the right prep time v. test time?• What are the right parts of the app to focus

on?– Can we pinpoint problematic functionality?– Can we focus on changed code?– Should we focus on buggy parts?

• Where is the method in the madness?

Types of exploratory testing

• Freestyle• Scenario• Strategy• Feedback

Exploratory testing

• Freestyle exploratory testing– Ad hoc exploration of an application’s features– No rules, not necessary to account for coverage,

past tests, etc.– Good for:• familiarizing yourself with an application to prepare for

more sophisticated methods• quick smoke tests, sometimes finds bugs

Exploratory testing

• Scenario-based exploratory testing– An extension of traditional scenario testing– Start with:• user stories• e2e scenarios• test scenarios

– Then widen the scope and inject variation as you execute the scenarios

Exploratory testing

• Strategy-based exploratory testing– Freestyle testing combined with known bug

finding techniques• documented strategies like boundary values and

combinatorial testing• intuitive instinct from experience or familiarity with the

application

– Focus is on learning tips and tricks and recognizing when to apply them

Exploratory testing

• Feedback-based exploratory testing– Starts out freestyle until a test history is built up– That feedback then guides future exploration• Test history• Coverage• Churn• …

– Tools are often an integral part of this type of testing as they ‘remember’ history for us

The tourist metaphor

• You’re visiting Hyderabad for the first time– You can go it alone– You can buy a guidebook– You can hire a guide– You can take the bus tour

• On return visits you can– Repeat the best tours– Choose to visit new places

The tourist metaphor

• You’re testing an application– What do you use as a guide?– Don’t just wander the app• Always have a goal in mind• When presented with a choice, allow the goal to guide

you• Track which goals work for you

– Coverage of code, features, UI– Find important bugs– Force the software to best exhibit its functionality

The guidebook tour

• Guidebooks help tourists navigate choices– Boil down the possibilities to a ‘best of’ list– Many tourist follow this advice exclusively

• Use– User manual, follow its advice– Online help (the F1 tour)– Third party advice (the blogger’s tour)– Detractors (the pundit’s tour)– Competing products (the competitor’s tour)

The antisocial tour

• Tours have rules and there are places off limits• Antisocial tourists ignore those rules – Enter illegal input (the crime spree tour)– Enter the least likely input (the opposite tour)– Choose an input, change your mind, use undo, use

cancel, then do it all over again

The back alley tour

• Explore the back alleys of your app• Do you track feature usage?– Tour the least popular features– Mix and match popular and unpopular features

(mixed destination tour)

The couch potato tour

• Being a nonparticipant on a tour is a waste of money• Sometimes for a tester, it finds bugs– Do as little as possible on every screen– Leave fields blank, enter as little info as possible, always

take the shortest path• Why?– Default values have to be set, used– Decisions have to be made by the software– Just because you aren’t working hard doesn’t mean the

software isn’t either

The obsessive compulsive tour

• OCD tours would likely be quite unpopular for real tourists– Avoiding sidewalk cracks– Walking the same street over and over obsessively

• But testers can use repetition to their advantage– Developers often code based on an expected path– Repeating inputs individually and in sequence can

often lead to bugs

The all nighter tour

• The ‘clubbing tour’ is about staying out all night, a real test of the constitution

• Software should stay out all night too– Keep it working, never shut it down– Keep files open without saving or closing them

• This tests time-out functionality, finds memory problems and leaks that are masked by shutdown-restart

The TOGOF tour

• Test-One-Get-One-Free• Run multiple copies of the app• Put them through their paces• Try and open the same files, access the same

resources … can they coexist?• Why TOGOF? Because if you find a bug in one

copy, you’ve broken them all

The saboteur • The sabotage tour• Ask the software to do something

– Access a resource– Open a file– Perform a tasks

• Understand the resource it uses– Memory– Files – Network

• Then prevent it from doing so– Take away memory– Delete/rename files– Turn off the network

The collector’s tour

• Some people need to do it all– Pictures of landmarks, samples at a wine tasting,

signatures from the Disney characters– They don’t want to miss anything

• Testers can learn from this– Force the software to produce every output

• Buy items from every department• Put every special character in an email• Fill a document with every possible graphic• …

The supermodel tour

• This tour goes only skin deep– Ignore the function and meaning and look only at

the interface– It’s about first impressions

• Watch the interface elements– Do they look good and render properly?– Are interface elements consistent across the app?– As you make changes does the UI refresh properly?

Types of exploratory testing

• Scenario– Start with a scenario

• Strategy– Start with accumulated

testing knowledge

• Feedback– Test for a while– Build up a history– Use the history to guide

additional testing

• Then inject variation– Freestyle guidance– Tour metaphors

Obstacles to exploratory testing

• Where’s the accountability?• How do we track completeness?• What metrics are important?

WE NEED TOOLS!

Q/A

GSJGD