The Awkward Teenager of Testing: Exploratory Testing

by Shawn Faunce

The Exploratory Testing Handbook

Shawn M. Faunce &

Raj Indugula

Reduce your risk as you embrace DevOps and

Continuous Delivery

L E A N P U B . C O M

March 31st – Initial Release

December 31st – Final Release

AWKWARD

Tested = Checked + Explored

Charters

Target Areas

Reso

urce

s

Discovery

Mind M

aps

EcoSystem Heurist icsG o l d i l o c k s

Follow the DataNouns and Verbs

Sta rva t i on

Reconnaissance Charter

A c t i v e R e a d i n g Ambi

guity

Ana

lysis

Risk Heuristics

We think we understand exploratory testing just as we think we understand teenagers,

because “we have been there”.

Key Concepts

Adoption

Heuristics

Mind Map

Charter

Exploratory Testing

TalkTalk

Adoption

Heuristics

Mind Map

Charter

Exploratory Testing

TalkTalk

Part 1: Exploratory Testing

Why Exploratory Testing

What is Exploratory Testing?

• Tested = Checked + Explored

• Going off-script to find that undiscovered bug.

• What you do to promote code to the next testing level.

• High-value testing that relies on a tester’s intuition, skill, and testing experience.

Part 2: Charters

Adoption

Heuristics

Mind Map

Charter

Exploratory Testing

TalkTalk

ResourcesTargetAreas

ToDiscover

Info

Gives us a common, light-weight, solid foundation from which to venture forth. They are a framework that aides us in gathering insights while we wander and help us avoid getting lost.

A Spot, not a travel itinerary

You must plan ahead what you will need

You must think

You seek treasures not defects

Charters

Are You Smarter Than a 5th Grader

Who was asked to explore the area west of the Mississippi taking with them only what could fit in a canoe (guns, ammo, furs, food) to discover the natural resources?

Explore the area west of the Mississippi

With whatever fits in a canoe (guns, ammo, furs, food)

to discover the land’s natural resources

Explore Sign In

With a state table and a valid account

to discover if I am able to access and view information only when I am properly signed in

Explore <target area>

With <resources>

to discover <information>

Example Charter

Part 3: Mind Map

Adoption

Heuristics

Mind Map

Charter

Exploratory Testing

TalkTalk

GalacticModelingLanguage

EcoSystemInformationRadiator

Mind Maps are a visual representation of your system. They are not detailed architectural diagrams. They help orient your exploratory testing.

3 parts:1) Boxes2) Lines3) TextKeep it simpleEasy to update

Conversation starters

As important as a Scrum board

EcoSystem = Mind Map + Infrastructure

Mind Maps

Google.com

Fun Stuff

Images

Grid

Notification

Gmail

Icon Array

More Area

Settings

Terms

Privacy

About

Business

Ads

Stuffy Stuff

Search Area

Search Box

Search Btn

Lucky Btn

Other

Results Page

Doodle

Social Media

Example Mind Map

Sign In

Explore Sign In

With a state table and a valid account

to discover if I am able to access and view information only when I am properly signed in

Part 4: Heuristics

Adoption

Heuristics

Mind Map

Charter

Exploratory Testing

TalkTalk

Heuristic

Heuristic DefinitionNouns and Verbs List out and number all nouns and verbs in your system

Roll 1 dice for nouns and 1 dice for verbs; the result is what you test

Never and Always Apply “Never” and “Always” conditions to the “ilities” (Usability, Scalability, Reliability, etc.)

Follow the Data Apply “Create,” “Read,” “Update,” and “Delete” to your data

State Table List States and Events in a table, explore possible odd combinations

Goldilocks Try small (including zero), medium, large data entry

Format Explore fields with special formatting

Navigation Try keyboard, mouse, and browser buttons

Example Heuristics

Explore Sign In

With a state table and a valid account

to discover if I am able to access and view information only when I am properly signed in

Explore Search Area

With nouns and verbs heuristic

to discover unexpected behavior when I interact in a random fashion.

Explore Google.com

With never and always heuristic

to discover if the active control is visually annotated

Explore Search Box

With the Goldilocks heuristic and randomtextgenerator.com

to discover effect on search results with various sizes of search text

Explore Grid

With navigation heuristic and the Safari and Chrome browsers

to discover if I can navigate successfully using only the TAB, Shift-TAB, and Enter Keys

Part 5: Adoption

Adoption

Heuristics

Mind Map

Charter

Exploratory Testing

TalkTalk

Adoption RoadmapCo

mpl

exity

Practice Adoption

Charters Mind Map Heuristics

FoundationalPractices

IntermediatePractices

AdvancedPractices

Doc Recons1 charter per personPaired testing

Expand the mind mapUpdates are occuringPlotting charters

Heuristics are used on some charters

Team prioritizes use of heuristics

Charters existsTarget areas tied to

mind mapBasis for metrics

1 page map exists

Mind map the heuristicsInclude hardware and software architecture

N/A

N/A

Provide guidance, not step by step actions Read like a test script

Can be completed in one session Are not specific enough to know what has or has not been tested in the charter execution

Are aligned with your stakeholder’s concerns Ignore high risk areas and do not have input from your stakeholders

Use a common language or terms to describe the target areas so that others know what has or has not been explored

Are ambiguous as to where you have been or refer to terms only your team uses

Are recorded and made visible so that future charters can use the results from those charters as inputs into new charters

Cut’n paste the same charters and look more like a traditional smoke test.

Demonstrate some thought about what you are trying to discover or why you are going to a certain area

Use vague or broad terms about what you hope to discover, e.g., “to discover bugs”

Complement and augment the other components of your testing strategy

Repackage your test scripts and scenarios

Good Charters Bad Charters

Challenges and solutions transitioning a traditional test team into an exploratory test team on a large, heavily interfaced system.

I’ve a feeling we’re not in Kansas anymoreChallenge: Changing a mindset Solution: Framework, cadence, training, discussions, and metrics

Why do you want access to my system?Challenge: You need “visas” to exploreSolution: You need allies, and strong ones

Yet another build!!!?Challenge: Continuous Delivery means continuous deploymentsSolution: Test in parallel with and after production deployments

What is truth?Challenge: Acceptance criteria vs. Desired OutcomeSolution: Find someone who cares

Boldly going where no one has gone beforeChallenge: Indigenous population may be indigentSolution: “Marketing” brochure and chocolate chip cookies

Notes From the Trenches

References

3 Titans:• Elisabeth Hendrickson• James Bach• Cem Kaner

Elisabeth HendericksonHendrickson, Elisabeth, Explore It!: Reduce Risk and Increase Confidence with Exploratory Testing

James Bachhttp://www.satisfice.com/blog/archives/category/exploratory-testing

Cem Kanerhttp://www.kaner.com/pdfs/QAIExploring.pdf

?