Start, Grow, and Perfect Exploratory Testing on Your Team

Kevin DunneDirector of Product Strategy

QASymphony

Kimberly StockettQA Manager

KMS Technology

Agenda

1. What is exploratory testing (ET)?

2. What is NOT ET?

3. How can I start an ET practice?

4. How can I expand the footprint of my ET practice?

5. How can I measure & improve the effectiveness of my ET practice?

What is exploratory testing?

1. Parallel test planning, test design, and test execution

2. Specific yet flexible

3. Aligned towards investigation of potential opportunities

4. Values depth and attention to detail during testing

5. Fosters knowledge sharing and awareness

Parallel Planning, Design & Execution

Unlike traditional testing techniques, planning, design, and execution happen concurrently, allowing efficiencies of time as well as flexibility in approach

Plan Design Execute Report

Plan

DesignExecute

Report

Specific yet Flexible

Exploratory testing provides a specific lens through which to perform testing – whether that be a user persona, functionality, criteria (i.e. localization), etc.

However, it allows testers to use the tool as an end user would, not necessarily as the product owner envisioned it

Manual Scripted Testing

I tested the application as the script prescribedExploratory Testing

I tested the application as the end user would

Investigating Opportunities

Exploratory testing rewards testers who identify unknown areas of “opportunity” within the application, as they are essential in maintaining a backlog of future test charters

Manual Scripted Testing Exploratory Testing

Knowledge Sharing

Exploratory testing relies on knowledge sharing to reach full potential – developing testers who understand the impact of more areas of the application allows them to identify more areas of risk and opportunity

Plan

DesignExecute

Report Transfer Learning

Example Questions to Ask

• Have you seen this before?

• What am I not considering?

• Why would someone do this?

• How would you have tested this?

What is Not Exploratory Testing

1. Exploratory testing is NOT unstructured testing

2. Exploratory testing is NOT the only form of testing

3. Exploratory testing is NOT throwaway work

4. Exploratory testing is NOT impossible in a regulated environment

ET is NOT Unstructured Testing

While exploratory testing allows for flexibility in the exact path of the application that is tested, it is NOT unstructured, in that it still contains parameters such as:

1. A goal of the exploration

2. A log of the activity performed

3. A lens through which the testing is performed (i.e. a user persona)

Performing exploratory testing without involving some parameters such as the above allows a greater risk of unsuccessful implementation of exploratory testing

ET is NOT the Only Form of Testing

Exploratory testing is best suited as a complement to automated and manual scripted test cases. It can feed these types of testing to create greater depth in testing, and also to identify any potential gaps in coverage.

Potential New Feature Testing Cycle

Code Developer Unit Test

Exploratory Testing

Manual Scripted

Test

Automation Regression

Test

Exploratory Testing

Feature “Delivered”

“Let’s make sure this is worth writing

scripts against yet”

“Let’s make sure were still testing all

aspects of this”

ET is NOT Throwaway Work

Exploratory testing does NOT need to be extra work done on top of other testing methods – it can count on its own towards testing progress and coverage if properly accounted for. Some of the necessary information needed to manage it:

1. Charter2. Session Sheet3. Oral report4. Debrief5. Data Files6. Logs

"Any testing approach is manageable when you choose to manage it.” – Michael Boltonhttp://www.developsense.com/blog/2010/01/exploratory-testing-is-accountable/

ET is NOT Impossible in a Regulated Environment

Contrary to rumor and popular belief, exploratory testing is not only allowed in most regulated environments, it is also essential.

Starting an Exploratory Testing Practice

1. Who - choosing the pilot group

2. What - choosing the application

3. Where - choosing the team/location

4. When - choosing the release cycle

5. How - choosing the Exploratory Testing framework/approach

Who will be the first to explore?

Finding the right group of initial members to pilot exploratory testing is one of the most critical aspects of successful implementation. Pilot members should be:

• Comfortable with uncertainty and flexible

• Well connected in the organization

• A group that covers the ranges of skill and experience on the team

• Solid communicators and willing to speak up about their concerns

• Attuned towards new experiences as well as personal and team improvement

What will be the first application we explore?

The pilot application should be one that minimizes the risk to the business of serious impact if exploratory testing is unsuccessful. Some factors to look out for in a pilot application are:

SizeSmall to medium sized application – easier to track coverage

Time on MarketMature application – easier to see net impact of explorations

Business CriticalityNot the flagship product – executives will allow a longer period to pilot

Where will the first exploratory team be located?

To ensure the highest likelihood of success, it is important to choose a team that is located centrally with easy access to developers and visibility to executives overseeing the implementation

Centrally LocatedAllow for collaboration and knowledge sharing early on

Access to DevelopersWhen developers are near by, testers can provide feedback quickly

Executive VisibilityLocating the team near execs means better visibility of impact

When will we first introduce exploratory testing?

It is critical to introduce exploratory testing during a time where the pilot group can contribute their full attention to the new initiative and they are not being pulled in other directions:

• Avoid “busy season” – typically around big end of year releases• Avoid releases/sprints that overlap with long holidays• Avoid times of the year when application usage is high (i.e. year end closing for an

accounting application)

Be cautious of implementing exploratory testing while other change initiatives are going on. It can be helpful as there is a change “tailwind”, but be aware exploratory testing might be falsely attributed to other initiatives’ failure

How will we structure our exploratory testing?

Introducing exploratory testing within a framework will greatly increase the odds of success, and will reduce fear and uncertainty among the practitioners as well as executives.

Session Based Test Management is a popular framework for this, as it tracks all the important data on testing:

More info on SBTM: http://www.satisfice.com/articles/sbtm.pdf

• Session charter (includes a mission statement, and areas to be tested) • Testers involved Date and time executed• Task breakdown• Data files• Test notes• Issues • Bugs

Expanding the Footprint

1. Build champions within the organization

2. Share best practices and tips

3. Understand the limits and opportunities

Team #3 Team #3 Team #3

Build Champions within the Organization

There are two main ways we can build champions to spread the word about exploratory testing in our organization:

Initial Team

New Team

New Team

“Bowling Pin” Model “Infiltration” Model

Initial Team

Tester #1

Tester #2

Tester #3

Share Best Practices and Tips

A common threat in Agile organizations is the “information hoarding.” As loyalty to the Agile team grows, teams can lose site of the importance of improving the organization as a whole, and strive instead to improve there position as the best team.

Ways to mitigate against this include:

• Greater alignment with organizational goals• Transfer team members periodically

• Reward teams for sharing knowledge• Provide greater visibility across teams

Understand the Limits and Opportunities

There is a good chance that you will find early success with exploratory testing.

However, it is important to temper your excitement to roll out exploratory testing across your organization.

Executives often have an incredible memory when it comes to failed initiatives. Often, it is better to follow a measured approach than risk having exploratory testing barred from your organization entirely.

Measure and Improve ET Practices

1. What are we trying to accomplish by implementing ET?

2. How will we measure ET’s ability to accomplish these goals?

3. How will accomplishing these goals improve our ability to expand ET in the organization?

Benefits of Exploratory Testing

1. Increases focus on unique scenarios with greater potential to find new bugs

2. Drives engagement from testers and fosters critical thinking during the testing process

3. Boosts team morale and provides a change of pace from manual scripted testing

4. Encourages testers to think as end users and the "voice of the customer" and drive improvement in the product

SMART Goals

• Specific• Measurable• Attainable• Relevant• Time-bound

Making the Case for Exploratory Testing

• Claim: ET will create more educated testers• Metric - increase in average # of testers capable of test a given

scenario• Claim: ET will provide faster feedback on new features

• Metric - reduction in user stories revised in subsequent sprints• Claim: ET will increase the alignment with customer needs

• Metric - reduction in support tickets/feature requests• Metric - increase in net promoter score/referenceable customers

• Claim: ET will garner communication product, dev, and testing• Metric - reduction in open defect time• Metric - increase in acceptance criteria met

Continuous Improvement

The path to success with exploratory testing does NOT end when implementation is complete.

You must continually improve to make sure that your program continues to add value in the organization

Plan

Implement

Evaluate

Reassess

Continuous Improvement

KMS has successfully implemented Exploratory testing techniques for all clients seeing benefit in quality and efficiency to the act of testing with an added bonus of boosted morale for the QA Analysts.

• Benefits we have seen

• Time boxed testing creates structure and limits

• Risk is apparent from application reviews

• Faster and deeper product understanding with hands on exploration

• Proactive over reactive approach in discovering bugs or workflow issues

• Items to keep in mind

• Worm holes in testing – knowing when to end a testing path is based on expereince of the QA Analyst

• Experiences drives when to annotate in the tool (QASymphony eXplorer) to make defect referencing easier

• Session definition – too much or too little

KMS is a fast growing global service provider with over 500 professionals trained on QA Symphony products. We are a Platinum Partner with QASymphony with over 3 years experience in Exploratory Testing Techniques.

About KMS Application of ET

Efficiency

In a month long project consisting of 4 1 week sprints the team uncovered 146 bugs in 143 hours averaging 1.02 bug/hour

Quality

The team completed Mind Maps outlining 154 Charters completed 144 charters in 143 true testing session hours identified high priority bugs

Dramatic & Measurable Results from ET

Value (Bugs Found)

20 High priority at 14%

57 Medium Priority at 39%

66 Low Priority at 45%

3 Closed as N/A at 2%

Resources/Thought Leaders

• James Bach - http://www.satisfice.com/• Jonathan Bach - https://jonbox.wordpress.com/• Michael Bolton - http://www.developsense.com/• Paul Holland -

http://testingthoughts.com/blog/author/testthought• Keith Klain - http://qualityremarks.com/• Brian Osman - https://bjosman.wordpress.com/

Questions?Kevin Dunne

kevindunne@qasymphony.comTwitter: @kevindunneQA

Linkedin: www.linkedin.com/in/kevindunneQABlog: http://www.qasymphony.com/blog/

Kimberly Stockettkimberlystockett@kms-technology.com