SAST 2002-09-19 Managing ET

download SAST 2002-09-19 Managing ET

of 55

  • date post

    06-Apr-2018
  • Category

    Documents

  • view

    214
  • download

    0

Embed Size (px)

Transcript of SAST 2002-09-19 Managing ET

  • 8/3/2019 SAST 2002-09-19 Managing ET

    1/55

    SAST 2002-09-19Managing ET & Case 2002 Amland Consulting 1

    Managing Exploratory Testing

    Parts of this presentation have been excerpted from:Rapid Software Testing, by James Bach,james@satisfice.com

    Black Box Software Testing, by Cem Kaner, kaner@kaner.com

    High Speed WEB Testing, by Jarle Vga,jarle.vaga@xit.no

    Stle Amland, Amland Consulting,

    Hulda Garborgsv. 2, 4020 STAVANGER, NORWAY

    Mob:+47 905 28 930, Phone: +47 51580587, Fax: +4751585524

    e-mail: stale@amland.no WEB: www.amland.no

    SAST meeting September 19, 2002, Stockholm, Sweden

    mailto:james@satisfice.commailto:kaner@kaner.commailto:jarle.vaga@xit.nomailto:stale@amland.nohttp://www.amland.no/http://www.amland.no/mailto:stale@amland.nomailto:jarle.vaga@xit.nomailto:kaner@kaner.commailto:james@satisfice.com
  • 8/3/2019 SAST 2002-09-19 Managing ET

    2/55

    SAST 2002-09-19Managing ET & Case 2002 Amland Consulting 2

    Software Testing:

    There is artistry. There is craft. There isscience. It is all threeas with all

    technologies.

    If you understand the science and havemastered the craft, then it is meaningful toinvestigate, espouse, and perfect the art.

    Boris Beizer

  • 8/3/2019 SAST 2002-09-19 Managing ET

    3/55

    SAST 2002-09-19Managing ET & Case 2002 Amland Consulting 3

    Software Testing:

    Testing is the process of executing a programwith the intent of finding errors.

    Testing is an extremely creative andintellectually challenging task.

    Glenford Myers inThe Art of Software Testing

  • 8/3/2019 SAST 2002-09-19 Managing ET

    4/55

    SAST 2002-09-19Managing ET & Case 2002 Amland Consulting 4

    Presentation Outline

    What is Exploratory Testing?

    How to do Exploratory Testing?

    How to Manage Exploratory Testing?

    High Speed WEB Testing Case Study

  • 8/3/2019 SAST 2002-09-19 Managing ET

    5/55

    SAST 2002-09-19Managing ET & Case 2002 Amland Consulting 5

    Different testing approachesSkeptical approaches

    Analytical Approaches

    Information-driven approaches

    Time-honored but less effective approaches

    Experiential and intuitive approaches:Lets think blue-sky, speculate and follow our intuition.

    We have good hunches about where the bugs are lurking.

    Lets jump in an explore the systems behavior hands-on, sowe can decide how to test it.

    Lets find the important bugs fast, and worry about the testpaperwork later.

    Ross Collard (2002)

  • 8/3/2019 SAST 2002-09-19 Managing ET

    6/55

    SAST 2002-09-19Managing ET & Case 2002 Amland Consulting 6

    What is Exploratory Testing?"Exploratory testing involves simultaneously learning,

    planning, running tests, and reporting / troubleshootingresults."

    Dr. Cem Kaner (2001)

    "Exploratory testing is an interactive process of concurrentproduct exploration, test design and test execution.

    To the extent that the next test we do is influenced bythe result of the last test we did, we are doing

    exploratory testing.

    James Bach, Satisfice (2001)

    http://www.kaner.com/
  • 8/3/2019 SAST 2002-09-19 Managing ET

    7/55

    SAST 2002-09-19Managing ET & Case 2002 Amland Consulting 7

    Said about eXtreme Programming

    Agile software development is not conventional software developmentdone more quickly or done on tippie-toe. Agile software development issoftware done differently.

    Ron Jeffries, (e-mail on agile-testing list, April 24, 2002)

    proven (no single technique is new)application oriented

    planned and disciplined

    controllable and reliable

    risk minimizingTwo sides of extreme programming:

    for the developer: freedom, flexibility, fun

    for the manager: controllability, reliability, high qualityMartin Lippert (University of Hamburg), ICSTEST 2002

  • 8/3/2019 SAST 2002-09-19 Managing ET

    8/55

    SAST 2002-09-19Managing ET & Case 2002 Amland Consulting 8

    The eXtreme Programmingand Exploratory Testing Analogy:

    Agile software testing is not conventional (scripted) software testingdone more quickly or done on tippie-toe.

    Exploratory Testing:proven (no single technique is new)

    application oriented

    planned and disciplined

    controllable and reliable

    risk minimizingTwo sides of Exploratory Testing:

    for the tester: freedom, flexibility, fun

    for the manager: controllability, reliability, high quality

  • 8/3/2019 SAST 2002-09-19 Managing ET

    9/55

    SAST 2002-09-19Managing ET & Case 2002 Amland Consulting 9

    ET vs. Scripted Testing

    Fully

    Scripted

    Testing

    Ad-hoc

    Testing

    AutomatedTests

    Bug

    Hunting

    Exploratory Testing

    Jarle Vga (2002)

  • 8/3/2019 SAST 2002-09-19 Managing ET

    10/55

    SAST 2002-09-19Managing ET & Case 2002 Amland Consulting 10

    What is Scripted Testing?Small (but realistic) example:

    How to script and test this login?(Functional tests only not security!)

  • 8/3/2019 SAST 2002-09-19 Managing ET

    11/55

    SAST 2002-09-19Managing ET & Case 2002 Amland Consulting 11

    When to use Exploratory Testing? (1)There is little or no specifications and /or requirements

    You have little or no domain knowledgeYou dont have time to specify, scriptand test

    Uncertainty and Time Pressure!

  • 8/3/2019 SAST 2002-09-19 Managing ET

    12/55

    SAST 2002-09-19Managing ET & Case 2002 Amland Consulting 12

    When to use Exploratory Testing? (2)A common goal of exploration is to probeforweakareasof the program.

    Test teams resource consumption per week:25% of the groups time developing new tests

    50% executing old tests (including bugregression)

    25% on exploratory testing

    Cem Kaner (2001a)

  • 8/3/2019 SAST 2002-09-19 Managing ET

    13/55

    SAST 2002-09-19Managing ET & Case 2002 Amland Consulting 13

    Testing is done in Context1. The value of any practice depends on its context.2. There are good practices in context, but there

    are no best practices.

    3. People, working together, are the most importantpart of any project's context.

    4. Projects unfold over time in ways that are often notpredictable.

    5. The product is a solution. If the problem isn't solved, theproduct doesn't work.

    6. Good software testing is a challenging intellectual process.7. Only through judgment and skill, exercised

    cooperatively throughout the entire project,are we able to do the right things at the righttimes to effectively test our products.

    http://www.context-driven-testing.com/

  • 8/3/2019 SAST 2002-09-19 Managing ET

    14/55

    SAST 2002-09-19Managing ET & Case 2002 Amland Consulting 14

    Presentation Outline

    What is Exploratory Testing?

    How to do Exploratory Testing?

    How to Manage Exploratory Testing?

    High Speed WEB Testing Case Study

  • 8/3/2019 SAST 2002-09-19 Managing ET

    15/55

    SAST 2002-09-19Managing ET & Case 2002 Amland Consulting 15

    Doing Exploratory TestingKeep your mission clearly in mind.

    Keep notes that help you report what you did,

    why you did it, and support your assessmentof product quality.

    Keep track of questions and issues raised inyour exploration.

    To supercharge your testing, pair upwith another tester and test the samething on the same computer at thesame time.

    From Rapid Software Testing, copyright 1996-2002 James Bach

    Charter/Mission

    Notes, Risks, Issues,Questions and Errors

    Execution:Pairs in Sessions

  • 8/3/2019 SAST 2002-09-19 Managing ET

    16/55

    SAST 2002-09-19Managing ET & Case 2002 Amland Consulting 16

    Charter: A clear mission for the sessionA charter may suggest what should be tested, how itshould be tested, and what problems to look for.

    A charter is not meant to be a detailed plan.

    General charters may be necessary at first:Analyze the Insert Picture function

    Specific charters provide better focus, but take moreeffort to design:

    Test clip art insertion. Focus on stress and flow techniques,and make sure to insert into a variety of documents. Wereconcerned about resource leaks or anything else that mightdegrade performance over time.

    From Rapid Software Testing, copyright 1996-2002 James Bach

    Charter/Mission

    Notes, Risks, Issues,Questions and Errors

    Execution:Pairs in Sessions

  • 8/3/2019 SAST 2002-09-19 Managing ET

    17/55

    SAST 2002-09-19Managing ET & Case 2002 Amland Consulting 17

    Taking NotesTest Coverage Outline/Matrix

    Evaluation Notes

    Risk/Strategy ListTest Execution Log

    Issues, Questions & AnomaliesIt would be easier to test if you changed/added

    How does work?Is this important to test? How should I test it?

    I saw something strange

    From Rapid Software Testing, copyright 1996-2002 James Bach

    Charter/Mission

    Notes, Risks, Issues,Questions and Errors

    Execution:Pairs in Sessions

  • 8/3/2019 SAST 2002-09-19 Managing ET

    18/55

    SAST 2002-09-19Managing ET & Case 2002 Amland Consulting 18

    ET DocumentationPlanning:

    Charter overview of what to test (plan)

    Migh be a flip chart on the wall

    Mission What are we looking for?

    ExecutionNotes what happened during testing?

    What did I do? Why did I do it?

    Used to assess product quality after test.Data files input data used for testing

    Bug reports enough details to recreate the test / bug

    Track questions and Issues

    Charter/Mission