© Workroom Productions 2002 Slide 1 Further Adventures in Session-based Testing James Lyndsay...

© Workroom Productions 2002

www.workroom-productions.com

Further Adventures in Session-based Testing

James LyndsayWorkroom Productions

[email protected]



?What?



Session-Based Testing?

Manages and controls unscripted testing

• Limited Duration• Directed Exploration• Recorded Activities

Supports ‘Agile’ ProcessAllows quick metricsHas wider uses



Adventures?

Two case studies

• Permanent team - • 20 coders, 4 testers• Inexperienced and under-motivated• Internet-based product, ASP model

• Part-time team• 2-6 coders, 0-3 testers bought in from pool of 6-8• No permanent test team• Windows client / server product around known

applicationShared business context

• Rapid change• Commercial constraints; budget, time-to-

market



Further?

Paper - Adventures in Session-based Testing

• Contains detailed methods• Contains problems and (some)

solutions• Now a year old

STARWest 2002 -Further Adventures . . .

• Methods• Tools• Training, Coaching



Strategy

Stay within business constraintsDeal with change rather than enforce stasisRapidly transfer knowledge to allow

independenceFour goals

• Control the scope• Control the work• Measure risk and coverage• Promote ongoing improvement


www.workroom-productions.com1

Methods



Methods: Overview

Control and recording

• Control Scope• Control Work• Recording a Session

Assessment and Measurement

• Assessing Risk• Assessing Coverage

Infrastructure

• Real-time Metrics• Social techniques



Example: Test Points (Excel-based tool)

Test / Test Group ID Description/instructions FC L RiskEstimated

cost Who Date planned Status Actual cost

SM-I-W20F20E136 Smoke test for release E1.3.6 H L M 0.8 James 7-Aug planned

STR-01 Stress test: Multiple users in one doc - automated H L M 1.0 James 7-Aug plannedSTR-02 Stress test: Multiple users in one doc H H H 4.0 Many 7-Aug plannedSTR-03 Stress test: One user in multiple docs - less important H L M 1.0 James 7-Aug plannedSTR-04 Stress test: Many users observing and working in many documentsH H H 4.0 Many 7-Aug planned

AT-01 Audit trail (both) presentation, usability, interaction with explorer, word, othersL H M 2.0 James 7-Aug plannedAT-02 Audit trail recording events within a doc H L M 0.5 James 7-Aug planned

NC-01 Narrowcast - reproduce and vary new bug H L M 1.0 James 7-Aug plannedNC-02 Further narrowcast tests (may be covered by stress test) H L M 1.0 James 7-Aug left out this time

UE-01 Test customized template (once delivered) H L M 1.0 James 7-Aug plannedUE-02 Test 'Get my doc into LD' use case H L M 1.0 James 7-Aug plannedUE-03 Test 'Get my doc out of LD' case H L M 1.0 James 7-Aug plannedUE-04 Test 'Insert a BLOB' use case H L M 1.0 James 7-Aug plannedUE-05 Test 'Bullets and numbering' use case H L M 1.0 James 7-Aug plannedUE-06 Test 'Setting permissions' use case H L M 1.0 James 7-Aug planned

WR-01 Single-user wrapper tests - automated for coverage H L M 1.0 James 7-Aug plannedWR-02 Tabs Indents H H H 1.0 James 7-Aug plannedWR-03 Bullets and numbers - also see enh #32 L H M 1.0 James 7-Aug plannedWR-04 User Settings - change/restore L H M 2.0 James 7-Aug plannedWR-05 Usability, interaction of error messages and conditions L H M 1.0 James 7-Aug plannedWR-06 Test effect of error conditions on wrapper L H M 2.0 James 7-Aug planned

BLB-01 Inserting Word BLOB - automate for coverage H L M 1.0 James 7-Aug plannedBLB-02 Editing § around BLOBs H L M 1.0 James 7-Aug plannedBLB-03 Editing BLOB contents H H H 1.0 James 7-Aug plannedBLB-04 Editing BLOB - copy/paste - automate for coverage H H H 1.0 James 7-Aug planned



Methods: Control

Scope - made of Test Points

• Test Point - piece of work• Risk, Estimated cost• Four groups

• New tests• Regression tests• Retests for fixes• Investigatory tests

Work done - made of Test Sessions

• Test Session - piece of time • Actual cost, Coverage



Example: A Test Session

Test Description

Test ID Risk Date + time done

Your name IDs, Machines, Files and other resources used

Estimated time

Actual time

How much more testing does this need?

% complete?

Plan

Notes



Methods: Control

Recording a Test Session

• Plan: Approach, Scope, Expectations• Record: Conditions, Actions,

Observations• Bug details• Actual Cost• How done are we? Coverage

Allows

• Review, coaching• Forgetting• Retrospectives



Methods: Assessment

Risk

Coverage

• Ask the Experts - the testers!• How done are you? How much is left?• Collect many subjective few objective

assessments • Overall figures can go down as well as up• Ongoing learning - estimates improved

Likelihood of failureRisk High Low

Cost of failure High 3 2Low 2 1



Example: Metrics (Excel-based tool)We are 95% complete38% of our tests have been successfulWe think we have 7 hours left

No. failed tests 55No. still not fixed #NAME?No. fixed not tested 1

% planned tests complete 95%Time taken so far 125.15Est. time for these 100.7

total leftPre-test estimate of time 105.7 -19.45Post-test new guess 132 7

All tests Planned tests Completed Tests Successful CompletionEst. Actual %Done %Successful

Number Hours Number Hours Number Hours spent Number Number Number Total 139 182.5 94 92.6 89 106.1 34 95% 38%

H High risk 26 20.1 23 17.1 20 18.9 6 87% 30%M Med risk 68 80.4 48 46 46 54.5 18 96% 39%L Low risk 45 82 23 29.5 23 32.8 10 100% 43%

All retests Planned retests Fix tested Successfully fixed CompletionActual %Done %Successful

Number Hours Number Hours Number Hours spent Number Number Number Total 64 13.3 63 13.1 63 19.05 49 100% 78%

The spreadsheet looks consistent



Methods: Support

Real-time metrics

• Immediate feedback• State of testing, not of system• Fast, frequent, informed re-planning

Soft skills/Social Techniques

• Experimental Approach• Ongoing learning• Communication• Empowerment


www.workroom-productions.com2Tools



Tools

Freely available - under GPLExcel spreadsheet

• Single-user, single location• Download from:

Web: PHP/MySQL

• Multiple user• Download from:


www.workroom-productions.com3

Training and coaching



Ongoing learning and development

Training - from unskilled to skilled

• Skilled involvement - pair expert / novice

• Show - practice - challenge• Careful choice of sessions

Coaching - from good to great!

• Peer involvement, neutral assessment• Challenge - innovate• Discussion, metrics, personal

investigation



Supporting the learning process

Session-based testing

• Introduces discipline and framework• Helps overlap training and testing• Helps match challenge to individual

Experts, managers - get involved!Sometimes compromise:

• Good bugs found quickly, or• Better testers long-term

Feedback is integral

• Before testing - techniques, planning, charter

• After testing - session log, bug log• Stats and metrics - not just for management



Encouraging the team

Involve the test team

• Allocate responsibility / ownership for tests

• Allow direct influence on clear, open metrics

• Relate metrics to business decisionsShare successes, failuresReward discoveryDiscourage blind faith, boredomStep away - look at the wider picture



Changes: Case Study 1

Active learning process is encouraged

• Feedback has improved estimating skills• Session review has improved test techniques• Test Point ownership has improved planning

and scoping skills• Weekly 2-hour self-training session as a team• Greater interest in testing, greater motivation

Useful, visible, up-to-date metrics

• Increase openness, reduce surprise, blame• Joint ownership of decisions, successes,

failures


www.workroom-productions.com!

Conclusion



Conclusion

Communication

• Sessions gave us the tools to communicate• Better communication improved trust and

effectivenessOngoing improvement

• Sessions allow directed and disciplined approach• Feedback improved techniques and estimation• Empowerment increased morale and confidence

Openness

• All figures available, all of the time, always current• Encouraged trust and interest• Allowed neutral feedback



Conclusion

Test sessions provide:Control of unscripted testing

Framework for ongoing improvement

Overall lessons

• Simple measures are the best• Favour effective communication and

involvement over knee-jerk documentation

• Unobtrusive, immediate metrics allow real-time control and feedback



More

Further details:

• Paper, updates, references etc.• www.workroom-productions.com

• Contact: • James Lyndsay

• [email protected]

© Workroom Productions 2002 Slide 1 Further Adventures in Session-based Testing James Lyndsay...

Documents

Transcript of © Workroom Productions 2002 Slide 1 Further Adventures in Session-based Testing James Lyndsay...