Geoff Thompson - Why Do We Bother With Test Strategies
-
Upload
eurostar-software-testing-conference -
Category
Software
-
view
103 -
download
0
Transcript of Geoff Thompson - Why Do We Bother With Test Strategies
Listen | Challenge | Understand | Interpret | Create
Private & Confidential Experimentus Ltd 85 Tottenham Court Road London W1T 4TQ T: +44 (0)870 770 6099 www.experimentus.com
Why do we bother with Test Strategies?Why do we bother with Test Strategies?
Presented by:Geoff Thompson
EuroSTAR 200813th November
Agenda
Drivers for change?
What had to change?
What we did first
The problem
How we made it work
© Experimentus Ltd 2008
2
Drivers for change
© Experimentus Ltd 2008
3
4
© Experimentus Ltd 2008
The Business was demanding a higher
standard to remain competitive
The Business wanted clearer
identification of Risk to quality and Risk
Mitigation
The IT department was under pressure
to do more for less
There was a continual drive to
increase in productivity
- Cost Reduction
- Risk Management
- Improving Quality
- Revenue Enhancement
5© Experimentus Ltd 2008
So they were looking for change to avoid the
wrong solution going live late!
They wanted software that just worked!
6© Experimentus Ltd 2008
What had to change?
7© Experimentus Ltd 2008
How did we find out?
We reviewed many assessment methods
We looked at TPI ® and TMM
Selected TMM because:
– It was a research project developed by Illinois Inst. of Technology
– It describes the processes, which ensure a well planned and controlled test project
– Has very clear 5 levels of test maturity
– Is based on the evolutionary testing model
– Has been empirically validated
– Inspired by the CMM (complementary)
8© Experimentus Ltd 2008
TMM
9© Experimentus Ltd 2008
The 5 levels of TMMi
10
How does it work?
Maturitylevel
Maturitylevel
Indicates Testing Processcapability
Indicates Testing Processcapability
Contains Key process
areas
Contains Key process
areas
Achieves (Sub) GoalsAchieves (Sub) Goals Contains Key practices
Contains Key practices
Describe ImplementationDescribe Implementation
© Experimentus Ltd 2008
For more information see
www.tmmifoundation.org
11© Experimentus Ltd 2008
The recommended improvements
Needed a Test Policy
Needed a Testing Strategy/Framework
Needed a test process definition
Needed a defined Risk Based Approach to Testing
Needed a career path for testers
Needed to complete the roll out of TestDirector
Needed to implement reviews and static analysis
12
So what happened next?
© Experimentus Ltd 2008
What we did first
13© Experimentus Ltd 2008
TESTING POLICY
TESTING DEFINITION Risk Based Testing is the process by which we explore and understand the status of the business benefits and the business risks associated with a release of software or of new or upgraded hardware, by :-
1. Making a prioritised list of risks,
2. Performing tests that explore each risk,
3. As risks evaporate and new ones emerge, adjust the test effort to stay focused on the current crop.
TESTING SYSTEM Development and execution of a test plan in accordance with The Legal & General Test Strategy :-
i. Testing will be organised in line with the Testing V model,
ii. The different levels (phases) of testing will be clearly defined and understood,
iii. Testing will be an integral part of the Project Lifecycle,
iv. Formal test techniques will be used at all levels (phases) of testing,
v. Appropriate training in testing will be provided,
vi. Defined roles and responsibilities and Career paths will be available,
vii. Appropriate Entry and Exit criteria will be defined, communicated and enforced for all test levels (phases),
viii. The Test environment requirements,
ix. Test tools will be used where appropriate,
x. All testware will be maintained under Configuration Management, to enable audit and reuse,
xi. Metrics and measures will be recorded and used to enable test improvement and better estimating for test activity,
xi As well as the process for Incident Management , the definition of fault priority and severity rating will be defined.
MEASUREMENT OF TESTING EFFECTIVENESS Test Effectiveness will be measured by comparing the volume of Priority 1, 2 and 3 faults found in the first 6 months of live running versus the volume of priority 1, 2 and 3 faults found during testing - DEFAULT DETECTION RATE (DDR).
MEASUREMENT OF TEST EFFICIENCY Test Efficiency will be measured by determining the total cost of testing (both Business and BIS time – both planning and execution but excluding fix costs) as a ratio of the total design and build costs (including Project Management) – TEST EFFICIENCY RATE (TER).
TESTING STANDARDS DDR of less than XXXX TER of XXXX or less
TEST PROCESS IMPROVEMENT Quality reviews will take place at key stages in the ‘test projects’ lifecycle.
Specific testing focused Post Project reviews will be held after every release of software or of new or upgraded hardware. ------------------------------------------- ---- -------------------------------------------
We agree to the general principals outlined above, and changes or additions made will be subject to change control.
Manager )________________
Dated 1 st
© Experimentus Ltd 2008
14
Test Tools
Reviews
Generic Test Process
Measures and Metrics
HR
Implementation and
Communications
Risk Based Testing
Corporate Test Strategy
© Experimentus Ltd 2008
15
Corporate Test Strategy
Generic Test Process
Test Tools HR Reviews
Implementation steps
Implementation steps
Implementation steps
Implementation steps
Pros- Quicker delivery- Early buy-in- Earlier benefit realisation
Cons- Rework- Harder to manage Strategic vision- Delivery costs are larger- Loss of confidence due to rework- Support costs are higher
Delivery choices - Vertical
Implementation steps
© Experimentus Ltd 2008
16
Corporate Test Strategy
Generic Test Process
Test Tools HR Reviews
Implementation steps
Implementation steps
Implementation steps
Cons- Slower delivery- Needs commitment and trust
Pros- Minimal rework- Clear strategic view- Clear understanding of impacts between deliveries- Greater acceptance as full picture understood- Quick wins giving early buy-in
Delivery choices - Horizontal
© Experimentus Ltd 2008
17
The problem
18© Experimentus Ltd 2008
Up to this point we had tremendous support
Particularly from the Project and Programme Management Community
BUT
The process and templates just didn't get used to provide the greatest benefit so were in danger of being bypassed by everyone
WHY?
19© Experimentus Ltd 2008
There were two problems in the
implementation of the process improvements:
Testers
Project and Programme Managers
20© Experimentus Ltd 2008
They couldn't plan!
Common issues
– Requirements
– Estimating
– Activity plans
– Resource requests
– Budgeting
Planning was traditionally successful but not effective, when undertaken two weeks before test execution was due to start, when everything was known!
21© Experimentus Ltd 2008
Lessons learnt
Test resources were traditionally put in place long after the project has started, and often just before test execution starts when...
– All requirements are defined and almost built
– Budget has been given rather than asked for
– Resources are set
– Timescales are set, and probably squeezed
– Basically everything is in place to enable accurate plans to be drawn up!
Development Managers provided mentoring to Test Managers using their experience of planning when all that was available was a high level requirement
Test Managers were taught
– High level estimating
– Early Project Planning22© Experimentus Ltd 2008
The Project and Programme Managers
Couldn't understand the deliverables from testing
– Test Strategy?
– Test Plans?
They knew they needed a Test Strategy but didn't know why – ‘we always have one’
They were not clear when each deliverable should be produced
They still didn't invite testing in at the start of a project
23© Experimentus Ltd 2008
Lessons learnt
Align Test deliverables with Project Deliverables
We reintroduced the section for an overview of testing that had been removed from the Prince2 template Quality Plan
Test Policy remained – Test Strategy abolished and replaced with a documented Test Process, and a Master Test Plan
Where appropriate rename Test Deliverables to match Project Deliverable
So lets see what we did to achieve this
24© Experimentus Ltd 2008
Testing Deliverable Breakdown
25© Experimentus Ltd 2008
Master Test Plan
Component Test /Component
Integration Test
WorkpackagePlan
SystemIntegration Test
System Test /Business
Acceptance Test
OperationalAcceptance Test
Co-ExistenceTest
ProductionAcceptance Test
TestSpecification
Tested/IntegratedComponents
Actual Results
WorkpackagePlan
TestSpecification
IntegratedSystems
Actual Results
Test ClosureReport
Test Incidents
WorkpackagePlan
TestSpecification
Tested Solution
Actual Results
Test ClosureReport
Daily IncidentReport
Daily ProgressReport
Test Incidents
Team Plan
TestSpecification
OperationallyTested Solution
Actual Results
Test ClosureReport
Daily IncidentReport
Daily ProgressReport
Test Incidents
Team Plan
TestSpecification
Co-ExistenceTested Solution
Actual Results
Test ClosureReport
Daily IncidentReport
Daily ProgressReport
Test Incidents
Team Plan
TestSpecification
TestedImplementation
Actual Results
Test ClosureReport
Test Incidents
Handover Report
Weekly TestSpecificationStatus Report
Early Test Planning – the inputs and outputs
Customer’s QualityExpectations
Acceptance Criteria
Quality Approachto Project Deliverables
Quality Plan Summary
Approved Testing Budget
Project Plan
Project Controls
Project Risks
Inp
uts
to
Tes
t P
lan
nin
g
Testing Budget
Test Stages forProject Plan
Project Controls
Testing Risks
Original idea and estimate produced
High Level Terms of Reference
Project Initiation Document
Quality Plan
ApprovedBusiness Case
Business Requirements
Testing EstimateIn
pu
t fr
om
Tes
tin
g
Test Policy
Test Managers Roles and
Responsibilities
EstimatingMetrics
© Experimentus Ltd 2008
26
Master Test Planning - the inputs and outputs
MTP or PID
PID
Master Test Plan
Acceptance Test Work Package
Team Plan
OperationalAcceptance Test Work Package
System Integration Test Work Package
System Test Work Package
Component Integration TestWork Package
Component Test Work Package
Team Plan
Team Plan
Team Plan
Team Plan
Team Plan
Risk Based TestingAssessment
Customer’s QualityExpectations
Acceptance Criteria
Quality Approachto Project Deliverables
Quality Plan Summary
Approved Testing Budget
Project Plan
Project Controls
Project Risks
Inp
uts
to
Tes
t P
lan
nin
g
27
© Experimentus Ltd 2008
Learning styles
Having a process isn't the full answer though!
Humans will initially learn the process verbatim, and will expect significant help to be able to adapt the processes to their environment
We first have to learn, before we can be agile with our use of the process
This takes time, which is why the process improvement experts suggest an 18 month timeline before process and culture come together to be at their most efficient
28
© Experimentus Ltd 2008
Reflect ive Observation Understanding
Concrete Experience
Following the rules
Active Experimentation
Doing
Abstract Conceptualisat ion
Thinking
Learning styles
This is where no real evaluation of whether the
process is right or not occurs
At this point the process usage is
second nature, with all users being able to evaluate what is
right and wrong.
29
© Experimentus Ltd 2008
No matter how adaptable to change people are, there will always be cultural issues which if not managed will inhibit the benefits of the change being realised
Remember...
© Experimentus Ltd 2008
That to make any process change work, agility in its delivery is the key – its no good enforcing change that no one can understand or is able to use
En
thu
siasm
Maximum Hype
Initial Invention
ActualUse
Dis -illusion-
ment
PracticalBenefit
RIP
30
Thank You Any questions?
© Experimentus Ltd 2008
31