Improving the Test Process T-Systems
-
Upload
billy-decy-william -
Category
Documents
-
view
12 -
download
0
description
Transcript of Improving the Test Process T-Systems
Test Process Improvementusing the TPI®-Next Model:
Graham Bath
Graham Bath ANZTB Test 2010V.1.0 Page 2 SOFTEC 2010 Graham Bath© T-Systems GEI, PDC Test, 2010
Trademarks and ReferencesTrademarks
The following registered trademarks and service marks are used in this tutorial:
CMM ®, CMMI ®, TMMi ®, IDEALSM, ISTQB ®, TPI ®
CMM and CMMI are registered in the U.S. Patent and Trademark Office by Carnegie Mellon University.
IDEAL is a service mark of Software Engineering Institute (SEI), Carnegie Mellon University
ISTQB is a registered trademark of the International Software Testing Qualifications Board
TMMi is a registered trademark of TMMi Foundation
TPI is registered trademark of Sogeti Nederland B.V.
References
[TPI Next] TPI Next: Business driven test process improvement, de Vries et al, UTN Publishers, ISBN 9789072194978
[Expert ITP] ISTQB Expert level syllabus, “Improving the test process”, Graham Bath, Erik van Veenendaal,Isabel Evans (available from www.istqb.org)
[Craig02] “Systematic Software Testing”, Craig, Rick David; Jaskiel, Stefan P., Artech House, 2002, ISBN: 1-580-53508-9
[Black03] “Critical Testing Processes”, Rex Black, Addison-Wesley, 2003, ISBN: 0-201-74868-1
Graham Bath ANZTB Test 2010V.1.0 Page 3 SOFTEC 2010 Graham Bath© T-Systems GEI, PDC Test, 2010
Test Process Improvement with TPI-Next
Contents.
IntroductionSetting the scene for test process improvementUsing models for improvementA closer look at the TPI®-Next ModelThe improvement cycleUsing TPI-NextCritical Success FactorsThe ISTQB® Expert Level syllabus “ITP”Key messages
Graham Bath ANZTB Test 2010V.1.0 Page 4 SOFTEC 2010 Graham Bath© T-Systems GEI, PDC Test, 2010
Setting the Scene.How well do we perform test processes?
Common complaints:
We find the faults too late
Testing is too expensive
Testing isn’t systematic enough
Something’s wrong with our testing
What the testing organization does is a mystery to me
Graham Bath ANZTB Test 2010V.1.0 Page 5 SOFTEC 2010 Graham Bath© T-Systems GEI, PDC Test, 2010
Setting the Scene.What is “improvement”?
Many of these common complaints are symptoms.
The causes can be varied, but more often than not, the test process is in need of improvement.
What is “improvement”?
It depends on who you ask.CustomersManagersUsersOperations staffTesters
We need to consider all of these people (stakeholders)
Graham Bath ANZTB Test 2010V.1.0 Page 6 SOFTEC 2010 Graham Bath© T-Systems GEI, PDC Test, 2010
Test Process Improvement with TPI-Next
Contents.
IntroductionSetting the scene for test process improvementUsing models for improvementA closer look at the TPI®-Next-ModelThe improvement cycleUsing TPI-NextCritical Success FactorsThe ISTQB Expert Level syllabus “ITP”Key messages
Graham Bath ANZTB Test 2010V.1.0 Page 7 SOFTEC 2010 Graham Bath© T-Systems GEI, PDC Test, 2010
Using models for test process improvement
“Best practices” in software testing described as a model
The assumption is: “Follow the model and you will improve”
Different levels of test process maturity are defined
Very bad„initial“
Very good„optimal“
Use the model to find out
Where the test process is now
Where should it be improved
What steps to take
Graham Bath ANZTB Test 2010V.1.0 Page 8 SOFTEC 2010 Graham Bath© T-Systems GEI, PDC Test, 2010
Using models for test process improvementTwo principal types of model can be used
1. Process modelsStep-by-step approachSteps to be taken are defined by the model
Examples:Test Maturity Model Integrated (TMMi)Test Process Improvement Model (TPI-Next)
2. Content modelsDefine specific activities which can benefit a test process if applied well Do not require that improvements occur in a specific order
Examples:The Systematic Test and Evaluation Process (STEP), [Craig02] Critical Testing Process (CTP), [Black03]
Graham Bath ANZTB Test 2010V.1.0 Page 9 SOFTEC 2010 Graham Bath© T-Systems GEI, PDC Test, 2010
Test Process Improvement with TPI-Next
Contents.IntroductionSetting the scene for test process improvementUsing models for improvementA closer look at the TPI®-Next-ModelThe improvement cycleUsing TPI-NextCritical Success FactorsThe ISTQB Expert Level syllabus “ITP”Key messages
Graham Bath ANZTB Test 2010V.1.0 Page 10 SOFTEC 2010 Graham Bath© T-Systems GEI, PDC Test, 2010
Test Maturity Matrix
Structure of the TPI-Next Model.
Key AreasC
lusters
Improvementsuggestions Enablers
Maturity Levels
Checkpoints
Go to contents
Graham Bath ANZTB Test 2010V.1.0 Page 11 SOFTEC 2010 Graham Bath© T-Systems GEI, PDC Test, 2010
Key Areas represent different aspects of the test Process
StakeholderRelations
TestManagement
TestProfession
Reporting
Communication
Test organization
Test strategy
Degree ofinvolvement
Stakeholdercommittment
Test processmanagement
Estimating andplanning
Metrics
Defectmanagement
Testwaremanagement
Methodologypractice
Testerprofessionalism
Test case design
Test tools
Test environment
DetailsAppendix D
Graham Bath ANZTB Test 2010V.1.0 Page 12 SOFTEC 2010 Graham Bath© T-Systems GEI, PDC Test, 2010
Maturity levels
Controlled
Initial
Efficient
Optimizing
Stages of increasing test process maturity
Key areas can be assessed at any maturity level
Each maturity level of a given key area:is achieved by fulfilling all specific checkpointshas associated improvement suggestionshas associated enablers
„Doing the right things“
„Ad-hoc activities“
„Doing things the right way “
„Continuously adapting toever-changing circumstances“
Graham Bath ANZTB Test 2010V.1.0 Page 13 SOFTEC 2010 Graham Bath© T-Systems GEI, PDC Test, 2010
Checkpoints
Stakeholdercommittment 01.c.1 The principal stakeholder is defined
(not necessarily documented) and known to the testers.
01.c.2 Budget for test resources is granted by and negotiable with the principal stakeholder.
01.c.3 Stakeholders actually deliver the committed resources.
01.c.4 The principal stakeholder is responsible for a documented product risk analysis (the input for the test strategy).
Checkpoints for Controlled
objective statementscan be confirmed by a simple ‘yes’ or ‘no’.
TPI NEXT contains:157 Checkpoints across16 Key Areas and 3 Maturity Levels.
Key Area 1:
Source : [TPI Next]
Graham Bath ANZTB Test 2010V.1.0 Page 14 SOFTEC 2010 Graham Bath© T-Systems GEI, PDC Test, 2010
Clusters
It makes no sense to improve the test process by proceeding sequentially according to key area (i.e. improving key area 1, then key area 2 etc.)
Clusters are defined to help us to improve the test process in logical steps
Clusters define an improvement path for test process improvementcan also be defined by the user to place emphasis on specific business goals (Business-driven approach)
Graham Bath ANZTB Test 2010V.1.0 Page 15 SOFTEC 2010 Graham Bath© T-Systems GEI, PDC Test, 2010
Clusters
A cluster is a group of related checkpoints across different key areasTPI-Next defines 13 “base” clusters which relate directly to the test processThe clusters are shown on a Test Maturity matrix (here, just a section is shown. More on the Test Maturity Matrix follows later) Cluster A, for example, contains 12 checkpoints from 8 different key areas
Source : [TPI Next]Source : [TPI Next]
Graham Bath ANZTB Test 2010V.1.0 Page 16 SOFTEC 2010 Graham Bath© T-Systems GEI, PDC Test, 2010
ClustersExample for base cluster A
KA Checkpoint1 01.c.1 The principal stakeholder is defined (not necessarily documented) and known to the testers.2 02.c.1 The test assignment, scope and approach are negotiated early with the principal stakeholder
as one of the first test activities.3 03.c.1 The principal stakeholder agrees with the documented test strategy.3 03.c.2 The test strategy is based on a product risk analysis.4 04.c.1 People involved know where to find the persons (or department) responsible for test services.6 06.c.1 The reporting contains aspects of time and/or costs, results and risks.7 07.c.1 At the start of the test project a test plan is created. The test plan includes at least the test assignment,
the test scope, the test planning, the roles and responsibilities.7 07.c.2 The test plan is agreed with the principal stakeholder.10 10.c.1 The defect lifecycle is defined (including a retest) and applied10 10.c.2 The following items are recorded for each defect: unique ID, related test case ID (if applicable),
person reporting the defect, date, severity category, description (the actions to reproduce the defect, expected and observed result) and defect status.
14 14.c.1 The test cases are recorded on a logical level.14 14 c.2 The test cases consist of a description of: a) initial situation, b) change process = test actions to be performed,
c) predicted result.
Base Cluster A contains 12 checkpoints from 8 different key areas (KA)Achieving these checkpoints is a first step away from initial “ad-hoc” testing
Source : [TPI Next]
Graham Bath ANZTB Test 2010V.1.0 Page 17 SOFTEC 2010 Graham Bath© T-Systems GEI, PDC Test, 2010
Test Maturity Matrix
BenefitsVisual representation of the overall test processEasy to show what the current maturity is across all key areasPlanning and showing improvement paths Provides a instrument for discussion with stakeholders
Two different viewsCheckpoint viewCluster view
Graham Bath ANZTB Test 2010V.1.0 Page 18 SOFTEC 2010 Graham Bath© T-Systems GEI, PDC Test, 2010
Test Maturity Matrix: Checkpoint view
Checkpoint view to show achieved maturity (all checkpoints shown)
The matrix shown above is blankCurrent maturity is shown by coloring in the checkpoints achieved
Source : [TPI Next]
Graham Bath ANZTB Test 2010V.1.0 Page 19 SOFTEC 2010 Graham Bath© T-Systems GEI, PDC Test, 2010
Test Maturity Matrix: Cluster view
Clusters define an improvement path for test process improvement
The matrix shown above is blankSource : [TPI Next]
Graham Bath ANZTB Test 2010V.1.0 Page 20 SOFTEC 2010 Graham Bath© T-Systems GEI, PDC Test, 2010
Enablers
Clarify how other SDLC processes can support improvement of the test process Creates a link to other software process improvement (SPI) models such as CMMI and ISO 15504
Example
Test strategy, Controlled : Use risk management for product risks
Graham Bath ANZTB Test 2010V.1.0 Page 21 SOFTEC 2010 Graham Bath© T-Systems GEI, PDC Test, 2010
Improvement suggestions
Tips on how to achieve a particular maturity level for a given key areaExample (summarized)
Locate the person who orders test activities or is heavily dependant on test resultsResearch the effect of poor testing on production and make this visible to stakeholders. Show which defects could have been found earlier by testing. Indicate what the stakeholders could have done to avoid the problems.Keep to simple, practical examples.Focus on “quick-wins”
Stakeholder commitment, Controlled
Graham Bath ANZTB Test 2010V.1.0 Page 22 SOFTEC 2010 Graham Bath© T-Systems GEI, PDC Test, 2010
Test Process Improvement with TPI-Next
Contents.
IntroductionSetting the scene for test process improvementUsing models for improvementA closer look at the TPI-Next®-ModelThe improvement cycleUsing TPI-NextCritical Success FactorsThe ISTQB Expert Level syllabus “ITP”Key messages
Graham Bath ANZTB Test 2010V.1.0 Page 23 SOFTEC 2010 Graham Bath© T-Systems GEI, PDC Test, 2010
The improvement cycle
Activities.
Identify the need and objectivesPlan and prepare the assessmentPerform the assessment (interviews)Analyse results and make recommendationsCreate and present assessment reportEstablish an improvement planMonitor progress, learn and set new objectives
Graham Bath ANZTB Test 2010V.1.0 Page 24 SOFTEC 2010 Graham Bath© T-Systems GEI, PDC Test, 2010
The need for improvement often results froma negative event (e.g. production failure)a negative realization (e.g. “our testing is not effective”)a planned activities (e.g. improvement is part of company policy)
The need for improvement results in the definition of objectives, e.g.More efficiency in all aspects of the test process.Generally lower costs (but not at the expense of quality!)Better transparency of achieved test coverage.More effective use of available testing staff. It may be sufficient to just say „improve on our current situation“?
The TPI-Next model allows key areas to be priorized to account for specific business objectives
Identify the need and objectives
Graham Bath ANZTB Test 2010V.1.0 Page 25 SOFTEC 2010 Graham Bath© T-Systems GEI, PDC Test, 2010
Capture objectives in the TPI-Maturity Matrix
Objectives are shown here in blue
For example: Establish testing at the “controlled” maturity level
Graham Bath ANZTB Test 2010V.1.0 Page 26 SOFTEC 2010 Graham Bath© T-Systems GEI, PDC Test, 2010
Identify the need and objectivesPlan and prepare the assessmentPerform the assessment (interviews)Analyse results and make recommendationsCreate and present assessment reportEstablish an improvement planMonitor progress, learn and set new objectives
The improvement cycle
Activities.
Graham Bath ANZTB Test 2010V.1.0 Page 27 SOFTEC 2010 Graham Bath© T-Systems GEI, PDC Test, 2010
Plan the assessment
Week 1 Week 2 Week 3 Week 4
Establish objectivesand set scope
Plan & Prepare forassessment
Perform Assessment
Analyse results andmake
recommendations
Create and present
assessmentreport
1 day
2 days
Total effort:10 days
2 days
1 day
4 days
Typical example
Graham Bath ANZTB Test 2010V.1.0 Page 28 SOFTEC 2010 Graham Bath© T-Systems GEI, PDC Test, 2010
Separate interviews for stakeholders:
EssentialTest manager 2 hours (minimum)Tester 1 1,5 hours (minimum)Tester 2 1,5 hours
Optional (1 hour each, as required)Test environment managerProject leaderRelease managerAutomation specialists
Plan the interviews
Where possible:In this sequence.On the same day
Graham Bath ANZTB Test 2010V.1.0 Page 29 SOFTEC 2010 Graham Bath© T-Systems GEI, PDC Test, 2010
Prepare for the assessment interviewsMaybe you can review some test documents beforehand
Test plansTest specifications
Get some opening questions preparedPrevent “black-outs”
Create summaries of the checkpointsMindmapsKeywordsPath diagrams (sequence of key areas)
Graham Bath ANZTB Test 2010V.1.0 Page 30 SOFTEC 2010 Graham Bath© T-Systems GEI, PDC Test, 2010
Identify the need and objectivesPlan and prepare the assessmentPerform the assessment (interviews)Analyse results and make recommendationsCreate and present assessment reportEstablish an improvement planMonitor progress, learn and set new objectives
The improvement cycle
Activities.
Graham Bath ANZTB Test 2010V.1.0 Page 31 SOFTEC 2010 Graham Bath© T-Systems GEI, PDC Test, 2010
Perform the assessment (interviews)Cover all checkpoints for all key areas at the Controlled maturity level
Ask the right questions
Ask the questions right Interviewing skills are essential!Open questions are better than closed questionsConduct a conversion, not an inquisition
React to events and the answers you getControl the interview but be flexible
Don’t be afraid to deviate from the modelIf it doesn’t make sense to ask a question, don’t ask itIf it makes sense to ask an additional question, ask it
Graham Bath ANZTB Test 2010V.1.0 Page 32 SOFTEC 2010 Graham Bath© T-Systems GEI, PDC Test, 2010
Identify the need and objectivesPlan and prepare the assessmentPerform the assessment (interviews)Analyse results and make recommendationsCreate and present assessment reportEstablish an improvement planMonitor progress, learn and set new objectives
The improvement cycle
Activities.
Graham Bath ANZTB Test 2010V.1.0 Page 33 SOFTEC 2010 Graham Bath© T-Systems GEI, PDC Test, 2010
Analyze results
The TPI-Next Model can helpScoring toolBuilt-in improvement suggestions
This does not replace the skills of the test process improverAbility to take project-specific aspects into account Can adapt to different development life-cycles (sequential, agile)
Graham Bath ANZTB Test 2010V.1.0 Page 34 SOFTEC 2010 Graham Bath© T-Systems GEI, PDC Test, 2010
Use the TPI-Matrix (Checkpoint view) to show the assessment results
No key areas achieve controlled maturity at presentOnly the stakeholder commitment is close to the controlled levelFour key areas are still at the ad-hoc “initial” level
Graham Bath ANZTB Test 2010V.1.0 Page 35 SOFTEC 2010 Graham Bath© T-Systems GEI, PDC Test, 2010
Identify the need and objectivesPlan and prepare the assessmentPerform the assessment (interviews)Analyse results and make recommendationsCreate and present assessment reportEstablish an improvement planMonitor progress, learn and set new objectives
The improvement cycle
Activities.
Graham Bath ANZTB Test 2010V.1.0 Page 36 SOFTEC 2010 Graham Bath© T-Systems GEI, PDC Test, 2010
Create an assessment report
Minimum content
Management summaryTPI-Matrix showing objectives and achieved maturity levelResults for each key area
What was goodWhat could be improved
Proposed improvement stepsSpecific recommendations
PriorityCost of implementationImpact on specific objectives
A test improvement plan is usually agreed using the recommendations
Graham Bath ANZTB Test 2010V.1.0 Page 37 SOFTEC 2010 Graham Bath© T-Systems GEI, PDC Test, 2010
Assessment report: Results for each key area
Positive aspectsA test manager role is defined for this projectA number of testers are availableA project hand book defines the roles and responsibilitiesAll testing staff have in-depth knowledge of the application
Problem areasNo specific test trainingTesting skills are not evaluatedTest managers are relatively inexperienced and have no formal training
Recommendations
For example: Key Area: Tester professionalism, Level: controlled
Will be completed later
Graham Bath ANZTB Test 2010V.1.0 Page 38 SOFTEC 2010 Graham Bath© T-Systems GEI, PDC Test, 2010
Propose improvement steps Use the TPI-Matrix (Cluster view)
Step1: Achieve remaining checkpoints for cluster B
Graham Bath ANZTB Test 2010V.1.0 Page 39 SOFTEC 2010 Graham Bath© T-Systems GEI, PDC Test, 2010
Step2: Achieve remaining checkpoints for cluster C
Propose improvement steps Use the TPI-Matrix (Cluster view)
Graham Bath ANZTB Test 2010V.1.0 Page 40 SOFTEC 2010 Graham Bath© T-Systems GEI, PDC Test, 2010
Step3: Achieve checkpoints for cluster D
Propose improvement steps Use the TPI-Matrix (Cluster view)
Graham Bath ANZTB Test 2010V.1.0 Page 41 SOFTEC 2010 Graham Bath© T-Systems GEI, PDC Test, 2010
Step4: Achieve Controlled maturity level by completing checkpointsfor cluster E
Propose improvement steps Use the TPI-Matrix (Cluster view)
Graham Bath ANZTB Test 2010V.1.0 Page 42 SOFTEC 2010 Graham Bath© T-Systems GEI, PDC Test, 2010
Add recommendations to the assessment report
Positive aspectsA test manager role is defined for this projectA number of testers are availableA project hand book defines the roles and responsibilitiesAll testing staff have in-depth knowledge of the application
Problem areasNo specific test trainingTesting skills are not evaluatedTest managers are relatively inexperienced and have no formal training
Testers should complete the ISTQB Foundation Level training Test managers should complete the ISTQB Advanced Level training within the next 18 monthsLine management should employ a consultant to evaluate testing skills each yearThe training will cost approximately 20,000 USD
Recommendations
For example: Key Area: Tester professionalism, Level: controlled
Graham Bath ANZTB Test 2010V.1.0 Page 43 SOFTEC 2010 Graham Bath© T-Systems GEI, PDC Test, 2010
Tips for proposing recommendationsGather all recommendations into a single list
Prioritize Which recommendations do you consider urgent?The customer sets the final priority
Costs, benefits Will it have any noticeable effect?Why would I want to implement
Estimate implementation risks
Estimate chances of success
Indicate when payback can be expected (short term / long term)
Graham Bath ANZTB Test 2010V.1.0 Page 44 SOFTEC 2010 Graham Bath© T-Systems GEI, PDC Test, 2010
Identify the need and objectivesPlan and prepare the assessmentPerform the assessment (interviews)Analyse results and make recommendationsCreate and present assessment reportEstablish an improvement planMonitor progress, learn and set new objectives
The improvement cycle
Activities.
Graham Bath ANZTB Test 2010V.1.0 Page 45 SOFTEC 2010 Graham Bath© T-Systems GEI, PDC Test, 2010
Establish an improvement plan
Sets priorities
Obtain resources IndividualsImprovement team
Plan the improvement activitiesMilestonesDeliverablesMeasures for success
Graham Bath ANZTB Test 2010V.1.0 Page 46 SOFTEC 2010 Graham Bath© T-Systems GEI, PDC Test, 2010
Identify the need and objectivesPlan and prepare the assessmentPerform the assessment (interviews)Analyse results and make recommendationsCreate and present assessment reportEstablish an improvement planMonitor progress, learn and set new objectives
The improvement cycle
Activities.
Graham Bath ANZTB Test 2010V.1.0 Page 47 SOFTEC 2010 Graham Bath© T-Systems GEI, PDC Test, 2010
Monitor progress, learn and set new objectivesThe situation at a later date could be:
Controlled maturity has now been achieved
Our overall objective now is to achieve the Efficient maturity level
Repeat the cycle
Graham Bath ANZTB Test 2010V.1.0 Page 48 SOFTEC 2010 Graham Bath© T-Systems GEI, PDC Test, 2010
Test Process Improvement with TPI-Next
Contents.IntroductionSetting the scene for test process improvementUsing models for improvementA closer look at the TPI-Next®-ModelThe improvement cycle Using TPI-NextCritical Success FactorsThe ISTQB Expert Level syllabus “ITP”Key messages
Graham Bath ANZTB Test 2010V.1.0 Page 49 SOFTEC 2010 Graham Bath© T-Systems GEI, PDC Test, 2010
Practical session: Contents.
Background and expectationsAssessment planInterviewsResults and analysisReport
Background and expectationsT-Bank is a new on-line banking system which was started six months ago. and the first delivery of software has already been tested. It is planned that the system will be operational within two years. Company standards require that a first test process assessment using the TPI-Next model should now be conducted. It is expected that the test process has achieved cluster A in the TPI-Next model.It is considered desirable (“nice to have”) at this stage to have at least partially completed subsequent clusters B and C. At this early stage, clusters beyond C are not considered to be in scope, but a perspective should be given on how to improve before the next assessment takes place in six months.The company standards require that a controlled test process is established by the time the next assessment takes place.The base practices of the test process are in focus for this assessment. The key areas are all set to “neutral” priority.
Objectives shown on the TPI-Next Maturity MatrixInitial
1 Stakeholder commitment2 Degree of involvement3 Test strategy4 Test organization5 Communication6 Reporting7 Test process management8 Estimating and planning9 Metrics
10 Defect management11 Testware management12 Methodological practice13 Tester professionalism14 Test case design15 Test tools16 Test environment
Cluster A "must achieve"Cluster B "nice to have"Cluster C "even nicer to have"
Key Area Controlled
We need to achieve „controlled“ before the next assessment: this sets our overall scope for the current assessment
Graham Bath ANZTB Test 2010V.1.0 Page 52 SOFTEC 2010 Graham Bath© T-Systems GEI, PDC Test, 2010
Practical session: Contents.
Background and expectationsAssessment planInterviewsResults and analysisReport
Graham Bath ANZTB Test 2010V.1.0 Page 53 SOFTEC 2010 Graham Bath© T-Systems GEI, PDC Test, 2010
Assessment planFeb.25 March 1 March 8 March 15
Establish objectivesand scope
Prepare forAssessment
Createassessment
report
1 day
3 days
Total effort:11,5 daysTalk to the customer(Mr. Primbo) and withinternal quality management
Perform Assessments:•Test manager•Tester
Make arrangementsGather templates for assessment Perform required tailoring
1/2 day
3 days
Preliminary analysisand recommendations
3 days Detailed analysisand recommendations
Presentation(1 day)
March 19
Graham Bath ANZTB Test 2010V.1.0 Page 54 SOFTEC 2010 Graham Bath© T-Systems GEI, PDC Test, 2010
Practical session: Contents.
Background and expectationsAssessment planInterviewsResults and analysisReport
Graham Bath ANZTB Test 2010V.1.0 Page 55 SOFTEC 2010 Graham Bath© T-Systems GEI, PDC Test, 2010
Information to participantsI will read through an interview which I had with Mr.Testwell, the Test Manager.
Please listen carefully to the interview.
Make notes as you go along if you want to.
At the end of the interview, we will judge how good the test process is, based on what we learned from Mr. Testwell.
We will need the red and the green cards for that.
Graham Bath ANZTB Test 2010V.1.0 Page 56 SOFTEC 2010 Graham Bath© T-Systems GEI, PDC Test, 2010
Interview with Mr.Testwell (Test Manager)
Script
Graham Bath ANZTB Test 2010V.1.0 Page 57 SOFTEC 2010 Graham Bath© T-Systems GEI, PDC Test, 2010
Practical session: Contents.
Background and expectationsAssessment planInterviewsResults and analysisReport
Graham Bath ANZTB Test 2010V.1.0 Page 58 SOFTEC 2010 Graham Bath© T-Systems GEI, PDC Test, 2010
What did we learn from Mr.Testwell?(Test Manager)
Lets judge the test process now.
We will do this by using the TPI-Next scoring tool and our cards.
This is just a quick check on what we think right now
In real life we would have more time!!
Graham Bath ANZTB Test 2010V.1.0 Page 59 SOFTEC 2010 Graham Bath© T-Systems GEI, PDC Test, 2010
This is how our TPI-Matrix might appear after the test manager interviewCluster view generated by TPI-Next scoring tool
This is how our TPI-Matrix might appear after the test manager interview
Other maturitylevels not in scopefor this assessment
Initial
1 Stakeholder commitment2 Degree of involvement3 Test strategy4 Test organization5 Communication6 Reporting7 Test process management8 Estimating and planning9 Metrics
10 Defect management11 Testware management12 Methodological practice13 Tester professionalism14 Test case design15 Test tools16 Test environment
Key not achievedachievednot evaluated
111
Key Area
1 2 3
Controlled
41 2 3 4
1 2 3 41 2 3 4
1 2 31 2 3 4
1 2 3 4
1 2 31 2 3 4
1 2 3 4
1 2 31 2 3 4
1 2 3 4
1 2 31 2 3
1 2 3 4
Other interviews should cover these gaps
Graham Bath ANZTB Test 2010V.1.0 Page 61 SOFTEC 2010 Graham Bath© T-Systems GEI, PDC Test, 2010
What did we learn about using the TPI-Next model?
Graham Bath ANZTB Test 2010V.1.0 Page 62 SOFTEC 2010 Graham Bath© T-Systems GEI, PDC Test, 2010
Test Process Improvement with TPI-Next
Contents.
IntroductionSetting the scene for test process improvementUsing models for improvementA closer look at the TPI-Next®-ModelThe improvement cycleUsing TPI-NextCritical Success FactorsThe ISTQB Expert Level syllabus “ITP”Key messages
Graham Bath ANZTB Test 2010V.1.0 Page 63 SOFTEC 2010 Graham Bath© T-Systems GEI, PDC Test, 2010
Critical success factors
Many factors can influence the success of a test process improvement programIgnore any one of them and our improvement efforts may fail.
1: Choose the right approach
2: Consider improvement as a process
3: Obtain sponsorship
4: Understand people and get “buy-in”
5: Manage expectations
6: Manage change as a process
7: Acquire the right skills and knowledge
Graham Bath ANZTB Test 2010V.1.0 Page 64 SOFTEC 2010 Graham Bath© T-Systems GEI, PDC Test, 2010
Test Process Improvement with TPI-Next
Contents.IntroductionSetting the scene for test process improvementUsing models for improvementA closer look at the TPI-Next®-ModelThe improvement cycleUsing TPI-NextCritical Success FactorsThe ISTQB Expert Level syllabus “ITP”Key messages
Graham Bath ANZTB Test 2010V.1.0 Page 65 SOFTEC 2010 Graham Bath© T-Systems GEI, PDC Test, 2010
ISTQB® Certified Training: structure
TestAnalyst
TestManager
Technical Test
Analyst
Certified Tester Foundation Level
Certified Tester Advanced Level
Test Manager
Test Analyst
TechnicalTest
Analyst
Certified Tester Expert Level
Improving the Test Process
TestManagement
TestAutomation
TTCN-3…
Graham Bath ANZTB Test 2010V.1.0 Page 66 SOFTEC 2010 Graham Bath© T-Systems GEI, PDC Test, 2010
Analytical-based Improvement
Selecting the approach for test process Improvement
The improvement process
Organization, roles and skills
Managing change
Critical success factors
Model-based Improvement
Fundamental approaches and frameworks
The ISTQB® Expert Level Syllabus.Improving the Test Process: Principal Contents
Graham Bath ANZTB Test 2010V.1.0 Page 67 SOFTEC 2010 Graham Bath© T-Systems GEI, PDC Test, 2010
Test Process Improvement with TPI-Next
Contents.
IntroductionSetting the scene for test process improvementUsing models for improvementA closer look at the TPI-Next®-ModelThe improvement cycleUsing TPI-NextCritical Success FactorsThe ISTQB Expert Level syllabus “ITP”Key messages
Graham Bath ANZTB Test 2010V.1.0 Page 68 SOFTEC 2010 Graham Bath© T-Systems GEI, PDC Test, 2010
Using a model to improve the testing process is a tried and trusted approach.
The TPI-Next model has been shown to provide benefit to those using it.
Use it as a tool to guide you through test process improvements.
Talking to and understanding different stakeholders is essential.
Key messages.
Thanks for participating!
Graham Bath
Graham Bath ANZTB Test 2010V.1.0 Page 70 SOFTEC 2010 Graham Bath© T-Systems GEI, PDC Test, 2010
Appendix:
Checkpoints to be assessed in the interview with Mr. Testwell
Graham Bath ANZTB Test 2010V.1.0 Page 71 SOFTEC 2010 Graham Bath© T-Systems GEI, PDC Test, 2010
Key Area 1: Stakeholder commitmentMaturity level:
01.c.1 The principal stakeholder is defined (not necessarily documented) and known to the testers.
01.c.2 Budget for test resources is granted by and negotiable with the principal stakeholder.
01.c.3 Stakeholders actually deliver the committed resources.01.c.4 The principal stakeholder is responsible for a documented product risk
analysis (the input for the test strategy).
Checkpoints
Controlled
Source: [TPI Next]
Graham Bath ANZTB Test 2010V.1.0 Page 72 SOFTEC 2010 Graham Bath© T-Systems GEI, PDC Test, 2010
Key Area 2: Degree of involvementMaturity level:
Checkpoints
Controlled
02.c.1 The test assignment, scope and approach are negotiated early with the principal stakeholder as one of the first test activities.
02.c.2 Test activities are started early, timely before test execution, with the goal of keeping the test activities of the project's critical path.
02.c.3 A tester is involved in project planning: dependencies between the test process and other processes are taken into account.
02.c.4 A tester is involved in the analysis and mitigation of overall project risks.
Source: [TPI Next]
Graham Bath ANZTB Test 2010V.1.0 Page 73 SOFTEC 2010 Graham Bath© T-Systems GEI, PDC Test, 2010
Key Area 3: Test strategyMaturity level:
Checkpoints
Controlled
03.c.1 The principal stakeholder agrees with the documented test strategy.
03.c.2 The test strategy is based on a product risk analysis.03.c.3 There is a differentiation in test levels, test types, test coverage and test
depth, depending on the analyzed risks.
03.c.4 For retests and regression testing a simple strategy determination takes place.
Source: [TPI Next]
Graham Bath ANZTB Test 2010V.1.0 Page 74 SOFTEC 2010 Graham Bath© T-Systems GEI, PDC Test, 2010
Key Area 4: Test organizationMaturity level:
Checkpoints
Controlled
04.c.1 People involved know where to find the persons (or department) responsible for test services.
04.c.2 There is a structure of control and accountability within the test organization.
04.c.3 Test tasks and responsibilities are defined (and documented) and are assigned to a person or organizational unit.
04.c.4 The products and services of the test organization are clear to its clients.
Source: [TPI Next]
Graham Bath ANZTB Test 2010V.1.0 Page 75 SOFTEC 2010 Graham Bath© T-Systems GEI, PDC Test, 2010
Key Area 5: CommunicationMaturity level:
Checkpoints
Controlled
05.c.1 Every team member is aware of decisions being made and of internal progress.
05.c.2 The test team actively gathers relevant information from stakeholders.
05.c.3 It is possible to trace back points of action, agreements and decisions of the test team.
05.c.4 The test team deliberates with stakeholders about progress, product quality and risks and signals proactively potential delays as well.
Source: [TPI Next]
Graham Bath ANZTB Test 2010V.1.0 Page 76 SOFTEC 2010 Graham Bath© T-Systems GEI, PDC Test, 2010
Key Area 6: ReportingMaturity level:
Checkpoints
Controlled
06.c.1 The reporting contains aspects of time and/or costs, results and risks.
06.c.2 The reporting frequency and content matches the basic demands ofstakeholders for the decision making process.
06.c.3 Reporting is done in writing.
Source: [TPI Next]
Graham Bath ANZTB Test 2010V.1.0 Page 77 SOFTEC 2010 Graham Bath© T-Systems GEI, PDC Test, 2010
Key Area 7: Test process managementMaturity level:
Checkpoints
Controlled
07.c.1 At the start of the test project a test plan is created. The test plan includes at least the test assignment, the test scope, the test planning, the roles and responsibilities.
07.c.2 The test plan is agreed with the principal stakeholder.
07.c.3 Each test activity is monitored and when necessary adjustments are initiated.
07.c.4 The test plan is agreed with the relevant stakeholders.
Source: [TPI Next]
Graham Bath ANZTB Test 2010V.1.0 Page 78 SOFTEC 2010 Graham Bath© T-Systems GEI, PDC Test, 2010
Key Area 8: Estimating and planningMaturity level:
Checkpoints
Controlled
08.c.1 For test effort estimation, simple techniques are used such as ratios.
08.c.2 For each test activity there is an indication of the period in which it runs, the resources required and the products to be delivered. Activities to be identified are: test planning and management, defining test cases and executing test cases.
08.c.3 The dependencies between test phases or test activities are plotted in a test planning. A certain overlap of test phases and test activities is allowed.
08.c.4 Test estimations and test planning are discussed with the principal stakeholder.
Source: [TPI Next]
Graham Bath ANZTB Test 2010V.1.0 Page 79 SOFTEC 2010 Graham Bath© T-Systems GEI, PDC Test, 2010
Key Area 9: MetricsMaturity level:
Checkpoints
Controlled
09.c.1 In the test process metrics are defined and used to estimate and control the test project.
09.c.2 The necessary input for the metrics is recorded uniformly and the defined metrics are systematically stored.
09.c.3 The input (data) for metrics is probably accurate.
Source: [TPI Next]
Graham Bath ANZTB Test 2010V.1.0 Page 80 SOFTEC 2010 Graham Bath© T-Systems GEI, PDC Test, 2010
Key Area 13: Tester professionalismMaturity level:
Checkpoints
Controlled
13.c.1 Testers have received specific test training and/or have sufficient experience in the field of structured testing.
13.c.2 Testers are familiar with the adopted test method and they apply it.
13.c.3 All required expertise - whether industry, business or technical - is available to the test team.
13.c.4 Testers are regularly evaluated on specific testing skills as well as general IT abilities in an employee performance appraisal.
Source: [TPI Next]