The Testing Maturity Model in the Real World...Our TMM Assessment Model Our approach involved: •...
Transcript of The Testing Maturity Model in the Real World...Our TMM Assessment Model Our approach involved: •...
© Marks and Spencer plc 2004
The Testing Maturity Modelin the Real World
W15
Brian Wells & Andrew Goslin1 December 2004
© Marks and Spencer plc 2004
Objectives
The objectives today are to:• Tell you why we needed to assess test process
capabilities• How we used the Test Maturity Model (TMM) method
to undertake a pilot• What was achieved
and• Our views on taking TMM Forward
© Marks and Spencer plc 2004
Agenda
• The Motivators!• CMM/CMMi/TMM Sister Act!• What is TMM and our TMM Assessment
Method• The Maiden Flight of (our) TMM!• What happened in-flight and after landing!• Scheduled Flights for TMM?• Conclusions
© Marks and Spencer plc 2004
How do you assess the quality of the testing process that demonstrates quality?
© Marks and Spencer plc 2004
Motivators
• What were the testing capabilities of our partners & internally?
• Needed an objective robust assessment method, clearly understood by all
• Wanted to identify improvement opportunities in an action plan with realistic timescales
• Need to develop a structured test process improvement framework
• Extend institutionalisation of good practices• Needed a seamless project lifecycle…
but geographically challenged!
© Marks and Spencer plc 2004
CMM/CMMi/TMM Sister Act!
Considered:• Test Process Improvement (TPI) – continuous model
and flexible to organisation• Test Improvement Model (TIM) – Based on CMM but
not as extensive in KPAs nor as detailed• SPICE – Doesn’t address testing in sufficient detail• Also looked at others, but…• Test Maturity Model
• Clear mapping to CMM/CMMi already.• Contains required depth of most aspects of test
and test-related processes
© Marks and Spencer plc 2004
CMM/CMMi/TMM Sister Act - 2
We decided on TMM. Why?• Used experience and structure of CMM assessments
and trained CMM Assessors• TMM can be used as a staged or continuous
assessment model• TMM could integrate into CMMi• TMM can be used in isolation…• In reality supports CMMi process maturity, but
concentrates on testing!
© Marks and Spencer plc 2004
The TMM Structure
5 Levels of maturity, each containing:• Maturity Goals (KPA)• Purpose of Maturity Goal• Maturity sub-goals (Generic & Specific Goals)• Questions (Key Practices)• Critical Views
(Sub practices based on 3 roles: Manager, Developer/Tester & User/Client)
© Marks and Spencer plc 2004
TMM Structure Diagram
LevelsContainIndicate
Testing Capability Maturity GoalsSupported by
Maturity Sub-GoalsAchieved by
Activitiers / Tasks / Responsibilities
Critical ViewsImplementation and
Organisation Adaption
AddressOrganised by
Manager Developer / Tester User / Client
© Marks and Spencer plc 2004
The TMM Structure
© Marks and Spencer plc 2004
TMM Text Book Approach
• Service Mark of Illinois Institute of Technology• Text Book indicates that TMM questionnaires be sent
out to organisations, supported by follow-up interviews if required
• No requirement mandated for supporting artefacts• Assessment based on questions and responses;
evaluation of responses against Critical Views• Critical views based on Senior manager,
Developer/Tester and User/Client• Has basic ranking system: Satisfied / Not Satisfied
© Marks and Spencer plc 2004
Our TMM Assessment Model
Our approach involved:• Organising TMM into CMM/CMMi structure• Assessment through structured interviews of project /
organisational roles• Trained CMM Assessors• Requires for submission of supporting artefacts• Review and consensus of information by assessment
team• Consolidated assessment workbook• Follow CMM/CMMi ratings process and procedures• Structured feedback of findings
© Marks and Spencer plc 2004
TMM - Learning to fly…
• Must obtain clear set of objectives from sponsor before undertaking TMM assessment
• All information is confidential and de-personalised at all levels before presentation
• Format is...• Briefing Session• Interviews• Review of artefacts• Scoring by consensus• Structured feedback including action plan
© Marks and Spencer plc 2004
TMM – Our first take-off
• M&S are working towards CMMi Level 3• All preferred suppliers were CMMi Level 5
certified• We assessed to TMM Levels 2 & 3 only• Everyone had some exposure to CMM/CMMi,
this greatly assisted buy-in of unknown• Some scepticism at beginning, but this
evaporated quickly• All found the experience very useful and results
focused everyone
© Marks and Spencer plc 2004
TMM – Our maiden flight
• All indicated some failures at Level 2 and 3
• Some activities are done well by all…
• No-one does some of the required activities…
• About 50% of the activities were done well by most
• Best practices were highlighted for sharing
© Marks and Spencer plc 2004
TMM – Our first landing
• We established our capability baseline which was positively accepted
• Identified…• Common strengths & weaknesses• Best practices
• Identified other improvement areas
• Easily defined process improvement road map
• Road map was readily accepted and implementation is in progress
© Marks and Spencer plc 2004
How was the flight?
• What went well• Preparation• Structured• Assessment method / process• Credibility, understanding & acceptance• Knowledge of organisations’ at many levels
• What did not go well…
© Marks and Spencer plc 2004
How was the flight?
• Learning's & Improvements• Need to streamline summarisation of results• Supporting tool
• Pitfalls• Time to complete formal assessments• Relative expense• Marketing longer term benefits
• Issues• Needs to be supported by a robust review / QA
process
© Marks and Spencer plc 2004
Our Conclusions
• CMMi level 5 is not necessarily a good indication of Test Process Maturity!
• Practical, clearly structured framework for test process improvements resulted
• TMM is the best structured and most robust method for test process assessment
• Time consuming, but comprehensive
© Marks and Spencer plc 2004
Our Conclusions
• Unexpected findings• Two-way insight into organsations’ culture!• Partners – high turn-over of test resources• Day to day on the ground issues• Immediate ‘micro-process’ improvements
• Intend a follow-up with formal assessment in 12-18 months
• While it was hard work - It was fun!
© Marks and Spencer plc 2004
Scheduled Flights for TMM?
• We intend scheduled flights for TMM and flying in formation with CMMi assessments
• Value and benefits readily accepted by stakeholders
• Investigating a TMM Forum:• Collaboration of practitioners• Common assessment approach and tools• International acceptance as a standard for Test
Process assessments?
© Marks and Spencer plc 2004
Final thoughts…
© Marks and Spencer plc 2004
How to contact us?
• Brian Wells• Mobile +44 (0)7768 855817• Email [email protected]• Andrew Goslin• Mobile +44 (0)7795 266290• Email [email protected]
© Marks and Spencer plc 2004
References
• “Practical Software Testing” by Ilene Burnstein: ISBN 0387951318
• “A Comparison of Test Process Assessment Models” by Ron Swinkel, Frits Phillips Institute dated 17 November 2000
• CMM/CMMi, SEI