Speaker Cover Merge - StickyMinds · assignments covering a wide range of applications and tools....
36
BIO PRESENTATION International Conference On Software Test Analysis And Review May 14-18, 2007 Orlando, FL USA TK2 5/16/2007 4:30:00 PM "BUILDING THE TEST MANAGEMENT OFFICE" Geoff Horne iSQA
Transcript of Speaker Cover Merge - StickyMinds · assignments covering a wide range of applications and tools....
BIO PRESENTATION
International Conference On Software Test Analysis
And Review May 14-18, 2007 Orlando, FL USA
TK2
5/16/2007 4:30:00 PM
"BUILDING THE TEST
MANAGEMENT OFFICE"
Geoff Horne iSQA
Geoff Horne Geoff Horne is based in New Zealand and has founded and run two testing companies which have grown to enjoy an international clientele. He has over 28 years experience in IT including software development, sales and marketing and IT and project management. In 1994, almost by accident, he found himself involved in testing a complex fault management system which led to further testing and QA assignments covering a wide range of applications and tools. Geoff’s companies were subsequently founded to bring a full range of testing consultancy services to the IT industry. Of late, he has focused on a few select clients running complex test projects in a programme test management capacity. Geoff has written a variety of white papers on the subject of software testing and has been a regular speaker at the Star testing conferences. He is married with four children and in his spare time (which there is not a lot of) enjoys writing and recording original contemporary Christian music.
www.isqa.com
BUILDING THE
TEST MANAGEMENT OFFICE (TMO)
Geoff Horne
Managing Director
www.isqa.com
Strategy Defect management & resolution
Methods, standards & processesVendor management
Process adherenceTools implementation & management
Communications Environments management
Status reporting Planning Release management
Coverage management Issues resolution
Budget & cost management Schedule management
Change request management Configuration management
Escalation management Support management Review & approval management Human resources management
Quality assurance
www.isqa.com
The typical Test Management inter-relationship model:
www.isqa.com
Sliced:
www.isqa.com
Sliced and diced:
www.isqa.com
Strategy Defect management & resolution
Methods, standards & processesVendor management
Process adherenceTools implementation & management
Communications Environments management
Status reporting Planning Release management
Coverage management Issues resolution
Budget & cost management Schedule management
Change request management Configuration management
Escalation management Support management Review & approval management Human resources management
Quality assurance Automation
www.isqa.com
Strategies Defect management & resolution
Methods, standards & processesVendor management
Process adherenceTools implementation & management
Communications Environments management
Status reporting Planning Release management
Coverage management Issues resolution
Budget & cost managementSchedule management
Change request management Configuration management
Escalation management Support management Review & approval management Human resources management
Quality assurance Automation
www.isqa.com
Defect management & resolution
Tools implementation & management
Environments management
Coverage management
Budget & cost management
Configuration management
Review & approval management
www.isqa.com
Defect management & resolution
Tools implementation & management
Environments management
Coverage management
Budget & cost management
Configuration management
Review & approval management
www.isqa.com
Defect management & resolution
Tools implementation & management
Environments management
Coverage management
Budget & cost management
Configuration management
Review & approval management
www.isqa.com
Defect management & resolution
Tools implementation & management
Environments management
Coverage management
Budget & cost management
Configuration management
Review & approval management
www.isqa.com
Environments management
Defect management & resolution
Tools implementation & management
Budget & cost management
Release management
Communications
Vendor management !Coverage management
Configuration management
Support management Review & approval management
www.isqa.com
What is a Test Management Office?
• A centre of governance for the testing subproject
• Support team & mechanism for Test Managers, Test Analysts etc.
• Carries responsibility for selected testing functions that may not realistically be carried by Test Manager(s)
• Operates as a discrete team within the testing function
• TMO has an operational & delivery responsibility as opposed to being a purely administrative function
www.isqa.com
Why use one?
• Removes some of the management overheads from Test Managers & Analysts
• Enables test teams to focus on key testing activities
• Streamlines & spreads workloads
• Provides a central control of all test resources
• Sometimes a necessity to keep on top of testing resources, assets and facilities
www.isqa.com
When should a TMO be used ?
• Large & complex testing projects
• Integration with other technologies eg. telecommunications networks
• Multi-workstream testing projects
• Projects with multiple & disparate applications, platforms, technologies, environments etc.
• No resources, infrastructures, processes within functions required to support testing eg. environment management, configuration management, release management etc.
• Lack of expertise or commitment within same.
www.isqa.com
Which test functions can be covered by a TMO?Strategies Defect management & resolution
Methods, standards & processesVendor management
Process adherenceTools implementation & management
Communications Environments management
Status reporting Planning Release management
Coverage management Issues resolution
Budget & cost management Schedule management
Change request management Configuration management
Escalation management Support management Review & approval management Human resources management
AutomationQuality assurance
www.isqa.com
Should there be a separate TMO Manager?
• Depends on the size of the TMO and functions it covers.
• What type of person; line manager, test manager, project manager?
• Can the TMO Manager be a non-testing professional?
• Can the Test Manager also be the TMO Manager?
www.isqa.com
What type of roles are the management of each TMO function?
• Should all TMO function managers be testing professionals?
• Should any of the them be testing professionals?
• Where is the balance?
www.isqa.com
How should a TMO operate?
• Clear distinction of responsibilities between Test Manager & TMOManager
• When should managers operate within the TMO and when outside?
• Can the TMO Manager be responsible for one or more TMO functions?
• Who/where should the TMO report?
www.isqa.com
Steps for building a TMO:
• Analyse requirement and determine need
• Select project functions that should come under the TMO
• Select/acquire TMO function managers
• Develop a TMO charter that clearly defines:
• What the functions are that will be under the TMO & why
• How each function will inter-relate with other project streams
• How the TMO will function with the Test Manager(s)
• Where there are overlaps with other functional teams
www.isqa.com
Analyse requirement & determine need:Function Role Covered? How?Test strategies TCM Yes Test ManagerDefect management & resolution LPM Yes Test ManagerVendor management LPM No TMOMethods, standards & processes TCM No Engage iSQA Tools implementation & management SPM Yes Software Support TeamProcess adherence TCM Yes Test ManagerEnvironments management SPM No TMORelease management SPM No TMOCommunications LPM Yes Project AdminStatus reporting LPM Yes Test ManagerTest planning LPM Yes Test ManagerIssues resolution LPM Yes Test ManagerCoverage management TCM No TMOTest schedule development & management LPM No TMOBudget & cost management SPM Yes Project Management OfficeConfiguration management SPM No TMOChange request management LPM Yes Software Development TeamSupport management LPM Yes Project Vendor Liaison OfficerEscalation management LPM Yes Test ManagerHuman resources management SPM Yes HR DeptReview & approval management LPM No TMOQuality assurance SPM No Engage iSQA
Line/project manager (LPM)Test consultant/manager (TCM)Specialist manager (SPM)
www.isqa.com
Select functions for TMO:Function Role Covered? How?Test strategies TCM Yes Test ManagerDefect management & resolution LPM Yes Test ManagerVendor management LPM No TMOMethods, standards & processes TCM No Engage iSQA Tools implementation & management SPM Yes Software Support TeamProcess adherence TCM Yes Test ManagerEnvironments management SPM No TMORelease management SPM No TMOCommunications LPM Yes Project AdminStatus reporting LPM Yes Test ManagerTest planning LPM Yes Test ManagerIssues resolution LPM Yes Test ManagerCoverage management TCM No TMOTest schedule development & managem LPM No TMOBudget & cost management SPM Yes Project Management OfficeConfiguration management SPM No TMOChange request management LPM Yes Software Development TeamSupport management LPM Yes Project Vendor Liaison OfficerEscalation management LPM Yes Test ManagerHuman resources management SPM Yes HR DeptReview & approval management LPM No TMOQuality assurance SPM No Engage iSQA
Line/project manager (LPM)Test consultant/manager (TCM)Specialist manager (SPM)
www.isqa.com
Resource TMO:
Function Role Fulltime? How?Vendor management LPM No Second Helen to TMOEnvironments management SPM Yes Appoint Test Environment ManagerRelease management SPM Yes Appoint Release ManagerCoverage management TCM No Engage iSQA as part of TMOTest schedule development & management LPM Yes Second Jim from Project XConfiguration management SPM Yes Appoint Test Configuration ManagerReview & approval management LPM No Second John from PMO
www.isqa.com
Develop TMO Charter:
Table of Contents
1 Introduction 12 TMO Team Overview 23 Purpose 3
Responsibilities 3Team Structure 4TMO Accept/Reject Processes 5
3 Test Approach 6Test Phases 6Objectives 7
4 Environments 8Requesting Test Environments 8Deployment of Code into Test Environments 9
5 Release Management 10Release No. Structure 10Release Processes 11Release Notifications 13Releases into Production 14
www.isqa.com
Example
• Local subsidiary of a worldwide mobile network operator• $NZ50m+ project• Frequent changes of senior project staff• Multiple vendors• Multiple test environments• Multiple test workstreams including multiple technologies• No recognition of formal testing methodologies• Flawed test strategy• Test phase would commence without properly completing previous
phases in order to maintain deadlines• Totally reactive project
www.isqa.com
Logical Test Environm ents
Test Configuration Coordination
High Level Test Planning(Across SIT / UAT / OAT / Perf / MIT)
Test
Sch
edul
e C
oord
inat
ion
Environm ent CoordinationTest SME Coordination
Release Coordination
Coverage & Defect Coordination
SIT Planning UAT Planning OAT Planning PRFTPlanning MIT Planning
Test Planning
SME Test Resources
Physical Test Platforms
Test Platform Configuration
Test Applications / Libraries
Envi
ronm
ent S
tatu
s
Test Data (e.g Subscribers, Re-charge Cards, MSISDNs)
Test Devices (e.g. Handsets, SIMs)
SIT Execution UATExecution
OATExecution
PRFTExecution MIT Execution
Test Execution
Build & PgmSchedule
andBuild Team
Program Frodo TMO ProcessesVersion: V0.2Date: 5 May 2006Author: Arno Andres
www.isqa.com
TMO Structure:
TMO Manager• TMO Owner/Driver • Test Strategy• Testing Quality Assurance• Test Scheduling & Co-
ordination• Support Management
TMO Administrator• Reporting• Communications
Defect & Coverage Manager• Test Coverage Management• Defect Management• Issues Management
Environments Managers (2)• Test Environments• Test Equipment
Release Manager• Test Releases
Configuration Manager• Test Configuration• Parameters
Triage Manager• Owner of test blockages• See blockages through to
completion
www.isqa.com
TMO Structure:
TMO Manager: senior test consultant & managerTMO Administrator: new hire, no previous testing experienceDefect & Coverage Manager: experienced defect & issues managerEnvironments Managers: experienced environments managers,
one with solid testing experienceRelease Manager: experienced release managerConfiguration Manager: experienced configuration managerTriage Manager: experienced project manager and ace
problem solver
www.isqa.com
Lessons learned & how to avoid
• Anything to do with TMO functions was thrown onto or any delays always blamed on TMO:
• Other managers & personnel abdicating responsibility• Our Charter was not clear enough • Took us a while to develop an Accept/Reject process• We lacked adequate tools to properly track issues• We were slow to escalate issues
What could we have done better:
• Developed our Charter including all processes up front• Ensure the Charter was properly communicated across the project and
correct expectations set as to the TMO role• Established TMO at the project outset rather than part way through• Flagged the needs for tools sooner
www.isqa.com
Lessons learned & how to avoid cont.
• Needed to prevent TMO becoming an unnecessary management layer &bottleneck
• Decisions often required the Environments, Release & Configuration Managers to concur & they where based in multiple locations
• Set up an issues “triage” function which became more of a hindrance than benefit
What could we have done better:
• Gained buy-in from external parties sooner• Implemented a “heads-up” process• Pushed the “triage” function out onto the relevant build teams rather than
manage within the TMO• Endeavour to locate TMO personnel in same location if possible
www.isqa.com
Lessons learned & how to avoid cont.
• Having TMO managers hooking into existing infrastructures, teams & personnel
• TMO managers often had difficulties in obtaining resources from relevant infrastructure teams
• Non-project management personnel de-prioritised TMO requests as they didn’t come from a senior project manager
What could we have done better:
• Escalated blockages sooner• Gained buy-in from external parties sooner• Implemented a “heads-up” process• All of the previous points as well!
www.isqa.com
Summary
• Needs to be established and clear Charter developed at the outset of a project
• Need to carefully plan and structured to avoid:• Dumping ground• Blame game• Delay agent• Becoming an “island”
• Needs to suitably tooled & resourced• Needs to fit into a project & not onto it• Not the panacea of all large test project problems
However…..
www.isqa.com
Summary
• Some projects cannot run without a TMO
• Provides a central point of governance & control of all testing resources, assets and facilities
• Removes overheads from testing personnel
• Enables test teams to focus on key testing activities
• Streamlines & spreads workloads
www.isqa.com
BUILDING THE
TEST MANAGEMENT OFFICE (TMO)
Geoff Horne
Managing Director
