Special Interest Group in Software TestingAustralia and New Zealand Testing Board 27 August 2013...

Australia and New Zealand Testing Board www.anztb.orgAustralia and New Zealand Testing Board www.anztb.org

27 August 2013

Sydney

Facilitators: Leanne Howard & Steve Toms

Special Interest Group in

Software Testing


Agenda5:30pm Welcome Leanne Howard

5:40pm ANZTB update

5:50pm Multi-vendor Management with Matt Wiggins

6:20pm Networking & Refreshments

6:50pm Test Management with Atul Bhatia

7:20pm Close


ANZTB’s Mission Statement

• The ANZTB offers sought after certification, dependable training accreditation and career-enhancing support for software testing professionals throughout Australia and New Zealand.

• The ANZTB will exclusively adopt the qualifications of the ISTQB in its national qualifications.

Australia and New Zealand Testing Board www.anztb.org

Our Team

Copyright © ANZTB

1. David Hayman, Chairman – Auckland

2. Steve Toms, Vice Chair & Training Provider Liaison Officer – Sydney

3. Chris Carter, Board Member & ISTQB President – Sydney

4. David Fuller, Board Member – Sydney

5. Graeme Mackenzie, Treasurer & Accreditation Chair – Wellington

6. Ian Ross, Governance Officer – Christchurch

7. Carol Cornelius, Examination Chair – Wellington

8. Thomas McCoy, Marketing Chair – Canberra

9. Marie Walsh, SIGiST Coordinator – Brisbane

1. Cheryl Downes – Melbourne

2. Farid Vaswani – Auckland

3. Nigel Saunders – Auckland

4. Leanne Howard – Sydney

5. Veronica Belcher – Brisbane

6. Stanley Johannes – Christchurch

7. Tracy Merlino – Melbourne

8. Sarah Benstead – Perth

Board Members

Associate Members


Copyright © ANZTB

AdelaideAucklandBrisbaneCanberraChristchurchHamiltonHobartMelbournePerthSydneyWellington

ANZTB operates in cities in Australia & New Zealand


Certified Testers in Australia & New Zealand

0

200

400

600

800

1000

1200

1400

1600

1800

2004 2005 2006 2007 2008 2009 2010 2011 2012 2013

Advanced

Foundation

ISEB


Certified Advanced Testers inAustralia and New Zealand

Copyright © ANZTB

0

50

100

150

200

250

300

2008 2009 2010 2011 2012 2013

ATTA

ATM

ATA


ANZTBANZTB

SIGiSTsSIGiSTs

ConferenceConference

Accreditation of Training Providers


Create & run exams

Create & run exams

Participate in ISTQB

international working parties



Support the development

of syllabi


of syllabi


ANZTBANZTB

SIGiSTsSIGiSTs




Create & run exams

Create & run exams






of syllabi


of syllabi

www.anztb.org/trainingproviders.php


ANZTB Accredited Training

Providers

ANZTB Accredited Training

Providers

AJQAJQ

AltecAltec

DisqoverDisqover

iChipSolutions

iChipSolutions

K.J. Ross & AssociatesK.J. Ross & Associates

Knight ErrantKnight Errant

LogitrainLogitrain

PlanitPlanit

Revolution ITRevolution IT

SafetyNetSafetyNet

Software EducationSoftware Education

The Testing ConsultancyThe Testing Consultancy


• Confidence that all training materials comply with relevant syllabus and version

– Slides

– Course handbook

– Exercises

• Trainers must have relevant software testing certification, experience and skills

• Organisations must have access to facilities that are suitable learning environments

• Accredited trainers can book exams onsite

Copyright © ANZTB

Why is Accreditation Important to you?


ANZTBANZTB

SIGiSTsSIGiSTs




Create & run exams

Create & run exams






of syllabi


of syllabi


Advancing Expertise in Software Testing

When: Friday, 16 May 2014

Where: Sofitel Wentworth Sydney

Our one-day conference will feature local and international testing experts, discussing the latest advances in the profession.

Copyright © ANZTB

ANZTB Test 2014 Conference


ANZTBANZTB

SIGiSTsSIGiSTs




Create & run exams

Create & run exams






of syllabi


of syllabi


Copyright © ANZTB

ANZTB SIGiSTsSIGiSTs are run 3 to 4 times a year across Australia and New Zealand in the following cities:

• Adelaide • Auckland

• Brisbane • Canberra

• Christchurch • Hobart

• Melbourne • Perth

• Sydney • Wellington

� Up to 30 SIGiSTs per year and growing.

� All past presentations are available on the ANZTB download site: http://anztb.org/downloads.php

� If you would like to be involved – please speak to your facilitator


ANZTBANZTB

SIGiSTsSIGiSTs




Create & run exams

Create & run exams






of syllabi


of syllabi

Exams are written in:

- Wellington

- Auckland

- Brisbane

- Melbourne

In 2013 this will

expand to:

- Perth


• Automatic inclusion on the ANZTB certified tester’s list

• http://www.anztb.org/testers_foundation.php

• Employers check qualification levels

Copyright © ANZTB17

Why take ISTQB certification with the ANZTB?


Guest PresentationMulti vendor management

By: Matt Wiggins


Multi-Vendor Programme Delivery – Getting the Bestof both Worlds

• DEFINITION OF MULTI-VENDOR OUTSOURCING (MVO)

• “Multi-Sourcing is the disciplined provisioning and blending of business and IT services from the optimal set of internal and external providers in the

pursuit of business goals.”• (Gartner)

Footer August 29, 201320



• DEFINITION OF MULTI-VENDOR OUTSOURCING (MVO)

• “Multi-Vendoring is the process of selecting and managing two or more

distinct vendors to deliver one or more IT services, processes or functions.”

(Trent Jordan Heath, Queensland University of Technology)




• MATURITY OF MV-INDUSTRIES


22%

4%

33%4%

4%

11%

11%

11%

Industries (Survey Deloitte)

Energy and utilities Telecommunications

Banking, financial services, insurance Life siences or healthcare

Consumer Products Industry products

Manufacturing Government



• OUTSOURCED IT FUNCTIONS/SERVICES


11%

11%

10%

11%

8%11%

13%

6%

6%

10%

3%

IT Functions (Survey Deloitte)

Application maintenance Application development Data center management Help desk support

Call center operations Desk support Infrastructure services Network architecture

Data security services Telecom services Other



• BENEFITS & CHALLENGES


BENEFITS

CHALLENGES



• BENEFITS OF MVO


Cost effective and efficient: competitive results at the right price for customer

Selection of best-of-breed providers for different IT functions

Flexible change of service provision in case of service quality deterioration

Minimization of Business risk: more than one vendor

Functions shared by internal and external resources



• BENEFITS OF MVO


Enhancement of service delivery: strong competition secures innovation

Geographical Flexibility

Different time zones maximise the working day

Allows customer time to focus on core Business



• CHALLENGES OF MVO

Governance related risks

• Creation of new governance: need a highly skilled team

• Increased complexity and vendor management

• Interoperability between competitive vendors

• Integration of system and infrastructure

Differences in corporate culture

Customer confusion: confusing web of vendors and service providers with individual requirements

Availability of business partners specialised in required domain




• CHALLENGES OF MVO


Inconsistencies due to lack of service quality measurements

Lack of Business Intelligence due to vendor’s geographical dispersion: data analysis as a time and labor-intensive process

Business Nature: contract terms, individual warranty periods

Synchronisation of deliverables

Language and cultural barriers

Time zone differences



• KEY PRINCIPLES FOR MANAGING MVO


GOVERNANCE

COMMITMENT

PEOPLE AND TRAINING

PROCESSES AND METHODS

TOOLS

PROJECTMANAGEMENT



• KEY PRINCIPLES FOR MANAGING MVO

Footer

• GOVERNANCE (Key factor for success)

� Clear set of Key Performance Indicators (KPI’s)

� Interfaces of retained and outsourced functions

� Well-defined contract terms and agreements

• PROJECT MANAGEMENT

� Considered selection of partners (domain expertise)

� Increased effort in a strong partnership

� Open discussion about goals of partnership

� Shared vision of success

� Weekly/monthly assessment of the vendor’s performance

August 29, 201330


Multi-Vendor Programme Delivery – Getting the Best of both Worlds

• TESTING GOVERNANCE STRUCTURE


Steering Committee

Programme Director

• Test Advisory

Board (TAB)

Programme

Test Managers

Portfolio Test

Teams

MULTIVENDOR

TAB Attendance

Test Programme Manager (Chair)

TAB stakeholder community

(approval/authority for testing

artefacts and go/no go

entry/exit criteria quality gates:

� Business Stakeholders

� Programme Management

� Enterprise Testing Senior

Management

� Architecture

� Scrum Master (for Agile

delivery streams)

TAB Subject Matter Expertise

Community (as required, per

agenda):

� Portfolio Test Managers

� Test Managers

� Other relevant SME’s



• RISK & ISSUE MANAGEMENT


Programme Director

Test Advisory

Board (TAB)

Test Managers

Portfolio Test Teams

Programme Test

Manager



• KEY PRINCIPLES FOR MVO

• PROCESSES AND METHODS• PROCESSES AND METHODS

� Structured processes and methods� Well-defined management review and feedback mechanism� Enhace communication

• PEOPLE AND TRAINING

� Leadership across all system integrators� Clear definition of responsibilities/roles� Shared corporate culture and terminology

• COMMITMENT

� Well-defined business objectives and expectations � Evaluation of constraints

• TOOLS� Standardise information shared through integrated tools, techniques and

methods




• Vendor Management Tips


1.• Don't be afraid to share your priorities with a vendor

2.• Create a competitive environment (at least in the eyes of

the vendor) by seeking credible alternatives

3.• Use vendors to help you build a business case

4.• Treat your potential partners as such - not as vendors

5.• Find a way to build a relationship



• Vendor Management Tips


6.• Ask for ongoing responsibility

7.• Always get competitive bids

8.• See the other side

9.

• Make it a win-win

• Dollars & Cents - Don't pay for features you won't use

10.

• Tie guarantees to money

• Focus on value, not price



Any Questions?



Networking & Refreshments


Guest PresentationTest Management

By: Atul Bhatia


Synopsis

• The discussion is focussed on the different types of programs, understand the role of program test manager, challenges faced by the Program Test Manager and how to bring together the disparate streams of testing across large programs so that they can work together successfully and drive the program forward.

• It also provides some best practices and lays emphasis on organisational conditions which will lead to effective management and reporting of quality when testing is not directly managed by you

29/08/2013Infosys Techonologies


AGENDA

� Program Definition

� Challenges Faced by Testing for a Program

� Best Practices to manage Testing for large Program

� Test management by Metrics

� Managing Risks



WHAT IS A PROGRAM

� Projects and activities which are related to each other to produce a unique product, service or result

� Test Management of a Program will involve managing testing of multiple projects often working in Silos at multiple locations

� Each testing team is responsible for their own test planning and execution



PROGRAM STRUCTURES

PORTFOLIO A

Project 1

Project 2

PRODUCT B

Project 3

Project 4

Program comprising of projects impacting only one functional area/portfolio

Program comprising of projects impacting only one functional area/portfolio

Program Management

Portfolio A Project 1

Portfolio B Project 2

Portfolio C Project 3



Program Test Manager-Responsibilities

Are they responsible for Day to Day Test Management???

Responsibilities depend on nature of Program and Organisation structure

Program Test Managers Should:

� Defining Test Strategy at a program level (Prescriptive Only)

� Ensure compliance to the test management processes laid by Test Strategy

� Identification of stakeholders and managing expectations

� Manage program level testing risks and issues

� Identify gaps and interdependencies between various projects under a program

� Defining Test Processes and guidelines for status reporting, defect management

� Defining quality gates

� Enable improvement of Testing Processes



CHALLENGES OF MANAGINGA LARGE TESTING PROGRAM

� Silo Project Teams

� Multi Vendor Teams

� Different Geographies

� Different Project Delivery Methodologies

� Implementing consistent processes across the program

� Managing Quality with constraints of scope, schedule and budget

� Reporting and Communication

� Influence on project resources



BEST PRACTICESARE THERE ANY SET OF DEFINED RULES????

Probably None!!!

Test Approach should be driven by Context

BASIC RULES:

� Understand the Organisation Structure

� Understand the Program Structure

� Develop understanding of current testing processes and testing maturity of the organisation

� Acknowledge the differences in project management and test management processes in different portfolios

� Identify the right people and right testing processes

� Manage stakeholders expectations



PROGRAM TESTSTRATEGY

• Identifies the testing approach

• Prescribe the different types of test phases for a program

according to the needs of the program

• Establishes the QA process required to support the testing

approach

• Establishes reporting and communication framework

• Specifies Roles and Responsibilities of test team with in

the Program

• Identifies gaps and interdependencies between the

projects

• Specifies metrics and measurement procedures29/08/2013Infosys Techonologies


PROGRAM TESTSTRATEGY

� Defines and Agrees program level processes impacting

testing

� Defect Management

� Test data management

� Environment Management

� Release Management (Defect Releases)

� Change Management (Schedule or Scope)

� Risk & Issue Management

� Defines deliverables to be produced by individual project

testing teams

� Establishes Entry and Exit criterion for different phases of

testing

� Defines different metrics which need to be captured

� Recommends the usage of tools29/08/2013Infosys Techonologies


Test Management by Metrics

Software metrics can be broadly classified into 3 categories

� Product metrics: Portray its size, complexity, design aspects, performance and quality

� Process metrics: Measure and Enhance processes of software development, maintenance and testing

� Project metrics: Provides information about various project characteristics and its execution

Software testing metrics are a subset of overall metrics umbrella that focus primarily on the quality facets of process and the product

� Process quality metrics

� Product quality metrics



Process Quality Metrics

• Test Progress Curve – S Curve

• Test Scores Progress Curve – S Curve

• Test Execution Productivity Trends

• Average Defect Turnaround time

• Average Defect Response Time



Process Effectiveness Metrics

• Test Coverage component wise

• Defect removal efficiency

• Defects Rejected

• Requirement Volatility Index

• Defects found per test case executed



Product Quality Metrics

• Testing defect arrival rate

• Quality Quotient Trends

• Component Sensitivity Analysis

• Component wise defect distribution



Additional defect management metrics

• Defects severity distribution

• Defects priority distribution

• Defects state distribution

• Showstoppers trend



Risk Management

Testing is a means to identify business risks and eliminate them before

the code is deployed in production

RISK BASED TESTING:

• Is a methodology to access risks associated with business requirements and driving the testing and test strategy based on associated risks

• Risk Based Testing is

� Mitigate delivery risk

� Improved delivery quality

� Optimize testing cost



Implementation of RBT


• Ensure participation from the right stakeholders and agreement

• Use of metrics to measure risks

• Identify and implement proper framework for Risk Based Testing

• Use proper tools to measure risks

• Measure implementation efficiency of Risk Based Testing

techniques


Success Indicators

• Processes defined by Program Test Strategy are implemented with in the program that meet and support the testing needs of the program

• Processes are seen as efficient and effective

• Organisation Structure is in place to support the testing needs of the Program

• Role of Program Testing Team is clearly articulated and understood

• Delighted stakeholders and sponsors



LESSONS LEARNT

• There is no recipe for ensuring the success

• Test approach should be driven by context

• Management by metrics provides effective way to manage different

testing streams across large programs

• Ensure effective implementation of testing process across all streams



Discussion


[email protected] Details: +61 437074522


• The aim of our Specialist Interest Groups is to:

– Discuss White Papers

– Discuss Technology or Technique changes

– Share Information

– Network

SIGiSTs are sponsored by ANZTB and are in no way linked to any particular training provider or third party.

Copyright © ANZTB

Interest GroupsSpecialist


• Volunteers for facilitation of the next SIGiST?

• Volunteers to submit white papers for presentation and discussion?

The SIGiST cannot succeed without your input.

Copyright © ANZTB

Future SIGiSTs


Any questions?


What did you think of the SIGiST?

Please feel free to Tweet now.

#ANZTBSIGIST

www.anztb.org

Copyright © ANZTB

ANZTB SIGiST

Special Interest Group in Software TestingAustralia and New Zealand Testing Board 27 August 2013...

Documents

Transcript of Special Interest Group in Software TestingAustralia and New Zealand Testing Board 27 August 2013...