Are you in control of Testing, or does Testing control you?

Mike SmithIndependent Consultant

40th Year in IT!

I’ve lived through a few technology eras!

Getting the balance right can be difficult!

Trying to stay relevant

• Tool Author – Test Process Management (T-Plan)• Businessman/Entrepreneur/Tester/Innovator• UK Representative to ISTQB• Chairman ISTQB Advanced Level Syllabus Working Group• Founder member of CAT (Certified Agile Tester) SIG• Practical Assessments (R&D Project)• Web TV & Digital Marketing in sport• Professional sports governance (in my spare time!)• Writing a book!

Presentation Background

• Lack of control over projects/initiatives in IT, Business & Sport

• Common faults over this broad perspective• This presentation focuses on software testing as a form

of measurement• But it uses analogies and common characteristics across

other perspectives • I will use some examples and my experiences to help

highlight key points

Presentation Background

Concerns over effectiveness of software testing as a measurement process

The subject of my book – Measurement & Testing!

My theme for a book

“Effective Measurement and Testing Delivers Control of Anything”

Measurement & TestingMy Axioms

• Testing is more than just Testing!• Software Testing can learn from other forms of

Measurement & Testing• Other forms of Measurement & Testing can learn

from Software Testing• Agile Development is helping the case for

Measurement & Testing• Software Testing took a wrong turn when it

became obsessed with Test Cases!

Another Testing Axiom

•The answer to the ultimate question of Life, The Universe and Everything?

Answer: “42”(Hitchhikers Guide to the Galaxy – Douglas Adams)

•The answer to any question about Software Testing?

Answer: “It Depends!”(Dale Perry – SQE Inc. USA)

3 Key Words

Control

Measurement

Testing (& Software Testing)

Control – Some definitions

•The power to influence or direct people's behaviour or the course of events. (lead)

•Determine the behaviour or supervise the running of. (lag & lead)

Control vs ManagementWhat’s the difference?

Corporate Governance Definition1.Strategic Direction of the Business

1.Effective Monitoring and Control of the Management of the Business

2.Accountable to the Shareholders of the Business

Control vs Management?

Being in Control – Can you answer these questions?

•What are my measures of success?•How will I know when I’ve met them?•How sure will I be that I’m going to meet them at any point in time?•How do I know where I am against them at any point in time?•Can I manage change effectively?•For an IT project, how useful are Test Cases to help me keep in control?

Successful Outcomes?

• Reach destination port• Get there on schedule• All passengers & crew are safe• Passengers have not been (too) seasick!• Within budget (economical with fuel)• Obstacles avoided!

How will I know when I’ve met the successful outcomes?

• What are the key measures & targets?• Speed?• Direction?• Position?• Quality Attributes?

Keeping the ship under control!Key Drivers

Initiatives taken based these key goal & performance drivers

Other measures to support the key drivers for Control

There may be many management systems to support these measures

A complex measurement & management framework is needed

Simple Hierarchical Model does not support the complex relationships

What is Business Performance Measurement & Management?

• ‘Translating Strategy into Action’– Kaplan & Norton

• Use of Business Balanced Scorecard– Different perspectives

• Drives behaviour• Measures outcomes• Links actions to strategy• Leads to predictable outcomes

Dangers of Performance Measurement & Management?

Measures that are badly constructed can destroy an organisation’s performance… effort may be misdirected into unproductive actions and a great deal of time and effort can be wasted.

‘Balanced Scorecard in a week – Mike Bourne, Pippa Bourne’

5 Key principles

1. Generic application2. Objectives, Measures & Targets, Initiatives3. Separate “What” & “How”4. Cascading Scorecards

– One person’s ‘How’ is another person’s ‘What’– Measures & Targets become objectives for next

person 5. Lead & Lag Indicators

– Goal Indicators (reactive)– Performance Indicators (predicative)

Scorecard Objectives, Measures, Targets & Initiatives

InitiativesObjectives TargetsMeasures

WHY WHAT HOW

Scorecards linked bycascading ‘whats’ & ‘hows’

HOWWHAT

HOWWHAT

HOWWHAT

• Could be 1:many or

many:many

• Simplest way is 1:1

HOW

HOWWHAT

HOWWHAT

HOWWHAT

WHAT

Summary of Outcome Driven Measurement Framework/Model

• Separate “What” & “How”• Complex Relationships in Model• Information Traceability matrix Top to

Bottom• Control of Complex Organisations &

Systems• Good fit with Agile Development

Practices!

Now consider Software Testing Model

• What are the Measures & Targets that support the Software Testing Process?

• Test Cases?• Test Objectives?• Test Requirements?• Test Conditions?• Testers focus on Test Cases, but are they the

best Measures/Targets to drive behaviour?• Are they the best to manage risk and change?

The problem with Test Cases

Test(ing) & Software Testing

•The means by which the presence, quality, or genuineness of anything is determined; a means of trial (lag)

•Software testing is a process of executing a program or application with the intent of finding the software bugs. (lag) It can also be stated as the process of validating and verifying that a software program or application or product meets the business and technical requirements that guided it's design and development. (lag & lead)

•Testing is a risk reduction process (lead)

The problem with Test Cases

Test caseA set of input values, execution preconditions, expected results and execution postconditions, developed for a particular objective or test condition, such as to exercise a particular program path or to verify compliance with a specific requirement. [After IEEE 610]

The problem with Test Cases

• How big?• When can they be defined?• They may help me Test, but do they help me

Measure?• Whose Measures are they?• Do they separate the What from the How?• History of Test Cases (‘Bottom-up’)• What else fails if my Test Case fails - traceability?• Tool problems – wrong model• What if none defined?

The problem with Test Cases

1479 test cases: but are they “useful”? Now, let’sstart with a

classificationtree

1479 test cases, so

it must be good,right? Test

specificationprocess

Documentationto agreecoverage

A Simple Hierarchical World

A Complex Relational World!

A simple world became a complex world!

Test Model described by ‘methods’ people at large pan-European financial institution.

Test RequirementsRequirements

WHY

Test Design Specifications

Test Case Specifications

WHAT WHAT HOW

• “From Requirements to Test Case Specification”

in test management tool

A complex model to support software testing

May or may notNeed complex

model:

IT DEPENDS!

Guidelines for a better model

• Need to separate What & How• Model needs Complex Relationships – many:many• Test Conditions can be Considered targets for success,

organised by Measurement Attributes– Functional, Behavioural

• Can and should be defined early• Ability to cope with change• Know where you are at any point in time• Delivers Control of Complex Systems• Good fit with Agile Development Practices!• Business Outcome Driven Development

What factors affect the level of measures and targets in a measurement framework?

• Risk• Size & Complexity• Dev Method• Test Level• Manageability & Maintainability• Tool usage• Ability to cope with change• Quality and level of artefacts• Regulatory requirements…....

What factors affect the level of measures and targets in a measurement framework?

IT DEPENDS!!Setting good measures and targets is

difficult - it requires strong analytical skills

People often neglect the –ve measures and focus on the +ve

What can the world of Measurement learn from Software Testing?

• Software Testing has been developed as a discipline for many years

• Very detailed processes and techniques compared with other forms of Measurement & Testing

• Software Test Techniques– Support Analysis and Design Activities

• Airport Example

Other Experiences

• Making a new Financial Exchange Live

• An Office Move• ISTQB Syllabus Development• Practical Assessments• Governance in Sport• Mindset change and silos!

WHAT’S MISSING?

General Model

Control

Management &Information

Systems

Measurement Framework

Measures &Targets

Objectives

Initiatives

WHAT’S MISSING?

Testing Model

Control

Tests and Test Cases

Test Requirements/Test Conditions

Measures & TargetsSeparated from TestsMany:Many Reqs:Tests

Objectives/Requirements

Initiatives

WHAT’S MISSING?

New Financial Exchange

New Market

Tests and Test Cases

Regulator Approval

SPECIFICATION

MeasurementFramework

RetrospectiveTest Conditions inComplex Model

WHAT’S MISSING?

ISTQB Syllabi (Presented in Prague 2011)

Certificates(FL,AL,EL)

LearningObjectives &

Syllabus

VISION / STRATEGY

SPECIFICATION

Business Outcomes

REQUIREMENTS

WHAT’S MISSING?

Sports Club Constitution Review

Need to Change

Old Subject Matter Experience

VISION / STRATEGY

CONSTITUTION/RULES

Terms of Reference

REQUIREMENTS

My ‘Axioms’

“Test Driven Everything”

“Business Outcome Driven Development”

“Effective Measurement and Testing Delivers Control of Anything”

“Top Down, meets Bottom Up”

References

• Testing Entities Paper – John Kent• IEEE Paper – ‘Keystone Testing Entity’ Mike

Smith & Neil Thompson• Value Flow Scorecards – Mike Smith & Neil

Thompson• Balanced Scorecard in a week – Mike & Pippa

Bourne• My book on Measurement & Testing – when?!

What can you take away from this?

• If you are building a pure hierarchical model to manage testing, think again

• Can I optimise my model to give better control without it becoming unmanageable?

• Am I relying too much on my testing tools – do I need something else?

• Can I use my analytical skills to engage with the business better?

• I encourage you to think about these concepts, even if you think you are in control…...

If you think you’ve found a good move,

look for a better one!

My Measures of Success for this talk?!

1.Standing Ovation?2.Loud Applause?3.Polite Applause?4.Silence?5.I won’t be let into Russia again?!!