Start with Quality – an Agile Tester’s Case Study

Increasing defect trend - 2013 critical bugs

Decreasing defect trend - 2014 critical bugs

Agile mindset

Before:

Product changes based on internal SME’s knowing what changes needed to be made, which was usually based on noisy customers, gut feel and stabs in the dark.

The only metrics measured where budgeted sales vs actual sales, and churn. Team was accountable for missing targets.

Missed release deadlines blamed for missed sales targets, and poor developer productivity blamed for missing delivery deadlines.

Development cycle of about 15 weeks of development, 3-5 weeks of testing and then a release.

Team took no responsibility because the team was held accountable to deadlines they didn’t agree to, and for solutions & features they weren’t consulted on.

What we saw often was short term solutions, ending up in long term problems.

Agile mindset

Now:

Autonomy and accountability. Manager’s role is to embrace collective responsibility and to help their team excel and develop into leaders.

Measure our economic progress: Understand what we value: customer growth, revenue growth, user engagement, higher employee retention.

Do a value proposition canvas and lean canvas business model for all new products and large features. Is this the right action at the right time?

Have a hypothesis for each new feature and compare progress metrics with actual outcomes. Have metrics indicating business health (LTV, CAC, LTV:CAC, MRR, Churn,

Customer Happiness Index).

Experiment with changes to the way we work then measure to test our hypothesis. Constraining the feature pipeline, ready criteria, done criteria, agile testing methodology, moved towards continuous

testing and deployment.

Limit work in progress: Agile testing, deployment to test servers every week, deployment to production every sprint (2 weeks).

Distributed leadership and responsibility where we help each other excel.

Digging deeper to find systematic causes, allowing us to come up with more permanent solutions.

Agile is a direction, not a place

The important thing isn’t how you work.

The important thing is how you improve the way you work!

Week 1 Week 2 Week 3 Week 4 Week 5 Week 6 Week 7 Week 8

Release

Test Fix %&@#!

Before:

Test at the end

Week 1 Week 2 Week 3 Week 4 Week 5 Week 6 Week 7 Week 8

ReleaseTried:

Test continuouslyTest Fix %&@#!

Test Fix Time

Saved

Week 1 Week 2 Week 3 Week 4 Week 5 Week 6 Week 7 Week 8

Release

What we do now

Release Release Release

Week 1 Week 2 Week 3 Week 4 Week 5 Week 6 Week 7 Week 8

ReleaseWhat we ended

up with

Test Fix %&@#!

Fix %&@#!

Test Fix Time

Saved

Building a product

Document Plan

Constrain Pipeline

Prioritise

Build MVP

Iterate to PMF

Optimise

Building a product

Is this the right action

at the right time?

How do you ensure that your product works?

1.Understand the problem

2.Iterate until you’ve solved it

Who are the stakeholders?

How will we know when we’ve solved it?

What need do they have that we want to solve?

How will we know if we’re moving in the right direction?

Minimise the distance to MVP

Deliver, measure, adjust continuously

Henrik Kniberg

How do you ensure that your product works?

1.Understand the problem

2.Iterate until you’ve solved it

Who are the stakeholders?

How will we know when we’ve solved it?

What need do they have that we want to solve?

How will we know if we’re moving in the right direction?

Minimise the distance to MVP

Deliver, measure, adjust continuously

Henrik Kniberg

The Value Proposition CanvasDocument

Plan

Constrain

Pipeline

Prioritise

Build

MVP

Iterate to

PMF

Optimise

How do you ensure that your product works?

1.Understand the problem

2.Iterate until you’ve solved it

Who are the stakeholders?

How will we know when we’ve solved it?

What need do they have that we want to solve?

How will we know if we’re moving in the right direction?

Minimise the distance to MVP

Deliver, measure, adjust continuously

Henrik Kniberg

The Business Model CanvasDocument

Plan

Constrain

Pipeline

Prioritise

Build

MVP

Iterate to

PMF

Optimise

Constrain the Feature PipelineDocument

Plan

Constrain

Pipeline

Prioritise

Build

MVP

Iterate to

PMF

Optimise

How do you ensure that your product works?

1.Understand the problem

2.Iterate until you’ve solved it

Who are the stakeholders?

How will we know when we’ve solved it?

What need do they have that we want to solve?

How will we know if we’re moving in the right direction?

Minimise the distance to MVP

Deliver, measure, adjust continuously

Henrik Kniberg

Build MVP and Verify

Problem InterviewFormulate Testable

Hypotheses

Understand ProblemIs there a problem worth solving and

how do customers solve the problem

today?

Hypotheses Build Measure / Learn

Articulate Problem

Solution InterviewFormulate Testable

Hypotheses

Validate DemoDefine a possible solution using a

demo to help customer visualise?

Build Demo

MVP InterviewFormulate Testable

Hypotheses

Validate Solution

QualitativelyUse feedback from demo to distil

down to the products essence (MVP)

which we should then build.

Build Smallest

Solution (MVP)

Verify

Quantitatively

Validate Solution

QuantitativelyPositive qualitative signal from

customers grants us permission to

move forward.

Validated Learning Loop 1

Validated Learning Loop 2

Validated Learning Loop 3

Validated Learning Loop 4

Document

Plan

Constrain

Pipeline

Prioritise

Build

MVP

Iterate to

PMF

Optimise

Make sure the backlog items are testable & valuable

As a ConsultantI want to assign soil sample collection jobs to a userso that collectors know what jobs are assigned to them.

Prototype: How to demo/test done:1) Select one or more or jobs from

the unassigned jobs list.2) Click “assign jobs”3) Start typing a user’s name

and/or4) Select the user from the list of

users5) Click “Assign to selected user”

Team Layout

Developers

Testers

Product Managers

Designer

Team Layout

Black Team

Black TeamBlue Team

Floating

Developers

Testers

Product Managers

Designer

Blue Team

Continuous Delivery

Automatic

Manual

Feature Toggles

Henrik Kniberg

Henrik Kniberg

Test backlogStep 1: Decide what needs to be tested

● Add new sample job● Delete sampling job● Assign sample job to a user● Unassign sample job from a user● Mark a sample job as complete● Filter sample jobs● Sort sample jobs● Which jobs do I have to do?● Viewing a sample collection job● Editing a grid on a sample collection job

Step 2: Classify each test

Test Case Risk Manual Test

Cost

Automation

Cost

Add new sample job High 0.5 hrs low

Delete sampling job High 0.5 hrs low

Assign sample job to a user Med 2 hrs low

Unassign sample job from a user Med 0.5 hrs low

Mark a sample job as complete High 0.5 hrs low

Filter sample jobs Med 0.5 hrs high

Sort sample jobs Low 0.5 hrs low

Which jobs do I have to do? Med 2 hrs high

Viewing a sample collection job High 2 hrs low

Pay every

timePay once

Step 3: sort the list

Test Case Risk Manual Test

Cost

Automation

Cost

Viewing a sample collection job High 2 hrs low

Assign sample job to a user Med 2 hrs low

Mark a sample job as complete High 1 hr medium

Add new sample job High 0.5 hrs low

Delete sampling job High 0.5 hrs low

Unassign sample job from a user Med 0.5 hrs low

Sort sample jobs Low 0.5 hrs low

Filter sample jobs Med 0.5 hrs high

Which jobs do I have to do? Med 0.5 hrs high

Automate first!

Don’t bother

automating

Automate later!

Bug fixing process

Bug found!

Before

Log it

Release

test

Fix bugs

Bug found!

Now

Don’t log it.

Fix it NOW!

Top-down and bottom-up development

MindsetQuality

AssuranceQuality

Assistance

Manual testAutomatic

test

Functional test

Exploratory test

RequirementsCustomer

needs

Late involvement

Early involvement

Long feedback loop

Short feedback loop

Find defectsPrevent defects

Henrik Kniberg

Metrics Before and After

• Defects

• Customer Satisfaction

• Employee Satisfaction

• Customer Retention

• Sales

• New Customers

• Less Work Better Results