Regression Testing – What to Automate and how

Assurance with Intelligence

Paul GerrardGerrard Consulting1 Old Forge CloseMaidenheadBerkshireSL6 2RD UK

e: paul@gerrardconsulting.comw: http://gerrardconsulting.comt: 01628 639173

In this session, I want to re-trace our early steps to

automationIs life hard at the end because we didn’t think it through the

beginning?

Assurance with Intelligence Slide 3

Automating regression tests

• Regression tests are the most likely to be stable and run repeatedly so:– Automation promises big savings of time– Automation ensures reliable execution and results

checking– Stable tests/software are usually easiest to

automate

• Hold on! Paradox ALERT!!!!– We regression test because things change– Chaotic, unstable environments need it the most– When things change automation is HARD!

Slide 4Slide 4 Assurance with Intelligence

What is the regression testing thought process?

Assurance with Intelligence Slide 5

How I think about regression testing• Establish what we mean by regression

testing• Understand why regressions occur• Combating regression – there are choices• Who are the stakeholders for regression

testing and what do they want?• Context of regression testing• Regression test policy• Selecting what to regression test• Selecting what regression tests to automate

Assurance with Intelligence Slide 6

A common experience?

• Regression testing isa) Painfulb) Boringc) A waste of time

• “Let’s get a tool in”• Automating 10% of tests is easy• Automating 90% of tests is hard

(impossible?)• “Why are we doing this?”• “Let’s do it manually”

Assurance with Intelligence Slide 7

Let’s look at each thought process

Assurance with Intelligence Slide 8

(What is...) regression testing?• When a software fault is fixed, it often happens

that 'knock-on' effects occur• We need to check that when changing the faulty

code, the developer has not introduced new faults• There is a 50% chance of introducing regression

faults• Regression tests tell us whether new faults have

been introduced– i.e. whether the system still works after a change to

the code or environment has been made.

• When environments are changed, we might also regression test.

Slide 9Slide 9 Assurance with Intelligence

Regression testing 2

• “Testing to ensure a change has not caused faults in unchanged parts of the system"

• Not necessarily a separate stage• Regression testing most important

during maintenance activities• Regression tests are usually the first

to be considered for automation.

Slide 10Slide 10 Assurance with Intelligence

How do regressions occur?

• Something changes and affects ‘working’ software

• An enhancement is implemented

• A bug is fixed• Two bug fixes interfere

with each other• One programmer

overwrites another programmer’s fix (configuration management)

• Something in the environment setup

• Test data changes in some way

• A test changes in some way

• Test configuration/setup changes

• Etc. etc.

• Can you think of others?

Assurance with Intelligence Slide 11

Code fixes have a 50% chance of introducing side-effects in working

softwareAncient history – who cares whether it’s 10% or 90%?

Regressions have a disproportionate impact on effort, confidence, morale

Assurance with Intelligence Slide 12

Combating regression - choices• Several options:

– Don’t change anything!– Impact analysis (of code/data//test changes)– Regression testing (to detect unplanned changes)

• Impact analysis is tricky; not 100% reliable but...– Compare source code – see what’s changed– Module dependencies – see what’s affected– Rigour in managing environments and data– Rigour in managing source code

• Regression testing choices– Manual v automated– Test first or test last– All tests or a selection of tests– Others?

Assurance with Intelligence Slide 13

Who are our stakeholders?

• Stakeholders are people interested in the outcome of test• What do they want? Ideally, nothing!• Sponsors, customers

– Regressions are an immediate setback/delay just when they thought you could move the project forward

• Users and testers– “All our good work is undermined!”

• Developers– They’ve missed something obvious, embarrassing; unplanned

work – could be trivial or substantial

• Tech support– “Users are going to be furious!”– “We feel really let down by dev, maintenance, testers”

Assurance with Intelligence Slide 14

Context of regression testing• Test- or behaviour-driven development:

– All tests are automated; these are our safety-net– Tests allow us to refactor with confidence

• Integration/System/Acceptance testing 1st pass– RT after main testing done – confirmation that all

tests pass with all fixes in place

• Maintenance phase– Minor fixes (don’t get impact analysed)– Enhancements fit with existing system– Refactoring/conversion/interface changes.

Assurance with Intelligence Slide 15

Regression test policy

• Common:– Business critical, high risk, error-prone

areas– FIFA – first in, first to automate?– LIFA – last in first to automate– Easily automatable

• But let’s step back a little– Focus on stakeholders– What do THEY actually want?

Assurance with Intelligence Slide 16

What would stakeholders want?• Sponsors, customers

– Want to avoid delays– Is the most critical functionality working?

• Tech support– Are we inconveniencing the users?– Is the most critical functionality working?

• Users and testers– Can we trust the results of our early (high priority tests)– (Maybe we aren’t so worried about the rest?)– Is the most critical functionality working?

• Developers– Want instant feedback (so they can fix with least hassle)– The best case for them doing their own automated tests!

Assurance with Intelligence Slide 17

If a developer ran your test team...• Whose problem is regression anyway?• It’s developers’ problem not the testers’!• Dev have been asked to fix/enhance/refactor

code without screwing up• How would a developer address the

regression problem?– Design a harness and code to execute transactions

at the API level– Easier, more maintainable, less sensitive to

cosmetic changes

• A reusable asset that allows refactoring.

Assurance with Intelligence Slide 18

The real problem to solve?

• If the problem is identifying change where there shouldn’t be any...– With a few caveats, it doesn’t matter what tests we

run as long as we cover the important functionality – and that it behaves the same way as it did last time

– We still need to demonstrate the user interface isn’t adversely affected and the end-to-ends still work

– Is it POSSIBLE and BETTER to do this manually?– Why use a framework on top of a test execution tool

on top of a browser to test server-based code? Too complex!

• Where does that leave automated test execution – on the shelf? (It’s probably there already!)

Assurance with Intelligence Slide 19

Assurance with Intelligence Slide 20

Techniques for Testing GUI Applications• Eurostar 1997: the most popular paper

on my website is referenced by Wikipedia

• http://gerrardconsulting.com/sites/default/files/Techgui.pdf

• Won’t repeat the whole story here...• But most of the 1997 slides are still

relevant• Have we made NO progress since

1997? Assurance with Intelligence Slide 21

Selecting what regression tests to automate

1997

Assurance with Intelligence Slide 22

Manual v automatic scripts - very rough guideline

Manual Scripts Automatic Scripts

Check list (application conventions)

Integration

Synchronisation

Client/server communication

Low-volume/complex EP/BV

Check list (object states, menus, standard features)

Navigation

Integration (for soak tests)

State transition

Simple EP/BV and decision tables.

What aspects to automate?

• Pareto law:– go for the easy scripts first– don’t waste time scripting low volume

complex scripts at the expense of high volume simple ones

• Consider hybrid approach:– use the tool to do navigation and data

entry– do the checking on-line or at end of test.

Automated test scripts

• Coded scripts– navigation through the system, some

checklist test cases can be can be coded directly

– relatively easy to maintain

• Recorded scripts– automated scripts should be documented– then used to record the automated script– less easy to maintain

• Integrate all scripts into automated suite.

Migration of manual scripts

• Automated scripts are easier to maintain– if designed to be maintained– Means you have to build test automation

scaffolding before you script tests– Automation projects are software

development projects, not test projects

• Allow for maintenance cost before deciding to migrate functional scripts– delay migration until system is stable– soak and regression tests, as required.

Conclusion

• The regression problem is a developers’ problem that can be solved by:1. Developer discipline2. Better impact analysis3. Automated testing at an API level4. Manual/automated testing (critical/E2E

level) for visibility to stakeholders

• Is this what the big product companies do?

Assurance with Intelligence Slide 27

Regression Testing – What to Automate and how

Assurance with Intelligence

Paul GerrardGerrard Consulting1 Old Forge CloseMaidenheadBerkshireSL6 2RD UK

e: paul@gerrardconsulting.comw: http://gerrardconsulting.comt: 01628 639173