Automated Web Service Testing in Agile

Edmund Barton

Contents

1. TEC & Industry Training – the “puzzle”

2. ITR as a solution to “the puzzle”

3. Our Agile Team 3. Our Agile Team

4. Technology & Tools

5. A Test Approach in Agile

6. What went well

7. What didn’t

8. Lessons Learned

Take-homes

• Where modern software development is

• Where functional testing is heading

• How to reduce the testing “bottleneck”

• Insight into a project that used test automation well

1. TEC & Industry Training – the “Puzzle”

The Client

Tertiary Education Commission

Multi-$bn budget across the Tertiary Education Sector

1. TEC & Industry Training – the “Puzzle”

Governmental body

Industry Training

• Funded by TEC

• Workplace training programmes

1. TEC & Industry Training – the “Puzzle”

• Modern Apprenticeships

Industry Training Organisations – ITOs

Question?

• How do we know how well ITOs are performing?

1. TEC & Industry Training – the “Puzzle”2. ITR as a solution to “the Puzzle”

The Answeris to collect data

... like a vac

I.T.R. or Industry Training Register

Collects data about all Industry Training

TEC gets a real-time view of Industry Training

2. ITR as a solution to “the Puzzle”

TEC gets a real-time view of Industry Training sector

• Accurate measures of ITOs performance

• Checks data against TECs policy

Data source for further analysis of Tertiary Sector

I.T.R. as an IT solution

ITR is a Web Service, from scratch

No User Interface

2. ITR as a solution to “the Puzzle”

Application

e.g.

Student

Management

System

Web

Service

XML SOAP Messages

Building ITR

2. ITR as a solution to “the Puzzle3. Our Agile Team

A candidate for Agile?

Requires Flexibility

3. Our Agile Team

ITR’s Agile Team

Requirements

partially

Therefore a

candidate for Agile

3. Our Agile Team

partially

flexible

Resources

3-6 developers

2-3 testers

Timescale

Fixed April ‘10

to Feb ‘11

candidate for Agile

Agile Team

“The Business”

Development / Project Management

3. Our Agile Team

Development / Project Management

Test / QA - Qual IT

ITR Project goes Agile

Adopted SCRUM

Testing at the same time as Development

3. Our Agile Team

Multiple User Stories per iteration

Software “Demo” after each iteration

Question:

• But there’s no UI, how did we demo progress?

What was the role of the tester in this project?

3. Our Agile Team

Role of the Tester in Agile

Test Automation Engineer

Distributor of knowledge

3. Our Agile Team

Project “glue”

Story quality gate

Usual defect wrangling / management

Facilitator of business acceptance during iteration

Example Agile User Story

As a generic example:

As a:Consumer of ITR

3. Our Agile Team

I want to:receive an error message if I submit an invalid message

So that:I know I’ve made a mistake

List of Acceptance Criteria

conditions where this story applies

Agile Project Approach

Fixes

This high-level process applies to each user story

3. Our Agile Team

Development Test

Bugs

Fixes

Application Feature

AutomatedTest

Process Outputs

Plan / Design

Agile Project Approach

Test all the way through the iteration

More efficient use of test resources

3. Our Agile Team

Use “Test Driven Development”

Reliance on Automation technology

What technology was used?

... here comes the science bit

3. Our Agile Team 4. Technology & Tools

Agile Development Technologies

4 . Technology & Tools

We assume that:

• An application will function the same in each environment (if we control data and configuration)

Agile Development Technologies

Each team member (inc Test) has:

• own VM (a Build workstation)

4 . Technology & Tools

• own VM (a Build workstation)

• Visual Studio 2008

• Team Foundation Server

• Common test data

• Build Server status

• Team City

s

Build Server (Team City)

Provides feedback on the current state of the “solution” i.e. code doesn’t compile

4 . Technology & Tools

Team

Foundation

Server

Source code(under source control)

Team City

Build Server

Build Server (Team City)

Nightly Builds (Team City)

• Run ALL automated functional tests (Regression suite)

• Nightly feedback on state of “solution” i.e. some new

4 . Technology & Tools

• Nightly feedback on state of “solution” i.e. some new code regressed existing functionality

Broken builds

• Buy / Make the team coffee

Successful Nightly Builds

Essential for Agile software development

• Tell us we have working software

• Could be deployed right now!

4 . Technology & Tools

• Could be deployed right now!

New code hasn’t regressed other parts of the application

Rely heavily on the automated functional tests

Really important to have good test coverage

4. Technology & Tools5. A Test Approach in Agile

What was the testing approach?

Test Automation approach

Test Engineer has 3 key tasks:

• Create and support the automation framework

5. A Test Approach in Agile

• Create automated tests – validate features

• Add / remove from regression suite as project changes

• Can be a lot of work (ITR has over 2000 test cases)

Example of Tools

Open Source

5. A Test Approach in Agile

Front end for Web Services – the missing UI

Large community of users – easy to find help

Data Driven test cases

Example of framework

ITR written in .NET

Use NUnit to execute SOAP UI test cases

5. A Test Approach in Agile

ITR

Web

Service

SOAP Messages

Test suite project

.NET test

method

ITR

database

Example of testing

Really grey-box testing, not fully open white box testing but more in-depth than black box

Achieved 100% requirements coverage

5. A Test Approach in Agile

Achieved 100% requirements coverage

SOAP UI projects self documenting

Traceable via “Spreadsheet of Doom”

Agile – Approach to change

Agile projects change

Complete manual testing unfeasible

5. A Test Approach in Agile

Maintenance overhead?

Yes and No

• Yes – but don’t worry about throwing tests away

• Yes – but don’t maintain, rebuild

• No – just reflects churn within the project

So what worked well?

5. A Test Approach in Agile 6. What went well

… what’s so flash?What worked well?

1. Houston, we have a problem ...

6. What went well

How to make it faster?

Other options failed to make a difference:

• Database tweaking

• Disabling logging

6. What went well

• Disabling logging

• Better hardware

Key architectural components were too slow

ITR needed more than just refactoring

Improving performance

Developers:

• Rip out the poorly performing frameworks and re-engineered some of the architecture

6. What went well

engineered some of the architecture

• Re-factor poorly performing code

Use the automated functional test suite as the only validator of behaviour

Test First Development

A real improvement

Result

• Massive performance

6. What went well

performance improvement

• 13x increase in message processing speed

2. Better Deployments

Deploying wrong version of code = bad

Build server set up to automatically:

6. What went well

• run the regression suite

• check every test passed

• build a deployment package with correct versions

2. Better Deployments

Greatly simplified the deployment

Over 50 deployments

6. What went well

Guaranteed the functionality of the deployment package:

• All deployment problems were configuration related

• much easier to fix

• Zero deployment functionality problems

3. Using SOAP UI Tests for Business Acceptance

Web service has no UI

• Difficult to demo to the business

6. What went well

Demo the features created during the Agile iteration

Improved Agile acceptance “gates”

So what worked well?

6. What went well7. What could have been improved

• … what’s so flash?What could have been done better?

1. Agile troubles – Web Service

Fixed Time / Fixed Requirements – not enough resources. Requirements

partially

7. What could have been improved

Project struggled to be Agile

partially

flexible

Resources

3-6 developers

2-3 testers

Timescale

Fixed April ‘10

to Feb ‘11

1. Agile troubles – A Web Service

Web Services have set list of operations

SOAP Messages

7. What could have been improved

SOAP Messages

ITR

Web

Service

ITR Client

Cannot just remove them – they are mandatory

operation

2. Agile troubles – No Refactoring

Refactoring was discouraged in favour of more User Stories – “Velocity”

Iteration after iteration of on-the-fly decisions

7. What could have been improved

Iteration after iteration of on-the-fly decisions produces a terrible codebase

Often difficult to accommodate new requirements

Agile progress or “velocity” is not more important than code quality.

Address code quality as you go

So what worked well?

7. What could have been improved8. Lessons Learned

• … what’s so flash?Lessons Learned

1. Lessons about Agile Projects

• If the project isn’t a candidate, don’t bend it to fit

• Needs a strong PM with Agile experience

8. Lessons Learned

• Testing does fit – Testers become part of the development team

• Test Automation pretty much essential on projects of this scale

2. Lessons about Agile Development

• Agile projects are not just about “velocity”, quality must also be a driver

• Consequence of design decisions being made

8. Lessons Learned

• Consequence of design decisions being made on-the-fly is poor code quality

• There’s no getting away from a need to plan / design and architect

3. Lessons about Test Automation

Steep Learning Curve

Highly valuable

8. Lessons Learned

• Time Saved on Project Timelines

• Confidence in development for Business

• Test Driven Development and Regression

A final word from the Developers

From the post-implementation review

• “The testing was awesome”

8. Lessons Learned

Questions / Comments