Acceptance Testing vs. Unit Testing:

A Developer’s Perspective

Owen Rogers

orogers@thoughtworks.com

Rob Styles

rob@objectagile.com

2

What are Acceptance Tests?

• Tests owned and defined by the customer to verify that a story is complete and correct.

3

Why we use Acceptance Tests:

• Increase team confidence that the system is correct.• Structures communication between customer and

developers.• Help customer think through requirements.• Informs developer estimates.• Indicates how to split a story.• Concrete definition of story completion (“knowing when

we’re done”).• Early involvement of QA.• Automated regression testing for free.

4

Why other people don’t:

• Customer participation.• Management buy-in.• QA support.• More tests for developers to implement.• No common framework.

5

ATs and UTs are the same though, right?

• Both can and should be written first, before implementation.

• Both should be automated by containing assertions for validating expected results.

• Both typically use fixture setup and teardown code to run before and after the execution of the tests.

• Both can be implemented using an xUnit framework.

6

Difference: ATs are written by the customer.

• UTs only need to be read and understood by other developers.

• ATs must be defined in a language familiar to the customer.

• ATs require a customer meta-language:• Generic enough to capture requirements• Abstract enough to be maintainable• Simple enough to be easily understood

7

Example Acceptance Tests.

fit.ActionFixture

start eg.Page

enter location http://google.com

check title Google

enter link Jobs

check title About Google

enter link Press

enter link Review

check title Google Press Room

FIT Acceptance Test: FAT Acceptance Test:

• Request: “http://google.com”

• Check title: “Google”

• Click link: “Jobs”

• Check title: “About Google”

• Click link: “Press”

• Click link: “Review”

• Check title: “Google Press Room”

8

Difference: Customers don’t use an IDE.

• UTs are implemented in the language of development using an IDE.

• ATs should be defined using a tool accessible to the customer (Excel, web page, wiki).

• ATs must be retrieved from the tool and parsed.• Ideally, the customer should be able to run the ATs from

this environment.

9

Interacting with customer tool.

Excel

Wiki

AT Framework

Extract Test (.html)

Extract Test (.csv)

Execute TestWrite Test

10

Difference: ATs are not directly executable.

• UTs invoke code-under-test directly.• AT definition is abstracted from its implementation.• ATs specify what needs to be done, but not how to do it.• ATs require an implementation layer to invoke the

application.• AT definitions must be interpreted – parsed and mapped

onto corresponding implementation.• Tests must be definable and executable prior to

implementation.

11

Test Definition / Test Implementation

• Request: “http://google.com”

• Check title: “Google”

• Click link: “Jobs”

• Check title: “About Google”

• Click link: “Press”

• Click link: “Review”

• Check title: “Google Press Room”

Interpreter

public void Request(string url) { }

public void CheckTitle(string title) { }

public void ClickLink(string link) { }

Test: Verify Google Links Class: WebPageFixtureAT Framework

12

Difference: ATs test through the same interface.

• UTs test at level of a single unit (class).• ATs test at level of user/client interaction.• All tests go through a common gateway (Http, Soap, UI

controls or application façade)• Considerable opportunity for reuse across test

implementations.• ATs need a framework in the technology of the interface

it drives: Web, GUI, Web Service, Messaging etc.

13

Testing through the application gateway.

Application G

ateway

ApplicationDefinition Implementation

Application G

ateway T

est Client

Acceptance T

est Interpreter

ATD

ATD

ATimpl

ATimpl

ATimpl

ATD

14

Difference: ATs test the Users’ Experience.

• UTs should test all application logic in detail.• ATs encourage the customer to consider the all aspects

of user experience.• ATs should test:

• Interaction & Flow• Performance• Error Handling• Security

• ATs don’t test:• Usability• Look & Feel

15

Difference: ATs are end-to-end.

• UTs test single unit in isolation.• ATs test interaction of all layers in the system.• ATs require external systems to be operating correctly.

16

Difference: ATs are process-driven.

• Each UT typically tests a single public method.• ATs consist of a sequence of steps simulating a client

interacting with the system.• ATs may test the same steps repeatedly in different

variations with different data.• Considerable overlap across different ATs.• Single point of failure will break multiple ATs.

17

Difference: ATs have serious side-effects.

• For UTs, setup and teardown code should be common only to a single fixture.

• ATs interact extensively with external systems and the environment.

• ATs typically require separate setup and teardown per test as different tests will have different side-effects.

• Setup and teardown routines are commonly shared across tests and even across stories.

18

Difference: ATs suffer concurrency issues.

• UTs should be largely independent of their environment.• ATs require that the environment is setup as expected

by the tests. • Problems can result from different users running ATs

concurrently.• Ensure sufficient variation in the data used for each test

to prevent collisions.

19

Difference: Failing ATs don’t break the build.

• UTs must pass when checked in – failure breaks the build.

• ATs may be written at the start of a story and may not pass until the very end of the iteration.

• We mark ATs as [Ignore] until they have been fully implemented.

20

Difference: ATs don’t go quietly

• UTs are designed to run silently, with all verification encoded in the assertions.

• Customer requires visual feedback to give reassurance that test does what is expected.

• ATs need to provide step-by-step visibility of the contents and results of running the tests.

21

Difference: ATs are useful outside of dev environment.

• UTs are primarily useful to developers for ensuring the code does what was intended of it.

• ATs on the other hand can provide a good assessment of the overall health of the system.

• This makes them useful for assuring deployments to testing environments are sound.

• To do this usually requires environmental parameters to be extracted to a configurable location.

22

Difference: ATs mock out systems, not units.

• Introducing mocks for dependencies in UTs helps to decouple the code.

• We often have the same needs in ATs to mock third party interfaces that are not always available.

• Mocking is very hard because the AT is outside of the application context so a specific interface for the system being mocked has to be created.

• Mocking is still important to maintain customer’s trust in the solution.

• Introduce mocks to help simulate failure conditions.

Implementation

Mo

ck

Application Interface

Unit TestFixture

Setup

Teardown

UnitImplementation

impl

impl

impl

impl

impl

impl

UT

UT

UT

UT

UT

UT

AcceptanceTest Definition

ATimpl

ATD

ATimpl

ATD

ATimpl

ATD

setup

teardown

setup

teardown

setup

teardown

Acceptance T

est Client