…building your own AT framework…

21 November, 2003

Sean Hanly shanly@exoftware.com

Acceptance Tests…

- Story of a “Story”

- AT Demonstration

- Pattern- Anti-Patterns- AT Framework Patterns- Process Patterns

- Interactive Session

anything else…

The Story of a "Story”

XP Process Overview

Years

MonthsRelease

IterationWeeks

ImplementationDays

Stories

Stories

Story Story Story Story

Tasks

Test Cases

Project

Release Planning

this is a short description of a customer’s

requirementthis is a short description

of a customer’s requirement

this is a short description of a customer’s

requirementthis is a short description

of a customer’s requirement

this is a short description of a customer’s

requirementthis is a short description

of a customer’s requirement

this is a short description of a customer’s

requirementthis is a short description

of a customer’s requirement

this is a short description of a customer’s

requirementthis is a short description

of a customer’s requirement

this is a short description of a customer’s

requirementthis is a short description

of a customer’s requirement

this is a short description of a customer’s

requirementthis is a short description

of a customer’s requirement

this is a short description of a customer’s

requirement

this is a short description of a customer’s

requirementthis is a short description

of a customer’s requirement

this is a short description of a customer’s

requirementthis is a short description

of a customer’s requirement

this is a short description of a customer’s

requirementthis is a short description

of a customer’s requirement

this is a short description of a customer’s

requirementthis is a short description

of a customer’s requirement

this is a short description of a customer’s

requirementthis is a short description

of a customer’s requirement

this is a short description of a customer’s

requirementthis is a short description

of a customer’s requirement

this is a short description of a customer’s

requirementthis is a short description

of a customer’s requirement

this is a short description of a customer’s

requirement

this is a short description of a customer’s

requirementthis is a short description

of a customer’s requirement

this is a short description of a customer’s

requirementthis is a short description

of a customer’s requirement

this is a short description of a customer’s

requirementthis is a short description

of a customer’s requirement

this is a short description of a customer’s

requirementthis is a short description

of a customer’s requirement

this is a short description of a customer’s

requirementthis is a short description

of a customer’s requirement

this is a short description of a customer’s

requirementthis is a short description

of a customer’s requirement

this is a short description of a customer’s

requirementthis is a short description

of a customer’s requirement

this is a short description of a customer’s

requirement

customer + developers + tester + interaction designer

V(elocity) = 20

Release Plan

this is a short description of a customer’s

requirement

this is a short description of a customer’s

requirement

this is a short description of a customer’s

requirement

this is a short description of a customer’s

requirement

1

this is a short description of a customer’s

requirement

this is a short description of a customer’s

requirement

this is a short description of a customer’s

requirement

this is a short description of a customer’s

requirement

2

this is a short description of a customer’s

requirement

this is a short description of a customer’s

requirement

this is a short description of a customer’s

requirement

this is a short description of a customer’s

requirement

6

this is a short description of a customer’s

requirement

this is a short description of a customer’s

requirement

this is a short description of a customer’s

requirement

this is a short description of a customer’s

requirement

Bucket

…

5

5

5

5

5

5

5

5

5

5

5

5

5

5

5

5

Iteration Planning

1

this is a short description of a customer’s

requirementthis is a short description

of a customer’s requirement

this is a short description of a customer’s

requirementthis is a short description

of a customer’s requirement

+AutomatedAcceptance

Tests

customer + developers + tester + interaction designer

5

Implementation Planning

Test

Refactor Code

failingacceptance

test

passing acceptancetest

this is a short description of a customer’s

requirement

acceptancetests

5

The Story

A good user story:I Independent N Negotiable

V Valuable / Vertical

E Estimable

S Small

T Testable

Initial Story Presentation(Release Plan)

customer wants the ability to have a rules

engine that allows rules to be evalauted against a clients financial position

New Story

customer wants the ability toanalyse the amount of casha client is currently holding

-too little- too much- just right

(depends on lifestyle cost and attitude to risk) 25

A good user story:I Independent N Negotiable

V Valuable / Vertical

E Estimable

S Small

T Testable

2

Release Plan

this is a short description of a customer’s

requirement

this is a short description of a customer’s

requirement

this is a short description of a customer’s

requirement

this is a short description of a customer’s

requirement

1

this is a short description of a customer’s

requirement

this is a short description of a customer’s

requirement

this is a short description of a customer’s

requirement

this is a short description of a customer’s

requirement

this is a short description of a customer’s

requirement

customer wants the ability toanalyse the amount of casha client is currently holding

this is a short description of a customer’s

requirement

this is a short description of a customer’s

requirement

6

this is a short description of a customer’s

requirement

this is a short description of a customer’s

requirement

this is a short description of a customer’s

requirement

this is a short description of a customer’s

requirement

Bucket

…

15

15

25

10

15

25

10

15

10

25

15

15

15

25

5

10

V(elocity) = 65

Release Plan

2

this is a short description of a customer’s

requirement

this is a short description of a customer’s

requirement

this is a short description of a customer’s

requirement

this is a short description of a customer’s

requirement

1

this is a short description of a customer’s

requirement

customer wants the ability toanalyse the amount of casha client is currently holding

this is a short description of a customer’s

requirement

this is a short description of a customer’s

requirement

this is a short description of a customer’s

requirement

this is a short description. of a customer’s

reuqirement

this is a short description of a customer’s

requirement

this is a short description of a customer’s

requirement

6

this is a short description of a customer’s

requirement

this is a short description of a customer’s

requirement

this is a short description of a customer’s

requirement

this is a short description of a customer’s

requirement

Bucket

15

15

25

10

15

25

10

15

10

25

15

15

15

25

5

10

V(elocity) = 65

Iteration Two

+screen design

+interaction tests

+engine tests

25

customer wants the ability toanalyse the amount of casha client is currently holding

-too little- too much- just right

(depends on lifestyle cost and attitude to risk)

Screen Design

Interaction Tests (example)//Overview“Analyses a clients cash balance to see if they are holding too much cash given their attitude to risk and lifestyle cost…”

//setup client dataUserClicksMainMenu MenuFinancialObjectivesUserInputsText FinancialObjectivesAttitudeToRisk "3 - Low Return - Long " "Term Investment" UserClicksMainMenu MenuCurrentBalanceSheetUserInputsText CurrentBalanceSheetTotalCash 30000 UserClicksMainMenu MenuFinancialObjectivesUserInputsText FinancialObjectivesLifestyleCost 25000

//cash ruleTestValueOfText AnalyseObservation "Given your attitude to risk, you " "should maintain a cash balance no " "greater than €12,500."TestValueOfText AnalyseRecommendation "Consider moving €17,500 from cash " "accounts to investable assets."TestValueOfText AnalyseDestination "Go to Invest Capital Sum and move " "excess cash to cash deposit unless " "holding cash to buy an asset." //hyperlinkUserClicksControl AnalyseDestinationTestValueOfLabel WorkAreaTitle "Invest Capital Sum"

Engine Test (example)

Test Name

Cash AttitudeTo Risk

Lifestyle Cost

Recommended Cash Balance

Disposable Cash

Textual

Analysis

Test 1 30,000 6% 25,000 17,500 17500 Given your attitude to risk, you should maintain a cash balance no greater than €12,500. Consider moving €17,500 from cash accounts to investible assets.

Test 2 10,000 6% 25,000 -2,500 -2500 Given your attitude to risk, you should maintain a cash balance of €12,500. Consider moving €2,500 from Liquid Assets to a Cash Account.

Test 3 15,000 8% 30,000 5,000 5000 Given your attitude to risk, you should maintain a cash balance no greater than €10,000. Consider moving €5,000 from cash accounts to investible assets.

Test 4 10,000 8% 30,000 0 0 Given your attitude to risk, you have an ideal cash balance.

Demonstration

Automated Acceptance Tests

• Unambiguous, Transparent

• Controls Scope Creep

• No Assumptions

• “When do you stop?”

• Surfaces and defines requirements

• Executable Requirements Documentation

• Executable – Asset, Regression Suite

• Test Infecting your Customer

AT Mini-Patterns

Anti-Patterns

LunaticsRunningTheAsylum

• Problem– The developers create an AT framework that suits the way they think

and the technologies they like or are currently in vogue. As an example we delivered a XML, Cocoon, Java Reflection based framework to our client!!!

• Solution– Eat our own dogfood, i.e. listen to your customer whether they be a

business user, tester, interaction designer. They know what they want to achieve and how they would like it to work, not you!

– Do give them alternatives and possibilites. They may not always be aware of what is possible.

what happens when the lunatics run the asylum!!

RecordWhatAndPlaybackWhat

• Problem– Trying to use a GUI based record and playback tool

means that• AT’s cannot be created upfront and • where they are, they the tend to be extremely fragile

due to change i.e. constant re-recording and consequent frustration.

• Solution– Create an AT framework that allows upfront executable

requirements definition e.g. ACT and FIT. – However, record and playback can give you a very cheap

regression suite for legacy code bases.

MetaLanguageMadness(JAct, Tython )

• Problem– In the effort to create an AT frameowrk the “lunatics”

invent a whole new programming language. This results in un-necessary AT complexity for the customer and the temptation to overload the tests with actual logic, breaking the DRY principle.

• Solution– Remember who the framework will be used by and what

the goal of the framework is.– Create a definition language not a programming

language.

Other anti-patterns

• Test the system not the OS i.e. figure out the boundaries

• AllOrNothing – 80% is ok• Not being subtle enough in setup and

teardown– Using a copy of the complete database

• Testers/Customers will not write code• Allow for some verbosity e.g. FIT• Overly generic e.g. a GUI scripting

language

Patterns

CreateADomainLanguage(AddACommandLine)

• Problem– When creating an AT framework it is difficult to see how to expose

the system to testing in a manner that allows for definition of requirements and ease of testing. Here intimacy and testability always seem to be the problem. Additionally the framework has to talk to the customer/tester and reflect the domain. Importantly this often means allowing for scripting of actual recognizable usage scenarios.

• Solution– Create a simple verb based, (similar to OS commands), scripting

language for the application– Create high level domain verbs, e.g. CreateCustomer, not low

level calls, e.g. CallSessionBean “jndi::/Customer” createCustomer

– Hook directly into the environment– Verbs should act as client/actor to all parts of the system and be

conversational in nature

DomainLanguageInsights

• Improves Metaphor

• In the beginning there was the command line – intimacy

• Reusable

• Improves Design

• Becomes a way of interacting with the system

• Leverages existing test code

Example

//setupSetPretendTime “2003,7,23,13,5,6”DeleteAllRecordsInTheDatabase

SetPhoneBalance 353879793001 1000SetPhoneBalance 353879793002 1000SetPhoneBalance 353879793003 1000

//do workSubmitUrl “http://AddNickName…” SubmitUrl “http://AddNickName…”SubmitUrl “http://UpLoadHighScore…”

//check dbTestNumberOfRecordsInDatabase 3TestCustomerRecordEquals “1,8,…”TestCustomerRecordEquals “2,8,…”TestCustomerRecordEquals “3,8,…”

do batch workSetPretendTime “2003,7,24,13,5,6”

RunBatchJob

//check files, database and account balances

CompareFiles expected.out actual.out

TestCustomerRecordEquals “1,10,…”TestCustomerRecordEquals “2,10,…”TestCustomerRecordEquals “3,10,…”

CheckPhoneBalance 353879793001 1000CheckPhoneBalance 353879793002 1000CheckPhoneBalance 353879793003 0985

BuildInTestability

• Problem– The system in question is difficulut to test because there are no

testing hook’s available.

• Solution– When desiging/building the system add testability. Easiest done

when driving the actual development from acceptance tests. Examples include:

• Recording API’s• Query API’s• Determinism• Fakeability• Undoability• State Management

Legacy: Characterize It

• Problem– Before adding to or refactoring we need a set of acceptance tests

that provide enough feedback to ensure correctness. However, the effort to bring the systen under unit test / acceptance test is prohibitive.

• Solution– Create a coarse grained set of characterization tests that create an

invariant that immdeitely gives feedback as the system as it is changed e.g.

• Log File Comparison• Database Comparison• Record and Playback

Legacy: AddTracerBullets

• Problem– While adding acceptance tests to Legacy Code you find that there

are no testable post conditions.

• Solution– Add safe tracer bullets, i.e. not behaviour changing, to the code

base. As an example add log statements to the major entry and exit points to your system.

Other mini patterns

• TestsAreSource• BringDataUnderControl• IntegrateWithBuild• It’s all right to fake it• Trust your code• Script in the implementation environment• Add determinism• Allow for some verbosity e.g. FIT• Add baseline capability

Process Patterns

Process Patterns

• TestsAreTheRequirements

• TestsAreConversationPieces

• AddSpanners

• EdgeCaseObsession

• HappyPath

Framework Round Up

Frameworks

• Data– FIT– CSV / Excel Driven

• Script– GUI based tools– ACT

• Record Playback– nuff said!– …

ACT Demo

Bank System(stories to date)

allow for the creation of a new customer account

(open balance should be zero)

allow a customer to deposit money into an

account

allow a customer to withdraw money from an

account

Bank System(new story)

integrate deposit functionality with a fraud

checking system