Get Full Value from Your Automated Testsxunitpatterns.com/~gerard/StarCanada2015Tutorial.pdf ·...
Transcript of Get Full Value from Your Automated Testsxunitpatterns.com/~gerard/StarCanada2015Tutorial.pdf ·...
1
1 Copyright 2015 Gerard Meszaroshttp://StarTutorial2015.testAutomationPatterns.com
Get Full Value from YourAutomated Tests
Gerard MeszarosIndependent Consultant
CTO of [email protected]
These Slides:http://StarTutorial2015.testAutomationPatterns.com
2 Copyright 2015 Gerard Meszaroshttp://StarTutorial2015.testAutomationPatterns.com
This is final version of my tutorialslides and handouts.
See you in Vancouver June 23rd!
Gerard
2
3 Copyright 2015 Gerard Meszaroshttp://StarTutorial2015.testAutomationPatterns.com
Product & I.T.
I.T.
EmbeddedTelecom
My Background
Gerard [email protected]
•Software developer
•Development manager
•Project Manager
•Software architect
•OOA/OOD Mentor
•Requirements (Use Case) Mentor
•XP/TDD Mentor
•Agile PM Mentor
•Test Automation Consultant & Trainer
•Lean/Agile Coach/Consultant
4 Copyright 2015 Gerard Meszaroshttp://StarTutorial2015.testAutomationPatterns.com
Product & I.T.
I.T.
EmbeddedTelecom
My Background
Gerard [email protected]
•Software developer
•Development manager
•Project Manager
•Software architect
•OOA/OOD Mentor
•Requirements (Use Case) Mentor
•XP/TDD Mentor
•Agile PM Mentor
•Test Automation Consultant & Trainer
•Lean/Agile Coach/Consultant
3
5 Copyright 2015 Gerard Meszaroshttp://StarTutorial2015.testAutomationPatterns.com
Agenda• Motivation?
– Why we need to rethink how we test and automate• Overall Strategy
– Tests as Specification, made executable• Tactics and Practices
– Scope vs. Detail, Refactoring, Collaboration• Specification Hands-On
– A chance to try this out• Automation and Tools
– How do we automate these specifications?
6 Copyright 2015 Gerard Meszaroshttp://StarTutorial2015.testAutomationPatterns.com
Functional Test Automation?
4
7 Copyright 2015 Gerard Meszaroshttp://StarTutorial2015.testAutomationPatterns.com
Unit Test Automation
8 Copyright 2015 Gerard Meszaroshttp://StarTutorial2015.testAutomationPatterns.com
Effectiveness of Automation
5
9 Copyright 2015 Gerard Meszaroshttp://StarTutorial2015.testAutomationPatterns.com
Happiness With Test Automation?
10 Copyright 2015 Gerard Meszaroshttp://StarTutorial2015.testAutomationPatterns.com
So Why Are You Unhappy?
6
11 Copyright 2015 Gerard Meszaroshttp://StarTutorial2015.testAutomationPatterns.com
The Agile Test Problem
RequirementsDevelopment
Testing
12 Copyright 2015 Gerard Meszaroshttp://StarTutorial2015.testAutomationPatterns.com
The Agile Test Problem
• As development increments reduce induration, testing needs to be reducedaccordingly
TestingDevelopment
Requirements
TestingDevelopment
Requirements
7
13 Copyright 2015 Gerard Meszaroshttp://StarTutorial2015.testAutomationPatterns.com
The Agile Test Problem
... and traditional approaches to testing nolonger work
TestingDevelopment
Requirements
TestingDevelopment
Requirements
TestingDevelopment
Requirements
TestingDevelopment
Requirements
14 Copyright 2015 Gerard Meszaroshttp://StarTutorial2015.testAutomationPatterns.com
ROI of (Automated) Tests
ROI = Value / Cost
ROI: Return on Investment
8
15 Copyright 2015 Gerard Meszaroshttp://StarTutorial2015.testAutomationPatterns.com
Exercise: Value of Tests1. What value can automated tests provide?
2. How can this value be increased?
16 Copyright 2015 Gerard Meszaroshttp://StarTutorial2015.testAutomationPatterns.com
Exercise: Cost of Tests1. What are the costs associated with
automated tests?
2. How Can Cost of Tests be Reduced?
9
17 Copyright 2015 Gerard Meszaroshttp://StarTutorial2015.testAutomationPatterns.com
Value of Tests• Validate correct functionality was built
– Built the right product
• Verify implementation is correct– Built the right product right
18 Copyright 2015 Gerard Meszaroshttp://StarTutorial2015.testAutomationPatterns.com
Value of Tests• Validate correct functionality was built
– Built the right product– After the fact is much too late to discover this!– Should be discovered before development starts.
• Verify implementation is correct– Built the right product right– Tests as Defect Detection– But, Cannot test quality into the product
10
19 Copyright 2015 Gerard Meszaroshttp://StarTutorial2015.testAutomationPatterns.com
Business
Test “After”
• Different bosses & definitions of success
Development TestTeam
20 Copyright 2015 Gerard Meszaroshttp://StarTutorial2015.testAutomationPatterns.com
Test&Fix Cycle Battleship™
DevelopmentIndependent
Test Team
Requirements
B6! Missed!
Requirements
11
21 Copyright 2015 Gerard Meszaroshttp://StarTutorial2015.testAutomationPatterns.com
Result of Test “After”• Hard to predict when product will be “done”
– Since unknown number of test&fix cycles required• Lots of effort wasted building the wrong
product and then fixing it– Why not build the right product the first time?
• Product is not designed for testability– Because there is no “testability spec”– Therefore, automated testing is difficult
22 Copyright 2015 Gerard Meszaroshttp://StarTutorial2015.testAutomationPatterns.com
Conway’s Law• “The Architecture of System will resemble the
Organization that Built it.”
DevelopmentTeam
TestTeam
BusinessTeam
12
23 Copyright 2015 Gerard Meszaroshttp://StarTutorial2015.testAutomationPatterns.com
Resulting Architecture
Specifi-cations
ProductCode
TestCode
24 Copyright 2015 Gerard Meszaroshttp://StarTutorial2015.testAutomationPatterns.com
Resulting Tests Are:• Forced to find sub-optimal ways to interact
with the Product Code– Via interfaces not designed for tests– Can only test the whole system, not the parts that
contain the logic we want to test• Forced to find ways to control the state of the
system– Can’t set state directly; must use convoluted paths to
set the state
13
25 Copyright 2015 Gerard Meszaroshttp://StarTutorial2015.testAutomationPatterns.com
Where Does This Leave Us?• Tests are more complex than they should be• Test coverage is poor so important bugs are
missed• Tests are fragile with frequent “false positive”
test failures• Tests require frequent, expensive
maintenance as code changes• Tests take longer to run
26 Copyright 2015 Gerard Meszaroshttp://StarTutorial2015.testAutomationPatterns.com
Agenda– Overall Strategy• Motivation?
• Overall Strategy– Tests as Specification, made executable
• Tactics and Practices– Scope vs. Detail, Refactoring, Collaboration
• Specification Hands-On– A chance to try this out
• Automation and Tools– How do we automate these specifications?
14
27 Copyright 2015 Gerard Meszaroshttp://StarTutorial2015.testAutomationPatterns.com
Example-Driven Development• Business & Testers come up with examples;
Developers help them formalize it.• Developers write interpreters for examples;
design product code to support them• Developers write product code guided by
execution of the examples
This is Like TDD but at a Very Large ScaleSometimes called Behaviour-Driven Dev’t (BDD)
28 Copyright 2015 Gerard Meszaroshttp://StarTutorial2015.testAutomationPatterns.com
ExampleInterpreter
Code
D,TB,T,D
ProductCode
D
Example-Driven Development
This is Like TDD but at a Very Large ScaleSometimes called Behaviour-Driven Dev’t (BDD)
PotentiallyExecutableExamples
15
29 Copyright 2015 Gerard Meszaroshttp://StarTutorial2015.testAutomationPatterns.com
Example-Driven Development• Runtime View:
ProductCode
ExampleInterpreter
Code
B,T,D D,TDExample
ExecutionResults
PotentiallyExecutableExamples
30 Copyright 2015 Gerard Meszaroshttp://StarTutorial2015.testAutomationPatterns.com
Example-Driven Development• Feedback Loops:
PotentiallyExecutable
Specifications ProductCode
ExampleInterpreter
CodeHow they say
ExampleExecutionResults
What they say
16
31 Copyright 2015 Gerard Meszaroshttp://StarTutorial2015.testAutomationPatterns.com
Example-Driven Development• Feedback Loops:
PotentiallyExecutable
Specifications ProductCode
ExampleInterpreter
Code
ExampleExecutionResults
What they say
How they say
Isn’t This “Just”Test AutomationBy Another Name?
32 Copyright 2015 Gerard Meszaroshttp://StarTutorial2015.testAutomationPatterns.com
Life Cycle of an Example
UserGoal
Feature
StoryTitle
StoryNarrative
StoryScenarios
StoryExamples
ExecutableExamples
SatisfiedExamples
17
33 Copyright 2015 Gerard Meszaroshttp://StarTutorial2015.testAutomationPatterns.com
Activities on an Example
UserGoal
Feature
StoryTitle
StoryNarrative
StoryScenarios
StoryExamples
ExecutableExamples
SatisfiedExamples
FormalizationAutomation
ProductDevelopment
Make ConcreteAdd Data
DefineAcceptance
Criteria
WorkBreakdown
ProductDesign &Validation
Elaborate
34 Copyright 2015 Gerard Meszaroshttp://StarTutorial2015.testAutomationPatterns.com
The Goal:• All functionality is clearly specified and
covered by automated checking– Of product against the examples/specification
• All traditional functional testing is now“regression testing”– Since developers had it working before they declared it
“done”• Human testers can focus on exploratory
testing– Instead of running the same functional tests every
iteration
18
35 Copyright 2015 Gerard Meszaroshttp://StarTutorial2015.testAutomationPatterns.com
Exercise: What’s in a Name• How does calling it an “example” or “spec”
change your expectations and approach?
36 Copyright 2015 Gerard Meszaroshttp://StarTutorial2015.testAutomationPatterns.com
Agenda– Tactics & Practices• Motivation?
• Overall Strategy• Tactics and Practices
– Getting the Terminology Right;» Managing Scope vs. Detail» Use the terminology of the domain
– Refactoring Examples (Simplifying, Clarifying)– Collaborative Process
• Specification Hands-On– A chance to try this out
• Automation and Tools– How do we automate these specifications?
19
37 Copyright 2015 Gerard Meszaroshttp://StarTutorial2015.testAutomationPatterns.com
How to Make Examples Understandable• Traditional automated tests are code!
– Provide all details that product code requires– Way more detail than reader can easily understand
• We need to focus on reader’s needs– Not the computer’s or products needs
• But “The Devil is in the Details”– Yes, we need to specify the details somewhere– But not everywhere!– Can be hidden in test harness for most examples– And specified in just a few very detailed examples
38 Copyright 2015 Gerard Meszaroshttp://StarTutorial2015.testAutomationPatterns.com
Test Automation Pyramid• Large numbers of very
small unit tests– Ensures integrity of code
• Smaller number offunctional tests for majorcomponents– Verify integration of units
• Even fewer tests for theentire application &workflow– Ensure application(s)
support users’requirements
• Tools to supporteffective exploratorytesting
Unit Tests
ComponentTests
SystemTests
Pyramid originally proposed by Mike Cohn
Exploratory Tests
20
39 Copyright 2015 Gerard Meszaroshttp://StarTutorial2015.testAutomationPatterns.com
Behavior Specification at Right LevelNeed to specify at the right detail and scope.
Broad Narrow
Detail
High
L
ow
Scope
Unit Tests
ComponentTests
SystemTests
40 Copyright 2015 Gerard Meszaroshttp://StarTutorial2015.testAutomationPatterns.com
Behavior Specification at Right LevelNeed to specify at the right detail and scope.
Broad Narrow
Detail
High
L
ow
Scope
ComponentTests
SystemTests
21
41 Copyright 2015 Gerard Meszaroshttp://StarTutorial2015.testAutomationPatterns.com
ComponentTests
SystemTests
Behavior Specification at Right Level
Rules &Algorithms
Multi-Use Case
WorkflowsTransactions(Use Cases)
IncompleteSpec
Toomuch detailUnmaintainable
Detail
High
Low
Broad NarrowScope
• Specify broad scope at minimum detail• Specify most detailed req’ts at narrowest scope
42 Copyright 2015 Gerard Meszaroshttp://StarTutorial2015.testAutomationPatterns.com
Use Cases & User Stories
ConfigureNotification
Rules
SuspendNotification
ResumeNotification
AccountHolder
ProcessTransaction
TransactionSettlement
Not
ify
base
d on
cha
rge
Type
Not
ify
base
d on
cha
rge
Cont
inen
tN
otif
y ba
sed
on c
harg
e Co
untr
yN
otif
y ba
sed
on c
harg
e Ci
ty &
Sta
te
Susp
end
Not
ific
atio
n
22
43 Copyright 2015 Gerard Meszaroshttp://StarTutorial2015.testAutomationPatterns.com
Testing Notifications• Open MegaBank app• Log in as “BobMa” with password ******• Click on “Manage Notifications” tab• Click on “Add New Rule” button• Select account “10035692877”• Type “Default Rule” into field “rule name”• Type “1000” into field “threshold amount”• Click on “all transaction types” radio button• Click on “all locations” radio button• Click on “save changes” button
And this is just the first step in the overall workflow!
44 Copyright 2015 Gerard Meszaroshttp://StarTutorial2015.testAutomationPatterns.com
Testing Notifications – 1/2Given:
User and Accounts
Anotherattempt:
When:Notification
Rule is Configured
Then:Notification RuleConfigured and
Active
Then:Notification Rule
is Active
23
45 Copyright 2015 Gerard Meszaroshttp://StarTutorial2015.testAutomationPatterns.com
Testing Notifications – 2/2
Then: ExpectedNotifications
When: TheTransactions tobe processed
Use Case:Process
Transactions
Anotherattempt:
46 Copyright 2015 Gerard Meszaroshttp://StarTutorial2015.testAutomationPatterns.com
Issues With This Test• Difficult to understand which TX’s should notify
– because cause (rules) and effect (notification) are far apart• Only verifies one simple combination of rules
– We will require many more tests to test all the othercombinations
– Lots of repetition of workflow & data across test cases• Simplest workflow;
– More complex workflows will be even longer and harder tounderstand
• Tests will take a long time to run– Due to need to configure first, then run transaction
processing
24
47 Copyright 2015 Gerard Meszaroshttp://StarTutorial2015.testAutomationPatterns.com
How Can We Improve Tests?• Make tests shorter
– Fewer steps• Make tests less detailed
– Omit unnecessary information• Focus tests on specific aspects of behaviour
– Test/Spec Algorithms & Rules directly– Test less functionality, but more thoroughly
Fine advice, but too vague
Difficult without Test-Friendly Architecture
48 Copyright 2015 Gerard Meszaroshttp://StarTutorial2015.testAutomationPatterns.com
How Can Example Mindset Help• Describe overall workflow
– with much less detail• Provide more details
– For individual transactions• Provide even more details
– For business rules.• But not embedded in the workflow
specifications
25
49 Copyright 2015 Gerard Meszaroshttp://StarTutorial2015.testAutomationPatterns.com
Changing Level of Abstraction/Detail• Need to Reduce Detail or Reduce Scope
Rules &Algorithms
Multi-Use Case
WorkflowsTransactions(Use Cases)
Toomuch detailUnmaintainable
IncompleteSpec
Broad Narrow
Detail
High
L
ow
Scope
50 Copyright 2015 Gerard Meszaroshttp://StarTutorial2015.testAutomationPatterns.com
Omitting Unnecessary Information• Ask not what you can include in an example…
• Ask instead:
• What can I leave out of this example?
“If it isn’t essential to conveying the essence ofthe behavior, it is essential to not include it.”
26
51 Copyright 2015 Gerard Meszaroshttp://StarTutorial2015.testAutomationPatterns.com
Refactoring Workflow ExampleGiven:
User and Accounts
Example:
When:Notification
Rule is Configured
Then:Notification RuleConfigured and
Active
Then:Notification Rule
is Active
52 Copyright 2015 Gerard Meszaroshttp://StarTutorial2015.testAutomationPatterns.com
Example:Refactoring Workflow Example
27
53 Copyright 2015 Gerard Meszaroshttp://StarTutorial2015.testAutomationPatterns.com
Then: ExpectedNotifications
Example:
When: TheTransactions tobe processed
Use Case:Process
TransactionsRefactoring Workflow Example
54 Copyright 2015 Gerard Meszaroshttp://StarTutorial2015.testAutomationPatterns.com
Example:Refactoring Workflow Example
28
55 Copyright 2015 Gerard Meszaroshttp://StarTutorial2015.testAutomationPatterns.com
Example:Refactoring Workflow Example
56 Copyright 2015 Gerard Meszaroshttp://StarTutorial2015.testAutomationPatterns.com
Example: Given:User &
Thresholds
When:Transactions
Are Processed
Then:We Expect
Notifications
Refactoring Workflow Example
29
57 Copyright 2015 Gerard Meszaroshttp://StarTutorial2015.testAutomationPatterns.com
Example: Given:User &
Thresholds
When:Transactions
Are Processed
Then:We Expect
Notifications
credit
credit
9991113333
Refactoring Workflow Example
58 Copyright 2015 Gerard Meszaroshttp://StarTutorial2015.testAutomationPatterns.com
9991113333
Example:
Broad Scope (Multi-Actor);Minimum Detail (per Actor/Transaction);
Given:User &
Thresholds
When:Transactions
Are Processed
Then:We Expect
Notifications
credit
credit
Refactoring Workflow Example
30
59 Copyright 2015 Gerard Meszaroshttp://StarTutorial2015.testAutomationPatterns.com
Specifying Transaction Details• So, where should omitted details go?
Rules &Algorithms
Multi-Use Case
WorkflowsTransactions(Use Cases)
IncompleteSpec
Toomuch detailUnmaintainableBroad Narrow
Detail
High
L
ow
Scope
60 Copyright 2015 Gerard Meszaroshttp://StarTutorial2015.testAutomationPatterns.com
Single Use Case ExampleUse Case:
ManageNotifications
Medium Scope (Single Actor)Medium Detail (Transaction, not UI);
Example:
Given: User & Accounts
When: Customer sets up threshold
Then: We ExpectLogging,
Updated Statuses
Then:We Expect Logging& Updated Statuses
31
61 Copyright 2015 Gerard Meszaroshttp://StarTutorial2015.testAutomationPatterns.com
Specifying Business Rules• Need the more detail; need to reduce scope
Rules &Algorithms
Multi-Use Case
WorkflowsTransactions(Use Cases)
IncompleteSpec
Toomuch detailUnmaintainableBroad Narrow
Detail
High
L
ow
Scope
62 Copyright 2015 Gerard Meszaroshttp://StarTutorial2015.testAutomationPatterns.com
Kinds of Business Rules & Algorithms• Generation of Choices
– Pick List Contents» Payment methods allowed,
• Validation of inputs– user names, passwords, customer names, addresses
• Calculation of derived values– Cost of order, taxes
• Basis of decisions– Should you do something
» Should you notify?– How should you do something?
» How should you notify? E-mail? Text? Phone Call?
32
63 Copyright 2015 Gerard Meszaroshttp://StarTutorial2015.testAutomationPatterns.com
Business Rule Example
Process TransactionThreshold per Charge Type
Example:
Configuration
64 Copyright 2015 Gerard Meszaroshttp://StarTutorial2015.testAutomationPatterns.com
Business Rule Example
Process TransactionThreshold per Charge Type
Narrow Scope (Single Rule)High Detail (Everything that matters)
Example:
Configuration Given theserules
Then: Theanswer should be
When we askNotificationRequired?with this transaction:
33
65 Copyright 2015 Gerard Meszaroshttp://StarTutorial2015.testAutomationPatterns.com
Agenda– Specification Hands-On• Motivation?
• Overall Strategy• Tactics and Practices• Specification Hands-On
– Workflow Specification– Transaction Specification– Business Rules Specification
• Automation and Tools– How do we automate these specifications?
66 Copyright 2015 Gerard Meszaroshttp://StarTutorial2015.testAutomationPatterns.com
Exercise – Write Some Examples• Write a Workflow Example
– Identify a few scenarios– Identify what would cause different scenarios to be executed– Write a specification for one of them– Identify any business rules or calculations that would be involved; add
them to your list of algorithms and rules
• Write Examples for a Business Rule or Two– Describe the behaviour of an algorithm or business rule identified in
your workflow examples using tables.– Focus on what the inputs/outputs are, not where they came from
34
67 Copyright 2015 Gerard Meszaroshttp://StarTutorial2015.testAutomationPatterns.com
Exercise Logistics• Pair up with someone
– Avoids getting stuck; learn from each other• Pick an Application At Least One of You Knows
– Could be your own application– Or a public domain one like Amazon
• Pick a couple of related workflows– Choose variations of same workflow so there is some
common steps– But there should be some different steps too.
68 Copyright 2015 Gerard Meszaroshttp://StarTutorial2015.testAutomationPatterns.com
Exercise Logistics (2)• You Can Choose to Work:
– Outside In: Workflow example first, then Business Ruleexample
– All at Once: Put details into Business Rule examplewhile sketching out Workflow in higher level example
• Don’t Worry About How to Automate– We’ll discuss this later
35
69 Copyright 2015 Gerard Meszaroshttp://StarTutorial2015.testAutomationPatterns.com
List of Workflow Scenarios
70 Copyright 2015 Gerard Meszaroshttp://StarTutorial2015.testAutomationPatterns.com
List of Business Rules & Algorithms
36
71 Copyright 2015 Gerard Meszaroshttp://StarTutorial2015.testAutomationPatterns.com
Workflow Example Template
72 Copyright 2015 Gerard Meszaroshttp://StarTutorial2015.testAutomationPatterns.com
Workflow Example Template
37
73 Copyright 2015 Gerard Meszaroshttp://StarTutorial2015.testAutomationPatterns.com
Business Rule Examples Template
74 Copyright 2015 Gerard Meszaroshttp://StarTutorial2015.testAutomationPatterns.com
Business Rule Examples Template
38
75 Copyright 2015 Gerard Meszaroshttp://StarTutorial2015.testAutomationPatterns.com
Example: List of Scenarios• Basic Notification Workflow
– Notify user of a transaction above threshold– Don’t notify user of a transaction below threshold– Changing threshold for one user doesn’t affect threshold
for other users• Suspend & Resume Notifications
– No notification is sent while notifications are suspended– Notifications are sent for new transactions when
notifications are resumed– Suspending one user doesn’t affect other users
notification state
76 Copyright 2015 Gerard Meszaroshttp://StarTutorial2015.testAutomationPatterns.com
Example: List of Business Rules & Algorithms
• Content Generation– What is the list of accounts associated with this user’s
login credentials?– What types of transactions should the user be able to
pick?• Notification Decision
– Should we notify the user about this transaction?– By what means should we notify the user?
39
77 Copyright 2015 Gerard Meszaroshttp://StarTutorial2015.testAutomationPatterns.com
Example: Notification Workflow (initially)Given BobMa has set notification threshold to $10,000
for all transactions on his accountWhen the bank processes debit for 15,000 to his accountAnd the bank processes debit for 9,999 to his accountAnd the bank process debit for 10,000 to his accountThen BobMa receives notification for debit 15,000And BobMa receives notification for debit 10,000When the bank processes debit for 10,000 to Fred’s
accountThen FredS receives no notification
78 Copyright 2015 Gerard Meszaroshttp://StarTutorial2015.testAutomationPatterns.com
Example: Notification Workflow (simplified)Given BobMa has set notification threshold to $10,000 for all
transactions on his accountWhen the bank processes debit for 10,000 to his accountThen BobMa receives notification for debit 10,000
Given BobMa has set notification threshold to $10,000 for alltransactions on his account
When the bank processes debit for 9,999.99 to his accountThen BobMa receives no notification
Given BobMa has set notification threshold to $10,000 for alltransactions on his account
And FredS has not set notification threshold on his accountWhen the bank processes debit for 10,000 to his accountThen FredS receives no notification
Positive Case
negative Case:< Threshold
negative Case:Another user
40
79 Copyright 2015 Gerard Meszaroshttp://StarTutorial2015.testAutomationPatterns.com
Example: Suspension WorkflowGiven BobMa has set notification threshold to $10,000
for all transactions on his accountWhen the bank processes debit for 15,000 to his accountThen BobMa receives notification for debit 15,000
When BobMa suspends notification on his accountAnd the bank processes debit for 15,000 to his accountThen BobMa receives no notification
When BobMa resumes notification on his accountAnd the bank processes debit for 15,000 to his accountThen BobMa receives notification for debit 15,000
80 Copyright 2015 Gerard Meszaroshttp://StarTutorial2015.testAutomationPatterns.com
Example: Suspension WorkflowGiven BobMa has set notification threshold to $10,000
for all transactions on his accountWhen the bank processes debit for 15,000 to his accountThen BobMa receives notification for debit 15,000
When BobMa suspends notification on his accountAnd the bank processes debit for 15,000 to his accountThen BobMa receives no notification
When BobMa resumes notification on his accountAnd the bank processes debit for 15,000 to his accountThen BobMa receives notification for debit 15,000
Over threshold
Over threshold
Over threshold
41
81 Copyright 2015 Gerard Meszaroshttp://StarTutorial2015.testAutomationPatterns.com
Example: Suspension Workflow (simplified)Given BobMa has set up notification for all transactions
on his accountWhen the bank processes an over-threshold-txThen BobMa receives notification for that tx
When BobMa suspends notification on his accountAnd the bank processes an over-threshold-txThen BobMa receives no notification for that tx
When BobMa resumes notification on his accountAnd the bank processes an over-threshold-txThen BobMa receives notification for that tx
82 Copyright 2015 Gerard Meszaroshttp://StarTutorial2015.testAutomationPatterns.com
Example: Business Rule Spec
Process TransactionThreshold per Charge Type
Configuration
42
83 Copyright 2015 Gerard Meszaroshttp://StarTutorial2015.testAutomationPatterns.com
Example: Single Use-Case Test Use Case:Manage
Notifications
Medium Scope (Single Actor)Medium Detail (Transaction, not UI);
Given: User & Accounts
When: Customer sets up threshold
Then: We ExpectLogging,
Updated Statuses
Then:We Expect Logging& Updated Statuses
84 Copyright 2015 Gerard Meszaroshttp://StarTutorial2015.testAutomationPatterns.com
Exercise – Simplify Your Examples• Simplify Your Workflow Examples
– Review your list of workflow scenarios– Write a specification for another workflow example– Refactor both examples, simplify & clarify– Ruthlessly remove anything that isn’t needed
• Move Complexity into Transaction Examples– If Workflow examples are too complex (too much detail, more than
one steps per user/actor), try moving details/steps into examples forsingle transactions
• Simplify Your Business Rule Examples– Review your rule/algorithm examples– Add additional examples as needed– Ruthlessly remove anything that isn’t needed
43
85 Copyright 2015 Gerard Meszaroshttp://StarTutorial2015.testAutomationPatterns.com
Agenda– Automation and Tools• Motivation?
• Overall Strategy• Tactics and Practices• Specification Hands-On• Automation and Tools
– How do we automate these specifications?– What tools can we use to automate them?
86 Copyright 2015 Gerard Meszaroshttp://StarTutorial2015.testAutomationPatterns.com
Automating Workflow Example – via User InterfaceGiven BobMa has set notification threshold to $10,000
for all transactions on his accountLogin as BobMaGo to notification configuration pageSelect chequing accountSelect “edit notification rules”Choose action “add new rule”Set tx type to “All” and threshold to “10,000”…
When the bank processes debit for 10,000 to BobMa’saccount
Add transaction record for “debit” “10,000” “bobma’s account” totransaction queue
Run the batch job “processTransactionQ”
Then BobMa receives notification for debit 10,000Run DB query “select * in notifications where date=today and user=“BobMa”Ensure count is 1 and record.amount=10,000
44
87 Copyright 2015 Gerard Meszaroshttp://StarTutorial2015.testAutomationPatterns.com
Test-After Architecture• Must test through User Interface
• Must load lots of data
System Under Test
Should weNotify?Process
Transaction
ConfigureNotificationThreshold
NotificationRules
TransactionInterface
ConfigurationUser
Interface
NotificationLog
WorkflowTest
DoNotification.
88 Copyright 2015 Gerard Meszaroshttp://StarTutorial2015.testAutomationPatterns.com
Example-Driven Architecture• Need to provide API’s to invoke functionality
• Bypassing the UI• Supports workflow and
use case / transaction example automation
System Under Test
Should weNotify?
DoNotification.
ProcessTransaction
ConfigureNotificationThreshold
NotificationRules
NotificationLog
TransactionInterface
ConfigurationUser
InterfaceWorkflowExample
WorkflowKeyword
Interpreter
45
89 Copyright 2015 Gerard Meszaroshttp://StarTutorial2015.testAutomationPatterns.com
Automating Workflow Specification – via API
Given BobMa has set notification threshold to $10,000for all transactions on his account
sut.setNotificationThresholdForUserAccount( “bobma”, “chequing”, 10,000);
When the bank processes debit for 10,000 to his accountsut.processTransactions( [ {“bobma”,”chequing”,”debit”, 10,000} ] );
Then BobMa receives notification for debit 10,000allTx = sut.getTransactionNotificationsForAccount( “BobMa”, “chequing”);assertAllTx(alltx, {“BobMa”, “chequing”, 10,000});
Before Eachsut = new MegaBankNotificationsAPI();bobma = sut.createUser(“bobma”, ….);sut.addUserAccount(“bobma”, “checking”, …..);
sut =
90 Copyright 2015 Gerard Meszaroshttp://StarTutorial2015.testAutomationPatterns.com
Automating Workflow Specification – via API(made parameterized)
Given {{user}} has set notification threshold to {{amount}} forall transactions on his account
sut.setNotificationThresholdForUserAccount( {{user}},“chequing”, {{amount}});
When the bank processes debit for {{amount}} to his accountsut.processTransactions( [ { {{user}}, ”chequing” , ”debit”, {{amount}} } ] );
Then {{user}} receives notification for debit {{amount}}allTx = sut.getTransactionNotificationsForAccount( {{user}} , “chequing”);assertAllTx(alltx, { {{user}} , “chequing”, {{amount}} });
Before Eachsut = new MegaBankNotificationsAPI();bobma = sut.createUser(“bobma”, ….);sut.addUserAccount(“bobma”, “checking”, …..);
sut =
46
91 Copyright 2015 Gerard Meszaroshttp://StarTutorial2015.testAutomationPatterns.com
EDA – Single Tx Examples• Same API supports Transaction (Use Case)
examples: System Under Test
Should weNotify?
DoNotification.
ProcessTransaction
ConfigureNotificationThreshold
NotificationRules
NotificationLog
TransactionInterface
ConfigurationUser
Interface
Config’nTX
Example
Config’nKeyword
Interpreter
92 Copyright 2015 Gerard Meszaroshttp://StarTutorial2015.testAutomationPatterns.com
Business Rule Example
Process TransactionThreshold per Charge Type
Example:
Configuration
47
93 Copyright 2015 Gerard Meszaroshttp://StarTutorial2015.testAutomationPatterns.com
ConfigureNotificationThreshold System Under Test
EDA – Business Rules• Encapsulation of Rules in “Rules Component”
• which can be accessed directly via the Test
DoNotification.
ProcessTransaction
NotificationRules
NotificationLog
TransactionInterface
ConfigurationUser
Interface
NotificationMethod Example
NotificationRules Fake
NotificationRule Example
Should weNotify?
94 Copyright 2015 Gerard Meszaroshttp://StarTutorial2015.testAutomationPatterns.com
EDA – Business Rules• Encapsulation of Rules in “Rules Component”
• which can be accessed directly via the Test
• Component Accepts Rules via “Data Injection”– Simplify Automation by avoiding need to configure
• Ensures Design-for-testability– But Only When Automation is a Dev’t Responsibility
Should weNotify?
NotificationRules Fake
NotificationRule Example
48
95 Copyright 2015 Gerard Meszaroshttp://StarTutorial2015.testAutomationPatterns.com
public class NotificationRequired ColumnFixture {
public String account;
public String ChargeType;
public Currency amount;
public notify( ) {NotificationDecider nd =
new NotificationDecider ();return nd.shouldNotify(account, chargeType, amount);
}}
Sample Fit Column Fixture
96 Copyright 2015 Gerard Meszaroshttp://StarTutorial2015.testAutomationPatterns.com
Exercise: Automate Your Examples• For each of your workflow steps,
– implement a keyword in psuedo-code calling the APIyou would like the system to expose
– Pass all the information your test provided, plus– Pass default values for any info the system would
require• For each of your business rule tables,
– implement a keyword or fixture in psuedo-code callingthe business rule component API you would like thesystem to expose
– Pass input values provided by example– Ensure output values specified by example are returned
49
97 Copyright 2015 Gerard Meszaroshttp://StarTutorial2015.testAutomationPatterns.com
Workflow Keyword ImplementationKeyword:
Keyword:
Keyword:
Keyword:
98 Copyright 2015 Gerard Meszaroshttp://StarTutorial2015.testAutomationPatterns.com
Workflow Keyword ImplementationKeyword:
Keyword:
Keyword:
Keyword:
50
99 Copyright 2015 Gerard Meszaroshttp://StarTutorial2015.testAutomationPatterns.com
Business Rule Table ImplementationInput Columns:
• What would you do with them?
Output Columns:
• Where would you get them from?
100 Copyright 2015 Gerard Meszaroshttp://StarTutorial2015.testAutomationPatterns.com
Business Rule Table ImplementationInput Columns:
• What would you do with them?
Output Columns:
• Where would you get them from?
51
101 Copyright 2015 Gerard Meszaroshttp://StarTutorial2015.testAutomationPatterns.com
Agenda– Automation and Tools• Motivation?
• Overall Strategy• Tactics and Practices• Specification Hands-On• Automation and Tools
– How do we automate these specifications?– What tools can we use to automate them?
102 Copyright 2015 Gerard Meszaroshttp://StarTutorial2015.testAutomationPatterns.com
Tools Selection• Tools must support the strategy
– Support business/testers to write specs/tests inbusiness language» More or less “natural” language
– Let them write specs before the specs are automated orthe system is built» This disqualifies Capture & Replay tools
– Can be automated by technical people, later» Separation of business skills from technical skills» e.g. Table or Keyword-based tools
– Can be run by anyone, especially developers» This disqualifies expensive tester-specific tools
52
103 Copyright 2015 Gerard Meszaroshttp://StarTutorial2015.testAutomationPatterns.com
Business Rules Tools - A Sampling• Fit
– Column Fixture• SLIM (Fitnesse)
– Decision Table• Robot Framework
– Parameterized Testcase
104 Copyright 2015 Gerard Meszaroshttp://StarTutorial2015.testAutomationPatterns.com
Workflow Spec Tools (my fav’s)• Gherkin (Given-When-Then terminology)
– Cucumber (various tech-specific implementations)– JBehave (Java/JVM specific)– SpecFlow (.Net)
• Other GWT (Given-When-Then terminology)– Robot Framework GWT
» https://blog.codecentric.de/en/2009/11/givenwhenthen-and-example-tables-using-the-robot-framework/
• Robot Framework– Keyword-based Testcases
Resource: Agile Alliance Functional Testing Tools program
53
105 Copyright 2015 Gerard Meszaroshttp://StarTutorial2015.testAutomationPatterns.com
More Complete Survey of Tools
• Agile Alliance Functional Testing Toolsprogram Yahoo! group– https://groups.yahoo.com/neo/groups/aa-ftt/info
• Agile Alliance Functional Testing Toolsspreadsheet– https://docs.google.com/spreadsheet/ccc?key=0Apag_J
97l3CTdHR0WS1sZGFVaFA0dEpXYWRqLXBxV3c#gid=1
106 Copyright 2015 Gerard Meszaroshttp://StarTutorial2015.testAutomationPatterns.com
Exercise: Automation Tool AssessmentAssess each of the “test automation” tools your
organization currently uses (or is consideringfor future use) for suitability to this approach.
Use the worksheet on the next page.
54
107 Copyright 2015 Gerard Meszaroshttp://StarTutorial2015.testAutomationPatterns.com
Automation Tool AssessmentTool Name Supports
naturallanguage?
Can be specifiedbefore system isbuilt?
Automation canbe done byDevelopers?
Can be run byDevelopersToo?
108 Copyright 2015 Gerard Meszaroshttp://StarTutorial2015.testAutomationPatterns.com
Summary of Approach• Business & Testers come up with examples;
Developers help them formalize it.– Ensures everyone agrees on what needs to be built
• Manage Scope vs. Detail in examples– Different kinds of examples for each of:
» Workflow sequences» Transaction interactions» Business rules and algorithms
– Avoids long, hard-to-understand examples
55
109 Copyright 2015 Gerard Meszaroshttp://StarTutorial2015.testAutomationPatterns.com
Summary of Approach• Developers write interpreters for examples;
design product code to support them– Ensures the architecture supports automation
• Developers write product code guided byexecution of the examples– They know what needs to be built– They can run the examples frequently to know how they
are doing
110 Copyright 2015 Gerard Meszaroshttp://StarTutorial2015.testAutomationPatterns.com
Summary of Benefits• Examples encourage collaboration between all
project roles.• Built-in automation helps avoid Fragile Tests• Automated Examples help us prevent defects
being introduced through misunderstandingof requirements
• Automated Examples ensure complete “test”coverage of functionality
• Frees up testers to do more ExploratoryTesting
•
56
111 Copyright 2015 Gerard Meszaroshttp://StarTutorial2015.testAutomationPatterns.com
Closing Thoughts• Are you automating to find defects or prevent
them?• Are your automated tests good examples?
– Why not? What would you need to change?• Are your tests low maintenance?
– Why not? What causes them to break?– What could you change…– … to make them break less often?– .... to reduce the impact of breakage?
112 Copyright 2015 Gerard Meszaroshttp://StarTutorial2015.testAutomationPatterns.com
Thank You!Gerard Meszaros
[email protected]://www.xunitpatterns.com
Slides:http://StarTutorial2015.testAutomationPatterns.com
Call me when you:• Want to transition to Agile or Lean• Want to do Agile or Lean better• Want to teach developers how to test• Need help with test automation strategy• Want to improve your test automation
Jolt Productivity Awardwinner - Technical Books
Available on MSDN:
57
113 Copyright 2015 Gerard Meszaroshttp://StarTutorial2015.testAutomationPatterns.com
References & Resources - Tools• AAFTT
– https://groups.yahoo.com/neo/groups/aa-ftt/info
• Cucumber– https://cucumber.io/– http://www.specflow.org/
• Fitnesse– http://www.fitnesse.org/
• Fit– http://fit.c2.com/– http://en.wikipedia.org/wiki/Framework_for_integrated_test– https://code.google.com/p/refit/
• Robot Framework– http://robotframework.org/
• Robot Framework GWT– https://blog.codecentric.de/en/2009/11/givenwhenthen-and-example-
tables-using-the-robot-framework/
114 Copyright 2015 Gerard Meszaroshttp://StarTutorial2015.testAutomationPatterns.com
References & Resources - Reading• Acceptance Test Engineering
– Grigori Melnik, Gerard Meszaros• Agile Testing + More Agile Testing
– Lisa Crisping, Janet Gregory• The Cucumber Book: Behaviour-Driven Development for
Testers and Developers– Matt Wynne and Aslak Hellesøy
• Fit for Developing Software: Framework for IntegratedTests– Ward Cunningham, Rick Mugridge
• Lean-Agile Acceptance Test Driven Development– By Ken Pugh
• Test Driven .NET Development with FitNesse• Gojko Adzic
58
115 Copyright 2015 Gerard Meszaroshttp://StarTutorial2015.testAutomationPatterns.com
Robot Framework GWT example*** Settings ***Resource ${RESOURCES}/BDD.txt
*** Keyword ***Example
[Arguments] ${periodClosed} ${periodOpenAndModified} ${importDay} ${oldManagerValidUntil}${newManagerValidFrom}
Given initialized criteria for bonus commercialAnd a branch B with branch manager M_OLD and employee E1And evaluation for E1 for period ${periodClosed} which is closedAnd evaluation for E1 for period ${periodOpenAndModified} which is open and modified
When M_NEW becomes new manager of branch BAnd import service is called on ${importDay}
Then the new branch manager of branch B is M_NEW valid from ${newManagerValidFrom}And branch manager M_OLD manages employee E until ${oldManagerValidUntil}And branch manager M_NEW manages employee E from ${newManagerValidFrom}And Evaluations for E1 still have the same content
| *Test Case* | | *Closed Period* | *Open Period* | *Run Import On* | *Old Manager Stops* | *New Manager Starts* || 1 | Example | 1.11.2009 - 30.11.2009 | 1.12.2009 - 31.12.2009 | 11.11.2009 | 30.11.2009 | 1.12.2009| 2 | Example | 1.11.2009 - 30.11.2009 | 1.12.2009 - 31.12.2009 | 1.11.2009 | 31.10.2009 | 1.11.2009| 3 | Example | 1.11.2009 - 30.11.2009 | 1.12.2009 - 31.12.2009 | 1.12.2009 | 30.11.2009 | 1.12.2009
116 Copyright 2015 Gerard Meszaroshttp://StarTutorial2015.testAutomationPatterns.com
Summary of Exercises• Value of Automated Tests• Cost of Automated Tests• What’s in a Name?• Write a Workflow Example• Write a Business Rule Example• Refactor a Workflow Example• Automate Execution of Your Examples• Automation Tool Assessment