5/13/16

1

© 2016 LogiGear Corporation. All Rights Reserved

STAREAST 2016, Track Session W3 Orlando, Florida

Wednesday, May 4th, 2016 11.30 AM – 12.30 PM

"Anti-Patterns" for Automated Testing

Mr. Playback

Hans Buwalda LogiGear

hans@logigear.com @hansbuwalda

www.happytester.com

© 2016 LogiGear

Domain Language Approaches: Keywords

4 actions, each with an action keyword and arguments

read from top to bottom

fragment from a test with actions

acc nr first last

open account 123123 John Doe

acc nr amount

deposit 123123 10.11deposit 123123 20.22

acc nr expected

check balance 123123 30.33

•  The test developer creates tests using "actions" with keywords and arguments

•  Checks are, as much as possible, explicit (specified expected values) •  The automation task focuses on automating the keywords, each keyword

is automated only once •  This technique can be very scalable. A similar approach is behavior

based testing, which also works with human readable tests, but is more verbose

5/13/16

2

© 2016 LogiGear

High Level Test Design - Test Development Plan

Objectives

Test Module 1

Test Cases

Test Module 2 Test Module N

Actions

. . .

AUTOMATION

Objectives Objectives

interaction test business test

Overview Action Based Testing define the "chapters"

create the "chapters"

create the "words"

make the words work

Test Cases Test Cases

window control valueenter log in user name jdoeenter log in password car guy

window control property expectedcheck property log in ok button enabled true

user passwordlog in jdoe car guy

first last brand modelenter rental Mary Renter Ford Escape

last totalcheck bill Renter 140.42

© 2016 LogiGear

§  Business objects and business flows -  objects are like cars, invoices, locations, etc -  flows are like create, fulfill, pay and close an order

§  Other tests -  functions and features, like premium calculation or PDF output -  administration, users, security, authorizations -  graphics -  technologies, protocols, ... -  customization, extensibility -  interoperability -  . . .

§  Business versus interaction -  differentiate within business objects and all other categories -  interaction can be further differentiated into: values, UI, keyboard, etc -  also, for every category, look at negative tests, flows, aggressive

tests, etc

Examples of considerations

5/13/16

3

© 2016 LogiGear

Example Top Level Structure Project create, retrieve, update, delete / deactivate ("CRUD")

copy, move categorize, enumerate, identify convert, serialize, export/import, ...

UI, dialogs, forms, pages input (validation, defaulting, dependencies) flows (primary paths, alternate paths) keyboard shortcuts, keyboard controls, ...

. . .

<business object 1>

Lifecycles, data operations

Interaction

Functions and Features

Technologies, protocols, controls

Data (handling, quality, ETL, ...)

Security, authorization, admin

Graphics, multi-media, charts, ...

Load, performance

Interoperability

Customizing, extensibility

Business Flows

Concurrency, race conditions, ...

Business Objects

processes, transactions, end-to-end, day in the life, combinations of flows, ...

calculations, analyses, PDF output, ...

© 2016 LogiGear

Eye on the ball, Scope

§  Always know the scope of a test module

§  The scope should be unambiguous

§  The scope determines many things: - what the test objectives are - which test cases to expect - what level of actions to use - what the checks are about and which events should

generate a warning or error (if a “lower” functionality is wrong)

5/13/16

4

© 2016 LogiGear

Example of a Test

Step Description Expected

     

step  16 Open  http://www.bigstore.com The  "BIG  Store"  main  page  is  displayed,  with  a  "sign  in"  link

step  17 Click  on  "Sign  In",  upper  right  corner A  sign  in  dialog  shows,  the  "Sign  in"  button  is  disabled

step  18 Enter  "johnd"  in  the  user  name  field The  "Sign  In"  button  is  still  disabled

step  19 Enter  "bigtester"  in  the  password  field Now  the  "Sign  In"  button  is  enabled

step  20 Click  on  the  "Sign  in"  button The  page  now  shows  "Hello  John"  in  the  upper  right  corner

step  21 Enter  "acme  watch"  in  the  search  field The  "Search"  button  is  enabled

step  22 Click  on  the  "Search"  button 5  watches  of  Acme  Corporation  are  displayed

step  23 Double  click  on  "Acme  Super  Watch  2" The  details  page  of  the  Acme  Super  Watch  2  is  displayed

step  24 Verify  the  picture  of  the  watch The  picture  should  show  a  black  Acme  Super  Watch  2

step  25 Select  "red"  in  the  "Color"  dropdown  list The  picture  now  shows  a  black  Acme  Super  Watch  2

step  26 Type  2  in  the  "Order  quantity"  textbox The  price  in  the  right  shows  "$79.00    +  Free  Shipping"

step  27 Click  on  "Add  to  cart" The  status  panel  shows  "Acme  Super  Watch  2"  added

step  28 Click  on  "Check  out" The  Cart  Check  Out  open,  with  the  2  Acme  Super  Watches

     

© 2016 LogiGear

GWT scenario

Scenario: Applicant is at Stop Test alert and chooses to Stop Test Given User turns off Password required option for Drive Test And User has logged in by Traffic Applicant account And User is at the Assessments Take a Test page And User clicks the Traffic Test link And User clicks the Next button And User clicks the Sheet radio button in Mode page if displayed And User clicks the Start button And User waits for test start And User clicks the Stop Test button When User clicks the Confirm Stop Test button

And User enters the correct applicant password And User clicks the Confirm Stop Test button

Then The Test is Over should be displayed in the Message label Then Value of the Message label should be The test is over. And The Welcome to Traffic Testing page should be displayed

5/13/16

5

© 2016 LogiGear

What is an "anti-pattern"

§  A practice considered harmful -  opposite of a "pattern"

§  Has an easy to remember name

§  Can be helpful to spot or describe risks

§  In the case of automated tests, the risks you usually try to avoid are in areas like: -  effectiveness -  efficiency -  readability -  manageability -  maintainability -  etc

© 2016 LogiGear

Using the anti-patterns in this presentation

§  Experimental, input welcome

§  There are overlaps -  in not-so-great test designs multiple patterns tend to apply

simultaneously

§  Input for design decisions, not absolute truths -  "decisions have consequences", but those can be acceptable in

sometimes -  not everything applies to all situations, not every anti-pattern

occurrence is a bad thing -  see them more as a starting point of a discussion

§  See anti-patterns as symptoms -  lack of an overall high level test design -  lack of experience of the test developer

5/13/16

6

© 2016 LogiGear

"Anti-patterns" (informal)

§  Interaction Heavy – Not having many business tests §  Lame – No depth or variety, no testing techniques used §  Enter Enter Click Click – Test steps are too detailed §  No Life – Missing life cycle steps of business objects §  Clueless – No clear scope for the tests §  Cocktail – Interaction tests mixed with business tests §  Over-Checking – Checks not relevant for the scope §  Sneaky Checking – Checks hidden in actions §  Action Explosion – Many actions, hardly different §  Mystery Actions – Cryptic actions, unclear what they do §  Techno – Actions and tests that look like code §  Passive Timing – Hard-coded "sleeps" for timing issues

© 2016 LogiGear

Interaction Heavy

§  Not having many business tests

§  What to look for: -  seeing only test cases like these:

•  can I successfully create a new customer •  can I successfully update an order •  ...

-  seeing the test tree follow the UI organization

§  The UI is the most visible part of an application. It is not uncommon to have a "what you see is what you test" tendency

§  What is missing is a clear understanding and verification if the application is fit for business

5/13/16

7

© 2016 LogiGear

Lame

§  No depth or variety, no testing techniques used

§  What to look for: -  boredom -  no suprises -  no techniques used -  not many, and trivial, bugs

§  Consider exploratory approaches, not just for manual testing via the UI, but also for test case design -  in other words: make it more fun -  involve domain experts

© 2016 LogiGear

Enter Enter Click Click

§  Test steps are too detailed §  What to look for:

-  long sequences with only built-in low level actions like "enter", "click", "select menu", etc

-  high sensitivity to UI changes -  not able to understand the goal of the tests easily

§  Example: -  an application in the oil industry -  over 6000 test cases with a length of 2000 test lines or

more

§  Rather than trying to optimize, create a high-level test design first ("from scratch")

5/13/16

8

© 2016 LogiGear

No Life

§  Missing life cycle steps of business objects -  like CRUD: Create-Read-Update-Delete

§  What to look for: -  is it clear what the business objects are? -  do they show in the hierarchy -  in particular are there modifications ("update") and

removal/closure ("delete") tests? -  is there variation in such tests

§  Life-cycle tests are usually not difficult for define

§  I usually prefer to design life-cycle tests as business tests, not as interaction tests

© 2016 LogiGear

Clueless

§  No clear scope for the tests §  What to look for:

-  look through the test lines. Assuming you know the domain, is it clear to you what the purpose of the test is?

-  is the scope mixed. For example a customer is created, and tested, and a order is placed by that customer in the same test

§  Having clear scopes for tests (in particular test modules) helps organize and maintain them

§  It also helps follow up -  for example: "Credit Card Payments" has fails

5/13/16

9

© 2016 LogiGear

Cocktail

§  Interaction tests mixed with business tests

§  This is quite common

§  What to look for -  a test is testing business rules and processes, like

calculation of a home loan premiums in various situations -  and at the same time is doing UI checks

§  This anti-pattern is usually a symptom of other anti-patterns (like "clueless")

© 2016 LogiGear

Over-Checking

§  Checks not relevant for the scope

§  Testers, and their management, often believe everything a test script goes through needs to be tested

§  Automation engineers want to make sure the navigation is still ok (consider using an "assert" action instead)

§  The result can be less clarity and harder maintenance

§  In a clear test design over-checking can quite easily be avoided

5/13/16

10

© 2016 LogiGear

Example of a Test Module

TEST MODULE Order processing

start system

TEST CASE TC 01 Order for tablets

user passwordlogin jdoe doedoe

windowcheck window exists welcome

order id cust id article price quantitycreate order AB123 W3454X tablet 198.95 5

order id totalcheck order total AB123 994.75 . . .

© 2016 LogiGear

Sneaky Checking

§  Checks hidden in actions

§  What to look for: -  in action definitions or scripts there are checks that are

not visible in the test module -  it are often interaction checks, like "does this window

exist", after a click

§  Such checks may influence your statistics, and increase maintenance sensitivity

§  If it is "just to make sure", use an "assert" action: -  producing a warning or an error if the condition isn't met -  in most cases this is not needed, the tool will complain

5/13/16

11

© 2016 LogiGear

Action Explosion

§  Many actions, hardly different §  What to look for:

-  every step and variation of a step has its own action -  many actions are used only once or twice -  there are about as many actions as tests

§  It is the opposite of the "enter enter click click" -  the challenge is to find a balance between not-enough-

actions and too-many-actions

§  Example: open financials open financials - payments tab open financials - payments tab - approval sub tab open financials - payments tab - approval sub tab - approved changes panel

© 2016 LogiGear

Consider using "mid-level" actions

§  "Low level": detailed interaction with the UI (or API) -  generic, do not show any functional or business logic -  examples: "click", "expand tree node", "select menu"

§  "High level": a business domain operation or check on the application under test -  hide the interaction -  examples: "enter customer", "rent car", "check balance"

§  "Mid level": common sequences at a more detailed application level -  usually to wrap a form or dialog -  for use in high level actions -  greatly enhance maintainability -  example: "enter address fields"

enter customer

enter address fields

enter select set . . . . . .

5/13/16

12

© 2016 LogiGear

Mystery Actions

§  Cryptic actions, unclear what they do

§  What to look for: -  action or arguments that are not clear to those who know

the domain -  generic actions like "check invoice" -  are not business related or UI related

§  Unclear actions: -  might be used a wrongly -  might not be found, delimiting their re-use -  hinder impact analysis and follow up

© 2016 LogiGear

Techno

§  Actions and tests that look like code -  "preferably" C++ code -  work well to impress people, but not for much else

§  What to look for: -  not English or another natural language -  all kinds of special characters -  lack of spacing -  in general names that are _NOts0EasY_2REad

§  Often a result of developers at the helm of the test design

5/13/16

13

© 2016 LogiGear

§  Hard-coded "sleeps" for timing issues

§  What to look for -  "sleep" or "wait" actions -  sometimes hidden in other actions -  arguments are sometimes variables, this is not much of an improvement

§  Passive timing -  big nono, the number one archenemy of automation J -  at best slows down your tests -  but if not long enough make the test unstable -  in VM's the problem is even worse than in physical machines -  if used, the team should clearly escalate it, define it as a risk

§  Better approach: Active timing -  wait for a measurable event -  usually the wait is up to a, generous, maximum time -  for UI many such waits are built into the automation tool -  consider involving developers to help you (see my slides of my "BIG Testing" tutorial for

more suggestions)

Passive Timing

© 2016 LogiGear

Be careful in communication!

§  Avoid even the hint of attack

§  Make sure all people involved understand, and appreciate, that anti-patterns are applied

§  In particular watch your writing, like emails and reports -  good fuel for "flaming" -  avoid writing something like: these tests are "lame" -  talk about test design -  teamwork rules suppreme

5/13/16

14

© 2016 LogiGear

Summary

§  Automation success depends on design decisions, most of which are test design decisions

§  Anti-patterns can help identify risks, and start discussions

§  Anti-patterns are not (meant to be) absolute rules

§  Test design and automation are team work, part of a larger route to success