Make automated testing agile and scalablefiles.meetup.com/18247475/Omaha QA Event 2015 02 19 Hans...

Make automated testing

agile and scalable

Hans Buwalda

Omaha Software Testing and QA user group

February 21st 2015

©2015 LogiGear Corporation

WELCOME

• Introduction/Welcome

• Cadence

• Announcements– Speakers

– Hosts

– Leaders

• Logistics

• Sponsors– Deliveron

• Feature Speaker Introduction

SPONSOR - DELIVERONDeliveron is a consulting organization that delivers business-driven technology

solutions. Our high success rate is the result of our:

• Project leadership and guidance

• Agile delivery framework and ALM best practices

• Team approach with client engagements

• Integrated client partnership and collaboration

• Deep expertise in core Microsoft technologies

• Strategic offerings

Our solutions span several areas:

• Custom Application Development

• Microsoft Cloud & Azure

• Agile, ALM, & DevOps

• Collaboration & Analytics

• Application Integration


Who is your speaker

• Partner of Deliveron

• Software testing company, around since 1994

• Testing and test automation services:– consultancy, training

– test development and automation services

– focus on big and complex testing projects

• Product line: TestArchitect™, TestArchitect for Visual Studio™

– integrating test development with test management and automation

– based on modularized keyword-driven testing

www.logigear.comwww.testarchitect.comwww.deliveron.com

• Dutch guy, in California since 2001

• Background in math, computer science, management

• Since 1994 focusing on automated testing– keyword testing, agile testing, big testing

Hans Buwalda

LogiGear Corporation

hans @ logigear.comwww.happytester.com


• No more than 5% of all test cases should be executed

manually

• No more than 5% of all efforts around testing should

involve automating the tests

The 5% Rules of Test Automation


Business Approach to Testing

• Business goals for testing and automation– Time-To-Market

– Quality-To-Market

– Control

• Agree on a method– integrated method for testing and automation

– testers in the driver seat

– maximum flexibility, best technology

– minimal technical efforts

– maximum maintainability

• Then select the tooling to support it– providing a framework for the testing and the automation

– encouraging testers to follow the method

– not standing in the way of the engineers


Don't Automate Manual Testing


Don't just automate manual testing


Don't just automate manual testing

Good automated testing is not the same as automating good manual testing. . .

Record and Playback

select window "Logon"

enter text "username", "administrator"

enter text "password", "testonly"

push button "Ok"

select window "Main"

push button "New Customer"

expect window "Customer Information"

select field "First Name"

type "Paul"

select field "Last Name"

type "Jones"

select field "Address"

type "54321 Space Drive"

.

.

.

Scripting

Test CaseDesign

Test CaseAutomation

Test CaseExecution

TEST DESIGNER

AUTOMATION ENGINEER

PLAYBACK


Example scripting

/// <summary>/// AddItems - Use 'AddItemsParams' to pass parameters into this method./// </summary>public void AddItems(){

#region Variable DeclarationsWinControl uICalculatorDialog =

this.UICalculatorWindow.UICalculatorDialog;WinEdit uIItemEdit =

this.UICalculatorWindow.UIItemWindow.UIItemEdit;#endregion

Keyboard.SendKeys(uICalculatorDialog,this.AddItemsParams.UICalculatorDialogSendKeys,ModifierKeys.None);

Keyboard.SendKeys(uIItemEdit,this.AddItemsParams.UIItemEditSendKeys,ModifierKeys.None);

}

• State of the art, but stuff for coders . . .


Action Based Testing

• Modularized keyword-driven method to describe

behaviors as a series of keyword-based actions

• Particularly suited for large and complex projects

• High degree of agility, even when used in traditional

projects

• High level of automation, 95% or more

• Puts the focus on test design, less on technology


Actions

4 actions, each with an action keyword and arguments

read from top to bottom

fragment from a test with actions

acc nr first last

open account 123123 John Doe

acc nr amount

deposit 123123 10.11

deposit 123123 20.22

acc nr expected

check balance 123123 30.33

• The test developer creates tests using actions with keywords and arguments

• Checks are, as much as possible, explicit (specified expected values)

• The automation task focuses on automating the keywords, each keyword is automated only once

• note: instead of keywords one can also use "gherkin" scenarios (BDD), in particular if the tests are smaller

Test Module Plan

Tests

Objectives

Test Module 1

Actions

. . .

AUTOMATION

user password

log in jdoe StarEast

first last brand model

rent car John Renter Ford Escape

rent car John Renter Chevrolet Volt

last total

check bill Renter 140.42

interaction test business test

window control value

enter log in user name jdoe

enter log in password StarEast

window control property expected

check prop log in ok button enabled true

car guycar guy

Action Based Testing

Tests

Objectives

Test Module 2

Tests

Objectives

Test Module n


Why Better Test Design?

• Quality and manageability of test – many tests are often quite "mechanical" now

– one to one related to specifications, user stories or requirements, which often is ok, but lacks aggression

– no combinations, no unexpected situations, lame and boring

– such tests have a hard time finding (interesting) bugs

• Better automation– when unneeded details are left out of tests, they don't have to be

maintained

– avoiding "over checking": creating checks that are not in the scope of a test, but may fail after system changes

– limit the impact of system changes on tests, making such impact more manageable

I have become to believe that successful automation is usually less of a technical challenge as it is a test design challenge.

The 3 "Holy Grails" of Test Design

• Effective breakdown of tests into test modules– each with a focused and differentiated scope

• Right approach for each test module– which test design technique

– who to involve, etc

• Proper level of detail in the test lines– what actions

– what checks


What's the trick...


What's the trick...

• Have or acquire facilities to store and organize you

content

• Edit your stuff

• Decide where to put what

– assign and label the shelves

• Put it there

• If the organization is not sufficient anymore, add to it or

change it


Breakdown Criteria

• Straightforward Criteria– Functionality (customers, finances, management information, UI, ...)

– Architecture of the system under test (client, server, protocol, sub systems, components, modules, ...)

– Kind of test (navigation flow, negative tests, response time, ...)

• Additional Criteria– Stakeholders (like "Accounting", "Compliance", "HR", ...)

– Complexity of the test (put complex tests in separate modules)

– Technical aspects of execution (picture checks, special hardware, multi-station, ...)

– Overall project planning (availability of information, timelines, sprints, ...)

– Risks involved (extra test modules for high risk areas)

– Ambition level (smoke test, regression, aggressive, …)


• Business objects and business flows– objects are like cars, invoices, locations, etc

– flows are like place, fulfill, pay and close an order

• Other tests– functions and features, like premium calculation or PDF output

– administration, users, security, authorizations

– graphics

– technologies, protocols, ...

– customization, extensibility

– . . .

• Business versus interaction– differentiate within business objects and other categories

– interaction can be further differentiated into: values, UI, keyboard,

etc

Example of considerations


Example Top Level Structure

Project create, update, delete/closecopy, movecategorize, enumerate, identifyconvert, serialize, export/import, ...

UI, dialogs, forms, pagesinput (validation, defaulting, dependencies)flows (primary paths, alternate paths)keyboard shortcuts, keyboard controls, ...

. . .

<business object 1>

Lifecycles, data operations

Interaction

Functions and Features

Technologies, protocols, controls

Data (handling, quality, ETL, ...)

Security, authorization, admin

Graphics, multi-media, charts, ...

Load, performance

Integration

Customizing, extensibility

Business Flows

Concurrency, race conditions, ...

Business Objects

processes, transactions, end-to-end, day in the life,combinations of flows, ...

calculations, analyses, PDF output, ...


Example of business level test module

• Consists of an (1) initial part, (2) test cases and (3) a final part

• Focus is on business functionality, with a clear business scope

• Navigation details are avoided

TEST MODULE Car Rental Payments

user

start system john

TEST CASE TC 01 Rent some cars

first name last name car

rent car John Doe Ford Escape

rent car John Doe Chevvy Volt

last name amount

check payment Doe 140.4

FINAL

close application


Example of an interaction level test

module

• Interaction details that are the target of the test are not hidden

• Focus is not on business ("is the payment amount correct"), but on

interaction ("can I retrieve the payment amount")

TEST MODULE Screen Flow

user

start system john

TEST CASE TC 01 Order button

window button

click main create order

window

check window exists new order

FINAL

close application


Example Script for an Action

# get table object, column number and column count

windowName = LIBRARY.NamedArgument("window")

tableName = LIBRARY.NamedArgument("table")

columnName = LIBRARY.NamedArgument("column")

table = ABT.OpenElement(windowName, tableName)

column = table.GetColumnIndex(columnName)

rowCount = table.GetRowCount()

# check the sort order, row by row

previous = table.GetCellText(0, column)

for i in range(1, rowCount):

current = table.GetCellText(i, column)

if current < previous :

LIBRARY.AdministerCheck("order", "sorted", "fails " + str(i+1), 0)

return

previous = current

LIBRARY.AdministerCheck("order", "sorted", "all rows in order", 1)

The following action script will verify whether the rows in a table are sorted:

find the table in the UI

if a value is smaller than before, fail the test

if all rows are ascending, pass the test

get arguments from the test line

def action_checkSortOrder():


Using the new action

• By keeping an action generic it can be applied for a

variety of situations

• Some examples of using "check sort order":

window table column

check sort order view orders orders table ID

window table column

check sort order annual results regions revenue

window table column

check sort order inventory cars price

window table column

check sort order registration students last name

Example application

A Test Module for the application

We click a tree node, and then do a check

The actions here are built-in in the framework

Making a new "action"

This action definition uses existing actions to create a new action called "check bitrate"

Argument names can be used in cell expressions, that start with "#", and support the usual string and numeric operators

create a node path from the first two argumentsthe expected value is given

by the 3rd argument

the arguments of the new action

name of the new action

Using the action in a test

These test lines don't care about the navigation in the UI of the application, the focus is functional: verify data

Such functional tests are easier to read with high level actions, and the reduced dependency on navigation makes them (much) easier to maintain in the long term


Low-level, high-level, mid-level actions

• Low-level: detailed interaction with the UI (or API)– generic, do not show any functional or business logic

– examples: "click", "expand tree node", "select menu"

• High-level: represent a business function specific to the scope of the test– hide the interaction

– examples: "enter customer", "rent car", "check balance"

• Mid-level: auxiliary actions that represent common sequences of low level actions– usually to wrap a form or dialog

– greatly enhance maintainability

– example: "enter address fields"

enter customer

enter address fields

enter select set . . .. . .

Identifying controls

Identify windows and controls, and assign names to them

These names encapsulate the properties that the tool can use to identify the windows and controls when executing the tests


Mapping an interface

• An interface mapping (common in test tools) will map windows and controls to names

• When the interface of an application changes, you only have to update this in one place

• The interface mapping is a key step in your automation success, allocate time to design it well, in particular naming and choosing identifying properties

INTERFACE ENTITY library

interface entity setting title {.*Music Library}

name class label

interface element title text Title:

interface element artist text Artist:

interface element file size text File size (Kb):

name class position

interface element playing time text textbox 4

interface element file type text textbox 5

interface element bitrate text textbox 6

name class position

interface element music treeview treeview 1


Hidden Identifying Property


Mapping the interface

• An interface mapping will map windows and controls to names

• When the interface of an application changes, you only have to update this in one place

• The interface mapping is a key step in your automation success, allocate time to design it well


Stable Automation

• Test your automation– develop separate tests for the actions themselves

• Let developers supply values for hidden identifying properties of UI element, to ease automated identification

– like “accessible name” in .Net and Java, or “id” in Web controls

– a lot of time is spent on identifying and re-identifying UI elements

• Use "active" timing– wait based on conditions in the system under test, avoid fixed “sleeps”

– if needed request developers to provide hooks

• Use automation to identify differences between versions of the system under test

– test in a loop if all control can still be found

• Keep an eye on the test design– unsuccessful automation virtually always can be traced back to poor test design


• Identifying properties

• Hooks for timing

• White-box access to anything relevant:– input data (ability to emulate)

– output data (what is underlying data being displayed)

– random generators (can I set a seed?)

– states (like in a game)

– objects displayed (like monsters in a game)

• Testability should be a "must have" requirement– first question in a development project: "how do we test this"?

Testability, some key items


Variables and expressions with keywords

• This test does not need an absolute number for the available cars,

just wants to see if a stock is updated

• As a convention we denote an assignment with ">>"

• The "#" indicates an expression

TEST CASE TC 02 Rent some more cars

car available

get quantity Chevvy Volt >> volts




car expected

check quantity Chevvy Volt # volts - 2


Data driven testing with keywords

• The test lines will be repeated for each row in the data set

• The values represented by "car", "first" and "last" come from the

selected row of the data set

DATA SET cars

car first last value

Chevvy Volt John Doe 40000

Ford Escape Mary Kane 22500

Chrysler 300 Jane Collins 29000

Buick Verano Tom Anderson 23000

BMW 750 Henry Smyth 87000

Toyota Corolla Vivian Major 16000

TEST CASE TC 03 Check stocks

data set

use data set /cars

car available

get quantity # car >> quantity


rent car # first # last # car

car expected

check quantity # car # quantity - 1

repeat for data set


"Lead Deputy" Testing

• For "multi station" testing, when multiple machines have to

participate in a test in real-time

• For example if a supervisor needs to approve a withdrawal in a bank

teller system

• Can be "sync" and "parallel" (with a rendezvous point)

acc nr amount tx id

withdraw 123123 10,000 >> tx

name

use deputy supervisor

tx id amount

approve # tx 10,000

use lead


Non-UI Testing

• Examples– application programming interfaces (API’s)

– embedded software

– protocols

– files, batches

– databases

– command line interfaces (CLI’s)

– multi-media

– mobile devices

• Impact is mainly on the automation– test design should in most cases be transparent towards the

specific interfaces

• Often non-UI automation can speed up functional tests

that do not address the UI

testing devices

Multiple System Access

SystemUnder Test

Action Automation

APIaccess

protocolaccess

UIaccess

databaseaccess

Test Modules

Device Testing

Software Under Test

AgentABTAutomation

InterfaceInfo

Testing HostDevice

Android


Life Cycles

• Product life cycles, rather than task life cycles

• The project planning and execution largely determines when the

products are created

systemdevelopment

testdevelopment

testautomation


Typical Time Allocation

TEST DEVELOPMENT

AUTOMATION

time

effo

rts


ABT in Agile

Test ModuleDefinition(optional)

Test Module Development

Interface Definition

Action Automation

Test Execution

Sprint ProductsProduct Backlog

Test re-use

Automation re-use

product owner

teamprod owner

& team

User storiesDocumentation

Domain understanding

Acceptance CriteriaPO Questions

SituationsRelations

Agile life cycle

Test development

Main Level Test Modules

Interaction Test Modules

Cross over Test Modules


Using ABT in Sprints (1)

• Aim for "sprint + zero", meaning: try to get test

development and automation "done" in the same sprint,

not the next one

– next one means work clutters up, part of team is not working on the

same sprint, work is done double (manually and automated), ...

• Agree on the approach:

– questions like does "done" include tests developed and automated?

– do we see testing and automation as distinguishable tasks and

skillsets?

– is testability a requirement for the software?



• Just like for development, use discussions with the team and

product owners

– deepen understanding, for the whole team

– help identify items like negative, alternate and unexpected situations

• Start with the main test modules, that address the user stories

and acceptance criteria

– try to keep the main test modules at a similar level as those stories and

criteria

– test modules can double as modeling device for the sprint

• Plan for additional test modules:

– low-level testing of the interaction with the system under test (like UI's)

– crossing over to other parts of the system under test



• Create good starting conditions for a sprint:

– automation technology available (like hooks, calling functions, etc)

– how to deal with data and environments

– understanding of subject matter, testing, automation, etc

• Do interface mapping by hand, using developer provided

identifications

– saves time by not having to use the viewer or other spy tools

– recording of actions (not tests) will go better

• Make testing and automation part of the evaluations and

hardening sprints


Outsourcing and Agile

• Model 1: Full team outsourcing– development, testing and automation

– automated tests can be positioned as part of the delivery

• Model 2: Integrated team:– needs online tool like Jira or Rally

– you must have shared meetings

– advantage: more project time

• Model 3: "2nd unit"– off-shore team works under control of one or more sprint team members

• Model 4: Test Production and management– off-shore team takes the deliveries of the primary team, creates/automates more tests,

and executes and maintains them


Summary

• Keywords is one of the techniques for automated testing, in addition

to record & playback and scripting

• In itself keywords are not a silver bullet. It needs a good approach,

careful planning and good organization to be successful, maintainable

and scalable

Make automated testing agile and scalablefiles.meetup.com/18247475/Omaha QA Event 2015 02 19 Hans...

Documents

Transcript of Make automated testing agile and scalablefiles.meetup.com/18247475/Omaha QA Event 2015 02 19 Hans...