Pitfalls and tips with Pitfalls and tips with browser based browser based

testIngtestIng

My Background

several projects with functional testing

consulted at organizations to introduce functional testing and builds

heavily involved in trying to improve effectiveness and reduce maintenance

developed a framework over several projects that made use of my learnings

Common uses of functional tests

smoke/sanity tests for most valuable and risky features

black box/regression tests

post deployment checks and rollbacks

multi browser testing

what do we want from tests?

correctness (when my tests work, my application is not broken)

robustness (when my tests break, my application is broken)

what do we focus on?

what’s valuable? (it matters to us that people can do this)

what’s risky? (if this breaks, it really affects us)

so whats the so whats the problem?problem?

five problemsfive problems

1. slow tests1. slow tests

BecauseBecause

uses the uses the entire stackentire stack

low spec hardwarelow spec hardware

tons of setuptons of setupand teardownand teardown

talking to the talking to the browser is slowbrowser is slow

soso

very long feedback very long feedback looploop

can delay can delay deploymentsdeployments

2. Brittle tests2. Brittle tests

UI UI changes changes

a lota lot

tests over tests over specifyspecify

3. Random failures3. Random failures

race conditionsrace conditions

unreliable unreliable dependenciesdependencies

4. tests are hard to 4. tests are hard to debugdebug

too many too many tests breaktests break

error messageserror messagestell you nothingtell you nothing

5. test code becomes 5. test code becomes unmaintainableunmaintainable

insufficient insufficient commitment to commitment to

testingtesting

inadequate inadequate understanding understanding

of testingof testing

what happens

there is an initial stream of tests

tests break unnecessarily

tests become hard to maintain

team loses confidence and motivation

some teams even abandon testing via the UI

tips and trickstips and tricks

Treat test codelike real code

speed

maintainability

A lot of the techniques we A lot of the techniques we use with real code work use with real code work

with test codewith test code

similar levels of similar levels of commitmentcommitment

test test creation/maintenancecreation/maintenancecan be part of the dev can be part of the dev

cyclecycle

Use Ruby

maintainability

development time

use a headless browser when possible

speed

Use selenium RC (if you use selenium)

avoid brittleness

maintainability

Don’t record and replay

(if you use selenium)

avoid brittleness

maintainability

development time

Introduce an Abstraction layer

speed

maintainability

Categorize/tag tests

speed

maintainability

Use the page object pattern

maintainability

Parallelize

speed

separate data for tests

speed

avoid random failures

clump data into common location

maintainability

Push setup and teardown out of tests

speed

avoid brittleness

development time

debuggability

Only exercise parts of the application you

want to testspeed

avoid brittleness

debuggability

Build stub services

speed

avoid random failures

don’t over specify

avoid brittleness

debuggability

run on better hardware

speed

slice vertically

debuggability

Use the right locator

maintainability

avoid brittleness

speed

learn xpath/css selectors

avoid brittleness

maintainability

fail later

speed

use polls over sleep

speed

avoid random failures

use existing waits of selenium

speed

avoid random failures

avoid brittleness

avoid random

Don’t use assertions to identify transitions

avoid brittleness

debuggability

avoid brittleness

avoid random

screenshots and html dumps

debuggability

xpath/css selector tester

development time

firebug or w/e

development time

maintainability

talk to me if you’re talk to me if you’re testing tabular datatesting tabular data

I’m always available I’m always available for questionsfor questions

Questions?Questions?

thank youthank you