Improve Your Automated Web Application Testing

Scott McMaster

Google

Kirkland, Washington USA

About Me• Software Design Engineer @ Google.

– Building high-traffic web frontends and services in Java.

– AdWords, Google Code

• Ph.D. in Computer Science, U. of Maryland.

• Formerly of Amazon.com (2 years), Lockheed Martin (2 years), Microsoft

(7 years), and some small startups.

• Frequent adjunct professor @ Seattle University, teaching software

design, architecture, and OO programming.

• Author of technical blog at http://www.scottmcmaster365.com.

Testing and Me

• Doing automated testing since 1995.

• Ph.D. work in test coverage and test suite maintenance.

• Champion of numerous unit, system, and performance

testing tools and techniques.

• Co-founder of WebTestingExplorer open-source automated

web testing framework (www.webtestingexplorer.org).

Agenda

• Web Testing Challenges

• The Present: Selenium/WebDriver Automation.

– Best Practices

– Things to Avoid

• The Future? Automated Exploratory Testing with

WebTestingExplorer

A Few Words About Testing @ Google

• At Google, test engineers are developers. They develop:

– Test code.

– Test frameworks and productivity tools that are used by software design engineers.

• Test code quality == development code quality.

– Test code must be maintainable, robust, and meet other general software engineering

guidelines.

• For more on “How Google Tests Software”, see the series of blog posts by James

Whittaker (starting at http://googletesting.blogspot.com/2011/01/how-google-

tests-software.html).

Assumptions in This Talk

• You have at least some background in:

– Web application development and/or testing.

– Basic Selenium and/or WebDriver usage.

• And you would like to improve your testing

through the user interface!

How Would You Test This?

Some Things on That Page• Dynamic content and hundreds of HTML elements.

• User content (reviews, likes).

• Multiple dropdown menus.

• Search.

• Security (login).

• State (shopping cart).

• Embedded video.

It’s Even Worse

• After we write the tests, customers /

designers will want changes.

– Our tests will probably break.

– Managers don’t want to wait for testing to fix

their broken tests before releasing the software.

!!!

Calm Down. What Can We Do?

• Write more robust, more maintainable tests!

Let’s Use a Slightly Simpler Example

Use Cases

• Enter the name of a country and load cities.

– Error message if not found.

• Select a city and load the current weather.

• The weather appears in a table.

• Signed-in user gets weather for their “favorite”

city.

Architecture

Rich Browser UI (GWT)

Frontend Server (GWT RPC servlet)

Remote Web Service (XML-over-

HTTP)User Database

What Testing Should We Automate?

• Unit?

– Yes!

• System?

– Yes!

• End-to-end?End-to-end?End-to-end?End-to-end?

– Yes! This will be the focus of this talk.

Web UI System Test Automation

• What tools do we use?

– Selenium WebDriver

– http://seleniumhq.org/projects/webdriver/

Selenium WebDriver“Driving a browser natively as a user would either locally or on a remote machine…”

• Selenium IDE• Browser plugin for play/record.• Mostly for non-programmers – don’t use this.

• Selenium RC / Selenium 1• Deprecated – do not start using this!

• Selenium 2 (WebDriver)• More concise API.• Uses browser’s native automation support, works in all

major browsers (but Firefox tends to work “best”).• Specially designed for rich (AJAX) web UIs.

• (For example, performs style calculations to help you only work with displayed elements.)

Selenium WebDriver Language Bindings

• Java

• Python

• C#

• Ruby

• PHP

• Perl

• Javascript

My examples will use Java.

“Simple” WebDriver Test: Spec• Load the homepage.

• Find the “country” text box.

• Type in the name of a country that does not exist.

• Find the “load cities” button.

• Click it.

• Wait (important!)

• Verify that the “cities” UI is not displayed.

• Verify that the “country not found” error is displayed.

“Simple” WebDriver Test: Code@Testpublic void testUnknownCountryName() { FirefoxProfile firefoxProfile = new FirefoxProfile(); WebDriver driver = new FirefoxDriver(firefoxProfile);

WebElement countryElement = driver.findElement(By.id(GWT_DEBUG + DebugIds.COUNTRY_NAME_ID)); countryElement.sendKeys("Does not exist"); WebElement loadCitiesButton = driver.findElement(By.id(GWT_DEBUG + DebugIds.LOAD_CITIES_ID)); loadCitiesButton.click();

WebDriverWait wait = new WebDriverWait(driver, 5000); wait.until(ExpectedConditions.visibilityOfElementLocated(By.id(GWT_DEBUG + DebugIds.NO_CITIES_ID)));

WebElement noCitiesElement = driver.findElement(By.id(GWT_DEBUG + DebugIds.NO_CITIES_ID)); assertTrue(noCitiesElement.isDisplayed());

WebElement cityRowElement = driver.findElement(By.id(GWT_DEBUG + DebugIds.CITY_ROW_ID)); assertFalse(cityRowElement.isDisplayed());}

Bad News / Good News

• This code is verbose and hard to maintain.

• But we can make it better!

Tip#1: Organize Your Locators

• Shortens the code and gives you one easy-to-find

place to change when the UI changes.private static interface WeatherHomeLocators {private static interface WeatherHomeLocators {private static interface WeatherHomeLocators {private static interface WeatherHomeLocators { By countryName = By.id(GWT_DEBUG + DebugIds.COUNTRY_NAME_ID); By loadCities = By.id(GWT_DEBUG + DebugIds.LOAD_CITIES_ID); By noCities = By.id(GWT_DEBUG + DebugIds.NO_CITIES_ID); By cityRow = By.id(GWT_DEBUG + DebugIds.CITY_ROW_ID); By city = By.id(GWT_DEBUG + DebugIds.CITY_ID); By loadWeather = By.id(GWT_DEBUG + DebugIds.LOAD_WEATHER_ID); By weatherCity = By.id(GWT_DEBUG + DebugIds.WEATHER_CITY); By weatherTemp = By.id(GWT_DEBUG + DebugIds.WEATHER_TEMP); By userName = By.id(GWT_DEBUG + DebugIds.USER_NAME); By login = By.id(GWT_DEBUG + DebugIds.LOGIN); By weather = By.id(GWT_DEBUG + DebugIds.WEATHER); By unknownUser = By.id(GWT_DEBUG + DebugIds.UNKNOWN_USER);}

…

WebElement countryElement = driver.findElement(WeatherHomeLocators.countryName);

Bonus Tip #1a: How to Locate

• In priority order, locate elements by:

1. id (ask developers to add these, make them stable, and

available in all builds)

2. name

3. css class

4. link text (but watch for dynamic content and localization)

5. Xpath expression

Tip #2: Use a Base Class for Tests

• Some things you can do in a base class:

1. Create the browser profile and WebDriver.

2. Safely close the WebDriver.

3. Improve on the WebDriver API.

Base Class Examplepublic class AbstractWebDriverTest {public class AbstractWebDriverTest {public class AbstractWebDriverTest {public class AbstractWebDriverTest {

protected static final String protected static final String protected static final String protected static final String GWT_DEBUG = "gwt-debug-";GWT_DEBUG = "gwt-debug-";GWT_DEBUG = "gwt-debug-";GWT_DEBUG = "gwt-debug-";

protected WebDriver driver;protected WebDriver driver;protected WebDriver driver;protected WebDriver driver;

@Before public void setUp() throws IOException {public void setUp() throws IOException {public void setUp() throws IOException {public void setUp() throws IOException { FirefoxProfile firefoxProfile = new FirefoxProfile();new FirefoxProfile();new FirefoxProfile();new FirefoxProfile(); driver = new FirefoxDriver(firefoxProfile);new FirefoxDriver(firefoxProfile);new FirefoxDriver(firefoxProfile);new FirefoxDriver(firefoxProfile); }

@After public void tearDown() {public void tearDown() {public void tearDown() {public void tearDown() { driver.close(); }

protected WebElement findElementOrFail(By by) {protected WebElement findElementOrFail(By by) {protected WebElement findElementOrFail(By by) {protected WebElement findElementOrFail(By by) { WebElement element = driver.findElement(by); if (element == null) {if (element == null) {if (element == null) {if (element == null) { org.junit.Assert.fail("Element " + by + " not found"); } return element;return element;return element;return element; }}

Other Good Base Class Methods

• Action methods:protected void click(By by) {protected void click(By by) {protected void click(By by) {protected void click(By by) { WebElement element = findElementOrFail(by); element.click();}

API improvements:protected void setText(By by, String text) {protected void setText(By by, String text) {protected void setText(By by, String text) {protected void setText(By by, String text) { WebElement element = findElementOrFail(by); WebElement element = findElementOrFail(by); WebElement element = findElementOrFail(by); WebElement element = findElementOrFail(by); element.clear(); element.clear(); element.clear(); element.clear(); element.sendKeys(text); element.sendKeys(text); element.sendKeys(text); element.sendKeys(text);}}}}

Rich assertions:protected void setText(By by, String text) {protected void setText(By by, String text) {protected void setText(By by, String text) {protected void setText(By by, String text) { WebElement element = findElementOrFail(by); WebElement element = findElementOrFail(by); WebElement element = findElementOrFail(by); WebElement element = findElementOrFail(by); assertTrue(element.isDisplayed()); assertTrue(element.isDisplayed()); assertTrue(element.isDisplayed()); assertTrue(element.isDisplayed());}}}}

Remember Our “Simple” Test?@Testpublic void testUnknownCountryName() { FirefoxProfile firefoxProfile = new FirefoxProfile(); WebDriver driver = new FirefoxDriver(firefoxProfile);

WebElement countryElement = driver.findElement(By.id(GWT_DEBUG + DebugIds.COUNTRY_NAME_ID)); countryElement.sendKeys("Does not exist"); WebElement loadCitiesButton = driver.findElement(By.id(GWT_DEBUG + DebugIds.LOAD_CITIES_ID)); loadCitiesButton.click();

WebDriverWait wait = new WebDriverWait(driver, 5000); wait.until(ExpectedConditions.visibilityOfElementLocated(By.id(GWT_DEBUG + DebugIds.NO_CITIES_ID)));

WebElement noCitiesElement = driver.findElement(By.id(GWT_DEBUG + DebugIds.NO_CITIES_ID)); assertTrue(noCitiesElement.isDisplayed());

WebElement cityRowElement = driver.findElement(By.id(GWT_DEBUG + DebugIds.CITY_ROW_ID)); assertFalse(cityRowElement.isDisplayed());}

Now It Looks Like…@Testpublic void testUnknownCountryName() { setText(WeatherHomeLocators.countryName, "Does not exist"); click(WeatherHomeLocators.loadCities);

WebDriverWait wait = createDefaultWait(); wait.until(ExpectedConditions.visibilityOfElementLocated(WeatherHomeLocators.noCities));

assertIsDisplayed(WeatherHomeLocators.noCities); assertIsNotDisplayed(WeatherHomeLocators.cityRow);}

Much Better!!!

Any More Problems?

• Today you select a country by:

– Typing a country name.

– Clicking the “Load Cities” button.

• What if we change it to:

– Select the country from a dropdown list.

• Our tests all break!

Tip #3: Use Page Objects

• Create higher-level action abstractions on a

per-page basis.

• Write tests that use those abstractions.

• Then, if the underlying UI changes, we can

update a single Page Object instead of many

individual tests.

Page Object Design Considerations

• Put your Locator definitions there.

• Give it access to your WebDriver API helpers.

• Put your page-specific action and (more

optionally) assertion methods there.

Page Object Examplepublic class WeatherHomePageObject {public class WeatherHomePageObject {public class WeatherHomePageObject {public class WeatherHomePageObject {

public static interface WeatherHomeLocators {public static interface WeatherHomeLocators {public static interface WeatherHomeLocators {public static interface WeatherHomeLocators {

public static final String public static final String public static final String public static final String GWT_DEBUG = "gwt-debug-";GWT_DEBUG = "gwt-debug-";GWT_DEBUG = "gwt-debug-";GWT_DEBUG = "gwt-debug-";

By countryName = By.id(GWT_DEBUG + DebugIds.COUNTRY_NAME_ID); By loadCities = By.id(GWT_DEBUG + DebugIds.LOAD_CITIES_ID); // etc. }

private WebDriverWrapper driverWrapper;private WebDriverWrapper driverWrapper;private WebDriverWrapper driverWrapper;private WebDriverWrapper driverWrapper;

public WeatherHomePageObject(WebDriverWrapper driverWrapper) {public WeatherHomePageObject(WebDriverWrapper driverWrapper) {public WeatherHomePageObject(WebDriverWrapper driverWrapper) {public WeatherHomePageObject(WebDriverWrapper driverWrapper) { this.driverWrapper = driverWrapper; this.driverWrapper = driverWrapper; this.driverWrapper = driverWrapper; this.driverWrapper = driverWrapper; }

public void loadCitiesForCountry(String countryName) {public void loadCitiesForCountry(String countryName) {public void loadCitiesForCountry(String countryName) {public void loadCitiesForCountry(String countryName) { driverWrapper.setText(WeatherHomeLocators.countryName, "Does not exist"); driverWrapper.click(WeatherHomeLocators.loadCities); }

public void assertCitiesNotLoadedDisplay() { driverWrapper.assertIsDisplayed(WeatherHomeLocators.noCities); driverWrapper.assertIsNotDisplayed(WeatherHomeLocators.cityRow); }}

Now It Looks Like…@Testpublic void testUnknownCountryName() { pageObject.loadCitiesForCountry("Does not exist"); pageObject.loadCitiesForCountry("Does not exist"); pageObject.loadCitiesForCountry("Does not exist"); pageObject.loadCitiesForCountry("Does not exist");

WebDriverWait wait = createDefaultWait(); wait.until(ExpectedConditions.visibilityOfElementLocated( WeatherHomeLocators.noCities));

pageObject.assertCitiesNotLoadedDisplay(); pageObject.assertCitiesNotLoadedDisplay(); pageObject.assertCitiesNotLoadedDisplay(); pageObject.assertCitiesNotLoadedDisplay();}

This change is subtle…but important!It makes your tests maintainable and

robust to change in the UI!

We are pretty happy, but…

That Seems Like a Lot of Work!

• It is! Selenium WebDriver tests are:

– Time-consuming to develop.

– Somewhat susceptible to being “flaky” (example: use of “wait”s).

– Highly susceptible to breaking when the UI changes.

– Sometimes hard to maintain.

• Consequently, we often focus more on unit testing, less on

end-to-end tests.

• Usually this is pretty successful, but can we do better?

Big Idea

What if we could generate and regenerate interesting tests automatically, and find bugs in the process?

WebTestingExplorer

• Supports Automated Exploratory Testing.

– AutomatedAutomatedAutomatedAutomated: Easy to replay (and regenerate) interesting

test cases.

– ExploratoryExploratoryExploratoryExploratory: Systematically discovers new application

states.

– TestingTestingTestingTesting: Includes test case generation, oracle generation,

real-time bug detection, and test case replay.

Advantages• Efficiently test through the user interface.

• Test like a real user.

• Don't miss integration issues and things that

are hard to observe.

– Behind the scenes in the browser.

– Or on other application tiers.

WebTestingExplorer Design

• Assumes a testable application.

– Example: We can ask developers to add id attributes.

• Built on top of WebDriver.

– Easy to integrate with existing WebDriver code.

• Highly configurable and extensible.

• Easy to skip/ignore uninteresting and/or unstable regions of

the web page.

– Examples: Ads, “Like” buttons, common headers/footers, etc.

WebTestingExplorer Workflow

•Two Complementary Phases:

1. Exploratory

– Incrementally searches for new application states.

– Writes out new test cases and oracles.

– Can search for bugs.

– Tester iterates on the configuration to guide the tool into new areas of the application

(the feedback-driven, exploratoryexploratoryexploratoryexploratory approach to testing).

2. Replay

– Executes tests generated by the Exploratory phase.

– For regression testing.

– A/B, comparison of actual and expected states.

State

• An abstraction of the current state of the web application, customizable by

the tester.

• Looking at “state” is key to how WebTestingExplorer generates test cases.

• State can be any combination of:

– Web browser state (HTML, DOM, cookies, ...)

– Web server state (session data, variables, ...)

– Database state (persistent application data)

– User-defined (but some state implementations are included and ready-to-

use!)

Exploratory Workflow• Iteratively and systematically moves through the web

application via the user interface.

• Monitors for different types of bugs.

• Generates increasingly-long and sophisticated test cases

(and oracles) for later replay.

– Driven by feedback derived from state.

• Tester iterates on the configuration to guide the tool into interesting new

areas of the application and away from uninteresting ones (the feedback-

driven, exploratoryexploratoryexploratoryexploratory approach to testing).

Automated Exploration• Changes in observed state between actions move the exploration forward into

longer test cases.

• High-level algorithm:

– Pick a candidate test case (a series of actions – click, set-text, select, etc.).

– Take a state snapshot.

– Do something.

– Take another state snapshot.

– Compare the “before” and “after” states.

– Different? Then:

• Generate a new test case.

• Enqueue new, longer candidate test cases to execute.

– Repeat until out of time, or all test cases of a specified length are executed.

Automated Error Detection (Oracles)

• HttpResponseOracle: Checks for allowed and disallowed

HTTP response codes.

• JSErrorCollectorOracle: Looks for Javascript runtime errors in

Firefox.

• FinalStateCheckOracle: Uses expected and observed final

state for regression testing.

• Testers can plug in custom oracle implementations.

– Example: If your application displays an error bar when an error

occurs, you can write an oracle that looks for the error bar!

Regression Testing• Exploration creates a test case.

– Test case includes saved final state for the configured state checkers.

• Replayer is configured with the FinalStateCheckOracle.

• Replayer replays the test case.

• At the end, a state snapshot is taken and compared to the

final state saved in the test case.

• If they are different, a failure is recorded.

Configuration• Selectors for Actionable and Stateful web elements.

• Pre-packaged and custom Oracles.

• Action sequence filters to eliminate redundant or uninteresting

exploration paths.

– Example: MaxRepeatedActionSequenceFilter eliminates action sequences

that repeat the same action more than a set number of times.

• Action generators to create custom actions on specific elements.

• Action sequence prioritizer for test case prioritization during exploration.

• Custom wait conditions to increase stability while executing an action

sequence.

Let’s Look at Some Code!

Custom Action Generator• If you have links to external web sites (e.g. www.google.com) , you

probably don’t want to write tests that navigate there.

new TagActionGeneratorConfig("a") { @Override public Set<Action> generateActions(WebElementWithIdentifier elementWithId) { Set<Action> actions = Sets.newHashSet(); String href = elementWithId.getElement().getAttribute("href"); if (href != null && href.startsWith("javascript:")) { actions.add(new ClickAction(elementWithId.getIdentifier())); } return actions; }}

• This will disallow test cases for all <a>’s that do NOT execute Javascript.• This is sufficiently useful that it is included as a library class:

• JavascriptAnchorActionGeneratorConfig

Another Example• If you have a textbox that takes login names, you want to try both valid

and invalid logins.

This will generate test cases with combinations of valid and invalid login names.

new IdentifierActionGeneratorConfig( new IdWebElementIdentifier(GWT_DEBUG + DebugIds.USER_NAME)) { @Override public Set<Action> generateActions(WebElementWithIdentifier elementWithId) { Set<Action> actions = Sets.newHashSet(); actions.add(new SetTextAction(elementWithId.getIdentifier(), "scott@example.com")); actions.add(new SetTextAction(elementWithId.getIdentifier(), "not a real user")); return actions; } })

A True Story

• I built the WeatherApp just for this conference.

• The first time I ran WebTestingExplorer against it, I

found a bug!

– An empty country name tries to load all the cities in the

world that have weather data!

– Not what I intended, not a good idea.

Test Case (XML Format)<?xml version="1.0" ?><TestCase> <url>http://127.0.0.1:8888/WeatherAppV1.html</url> <actionSequence> <actions> <SetTextAction> <keysToSend>Not a country</keysToSend> <identifier class="IdWebElementIdentifier"> <id>gwt-debug-CountryName</id> </identifier> <initial>false</initial> </SetTextAction> <ClickAction> <identifier class="IdWebElementIdentifier"> <id>gwt-debug-LoadCities</id> </identifier> <initial>false</initial> </ClickAction> </actions> </actionSequence> <finalState> <VisibleElementsState> <elementProperties> <!-- Lots more XML, properties and values of page elements -->

Test Case (Pretty Format)

1. Click the element with id gwt-debug-LoadCities2. Click the element with id gwt-debug-LoadCities3. Set the text in the element with id gwt-debug-CountryName to 'China'4. Set the text in the element with id gwt-debug-UserName to 'scott@example.com'5. Refresh the page

PASSED

WebTestingExplorer is Open Source Software

• http://www.webtestingexplorer.org

• http://code.google.com/p/webtestingexplorer/

• Contributors welcome!

Planned Features

• Parallel exploration.

• Code coverage integration.

• Different state abstractions.

– And advice and data on which ones work best.

Conclusions

• End-to-end testing of web applications is hard. But we can do

it!

• We can organize our Selenium WebDriver code to support

rapid development of concise, robust, maintainable tests.

– Organize Your Locators.

– Use a Base Class for Tests

– Use Page Objects

• And for automated exploratory testing:

– Try WebTestingExplorer.

Questions?Scott McMaster

Google

Kirkland, Washington USA

2012-12-20