Automated Acceptance Tests from ScratchDaniel Davis

• Daniel Davis

• Software Developer for 8 years

• Fun Fact:

– I recently went to see “Laser Cat” in DC

Who Am I?

3

Really though, who are you?

• Came from Java world

• Python developer for 2 years

• DevOps

– Lots of work with automation and quality

• First introduced to AAT about 2 years ago

– Now I use it at my client!

5

Deploy with CI

Website

Now that it’s up…how do I test that it works?

6

• Go to the website, click around, make sure

things look right…

• Problem: Scaling

– What happens as the site continues to grow?

– What happens as we deploy more frequently?

• We need a way to test the functionality of

the site

Simple!

Automated Acceptance Testing Can Help!!!

• Tests written at the functional/business

level

• Can be read by non-technical people

• Doesn’t have to be automated to provide

value!!!

What is Acceptance Testing

Unit – 70%

Integration

– 20%

AAT

10%

Where does AAT fit in?

Consider it this way…

11

Unit Testing Integration Testing

12

Automated Acceptance Testing

• Great for covering a lot of ground

• Not so good for very specific things

Ok, so how do I get started???

Popular Frameworks

• Java/JVM world: Cucumber-JVM/JBehave

• Ruby: Cucumber-Ruby

• Ruby on Rails: Cucmber-Rails

• Node.js/Javascript: Cucumber-Javascript

– AngularJS: Protractor not the same

• C#/.NET: SpecFlow

• Python: Behave

Cucumber

16

• Cucumber’s syntax and style is virtually

the same across all the platforms!

• Tonight’s examples will be in Python

Good News Everyone…

17

• One example, from start to finish

– This is how we solve problems

• Learning Objectives:

– Write an acceptance test

– Automate that test using Selenium

– Incorporate that test into your Continuous Integration

• Stay high level

– Not enough time to go deep

Let’s get started!

18

Example:theCatAPI.com

19

• Objective: Want to test Upvote/Downvote

functionality

• Follow Along:

– https://github.com/Ooblioob/aat_demo/

theCatAPI.com

Step #1: Write an Acceptance Test

22

• Install your cucumber tool of choice

• Directory structure:

project-root/

|-- features/

|-- steps/ (we’ll talk about this soon)

|-- pages/ (we’ll talk about this soon)

Project Setup

23

3 Layers to Cucumber

24

Gherkin (Feature)

“Glue” (Step)

Selenium Code

• A Human-readable syntax describing the

test

• Consists of a main “feature” and scenarios

describing the feature

Writing Gherkin

Feature: Title (one line describing the story/feature)

As a [role]

I want [feature]

So that [benefit]

Scenario: Title1 (for Behaviour 1)

Given [context or setup]

And [some more context]...

When [event occurs]

Then [expected outcome]

And [another outcome]...

Scenario: Title2 (for Behaviour 2)

…

Writing Gherkin

26

TheCatAPI.com

27

https://github.com/Ooblioob/aat_demo/blob/master/browser_tests/

features/upvote.feature

• Value even when not automated

– Clear, concise way to manually test

– Developer friendly

• Not extremely specific…

– What does it mean “a new cat image should be loaded…”?

– Declarative vs Imperative

– Don’t get bogged down in precision/verboseness

Observations

Step #2: Automating your Acceptance Tests

29

3 Layers to Cucumber

30

Gherkin (Feature)

“Glue” (Step)

Selenium Code

• Create an “environment.py” file in the

features directory

– Defines the test runner setup

• (optional) Create an environment.cfg file in

the features directory

– Holds editable configurations for your setup

• Create a “steps.py” file in the steps directory

– Contains the implementation of your steps

Project Setup

31

Environment.py

32

Full Example:

https://github.com/Ooblioob/aat_demo/blob/master/browser_tests/fe

atures/environment.py

• Setting up Selenium

– Select a driver, like Chromedriver, PhantomJS

or SauceLabs

• Setting up a logger

• Initializing the context object

• Set defaults

What goes in Environment.py?

33

• Parses each line of Gherkin

• Decide how to implement

Glue code (steps)

home.feature

steps/steps.py

Writing your implementation

Example:

https://github.com/Ooblioob/aat_demo/blob/master/browser_tests/featu

res/steps/steps_home.py

• Making references to context

– Way of passing information between steps

• What is “Home” in the context?

– Class representing the homepage

– Part of “Page-object model”

Observations

36

That pretty much it…When do we talk about Selenium?

37

Not the best practice…

38

Glue (step)

One Giant Selenium Class

Page-Object model

39

Glue (step)

Home

page

Docs

page

Reference

page

Utility

functions

Base Page

So what goes in a “Page” object?

40

“Hate It” Button

“Like It” ButtonTitle

Main

Image

• For each page element:

– Selectors: Id, CSS, XPATH, etc

– get() method

– commonly used attributes

• Define navigation to the page

– go()

• Define other actions / helpers

– Storing info, waits, etc.

Writing your pages

Writing Selenium Code

43

One more small detail…

44

• Need to setup the web driver

– Let’s use ChromeDriver

• https://sites.google.com/a/chromium.org/chromedri

ver/getting-started

Environment.py revisited

45

Demo Time!(Making it all work)

46

• Use page-object model to organize code

• Write simple, reusable methods to

implement pages

• Installed Chromedriver

Recap

Step #3: Put it into Continuous Integration!

48

• Now that we have tests

– Validate site is up and running

– Validate deployments

– Ensure new code doesn’t break existing

functionality (regression)

Why Continuous Integration?

Challenges

• PhantomJS

– Headless browser

– Based on Webkit

– Fast

• Works with Selenium

Solving the problem

51

• Black Box

– Can’t see problems when they happen

• Can’t handle certain things

– e.g. alerts

• Buggy, but getting better

Problems with PhantomJS

52

At this time, I don’t recommend it…

• “Cloudified” browsers

Sauce Labs

53

• Test different browser / os configurations

– Don’t need to own that hardware

• Integration with many CI tools

– Jenkins, Travis, CircleCI, Bamboo, Teamcity…

• Video recording/playback

– No longer a black box!!!

Advantages

54

Yes, but what about my Dev severs???

55

• Sauce Connect: test your Dev server too!

Advantages

56

So how do I set it up?

57

• Sign Up for an account:

– https://saucelabs.com/signup/trial

• Receive an API key

Sauce Labs Setup

• Modify your environment.py file to use

Sauce instead of Chrome

– Provide username and api key

– Select a browser and OS to use

Sauce Labs Setup

59

Sauce Labs Setup

60

https://github.com/Ooblioob/aat_demo/blob/master/browser_tests/f

eatures/environment.py

Sauce Labs Setup

61

https://github.com/Ooblioob/aat_demo/blob/master/browser_tests/f

eatures/environment.py

Demo Time!

62

One last topic

• Can only test what’s available through the

browser

• Selenium can be painful at times

– May not wait for things to load

– Debugging can be hard

Limitations of AAT

• Data

– Not a great choice for data-heavy applications

– Not impossible, but trickier:

• Launch a separate test server?

• Create a “testing mode” where transactions are

rolled back

– Might be better covered by Integration Testing

Limitations of AAT

65

• https://github.com/Ooblioob/aat_demo

• https://cukes.info/

• http://jenisys.github.io/behave.example/

• https://docs.saucelabs.com/reference/sauc

e-connect/

Resources

66

Questions?

67