Matan Goren and Yehuda MillerQA Engineers

Test Automation in a Continuous Deployment Environment

• More than 80 million registered users• 28 million family trees• Website, mobile apps (iOS and Android), desktop

apps (windows and Mac)• 2.5 billion profiles (names in family trees)• 4.5 billion historical documents and records.

Including the world’s largest collection of newspapers

• 200 million photos• 42 languages• 215 employees

Who we are

Where we were• All manual QA

• Repetitive tasks• Time consuming

• Two service packs a week• Holding back development• SPC size limits error handling

• Number of tests • Desktop: 538 (469)• Mobile-web 50 (0)

• Runtime of full suite: 37 min (51 min)• RND size

• Number of testers: 13 (7 automations)• Developers: 50

• Grids: 4 (1)• Tests run on prod and semi staging

env.• Continuous Deployment

Where we are today:

Gray Area for full image

“You don't have to be a genius or a visionary or even a college graduate to be successful. You just need a framework and a dream.”- Michael Dell

• Ruby• Watir-webdriver• Cucumber.io• Page-Object Gem• Grid Grouper• Selenium Grid/Grid Extras• Jenkins

Framework

• Easy language for inexperienced developers

• Open source community is very active• It ain’t Java!

Ruby

• Native to Ruby• Cleaner than Selenium• Adds methods not native to Selenium

• “Being able to select an element by a explicit identifier” –watirmelon.com

Watir-webdriver

Selenium:

Watir:

Why Watir is cleaner!

• Provides a simple interface to the define and interact with elements on a page.

• Works with both Watir and Selenium.• Simple way of introducing OOP without a

lot of technical knowledge required• Page factory module handles common

page classes action.

Page-Object gem

• BDD tool• Tests are written in plain English

• Makes for easy debugging• Easy move from Manual test to automatic test

• Helps soften the introduction to programming• “Top down” programming

• Write a step• Define the step• Make it work

• Pre and post test hooks

Cucumber.io

Cucumber Feature

Directly with Watir-WebdriverUsing page-object

Tags specify the suites

Step DefinitionsDirectly with Watir-Webdriver

Using page-object

Page-object example

Using page-object

Defining page-object

Verify page loaded with page-object

• Homegrown tool• Takes a given suite and distributes the test

across several groups• Utilizes open-source code from cucumber

and parallel_tests gems• Multiple reports problem is solved by a

report merger tool that was developed in house.

Grid Grouper

• Chuck Norris!• Was already in use by RND• Open source (lot’s of available plugins)• Schedule automated builds• Kick off manual builds with a few clicks

• Control build parameters (server, suite, etc).• Can build in any of our environments with any

combination of tags• Holds build history for a specified amount of time• Merged report is stored in each build.• HipChat and email notifications

Jenkins

Daily Build

Merged Report

Failed scenario

• Comprised of Selenium Grid and Selenium-Grid-Extras

• Selenium-Grid-Extras makes the basic setup streamlined

• Gives extra features such as:• Pulling in updated drivers for all browsers• Also for Selenium itself

The Grids

• We have 4 grids• Each grid has 4 VMs• The VMs include 1 hub-node and 3 other nodes• The hubs are Jenkins slaves• Each node can run 4 parallel tests on Chrome

• One grid is dedicated to CD• We can pull a grid “offline” for testing infra upgrades• Ability to have multiple test runs in parallel on different

environments and tags• Only runs through Jenkins

• Jenkins selects grids automatically based on availability and run history

How we use the grids

“How long does it take you to get one line of code into production?” - Wisdom of the Internet ;)

• We moved to CD in the last 6 months• On average we distribute code to

production 20 times per day.• Comes with challenges• Automated test on every code release• Only minimal tests on each release• Dev is responsible to run appropriate tests

PRE-RELEASE in staging

The CD mindset

• Risk reduction to production• Small incremental changes are easier to

monitor and revert in necessary• Has lower impact on the system

• Increasing R&D velocity• Avoiding wasted time on merges and complex

coordination before dist.• Allow R&D and product to experiment and

innovate more frequently.

RND Goals

• Provide a safety net• Avoid the bottleneck of manual sanity

testing• Reliability

• No flaky tests• No false failures

• QA E2E test should not be the new bottleneck

QA Goals

• QA are responsible for E2E tests only.• Dev write and maintain Unit tests• E2E tests are written during development

process • All QA members monitor CD suite results.• CD E2E tests are NOT acceptance tests

Responsibilities

• CD flow from commit to prod is ~25 minutes

• Current QA CD suite takes 1.5 minute in best case scenario.

• Runs in parallel to some of the jobs in the CD flow• Limited by time frame of parallel jobs• Still have room for further growth

Timing

• Minimal suite – No full regression on commit

• Occasional false failures due to env/site performance/human error

Risks

• Reliability• Velocity of tests• Pinpointing the failure• CD halt on failure.• Large scale tests refactoring due to

changes.

Challenges of CD

“A challenge only becomes an obstacle when you bow to it.” - Ray A. Davis

• Maintaining production data integrity• Reliance on pre-existing data• Running on local staging env• Fast adaptation to feature flags• Identifying failure reasons• Need a true BDD/TDD mindset

Challenges QA faces

• Testing on more browsers• IE• Spartan• Firefox

• Suites for our native apps• iOS• Android

• Upgrading the infra• Gems, grids, Ruby…

The road ahead…

Questions?

AA Milne

Thanks!We are hiring