Automated Acceptance Test~ How to build it stable & maintainable

Bryan

QA

Programmer

Automation Engineer

DevOps Evangelist & Facilitator

Big Data

Ad Exchange (RTB)

Low Latency

High Concurrency

(We are hiring)

Automated Acceptance Test In Continuous Delivery

• “Are we good to go live?”

- Test jobs are your “eyes and ears”

- Optimize for them!

• Test code equals production code

Common problems:

• Costly to maintain test buckets

• Spent much time running and debugging false alarms

Acceptance Test

“ An Executable Specification of System Behaviour ”

Culture~ Responsibilities?

< 100 ms

What’s AD Exchange

“ Don’t have a separate Testing/QA team! Quality is down to everyone - Developer owns Acceptance Tests!! ”

― David Farley, Pipeline Conf. 2015

“ Not enough to have acceptance test and acceptance test got to be created and maintained by developer

will make code more testable and maintainable. ”

― Jez Humble, Pipeline Conf. 2016

don’t hire too many dedicated testers

developer relies on them, lazier and write more bugs

hire people who can development and test functions

code review occupies a central position

no separate group of tester or QA people

relies on automated testing

“Testing is essentially the responsibility of the person who develops a given feature”

(tens of thousands of regression tests)

developer responsible for writing unit tests, regression tests and performance tests

regulated FX exchange London & Tokyo launched 2010

~ $2 trillion traded in 2015

one of UK’s fastest growing technology companies

high quality - very low rate of production issues

(order of magnitude below industry average) “lowest bug count we’ve ever found”

- independent analysis by a well-known tool vendor

2 million lines of code (50:50 test/production)

half the codebase < 18 months old

Process / Discipline~ Definition of done

Definition of Done

• Automation task and sprint end demo

• Code review

• Pipeline dashboard

- pipeline always in deliverable status

- statistic analysis report

• Build and unit tests

• Full acceptance against the change sets

• Only delivered if above passed

Definition of Done

• Code review of automated test scripts

• Loop test suite 20 times before code merge

Loop of Your Tests

stage concurrency: 1, name: ‘Test Looping before merge'docker.image('qa/chrome-slave:2.53.0').inside('-v /dev/shm:/dev/shm --privileged=true') { wrap([$class: 'Xvfb', additionalOptions: '-fbdir /tmp']) { git branch: 'develop', url: 'http://abc.com/qa/accept-test.git' withEnv(["baseUrl=${baseUrl}","mySQLUrl=${mySQLUrl}"]) { for(int i = 0; i < count; i++) { println(“Loop of: ${i}”) sh 'mvn -B test -fae —DsuiteXmlFile=testng.xml’ } } }}

Jenkins Scriptable Build & Multibranch Pipeline

Immutable Environment

• Dockerize environments for build & test - Always fresh - Quick - Immutable - Scalable & parallel

• Dockerize testing infrastructure

Strategies~ How to make good one

Test Automation Pyramid

src: http://www.ontestautomation.com/tag/mike-cohn/

Test Isolation• Reliability/Repeatability

- Provide consistent, repeatable results.

• Isolation

- Tests should not depend, or be affected by, the results of other tests: testContext

- users, accounts, advertiser names etc: alias name

- external services, 3rd-party integrations: stub, simulator

account: ‘Johnny’ ==> ‘Johnny_4534031’

book: ‘DevOps 101’ ==> ‘DevOps 101_1234567’

Parallelisation

• Commit stage < 15 mins

• Acceptance Test stage < 45 mins

• Fail FASTer!

• Radical parallelisation

- throw-away environments (e.g. containers)

Separation of Concern• Use Domain Specific Language (DSL)

- Focus on “What” not “How”

• Ease of Development

- Hide details of how the tests talk to the SUT

• Ease of Maintenance

- When tests break, we Identify the problem and fix it quickly.

‘What’ Not ‘How’

‘What’ Not ‘How’

‘What’ Not ‘How’

Focus on ‘What’

Not ‘How’

API Example@Given("^I create a advertiser with those fields fill : name '(.+)', state '(.+)', currency '(.+)'$") public void createAdvertiser(final String name, final String state, final String currency) { advertiserAPI.createDefaultAdvertiser(name, state, currency); }

package com.vpon.dsp.driver.web.rest.service.adv; public class AdvertiserServiceImpl { static { AdvertiserService advService = ServiceGenerator.createService(AdvertiserService.class); } public static String createAdvertiser(Advertiser obj) throws IOException{ Call<Advertiser> advExec = advService.createAdvertiser(obj); Response<Advertiser> resp = advExec.execute(); // id should not be null if created successfully if(resp.isSuccessful() && null != resp.body().id) { ...

}}}

package com.vpon.dsp.dsl.web; public String createDefaultAdvertiser(String name) { Advertiser obj = defaultAdvertiser(name); return AdvertiserServiceImpl.createAdvertiser(obj); }

DSL layer:

Driver layer:

Test case layer:

Implementation~ Right tools

API Test

“ API is more than just a status code, Verify every part of it.”

API Test

Concerns?• Support DevOps - high frequent delivery & BDD

• Maintainability - UI, API are volatile, avoid boilerplate code

• Tools: SoapUI, REST-assured, POSTMAN, Unirest

Test Case Example

Goal@end2endFeature: LineItem end-to-end system targeting test Background: An Advertiser, LineItem with different System targeting conditions created Given There is a 'advTargeting' advertiser and following 'System' targeting lineItems: | name | category | targeting | | LI_OS_iOS | OS Family | Apple iOS | | LI_LANG_Eng | Language | English | Scenario Outline: Bid accordingly with lineItem targeting setting Given I enable '<lineItemName>' lineItem When I send a bid request with '<criteria>' targeting only And I send a 'default' bid request with no targeting criteria Then I should receive '1' successful bid response Examples: | lineItemName | criteria | | "LI_OS_iOS" | "iOS" | | "LI_LANG_Eng" | "English" |

Retrofit

A type-safe HTTP client

for Android

and Java

GUI (Flaky) TestStability is super important for automation in CD

• 1% failure rate with 100 test scripts ( 0.99¹⁰⁰) = 63% chance of failure• You can’t get away with flaky tests in CI/CD• Spend much time debugging false alarms

Selenium + jQuery Selector• Implement JQueryBy class

- Tells Selenium how to find element with jQuery locator • If no jQuery in application page

- Inject one for testing • jQuery

- Powerful API for navigating and selecting content - Manipulate UI for easy assertion - CSS based, a whole lot better than XPath - Has “qualified” feature (ex: has, contains) for filtering

• No xPath locator - Slow & tight to DOM structure - Easily broken with html changes - Bad readability and hard to maintain

Selenium PageObject

• Separate test logic from web implementation - No html stuff (locator, Selenium code, …) in test case

• Increase maintainability - No need to update all hundreds of TCs when UI changes - Object classes are the only place to modify

• Increase stability - More tuning on the framework, it gets more stable

src: Martin

Handling Wait in UI TestUser Perform

Action if_Async

CheckPageReady() Check Ajax Request Complete

Wait Dynamic Element Presence Make Assertion

Check DB / Queue Status Match

Thread.sleep()

Handling Wait in UI Test• Each page loading is following the same ‘wait’ checking procedure • Each page can define it’s own ‘getPageLoadCondition()’public T initPage(Class<T> clazz) { T page = PageFactory.initElements(getDriver(), clazz);

checkPageReady(); //<== check for DOM, JS lib

ExpectedCondition pageLoadCondition = ((AbstractPage) page).getPageLoadCondition(); // for extended page does not want to check loading condition ... if (null != pageLoadCondition) { waitForCondition(pageLoadCondition); //<== check for page’s load condition } return page; }

Handling Wait in UI Testprivate static void checkPageReady() { /* for generic loading, but not all browsers compatible... final String chk1 = “(document.readyState==='complete'|| document.readyState===‘interactive')"; */ final String chk1 =

"(typeof jQuery != 'undefined') && ($(window).load(function() {return true;}))"; //for jQuery loaded and no ajax requests pending final String chk2 = "($.active === 0)"; final String condition = "return " + chk1 + " && " + chk2 + ";"; ExpectedCondition<Boolean> response = new ExpectedCondition<Boolean>() { public Boolean apply(WebDriver d) { return (Boolean)((JavascriptExecutor) d).executeScript(condition); } }; new WebDriverWait(getDriver(), LOAD_TIMEOUT).until(response); }

//for those with data grid loadingpublic static ExpectedCondition getDataLoadingCondition() { LogUtils.printDebugMessage(logger, "Before returning _loading wait condition..."); return new ExpectedCondition<Boolean>() { public Boolean apply(WebDriver d) { boolean result = (Boolean)((JavascriptExecutor) d).executeScript( "return ($(\"div[id$='_loading'][style*='display: none;']\").length===2" + " && $.active===0 “ + “ && $(\"div[id$='_loading'][style$='display: block;']\").length===0)" ); return result; } }; }

Wait Ajax Data Loading

(1) $.active === 0=> No ajax reqs pending(2) $(”div[id$=‘_loading’]…..”).length===0) => loading icon disappear

Wait Dynamic Web Elementpublic String getToolTipText() { SeleniumDriver.onPage(ByJQuery.jQuerySelector("div#content")); . . . . . //mouseover new Actions(getDriver()).moveToElement(e).build().perform(); String toolTipLocator = "div.tooltip[style$='display: block;']"; LogUtils.printMessage(logger, "trying to get tooltips of a warning ..."); return SeleniumDriver.waitUntilElementPresence( ByJQuery.jQuerySelector(toolTipLocator)).getText(); }

Capacity Testing• Performance testing for components• Long run and stress• Use production traffic - boost confidence level• Tools: Gor, Gatling

Reference[How google Test Software][Continuous Delivery: Reliable Software Releases][From research paper “Moving Fast with Software Verification - 2015”]

Others:https://dzone.com/articles/dev-centric-culture-breaking-down-the-wallshttp://blog.xebialabs.com/2015/06/22/guidelines-for-a-successful-test-strategy/http://www.androidwarriors.com/2015/12/retrofit-20-android-example-web.htmlhttps://gortool.com/http://gatling.io/#/

[About Vpon][About Me]

Q & A