By Arik Lerner & Waseem Hamshawi

Kubernetes your Tests! Testing with Docker on Google Cloud Platform

The JourneyBased On True Story!

{

“Name”: “Arik Lerner”,

“Role”: “Automation Infra Team Leader at LivePerson”,

“Data”: [

“3 years as Team Leader in Liveperson” ,

“2 years Java developer Reporting Platform Liveperson”

],

“Hobbies”: “Private pilot”

}

Active Prevention Department

{

“Name”: “Waseem Hamshawi”,

“Role”: “Senior Automation Infra Developer at Liveperson”,

“Data”: [

“2.5 years at Liveperson”,

“github: waseemh.github.io”,

],

“Hobbies”: “Running”

}

Waseem is Here!

Active Prevention Department

Short Story

From Manual Post Deployment Verification

TOAutomation

❏ Fast feedback

❏ Stable Environment

❏ Isolation

❏ Investigation tools - Hard to support (Video/HAR)

❏ Scale

❏ Efficiency

❏ Browser diversity

❏ Mobile Support

Requirements

Selenium HubRunning Test

Static VMs

Supported Requirements

Browser diversity

Mobile Support

Parallelism

Traditional Selenium Grid

Ranking:

Tooling

Deployment/Configuration

Feedback

Isolation

Efficiency

Scalability

❏ Less stable❏ Crashes - require restarts❏ Cancel build in progress leave open sessions

❏ Investigation tools - Hard to support (Video/HAR)

❏ Manual grid node Configuration/deployment

❏ Not scalable

❏ Slow feedback

❏ Not efficient - idle grid machines

Facing Issues

Cattle vs. Pets

VM

Container

● When they get ill you nurse them back to health

● You name them! And take of them!

● When they get ill you get another one!

● They are almost identical to one another

● Don't have specific names. Given numbers

● Can be simply replaced if they die

What Docker gives us ?

● Build it once run it everywhere!

● Resource Isolation

● Test Isolation

● Package format

● Implementation of a container

Docker

Chrome FirefoxVNC Server

Selenium Server Hub/Node JAR

Mobile Android

What is in a Selenium Container ?

VM

VM

VM

VM

NodesHUB

N Hub’s

Running Tests

Selenium Grid Architecture Makeover

Selenium Docker Images

hub.docker.com/r/selenium

DEMO

We need more than just Packaging and Isolation:

● Orchestration - Distribute containers

● Health Checks

● Resilience and Self-healing

● Scaling - making jobs bigger or smaller

Kubernetes Orchestrator

VM

VM

VM

VM

PodsHUB

N Hub’s

Running Tests

Selenium Grid Architecture Makeover

Testing on the Cloud!

Benefits of Cloud Platform

● Kubernetes cluster in a click of a button (GCP)

● No idle machines - Use when needed

● Costs per build/test - measurable

● Not part of Liveperson network

● Geolocation simulation

DEMO

Build Life Cycle

GCP - Selenium Grid lifecycle

Run tests

Kill Selenium Grid

Scale Down GCP Instances

Scale Out GCP Instances

Create Selenium Grid container cluster

Requirements

Requirements

❏ Fast feedback

❏ Stable Environment

❏ Isolation

❏ Investigation tools - Hard to support (Video/HAR)

❏ Scale

❏ Efficiency

❏ Browser diversity

❏ Mobile Support

Ranking:

Tooling

Deployment/Configuration

Feedback

Isolation

Efficiency

Scalability

❏ Investigation tools - Video tools not reliable and not stable.

❏ Feedback ❏ Improved BUT still needs to wait for nodes to

connect into Hub❏ Point of failures

❏ Hub/Nodes is more failure points in the architecture that not really necessary

Facing Issues

waseemh-mac$ java -jar selenium-server-standalone-2.53.1.jar -role

Selenium server can run in one of the following roles: hub as a hub of a Selenium grid node as a node of a Selenium grid standalone as a standalone server not being a part of a grid

Test Hub Selenium Server(Node)

Selenium Standalone

Non-Grid Solution● Fast → Deploy Selenium Standalone containers on-demand● Short-lived → Selenium containers are destroyed once test ends● Distribute and manage containers with Kubernetes

TestcreateRemoteWebDriverSession(caps)

VM

VM

VM

VM

new RemoteWebDriver(External Endpoint)

Test Cluster

?

github.com/LivePersonInc/ephemerals

Ephemerals

Ephemerals API

Deploy

Endpoint IP:Portnew RemoteWebDriver(Endpoint IP:Port)

createRemoteWebDriver(desiredCapabilities)

TestTestTestTestTestTestTestTestTest

JUnit Integration@Rulepublic EphemeralResource<RemoteWebDriver> seleniumResource =

new EphemeralResource( new SeleniumEphemeral.Builder(deploymentContext) .withBrowser(FIREFOX_42) .withVideoRecording(true) .build());

@Testpublic void test() { RemoteWebDriver remoteWebDriver = seleniumResource.get(); .....}

DEMO

Challenges● Appium crashes/unresponsive, adb server restarts

● Android SDKs environment installation/maintenance

● Appium↔WebDriver API Versions Incompatibility

● Difficult to parallelise mobile tests● Scale mobile tests ?!

Mobile Testing

Emulator

Ephemeral’s Way

Emulator

Appium WebDriver

Test

Install (APK URL)

Ephemerals API

createMobileDriver(capabilities) Deploy

Appium Endpoint IP:Port

new AndroidDriver(Appium IP:Port)

DEMO

Requirements

Ranking:

Tooling

Deployment/Configuration

Feedback

Isolation

Efficiency

Scalability

Requirements

❏ Fast feedback

❏ Stable Environment

❏ Isolation

❏ Investigation tools - Hard to support (Video/HAR)

❏ Scale

❏ Efficiency

❏ Browser diversity

❏ Mobile Support

Beyond Selenium...

Deploy any test dependency inside a container

● Web Servers (/modules/nginx)● Temporary Databases (/modules/mysql)● Mock Servers (/modules/wiremock)● Any Docker Image (/modules/custom)

Talking Together...

Web Application

Nginx Ephemeral Selenium Ephemeral

EP

Test Script

EP

WireMock Ephemeral

EP

RESTAPI

HTTP

Ephemerals - Future Plans

● REST API (for non-Java tests)

● Support more cluster management systems:

○ Apache Mesos

○ Docker Swarm

● More modules

Takeaways

● Enhance traditional Selenium tools

● Containerization - Short lived, throwaway instances

● Scalable tests with Containers Orchestration

● Benefits of Testing on Cloud Platforms

THANK YOU!