Parallel Test Runs with Appium on Real Mobile Devices – Hands-on Webinar

12 May 2015

Ville-Veikko HelppiTechnical Product Manager

[email protected]

Parallel Test Runs with Appium on Real Mobile Devices

W E B I N A R

Agenda

• Testdroid Update

• Statistics from Q1'2015

• Testing and Mobile Test Automation

• Appium – Client vs. Server Side

Execution

• Demonstration

• Q&A

TESTDROID UPDATE

Public Device Cloud on-demand devices

(multitenant)

Mobile app testing on thousands of real

Android and iOS devices hosted by

Bitbar

Private Device Cloud Reserved Devices

Hosted by Bitbar in the US and/or Europe

Devices chosen by and reserved only for

the Customer

On-Premise Device Cloud

Automated mobile app testing devices

hosted by the customer, usually 30-500 devices

1 Product – 3 Deployment Options

Testdroid Cloud Testdroid Enterprise Testdroid PrivateCloud

New Pricing Plans for Cloud

*) = Prices for annual commitment

Appium

• Client-Side Execution

• Server-Side Execution

Q1'2015 STATISTICS

Q1'2015 – Test Run Stats

Why Apps Fail?

iOS 8.0

iOS 8.0.2

iOS 8.1

iOS 8.1.1

iOS 8.2

iOS 8 ALL

KitKat 4.4

KitKat 4.4.2

KitKat 4.4.3

KitKat 4.4.4

KitKat ALL

Lollipop 5.0

Lollipop 5.0.1

Lollipop 5.0.2

Lollipop ALL

0 5 10 15 20 25 30 35

Failed test runs. Percentage (%).

Some Observations

1

2

Some Observations3

4

Some Observations

5

6

Android Fragmentation 2014

by Open Signal report July 2014

How Many Devices is Enough?

~90%market coverage can

be achieved with

128devices

~20%market coverage can

be achieved with

12 devices

US Market

25 Android devices= ~2/3 market

Global Market

60 Android devices= ~1/2 market

TESTING AND MOBILE TEST AUTOMATION

OS

versions

Chipsets

CPU +

GPU

Tens of

OEMs

Memory

Displays

(resolutions,

physical hw)

OEM

mods

Other

hardware

(connectivity

calibration)

Relation

to other

software

Where Test Automation Can Help

Correct behaviouracross platforms and browsers

Integration with web back-ends

Typically need to fully utilize HW (CPU+GPU)

Resource (e.g. battery) consumption

OpenGL ES 2/3

Functionality and usability

Screen orientations, connectivity, user profiles

Robustness

Robustness and security!

Brand

Compliances, verification with back-

ends and data

Different Mobile 'App Verticals'

Manual Testing is not Agile

Test Automation is Agile

Why Real Devices Are Must-to-Have

• Emulators cannot help you to test...

• User Experience

• Usability

• Hardware

• Software

• Infrastructure

0 % = the percentage of your app users that use emulator to run your app!

Manual vs. Automation

Manual Automation

Manual vs. Automation

Smaller coverage, More money burnt & time wasted, Error-prone

Manual Automation

Large coverage,

quickly completed,

Less money & time wasted, Exact results.

APPIUM

– CLIENT VS. SERVER SIDE EXECUTION

Family Tree of Android Test

Automation Frameworks

JUnit

Android Instrumentation Framework

Robotium

Espresso and Espresso v2

uiautomator

AppiumExtSolo

Calabash

What Framework Works You The Best?

Robotium uiautomator Espresso Appium Calabash

Android Yes Yes Yes Yes Yes

iOS No No No Yes Yes

Mobile web Yes

(Android)

Limited to x.y

clicks

No Yes (Android &

iOS)

Yes

(Android)

Scripting

Language

Java Java Java Almost any Ruby

Test creation

tools

Testdroid

Recorder

UI Automator

viewer

Hierarchy

Viewer

Appium.app CLI

Supported

API levels

All 16 => 8, 10, 15- All All

Community Contributors Google Google Active Pretty quiet

Selenium & Appium

Client Side Appium at

Testdroid CloudTest Script

Test Case

Desired Capabilities

{

“device”: “Android”,

“app”: “/Users/user/ApiDemos.apk”

“app-package”: “com.example.android.apis”

“app-activity”: “.ApiDemos”

}

Test Script

Test Case


WebDriverhttp://localhost:4723/wd/hub

Appium Server

4723

Device

Localhost

(DesiredCaps)

http://localhost:4723/wd/hub

*Testdroid Caps

http://appium.testdroid.com/wd/hub

(DesiredCaps)

Test Script

Test Case


WebDriver

From Localhost to Testdroid

Client Side Execution

Go tocloud.testdroid.com


Get a Device Name



Add Testdroid Desired Caps to test script

{

“testdroid_username”: “[email protected]”,

“testdroid_password”: “p4s$w0rd”,

“testdroid_project”: “My First Project”,

“testdroid_testrun”: “Test 1”,

“testdroid_device”: “iPad Mini 7.0.4 A1432”,

“testdroid_app”: “http://domain.com/app_v1.ipa”

.

.

“app”: “com.bitbar.testdroid.BitbarIOSSample”

}

Get a Device Name



driver = webdriver.Remote("http://appium.testdroid.com/wd/hub", desired_caps);

Point the Webdriver to http://appium.testdroid.com/wd/hub


Get a Device Name



Run the Test ScriptGet Results from Testdroid Cloud



Get a Device Name



Pull the Results from the Result URL

driver = webdriver.Remote("http://appium.testdroid.com/wd/hub", desired_caps);

Run the Test ScriptGet Results from Testdroid Cloud



Get a Device Name


Multiple Devices – Client Side

python testscript.py –device <devicename>1

python testscript.py –device <devicename>2

python testscript.py –device <devicename>n

Test Script

Test Cases

Instigator Script

deviceArray=[

“iPad 4 6.0.1 A1458”,“iPad mini 7.0.4 A1432”,...“iPhone 4S 6.1.3 A1387”,

]

21

3

4

5

6

7

WebDriver Session Request@http://appium.testdroid.com/wd/hub

WebDriver Session Response

Testrun

Configure Project

Appium Ready

Wait for Device toBecome Available

Device 1 Device 2 Device 3

Session

Map

Proxy

Device Cluster

Start Appium

Desired Capabilities, .apk / .ipa

SessionID

SessionID

Sessionid

Test

Script

Appium

Broker

Client Side – Behind the Scenes

Introducing "Server Side Appium"

DemonstrationAppium with Real Devices using Image

Recognition for Hill Climb Racing

Setup

• Using real Android devices at Testdroid Cloud

• Parallel test runs without a need to configure desired

capabilities

• Device groups (= set of devices used for runs) can be

manually created and configured

File Structure• pom.xml (maven)

• testdroid.properties (overwritten after submitted to Cloud)

• run-test.sh (shell script for execution)

• image files

15 seconds

Image Recognition

• Resolution agnostic implementation

• Identifies stretched and rotated images as well

Let's See How it Works!

THANK YOU!More information about frameworks, devices

and testing at www.testdroid.com

Parallel Test Runs with Appium on Real Mobile Devices – Hands-on Webinar

Technology

Transcript of Parallel Test Runs with Appium on Real Mobile Devices – Hands-on Webinar