HMUNAGAN@IN.IBM.COM

PRROHILL@IN.IBM.COM

MALLIKARJUN.GANIGI@IN.IB

M.COM

9985282049

9962979696

MPTE

AUTHORS

HAREESH MUNAGANURI

PRATIBHA ROHILLA

MALLIKARJUN GANIGI

ORGANIZATION –

IBM INDIA PVT Ltd.

MOBILE PERFORMANCE

TESTING EXPEDITE

SYNOPSYS

The phenomenal growth of mobile devices cannot be left

unnoticed by any Organization, intending to grow. Increasingly

Organizations, wanting to increase the reach of their

Applications.

However, this also poses challenges on the cost optimization

due to the ability to support various types of

mobile devices, and operating systems like Android, iOS, and

Windows.

This paper will explain below key points which will be useful for

industry to obtain the right tool to conduct performance testing

and monitoring for different mobile OS and devices also the

typical challenges

o KPIs to be considered for mobile performance testing.

o Different tools for android and IOS mobile testing.

o Setting up cost effective lab for mobile performance

testing.

o Monitoring tools to get the statistics to identify the

tailbacks.

Above best practices can overcome the challenges in Mobile

Performance Testing.

2

INTRODUCTION

Testing of mobile application must be performed according to certain industry standards to ensure its

stability and reliability, so that the applications can be accepted globally by the people using different

mobile operating systems, which helps user friendly and popular.

CHALLENGES IN MOBILE TESTING

Some of the typical challenges faced in any Mobile application are as shown in the diagram

KPI’S FOR MOBILE PERFORMANCE TESTING

Below are the important KPIs to be considered for mobile performance testing.

▪ App crash: The typical crash rate is around 1-2%, but this varies widely depending on the type of

app, its usage, maturity, etc.

▪ API latency: It is best practice to optimize the APIs to meet the turnaround time of 1 second. As

data generally shows that anything over 3-4 seconds total response time and the majority of user

(60% or greater) will abandon the transaction and may even delete your app altogether.

3

▪ App load per period: It is critical metric because you want to make sure that as the load increases,

your application performance doesn’t degrade. Load can be very spiky in nature for some apps,

so you need to know that your app can handle sudden changes in load without slowing down.

▪ Application time running without connection to permanent power (battery) (App + device + OS)

▪ Frequency of connections to server (every 5, 10, 30 minutes)

▪ Number of bytes needed for App data packet to server.

▪ Storage capacity for app data.

KNOW WHEN TO USE EMULATORS OR REAL DEVICE TESTING

While developers typically leverage emulators for testing, traditional QA knows there is no substitute for

real devices. This means both have their place as part of the process. For early testing and distributed

teams, nothing can replace the convenience of emulators. However, they typically don’t replicate many

core capabilities that should be tested. Recent advances with test cloud libraries of online devices replaced

many of the emulator advantages and eliminated the need to purchase and manage your own library of

devices. The bottom line is that both emulators and real device testing should play a key role.

Real Device Testing: Testing on real device allows you to run your mobile applications and checks its

functionality. Real device testing assures you that your application will work smoothly in customer

handsets.

Emulators: Emulator is a software program that allows your mobile to imitate the features of another

computer or mobile software you want them to imitate by installing them to your computer or Mobile.

PERFORMANCE TESTING TOOLS AND APPROACHES

Device Any Where

A revolutionary solution providing instant remote access to more than 2000 real devices in live global

networks, DeviceAnywhere empowers mobile application and content developers to create better, higher-

quality content faster than ever before. DeviceAnyWhere provides access to real mobile handsets that

are each connected to live networks. These devices are accessible remotely over the Internet, allowing

users to interact in real time to ensure all development, porting, testing, and monitoring needs are met.

The DeviceAnywhere service uses actual, physical handsets to provide developers with the same

experiences as having the devices in hand.

4

Test Quest

TestQuest is the leader in automated test solutions for mobile and wireless applications and devices. By

using TestQuest solutions, companies reduce test cycle times, improve test accuracy, collaborate to

accelerate new products to market and ensure a quality end user experience.

Mobile Device Perspective

Keynote Mobile Device Perspective enables you to measure the true end-to-end subscriber experience of

your mobile quality service from actual mobile devices. It allows you to validate service availability,

delivery times, throughput, as well as accuracy of content and services.

Steps for Recording in I-OS device through LR

1) Installation of Hotspot in Vugen Machine

2) Proxy Recording in LR - Protocol for Recording (Mobile Application Protocol or WebApp with Older

recording engine)

3) I-Phone /I-OS device setup

Installation of Hotspot:

Download and Install Connectify Hotspot to vugen machine

Click on “Start Hotspot” button to trigger the HotSpot

5

Proxy Recording in LR –

through Mobile Application Protocol open New Mobile app script in LR 11.52, click on Recording button

& Select “Proxy Recording”

Specify Port in the next window as below (e.g.7777 )

6

Click Finish button to start recording

I-Phone /I-OS device setup

Go To Settings – Wifi - & Select Connectify

Click Join Network, Enter password & Click Join

7

Configure HTTP proxy in the Wifi from OFF to Manual & provide Vugen machine’s Host Name & Port

Details

Install the certificate in mobile device by hitting the URL (http://<Hostnameof VuGenmachine>:

<port>/proxyroot.cer). Then Click Install as shown below.

Sample mobile browser based recording snapshot

8

EMULATORS

Android 1.5 SDK

What is Emulator ?

The Android SDK includes a mobile device emulator — a virtual mobile device that runs on your

computer. The emulator lets you develop and test Android applications without using a physical device.

Configure and Launch Emulator –

• once after downloading the package. Click on SDK (Software Development kit)page to upgrade to

latest version of software.

• Click on AVD.exe to create Android Virtual device

• Give AVD name (by this name the LR tool will recognize the emulator), Device, SD card Size: min

of 512 mb

• Click on start to Open the Device

9

Installing APK file in Emulator

• Go to Command Prompt

• Give the path of the Android Application using Command – ADB install filename.pak

Mapping AVD to LoadRunner:

• Open Vugen, select Mobile Http/Html protocol

• Click on Record Through Emulator

• In the recording wizard browse the path of Emulator and start Recording

Monitoring Tools

Spectra 2

- IMS/TISPAN, VoIP, and legacy/mobile core network test solutions for load/stress testing,

conformance testing, Media/QoS, and network monitoring.

Spectra2SE

- A cost-effective, software-only edition of Spectra2 for your PC or laptop

Spectra2XL

- High-performance chassis that produces extreme load/stress testing capacity

Spectra2XL

- Powerful 2G, 3G and WiMAX Functional and Load tester

K12-G35

- Powerful 2G, 3G and WiMAX Functional and Load tester

K15 Classic

- Powerful 2.5 and 3G Protocol Monitoring Tool

K18

- Monitors UMTS, UTRAN, and WiMAX

NSA

- To help you better optimize your 3G networks, Tektronix brings you the UTRAN Network

and Service Analyzer, a suite of real-time and off-line software applications designed to

troubleshoot and optimize those 3G networks and services supported by the K18 and K15

platform

10

K2Air

- The K2Air air interface probe provides real-time, promiscuous mode, air interface monitoring

capabilities via CPRI and is seamlessly integrated into the Network & Service Analyzer™

(NSA™) to provide vendor and carriers alike with a 360 degree, end-to-end view of LTE

network and services.

Mobile Testing Lab Setup Approach & Usage

Key challenges in Mobile testing are Increasing handset, application functionality compatible on

diversified platform and devices Mobile lab enable us to carry out detailed analysis on various tools,

simulators and develop innovative solution for our existing customers and potential customers

11

Remote Access: There is also opportunity to make the machines in these environment remotely

accessible from onsite and create virtual lab at offshore.

Lab Setup Process

➢ Identification of Lab Requirements

o Infra

▪ Tables

▪ Chairs

▪ Branding

o Network

▪ Cabling

▪ Connectivity

▪ Voice(vnets)

➢ Identification of Lab Location

➢ Approvals

➢ Raising PO & Procurement of Requirements

➢ Establishing Lab setup(infra & network)

➢ Setting up Lab Test Environments

➢ Pilot

➢ Live

Infrastructure & Network team need to get below things

12

Mobile Lab rough sketch

Benefits Of Mobile Lab

13

Suggested Reference

1. http://articles.techrepublic.com.com/5100-10878_11-5207561.html

2. http://www.safecomprogram.gov/NR/rdonlyres/5C74C631-ACF6-433F-B313-

C04D041A5489/0/Look_Future_Wireless_Communications_Beyond3G.pdf

3. http://www.forrester.com/Research/Document/Excerpt/0,7211,42749,00.html

4. http://www.tradingmarkets.com/.site/news/Stock%20News/2344437/

5. http://www.sybase.com/files/White_Papers/IDC_MobilePlatform_WhitePaper_Oct2008.pdf

6. http://gnokii.org/

7. http://www.pressreleasepoint.com/open-source-saas-collaboration-solution-provides-push-

email-and-cloudbased-mobile-sync-15-billion-sm

8. http://mite.keynote.com/product_overview/index.php

9. http://www.ciol.com/Developer/Testing/News-Reports/DeviceAnywhere-integrates-mobile-

testing/2609120397/0/

10. http://www.testquest.com/

11. http://www.testquest.com/objects/Library/appnote_net_perform.pdf

12. http://www.deviceanywhere.com/

13. http://www.mverify.com/resources/resources.htm

14. http://www.mverify.com/technology/mVerify-MTS-PerformanceTesting.pdf