Mobile Software Testing Challenges

Busra DenizSWE 550, Fall 2015

2014719135

Mobile World Statistics

Mobile phone users worldwide (in billions)

Mobile World Statistics

Number of mobile app downloads worldwide (in millions)

Mobile World Statistics

Worldwide mobile app revenues (in billion U.S. dollars)

Mobile in critical domains

Payment

Healthm-government

3 Questions

❖ RQ1 : Are mobile applications so different from traditional ones, so to require different and specialized new testing techniques?

3 Questions

❖ RQ1 : Are mobile applications so different from traditional ones, so to require different and specialized new testing techniques?

❖ RQ2 : What are the new challenges on testing mobile applications ?

3 Questions

❖ RQ1 : Are mobile applications so different from traditional ones, so to require different and specialized new testing techniques?

❖ RQ2 : What are the new challenges on testing mobile applications ?

❖ RQ3 : Which is the role automation may play in testing mobile applications ?

What is a mobile application ? ❖ A mobile application is vaguely defined as an application running on

mobile devices and/or taking in input contextual information. ❖ Mobility and context-awareness

➢ In mobile computing, an application is considered to be mobile if it runs on an electronic device that may move (mp3 readers, digital camera, mobile phones)■ limites resources, security, performance, reliability and finite energy source

➢ In context-aware computing, an application is aware of computing environment in which it runs, and adapts/reacts according to its computing, user, physical or time context

■ location, time, nearby objects, orientation, social contexts.

❖ App4Mobile & MobileApps

Applications

Apps4Mobile

Mob

ileA

pps

Context-a

ware

applic

ations

Mobile Applications

Mobile Application Peculiarities & ImplicationsType of Mobile Applications

PeculiarityImplications on Testing

Mobile Connectivity Reliability, performance, security and functional testing through different networks

Limited Resources Performance and functional monitoring

Autonomy Monitoring for energy consumption

User Interface GUI Testing

Context Awareness Context dependent functional and extra functional testing

Adaptation Adaptation correctness

New programming language New white-box and black-box testing,byte code analysis

New O.S.s Compatibility and O.S testing

Diversity of phones and OS Diversity coverage

Touch Screen Usability, and response to screen touch

Mob

ile A

pplic

atio

ns

Mob

ileA

pps

App

s4M

obile

Mobile Connectivity❖ Peculiarities :

➢ Connectivity is one of the most peculiar and critical characteristic of mobile➢ Different types of networks

■ Wi-Fi , 3G , LTE , Restrictive networks➢ Connection change between different type of these networks➢ Speed, security and reliability

➢ Interoperability : bluetooth

Mobile Connectivity❖ Peculiarities :

➢ Connectivity is one of the most peculiar and critical characteristic of mobile➢ Different types of networks

■ Wi-Fi , 3G , LTE , Restrictive networks➢ Connection change between different type of these networks➢ Speed, security and reliability

➢ Interoperability : bluetooth

❖ Implications on Testing : ➢ Reliability, performance, security and correct functioning strongly relies on connectivity➢ Functional and extra functional testing has to be performed in

■ Different connection type■ Different connectivity scenarios■ Connectivity changes scenarios with different connection types

➢ Not ignorable rate about connectivity issues

Mobile Connectivity - Real World Example

Limited Resources❖ Peculiarities :

➢ Mobile device resources are far away those on laptop or desktop computers

Limited Resources❖ Peculiarities :

➢ Mobile device resources are far away those on laptop or desktop computers

❖ Implications on Testing : ➢ Resource usage has to be continuously monitored and avoid performance degradation

➢ Actions have to be taken in case of resource shortage

Limited Resources- Real World Example

Autonomy❖ Peculiarities :

➢ Different mobile application may require very different energy consumption

➢ Reduced autonomy can be an issue

➢ Autonomy improvements can be a business advantage

Model Standby time Internet use

iPhone 6s Plus Up to 16 days Up to 12 hours on 3G,LTE, Wi-Fi

Autonomy❖ Peculiarities :

➢ Different mobile application may require very different energy consumption

➢ Reduced autonomy can be an issue

➢ Autonomy improvements can be a business advantage

❖ Implications on testing : ➢ Energy consumption of mobile applications have to be evaluated through testing or

continuous monitoring

Model Standby time Internet use

iPhone 6s Plus Up to 16 days Up to 12 hours on 3G,LTE, Wi-Fi

Autonomy - Real World Example

User Interface❖ Peculiarities :

➢ Developers have to follow some strictly defined user interface guidelines for producing mobile applications

➢ Different screen resolutions and dimensions

➢ Changing user interface guidelines

User Interface❖ Peculiarities :

➢ Developers have to follow some strictly defined user interface guidelines for producing mobile applications

➢ Different screen resolutions and dimensions

➢ Changing user interface guidelines

❖ Implications on testing : ➢ GUI testing

➢ Need to test same application on different mobile devices

User Interface - Real World Example

Context-Awareness❖ Peculiarities :

➢ MobileApps rely on sensed data provided by context providers

➢ Deal with huge amount of inputs that vary depending on the environment and/or user actions

➢ Brightness, temperature, altitude, noise level, type of connectivity, bandwidth, neighboring

devices, location, etc...

Context-Awareness❖ Peculiarities :

➢ MobileApps rely on sensed data provided by context providers

➢ Deal with huge amount of inputs that vary depending on the environment and/or user actions

➢ Brightness, temperature, altitude, noise level, type of connectivity, bandwidth, neighboring

devices, location, etc...

❖ Implications on testing : ➢ Testing whether application is going to correctly work

on any environment and under any input is a challenge!➢ May lead to combinatorial explosion

➢ Context specific test selection techniques and coverage have to be produced

➢ Not ignorable rate associated to contextual input

Context-Awareness - Real World Example

● Bluetooth problem reported by Customer side

● Problem is reproduced with only car bluetooth kit

● Working in a car to reproduce issue and fix it !

New Programming Language❖ Peculiarities :

➢ Programming languages of mobile applications have been designed for supporting mobility, managing resource consumption and handling new GUIs

➢ iOS , uses UIKit framework and dynamic language

➢ Android replace java.awt and java.swing APIs with new libraries, uses Dalvik opcode compiler optimized for mobile devices

➢ New language : Swift

New Programming Language❖ Peculiarities :

➢ Programming languages of mobile applications have been designed for supporting mobility, managing resource consumption and handling new GUIs

➢ iOS , uses UIKit framework and dynamic language

➢ Android replace java.awt and java.swing APIs with new libraries, uses Dalvik opcode compiler optimized for mobile devices

➢ New language : Swift

❖ Implications on Testing : ➢ Traditional structural test techniques need to be

revised in order to be applied to new mobile languages

➢ Bytecode analysis tools required to be updated

New(and rapidly evolving) Mobile OS❖ Peculiarities :

➢ Mobile applications run on top of new operation systems that are still only partially reliable

➢ Many are examples of applications failing due to operation systems problems ➢ New versions of mobile operating systems are frequently released

➢ New versions are not backward compatible with previous version

New(and rapidly evolving) Mobile OS❖ Peculiarities :

➢ Mobile applications run on top of new operation systems that are still only partially reliable

➢ Many are examples of applications failing due to operation systems problems ➢ New versions of mobile operating systems are frequently released

➢ New versions are not backward compatible with previous version

❖ Implications on testing : ➢ Failures can not be ascribable to mobile applications➢ Mobile applications have to tested on all versions of mobile operating system

New(and rapidly evolving) Mobile OS

New Mobile OS - Real World Example

● App Transport Security came within iOS 9

● Enforce secure network connections

● Minimum TLS version updated

Diversity ❖ Peculiarities :

➢ Hundreds of different mobile devices on market ■ produced by different vendors■ different software features and hardware components

➢ Specially, sensors are typically calibrated differently so that two different phones running the same application, in the same environment, may compute different outputs

➢ Supported device list : roughly 1228 https://support.google.com/googleplay/answer/1727131?hl=en

Diversity ❖ Peculiarities :

➢ Hundreds of different mobile devices on market ■ produced by different vendors■ different software features and hardware components

➢ Specially, sensors are typically calibrated differently so that two different phones running the same application, in the same environment, may compute different outputs

➢ Supported device list : roughly 1228 https://support.google.com/googleplay/answer/1727131?hl=en

❖ Implications on Testing : ➢ Testing techniques for maximizing the diversity coverage while minimizing the devices

under test have to be devised

Diversity - Real World Example

Diversity - Real World Example

Diversity - Real World Example Default Video resolution : 360 X 480

Exynos hardware which uses in Samsung devices causes problem while doing hardware video decoding (V8 codecs)

Wrong resolution : 368 X 480

Diversity - Real World Example -2● Highly critical no media

issue reported by customer in Japan

● Only reproducible on Sony Ericsson Xperia SO-02E !

● Device is only available in Japan !

● Device rejected due to Turkey’s import policy !!

Touch Screens❖ Peculiarities :

➢ Touch screens are main means for inputting user data into a mobile application➢ The system response time to a touch strongly depends on the device resource utilization ➢ May become slow in certain circumstances like busy processor

➢ Lots of reported issue related with lack of responsiveness of touch screens

Touch Screens❖ Peculiarities :

➢ Touch screens are main means for inputting user data into a mobile application➢ The system response time to a touch strongly depends on the device resource utilization ➢ May become slow in certain circumstances like busy processor

➢ Lots of reported issue related with lack of responsiveness of touch screens

❖ Implications on Testing : ➢ Testing techniques have to be introduced to

test the touch screen functioning under different circumstances and different mobile devices

➢ Resource usage, processor load, memory load, etc..

Touch Screens - Real World Example

Challenges on Testing Mobile Applications❖ Testing process

➢ Test selection➢ Test execution

❖ Testing artifacts ➢ Structural➢ Functional

❖ Testing levels➢ unit test➢ integration test

❖ Type of test➢ Performance & Reliability testing➢ Memory & Energy testing➢ Security testing➢ GUI testing

➢ Product line testing

Testing Process❖ Test Selection

➢ Challenge : Unpredictability and high variability of inputs

➢ Potentials & automation : new testing criteria are required to provide guidelines,rules,

strategies by which mobile test cases are selected so to maximize coverage in case of unpredictable and variable contextual input

➢ SOTA : Contexts that have a higher probability to generate faults are selected but coverage criteria for context-aware inputs seem to be still missing.

❖ Test Execution ➢ Challenge : Contextual information leads to challenge on how to execute test cases

including rich contextual inputs

➢ Potentials & automation : New capture-and-replay automated techniques can be realized for executing contextual input selected during the test selection phase

➢ SOTA : Android Robotium is an open source tool that enables the automated and black

box test execution of mobile application

Testing Artifacts❖ Structural (code base)

➢ Challenge : Mobile app languages add some specific constructs for managing mobility, sensing and energy consumption.

➢ Potentials & automation : new coverage criteria shall be though as a way to consider at best the new mobility , sensing and energy constructs. New bytecode analysis tool

➢ SOTA : The tool JaBUTi/ME is proposed

❖ Functional ➢ Challenge : Mobile application functional testing requires to specify both app and

environment

➢ Potentials & automation : State-based approaches to model different execution modes (e.g low battery, meeting, and flying mode )

➢ SOTA : Android Robotium, MonkeyRunner

Testing Levels❖ Unit Testing

➢ Challenge : no specific challenge seems to apply to unit testing of mobile apps

➢ Potentials & automation : there is the need and potential to realize automated testing

tools supporting and enabling the specification of contextual values and context-based coverage

➢ SOTA : JUnit / XCTest

❖ Integration Testing ➢ Challenge : Inter- and intra-application information flow must be followed

➢ Potentials & automation : Existing integration testing approaches have the potentials to be adapted for mobile applications

➢ SOTA : Mature research on integration testing of mobile applications is still missing today

Type of Testing ❖ Performance & Reliability testing

➢ Challenge : depends on mobile resources, on the device operational mode, connectivity quality and variability

➢ Potentials & automation : new techniques have to explicitly consider characteristics

related to changing context and different devices. Runtime analysis techniques can also

adopted to monitor the resources and connectivity state and prevent performance degradation

➢ SOTA : Berardinelli performance analysis tool

❖ Memory & Energy testing ➢ Challenge : memory leaks, active processes may reduce device battery

➢ Potentials & automation : Metrics for estimating energy and memory consumption could be devised so to automatically predict memory leak and energy outage

➢ SOTA : iOS Instruments tool

Type of Testing ❖ Security testing

➢ Challenge : Security is particularly relevant due to the mobility of the device into networks

with different security levels. A trojan might infact access to personal date, private networks and private contextual information

➢ Potentials & automation : traditional security testing approaches shall be revised so to keep in consideration contextual factors

➢ SOTA : The authors analyze the threats Android applications pose to the security and

privacy of enterprise and propose several approaches for defending enterprises against security risks

Conclusion❖ The need of automation in testing mobile applications is exacerbated by

two orthogonal aspects: ➢ Cost of testing : Automation is certainly among the most important

means for keeping the cost of testing low, while guaranteeing and adequate degree of dependability.

➢ Testing through the layers : current bugs are due to interoperability problems that exist today among the application, application framework, operating system, and hardware (sensoring) layers. The need of testing automation towards all the different layers, and able to clearly separate application-level failures from application framework or OS failures.

References❖ Software Testing of Mobile Applications : Challenges and Future Research

Directions➢ http://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=6228987

❖ Mobile Application Testing - Challenges and solution approaches through

automation

➢ http://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=6496451

❖ statistica.com

➢ http://www.statista.com/topics/1002/mobile-app-usage/

❖ developer.android.com

❖ developer.apple.com