XII Workshop de Teses e Dissertações em Qualidade de Software

1

Mechanisms to support automated testing of mobile

applications

Guilherme de Cleva Farto1,2,3

, André Takeshi Endo1

1Universidade Tecnológica Federal do Paraná (UTFPR)

Avenida Alberto Carazai, 1640 – 86300-000 – Cornélio Procópio, PR – Brasil

2Fundação Educacional do Município de Assis (FEMA)

Avenida Getúlio Vargas, 1200 – 19807-635 – Assis, SP – Brasil

3TOTVS Agroindústria

Rua Prudente de Moraes, 654 – 19806-160 – Assis, SP – Brasil

guilherme.farto@gmail.com, andreendo@utfpr.edu.br

Abstract. Due to the high number and diversity of users, new testing

approaches are necessary to reduce the occurrence of faults and ensure better

quality in mobile applications. The major objective of this project is to propose

mechanisms to support automated testing of mobile applications with an

emphasis on solutions developed for the Android platform. Specifically, we

will investigate the application of the Model-Based Testing (MBT) in order to

validate and improve the reliability of Android applications. The proposed

approach and tool will be evaluated in an industrial configuration with

professional developers of mobile applications.

1. Problem Characterization

Currently, there is a rapid growth in popularity of mobile devices, such as tablets,

smartphones, and e-readers, expanding the variety of applications that are developed with

the support of mobile computing technologies. A 2013 survey on sales of mobile devices

reports that the Android platform had an increase of 127% over the previous year with

approximately 121 million units sold [Gartner 2014]. Thus, Android took the lead, with

62% of systems and development environments for mobile applications, well ahead of

Apple’s iOS and Microsoft’s Windows Phone.

While mobile applications were initially developed for the entertainment industry,

there is a more widespread adoption in critical areas (e.g., financial systems, health care,

and industries) [Muccini et al. 2012]. As a large number and diversity of users as well as

critical systems have benefited from the mobility provided by these applications, the

occurrence of faults can result in human and economic loss. In this context, software testing

has been applied during the development process to minimize the occurrence of faults. The

activity of testing consists of designing test cases, executing the software with these test

cases, and examining the results with the central goal of detecting faults [Harrold 2000].

The mobile application testing provides many challenges to be overcome and has

received special attention from Software Engineering and Mobile Computing community

[Delamaro et al. 2006, Bo et al. 2007, Maji et al. 2010, Hu and Neamtiu 2011, Muccini et

al. 2012, Pathak et al. 2012, Yang et al. 2013, Liu et al. 2014]. Muccini et al. [2012]

WTDQS 2014

2

argue that there is a need for approaches specialized in testing of mobile applications. They

identified characteristics of mobile applications that influence the testing activity, such as

connectivity, limited resources, autonomy, user interface, context awareness, adaptation,

new programming languages and operating systems, diversity of settings, and touch

screens.

The dynamics of mobile applications and their use in critical environments demand

more accurate and repeatable tests. These characteristics can be obtained by the application

of formal approaches and automated tests. Formal testing is characterized by the adoption

of mathematical models to support the testing activity. Examples of formal models are

Finite State Machines (FSMs), Labelled Transition Systems (LTS), and Event Sequence

Graphs (ESGs) [Belli et al. 2006]. According to Hierons et al. [2009], the presence of

formal specifications and designs can lead to more efficient and effective testing. These

models also provide benefits to automation since their syntax and semantics are well-

defined and allow the development of tools to manipulate such models. In this context, this

project aims to investigate mechanisms to support automated tests for mobile applications.

Specifically, we plan the development of an approach and a supporting tool based on

concepts of model-based testing and on the Android platform.

2. Background

Several techniques can be applied during the development process in order to reveal faults

in the software artifacts. One of these techniques proposes the automatic generation of test

cases through a behavioral or structural model, named test model, of the software under

test (SUT); this approach is known as Model-Based Testing (MBT). The MBT process can

be more efficient because the tester can update the model and regenerate the test suite,

avoiding manual and error-prone changes [Utting and Legeard 2006]. The literature of

MBT reports a set of benefits resulting from its appropriate adoption, such as high fault

detection rate, reduced cost and time for testing, supporting of the requirements’ evolution,

and a high level of automation [Grieskamp et al. 2011].

Pieces of research on the MBT process suggest four main steps: (i) modeling, (ii)

test generation, (iii) concretization, and (iv) test execution [Pretschner and Philipps 2004,

El-Far and Whittaker 2001, Bouquet et al. 2006]. In modeling, the tester uses her/his

understanding of the software to design a model focused on testing. It is advisable the use

of requirements as the main source of information in order to maximize the independence

between the model and the SUT [Utting and Legeard 2006]. In test generation, the

algorithm to derive test cases from the model depends on the modeling technique adopted.

According to El-Far e Whittaker [2001], modeling techniques should have properties that

make both less costly test generation and easier automation. This step requires a tool to

receive the test model as input and automatically generate a set of test cases as output. The

generated test cases are abstract and not executable because they are in a different level of

abstraction of the SUT. In concretization, the tester provides means to transform abstract

test cases into ones executable in the SUT. To do so, abstract test cases could be

implemented using adapters [Pretschner and Philipps 2004]. In MBT, an adapter is a

software component that enables the execution of abstract test cases. In test execution, the

concretized test cases are executed in the SUT. The results are analyzed and corrective

actions can be taken if faults are revealed.

Event Sequence Graph (ESG). In this project, the expected behavior of the SUT

is going to be modeled by Event Sequence Graphs (ESGs). The choice of ESG as the test

XII Workshop de Teses e Dissertações em Qualidade de Software

3

modeling technique is backed up by ease of use and successful experience on MBT.

According to Belli et al. [2006], an ESG is a directed graph used to model possible

interactions between the events of the SUT and is formed by nodes that represent events

and by edges that represent valid sequences between these events. The ESG technique can

be learned in a short period, requires little manual work, and is supported by specific tools

[Belli et al. 2006]. Figure 1 illustrates an ESG model for the “cut-copy-paste” procedure.

The brackets represent the beginning and end of event sequences.

Figure 1. ESG for a “cut-copy-paste” procedure – adapted from [Endo 2013].

3. Related Work

The emergence of technologies and platforms for software expands or creates new fields of

study, despite the existing researches on software testing [Muccini et al. 2012]. Thus, the

mobile applications introduce new testing challenges that must be overcome and, as a

consequence, have been investigated by researchers of software engineering and mobility.

These studies can be divided into two lines. In the first line, traditional testing techniques

have been adapted to mobile applications. Delamaro et al. [2006] describe a strategy to

support structural testing of mobile applications and enable test execution through

emulators and physical devices in the Java Micro Edition (JME) platform. In functional

testing, Bo et al. [2007] implement a tool, named MobileTest, to automate the black-box

testing from an event-based approach to simplify and improve the design of test cases. Its

functionalities include the record of sensitive events, a schedule mechanism for regression

testing, and adapters for future devices to be added.

The second line investigates faults characteristic of mobile applications and, based

on them, new testing strategies are proposed. Maji et al. [2010] evaluate the reported

failures in Symbian and Android platforms resulting in: a detailed analysis of faults found,

a characterization of corrections made, and a comparison between the two operating

systems. Neamtiu and Hu [2011] describe an approach to test Android applications,

emphasizing the user interface’s faults. Random testing, instrumentation of virtual

machine, and log analysis were employed. Pathak et al. [2012] investigate software faults

related to excessive energy consumption on smartphones running Android and provide an

automatic solution to detect these problems using a data flow analysis algorithm. Yang et

al. [2013] propose a testing technique to identify and quantify faults related to excessive

waiting times for certain events in Android applications. For this, the authors rely on the

artificial insertion of delay instructions in typical problematic operations. Finally, Liu et al.

[2014] characterize a set of performance faults commonly identified in Android mobile

applications. The paper also presents a source code analyzer to detect the identified

performance fault patterns in Android applications.

WTDQS 2014

4

Notice that several researchers have investigated techniques for mobile application

testing, highlighting the most notorious adoption of the Google Android platform. So far,

few efforts have been spent to evaluate mechanisms for test automation in mobile

applications. Moreover, most of the papers are academic and there is a lack of research on

applying automated testing in industrial configurations. As mobile applications pose

challenges to software testing, many topics can be further explored.

4. Project Planning and its Current State

This section shows the activities planned to achieve the research project’s goal; its current

state is also presented. The activities are briefly described as follows.

1. Study of the Android platform: this activity aims to study the Android operating

system and how applications are developed for the given platform. In particular, we will

emphasize the tool named Android Studio. Android Studio is an integrated development

environment that provides mechanisms to design, code, debug and test Android

applications using emulators and different devices. Existing open source applications will

also be studied and collected as examples or case studies for future evaluations.

2. Study of the mobile application testing: this activity aims to understand the

challenges and limitations faced by testers during the development of mobile applications.

Another point is to identify tools that automate the testing of mobile applications,

specifically for Android. A starting point is the testing capabilities already provided by

Android Studio. Another interesting tool is Robotium [2014], which is a framework for

automating functional GUI testing in Android applications. Native capabilities for Android

test automation as Instrumentation and MonkeyRunner may also be studied [Android

Testing Fundamentals 2014]. Finally, a detailed literature review about mobile application

testing will be performed.

3. Conduction of an exploratory study: based on the elicited knowledge in previous

activities, we will conduct a first exploratory study to investigate the application of MBT in

mobile applications for the Android platform. This study aims to identify tools, lessons

learned, and limitations to be considered in the next activities.

4. Proposal of a testing approach: a model-based testing approach will be proposed to

verify mobile applications, taking into account the results obtained in previous activities.

The purposes of the approach are to maximize the fault detection rate, minimize the testing

rework, and reduce the test execution time. It is expected to maximize the fault detection

rate because it becomes possible to create and reuse test models to evaluate the SUT on

different configurations of mobile devices. The testing rework will be minimized because

the models can be updated and the test suite regenerated.

5. Development of supporting tools: this activity aims to develop supporting tools for the

proposed approach. Existing tools can also be modified to integrate with the developed

tool. A possible candidate would be the Astah tool; plug-ins could be developed to simplify

the modeling and test generation steps.

6. Case studies in an industrial setting: the evaluation of software testing approaches

basically involves two aspects: cost and effectiveness. We should consider the most

appropriate metrics to evaluate these aspects, e.g., the time elapsed to design and

implement the tests. The guidelines proposed in the literature [Wohlin et al. 2000,

Kitchenham et al. 2002] will guide the planning, execution, and analysis of experimental

XII Workshop de Teses e Dissertações em Qualidade de Software

5

studies performed. Initially, we plan to evaluate our proposal by conducting case studies in

industrial settings with real-world applications and experienced professionals.

7. Elaboration of scientific papers: this activity aims to report the findings of this

master’s project as scientific papers that will be submitted to journals and conferences of

the software engineering and mobility areas.

Current State of the Project. So far, we have worked on Activities 1, 2, and 3, described

previously. A summary of Activity 2 is shown in Section 3. As Activities 1 and 2 involve

literature review, future updates along the project are still necessary. Activity 3 was also

performed. In summary, we evaluated the adoption of MBT concepts and modeling with

ESG in the testing of Android applications. The preliminary results give evidences of the

feasibility of TBM to verify mobility solutions with Android, observing automatic

generation and execution of test cases, ability to detect faults, and reduced time for

regression tests. This study and obtained results have been compiled in a submitted paper.

Currently, the master’s candidate is working on Activities 4 and 5. Particularly, the result

of Activities 1-4 is planned to be presented as a master’s proposal in December 2014.

5. Expected Results

This section presents the results and benefits expected from this master’s project. The

desired results are described as follows: (i) Mechanisms to support automated testing of

mobile applications on Android: this result will be an approach based on MBT to verify

mobile applications developed on Android platform. (ii) Supporting tools: testing tools

will be developed and/or adapted to support the proposed approach. Furthermore, training

materials will be prepared to use the tools. (iii) Experimental evaluation: case studies will

be conducted in order to evaluate the applicability of the proposed approach, providing

evidence on cost and effectiveness, relevant to industrial adoption. (iv) Technology

transfer: we expect to transfer the technology developed to software companies that

develop Android applications.

We expect that the results bring the following benefits: (i) Strengthen the software

engineering area in the master’s program, encouraging the transfer of technology to

industry and disseminating the acquired knowledge through papers; (ii) collaboration with

other research centers; and (iii) cooperation with technology incubators to implement

software testing best practices in incubated companies that develop mobile applications.

References

Android Testing Fundamentals (2014) “Android Testing Framework”, available at

http://developer.android.com/tools/testing/testing_android.html, June.

Belli, F., Budnik, C. J. and White, L. (2006) “Event-based modelling, analysis and testing of

user interactions: approach and case study”, Software Testing, Verification & Reliability, v.

16, n. 1, pages 3–32.

Bo, J., Xiang, L. and Xiaopeng, G. (2007) “MobileTest: A Tool Supporting Automatic Black

Box Test for Software on Smart Mobile Devices”, In: Proc. of the Second Int. Workshop on

Automation of Software Test (AST).

Bouquet, F., Debricon, S., Legeard, B. and Nicolet, J. B. (2006) “Extending the unified

process with model-based testing”, In: 3rd Int. Workshop on Model Development,

Validation and Verification (MoDeVa), Genova, Italy, pages 2-15.

WTDQS 2014

6

Delamaro, M. E., Vincenzi, A. M. R. and Maldonado J. C. (2006) “A strategy to perform

coverage testing of mobile applications”, In: Proc. of the 2006 Int. Workshop on

Automation of Software Test (AST), pages 118–124.

El-Far, I. K. and Whittaker, J. A. (2001) “Model-based software testing”, In: Encyclopedia on

Software Engineering, Wiley, pages 825-837.

Endo, A. T. (2013) “Model-based testing of service oriented applications”, PhD dissertation,

ICMC/USP, São Carlos, SP, available at

http://www.teses.usp.br/teses/disponiveis/55/55134/tde-20062013-140259.

Gartner, Inc. Press release (2014), available at: http://www.gartner.com/newsroom/id/2674215.

Grieskamp, W., Kicillof, N., Stobie, K. and Braberman, V. A. (2011) “Model-based quality

assurance of protocol documentation: tools and methodology”, Software Testing,

Verification and Reliability, v. 21, n. 1, pages 55-71.

Harrold, M. J. (2000) “Testing: A Roadmap”, In: Proceedings of the Conference on The Future

of Software Engineering (ICSE), pages 61-72.

Hierons, R. M., Bogdanov, K., Bowen, J. P., Cleaveland, R., Derrick, J., Dick, J., Gheorghe,

M., Harman, M., Kapoor, K., Krause, P., Lüttgen, G., Simons, A. J. H., Vilkomr, S.,

Woodward, M. R. and Zedan, H. (2009) “Using formal specifications to support testing”,

ACM Computing Surveys (CSUR), v. 41, n. 2, pages 1-76.

Hu, C. and Neamtiu, I. (2011) “Automating GUI Testing for Android Applications”, In: Proc.

of the 6th Int. Workshop on Automation of Software Test (AST), pages 77–83.

Kitchenham, B. A., Pfleeger, S. L., Pickard, L. M., Jones, P. W., Hoaglin, D. C., Emam, K. E.

and Rosenberg, J. (2002) “Preliminary guidelines for empirical research in software

engineering”, IEEE Transactions on Software Engineering, v.28, 721–734.

Liu, Y., Xu, C. and Cheung, S.C. (2014) “Characterizing and Detecting Performance Bugs for

Smartphone Applications”, In: The 36th International Conference on Software Engineering

(ICSE), Hyderabad, India.

Maji, A. K., Hao, K., Sultana, S. and Bagchi, S. (2010) “Characterizing Failures in Mobile

OSes: A Case Study with Android and Symbian”, In: The International Symposium on

Software Reliability Engineering (ISSRE), pages 249–258.

Muccini, H., Di Francesco, A. and Esposito, P. (2012) “Software testing of mobile

applications: Challenges and future research directions”, In: The 7th International

Workshop on Automation of Software Test (AST), IEEE.

Pathak, A., Jindal, A., Hu, Y. C. and Midkiff, S.P. (2012) “What is keeping my phone awake?

Characterizing and detecting no-sleep energy bugs in smartphone apps”, In: The Int’l Conf.

Mobile Systems, App’s, and Services (MobiSys), pages 267-280.

Pretschner, A. and Philipps, J. (2004) “Methodological issues in model-based testing”, In:

Model-Based Testing of Reactive Systems, LNCS, pages 281-291.

Robotium (2014) “Robotium – The world’s leading Android test automation framework”,

available at https://code.google.com/p/robotium/, April.

Utting, M. and Legeard, B. (2006) “Practical model-based testing: A tools approach”, San

Francisco, CA, USA: Morgan Kaufmann Publishers Inc.

Wohlin, C., Runeson, P., Hoest, M., Ohlsson, M. C., Regnell, B. and Wesslon, A. (2000)

“Experimentation in Software Engineering: an Introduction”, Kluwer.

Yang, S., Yan, D. and Routev, A. (2013) “Testing for poor responsiveness in Android

applications”, In: Proc. Int’l Workshop on the Engineering of Mobile-Enabled Systems

(MOBS), pages 10-20.