Combinatorial Approach to Testing Mobile Applications: An Experimental Investigation Brandi Amstutz...
-
Upload
joseph-maslen -
Category
Documents
-
view
218 -
download
1
Transcript of Combinatorial Approach to Testing Mobile Applications: An Experimental Investigation Brandi Amstutz...
Combinatorial Approach to Testing Mobile Applications:
An Experimental Investigation
Brandi AmstutzAdvisor: Dr. Sergiy
Vilkomir
This research is supported by the National Science Foundation REU Site Award 1262933
Importance of Research
Mobile testing is a critical step in the development process where errors in an application can be discovered and corrected to increase reliability and functionality.
Traditional software testing strategies can be modified and implemented to solve the challenges associated with mobile device selection.
The amount of devices required to exhaustively represent today’s mobile market is extremely high due to a wide range of variability between available devices.
This investigative research analyzes the efficiency of mobile testing based on a combinatorial approach to device selection.
Introduction
Research Questions Addressed
Which device characteristics have the potential to influence application reliability?
Both hardware and software components have the potential to influence an application’s functionality.
▫ Hardware components considered for this approach include: Mobile device type Screen resolution Hardware manufacturer
▫ Software attributes included: Android OS version RAM memory
Introduction
Research Questions Addressed
How should device selection be executed to ensure an optimal set of devices?
This approach suggests parameter value specification based on user profiledata reflecting the current mobile market. Less popular parameter values can be grouped to reduce the total possible values without compromising proportionality
Introduction
Low11%
Medium23%
High36%
Extra High24.6%
Other (tvdpi and xxhigh)5.9%
Samsung; 55.9%
Sony; 8.1%
HTC; 6.3%
LG; 6.1%
Amazon; 3.6%Xiaomi; 2.5%Google; 2.5%Motorola; 2.4%
Pantech; 1.8%
Huawei; 1.2% Other; 9.5%
Figure 2. Device Manufacturer Distribution
Figure 1. Screen Resolution Distribution Source: Unity 3D Mobile Statistics
Research Questions Addressed
How many devices are sufficient for mobile testing?
Exhaustive testing strategies are limited by resource constraints and contain redundancy between device attributes. This research suggests a set of five devices selected based on device parameterization and combinatorial testing techniques for maximum coverage
Introduction
Device ParameterizationBased on user profile data we suggest to use the following device characteristics and their parameter values:
Materials and Methods: Step One
TypeDevice
ManufacturerAndroid OS
VersionScreen Resolution
RAM Memory
Smartphone HTCHoneycomb (3.0-
3.2)
Lowless than 426 dp x 320
dp
< 768 MB
Tablet MotorolaIce Cream
Sandwich (4.0)Medium
at least 470 dp x 320 dp≥ 768
MB
Samsung Jelly Bean (4.1-4.3)High
at least 640 dp x480 dp
Other OtherExtra High
at least 960 dp x 720 dp
Table 1. Selected Parameter Values
Device Selection Through Each-Choice coverage techniques and consideration of current market statistics we suggest the following two sets of selected devices:
Materials and Methods: Step Two
Based on user profile data the following devices were included due to their high level of popularity: Set A: Galaxy S2 i9100 and Galaxy Tab 2, 7.0Set B: Galaxy S2 i9100 and Galaxy Tab 10.1
All other devices have been added to provide total coverage of device characteristics
Table 2. Selected Device Sets
Application SelectionThree applications with similar functionality were selected for testing. The following applications used provide access to search, book and review travel accommodations ranging from car rental to discount vacation packages:
Materials and Methods: Step Three
Hotwire Travelocity BookIt.com
Device AccessPerfecto Mobile cloud services were used for remote access to devices for testing purposes. Tests were recorded and screen shots were saved for later analysis. http://www.perfectomobile.com/
Remote testing procedure using SaaS cloud-based device access:
Materials and Methods: Step Four
User provides
test input
Input is sent to devices selected by user
Audio & video of live test
execution is
recorded
Test results & video are sent to user for analysis
Device Access
Materials and Methods: Step Four
Figure 3. Perfecto Mobile Dashboard
Additional DevicesFor comparison, two sets of random devices equivalent in size to the Each-Choice sets were selected. The same testing procedures were executed on the ten additional devices:
Materials and Methods: Step Five
Table 3. Random Device Sets
Screenshot AnalysisScreenshots were analyzed from all sets of devices for fault identification
Results
Figure 4 (left) illustrates Fault H5 for the Hotwire application. The expected result is that the number of rooms should update on the main page when a user selects a new number of desired hotel rooms from the drop down menu. The fault occurs when the room quantity is not updated appropriately
Screenshot Analysis
Results
Figure 5 (right) illustrates fault T1 for the Travelocity.com application. The expectation is that the Travelocity menu bar with their gnome icon is displayed at the top of the home screen. The fault occurs when the menu bar is missing from view.
Screenshot Analysis
Results
Expected
Fault
Figure 6 (left) illustrates fault B1 of the Bookit.com application. The expectation is that a menu bar with current query's information and reference number is displayed at the top of the screen. The fault occurs when this menu bar is missing from view.
Fault Detection
Results
Fault detection totals are as follows:
Hotwire: 7 2 device independent 5 device dependent
Travelocity: 7 1 device independent 6 device dependent
Bookit.com: 4 1 device independent 3 device dependent
Fault Detection
Results
In 7 out of 12 cases (or 58%) the effectiveness of our approach was greater than the results of random testing.
In 4 out of 12 cases (or 33%) the effective ness of our approach was equivalent to the results of random testing.
Table 4. Fault Detection
Conclusions
The quantity of application faults detected by the selected set of devices was equal to or greater than the quantity detected by the random set of devices. Additionally, the random set of devices did not yield a fault that the selected set was not able to identify. This suggests that the device attributes selected were suitable based on their expected influence on application reliability.
Combinatorial testing strategies provided sufficient parameter value coverage with a test suite of five devices. This research suggests that this small set of devices is effective for mobile testing while also reasonable enough for resource constraints.
Conclusion
Questions?