Using Models to Automate Testing:

Experiences & Challenges

Dr. Muhammad Uzair KhanSenior Member IEEE

Quest Lab, FAST-NU Islamabad

Uzair.khan@questlab.pk

Pakistan Software Quality Conference, PSQC’17

April 08, 2017

About me!

• Assistant Professor @ FAST NU

– PhD from University of Nice, France and INRIA in 2011

• Head QUEST Lab

• Founding Member, Pakistan Software Testing Board

• Treasurer and Secretary IEEE Islamabad Section

• Over 20 high quality international research publications

• Over 14 years of industry and academia experience

– Expertise: Software Testing , UML, Empirical Software Engineering, Model

refactoring, Aspect Oriented Engineering

• Principle Investigator of ICT R&D Funded project – AGCT 2017 (26

Million PKR)

• Co-Principle Investigator of ICT R&D Funded project – MBT-Toolset

(15 Million PKR)

2

Software Quality Engineering and Testing

(QUEST) Lab

• Quest Lab, Est. June 2013, www.questlab.pk

• Industry-driven research on software quality engineering: testing,

model-driven development

• Targeting real industrial problems

• Several national and international industry partners

• International academic collaborations:

– Interdisciplinary Centre for Security, Reliability and Trust (SnT)

Luxembourg

– Simula Research Laboratory, Norway

• Several local and foreign funded projects

– Worth over Rs. 50 Million

• Research being published at high quality international venues

• Have offered 12+ fully funded MS scholarships

3

Applied Research in Software Engineering

• Useful research must have impact: Research-driven

innovation

• Close industrial involvement

– Conscious effort to understand the problems and practice of

industry

– Focus on pain points in industry

– Realistic Evaluation

– Long term industrial collaboration => impact

• Impact on Academia

– Should contribute to the learning of students

• Introduction of new courses/specializations

– Producing high quality graduates

– Producing high quality publications

4

Our Industry Collaboration Model

• Unlike other stereotyped academic research, at QUEST lab

research is driven by practical problems faced by

industry

• Strong industrial collaborations to develop and evaluate

candidate solutions

5

Software Testing

• Testing is the process of executing a program with the intent of finding

faults

• The challenge is to identify scenarios that trigger the faulty functionality of

software

Major steps of Testing

• Identifying Test Paths/Sequences

• Identifying Test Data

• Test Execution

• Test Evaluation

7

new Calculator(..).add(…)

“Windows”, 2, 2

4

Identifying Test Sequences

• What sequences of function calls to test?

– Out of billions of possible function call sequences for a class, which

call sequences to select?

– Important Decision

– A number of heuristics are typically used, e.g.,

• important function sequences based on application logic

• data-flow sequences

• checking the use-case flows

• checking the misuse-cases

Identifying Test Data

• What test data to execute?

– Consider a program taking two integers as input

• Possible number of input combinations: 264 * 264

• Which combinations to test?

• Decision is typically based on test adequacy criteria

– To maximize code’s control flow coverage

– To maximize code’s data flow coverage

– Test based on equivalence classes and boundaries

Computing Test Oracle

• How to obtain a test oracle (expected results)?

– Computing test oracle is challenging

– Simpler at unit testing level

– Expected outputs are based on test data

– Well documented specifications can help

Test Execution

• How to execute test cases?

• Manual execution

– laborious

• A number of frameworks support automated test execution

• Record and replay tools (e.g., for web applications)

• xUnit framework tools

Automated Test Execution

• How to execute test cases without manual intervention?

• A number of frameworks support automated test execution

• Record and replay tools (e.g., for web applications)

• xUnit framework tools

• Industry practice

– Large part of local industry struggling to achieve automated test

execution..

12

Automated Software Testing – Industry Practice

• Problems?

– Human intensive effort,

– Not scalable, not repeatable, not maintainable

– Limited confidence in testing, How to evaluate effectiveness of

testing?

• Typical solution:

– Testing is driven by availability and capability of the

resource

13

Why not automated other steps?

• Useful Test data, sequences, and oracle are

application specific

– Cannot be generalized

– Need to capture specification of the system

under test in a machine analyzable form

– Domain/application specific methodologies and

strategies for testing are required

• One widely accepted solution: Model-based

testing

– Our expertise at QUEST lab

14

Test Path/Sequence

Test Data

Test Oracle

Model-Based Testing

• Model-based Testing (MBT) uses models of the system to

completely automate the testing process

– The models typically describe some aspects of system under test

• MBT allows automated testing solutions for complex industrial

software systems

– Early

– Systematic

– Scalable

15

Test Case Generation and Execution

16

Test cases

SW Model

SW Code

Test cases

CompareOracle

Expected

Results

Results

What modeling

strategy?

What test data & sequence

generation strategy?

What oracle strategy?

Tailored Modeling

Methodology

Application specific/

control and data flow

analysis

Application

domain specific

Vending Machine Control Software -

State Machine

off

power-on/

light-on; m:=0

power-off/

light-off

not

emptyempty

inc/m:=1

dec[m=1]/m:=0

inc/m:=m+1

dec[m>1]/m:=m-1

money

busyidle

coffee

on

coffee[m>0]/start

after(5)/stop

coffee[m>0]/start;dec

17

State coverage

18

off

power-on/

light-on; m:=0

power-off/

light-off

not

emptyempty

inc/m:=1

dec[m=1]/m:=0

inc/m:=m+1

dec[m>1]/m:=m-1

money

busyidle

coffee

on

coffee[m>0]/start

after(5)/stop

coffee[m>0]/start;dec

Applied Research in MBT

• MBT is not something new!

• Well-accepted by international industry

– IBM, Microsoft, Cisco, Schlumberger, etc

• Next, some of the industrial applications of MBT we at Quest

lab have worked with

19

Automated Software Testing with Models

20

I - Video Game Testing

• Game Industry worth around $100 billion*

– Platform games a famous type of games

• Games are highly interactive, state-based, and are typically real-

time software

• Highly competitive market

21

*https://newzoo.com/insights/articles/

Video Game Testing – Current Practice

• Manual Testing – A common approach of testing games in

industry

• Testers play various game scenarios manually

– Preferably over long periods of time

– Observing various game scenarios

– Various game play rules need to be taken into account for testing

• Focus of functional testing:

– Functional correctness of games

– Identifying crash points

22

Video Game Testing - Problem

• Problem

– Lack of a scalable strategy to test functionality of platform games

• Issues with Current Practices

– Labor intensive: Manual testing by playing games for hours

– Non-systematic: Highly dependent on individual skills

– Boredom effect: Task gets non-rewarding over time

– Non-repeatable: playing the same game scenarios for every

change in the game

– Error prone: A number of interesting scenarios are missed

23

Video Game Testing – Automated Approach

• We developed an automated platform game testing approach

– Focuses on testing the game based on avatar’s behavior by

simulating the inputs required for playing games.

– Testers need to model the game play from users perspective

• We developed a detailed modeling methodology

– A UML profile for modeling platform games for testing

– A set of guidelines for modeling game concepts and behavior for

testing purposes

24

“Automated Model-based Testing Approach for Testing Games”, published in ACM/IEEE 18th

International Conference on Model Driven Engineering Languages and Systems (MODELS’15),

2015, Canada.

Modeling Mario Brothers

25

Video Game Testing – Models for Mario Bros

26

Fig. 3. Mario’s avatar state machine

Fig. 4. Run forward state machine.

V. CASE STUDY

To demonstrate the application of our proposed game

testing methodology, we applied it on two case studies. The

first case study is of an open source variant of a famous

platform game, Super Mario Brothers [9]. The second case

study is of an endless runner platform game being developed

by our local industry partner.

Mario game was first introduced in 1983 for Nintendo

console. Since then, a large number of versions of the game

have been introduced. In Mario brothers’ game, the avatar

moves through various obstacles and collects coins. While

moving, Mario dies when it hits a ghost or falls of the edge.

.

Video Game Testing

• Test Sequences (the user game play events) are automatically

obtained by traversing the models as a graph

• Test Data generated by solving constraints on guards

• Test Execution is automated by developing a simulator that can

execute user events

• Test Oracle

– System-generated events

– State transitions in the state machines (due to a change, signal, or

a time event) that are triggered as a result of the user-generated

events

– System crashes

27

Fig. 3. Mario’s avatar state machine

Fig. 4. Run forward state machine.

V. CASE STUDY

To demonstrate the application of our proposed game

testing methodology, we applied it on two case studies. The

first case study is of an open source variant of a famous

platform game, Super Mario Brothers [9]. The second case

study is of an endless runner platform game being developed

by our local industry partner.

Mario game was first introduced in 1983 for Nintendo

console. Since then, a large number of versions of the game

have been introduced. In Mario brothers’ game, the avatar

moves through various obstacles and collects coins. While

moving, Mario dies when it hits a ghost or falls of the edge.

.

A Sample Bug

28

II - Testing Real time Embedded Software

• Problem:

– RTES execute in complex environments

– A large number of possible environmental scenarios

• Practice:

– Manually writing simulators for SiL testing

– Manually system level test cases

29

Marine Seismic Acquisition SystemBottle Recycling Machine

Testing RTES - Solution

• Independent

– Black-box

• Behavior driven by

environment

– Environment model

• One model for

– Environment simulator

– Test cases and oracles

• Notation: Class Diagrams,

State machines, OCL

30

Environment

Simulator

Test cases

Environment Models

Test oracle

* *Environment Modeling and Simulation for Automated Testing of Soft Real-Time Embedded

Software”, in Software & Systems Modeling Journal (Sosym), 2015

Domain Model - Example

Behavioral Model - Example

32

Automated RTES Testing

• Test Sequence

– Search algorithms guide the selection of test sequences that take

the RTES closer to error states

• Test Data

– Test data generated based on search using constraint solvers

• Test Oracle

– Error states (states that should never happen) were added in the

environment state machine

• Test Execution

– Simulator is automatically generated from the environment models

33

III - Mobile Application Generation & Testing

• Problem

– Mobile applications need to support a large number of platforms

– Very tedious task to develop and maintain numerous versions

– Manually evaluating non-functional properties across various

platform takes significant effort

34

Mobile Application Generation & Testing

• Solution:

– Capture the core business logic with models

• A detailed modeling methodology specific to mobile applications

– UML Profile for modeling Mobile Applications

– UML Class Diagrams and State-machines

– Action Language: ALF

• Native application code generated for each platform

• Performance related properties are modeled and their

conformance automatically tested

35

*A product-line model-driven engineering approach for generating feature-based mobile applications, Journal of Systems

and Software, vol. 123, January 2017

Mobile Application Generation & Testing

Mobile Application Generation & Testing

Android Application Code Windows Application Code

Mobile Application – Modeling Performance

38

Mobile Application – Modeling Performance

39

Mobile Application – Performance Testing

• Test Sequences

– Search-based approaches and fuzzy logic are used to identify test

sequence that stress the SUT to violate the performance properties

• Test Data

– Test data is generated to support the generated test sequences

• Test Oracle

– The performance properties modeled are considered as oracle

• Test Execution

– Tests are automatically executed across various platforms for the

versions of the mobile applications

40

IV - Automotive System Testing

* “Automatic Generation of System Test Cases from Use Case Specifications”, in International Symposium

on Software Testing and Analysis (ISSTA’ 2015), USA

• A safety-critical automotive

software developed

• Monitors a car seat via an electrical

field sensor to classify the occupant

– disables the airbag for children and

unoccupied seats

– ensures airbag deployment for

adults

• Need to support ISO-26262

automotive standard for functional

safety

– All requirements must be tested

with test cases

Automotive System Testing

• Solution: Use case modeling with RUCM and OCL constraints

• RUCM: Restricted Use case Modeling Language

– Allows to specify scenario in a structured language

• Natural Language Processing to generate test cases

• OCL Constraints solving to generate test data

RUCM Use case

Derived OCL Constraints

Automotive System Testing - Automation

• Test Sequences

– Various scenarios of the use cases are automatically identified to

form the test sequence

• Test Data

– The OCL constraints are automatically solved to generate

scenarios to test the system

• Test Oracle

– The system actions of the use case are treated as oracle

• Test Execution

– Test scripts are automatically generated and executed on pre-

existing simulators

45

When to Automate?

• A large number of testing scenarios are automatable

– ROI?

– Large maintenance cycle

– Regression Testing

– Data driven Testing, e.g., web application

– Load and Performance Testing

– When the effort of manual testing is huge and redundant

– Systematic testing is a need

– Systems with complex nature and large environment variations

– Critical systems

• Learning curve should not hinder automation

– Need to start from somewhere

46

What to Automate?

• Everything that can be automated, should be automated?

• A large number of testing tools support automated test

execution

– xUnit framework tools: JUnit, GoogleTest, nUnit, etc

– Record & replay tools: Selenium, Test Complete, QTP, Watir, etc

– Should be used to reduce rework and support systematic test

execution and management

• Load testing is only practical when automated: LoadRunner,

JMeter

• Repeated and labor intensive tasks should be automated, e.g.,

manual white-box testing

• Useful Test data, sequences, and oracle are application specific

– Given the ROI and nature of the project, investment should be

made in exploring such strategies

47

Open Challenges

• How to automate usability testing?

• Test sequence, data and oracle generation approaches are

tailored to the application needs

– Tool support for Test Data Generation is improving

• EvoSuite, SBOS OCL Solver

– Test oracle in most cases is the difficult part

• Model-based testing approaches are promising

– Conformiq, Smartesting

– Still a huge potential for improvement

• Regression Testing approaches have room for a lot of

improvement

48

Future of Software Testing

• Current Software Testing Industry: Over $60 Billion

– 10% growth every year - a few of the industries having a double

figure growth rate for last decade

– Software Testing Job ranked among the top 10 In-demand jobs of

2016 (fortune.com)

• The role of software tester is changing

– More development, Developers are Testers, Test Driven

Development and DevOps, demonstrable skillset to be rewarded

• Cloud and IoT increasing the importance of software testing

• Tools gaining momentum

• Automated testing for tedious and repetitive tasks - A must

• A lot of research prospective!

49

Thanks!

50

Dr. Muhammd Uzair Khan

Quest Lab, FAST house, Islamabad

Uzair.khan@questkab.pk