The Thousands Shades of Model-Based Testing: Pitfalls, … · 2013. 12. 13. · The Thousands...

50
The Thousands Shades of Model-Based Testing: Pitfalls, Challenges, and Guidelines University of Luxembourg Interdisciplinary Centre for Security, Reliability and Trust Software Verification and Validation Lab (www.svv.lu) October 23rd, 2013 UCAAT, Paris, France Lionel Briand, IEEE Fellow FNR PEARL Chair

Transcript of The Thousands Shades of Model-Based Testing: Pitfalls, … · 2013. 12. 13. · The Thousands...

Page 1: The Thousands Shades of Model-Based Testing: Pitfalls, … · 2013. 12. 13. · The Thousands Shades of Model-Based Testing: Pitfalls, Challenges, and Guidelines University of Luxembourg

The Thousands Shades of Model-Based

Testing: Pitfalls, Challenges, and Guidelines

University of Luxembourg

Interdisciplinary Centre for Security, Reliability and Trust

Software Verification and Validation Lab (www.svv.lu)

October 23rd, 2013

UCAAT, Paris, France

Lionel Briand, IEEE Fellow

FNR PEARL Chair

Page 2: The Thousands Shades of Model-Based Testing: Pitfalls, … · 2013. 12. 13. · The Thousands Shades of Model-Based Testing: Pitfalls, Challenges, and Guidelines University of Luxembourg

Diversity

2

Page 3: The Thousands Shades of Model-Based Testing: Pitfalls, … · 2013. 12. 13. · The Thousands Shades of Model-Based Testing: Pitfalls, Challenges, and Guidelines University of Luxembourg

Objectives

• Report on 20 years of experience in automated

software testing & verification research and

innovation

• Explain why and how model-based testing is in many

situations—though not always—the best solution to

test automation

• Show why and how the tailoring of the MBT process

and technology to context is a key success factor

• Present representative project examples

• Identify challenges and lessons learned

• Guidelines 3

Page 4: The Thousands Shades of Model-Based Testing: Pitfalls, … · 2013. 12. 13. · The Thousands Shades of Model-Based Testing: Pitfalls, Challenges, and Guidelines University of Luxembourg

Testing vs. Verification

• Testing: The process of executing software with the

intent of finding and correcting defects

• Verification: The process of analyzing a

representation or model of the system specification or

design in order to reason about its properties

• Verification takes place in earlier phases than testing:

feasibility of requirements, design decisions …

• Will focus mostly on testing here, but there are many

common challenges, technologies, and lessons

learned

4

Page 5: The Thousands Shades of Model-Based Testing: Pitfalls, … · 2013. 12. 13. · The Thousands Shades of Model-Based Testing: Pitfalls, Challenges, and Guidelines University of Luxembourg

SnT Software Verification and Validation Lab

• SnT centre, Est. 2009: Interdisciplinary,

ICT security-reliability-trust

• 200 scientists and Ph.D. candidates, 20

industry partners

• SVV Lab: Established January 2012,

www.svv.lu

• 20 scientists (Research scientists,

associates, and PhD candidates)

• Industry-relevant research on system

dependability: security, safety, reliability

• Six partners: Cetrel, CTIE, Delphi, SES,

IEE, Hitec …

5

Page 6: The Thousands Shades of Model-Based Testing: Pitfalls, … · 2013. 12. 13. · The Thousands Shades of Model-Based Testing: Pitfalls, Challenges, and Guidelines University of Luxembourg

An Effective, Collaborative Model of Research and Innovation

Basic Research Applied Research

Innovation & Development

• Basic and applied research take place in a rich context

• Basic Research is also driven by problems raised by applied

research

• Main motivation for SnT’s partnership program 6

Page 7: The Thousands Shades of Model-Based Testing: Pitfalls, … · 2013. 12. 13. · The Thousands Shades of Model-Based Testing: Pitfalls, Challenges, and Guidelines University of Luxembourg

Collaboration in Practice

• Research informed by practice

• Well-defined problems in context

• Realistic evaluation

• Long term industrial collaborations

7

Page 8: The Thousands Shades of Model-Based Testing: Pitfalls, … · 2013. 12. 13. · The Thousands Shades of Model-Based Testing: Pitfalls, Challenges, and Guidelines University of Luxembourg

“Model-based”?

• All engineering disciplines rely

on abstraction and therefore

models

• In many cases, it is the only way

to effectively automate testing or

verification => scalability

• Models have many other

purposes: Communication,

support requirements and design

• There are many ways to model

systems and their environment

• In a given context, this choice is

driven by the application domain,

standards and practices,

objectives, and skills

8

Page 9: The Thousands Shades of Model-Based Testing: Pitfalls, … · 2013. 12. 13. · The Thousands Shades of Model-Based Testing: Pitfalls, Challenges, and Guidelines University of Luxembourg

Models in Software Engineering

• Model: An abstract and analyzable description of software

artifacts, created for a purpose

9

Requirements models Architecture models Behavioural

models Test models

• Abstract: Details are omitted. Partial representation. Much

smaller and simpler than the artifact being modeled.

• Analyzable: Leads to task automation

• Standards: UML, SySML, MARTE, BPMN, …

Page 10: The Thousands Shades of Model-Based Testing: Pitfalls, … · 2013. 12. 13. · The Thousands Shades of Model-Based Testing: Pitfalls, Challenges, and Guidelines University of Luxembourg

Test Automation Problem

Decomposition

Oracle Verdict

(correct/incorrect)

SW Under Test Driver

executes

creates

Stub(s)

uses

Outputs

Inputs

10

Page 11: The Thousands Shades of Model-Based Testing: Pitfalls, … · 2013. 12. 13. · The Thousands Shades of Model-Based Testing: Pitfalls, Challenges, and Guidelines University of Luxembourg

Automation Needs

• Test case generation

• Test oracle (verdict) generation

• Test stubs generation

• Test driver generation (test execution)

• Logging and analysis of test results

• Test suite evolution, e.g., requirements changes

11

Page 12: The Thousands Shades of Model-Based Testing: Pitfalls, … · 2013. 12. 13. · The Thousands Shades of Model-Based Testing: Pitfalls, Challenges, and Guidelines University of Luxembourg

Common Testing Technology

• Test case generation

• Test oracle (verdict) generation

• Test stubs generation

• Test driver generation (test execution)

• Logging and analysis of test results

• Test suite evolution, e.g., requirements changes

12

Page 13: The Thousands Shades of Model-Based Testing: Pitfalls, … · 2013. 12. 13. · The Thousands Shades of Model-Based Testing: Pitfalls, Challenges, and Guidelines University of Luxembourg

Consequences

• Test automation not scalable

• Expensive (generation and evolution), though costs

often hidden

• Error-prone

• Not systematic

• Not predictable

13

Page 14: The Thousands Shades of Model-Based Testing: Pitfalls, … · 2013. 12. 13. · The Thousands Shades of Model-Based Testing: Pitfalls, Challenges, and Guidelines University of Luxembourg

14

Mental Model

System or Environment

Models

Test suites & scripts

Test Stubs

Oracles

Test Objectives

Test ResultsTest Analysis

MBT Process

Page 15: The Thousands Shades of Model-Based Testing: Pitfalls, … · 2013. 12. 13. · The Thousands Shades of Model-Based Testing: Pitfalls, Challenges, and Guidelines University of Luxembourg

Testing Driven by Environment Modeling

15

Page 16: The Thousands Shades of Model-Based Testing: Pitfalls, … · 2013. 12. 13. · The Thousands Shades of Model-Based Testing: Pitfalls, Challenges, and Guidelines University of Luxembourg

• Three-year project with two industry partners

– Soft real-time systems: deadlines in order of

hundreds of milliseconds

• Jitter of few milliseconds acceptable

– Automation of test cases and oracle generation,

environment simulation

Tomra – Bottle Recycling Machine

WesternGeco – Marine Seismic

Acquisition System

Context

16

Page 17: The Thousands Shades of Model-Based Testing: Pitfalls, … · 2013. 12. 13. · The Thousands Shades of Model-Based Testing: Pitfalls, Challenges, and Guidelines University of Luxembourg

• Independent

– Black-box

• Behavior driven by

environment

– Environment model

• Test Engineers: Software

engineers

• No use of Matlab/Simulink

• One model for

– Environment simulator

– Test cases and oracles

• UML profile (+ limited use of

MARTE)

Environment

Simulator

Test cases

Environment Models

Test oracle

Environment Modeling and Simulation

17

Page 18: The Thousands Shades of Model-Based Testing: Pitfalls, … · 2013. 12. 13. · The Thousands Shades of Model-Based Testing: Pitfalls, Challenges, and Guidelines University of Luxembourg

Domain Model

18

Page 19: The Thousands Shades of Model-Based Testing: Pitfalls, … · 2013. 12. 13. · The Thousands Shades of Model-Based Testing: Pitfalls, Challenges, and Guidelines University of Luxembourg

Behavior Model

19

Page 20: The Thousands Shades of Model-Based Testing: Pitfalls, … · 2013. 12. 13. · The Thousands Shades of Model-Based Testing: Pitfalls, Challenges, and Guidelines University of Luxembourg

• Test cases are defined by

– Simulation configuration

– Environment configuration

• Environment Configuration

– Number of instances to be created for each component in

the domain model (e.g., the number of sensors)

• Simulator Configuration

– Setting of non-deterministic attribute values

• Bring the system state to an error state by searching for

appropriate values for non-deterministic environment

attributes

• Search metaheuristics to search the test case space

• Test oracle: Environment model error states (state invariants)

Automated Test Case Generation and Oracles

20

Page 21: The Thousands Shades of Model-Based Testing: Pitfalls, … · 2013. 12. 13. · The Thousands Shades of Model-Based Testing: Pitfalls, Challenges, and Guidelines University of Luxembourg

Testing Closed Loop Controllers

21

Page 22: The Thousands Shades of Model-Based Testing: Pitfalls, … · 2013. 12. 13. · The Thousands Shades of Model-Based Testing: Pitfalls, Challenges, and Guidelines University of Luxembourg

Complexity and amount of software used on vehicles’ Electronic Control Units (ECUs) grow rapidly

More functions

Comfort and variety

Safety and reliability

Faster time-to-market

Less fuel consumption

Greenhouse gas emission laws

22

Page 23: The Thousands Shades of Model-Based Testing: Pitfalls, … · 2013. 12. 13. · The Thousands Shades of Model-Based Testing: Pitfalls, Challenges, and Guidelines University of Luxembourg

Three major software development stages in the automotive domain

23

Hardware-in-the-Loop

StageModel-in-the-Loop

Stage

Simulink Modeling

Generic

Functional

Model

MiL Testing

Software-in-the-Loop

Stage

Code Generation

and Integration

Software Running

on ECU

SiL Testing

Software

Release

HiL Testing

Page 24: The Thousands Shades of Model-Based Testing: Pitfalls, … · 2013. 12. 13. · The Thousands Shades of Model-Based Testing: Pitfalls, Challenges, and Guidelines University of Luxembourg

Major Challenges in MiL-SiL-HiL Testing

• Manual test case generation

• Complex functions at MiL, and large and integrated

software/embedded systems at HiL

• Lack of precise requirements and testing objectives

• Hard to interpret the testing results

24

Page 25: The Thousands Shades of Model-Based Testing: Pitfalls, … · 2013. 12. 13. · The Thousands Shades of Model-Based Testing: Pitfalls, Challenges, and Guidelines University of Luxembourg

MiL testing

Requirements

The ultimate goal of MiL testing is to

ensure that individual functions

behave correctly and timely on any

hardware configuration

Individual Functions

25

Page 26: The Thousands Shades of Model-Based Testing: Pitfalls, … · 2013. 12. 13. · The Thousands Shades of Model-Based Testing: Pitfalls, Challenges, and Guidelines University of Luxembourg

A Taxonomy of Automotive Functions

Controlling Computation

State-Based Continuous Transforming Calculating

unit convertors calculating positions,

duty cycles, etc

State machine

controllers Closed-loop

controllers (PID)

Different testing strategies are required for

different types of functions

26

Page 27: The Thousands Shades of Model-Based Testing: Pitfalls, … · 2013. 12. 13. · The Thousands Shades of Model-Based Testing: Pitfalls, Challenges, and Guidelines University of Luxembourg

Controller Plant Model and its Requirements

=<

~= 0>=

time time time

De

sired

Va

lue

& A

ctu

al V

alu

e

Desired Value

Actual Value

(a) (b) (c)Liveness Smoothness Responsiveness

xy

z

v

w

27

Page 28: The Thousands Shades of Model-Based Testing: Pitfalls, … · 2013. 12. 13. · The Thousands Shades of Model-Based Testing: Pitfalls, Challenges, and Guidelines University of Luxembourg

MiL-Testing of Continuous Controllers

Exploration+Controller-

plant model

Objective

Functions

Overview

Diagram

Test

Scenarios

List of

RegionsLocal Search

Domain

Expert

time

Desired Value

Actual Value

0 1 2

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1.0

Initial Desired

Final Desired

28

Page 29: The Thousands Shades of Model-Based Testing: Pitfalls, … · 2013. 12. 13. · The Thousands Shades of Model-Based Testing: Pitfalls, Challenges, and Guidelines University of Luxembourg

29

Search Strategy

• Search:

• Inputs: Initial and desired values, configuration parameters

• Example search technique: (1+1) EA (Evolutionary Algorithm)

• Search Objective:

• Find worst case scenarios for liveness, smoothness,

responsiveness -> objective functions

• For each scenario -> simulation

• Result:

• worst case scenarios or values to the input variables that are

more likely to break the requirement at MiL level

• stress test cases based on actual hardware (HiL)

29

Page 30: The Thousands Shades of Model-Based Testing: Pitfalls, … · 2013. 12. 13. · The Thousands Shades of Model-Based Testing: Pitfalls, Challenges, and Guidelines University of Luxembourg

Random Search vs. (1+1)EA

Example with Responsiveness Analysis

Random (1+1) EA

30

Responsiveness Responsiveness

Iterations Iterations

Page 31: The Thousands Shades of Model-Based Testing: Pitfalls, … · 2013. 12. 13. · The Thousands Shades of Model-Based Testing: Pitfalls, Challenges, and Guidelines University of Luxembourg

• We found much worse scenarios during MiL testing than our

partner had found so far

• They are running them at the HiL level, where testing is much

more expensive: MiL results -> test selection for HiL

• But further research is needed:

– To deal with the many configuration parameters

– To dynamically adjust search algorithms in different

subregions

Conclusions

i.e., 31s. Hence, the horizontal axis of the diagrams in Figure 8 shows the number of

iterations instead of thecomputation time. In addition, westart both random search and

(1+1) EA from the same initial point, i.e., the worst case from the exploration step.

Overall in all the regions, (1+1) EA eventually reaches its plateau at a value higher

than the random search plateau value. Further, (1+1) EA ismoredeterministic than ran-

dom, i.e., thedistribution of (1+1) EA hasasmaller variance than that of random search,

especially when reaching theplateau (seeFigure8). In someregions (e.g., Figure8(d)),

however, random reaches its plateau slightly faster than (1+1) EA, while in some other

regions (e.g. Figure 8(a)), (1+1) EA is faster. We will discuss the relationship between

the region landscape and the performance of (1+1) EA in RQ3.

RQ3. We drew the landscape for the 11 regions in our experiment. For example, Fig-

ure9 showsthelandscape for two selected regions in Figures7(a) and 7(b). Specifically,

Figure 9(a) shows the landscape for the region in Figure 7(b) where (1+1) EA is faster

than random, and Figure 9(b) shows the landscape for the region in Figure 7(a) where

(1+1) EA is slower than random search.

0.30

0.31

0.32

0.33

0.34

0.35

0.36

0.37

0.38

0.39

0.40

0.70 0.71 0.72 0.73 0.74 0.75 0.76 0.77 0.78 0.79 0.80

0.10

0.11

0.12

0.13

0.14

0.15

0.16

0.17

0.18

0.19

0.20

0.90 0.91 0.92 0.93 0.94 0.95 0.96 0.97 0.98 0.99 1.00

(a) (b)

Fig.9. Diagrams representing the landscape for two representative HeatMap regions: (a) Land-

scape for the region in Figure 7(b). (b) Landscape for the region in Figure 7(a).

Our observations show that the regions surrounded mostly by dark shaded regions

typically haveaclear gradient between the initial point of thesearch and theworst case

point (see e.g., Figure 9(a)). However, dark regions located in a generally light shaded

areahaveanoisier shapewith several local optimum (seee.g., Figure 9(b)). It isknown

that for regions likeFigure9(a), exploitativesearch worksbest, while for those likeFig-

ure 9(b), explorative search is most suitable [10]. This is confirmed in our work where

for Figure 9(a), our exploitative search, i.e., (1+1) EA with σ = 0.01, is faster and more

effectivethan random search, whereas for Figure9(b), our search isslower than random

search. Weapplied amoreexplorativeversion of (1+1) EA where we let σ = 0.03 to the

region in Figure 9(b). The result (Figure 10) shows that the more explorative (1+1) EA

is now both faster and more effective than random search. We conjecture that, from the

HeatMap diagrams, we can predict which search algorithm to use for the single-state

search step. Specifically, for dark regions surrounded by dark shaded areas, we suggest

an exploitative (1+1) EA (e.g., σ = 0.01), while for dark regions located in light shaded

areas, we recommend a more explorative (1+1) EA (e.g., σ = 0.03).

6 Related WorkTesting continuous control systems presents anumber of challenges, and isnot yet sup-

ported by existing toolsand techniques [4, 1, 3]. Themodeling languages that havebeen

13

31

Page 32: The Thousands Shades of Model-Based Testing: Pitfalls, … · 2013. 12. 13. · The Thousands Shades of Model-Based Testing: Pitfalls, Challenges, and Guidelines University of Luxembourg

MBT Projects Sample (< 5 years)

32

Company Domain Objective Notation Automation

ABB Robot controller Safety UML Constraint Solver

Cisco Video conference Robustness UML profile Metaheuristic

Kongsberg Maritime Oil&gas, safety critical

drivers

CPU usage UML+MARTE Constraint Solver

WesternGeco Marine seismic

acquisition

Functional testing UML profile + MARTE Metaheuristic

SES Satellite operator Functional testing UML profile Metaheuristic

Delphi Automotive systems Testing

safety+performance

Matlab/Simulink Metaheuristic

Lux. Tax department Legal & financial Legal Requirements

testing

UML Profile Under investigation

Page 33: The Thousands Shades of Model-Based Testing: Pitfalls, … · 2013. 12. 13. · The Thousands Shades of Model-Based Testing: Pitfalls, Challenges, and Guidelines University of Luxembourg

Verifying CPU Time Shortage Risks in

Integrated Embedded Software

33

Page 34: The Thousands Shades of Model-Based Testing: Pitfalls, … · 2013. 12. 13. · The Thousands Shades of Model-Based Testing: Pitfalls, Challenges, and Guidelines University of Luxembourg

Today’s cars rely on integrated systems

• Modular and independent development

• Many opportunities for division of labor and

outsourcing

• Need for reliable and effective integration

processes 34

Page 35: The Thousands Shades of Model-Based Testing: Pitfalls, … · 2013. 12. 13. · The Thousands Shades of Model-Based Testing: Pitfalls, Challenges, and Guidelines University of Luxembourg

Integration process in the automotive domain

AUTOSAR Models sw runnables

sw runnables AUTOSAR Models

Glue

35

Page 36: The Thousands Shades of Model-Based Testing: Pitfalls, … · 2013. 12. 13. · The Thousands Shades of Model-Based Testing: Pitfalls, Challenges, and Guidelines University of Luxembourg

36

CPU Time Shortage in Integrated Embedded Software

• Challenge

– Many OS tasks and their many runnables run within a limited

available CPU time

• The execution time of the runnables may exceed the OS cycles

• Our goal

– Reducing the maximum CPU time used per time slot to be

able to

• Minimize the hardware cost

• Enable addition of new functions incrementally

• Reduce the probability of overloading the CPU in practice

36

5ms 10ms 15ms 20ms 25ms 30ms 35ms 40ms

5ms 10ms 15ms 20ms 25ms 30ms 35ms 40ms

(a)

(b)

Fig. 4. Two possible CPU time usage simulations for an OS task with a 5mscycle: (a) Usage with bursts, and (b) Desirable usage.

its corresponding glue code starts by a set of declarations

and definitions for components, runnables, ports, etc. It then

includes the initialization part followed by the execution part.

In the execution part, there is one routine for each OS task.

These routines are called by the scheduler of the underlying

OS in every cycle of their corresponding task. Inside each

OS task routine, the runnables related to that OS task are

called based on their period. For example, in Figure 3, we

assume that the cycle of the task o1 is 5ms, and the period

of the runnables r1, r2, and r 3 are 10ms, 20ms and 100ms,

respectively. Thevalue of timer is the global system time. Since

the cycle of o1 is 5, the value of timer in the Task o1() routine

is always a multiple of 5. Runnables r 1, r2 and r3 are then

called whenever the value of timer is zero, or is divisible by

the period of r 1, r 2 and r 3, respectively.

Although AUTOSAR provides a standard means for OEMs

and suppliers to exchange their software, and essentially

enables the process in Figure 1, the automotive integration

process still remains complex and erroneous. A major inte-

gration challenge is to minimize the risk of CPU shortage

while running the integrated system in Figure 1. Specifically,

consider an OS task with a 5ms cycle. Figure 4 shows two

possible CPU time usage simulations of this task over eight

time slots between 0 to 40ms. In Figure 4(a), there are bursts

of high CPU usage at two time slots at 0ms and 35ms, while

the CPU usage simulation in Figure 4(b) is more stable and

does not include any bursts. In both simulations, the total

CPU usage is the same, but the distribution of the CPU usage

over time slots is different. The simulation in Figure 4(b) is

more desirable because: (1) It minimizes the hardware costs

by lowering the maximum required CPU time. (2) It facilitates

the assignment of new runnables to an OS task, and hence,

enables the addition of new functions as it is typically done in

the incremental design of car manufacturers. (3) It reduces the

possibility of overloading CPU as the CPU time usage is less

likely to exceed the OS task cycle (i.e., 5ms) in any time slot.

Ideally, a CPU usage simulation is desirable if in each time

slot, there is a sufficiently large safety margin of unused CPU

time. Due to inaccuracies in estimating runnables’ execution

times, it is expected that the unused margin shrinks when the

system runs in a real car. Hence, the larger is this margin, the

lower is the probability of exceeding the limit in practice.

In this paper, we study the problem of minimizing bursts

of CPU time usage for a software system composed of a

large number of concurrent runnables. A known strategy to

eliminate high CPU usage bursts is to shift the start time

(offset) of runnables, i.e., to insert a delay prior to the start of

the execution of runnables [5]. Offsets of the runnables must

satisfy three constraints: C1. The offset values should not lead

to deadline misses, i.e., they should not cause the runnables to

run passed their periods. C2. Since the runnables are invoked

by OS tasks, the offset values of each runnable should be

divisible by the OS task cycle related to that runnable. C3. The

offset values should not interfere with data dependency and

synchronization relations between runnables. For example,

suppose runnables r1 and r 2 have to execute in the same time

slot because they need to synchronize. The offset values of r 1

and r 2 should be chosen such that they still run in the same

time slot after being shifted by their offsets.

There are four important context factors that are in line with

AUTOSAR [13], and have influenced our work:

CF1. The runnables are not memory-bound, i.e., the CPU

time is not significantly affected by the low-bound memory

allocation activities such as transferring data in and out of

the disk and garbage collection. Hence, our analysis of CPU

time usage is not affected by constraints related to memory

resources (see Section III-B).

CF2. The runnables are Offset-free [4], that is the offset of

a runnable can be freely chosen as long as it does not violate

the timing constraints C1-C3 (see Section III-B).

CF3. The runnables assigned to different OS tasks are

independent in the sense that they do not communicate with

one another and do not share memory. Hence, the CPU time

used by an OS task during each cycle is not affected by other

OS tasks running concurrently. Our analysis in this paper,

therefore, focuses on individual OS tasks.

CF4. The execution times of the runnables are remarkably

smaller than the runnables’ periods and the OS task cycles.

Typical OStask cycles arearound 1ms to 5ms. The runnables’

periods are typically between 10ms to 1s, while the runnables’

execution times are between 10ns = 10− 5ms to 0.2ms.

Our goal is to compute offsets for runnables such that the

CPU usage is minimized, and further, the timing constraints,

C1-C3, discussed earlier above hold. This requires solving

a constraint-based optimization problem, and can be done in

three ways: (1) Attempting to predict optimal offsets in a de-

terministic way, e.g., algorithms based on real-time scheduling

theory [6]. In general, these algorithms explore a very small

part of the search space, i.e., worst/best case situations only

(see Section V for a discussion). (2) Formulating the problem

as a (symbolic) constraint model and applying a systematic

constraint solver [14], [15]. Due to assumption CF4 above,

the search space in our problem is too large, resulting in

a huge constraint model that does not fit in memory (see

Section V for more details). (3) Using metaheuristic search-

based techniques [9]. These techniques are part of the general

class of stochastic optimization algorithms which employ

some degree of randomness to find optimal (or as optimal

as possible) solutions to hard problems. These approaches are

applied to awide range of problems, and are used in this paper.

I I I . SEARCH-BASED CPU USAGE M INIMIZATION

In this section, we describe our search-based technique for

CPU usage minimization. We first define a notation for our

problem in Section III-A. We formalize the timing constraints,

Page 37: The Thousands Shades of Model-Based Testing: Pitfalls, … · 2013. 12. 13. · The Thousands Shades of Model-Based Testing: Pitfalls, Challenges, and Guidelines University of Luxembourg

37

Using runnable offsets (delay times)

5ms 10ms 15ms 20ms 25ms 30ms 35ms 40ms

5ms 10ms 15ms 20ms 25ms 30ms 35ms 40ms ✗

Inserting runnables’ offsets

Offsets have to be chosen such that

the maximum CPU usage per time slot is minimized, and further,

the runnables respect their period

the runnables respect the OS cycles

the runnables satisfy their synchronization constraints

37

Page 38: The Thousands Shades of Model-Based Testing: Pitfalls, … · 2013. 12. 13. · The Thousands Shades of Model-Based Testing: Pitfalls, Challenges, and Guidelines University of Luxembourg

38

Meta heuristic search algorithms

Case Study: an automotive software system with 430 runnables

Running the system without offsets

Simulation for the runnables in our case study and

corresponding to the lowest max CPU usage found by HC

5.34 ms

Our optimized offset assignment

2.13 ms

- Search algorithms are used to search offset values balancing CPU

usage

- The objective function is the max CPU usage of a 2s-simulation of

runnables

- Single-state search algorithms for discrete spaces (HC, Tabu)

38

Page 39: The Thousands Shades of Model-Based Testing: Pitfalls, … · 2013. 12. 13. · The Thousands Shades of Model-Based Testing: Pitfalls, Challenges, and Guidelines University of Luxembourg

39

Conclusions

- We developed a number of search

algorithms to compute offset values

that reduce the max CPU time needed

- Our evaluation shows that our

approach is able to generate

reasonably good results for a large

automotive system and in a small

amount of time

- Due to large number of runnables and

the orders of magnitude difference in

runnables periods and their execution

times, we were not able to use

constraint solvers

- Current: Accounting for task time

coupling constraints with multi-

objective search trade-off between

relaxing coupling constraints and

maximum CPU time

39

Page 40: The Thousands Shades of Model-Based Testing: Pitfalls, … · 2013. 12. 13. · The Thousands Shades of Model-Based Testing: Pitfalls, Challenges, and Guidelines University of Luxembourg

Questions

• What kinds of models need to be developed to

support automated testing

• How expensive is test modeling?

• What are technologies enabling automated testing

based on models?

• How cost-effective is model-based testing (MBT)?

• What are the limitations of MBT?

• What are the open issues regarding MBT on which

research and innovation are still needed?

40

Page 41: The Thousands Shades of Model-Based Testing: Pitfalls, … · 2013. 12. 13. · The Thousands Shades of Model-Based Testing: Pitfalls, Challenges, and Guidelines University of Luxembourg

What kinds of models?

• What kinds of models need to be developed to support

automated testing?

• Four aspects:

– Notation

– Modeling Methodology

• Scope

• Level of detail

• Factors:

– Test objectives: Targeted faults, oracle.

– Domain

– Modeling skills and existing practice

• Standards: UML, SysML, MARTE, BPMN

– Often need to be tailored or specialized 41

Page 42: The Thousands Shades of Model-Based Testing: Pitfalls, … · 2013. 12. 13. · The Thousands Shades of Model-Based Testing: Pitfalls, Challenges, and Guidelines University of Luxembourg

How expensive is modeling?

• From a few days to a few weeks, really depends on

context

• Test models are much simpler than the systems they

purport to model

• They can serve other purposes as well, e.g.,

specification, certification

• The real question: modeling cost versus test

automation savings

42

Page 43: The Thousands Shades of Model-Based Testing: Pitfalls, … · 2013. 12. 13. · The Thousands Shades of Model-Based Testing: Pitfalls, Challenges, and Guidelines University of Luxembourg

What kinds of technologies?

• What are the enabler technologies for model-based testing?

• Goal: test case and oracle generation

– Find input values, sequences of events/operations satisfying

properties based on models, e.g., path in a state machine

– Derive oracles to detect failures at run-time, e.g., state

invariants, valid output sequences, metamorphic rules

– Timing or other performance measures may be relevant

• Goals can be often re-expressed as an optimization or

constraint solving problem

– Constraint solvers, e.g., IBM CPLEX

– Metaheuristic search, e.g., genetic algorithms

– Main challenge: Scalability

43

Page 44: The Thousands Shades of Model-Based Testing: Pitfalls, … · 2013. 12. 13. · The Thousands Shades of Model-Based Testing: Pitfalls, Challenges, and Guidelines University of Luxembourg

Cost-Effective?

• How cost-effective is MBT?

− Modeling training & Tools

− Modeling overhead

+ Test automation scales up

+ Regeneration of test suites when changes

+ More systematic, more confidence

+ Traceability: impact analysis, regression

• Cost-benefit results vary according to these factors

• My experience:

– The benefits far outweigh the costs, especially when

accounting for changes (e.g., requirements)

44

Page 45: The Thousands Shades of Model-Based Testing: Pitfalls, … · 2013. 12. 13. · The Thousands Shades of Model-Based Testing: Pitfalls, Challenges, and Guidelines University of Luxembourg

MBT Limitations?

• Not applicable when the

system or environment

cannot be easily

modeled with available

notations and tools

• Example: No (precise)

condition can be

identified to automate

oracles at run-time

• E.g., simulation, image

segmentation, scientific

computing

45

Page 46: The Thousands Shades of Model-Based Testing: Pitfalls, … · 2013. 12. 13. · The Thousands Shades of Model-Based Testing: Pitfalls, Challenges, and Guidelines University of Luxembourg

Open Issues?

• Scalability of test case generation

– Quick constraint solving

• Tailoring modeling notations and methodologies to specific

problems and domains

– The number of combinations of problems and domains is

large

– Hence potential problems in tailoring commercial tools

• Empirical studies

– There are very few credible, well-reported empirical studies

• Handling model changes

– Impact analysis

– Regression testing, e.g., selection, prioritization

46

Page 47: The Thousands Shades of Model-Based Testing: Pitfalls, … · 2013. 12. 13. · The Thousands Shades of Model-Based Testing: Pitfalls, Challenges, and Guidelines University of Luxembourg

FAQs

• What if the model is incomplete or incorrect?

– The purpose of MBT is automation, not proof

– The model may change as a result of failure

• Does MBT give me a proven test strategy?

– No, but it enables you to define and automate one

– There is not such thing as a universal, proven test strategy

• Can’t I just buy and apply some commercial MBT tool?

– How to apply MBT depends heavily on the domain, context,

and test objectives

– Heavy tailoring and investigation are required

• Isn’t MBT too expensive to introduce and tailor to our needs?

– manual testing (e.g., generation, oracle) is much more

expensive and less effective

47

Page 48: The Thousands Shades of Model-Based Testing: Pitfalls, … · 2013. 12. 13. · The Thousands Shades of Model-Based Testing: Pitfalls, Challenges, and Guidelines University of Luxembourg

Conclusions

• Despite much hype, the hardest testing problems (test case

generation, oracle) are not (really) solved yet in most contexts

• Modeling technology has matured (thanks in large part to OMG

standardization efforts around the UML and MDA)

• It is now easier to support Model-based testing (MBT) and

integrate it with other development activities

• MBT is a natural fit for companies using Model Driven

Engineering (MDE), but is also suitable for those that are not

– MBT is a good starting point for MDE

• In many situations, model-based testing is the only way to

achieve full test automation (scalability) – the question is not

whether to adopt MBT, but how.

• Much research and innovation is still required though and it

must involve collaborations between research and industry

48

Page 49: The Thousands Shades of Model-Based Testing: Pitfalls, … · 2013. 12. 13. · The Thousands Shades of Model-Based Testing: Pitfalls, Challenges, and Guidelines University of Luxembourg

Selected References

• L. Briand, Y. Labiche, and M. Shousha, “Using genetic algorithms for early

schedulability analysis and stress testing in real-time systems”, Genetic

Programming and Evolvable Machines, vol. 7 no. 2, pp. 145-170, 2006

• M. Shousha, L. Briand, and Y. Labiche, “UML/MARTE Model Analysis Method

for Uncovering Scenarios Leading to Starvation and Deadlocks in Concurrent

Systems”, IEEE Transactions on Software Engineering 38(2), 2012.

• Z. Iqbal, A. Arcuri, L. Briand, “Empirical Investigation of Search Algorithms for

Environment Model-Based Testing of Real-Time Embedded Software”, ACM

ISSTA 2012

• S. Nejati, S. Di Alesio, M. Sabetzadeh, L. Briand, “Modeling and Analysis of

CPU Usage in Safety-Critical Embedded Systems to Support Stress Testing”,

ACM/IEEE MODELS 2012

• S. Nejati, Mehrdad Sabetzadeh, D. Falessi, L. C. Briand, T. Coq, “A SysML-

based approach to traceability management and design slicing in support of

safety certification: Framework, tool support, and case studies”, Information &

Software Technology 54(6): 569-590 (2012)

• L. Briand et al., “Traceability and SysML Design Slices to Support Safety

Inspections: A Controlled Experiment”, forthcoming in ACM Transactions on

Software Engineering and Methodology, 2013

49

Page 50: The Thousands Shades of Model-Based Testing: Pitfalls, … · 2013. 12. 13. · The Thousands Shades of Model-Based Testing: Pitfalls, Challenges, and Guidelines University of Luxembourg

Selected References (cont.)

• Rajwinder Kaur Panesar-Walawege, Mehrdad Sabetzadeh, Lionel C. Briand:

Supporting the verification of compliance to safety standards via model-driven

engineering: Approach, tool-support and empirical validation. Information &

Software Technology 55(5): 836-864 (2013)

• Razieh Behjati, Tao Yue, Lionel C. Briand, Bran Selic: SimPL: A product-line

modeling methodology for families of integrated control systems. Information &

Software Technology 55(3): 607-629 (2013)

• Hadi Hemmati, Andrea Arcuri, Lionel C. Briand: Achieving scalable model-based

testing through test case diversity. ACM Trans. Softw. Eng. Methodol. 22(1): 6

(2013)

• Nina Elisabeth Holt, Richard Torkar, Lionel C. Briand, Kai Hansen: State-Based

Testing: Industrial Evaluation of the Cost-Effectiveness of Round-Trip Path and

Sneak-Path Strategies. ISSRE 2012: 321-330

• Razieh Behjati, Tao Yue, Lionel C. Briand: A Modeling Approach to Support the

Similarity-Based Reuse of Configuration Data. MoDELS 2012: 497-513

• Shaukat Ali, Lionel C. Briand, Andrea Arcuri, Suneth Walawege: An Industrial

Application of Robustness Testing Using Aspect-Oriented Modeling,

UML/MARTE, and Search Algorithms. MoDELS 2011: 108-122

50