Bio-inspired Robustness Testing Approaches

Seminário IC- Set/ 2010

Bio-inspired Robustness Testing Approaches

Eliane [email protected]

Overview of the talk

2

The Test and Fault Injection Group: who are we?

Robustness: what is it? Why is it important? Robustness testing activities of the group Part I: An evolutionary approach for

robustness tests generation Part II: Use of sequence alignment algorithms

as test oracles

The Test and Fault Injection Group

3

Distributed systems Laboratory (LSD)

Software Engineering and Dependability

(SED)

Test and Fault Injection group

(TIF)Fault-tolerant architectures

Distributed Systems

http://www.sed.ic.unicamp.br/

Motivation

4

Testing software to ensure that the required functionality was met … Example:

Requirement: When heading in a direction (up / down) an elevator

would service all received stop requests in that direction then it would reverse direction and verify if other stops in that direction had to be serviced

Test scenarios: Does the elevator goes up when a request up is given? Does the elevator goes down when a request down is

given? … is necessary but not enough

Why robustness testing?

5

Requirement: When heading in a direction (up / down) an

elevator would service all received stop requests in that direction then it would reverse direction and verify if other stops in that direction had to be serviced

Test scenarios: What if a request down is given and the elevator is on the first floor? What if a request up is given and the elevator is on the last floor?

How does the system behave in presence of erroneous or unexpected user inputs? internal or external failures? stressful conditions? attacks?

Robustness testing

Robustness

6

IEEE Std 610.12-1990 - Glossary of Software Engineering Terminology “Robustness is the degree to which a software

system or component can function correctly in the presence of invalid inputs or stressful environmental conditions.”

[Anderson85]: “[…] A computing system can be said to

be robust if it retains its ability to deliver service in conditions which are beyond its normal domain of operation”

7

In independent technical assessment To perform top-level investigation into the

quality of the software being developed and to see how the developing organization reacts when problems are encountered.

During component selection: To help in the decision of what component

better suit the system being developed

During software development To assess error handling mechanismsSource: SEI-CMU Report [CPK05]

When Robustness testing is useful?

Robustness testing characterization

8

Specified Inputs PossibleBehavior Outputs

Should work

Undefined

Should return error

ValidComponent under test

Success

FailureInvalid

Inspired on (Kropp, Koopman, Siewiorek 1998)

TIF – Test & Fault Injection Group

9

Robustness testing

Model-based testing

Test execution environme

nt

Fault injectio

n

Test results analys

is

Activity DiagramState Diagram

Exhaustive searchEvolutionary search

External faults:Accidental faultsMalicious faults

Interface faultsCommunication faults

Property-orientedBioinformatics algorithms

TIF – Research Projects and Tools

10

Model-based

testing

• Tools:• Condado• Plavis

• Projects• ATIFS• PLAVIS• QSEE• Harpia

Fault injectio

n –

Acci

dental Fault

s

• Tools• Jaca• WSInject

• Projects• ACERTE• Benchmarkin

g• WebMov• BioCORE• RobustWeb

Attack injection • Webmov

• RobustWeb

TIF – Collaborations with other research groups

11

Foreign groups: University of Coimbra Institut Telecom Sud-

Paris Université Paris-Orsay Laboratoire

d’Automatique et d’Analyse des Systèmes (LAAS), Toulouse

Universidad Politécnica de Valencia

Brazilian groups: INPE ITA UFRGS UFES UFAL

USP-S.Carlos

Part I – Robustness tests generation

12

Robustness testing

Model-based testing

Test execution environme

nt

Fault injectio

n

Test results analys

is


Exhaustive searchEvolutionary search





13


Should work

Undefined

Should return error


Success

FailureInvalid


Model-Based Testing (MBT)

14

Used to test software system based on a model of the system behavior

Test cases are generated from the system behavior model

State-based models: Commonly used to represent behavior of reactive

systems Techniques proposed since early sixties

Why MBT

15

Useful when source code is not available Applicable in all test phases: from Unit to

System testing Tests can be prepared early in development

cycle Test preparation can help in uncover requirements

faults Tests can be automatically generated

For robustness testing: Concerns with abnormal situations early in the

development cycle

MBT – State-based models

16

Useful to represent behavior of reactive systems: Control objects and systems Communication protocols User interactions …

Fonte: H. Robinson, StarWest 2006

Homepage

Imagepage Newspage

ImageTab

HomeTab

NewsTab

HomeTab

NewsTab

ImageTab

MBT – State-based models

17

Useful to represent behavior of reactive systems: Control objects and systems Communication protocols User interactions …

There are various models: FSM (Finite State Machines) EFSM (Extended-Finite State Machines) UML State Machines And others:

Statecharts State transition systems

EFSM

18

EFSM consists of: States Transitions

The following elements are associated with each transition: Events Guards (enabling conditions) Actions

State1 State2 Event(P) [Guard] / Actions

EFSM Testing Strategies

19

•W method•U method•D method• and variants

Search based

•State cover•Transition cover•Switch cover•Path coverage•Constrained path coverage

Graph based

•Random testing•Evolutionary testing

Automata

theoretic

Plavis platform

Condado tool

??? tool

Exhaustive enumeration approaches

20

Automata-theoretic approaches Equivalence between specification model and

implementation Graph-based approaches:

Coverage of model elements Exhaustive search of the state-space

Algorithms commonly used: Traveling salesman Depth-first search

Model-based robustness test cases generation

21

Specification(nominal)

Specification of robust behavior

Robustnesstest

cases

Test case generation

Unexpectedinputs

Inopportune inputsUnspecified inputsInputs that should return errors

Difficulties with exhaustive search

22

Model can be very big, in special, when representing data aspects and abnormal situationsRisk of state explosion Algorithms like TSM or DFS are no longer

adequate

Huge solution space Generation of large test suites

Big test cases High costs to execute and maintain

Search-based testing

23

How to deal with the huge solution space? Use of meta-heuristics to search the solution

space for automatic test case generation Various approaches:

Random testingEvolutionary algorithms (EA)

Genetic Algorithms (GA)Generalized Extreme Optimization Algorithms (GEO)

Have been extensively used to search input data for single functions

Evolutionary testing schema

24

Reinsert

Evaluate

Terminate?Selectfittest

solutions

Mutate

Initial solution

Output

No solution

Testexecution

and monitorin

g

Solution: input data

Traversed path

Code-based x EFSM-based evolutionary testing

25

Code-based: Generation:

Input data Controllability:

Control of code instructions

No control of states and context variables

Path feasibility: execute inputs to decide

whether the required element (instruction, decision, ...) has been traversed

EFSM-based: Generation:

Sequence of inputs Controllability:

States, event parameters and context variables

No control of code instructions

Path feasibility: Static approaches

path feasibility is an undecidable problem

Some existing approaches

26

MBT x CBT

Characteristics Evolutionary Algorithm

(Kalaji et al. 2009)

MBT: EFSM

Transition coveragePath sensitization

Genetic Algorithm (GA)

(Lefticaru et al. 2008)

CBT: Java Generate flowchartGenerate test data from the generated flowchart

GASimulated annealingParticle swarm optimization

(McMinn et.al 2005)

CBT: C Sequence of function callsState-based behavior

GA

(Derderian et al. 2005)

MBT: EFSM

Test paths with high likelihood to be feasible according to guards in transitions

GA

(Baresel et al.2003)

MBT: EFSMCBT: C code

Test sequence = signal wave form for the executable model Test data for C code

Random

GA(Guo et al. 2003)

MBT: FSM Unique Input-Output Sequences

GA

MOST-EFSM: our evolutionary model-based testing approach (Thaise Yano, PhD)

27

Generation: A solution = test sequence Test sequence = sequence of events starting at the initial

state + parameters Test sequence transition path Objectives: find a path that covers the test purpose +

minimize path size Controllability:

How to guide the search for an adequate path in the EFSM? Use of Dependency Analysis

Path feasibility Use of executable EFSM test paths dynamically derived

Steps of the approach

28

Create model

Validate model

Analyse dependences

Generate abstract test

cases

Generate executable test

casesÉrika Almeida - MSc

Test criterium

Example – ATM system

Test purpose

ATM – Automated Teller Machine [Korel et al. 2003]

Dependency analysis – ATM example

30

Shows control and data dependencies between transitions

Backward reachability analysis to obtain

Taffecting(tp)

Based on algorithm from K. Androutsopoulos et al.2009

Taffecting(t18) = {t1,t4,t8,t17}

seq={card(236,234,138), pin(236), spanish(), savings(), deposit(867)}path = { t1 t4 t6 t8 t18 }

Example - nominal inputs

seq={card(236,234,138), receipt(), pin(236), spanish(), pin(236), savings(), deposit(867)}path = { t1 t4 t6 t8 t18 }

Example - Inopportune inputs

seq={card(236,234,138), receipt(), pin(236), spanish(), pin(236), savings(), deposit(867)}path = { t1 t4 t6 t8 t18 }

Assumption: Unexpected input the machine remains in current state and generates null output

Example - Inopportune inputs

Some results

34

Sequence size x Number of Fitness Function evaluations

t3 t18

Work in progress Definition of other test criteria:

For the moment only test purpose coverage Transition coverage based on dependency analysis

Evaluation of the approach Cost – effectiveness use of benchmark models Scalability Use in real world applications

Web-services from an industry Teleduc

Open points and opportunities (1)

36

Development of a framework to support MOST How to guide users in adjusting the EA for his/her

problem?

Use of model transformation to easy dependency analysis One of the challenges in EFSM is the problem of

how to determine control dependency, as state machines may not have a final state Transform the model to improve its “analysability” Transform to a control flow graph [J.Li, E.Wong 2002]

use of MDA approaches

Open points and opportunities (2)

37

Comparison with the use of other meta-heuristics for EFSM-based testing: Given that each heuristic has its own strengths and limitations,

some might be more useful for some problems than others Random testing (e.g, hit-or-jump) Multi-objective Genetic Algorithms Multi-objective Ant Colony …

More long term … Use of more than two objectives Use of other approaches:

Memetic algorithms (MA): form of hybrid GA + individual learning procedures to perform local refinements

Hyper-heuristics: use of machine-learning techniques to automate the process of selecting, combining, adapting several heuristics to efficiently solve a problem

Part II – Result Analysis

39

Robustness testing

Model-based testing

Test execution environm

ent

Fault injectio

n

Test results analysi

s


Exhaustive searchHeuristic-based search





40


Should work

Undefined

Should return error


Success

FailureInvalid


How to identify robustness failures? (1) Comparison with a “golden run”

Component under test

Valid inputs

Outputs(golden run)

Component under test

Invalid inputs

Outputs

= FailureSuccess Yes No

What if the “golden run” is

faulty?

How to identify robustness failures? (2)

CRASH scale of the Ballista approach: Catastrophic: system crashes or reboots Restart: task execution hangs Abort: abnormal termination of task execution Silent: no exception reported when it should be Hindering: incorrect error code or delayed response

However …

Use of the CRASH scale: A system can fail without aborting or delaying

responses or sending error messages For example:

in an Elevator service, if there is a requestUP(floorID), then the Elevator should moveUp( ), stopAt(floorID) and opendoor( ) or else it stays in the same floor

In case of a request to a valid floor, if the Elevator opendoor( ) before stopAt(floorID), failure

A passive approach for robustness testing

44

Systemunder Test

Environment

Inputs

Outputs

Execution Traces

Inputs

OutputsClient

Points of Observation (PO)

Trace AnalyserSpecification(model or properties)

Verdict PassFailInconclusive

Analysis

Observation

Passive testing

45

A monitoring approach The steps:

Add observation mechanisms (PO) to the System under Test (SUT)

Execute the SUT and collect execution traces Filter the execution trace to keep only the events

of interest Analyse the filtered trace

46

•Execution trace is compared with a specification, given as a finite state machine (FSM)•Only control aspects are considered

Invariant analysis

• For EFSM-based specifications: consider the data aspects

• Intervals to represent possible variable values

• Predicates to represent assertions on variable values

Interval determina

tion• Invariant = property the SUT

must satisfy• Useful when no complete

specification of acceptable behavior may be available

Trace acceptatio

n

Passive Testing Approaches

47

A previous invariant analysis approach

Textual specification

Behavior model

I1 = RcvInvoke(TID = N)/?, *, TR-Invoke.res/{Ack (TID = N)}I2= RcvInvoke(TID = N) / TR.Invoke.ind, *, TR-Invoke.res / {Ack (TID = N) }

Invariants in the form of regular expressions

verification

[Bayse, Cavalli, Nuñez, Zahidi 2005]

Our proposal (Gizelle S. Lemos)

48

An approach for analyzing results of robustness testing:

1. Monitoring system execution in the presence of faults, collecting execution trace

2. Definition of the properties the system should verify even in the presence of faults

3. Verify whether the system model satisfies the required properties, if the model exists

4. Checking whether the execution trace satisfies the required properties

How to obtain the properties? What kind of properties to analyze? How to express the properties? How to analyze the trace?

Aspects to consider

How to obtain the properties?

50

Specification stating what a system should or should not do

Robustness specification should take into account Invalid and inopportune inputs:

exceptions, erroneous data, valid data at the wrong moment, missing events, late events

Outputs corresponding to the abnormal inputs: exception handlers, error messages, restart mechanism, …

For the moment, only safety properties are being considered: Something bad never happens during execution If something bad happens a robustness failure

is identified

What kind of properties?

How to express the properties

52

The (execution) trace is modeled as a sequence of events

The property (invariant) is also modeled as a sequence of events, much smaller than the trace

Properties can be represented by regular expressions Ex.: ((moveUp | moveDown), stopAtFloor,

opendoor )*

How to analyse the trace?

53

The trace satisfies the property if the property is a subsequence of the trace The events in the property may not be

consecutive in the trace, because of interleaving with other events that are not of interest for the property

Ex.: … moveUp(1, 4), requestDown(2, 1), …, stopAtFloor( 1, 4), movedown(2, 1), opendoor(1, 4)

How to analyse the trace?

54

Because of occurrence of unexpected events, some events in the trace may be different from those in the property, but also acceptable Events can be replaced by others

Ex.: in response to an incorrect input, a call to an exception handler is acceptable, but abort or hang is not

Events can also be deleted Ex.: in a fail stop behavior, the system stops

and no longer emits any output

The problem

55

In robustness testing we may not always know the exact behavior a system should have, but we know that some behaviors are more acceptable than others

Pattern matching

What algorithm to use?

56

In the literature longest common subsequence (LCS) problem: How to find the longest subsequence common to both

the trace and the property? Existing invariant analysis approaches use exact

pattern matching algorithms, which are adequate for conformance testing

For robustness testing inexact pattern matching algorithms are more appropriate Dynamic programming can be used when the number

of sequences is constant Match is solved in O(MxN) time, where M and N are the sizes

of both sequences

Dynamic programming is used for the alignment of two DNA sequences It is possible to take into account semantic

aspects, not allowable in pattern matching algorithms Use of a scoring system

It is possible to detect insertions, deletions or replacements of one or more inputs or outputs in the sequence

It is possible to obtain some statistics, as for example, number of matches and mismatches

The selected pattern matching algorithm

Alignment classification

58

Global: Useful when the sequences present almost the

same size: Stretch the sequences using gaps so that they have the

same size Semi global:

Sequences may not have the same size: Prefix or suffix are ignored use of gaps

Local: Useful for sequences of different sizes The focus is to find regions of high similarity

Example

59

Example

60

61

The DP algorithm Objective

Obtain the optimal alignment between two sequences The optimal alignment between sequences is the

one that maximizes the score

Where ma is the number of matches mi is the number of mismatches g is the number of gaps w is the weight for matches and mismatches gp is penalty assigned to gap occurrence

Scoring function

Schematic view of the approach

PassVerdict: Fail

Inconclusive

63

Filtered execution trace

Execution trace after encoding

Aligning the trace with an invariant

64

RcvInvoke / TR-Invoke.ind,*,RcvAck / { TR-Result.cnf,NULL }

14,30,22,28

An invariant

The invariant after encoding

Alignment example

What can be wrong?

65

False positives: Mark as violations some sequences that may be

correct

False negatives: Accept some sequences that are incorrect

Causes: Incorrect property used Inadequate scoring system Inadequate algorithm

On-going work

Assessment of the approach using: Benchmark programs

Number of false positives and false negatives

Simple regular expressions will not be enough Need other alignment algorithms (e.g. regular

expression constrained sequence alignment) Ex.: [requestUp] (moveUp, stopAtFloor, opendoor )*

Open points and opportunities

67

Use of the approach in real-world applications Web services robustness testing

Analysis of various properties Need to use multiple sequence alignment

algorithms

Some publications

69

Model-based testing

70

Martins, E., Sabião, S. B., and Ambrosio, A. M. ConData: A tool for automating specification-based test case generation for communication systems. Software Quality Control, 8(4):303–320, 1999.

ROCHA, Camila R.; MARTINS, Eliane. A strategy to improve component testability without source code. In Proc. Testing Component-based Systems Workshop (Net.ObjectDays Workshop), Erfurt (Germany), 2004.

Eliane Martins, Vanessa G. Vieira. Regression test selection for testable Classes. 5th. European Dependable Computing Conference, EDCC 2005, Budapest, Hungary, 20-22 April 2005, ”. Lecture Notes in Computer Science. Publisher: Springer-Verlag. Volume 3463/2005, pp453-470.

Ana Maria Ambrosio; Eliane Martins; Nandamudi L. Vijaykumar; Solon V. Carvalho. A Conformance Testing Process for Space Applications Software Services. AIAA (American Institute of Aeronautics and Astronautics) Journal of Aerospace Computing, Information, and Communication (1542-9423) 2006 vol. 3 no. 4, pages (146-158).

Ivan Perez, Eliane Martins, Júlio Esslinger Viégas. Uso de Modelos da UML em Testes de Componentes. VIII Workshop de Testes e Tolerância a Falhas (Em conjunto com o XXV SBRC - Simpósio Brasileiro de Computadores), Belém, PA, 29 de Maio de 2007.

T. Yano, E.Martins, F.L.Sousa. Generating Feasible Test Paths from an Executable Model Using a Multi-Objective Approach. In Proceedings of the 3rd International Workshop on Search-Based Software Testing, joint with IEEE International Conference on Software Testing, Verification and Validation (ICST), 2010. Paris

Trace Analysis

71

Martins, Eliane; Morais, Anderson Nunes Paiva; Cavalli, Ana. Generating Attack Scenarios for the Validation of Security Protocol Implementations, The 2nd Brazilian Workshop on Systematic and Automated Software Testing (SBES 2008 -SAST), Brazil, October 2008.

Cavalli, Ana ; Martins, E. ; Morais, Anderson Nunes de Paiva . Use of invariant properties to evaluate the results of fault-injection-based robustness testing of protocol implementations. In: 4th Workshop on Advances in Model Based Testing (A-MOST 2008), joint with 1st. IEEE International Conference on Software Testing, Verification and Validation (ICST), 2008, Lillehammer. Proceedings of the A-MOST, 2008.

F. Bessayah, A. Cavalli, E. Martins. A Formal Approach for Specification and Verification of Fault Injection Process. In Proceedings of The ACM International Conference on Interactive Sciences (ICIS'09),Seoul, Korea. November 24-26, 2009.

. G. S. Lemos, E. Martins, Robustness Testing Oracle using a Sequence Alignment Algorithm. First International Workshop on Software Test Output Validation (STOV), Co-located with ISSTA 2010, Trento, Italy.

Fault injection (1)

72

Eliane Martins, Regina O. Moraes. Validating an ODBMS Component using a High-Level Software Fault Injection Tool. 1st. Latin American Symposium on Dependable Computing (LADC), S.Paulo, Brasil, 21-24/10/2003.

Regina O. Moraes, Eliane Martins. Fault Injection Approach Based on Architectural Dependencies. Lecture Notes in Computer Science. Publisher: Springer-Verlag. Volume 3549 / 2005. Editors: Rogério de Lemos, Cristina Gacek, Alexander Romanovsky.

Regina Lúcia O. de Moraes, Eliane Martins, Elaine C. Catapani Poleti, Naaliel Vicente Mendes. Using Stratified Sampling for Fault Injection. 2nd. Latin-American Symposium on Dependable Computing (2nd LADC), Salvador, Bahia, Brazil, Oct/2005.

Regina Moraes, João Durães, Eliane Martins and Henrique Madeira. A Field Data Study on the Use of Software Metrics to Define Representative Fault Distribution. Workshop on Empirical Evaluation of Dependability and Security (WEEDS), collocated with The IEEE-IFIP International Conference on Dependable Systems and Networks (DSN-2006), Filadélfia, EUA, junho/2006.

R. Moraes, R. Barbosa, J. Durães, N. Mendes, E. Martins, H. Madeira. Do injected component interface faults represent software bugs?. 6th. European Dependable Computing Conference (EDCC) 2006. Coimbra, Portugal, Outbro/2006.

R. Moraes, J. Durães, E. Martins, H. Madeira. Experimental Risk Assessment and Comparison Using Software Fault Injection. DSN 2007. Edimburgo. Escócia. Junho/2007

Fault injection (2)

73

Morais, Anderson; Martins, Eliane; Cavalli, Ana; Jimenez , Willy. Security Protocol Testing Using Attack Trees. In: The 2009 International Symposium on Trusted Computing and Communications (EUC 2009 - TRUSTCOM), Vancouver, 29-31 August 2009.

Anderson Morais, Ana Cavalli, Eliane Martins, Attack scripts generation for security validation, Sécurité des Systèmes d’Information et les Environnements Collaboratifs (SEC-SY), joint with INFORSID 2010, 25 May 2010, Marseille, France.

F. Bessayah, A. Cavalli, W. Maja, E. Martins, A.W. Valenti. A Fault Injection Tool for Testing Web Services Composition. In Testing: Academic and Industrial Conference - Practice and Research Techniques (TAIC PART’10). Windsor,UK. To be published.

Example

74

Generated with Plavis: http://www2.dem.inpe.br/plavisFSM/main.php

Cruise control system (obtained in March/2010: http://sir.unl.edu/php/index.php)

Transition tree for the Cruise Control System

75

Inactive/Idle

Inactive/Idle

Standby/Running

Standby/Running

Cruising/Running

Cruising/Running

Standby/Running

Inactive/Idle

Standby/Running

Cruising/Running

Standby/Running

Inactive/Idle

Cruising/Running

Active/Running

Active/Running

Active/RunningengineOn

accelerator

brake

on

engineOff

brake

on

accelerator

engineOff

off

on

resume

accelerator

brake

engineOff

Inactive/Idle

Inactive/Idle

Inactive/Idle

Inactive/Idle

Inactive/Idle

Inactive/Idle

resume

accelerator

brake

on

off

engineOff

Active/Running

Active/Running

Active/Running

off

resume

engineOn

Cruising/Running

Cruising/Running

resume

engineOn

Standby/Running

Standby/Running

engineOn

off

Example of test cases generated by Plavis (1)

76

UIO method1 Seq: accelerator off engineOn brake on resume engineOff2 Seq: engineOn on accelerator3 Seq: engineOn resume brake4 Seq: engineOn on brake brake5 Seq: engineOn on engineOff engineOn6 Seq: engineOn on accelerator off brake7 Seq: engineOn accelerator brake8 Seq: brake engineOn9 Seq: engineOn engineOff engineOn1 0 Seq: engineOff engineOn

...Total: 29 test cases

Example of test cases generated by Plavis (2)

77

Switch-cover method

Total: 483 test cases

1 Seq: engineOn on off on accelerator resume accelerator engineOff

2 Seq: engineOn on accelerator resume brake on brake engineOff

3 Seq: engineOn on off on brake resume off brake engineOff4 Seq: engineOn on accelerator on off resume off engineOn engineOff

5 Seq: engineOn on off resume off on engineOff6 Seq: engineOn on brake resume accelerator on brake brake engineOff

...

Bio-inspired Robustness Testing Approaches

Documents

Transcript of Bio-inspired Robustness Testing Approaches