Model-Based Testing at ETSIModel-Based Testing at ETSI

Stephan SchulzETSI TC MTS chair

TC INT IMS Workshop Nov 2010 © ETSI 2010. All rights reservedTC INT IMS Workshop Nov 2010

SCOPE of the Technical Committee

Who is MTS?

Who is TC MTS?SCOPE of the Technical Committee

Methods for Testing and Specification (MTS) creates guidelines, frameworks, notations, and methodologies for specification and testing to help other ETSI committees to efficiently develop

their standards and test specifications.

Past Achievements

pTC MTS cooperates closely with ETSI’s Center for Testing & Interoperability (CTI).

Present Activities Possible Future AreasTTCN-3 Evolution

NotationsTTCN-3, TPLan, Descriptive SDL

TTCN 3 EvolutionEnhancements for LTE testing, real time

Performance testing

Further TTCN-3 Development

Update of best practice for specification

Test specifications for core IP technologies SIP, SIGTRAN, IPv6 (core, sec, mob, 4to6), H.225, H.248

Automated interoperability testingMethodology & framework

M d l B d T ti

p

ETSI security framework

Measurement framework formob, 4to6), H.225, H.248

Methodologies & guidelinesMaking Better Standards,IP testing framework

Model Based TestingConcepts for modelling

Securitye-Passport testing framework

Measurement framework for physical layers

Further work on Model Based Testing

22

e Passport testing framework and platform

Testing

More info: http://portal.etsi.org/mts

About Model-Based Testing (MBT)An umbrella of approaches that generate tests from models

Here: Test generation from system modelsSystem Model

Here: Test generation from system modelsIs about automating test design, i.e., produces test scripts and documentation

I li i

Generation

(TTCN-3) Test ScriptsIt compliments test execution

Considered today industrial best practice in software testing1 Test Execution Engine

(TTCN-3) Test Framework

gNumber of commercial tools available

ETSI TC MTS is creating standards on MBT since 2008 S t U d T t

Test Adaptation

MBT since 20082 TRs,1 ES, and more are to come

System Under Test

3

1 ”The Testing Tools Landscape: 2010. Functional Testing Tools Are Not Enough.” by M. Visitacion & M. Gualtieri. Forrester Research, Inc.

First ES: Concepts for Model Specification Identifies and collects all relevant concepts that model specifications developed for testing should include

It will essentially mandates concepts that MBT tools need to supportIt will essentially mandates concepts that MBT tools need to support to facilitate the generation of such testsConcepts are specified independent of a specific modelling language

Standard is currently in drafting stageStandard is currently in drafting stageFinal draft expected for TC MTS meeting in Dec 2010Key MBT tool vendors, industrial users, and research institutes who are active in this field are participating(Ericsson, Wipro, NSN, Siemens, SAP, MS, Conformiq, Smartesting, SeppMed, FOKUS, IRISA)

N t th t thi t d d i t d t b l th fi t i iNote that this standard is expected to be only the first in a series of ES’s for model-based testing tools

4

Classic vs. MBT Test Development

BaseSpec

TestPurposes

TTCN-3Tests

Reqstestcase t1() {

…}Specify

SystemModel

BaseSpec

GenerateReqs

5

testcase t1() {…

}

An Example MBT Tool Workflow

DevelopSystem Model

Direct & ReviewTest Design

Generate Scripts& DocumentationSystem Model Test Design & Documentation

Model Editor MBT Tool Test ExecutionTool(s)

6

Why is MBT interesting?

Model reuse Simplified test harness

Savings in test maintenance

Enabling new ways of working

Improved test Customizable

t t t

Higher test

coverage test sets

The business case for

MBT

Higher test design

productivityTest scripting

language independence

77

MBT

First Ideas on using MBT at ETSIShould not change ETSI’s existing way of standardizing tests!

Merely offer a new method for test development, e.g., in STFsModels specify externally observable behavior as defined by aModels specify externally observable behavior as defined by a base specification

But should be part of test specifications - not base specificationsIt is the generated tests that should be standardized

Models should be considered informative testing artifactsETSI TCs should set requirements for test selectionETSI TCs should set requirements for test selection

Perceived benefits of this approach includeImproved quality of ETSI test specifications More transparency in the ETSI test specification processAbility for ETSI members to reuse models for own test generation

8

The Road AheadCreation of further ETSI standards on model-based testing

Guide on the specification of models for testing in the context standardizationMinimum requirements on test generation/selectionConcretization of modelling concepts

O i ti f k h f h i i d t i l iOrganization of a workshop for sharing industrial experiences with model-based testingA MTS STF for developing a guide and performing case studies p g g p gwith other ETSI TCs/WGs in second half of 2011

So far TC INT and TC ITS have shown interest

9