Overview of ETSI Testing Methodology Anthony Wiles Manager ETSI Protocol and Testing Competence...
-
Upload
ryan-gonzalez -
Category
Documents
-
view
227 -
download
0
Transcript of Overview of ETSI Testing Methodology Anthony Wiles Manager ETSI Protocol and Testing Competence...
Overview of ETSI Testing Methodology
Anthony Wiles
Manager
ETSI Protocol and Testing Competence Centre
ETSI Testing Methodology - SINTESIO Seminar 2006 2
ETSIEuropean Telecommunications
Standards Institute
Sophia Antipolis, France
The home of ICT standardization
http://www.etsi.org
ETSI Testing Methodology - SINTESIO Seminar 2006 3
Interoperability is …
The ultimate aim of ICT standardisation The red thread running through the entire standards
development process, it’s not an isolated issue Not something to be somehow fixed at the end
The ability of systems and products to interwork is fundamental
ETSI approach ETSI produces with the required degree of parallelism
• Base Standards and • Test Specifications
Base standards should be designed with interoperability in mind Profiles designed to reduce potential non-interoperability Protocol conformance and Interoperability statements Two complementary forms of testing
• Conformance testing• Interoperability testing (formal IOT)
Testing provides vital feedback into the standardization work
ETSI Testing Methodology - SINTESIO Seminar 2006 4
Poor Interoperability is Expensive
Bad publicity in trade magazines Embarrassment for the manufacturer Annoyance of the end customer
In the past, interoperability failures meant:
Today, interoperability failures in the field means:
Front page headlines in the Financial Times Fall in manufacturers stock price Loss of investor confidence Unrecoverable damage to brand name Irretrievable loss of customers
We can no longer afford to get it wrong! We can no longer afford to get it wrong!
ETSI Testing Methodology - SINTESIO Seminar 2006 5
Specification and Testing is Part of ETSI’s Strategy (to get it right!)
ETSI membership believes in methodical testing A good example
GSM/UMTS is growing by 1 Million users per day! Can you imagine what would happen if users were
experiencing interoperability problems ? More than 1 Million Euros invested per year for writing
formal test specifications We have experienced what it means to get it right!
A bad example We have not always got it right! For GPRS we did not take a formal approach to testing First market products did not interoperate
We have learnt that testing is not cheap, but it pays in the long run
ETSI Testing Methodology - SINTESIO Seminar 2006 6
Technical Committee MTS (Methods for Testing and Specification)Development of methodologies, techniques and languages (including TTCN3)
• http://portal.etsi.org
ETSI’s Protocol and Testing Competence Centre (PTCC)Supports ETSI committees on the application of formal techniques in standards on a daily basisDevelopment of test specifications (conformance and interop)
• http://portal.etsi.org/ptcc/
ETSI Plugtests™ ServiceValidation of standards and prototypes through interoperability events
• http://www.etsi.org/plugtests
ETSI Resources for Interoperability
ETSI Testing Methodology - SINTESIO Seminar 2006 7
How Does the PTCC Help?
Assist ETSI Technical Bodies on the use of state of the art techniques for Specification, validation and testing Good working practices (Standards Engineering)
Pragmatic and flexible approach Based on experience
Help to develop usable methodologies For ETSI’s current and emerging needs
Knowledge transfer Quality through Continuity
ETSI Testing Methodology - SINTESIO Seminar 2006 8
Who PTCC Supports
Technical Bodies (TB) Technical Committees ETSI Projects Partnership Projects etc.
Chairmen, Rapporteurs, Individuals Working Groups (WG) STFs (Specialist Task Forces)
Experts are seconded from the ETSI membership• 25-30 experts in 2006
Produce test specifications (including validation) Short projects
• e.g., 2 man-months maintenance of VoIP tests Long projects
• e.g., UMTS testing 58 man-months per year over 3-5 years 15 – 20 PTCC STFs per year Responsible for approx. 2,7M€ of expert resource per year
ETSI Testing Methodology - SINTESIO Seminar 2006 9
Typical Standards Development Process in ETSI
(Unit) Conformance Testing
Interoperability Testing
Products mature from prototypes to commercial products
ETSI: Development of base standards
Certification
Industry
time
Conformance Test Specifications
Interoperability Test SpecificationsIterative feedback
Iterative feedback
ETSI Testing Methodology - SINTESIO Seminar 2006 10
PTCC Support for Specification Techniques
PTCC provides support to the ETSI TBs on the use of modern specification techniques in ETSI deliverables: UML (Unified Modeling Language) ASN.1 (Abstract Syntax Notation) XML (Extensible Markup Language) MSC (Message Sequence Charts) SDL (Specification and Description Language) Etc.
TTCN (Test and Test Control Notation)
ETSI Testing Methodology - SINTESIO Seminar 2006 11
Different Kinds of ETSI Test Specifications
Conformance Robustness
PerformanceInteroperability
Network Integration
RF/EMC
ETSI Testing Methodology - SINTESIO Seminar 2006 12
Typical ETSI PTCC Areas of Testing
Cellular: GSM and 3G (UMTS) terminals Now includes early IMS
WiFi: HiperMAN, HiperACCESS, WiMax
VoIP: H.323, SIP, SIGTRAN
Service Creation: OSA/Parlay (API, IDL)
IPv6: Core, Security, Mobility, v4-v6
Cordless phones: DECT
Radio communications: TETRA, DMR
Access terminals: FSK, SMS
Broadband: ISDN, DSL
Smartcards: Readers, cards, security modules
Intelligent Transport Systems (ITS): DSRC
Early NGN
Future: More Security, more NGN, GRID ...
ETSI Testing Methodology - SINTESIO Seminar 2006 13
Progressive Testing
All engineering disciplines accept that testing should be applied to progressively complex units From individual components to entire systems
This demands a number of different testing solutions There is no silver bullet
In the two extremes Just testing the components is not enough Just testing the system is not enough
Limitations are economic, not technical What can I afford to test? What can I afford not to test?
What should be covered by standardised test specifications What is the right kind of testing for the job
What kind of ‘thing’ is being tested? Device? System? Service? What is the most economic method(s)? What resources are (or must be made) available?
• Tools, test scripts, testbeds etc.
ETSI Testing Methodology - SINTESIO Seminar 2006 14
Conformance Testing
Is Black-Box testing Stimulation and Response
Tester Device
Point of Control and Observation (PCO)
Reqs.
ETSI Testing Methodology - SINTESIO Seminar 2006 15
Conformance Testing Tests is Usually Layer-byLayer
lat
ii
1 2 3
4 5 67 8 9
* 8 #
latigid
TestSystem
SUT API
MMI
L3
L2
PHY
SUT = System Under Test
IUT = Implementation Under Test
L3
Upper Tester
Lower Tester
ETSI Testing Methodology - SINTESIO Seminar 2006 16
Characteristics of Conformance Testing (1)
Is unit testing Tests a single ‘part’ of a device (e.g., a protocol layer)
• Often incrementally – layer-by-layer
Tests against well-specified requirements For conformance to the requirements in a base
specification or profile (standard) Usually limited to one requirement per test
Tests at a 'low' level At the protocol (message/behaviour) level (bits)
• Or service primitive, API, Interface
Tests over standardised interfaces May not be available to ‘normal’ user
Requires a test system (and executable test cases) Can be expensive (e.g., radio-based test system) Tests under ideal conditions
ETSI Testing Methodology - SINTESIO Seminar 2006 17
Characteristics of Conformance Testing (2)
High control and observability Means we can explicitly test error behaviour Can provoke and test non-normal (but legitimate)
scenarios Can be extended to include robustness tests
It is usually automated and tests are repeatable Conformance Testing is DEEP and NARROW
Thorough and accurate but limited in scope Gives a high-level of confidence that key components of
a device or system are working as they were specified and designed to do
ETSI Testing Methodology - SINTESIO Seminar 2006 18
Limitations of Conformance Testing
Does not prove end-to-end functionality (interoperability) between communicating systems Conformance tested implementations may still not interoperate
• This is often a specification problem rather than a testing problem! Need minimum requirements or profiles
Does not test a complete system Tests individual system components, not the whole
• A system is often greater than the sum of its parts!• Does not test functionality
– Does not test the user’s ‘perception’ of the system
Standardised conformance tests do not include proprietary ‘aspects’ Though this may well be done by a manufacturer with own
conformance tests for proprietary requirements
ETSI Testing Methodology - SINTESIO Seminar 2006 19
Conformance Testing Methodology ISO/IEC 9646 (parts 1-7)
Test Suite (Test Cases)
ATS
Req. checklist
ICS
Test Purposes
TSS & TP
testingTest
Report
logging and
analysis
Test System
Base Standard
(or Profile)
implementation
Product
Implementation Under Test
compilation
Executable Test Suite
(e.g., C++)
Industry
validation
ETSI Testing Methodology - SINTESIO Seminar 2006 20
Implementation Conformance Statement (ICS)
The ICS is a checklist of the capabilities supported by the Implementation Under Test (IUT)
Provides an overview of the features and options that are implemented in any given product
The ICS can be used to select and parameterise test cases and can be used as an indicator of basic interoperability between different products.
Conditional support e.g., Qn must be supported if Q1 supported then otherwise not.
Q# ICS Question Ref Status Support
Q1 Support of Feature F1 [x] Clause a OPTIONAL
Q2 Support of Feature F2 [x] Clause b MANDATORY
: : : :
Qn Support of Message M1 [y] Clause c CONDITIONAL
: : : :
Qk Support of message Parameter P1 [y] Clause d OPTIONAL
ETSI Testing Methodology - SINTESIO Seminar 2006 21
Profile ICS
Q# ICS Question Ref Status Profile Status Support
Q1 Support of feature F1 [x] Clause a OPTIONAL MANDATORY
Q2 Support of feature F2 [x] Clause b MANDATORY MANDATORY
: : : : :
Qn Support of Message M1 [y] Clause c CONDITIONAL MANDATORY
: : : : :
Qk Support of parameter P1 [y] Clause d OPTIONAL N/A
A profile ICS reflects how a (protocol) profile restricts the scope of a base standard by making certain options mandatory or excluding certain options.
ETSI Testing Methodology - SINTESIO Seminar 2006 22
Selecting Tests with the ICS
Q# ICS Question Ref Status Support
Q1 Support of Feature F1 [x] Clause a OPTIONAL
Q2 Support of Feature F2 [x] Clause b MANDATORY
: : : :
Qn Support of Message M1 [y] Clause c CONDITIONAL
: : : :
Qk Support of message Parameter P1 [y] Clause d OPTIONAL
A filled-in ICS can be used to select tests. For example, a test for an OPTIONAL feature which is not supported in a given product will be deselected from the test suite.
ETSI Testing Methodology - SINTESIO Seminar 2006 23
Parameterizing Tests with eXtra Information for Testing (IXIT)
Q# IXIT Question Ref Allowed Values Value
Q1 Network address [x] Clause a Valid IP address 12.34.56.76
Q2 Value of Timer T1 [x] Clause b 1 .. 7 s 5s
: : : : :
A filled-in IXIT can be used to parameterize tests. The Value column is used to specify explicit IUT or test system dependent values.
ETSI Testing Methodology - SINTESIO Seminar 2006 24
The Requirements Catalogue
Each Requirement is categorized as follows (example from IPv6): Requirement type:
• Mandatory (MUST, MUST NOT)
• Recommended (SHOULD, SHOULD NOT)
• Optional (MAY, MAY NOT)
Requirement target:• E.g., Host, Router, Node (Host or Router)
Requirement text Functional groupings
• E.g., Process Fragmented packet, Generate ICMPv6 Error Typetc.
A scalable database containing all requirement elements Web interface Browsing by function User-defined search Links to RFC and related test specification
ETSI Testing Methodology - SINTESIO Seminar 2006 26
Conformance Test Purposes
Test Purposes (TP) are natural language, precise descriptions of the purpose of the test in relation to a particular (base standard) requirement
Define the functionality being tested—the WHAT Do not define HOW to test the function Grouped into a logical structure (Test Suite Structure)
TSS & TP
One Requirement may spawn several TPs A conformance TP is written on the protocol level Specified in
Natural language, or ETSI’s Test Purpose Language (TPLan) They are not test code
ETSI Testing Methodology - SINTESIO Seminar 2006 27
TPLan Example for Conformance
TP id : TP_COR_0047_01Summary : ‘hop limit of one'RQ Ref : RQ_COR_0047Config : CF_02_CTC Ref : TC_COR_0047_01ensure that {--Stimulus when { IUT receives ‘Ipv6 packet' from ‘Host' containing ‘IPv6 Header' indicating ‘Hop limit' set to ‘1‘ }--Expected response then { IUT sends ‘ICMPv6 Time Exceeded' to ‘Host‘
containing ‘ICMP code' set to ‘ZERO‘ } }
ETSI Testing Methodology - SINTESIO Seminar 2006 28
Conformance Test Cases
Detailed TTCN-3 test script that implements test purpose can be compiled and executed
Is the HOW to test not WHAT to test Composition
a preamble test body (i.e., implementation of the Test Purpose) A postamble
Assigns test verdicts Handles unexpected behaviour as well as the behaviour in
the test purpose Can be distributed over parallel test components Can be entirely automated Configurable at run-time, e.g., SUT address
ETSI Testing Methodology - SINTESIO Seminar 2006 29
Abstract Test Suite
The Test Suite (ATS) is the collection of all Test Cases Most ETSI Test Suites are written in TTCN
Predominantly in TTCN-2 Progression to TTCN-3 for new work Not RF testing (generally not physical layer)
TTCN-3 Testing and Test Control Notation Allows tests to be specified in detail (test code) that is
independent of the (eventual) real test system on which it may be run
• i.e., TTCN-3 will run on any test system that supports a standardised TTCN-3 execution environment
ETSI Testing Methodology - SINTESIO Seminar 2006 30
What is TTCN-3?
Internationally standardized testing language Look and feel of a regular programming language Allows unambiguous implementation of tests Independent of execution environment due to
separate test system interface standards Wide range of applications from mobile (radio)
communications to Internet to software Good tool support Good book:
http://eu.wiley.com/WileyCDA/WileyTitle/productCd-0470012242.html
ETSI Testing Methodology - SINTESIO Seminar 2006 31
Example TTCN-3 Test Case
testcase TC_COR_0047_01() runs on Ipv6Node system EtherNetAdapter { f_cf02Up(); // Configure test system for HS->RT // No preamble required in this case f_TP_HopsSetToOne(); // Perform test // No postamble required in this case f_cf02Down(); // Return test system to initial state}
function f_TP_HopsSetToOne() runs on Ipv6Node { var Ipv6Packet v_ipPkt; var FncRetCode v_ret := f_echoTimeExceeded( 1, v_ipPkt ); if ( v_ret == e_success and v_ipPkt.icmpCode == 0 ) { setverdict(pass);} else { setverdict(fail); }}
function f_echoTimeExceeded(in UInt8 p_hops, out Ipv6Packet p_ípPkt ) runs on Ipv6Node return FncRetCode { var Ipv6Packet v_ipPacket; var FncRetCode v_ret; ipPort.send( m_echoReqWithHops(p_hops) ); alt { [] ipPort.receive( mw_anyTimeExceeded ) -> value p_ipPkt { return e_success } [] ipPort.receive { return e_error } }}
ETSI Testing Methodology - SINTESIO Seminar 2006 32
Executable Test Suites
Executable test suites are compiled from the abstract Either direct to binary, or More commonly to some intermediate programming
language (C++, Java, etc.)
ETSI does not deliver executables, but We try to ensure all code can be compiled And validated by executing the tests against a real
implementation on at least one test system• 5 test systems in the case of UMTS, for example
ETSI Testing Methodology - SINTESIO Seminar 2006 33
TTCN-3 Test System
ENCODER
DECODER
(N-protocol specific)
Adaptation Layers
TRITTCN-3 Runtime Interface
TCI TTCN-3 Control Interface
The Industry
Test Suite in TTCN-3
(source)
TTCN-3 Test Suite
(object)
Compilation
PICS etc
(reqs. catalogu
e)
Parameterisation
Selection
Underlying Protocol Stack
(N-1)
Connection to the SUT
Control / Logging
User
ETSI Testing Methodology - SINTESIO Seminar 2006 34
Interoperability Testing End-to-End Interoperability of devices
What is being tested? Assignment of verdicts?
Need to distinguish between Formal interoperability testing And informal interoperability events used to validate/develop
technologies
Device1 Devicen
Device2
ETSI Testing Methodology - SINTESIO Seminar 2006 35
SUT
API
MMI
L3
L2
PHY
API
MMI
L3
L2
PHY
QE = Qualified
Equipment
EUT = Equipment Under Test
Interoperability Testing Tests Functionality Between Devices
Interoperability Testing(of terminals)
1 2 3
4 5 67 8 9
* 8 #
QE EUT
1 2 3
4 5 67 8 9
* 8 #
Means of Communication (MoC)
ETSI Testing Methodology - SINTESIO Seminar 2006 36
Characteristics of Interoperability Testing
Is system testing Tests a complete device or a collection of devices
Shows that (two) devices interoperate Albeit within a limited scenario
Tests at a ‘high’ level (as perceived by users) Tests the ‘whole’, not the parts
• e.g, protocol stacks + applications Tests functionality
• Shows function is accomplished (but not how)
Does not necessarily require a test system Uses existing interfaces (standard/proprietary)
Interoperability Testing is BROAD and SHALLOW Less thorough but wide in scope Gives a high-level of confidence that devices (or components in
a system) will interoperate with other devices (components)
ETSI Testing Methodology - SINTESIO Seminar 2006 37
Limitations of Interoperability Testing
Does not prove interoperability with other implementations with which no testing has been done A may interoperate with B and B may interoperate with C. But it
doesn’t necessarily follow that A will interoperate with C. Combinatorial explosion
Does not prove that a device is conformant Interoperable devices may still interoperate even though they
are non-conformant
Cannot explicitly test error behaviour or unusual scenarios Or other conditions that may need to be forced (lack of
controllability) Has limited coverage (does not fully exercise the device)
Not usually automated and may not be repeatable
ETSI Testing Methodology - SINTESIO Seminar 2006 38
Interoperability Testing Methodology ETSI TS 102 237
Interop Test Cases
Test Report
logging and
analysis
Base Standard (or Profile)
implementation
Product 1
Qualified Equipment
Product 2
Equipment Under Test
implementation
IFS (functional checklist)
Interop Test Purposes
testing
ETSI Testing Methodology - SINTESIO Seminar 2006 42
Example Interoperability Test Description
Identifier:
Step Step
Test Purpose: TP_COR_1100_01 Reference: RQ_COR_1100 Configuration:
TD_COR_1100_01Summary EUT reassembles a fragmented packet of an original length less than 1500 octets
with { 'the MTU on Link1 set to 1400 octets' } ensure that { when { QE is requested to 'send data requiring a packet length greater than 1500 octets' } then { EUT indicates 'receipt of the same data without modification' } }
Pre-Test Conditions:
MTU set to 1400 octets on link1
Verdict
1
2
3
Cause QE to send an Echo Request to EUT with a packet size of 1450 octets and with each octet set to the hexadecimal value "F0"
CF_011_I
Check: Does protocol monitor show that the Echo Request was sent from QE to EUT?
Yes No
Check: Does QE receive an Echo Reply from EUT with the packet length the same as the Echo Request and with each octet containing the hexadecimal value "F0"?
Yes No
Pass Fail
Test Description
Observations
ETSI Testing Methodology - SINTESIO Seminar 2006 43
Automated IOP Testing Using TTCN-3
TestDriver
(TTCN-3)
TestDriver
(TTCN-3)
TestController(TTCN-3)
AT Command
s
AT Command
s
ETSI Testing Methodology - SINTESIO Seminar 2006 44
IOT with Conformance Verifiction(aka NIT - Network Integration Testing)
NWC1
NWC2
NWC3
Terninal E2E tests driven by human users
Terminal E2E tests over internal product API (automated)
UNI UNI
Network E2E tests over the UNI (automated)
Ra Rb
Conformance verification of reference points
ETSI Testing Methodology - SINTESIO Seminar 2006 45
Example NIT for IMS
End-to-End Testing
Mw Mw
UE1
Um Um
UE2
P-CSCF1
S-CSCF1 S-CSCF2
Cx
Mw
Access Network
HSS
I-CSCF P-CSCF2 Access Network
ETSI Testing Methodology - SINTESIO Seminar 2006 46
Conformance and Interoperability Testing are Complementary
ETSI experience As you move up a system stack the emphasis should change
from conformance to IOT
Lower layer protocols, infrastructure Emphasis on conformance
Middleware, enablers Combination of Conformance + IOT
Services, applications, systems Emphasis on IOT
Conformance testing as a pre-requisite to IOT
Interop Testing
Conformance Testing
Interoperability
ETSI Testing Methodology - SINTESIO Seminar 2006 48
TTCN-3 Standardshttp://www.ttcn-3.org
ES 201 873-1 (Z.140) TTCN-3 Core Language
ES 201 873-2 (Z.141) TTCN-3 Tabular Presentation Format (TFT)
TR 101 873-3 (will eventually be ES 201 873-3) (Z.142) TTCN-3 Graphical Presentation Format (GFT)
ES 201 873-4 (Z.143) TTCN-3 Operational Semantics
ES 201 873-5 TTCN-3 Runtime Interface (TRI)
ES 201 873-6 TTCN-3 Control Interfaces (TCI)
ES 201 873-7 and upwards Using ASN.1, XML, IDL, C/C++, with TTCN-3