AC-B Air Traffic Control & Business Systems GmbH © AC-B GmbH 2008 AMHS Conformance Testing AMHS...

AC-B Air Traffic Control & Business Systems GmbH

© AC-B GmbH 2008

AMHS Conformance Testing

AMHS Implementation Workshop Chennai, India, 15 – 16 December 2008

Rainer Hoffmann, AC-B GmbH

© AC-B GmbH 2008

AC-B Air Traffic Control & Business Systems GmbH Presentation Contents

EUR AMHS testing strategy

AMHS conformance test environment

Test experience

© AC-B GmbH 2008




Test experience

© AC-B GmbH 2008

AC-B Air Traffic Control & Business Systems GmbH EUR AMHS Testing Strategy

ICAO EANPG (AFSG) developed standards for AMHS testing What exactly will be tested? Which tests are performed? How are tests executed (under which conditions)? -> Make tests fair and transparent to everybody

3 Test phases are defined in the EUR AMHS Manual1. Conformance testing

2. Interoperability testing

3. Pre-operational testing

© AC-B GmbH 2008

AC-B Air Traffic Control & Business Systems GmbH Test phases (1)

1. Conformance tests wide test scope of standard (and optional) functions soon after system installation

(may be done in parallel with FAT/SAT)

2. Interoperability tests limited test scope - basic and important functions bilateral and trilateral tests when the network connection is established

3. Pre-operational tests a few check-ups close before cut-over run AFTN and AMHS link in parallel

© AC-B GmbH 2008

AC-B Air Traffic Control & Business Systems GmbH Test phases (2)

examine one system against the

specification (standard documents)

can be performed locally

examine two or more systems

communicating with each other

distributed over two or more locations

Conformance testing Interoperability testing

IUTSystem A

Test Tool

IUTSystem A

IUTSystem C

IUTSystem B

IUT:= Implementation Under Test

test automation possible

includes „negative“ test cases

bilateral coordination of staff presence

only „positive“ test cases

© AC-B GmbH 2008

AC-B Air Traffic Control & Business Systems GmbH Conformance testing saves overall test effort (1)

Example: 59 interoperability tests with 8 partners (neighbour domains)

Inte

rope

rabi

lity

Tes

ts

1

Inte

rope

rabi

lity

Tes

ts

2

Inte

rope

rabi

lity

Tes

ts

3

Inte

rope

rabi

lity

Tes

ts

4

Inte

rope

rabi

lity

Tes

ts

5

Inte

rope

rabi

lity

Tes

ts

6

Inte

rope

rabi

lity

Tes

ts

7

Inte

rope

rabi

lity

Tes

ts

8

Numberof tests

59

18

In this example

472 tests would need to be

coordinated

and performed

© AC-B GmbH 2008

AC-B Air Traffic Control & Business Systems GmbH Conformance testing saves overall test effort (2)

Inte

rope

rabi

lity

Tes

ts

1

Inte

rope

rabi

lity

Tes

ts

2

Inte

rope

rabi

lity

Tes

ts

3

Inte

rope

rabi

lity

Tes

ts

4

Inte

rope

rabi

lity

Tes

ts

5

Inte

rope

rabi

lity

Tes

ts

6

Inte

rope

rabi

lity

Tes

ts

7

Inte

rope

rabi

lity

Tes

ts

8

Con

form

ance

Tes

ts ...and reduce the number of interoperability tests

Start with conformance tests...

In this example

the overall number

has been reduced to

203 tests( = 59 + 144 )

Inte

rop

. T

es

ts

Inte

rop

. T

es

ts

Inte

rop

. T

es

ts

Inte

rop

. T

es

ts

Inte

rop

. T

es

ts

Inte

rop

. T

es

ts

Inte

rop

. T

es

ts

Inte

rop

. T

es

ts

Numberof tests

59

18

© AC-B GmbH 2008


Test Strategy A: Perform comprehensive interoperability testing

Number of test cases

Test cases per day

59 6

Test

cases Test days 2Partners 118 20 4Partners 236 39 8Partners 472 79

16Partners 944 157 40Partners 2360 393

Test effort comparison (1)

All test cases have to be performed with each partner!

© AC-B GmbH 2008

AC-B Air Traffic Control & Business Systems GmbH Test effort comparison (2)

Test Strategy B: Perform comprehensive conformance testing and a subset of interoperability testing Conformance tests Interoperability tests


Test cases per day


Test cases per day

59 12 18 6

Test

cases Test daysTest

cases Test daysOverall

days Savings2Partners 59 5 36 6 11 44%4Partners 59 5 72 12 17 57%8Partners 59 5 144 24 29 63%

16Partners 59 5 288 48 53 66%40Partners 59 5 720 120 125 68%

© AC-B GmbH 2008

AC-B Air Traffic Control & Business Systems GmbH Reasons for a conformance test tool

Improved test coverage, more test capabilities overcome scenario limitations encountered in the FIRST

interoperability trials enforce fault situations (= negative testing) to check IUT robustness,

e.g. erroneous AMHS parameters

Better fault analysis test tool can analyse captured AMHS messages, reports and probes test tool can act as a reference system

Easier regression testing less manual interaction, tests can be repeated very exactly (and

quickly) no need to use AMHS links to neighbour domains to verify your

software updates

© AC-B GmbH 2008




Test experience

© AC-B GmbH 2008

AC-B Air Traffic Control & Business Systems GmbH AMHS Test Tool Concept

Build a test tool with all necessary components to simulate an operational AMHS environment of a COM centre

Provide a test platform for loadable test scenarios modification or enhancements of standard tests reuse of test messages and test sequences easy creation of new tests

Minimise manual intervention by test automation

© AC-B GmbH 2008


COM Centre

AFTN/AMHS

Gateway

ATS

Message

Server

(MTA)

Generic COM-Center Environment (AMHS view)

Local AFTN

User Terminal

Monitor &

Control Position

AFTN Environment

AFTNUser (s)

AFTNUser (1)

AFTNUser (2)

AFTN

CenterAFTN (CIDIN)

interface

AMHS Environment

Local AMHS

User Agent

(UA)

Use

r A

gen

ts (

UA

)

UA 1

UA 2

UA 3

UA 4

UA m

UA o

AMHS (P1)interfaces

Message Transfer Agents (MTA)

MTA-1

MTA-n

MTA-2

© AC-B GmbH 2008


AFTN/AMHS

Gateway

ATS

Message

Server

(MTA)

AMHS Test Interfaces

Local AFTN

User Terminal

Monitor &

Control Position

AFTN Test Environment

AFTN (CIDIN)interface

AMHS Test Environment

AMHS (P1)interfaces

Local AMHS

User Agent

(UA)

IUT

TestMTA-1

TestMTA-3

TestMTA-2

AFTN

Test Appl.

Test Interfaces

© AC-B GmbH 2008


AMHS Test Tool

AFTN/AMHS

Gateway

ATS

Message

Server

(MTA)

AMHS Test Tool Architecture

Local AFTN

User Terminal

Monitor &

Control Position

AFTN

Test Appl.AFTN (CIDIN)

interface

AMHS (P1)interfaces

TestMTA-1

TestMTA-3

TestMTA-2

Local AMHS

User Agent

(UA)

IUT

Testlog

Me

ss

ag

eS

ou

rce

/Sin

k

Viewer/Editor

User Interface

Te

st

Co

ntr

ol

an

d E

va

lua

tio

n

Ap

pli

ca

tio

n

Test Interfaces

© AC-B GmbH 2008

AC-B Air Traffic Control & Business Systems GmbH Features of AC-B‘s AMHS Test Tool

Test interfaces AMHS over TCP/IP AMHS over ATN (option) AFTN over CIDIN/X.25 PVC AFTN over X.25 (option) AFTN over TCP/IP (option)

Test cases 59 conformance test scenarios according to EUR AMHS Manual (App. D) More than 200 prepared AMHS and AFTN test messages

Functions Generation of IPMs, IPNs, reports, probes with user-defined parameters Verification of IPMs, IPNs, reports, probes received from the IUT Automatic response to received messages and probes: DR, NDR, RN, NRN

© AC-B GmbH 2008

AC-B Air Traffic Control & Business Systems GmbH AMHS Conformance Test Scope

Test groups Submission operations Delivery operations Transfer operations Gateway operations

AMHS -> AFTN AFTN -> AMHS

Test classes Verification of normal communication Handling of communication errors

(negative testing – robustness tests)

© AC-B GmbH 2008

AC-B Air Traffic Control & Business Systems GmbH AMHS Test Tool - HMI examples (1)

Table tree view for test steps Details for a selected test stepRun control buttons

Indication of test step success / failure

© AC-B GmbH 2008


IPM

AMHS Test Tool - HMI examples (2)

Test Control and Evaluation Application

AMHS Viewer/Editor

MTE

IPN

Probe

Report

© AC-B GmbH 2008

AC-B Air Traffic Control & Business Systems GmbH AC-B‘s AMHS Test Services

AC-B offers AMHS test services to ANSPs and AMHS suppliers

The basic AMHS conformance test service package includes preparation of the AMHS test tool for the on-site test on-site test session to run a set of standardized AMHS conformance

test procedures test evaluation and test report test certificate (= conformance test statement)

Customized test service packages can be offered to perform regression testing, or to verify optional, customer-specific AMHS features

© AC-B GmbH 2008

AC-B Air Traffic Control & Business Systems GmbH AMHS conformance test sessions

9 conformance test sessions in 2006, 2007 and 2008 5 initial test session and 4 regression test regression tests were due to

normal (planned) product release upgrades, or unsatisfying initial test results

5 AMHS systems were tested DFS (Germany), Austrocontrol (Austria), MATS (Malta),

AFSBw (German Military), NAVIAIR (Denmark)

These 5 systems came from 5 different suppliers

© AC-B GmbH 2008

AC-B Air Traffic Control & Business Systems GmbH AMHS conformance test statistics

AMHS conformance test 1 2 3 4 5 6 7 8 9

Number of failed tests ( x out of 51 or 59): 12 3 4 20 0 4 0 8 5Submission, Delivery and Transfer Tests

1 CT101 Forward a submitted IPM2 CT201 Deliver an IPM to a local AMHS user3 CT202 Deliver an IPM containing erroneous ATS-message-header or ATS-message-text format4 CT203 Deliver an IPM containing empty or invalid IPM heading fields5 CT301 Transfer messages (IPMs and IPNs) 1 16 CT302 Transfer a report7 CT303 Transfer a probe 1 18 CT304 Reject a message, if DL expansion is prohibited 1 19 CT305 Loop detection

10 CT306 Generate a NDR, if transfer fails 1

Gateway Tests (AMHS to AFTN)11 CT401 Convert an incoming IPM to AFTN format12 CT402 Convert an IPM containing optional-heading-information in the ATS-message-header 113 CT403 Generate a DR for a successfully translated IPM 1 114 CT404 Generate a NDR, if implicit conversion is prohibited 1 115 CT405 Generate a NDR, if the ATS-message-header has a syntax error16 CT406 Convert or reject an IPM, if the ATS-message-text contains more than 1800 characters17 CT407 Convert or reject an IPM, if the ATS-message-text contains lines with more than 69 characters 118 CT408 Convert or reject an IPM, if the ATS-message-text contains characters not allowed by ICAO Annex 10 119 CT409 Reject an IPM with multiple body part20 CT410 Distribute an IPM to AMHS and AFTN users21 CT411 Expand a DL addressing both AMHS and AFTN users 1 1 122 CT412 Reject or split an incoming IPM addressing more than 21 AFTN users23 CT413 Remove an unknown address before conversion into AFTN format 1 124 CT414 Convert an incoming AFTN acknowledgement25 CT415 Incoming AFTN acknowledgement with unknown AFTN originator26 CT416 Incoming AFTN acknowledgement relating to a subject message without receipt-notification request27 CT417 Incoming AFTN acknowledgement without related subject message28 CT418 Convert an AFTN SVC “Unknown Addressee Indicator” to a NDR29 CT419 Incoming AFTN SVC “Unknown Addressee Indicator” without related subject message30 CT420 Processing of an incoming SVC QTA RPT Message 1 131 CT421 Probe Conveyance Test 1 1 132 CT422 Reject an IPM with unsupported content-type 1 1 1 1 133 CT423 Processing of the original-encoded-information-types (EIT) 1 1 1 1 1 134 CT424 Incoming IPM with extended body part of type "ia5-text-body-part" 1 1 1 135 CT425 Incoming IPM with extended body part type "general-text-body-part" and ISO 646 repertoire 1 136 CT426 Incoming IPM with extended body part type "general-text-body-part" and ISO 8859-1 repertoire 1 1

Gateway Tests (AFTN to AMHS)37 CT501 Convert an AFTN user message to AMHS format38 CT502 Convert an AFTN user message containing optional heading information39 CT503 Generate an AFTN service message of the type “Unknown Addressee Indicator”40 CT504 Incoming AFTN user message with unknown originator indicator41 CT505 Convert a receipt notification42 CT506 Incoming non-receipt notification43 CT507 Generate a NDR as a result of misrouted RN 1 144 CT508 Convert a non-delivery report (NDR)45 CT509 NDR conversion process failures 1

Naming and Addressing Tests46 CT601 Address conversion from AMHS CAAS- and XF-addresses to AFTN addresses47 CT602 Address conversion from AFTN addresses to AMHS CAAS- and XF-addresses48 CT603 Reject an IPM with invalid recipient address (CAAS like) 1 1 149 CT604 Reject an IPM with invalid recipient address (XF like)50 CT605 Reject an IPM with invalid originator address (CAAS like) 1 1 151 CT606 Reject an IPM with invalid originator address (XF like) 152 CT607 Asymmetric address conversion from AMHS CAAS- and XF-recipient addresses to AFTN addresses53 CT608 Asymmetric address conversion from AMHS CAAS- and XF- originator addresses to AFTN addresses

Handling of Non-Delivery Reports (NDR)54 CT701 Transfer a non-delivery report (NDR)55 CT702 Deliver a non-delivery report (NDR) to an AMHS user56 CT703 Handling of received non-delivery report (NDR) in the AFTN/AMHS gateway57 CT704 Transfer a NDR containing non-standard reason or diagnostic codes 158 CT705 Deliver a NDR containing non-standard reason or diagnostic codes to an AMHS user agent 159 CT706 Handling of NDR containing non-standard reason or diagnostic codes in the AFTN/AMHS gateway 1

© AC-B GmbH 2008

AC-B Air Traffic Control & Business Systems GmbH Experiences made

The first (initial) test of a product takes time Connection set-up is never easy b e p a t i e n t ... ! Inital test runs never flow smoothly do not get frustrated!

A renewed (second) test run can be done very efficiently (even in one day)

Some major problems that occured were not specific to the AMHS standard When we defined AMHS tests, we believed that base X.400

MHS products are mature that was wrong! Tests revealed problems also on X.400 level and even ISO

transport layer s u r p r i se ?

© AC-B GmbH 2008

AC-B Air Traffic Control & Business Systems GmbH Conclusions and recommendations (2)

Conformance testing is only one step in a series of tests stepwise testing at supplier and customer premises required conformance tests do not replace FAT/SAT and are no guarantee for

perfect AMHS operation (due to limited scope and specific test parameters)

The system‘s configuration is an issue the conformance test environment uses test parameters, e.g. for mta name,

domain names, AFTN and AMHS addresses later configuration changes may result in different behaviour (e.g. gateway

use of address mapping tables) configuration control is the supplier‘s responsibility

Recommendation: Perform the official conformance tests at the target system, best at

ANSPs premises

© AC-B GmbH 2008

AC-B Air Traffic Control & Business Systems GmbH Conclusions and recommendations (2)

Today‘s standardized conformance tests are absolutely necessary but not necessarily sufficient In many cases non-conformant behaviour was detected by the test tool and

the software could be updated (improved) before the official test run but... test scope limited mainly to AFTN/AMHS gateway and MTA only few MTA submission and delivery tests no standardized test for UA-MS/MTA and UA-UA communication

Recommendation: Enhance the scope of standardized conformance tests

© AC-B GmbH 2008

AC-B Air Traffic Control & Business Systems GmbH Invitation

Please consider AMHS support from AC-B in your implementation projects

AMHS Conformance Test Services

AMHS Seminars

The AMHS Book

AC-B Air Traffic Control & Business Systems GmbH © AC-B GmbH 2008 AMHS Conformance Testing AMHS...

Documents

Transcript of AC-B Air Traffic Control & Business Systems GmbH © AC-B GmbH 2008 AMHS Conformance Testing AMHS...