10th TTCN-3 User Conference, 7-9 June 2011, Bled, Slovenia AUTOSAR Conformance Tests - Feedback on...

10th TTCN-3 User Conference, 7-9 June 2011, Bled, Slovenia

AUTOSAR Conformance Tests - Feedback on their development and utilization

Alain Feudjio-Vouffo, Fraunhofer Fokus (Germany), [email protected] Alexandra Faucheux, Kereval (France), [email protected] Mathieu Morvan, Kereval (France), [email protected] Alain Ribault, Kereval (France), [email protected] Ashish Jadhav, Patni Computer Systems (India), [email protected] Philippe Robin, Technoveo (France), [email protected]

Facing Up to the Future of Testing

AUTOSAR conformance tests - feedback on their development and utilization10th TTCN-3 User Conference, 7-9 June 2011, Bled, Slovenia

Content of the presentation

Introduction: AUTOSAR CTSpecs release status AUTOSAR CTSpecs & abstraction from hardware platform Metrics on AUTOSAR CTSpecs TTCN-3 code and test environment Combination of conformance test and source code coverage Looking into the future

Theoretical view on combination and reuse of TTCN-3 suites to create new ones From ATS for module to ATS for stack and integrated software platform

Example of deployment in the automotive industry Conclusion

2


Introduction - AUTOSAR CTSpecs release status

AUTOSAR conformance test specifications (CTSpecs) have been released on 15 April as part of Release4.0R0002

Conformance tests are available for 43 modules (including the RTE)

A CTspec contains the specification of the conformance test suite the TTCN-3 scripts for dynamic test cases the configuration sets to be used when executing the Conformance Test Suite

Number of test cases 6820 Number of dynamic tests (TTCN3) 2754 Number of static tests 4066

3


AUTOSAR CTSpecs & abstraction from hardware platform (1/2)

4

TTCN-3 files

Adapters

COM

Module Under Test

HW Specific

COM

Module Under Test

AdaptersGenerator

TTCN-3Executable

TTCN-3Compiler

KEREVAL' Platform

ETSon PC

Instrumented MUTon board

Adapters are generated ETS is hardware independent

HW Specific

COM

Module Under Test

HW Specific

COM

Module Under Test

HW Specific

COM

Module Under Test

Execution of the CTS


Metrics on AUTOSAR CTSpecs TTCN-3 code and test environment

55

CTSpec, TTCN-3 scripts and Configuration Set(s)

Test Management System Reporting

CAPITALIZED or GENERATED

Test suite execution

results

Software, Hardware and Test design tool

Test suite

execution

Attestation

Configurations and environment

Sources

CUSTOMER

MANUAL EFFORTBSW preparation (configuration)

BSW integration

IntegrationReception

Effort during Execution

5%

95%

Manual Effort

Capitalized orGenerated

Source Code Inspection

Configuration Inspection

Operation Signature Tests

Dynamic Tests


AUTOSAR CTSpecs & abstraction from hardware platform (2/2)

Test Memory Management allows Test Suite to access memory of the board

6

HW Specific

COM

Module Under Test

Instrumented MUTon board

Test Memory Management

Adapters

COM

TTCN-3Executable

ETSon PC

Allocate / Deallocate

Read / Write / Compare /* ... */

var PointerAddr_ v_AddrPointer;

pt_TestMemoryAccess.call(TestMemoryAccess_Allocate:{v_Size}, nowait); alt { []pt_TestMemoryAccess.getreply(TestMemoryAccess_Allocate:{v_Size}) -> value v_AddrPointer { /* ... */ } } /* ... */ pt_Fee.call( Fee_Read:{v_BlockNumber, v_BlockOffset, v_AddrPointer, v_Size}, nowait ); /* ... */

TTCN-3 files

module FeeJo


Combination of conformance test and source code coverage

7

A Code Coverage Analysis tool (RTRT) is used during Conformance Test execution

Provides An indicator of tests relevance according

to structure An effort estimation to go over functional

validation from conformance testing Information on dead code and unspecified

code

Required in Safety Validation context ISO 26262 and DO178B

Thanks to coverage measurement, conformance testing can be part of validation process


Reuse of the TTCN-3 suites

Two types of reuse are envisaged for the module-level TTCN-3 ATS that are available for the 43 modules.

1) Reuse of the suites for creating the ATS for the later revisions of the AUTOSAR standard during the maintenance phase of the current release, and

2) Reuse of the suites for creating the ATS for the future releases of the AUTOSAR standard

The ATS has been created having maintainability as a major design criterion. TTCN-3 provides the high level artifacts in the ATS to enable the reuse.

Traceability between the specifications and the test cases and configuration parameters has been created to simplify the upgrade process.

8

RE-USE of Artifacts in the ATS

AUTOSAR Rel 4.0

Rev 2 Rev 3 …AUTOSAR Rel 4.1

Rev 0 Rev 1 … …


Combination of TTCN-3 suites to create new ones – extensibility

Different users of AUTOSAR might be interested in extending the existing ATS by having more tests that are vendor specific and do not exist in the current ATS for internal test compliance

The TTCN-3 suites are well structured and provide an ease for combining with other vendor specific test cases to extend the tests for internal test compliance

The TTCN-3 suite is well structured and provides ease of extensibility for test cases, configuration parameters and combinations of configuration parameters

9

AUTOSAR ATS

VendorSpecific Module Tests

Additional Tests

Required by OEMs

Extensibility of TTCN3 AUTOSAR ATS

(for internal test compliance)


Higher level Test Suites

The ATS is currently at module level Using TTCN-3 provides ease of extensibility to combine

test-cases and reuse code with additional test cases to obtain a test suite for a set of inter-connected modules

Some of the stubs and drivers would have to be replaced by actual modules and suites used for testing of stacks like the CAN stack, the Flexray stack, the communication stack, etc. can thus be prepared

In such type of higher level suites various AUTOSAR modules can be simultaneously tested

The test cases of the ATS can also be used to develop test suite for testing of a full AUTOSAR stack

The test cases of the ATS can also be used for developing test suites for testing of AUTOSAR compliant software components

10

FullAUTOSAR STACK

ATSfor

Combination of

Modules

Module ATS


Example of deployment in the automotive industry

Feedback on executions Average of 10 non conformities detected per Module Under Test at first CT execution

Typical non conformities detected on each module

11

Configuration data set

Source Code inspection

Static Compile Time

Dynamic test suite

• Parameter container error• Parameter range error• Parameter type error• Parameter missing

• Enum definition error• Parameter type error• include missing

• signatures or prototypes error• parameters code generation error (from XML information)• bad condition of function deactivation

• argument return• unexpected API invoked• expected API not invoked

Such non conformities have not been detected during validation of the module


Non-regression tests for any changes in:- Configuration or Implementation, - HW platform

Example of deployment in the automotive industry

12

Integrated with customers development plan Multi run conformance tests

Unit TestsUnit TestsCodingCoding

Conformance checking according to AUTOSAR description files (non-TTCN-3 tests)

Module Conformance checking

Confirmation test and attestation delivery

Functional design

SW Architecture design

Integration Tests

Functional Tests


Conclusions

Hardware dependent ETS is obtained from ATS and is used for testing the MUT, this ensures that the AUTOSAR CTSpecs are portable

The use of AUTOSAR CTSpecs TTCN-3 code and test environment by CTAs need a very low percentage of manual activities and up to 95% of activities can be tool-based

AUTOSAR Conformance testing can be combined with other tests like source code coverage so that independent test runs can be reduced resulting in overall reduction in test execution time

AUTOSAR ATS has maintainability and extensibility enabled by use of TTCN3 for efficient reuse in creating new ATS for future releases, ehancements and higher level testing

AUTOSAR Conformance testing can be integrated in the overall AUTOSAR development process so that testing can be performed in early stages of the development and can result in detecting non-conformances even at design stage which would improve the development efficiency

13

10th TTCN-3 User Conference, 7-9 June 2011, Bled, Slovenia AUTOSAR Conformance Tests - Feedback on...

Documents

Transcript of 10th TTCN-3 User Conference, 7-9 June 2011, Bled, Slovenia AUTOSAR Conformance Tests - Feedback on...