How to comply with ISO26262 efficiently – the case for highly automated testing

By Steve Barriault

Technical Sales & Marketing Manager –

Asia

© Vector Software Inc, all rights reserved.

Company introduction

• Founded in 1990 in Rhode Island by embedded engineers

• First release of VectorCAST in 1994

• VectorCAST provides unit and integration testing, as well as system test

coverage tools that are uniquely automated

• These tools can be run on a host, a simulator or directly on a target

• Have worldwide offices and representation

Georgia

Rhode Island

Arizona

Japan

Korea

China

France

Italy

London

Israel

SwedenNetherlands

India

Software testing under ISO 26262

• ISO 26262 is IEC 61508 adapted to the needs of

the Automotive Industry

• Adopts a similar approach to software testing and

code coverage requirements to other, longer-lived

standards (such as DO-178B)

• The challenge: meet its requirements in an

industry where deadlines are coming up much

faster than in Aerospace

– In order to achieve success, efficiency will be key!

Our experience with standards

• Our roots are in the Aerospace industry, where our

15+ year of experience enabled us to work with all

the leading organizations

• Our Automotive business is expanding quickly, with

a growing list of companies using VectorCAST

Tests in ISO 26262B

y S

co

pe • Unit test

• Integration test

• System testB

y G

oa

l • Requirements-based testing

• Interface testing

• Fault injection test

• Resource usage test(*)

Me

tric

Activitie

s • Structural code coverage

• Test-requirement association

(*) Source: Table 12 and 15:

Methods for software unit/integration testing

Requirement-base testing

• Ensures that the software fulfill its mission

– Sometimes called functional test

• Strongly recommended for all levels of ASIL, for both unit

and integration testing

• Stubbing can be performed to enhance your ability to test

low-level requirements in isolation

– They “replace” your existing code so you can better control your

inputs and outputs in the code

– But stubs can take a long time to be generated with scripting-based

tools

– With VectorCAST, the stubs are automatically generated in seconds,

with no user input whatsoever

Requirement-base testing

• The link between the requirement and the test case should

ideally be documented

– In VectorCAST, it can be. Our unit and integration tests can be linked

to specific requirements.

– The test case data (PASS|FAIL) that demonstrate requirements can

be uploaded to a requirement management system such as DOORS

VectorCAST

Test cases

DOORS®

requirements

Req. 1

Req. 2

Req. 3

Test 1

Test 2

Test 3

Execution on

host,

simulator

or target

PASS/FAIL

PASS/FAIL

PASS/FAIL!

External interface test

• External Interface Testing is a subset of functional testing.

• Highly recommended for all ASIL level, both unit and

integration

• It verifies that:

– Functions sent data out in the appropriate format and delivery

mechanism

– Functions that receive data in the appropriate format perform

correctly

– That the behavior when receiving data that is not formatted directly is

known

• Can also be tested by VectorCAST

Other types of test

• Fault injection test:

– Voluntarily inject arbitrary faults to test safety mechanisms

(ex: by corrupting values of variables)

– Recommended for unit/integration testing,

strongly recommended ASIL D (and C in integration)

– In VectorCAST, can provide test cases that have faulty values and

verify that the defensive code gets invoked

• Resource usage test:

– Often only doable on target or at least simulator

– Recommended for unit/integration testing

– Strongly recommended ASIL D

– Our superior degree of target integration can also help you do some

of this, but perhaps not all

Generating test case values

• Based on requirements

– Strongly recommended for all ASIL, unit/integration test

• Equivalence classes:

– This method may be used to partition possible input values of

external interfaces

– Strongly recommended ASIL B, C and D

– VectorCAST has a facility to generate automatically such partitioned

test cases

• Error guessing:

– Here, the tester tries to test errors that are suspected to be error

prone

– Only recommended all ASIL

– Also easily possible in VectorCAST

Generating test case values

• Analysis of boundary values

– Try values approaching, at, or crossing the boundaries, including out

of range value

– Can mean the type range or the functional range

– Strongly recommended for ASIL B, C and D

• VectorCAST has extensive tools to do this

– Auto-generation of MIN-MID-MAX test cases for all the extreme

variable type values

– Import from CSV functional range values – and execution of these in

test cases

Code coverage

• Lets you know when you have been “testing enough”

• Different criteria that require more or less test cases to

achieve

• VectorCAST supports all three criteria recommended by

ISO 26262 (and the “other criteria” – function/call coverage)

Sta

tem

ent • One test case

minimum to execute one line of code

Bra

nch • At decision

point, both TRUE and FALSE to be executed M

CD

C • All operands must independently affect the outcome of the condition

How coverage criteria stack up

• Statement

– Line of code executed at least once

• Branch

– Both the TRUE and FALSE

branches are executed

• MC/DC

– All operands can independently

affect the outcome

if((a || b) && c)

13© Vector Software, all rights reserved

if((a || b) && c) T F

1 test case required

2 tests cases required

if((a || b) && c) RESULT

F F T F

T F T T

F T T T

F T F F

At least n+1 test case

required

ab

c

Knowing what needs

to be done!

Green: Fully covered

(good)

Red: Not covered

(bad)

Orange: Partially

covered (?)

Statement

+

Branch

+

MCDC

+/- critical software

very critical software

What you get - Automaticity

• Unit test environments are generated automatically:

– All drivers and stubs generated with NO user input

• Constructing a test case is done through point and click OR CSV

– NO scripting of ANY kind

• Both Black Box and White Box are allowed

• Execution on target is done 100% automatically

– You click a button, and the rest is done for you

• Execution reports are generated 100% automatically

• Both GUI and command line are available

• Re-running any test environment in regression mode is automatic, even

if the test cases are modified, or if the underlying code is modified

– Regression testing is completely automated

What you get - Flexibility

• Full, guaranteed support for C/C++ of ANY complexity

• Users can control the value of ALL parameters, return values, global

data and data coming from stubs – even for pointers, exceptions, etc.

• Can test individual values, special values (NAN, positive infinity, etc),

range of values, list of values, even call code to generate Monte Carlo-

style of test cases

• Can create complex test cases that set state machines and test their

transition from one state to another (compound test cases)

• Creation of test cases from CSV

• Automatic test case generation based on basis path analysis, MIN-MID-

MAX, and more so as to give a leg up during structural coverage

• Code coverage is displayed in an easy-to-understand way

Other capabilities you get…

• The ability to test libraries – even if you don’t have access to the code!

• The capability to test as you develop (agile development), or even to first

write test cases before writing code (test-driven development)

• The debugger can be used to control test case execution (so the tool

becomes a test vector generator for debugging too)

• The ability to perform timing calculations, in some conditions can also

be done while other processes from the OS are executing

• The ability to stub library functions, if desired…

• … and much more

Quite simply, you are getting a complete test bench that

enables you to comply with ISO 26262 efficiently, so you

can still meet demanding deadlines!

What about tool qualification?

• ISO 26262 Part 8, Section 11 mentions tool qualification– The objective of the qualification of software tools is to provide evidence of

software tool suitability for use when developing a safety-related item or

element, such that confidence can be achieved in the correct execution of

activities and tasks required by ISO 26262

• Depends on how critical the tool’s reliability is to the quality

of the code

• Process more flexible than DO-178B, but if you need tool

qualification, Vector Software has a long experience of

providing this service

– We can provide you documents demonstrating the tool performance

as adequate in your environment: same compiler version, board,

debugger, and tool version

Conclusion

• ISO 26262 contains a number of recommendations that

have been proven effective in other industries– It does not reinvent the wheel but builds on a rich heritage to customize an

unique standard to the needs of the Automotive industry

• Unit/integration testing and code coverage can be very time-

intensive, which no one in this industry can afford!

• The only way to meet ISO 26262, increase code quality and

still meet your deadlines is to invest in test automation

– Fortunately, VectorCAST tools have both the automation and

flexibility required for you to achieve ISO 26262 compliance in a

timely manner

Questions?