SAFEWARESystem Safety and ComputersChap18:Verification of

SafetyAuthor : Nancy G. Leveson

University of Washington1995 by Addison-Wesley Publishing Company, Inc.

ISBN : 0-201-11972-2Presentation by Yean-Ru Chen at ESL of CS, CCU, Taiwan

2005/09/08

Outline(1/1) Introduction Hazard analysis

Dynamic Static

Conclusions

Introduction(1/2) Most verification techniques focus on

showing consistency between the code and the specification, but for safety, this approach is not enough.

We need to examine not only the relationship between the computer inputs and outputs, but also that between the inputs and the effect of the outputs on system behavior.

Introduction(2/2) Because of considering the effect

of the outputs on system behavior ,

“Hazard analysis” is needed to serves as the basis for this approach to safety verification.

Hazard analysis Hazard is a state or set of conditions of a

system that, together with other conditions in the environment of the system, will lead inevitably to an accident. (Chapter 9)

There are two main analysis ways: (1)Dynamic Analysis

In this analysis, the code or a model of the code is executed and its performance is evaluated.

(2)Static Analysis In static analysis, the code or model is

examined without being executed.

Dynamic Analysis(1/6) Testing! Here we’ll focus on software and

safety only. Process Considerations Limitations of Testing

Dynamic Analysis(2/6) Process Considerations:

There are several aspects of the system needed to be considered when we use the “Testing” approach:

Critical functions and variables. Boundary conditions. Special features, such as firewalls or

safety kernels upon which the protection of the safety-critical features is based.

Dynamic Analysis (3/6) Reaction of the software to system faults and

failures. The model of the system environment that

guides critical control and safety decisions. The operator interface.

Dynamic Analysis (4/6) In general, there are also several aspects for

the safety engineers to pay more attention : Review test plans Recommend tests based on the hazard analyses,

safety standards, previous accidents, and so on. Review the test results for any safety-related

problems that were missed in the analysis or in other testing.

Ensure that testing feedback is integrated into the safety reviews and analyses that will be used in design modifications.

Dynamic Analysis (5/6) Limitations of Testing:

The main limitation of testing is that testing is impossible in a real setting. Most testing must be done in a simulation mode, and there’s no way to guarantee that the simulation is accurate.

Dynamic Analysis (6/6) But these limitations do not mean

that we should not test or include testing in a safety program.

It merely means that there is a limit to the confidence that can be acquired through dynamic analysis and that testing may have to be augmented with static analysis if the limit is less than what is required.

Static Analysis Static analysis evaluates the software

without executing it. Instead, it examines a representation of the software.

There are two major means in this chapter : Formal Verification --- formal Software Fault Tree Analysis (SFTA) ---

informal

Formal Verification (1/4) We mainly concern about the formal

verification on software in this book. Formal verification essentially

provides a proof of consistency between two formal specifications of a system.

In safety, we have to use formal models and definitions of safety properties along with safety policies.

Formal Verification (2/4) A safety policy would describe the

rules for determining whether a system is safe and would use the safety constrains as a basis for decision making.

There are three general safety policies for which the specification could be checked: (Chapter 15)

Formal Verification (3/4) (I) There must be no paths to

unplanned hazardous states. (II) Every hazardous state must have a

path to a safe state. All paths from the hazardous states must lead to safe states. Time in the hazardous state must be minimized, and contingency action may be necessary to reduce risk while in the hazardous state.

Formal Verification (4/4) (III) If a safe state cannot be reached

from a hazardous state, there must exist at least one path from that state lead to a minimum risk state.

Major drawbacks of formal verification on software : Practicality and Feasibility.

Major advantages of formal verification on software : Analytical ,Mathematical and Formal.

Software Fault Tree Analysis(SFTA) (1/4)

FTA focuses on hazards rather than on software errors in general, and it can be applied to assembly or high-level language.

A leaf node of the software fault tree describes a behavior of the computer.

SFTA traces that behavior into the logic of the code and determines whether a path exists through the code that could cause the hazardous output.

Software Fault Tree Analysis(SFTA) (2/4) SFTA uses Backward Reasoning

approach. It follows the paths backward

through the program from the hazardous outputs to identify any inputs that might cause that output.

Software Fault Tree Analysis(SFTA) (3/4) Here is an example of backward

reasoning about variable values : X = 9

X =10 if a variable contains the value 10 after an

instruction adds one to that variable, then the variable must have held the value 9 before the instruction was executed.

X := X + 1

Software Fault Tree Analysis(SFTA) (4/4) SFTA uses fault tree templates to

generate the software fault trees. Here are 4 templates to be

introduced: Template for assignment Template for IF statement Template for WHILE statement Template for procedure call statement

Template for assignment Fig.18_4

Assignment causes event

or

Operand evaluation causes event

Exception causes eventChange in value causes event

Ex. of assignment statements and simple IF ( A Tree !)

(1) A := F(Y) (2) B := X-5.0 (3) IF A>B THEN FIRE

FIRE called

A > B

F(Y) > X – 5.0

A > X – 5.0

Template for IF statement Fig.18_6 Event in IF-TH

EN-ELSE

or

Condition FALSE, ELSE-part causes event

Condition evaluation causes event

Condition TRUE , THEN-part causes event

and and

Condition TRUE before IF-THEN-ELSE statement

THEN-part causes event

Condition FALSE before IF-THEN-ELSE statement

ELSE-part causes event

Ex. of IF statements( A Tree !)analyze for the unsafe condition “X>100” if a > b then X := f(X) ; else X := 100 ;

X>100 caused by IF statement

or

a>b , X := f(X) causes X>100

and and

a > b before IF-THEN-ELSE

X := f(X) causes X>100

a <= b before IF-THEN-ELSE

X := 100 causes X >100

a<=b , X := 100 causes X>100

Template for WHILE statement Fig.18_8 WHILE statement

causes event

or

Statement executed n times

Statement not executed

and and

Event before WHILE statement

Condition false before WHILE

n_th iteration causes event

Condition true before WHILE

Ex. of WHILE statements( A Tree !)analyze for the unsafe condition “z>100” (1/2) while b>x { b:=b-1; z:=z+10; }

WHILE statement causes z>100

or

Statement executed n times

Statement not executed

and and

z>100 before WHILE statement

b <= x before WHILE

z + 10(b-x) >100 before WHILE

b > x before WHILE

Ex. of WHILE statements( A Tree !)analyze for the unsafe condition “z>100”(2/2) Why z+10(b-x) > 100 will cause unsafe condition when “while loop” has been executed n times? Original value of z before loop : z Original value of b before loop : b The value of z after nth iteration :z’ The value of b after nth iteration :b’ z’ = z+10n, b’ = b – nSince iteration stopped, b’ = x (at least)So, z’ = z+10(b-x) If z’ > 100 ,then it’s unsafe.

Template for procedure call statement Fig.18_9

F (p1,p2,p3,…) causes event

or

Failure of F causes event

Execution of F with p1,p2,p3…causes event

Evaluation of parameters causes event

Uses and Evaluation of SFTA(1/2) If SFTA is used with a system simulator,

the critical software interfaces with the rest of the system can be identified and used to determine appropriate simulation states and events.

However, the practicality of SFTA use on large-scale software has not been demonstrated.

Uses and Evaluation of SFTA(2/2) One of the reasons is that SFTA is

basically a structured walkthrough of the code with an emphasis on hazardous behavior , so its practicality should be similar to that of other structured walkthrough techniques.

However, the entire code is not usually safety-critical, maybe small part of it need to be examined in SFTA.

Conclusions Those using computers for safety-

critical functions should be wary of relying too heavily on the techniques described in this chapter.

They are necessary but not sufficient!!