Software Satefy

8/21/2019 Software Satefy

1/49

Copyright 1995-2009, Dennis J. Frailey Software Safety 1

Introduction to Software

Safety

Dennis J. Frailey

Principal Fellow

Raytheon [email protected]


2/49

2Copyright 1995-2009, Dennis J. Frailey Software Safety 2

What This Talk is All About

This is an introduction to the topic ofsoftware safety

It is based on existing government and

commercial standards It is intended to explain what software

safety means, how software cancontribute to safety problems, and whattechniques are used to deal with safety-critical software


3/49


Why Be Concerned about SoftwareSafety

Can Software Harm Anyone?

Eeek!!!Its

Software!

Software by itself seemspretty innocuous but


4/49


Ways that Software can HarmSomeone

It can Control theBehavior ofDangerous Devices

Robots

Weapons

Security Doors atBuilding entry

Medical Devices

Chemical

Experiments

FactoryManufacturing Lines

It can SendInformation toPeople who doPotentiallyDangerous Things

Location of Airplanesfor ATC

Identification ofIntruders

It can Deceive

Internet scams

And on and on


5/49


Software is Often Used for Jobs that

Once Called for Human Judgment


6/49


Software Safety Starts withSystem Safety

Software is always part of a system A data base

A network

A vehicle

If the system can harm someone, thenthe software may be a factor in whetherthe system harms someone

So we have to start by analyzing thesafety issues of the system


7/497 Copyright 1995-2009, Dennis J. Frailey Software Safety 7

Basic System Safety Process

Identify the Potential Hazards

Decompose Hazard Threads through

Subsystem Components, including SW

Link/Trace to Requirements Generate Appropriate Mitigation Strategy

Implement the Mitigation

Verify that the Mitigation is Implementedand that it Functions as Expected

Document Safety Artifacts


8/498


Fault Tree Concept

Hazard

Potential

Cause ofHazard

Potential

Cause ofHazard

Potential

Cause ofHazard

Contributorto Potential

Cause


Cause


Cause


Cause


Cause


Cause


Cause


Cause


Cause

Keep going down until you reach a pointwhere you can do something about it.


9/499


An Example of aSystem Level Hazard List

Uncontrolledexplosion

Uncontrolled fire

Injury and/or illness

Blockage ofingress/egress paths

Structural failure

Collision

Uncontrolledactivation ofordinance

Electromagneticinterference

Hazardous/reactivematerials

Electrical energy Improper engagement

control (Fratricide)

Surface/aircontamination

Corrosion resulting inloss of strength orintegrity of exposedsurfaces

Batteries (exposure totoxic material,

explosion) Radiation (ionizing and

non-ionizing)

Uncontrolled/unsupervised robotic operations

Hazard


10/49Copyright 1995-2009, Dennis J. Frailey Software Safety 10

Notice how often the wordcontrol is used!

Software is often used to replacemechanical or human controls.

When software controls somethingit is often potentially safety critical.

But thats not the only waysoftware can be safety critical.


11/4911


Some Software Functionsthat May be Safety Critical

Assessment of overall system health (e.g, power-upand run-time monitors, heartbeat, program memoryCRCs, range checks, CPU health)

Enforcement of critical timing

Exception trapping and handling including, failure

/malfunction detection, isolation, and containment Functions that execute the systems response to

detected failures/malfunctions

Functions that enhance the systems survivability(preservation of core functionality)

Data quarantine/sanitization Range and reasonableness (sanity) checking

Tamper and cyber-attack proofing

Authentication for lethal actions

Inhibiting functions and interlocks

PotentialCause ofHazard


12/4912


Example SoftwareContribution to Hazard (1/3)

Example SWContribution

Symptom Example PotentialCause

IncorrectSafety-CriticalAlerts and

Warnings.

Safety Critical alertsare incorrect, or arenot triggered by Safety

Critical Events. Alertsfail to warn the user ofan unsafe condition,and or an UnsafeSystem State.

Software fails to alertthe operator to unsafecondition and/or

state. Alerts can beaudio, visual, or intext format in a log,etc..

Incorrect

Data TransferMessages(transmit andreceive).

Data transferred in the

wrong format and theSafety Critical Data areinterpreted incorrectly.

Failure to validate

data transfer with theappropriate parity,check sums to validateSafety Critical data.


13/4913




Symptom Example PotentialCause

Data StorageFailures(Safety

Critical Datacorrupted andor lost)

Safety-CriticalDisplays areconfusing, and/or

incorrect inpresenting SafetyCritical Data.

Safety critical data werenot properly check-summed; data

overwritten by mistake

Incorrectdata transferbetween

processors

Incorrect messagereceived

Failure to performverification checks in bothprocessors prior to

transferring SafetyCritical data.


14/49




DescriptionExample Potential

Cause

Timing andInterruptfailures

Interrupts occur at thewrong time andpotentially interrupt a

Safety Critical process,or introduce a potentialhazard.

Interrupts are out ofsynch with systemtime, and/or

interrupt a SafetyCritical Process/Pathwhich introduces apotential hazard.

Incorrect

Modes

Software signals the

system to fire a weaponwhen it should not.

Switch from training

to live mode is notcorrectly reflected inthe mode variable.


15/49Copyright 1995-2009, Dennis J. Frailey Software Safety 15

So What Should We Do AboutSafety Critical Software?

Mitigate the Risks


16/49


Potential SoftwareMitigation Strategies

Wrappers to ConstrainInputs/Outputs

Timers/Heartbeats/Counters

Fault Trapping

Error Trapping Command Retry

Parity/Checksums/CRCs

Redundancy/Voting

Monitoring

Synchronization/Timing

Timeout

Reasonableness

Procedures

Interlocks/Inhibits Range/Sequence

Checking

Transition Checking

Hysteresis for Man-

Machine Interactions toallow time for HumanInterpretation

Partitioning/Isolation

Redundant but

Dissimilar Algorithms Functional Separation

Failsafe Strategies

Fault/Error TolerantStrategies

Not a

Comprehensive List


17/49


How Seriously Must we TreatPotential SystemSafety Issues?

Severityof Hazard

(howmuch

damagecan be

caused?)

Hazard Probability(howlikely is the hazard?)

Hazard Response Levelor Risk Index

(how extensively our

product developmenttechniques must be

enhanced)

Mil-STD882D


18/49


This can be a very subjectiveevaluation

Any hazard should be avoided, butsome are worse than others

The safety community for anyapplication domain will, by and large,agree on a set of severity categories(see next slide)

Severity of Hazard (how muchdamage can be caused?)

d S i C i


19/49


Hazard Severity CategoriesCommonly Used inMilitary Safety Standards

Catastrophiccould result in death, permanenttotal disability, loss exceeding $1M, or severeenvironmental damage violating law or regulation

Criticalcould result in permanent partialdisability, injuries or illness affecting at least 3

people, loss exceeding $200K, or reversibledamage to environment violating law or regulation Marginalcould result in injury or illness

resulting in loss of 1 or more work days, lossexceeding $10K, or mitigatable environmentaldamage without violation of law or regulationwhere restoration activities can be accomplished

Negligiblecould result in injury or illness notresulting in lost workdays, loss exceeding $2K, orminimal environmental damage not violating law orregulations


20/49


How Seriously Must we TreatPotential SystemSafety Issues?

Severityof Hazard

(howmuch

damagecan be

caused?)


Hazard Response Levelor Risk Index

(how extensively our

product developmenttechniques must be

enhanced)

Mil-STD882D


21/49


Typical Hazard ProbabilitiesLevel Description Description Probability

A Frequent Will Occur 1 in 100

B Probable Likely to Occur 1 in 1000

C Occasional Unlikely butPossible

1 in 10,000

D Remote Very Unlikely 1 in 100,000

E Improbable Can AssumeWill Not Occur

1 in 1,000,000

Probability Level is Used to Select Appropriate Mitigation Actions



22/49


Hazard Risk Index CombinesSeverity and Probability

Typical Hazard Risk Index CalculationProbability A B C D E

Severity Frequent Probable Occasional Remote Improbable

I - Catastrophic U U M M N

II - Critical

U U M N N

III - Marginal U M N N N

IV - Negligible M N N N N

U - Unacceptable Mitigate at High CostM Marginal

RiskMitigate at Moderate Cost

N NoSignificant Risk

Mitigate at Low Cost


23/49


One can deduce the probability ofhardware failure by examining theproperties of materials, laws of physics,

data on material fatigue, etc.

But one cannot deduce the probability ofsoftware failure in such a manner.

So for software, most safety experts use adifferent approach (next slide).

The Problem with Probability forSoftware



24/49


How Seriously Must we TreatPotential SoftwareSafety Issues?

Severityof Hazard

(howmuch

damagecan be

caused?)

Level of Control(to whatdegree is softwareresponsible?)

Software Level of Rigor(how extensively oursoftware development

techniques must beenhanced)


25/49


The Software Level of Controlis ameasure of the degree to whichsoftware is responsible for the

behavior of a system or of a specificsystem action that may lead to asafety hazard

The higher the level of control, themore rigorous the process forsoftware development

Level of Control (to whatdegree is software

responsible?)


26/49


An Example of Software Levelsof Control(1/3)

I Autonomous/Time Critical- Softwareexercises autonomous controlover potentiallyhazardous hardware systems, subsystems, orcomponents without the possibility of interventionto preclude the occurrence of a hazard. Failure ofthe software or a failure to prevent an event leadsdirectly to a hazard's occurrence. This implies noother failure is required to cause the hazard.

IIa - Autonomous/Not Time Critical- Softwareexercises controlover potentially hazardous

hardware systems, subsystems, or componentsand allows time for interventionby independentsafety systems to mitigate the hazard. This impliesthat corrective action is possible and a secondfault or error is required for this hazard to occur.


27/49



IIb Information Time Critical- Softwaredisplays information requiring immediate operatoractionto mitigate a hazard. Software failures willallow or fail to prevent the occurrence of a hazard.This implies that the operator is made aware of

the existence of the hazard and intervention ispossible.

IIIa Operator Controlled- Software issuescommandsover potentially hazardous hardwaresystem subsystems or components requiringhuman action to completethe control function.There are several redundant, independent safetymeasures for each hazardous event.


28/49



IIIb Information Decision- Softwaregenerates informationof a safety-critical natureused to make safety-critical decisions. There areseveral redundant, independent safety measures

for each hazardous event.

IV Not Safety Software- Software does notcontrol safety-critical hardware systems,

subsystems, or components and does notprovide safety-critical information.


29/49


Summary of Software Levels ofControl

A software fault may:

(1)result directly in a hazard (LOC I),

(2)significantly impact the margin of safety(LOC IIa or IIb), but not result directly in ahazard, or

(3)have a minor impact on the margin ofsafety for a hazard (LOC IIIa or IIIb).


30/49


Hazard

Severity

Software Level of Control

I

Autonomous

/ Time

Critical

IIb

Information

/ Time

Critical

IIa -

Autonomou

s/ Not Time

Critical

IIIa -

Operator

Controlled

IIIb -

Information

Decision

IV - Not

Safety

Software

I -Catastrophic LOR-3

LOR-3

LOR-2

LOR-2

LOR-2

N/A

II - Critical

LOR-3

LOR-2

LOR-2

LOR-2

LOR-2

N/A

III - Marginal

LOR-2

LOR-2

LOR-2

LOR-1

LOR-1

N/A

IV -

Negligible LOR-1

LOR-1

LOR-1

LOR-1

LOR-1

N/A

Software Level of Rigor (howextensively our SW developmenttechniques must be enhanced)


31/49


SAFETY

CRITICAL

FUNCTIONS

TOP LEVEL

MISHAPS

REQUIREMENTS

& DESIGN

CHANGES

PERFORM

REQUIREMENTS

ANALYSIS

APPLICABLE

STANDARDS

(e.g., STANAG

4404)

DEVELOP TEST

PLAN &

PROCEDURES

OPERATIONS

& TEST

ANALYSIS

LESSONS

LEARNED

System Hazard Analysis

Operating & Support Hazard Analysis Safety Assessment Report

Safety Test Verification Report

Hazard Control Records

Computer Program Change Requests

INPUT EXTENT OF ANALYSIS PRODUCTS

SR/CA

Software Safety Requirements

Criticality Matrix

Traceability Matrix

Preliminary Hazard Analysis

Safety Requirements/ Criteria Analysis



REQUIREMENTS

COMPLIANCE

1

Techniques for Level of Rigor 1


32/49


SAFETY

CRITICAL

FUNCTIONS

TOP LEVEL

MISHAPS

REQUIREMENTS& DESIGNCHANGES

PERFORM

REQUIREMENTS

ANALYSIS

APPLICABLE

STANDARDS

(e.g., STANAG

4404)

DEVELOP TEST

PLAN &

PROCEDURES

SR/CA

DESIGN

ANALYSIS

OPERATIONS

& TEST

ANALYSIS

LESSONS

LEARNED

REQUIREMENTS

COMPLIANCE

Software Safety Requirements

Criticality Matrix Traceability Matrix





Criticality Matrix Update

Traceability Matrix Update

Safety Test Requirements

Subsystem Hazard Analysis

Hazard Control Records Computer Program Change Requests


Operating & Support Hazard Analysis

Safety Assessment Report Safety Test Verification Report




1



33/49


SAFETY

CRITICAL

FUNCTIONS

TOP LEVEL

MISHAPS

REQUIREMENTS

& DESIGN

CHANGES

PERFORM

REQUIREMENTS

ANALYSIS

APPLICABLE

STANDARDS

(e.g., STANAG

4404)

DEVELOP TEST

PLAN &

PROCEDURES

SR/CA

DESIGN

ANALYSIS

CODE

ANALYSIS

OPERATIONS

& TEST

ANALYSIS

LESSONS

LEARNED

REQUIREMENTS

COMPLIANCE

Software Safety Requirements Criticality Matrix

Traceability Matrix







Safety Test Requirements

Subsystem Hazard Analysis Hazard Control Records




Subsystem Hazard Analysis Report Update




Operating & Support Hazard Analysis Safety Assessment Report

Safety Test Verification Report




1



34/49


Most of the TechniquesDescribed Make Sense in aWaterfall Type of Development

Process

What about Agile

Development, IncrementalDevelopment, etc.?


35/49


Challenges for Agile andIncremental Development

Changes are constantly and frequentlybeing made, so things you do to assuresafety, such as comprehensive tests andverifications, may need to be done overand over again and may slow developmentdown significantly


36/49


SW Safety Strategy for Agile &Incremental Development (1/2)

Experimental and prototype functionalitycannot be used in critical applications untilformally developed and verified (partitioningmust be used to segregate

prototype/experimental functionality frommainstream functionality)

Place limitations on the use of functions that

have not yet met all safety criteria (e.g., allowonly non-critical applications to usecapabilities until all safety criteria aresatisfied)


37/49


SW Safety Strategy for Agile &Incremental Development (2/2)

Lift restrictions when functions havesuccessfully exited formal development andverification

Consider the economics of developing arobust, scaled-down version of a function forsafety critical applications

Partition critical functions from non-criticalfunctions (a failure/ malfunction in a non-critical function must not cause a failure/malfunction in a critical function)


38/49


All of the Above Assumes YouWrite the Software

(and therefore control the

development process)

What if Somebody Else Writes It?


39/49


Challenges for COTS Software

COTS (Commercial Off the Shelf)software examples

Operating systems

standard I/O drivers or interfaces

Board support packages COTS has these challenges:

You dont control the software developmentprocess, standards, etc.

You may not even be allowed to look at thesource code

T h i f S f t C iti l


40/49


Techniques for Safety CriticalCOTS Software

Safety Certificationthere are organizations

that will certify the safety or security of asoftware package.

Wrappers- that control the inputs and outputs ofCOTS software to inhibit likely failure modes.

Analysesto assess the reliability and failure

modes of COTS software. Extensive Testscovering a wide range of

operational conditions under limited resources,including stress testing, stability testing, andothers.

Limit use- to limited-complexity applicationswhere reliability can be readily and accuratelyassessed.

Redesign- to eliminate COTS from the design.


41/49


Challenges for FOSS Software

FOSS (Free or Open Source Software)examples Many components of a GUI

Compilers

FOSS has these additional challengesbeyond COTS: You may be legally required to report changes

back to the originator

The code may change over time with littlenotification or configuration control

You dont know who wrote the code or whattraps they might have embedded in the code


42/49


Other Software Safety Issues

Software development is rapidlychanging as we learn of new and fasterways to develop software

Faster development techniques often bypassthe safeguards needed to address safety

concerns

Software is often used in applicationswell beyond what was originally intended

How do you know the software is suitable for a

safety critical application? System safety is often not seriously

addressed

So software safety may not even be considered

F t I ith S ft


43/49


Future Issues with SoftwareSafety

Certifications or licensing for developersof safe software

Teaching safety issues in softwareengineering courses

Legal liabilities for those who developsoftware that has safety implications

C tifi ti Li i


44/49


Certification vs LicensingCertification

A certificate is a document whereby some

organization attests that you meet certainqualifications

A college diploma

A certificate for completion of a course

Microsoft certified system engineer

Cisco certified network architect

You get a certificate to show that you have somecapability, skill, or education

ASQ CSQE - Certified SW Quality Engineer

IEEE CSDP Certified SW Development Professional Any given certification is only as good as the

organization its from

Some employers require certifications for specifictypes of jobs

C tifi ti Li i


45/49


Certification vs LicensingLicensing

A license is a legal authorization to practice, usuallyin a field involving public safety

A medical license

A license to practice law

A plumbing license

A licensed professional engineer Licenses are granted by government agencies

Usually based on tests or other examinations

Sometimes also based on other factors, such asallowing only so many taxi licenses in a given city

Some organizations require that contractors havelicensed employees

For example, licensed civil engineers for highwayconstruction projects

Soft a e Enginee ing


46/49


Software EngineeringLicensing

Can you get licensed?

Texas currently licenses software engineers but theprocess for further licenses is on hold until anexamination is prepared at the national level

A national examination is in preparation, sponsoredby several organizations and under the technical

guidance of IEEE-USA Many states are expected to start licensing SW

engineers once the examination is available

An ABET accredited SW engineering degree is likelyto be preferred but not required

Must you be licensed? If SW licensing is ever required, it will likely be only

for safety critical software applications, such ascontrol of power plants, medical applications, etc.

Some Useful Reference


47/49


Some Useful ReferenceMaterial (1/2)

Patrick R.H. Place & Kyo C. Kang, Safety-CriticalSoftware: Status Report and AnnotatedBibliography, Software Engineering Institute,June 1993

Leveson. Nancy G., Safety-Critical SoftwareDevelopment. In T. Anderson, editor, Safe andSecure Computing Systems, pages 155-162.Blackwell Scientific Publications, 1989.

Leveson, Nancy G., Safeware: System Safety andComputers, Addison-Wesley, 1995, ISBN-13:978-0201119725

FAA System Safety Handbook, Appendix J:Software Safety

NASA-STD-8719.13A Software Safety

Some Useful Reference


48/49


Some Useful ReferenceMaterial (2/2)

IEEE 1228

IEEE Standard for Software SafetyPlans

EIA SEB6-A System Safety Engineering inSoftware Development

MIL-STD-882D Standard Practice for System

Safety RTCA, Inc., DO-178B, Software Considerations in

Airborne Systems and Equipment Certification

RTCA, Inc., DO-248B, Final report for Clarificationof DO-178B

The DACS Software Reliability Sourcebook Data &Analysis Center for Software


49/49

Questions?

Software Satefy

Documents

Transcript of Software Satefy