Risk Structures: Concepts, Purpose, and the Causality Problem
Transcript of Risk Structures: Concepts, Purpose, and the Causality Problem
Risk Structures: Concepts, Purpose,and the Causality Problem
Mario GleirscherUniversity of York, UKJune 26, 2019
Shonan, JP
Part I
Risk-aware Systems:Abstraction by Example
4.0 / Gleirscher / Shonan, JP/ June 26, 2019 52
Example: Air-traffic Collision Avoidance System (TCAS)
Example: Safe Autonomous Vehicle (SAV)
4.0 / Gleirscher / Shonan, JP/ June 26, 2019 53
Risk Mitigation / Intervention / Enforcement
Approach: Activesafety monitors,enforcementmonitors
System/ProcessModel
MDP,LTS, HA
RiskStructure
ActiveMonitor
Testmodel
ActiveMonitor
Implementation
MonitoredProcess
abstracts
fromabstracts
from
synthesisedfrom conform
sto
recognises/
enforces
property
RQ: How to build a mitigation monitor? / Which model to use?/ Which abstraction? / What do we need to verify?
4.0 / Gleirscher / Shonan, JP/ June 26, 2019 56
Example: Air-traffic CollisionAvoidance System (TCAS)
Example: Traffic Collision Avoidance System (TCAS) Ego
Risk Model R
Process Model P
abstractsfrom
p1
p4e.g. red ”1`severity1`benefit ą c
p6
rl,uq
movΛ: coll
1´Λ: fin
1 Risk Factor
0coll
tp1, p6ustart
colltp4u
collisiont��mov{ecollu
t?u
Σztmov{ecolluΣztmov{finu
2 Risk Factorsbefore
0ncoll
tp1, p3,p4, p6u
startncoll
tp2, p5u
near-collisiont���tmov{encollu
t���tmov{finu
Σzttmov{encolluΣzttmov{finu
p1start
p2
p3
p4 rl4,u4q
p5 rl5,u5q
p6
mov Λ2 :encoll
1´ Λ2 : fina1a2
(TCAS move)tmov Λ5 : encoll
Λ4: ecoll
a3Λ6 : fin
a5
a40ncoll, 0coll
ncoll, 0coll 0ncoll, coll
ncoll, coll
e ncoll��ecoll
ecoll e ncoll
mncoll
“
tmov
{fin
��m “
mov{fin
ěm
Risk Space
4.0 / Gleirscher / Shonan, JP/ June 26, 2019 57
Example: Traffic Collision Avoidance System (TCAS) Ego
Risk Model R
Process Model P
abstractsfrom
p1
p4e.g. red ”1`severity1`benefit ą c
p6
rl,uq
movΛ: coll
1´Λ: fin
1 Risk Factor
0coll
tp1, p6ustart
colltp4u
collisiont��mov{ecollu
t?u
Σztmov{ecolluΣztmov{finu
2 Risk Factorsbefore
0ncoll
tp1, p3,p4, p6u
startncoll
tp2, p5u
near-collisiont���tmov{encollu
t���tmov{finu
Σzttmov{encolluΣzttmov{finu
p1start
p2
p3
p4 rl4,u4q
p5 rl5,u5q
p6
mov Λ2 :encoll
1´ Λ2 : fina1a2
(TCAS move)tmov Λ5 : encoll
Λ4: ecoll
a3Λ6 : fin
a5
a40ncoll, 0coll
ncoll, 0coll 0ncoll, coll
ncoll, coll
e ncoll��ecoll
ecoll e ncoll
mncoll
“
tmov
{fin
��m “
mov{fin
ěm
Risk Space
4.0 / Gleirscher / Shonan, JP/ June 26, 2019 58
Example: Traffic Collision Avoidance System (TCAS) Ego
Risk Model R
Process Model P
abstractsfrom
p1
p4e.g. red ”1`severity1`benefit ą c
p6
rl,uq
movΛ: coll
1´Λ: fin
1 Risk Factor
0coll
tp1, p6ustart
colltp4u
collisiont��mov{ecollu
t?u
Σztmov{ecolluΣztmov{finu
2 Risk Factorsbefore
0ncoll
tp1, p3,p4, p6u
startncoll
tp2, p5u
near-collisiont���tmov{encollu
t���tmov{finu
Σzttmov{encolluΣzttmov{finu
p1start
p2
p3
p4 rl4,u4q
p5 rl5,u5q
p6
mov Λ2 :encoll
1´ Λ2 : fina1a2
(TCAS move)tmov Λ5 : encoll
Λ4: ecoll
a3Λ6 : fin
a5
a40ncoll, 0coll
ncoll, 0coll 0ncoll, coll
ncoll, coll
e ncoll��ecoll
ecoll e ncoll
mncoll
“
tmov
{fin
��m “
mov{fin
ěm
Risk Space
4.0 / Gleirscher / Shonan, JP/ June 26, 2019 58
Example: Safe Autonomous Vehicle(SAV)
Risk Mitigation / Intervention / Enforcement
Approach: Activesafety monitors,enforcementmonitors
System/ProcessModel
MDP,LTS, HA
RiskStructure
ActiveMonitor
Testmodel
ActiveMonitor
Implementation
Monitored
Process
abstracts
fromabstracts
from
synthesisedfrom conform
sto
recognises/
enforces
property
RQ: How to build a mitigation monitor? / Which model to use?/ Which abstraction? / What do we need to verify?
4.0 / Gleirscher / Shonan, JP/ June 26, 2019 62
SAV: Low Level Vehicle Dynamics
r 9p “ 0,9v “ 0s
v “ 0^ a “ 0
start
halt
r 9p “ v,9v “ as
0 ă v ă l^ a ą 0
accelr 9p “ v,9v “ 0s
v “ l^ a “ 0
drive atmax speed
r 9p “ v,9v “ as
0 ă v ă l^ a ď 0
decel
aą0
v = l
aă0
aą0
aď0
v = 0 ‖r 9α “ 0s
α “ 0start
neutral
r 9α “ f s0 ă y ă 5
turnleft
r 9α “ 0s
α ‰ 0
movestraight
r 9α “ f s´5 ă y ă 0 turn
right
y ą0
^|v|
ą0
9α“0
^α
‰0
yą0
^|v|
ą0
9α“0
^α
“0
yă0
^|v|
ą0
9α“0
^α
‰0
y ă0
^|v|
ą0
α“0
^9α
“0
Longitudinal dynamics LoDdrive
halt
accel
decel
driveat maxspeed
Lateral dynamics LaD(relative to route segment)
neutral
turnleft
turnright
movestraight
Overall low-level dynamics: drive ‖ LaD
4.0 / Gleirscher / Shonan, JP/ June 26, 2019 63
SAV: Situational Perspective of Urban Driving
Mode model of the driving activity:
basic || supplyPower
drive
driveAtL4 || requestTakeOverByDr driveAtL1 || operateVehicle
driveAtLowSpeed
driveAtL4Generic
exitTunneldriveThroughCrossing
parkWithRemote
autoOvertake driveAtL1Generic
manuallyOvertake
autoLeaveParkingLot
manuallyPark
leaveParkingLot
steerThroughTrafficJam
halt
start
Integration withlow leveldynamics:
In each mode,verify contract:inv ^ pre ñ
wppdrive ‖ LaD,inv ^ postq
4.0 / Gleirscher / Shonan, JP/ June 26, 2019 64
SAV: Risk Identification and Assessment
Knowledge sources for risk/hazard identification, e.g.
• accident reports• domain experts• situation/activity model• local dynamics model• control system architecture• control software
Analysis techniques, e.g.
• hazard identification: FHA, PHL, …• process/scenario analysis: HazOp, LOPA, BA, STPA, …• causal reasoning: ETA, FMEA, FTA, Bowties, …
4.0 / Gleirscher / Shonan, JP/ June 26, 2019 65
SAV: Situational Perspective of Urban Driving
Mode model of the driving activity:
basic || supplyPower
drive
driveAtL4 || requestTakeOverByDr driveAtL1 || operateVehicle
driveAtLowSpeed
driveAtL4Generic
exitTunneldriveThroughCrossing
parkWithRemote
autoOvertake driveAtL1Generic
manuallyOvertake
autoLeaveParkingLot
manuallyPark
leaveParkingLot
steerThroughTrafficJam
halt
start
Risk factors(Yap script):
1 HazardModel for "drive"{
3 OC alias "on occupiedcourse"
;5 CR alias "increased
collision risk";
7 CC alias "on collisioncourse"
;9 ICS alias "inevitable
collision state";
11 Coll alias "actualcollision"
;13 ES alias "perception
system fault";
15 }
4.0 / Gleirscher / Shonan, JP/ June 26, 2019 66
Risk Structures: Tool Support and Recent Publications
nColl
Coll
0
prvnCollcenColl
eColl
1 OperationalSituation "generic" {}
3 ControlLoop "Robot" for "generic" {emgBr alias "Emergency Brake";
5 }
7 HazardModel for "generic" {nColl alias "near-collision"
9 mitigatedBy (PREVENT_CRASH.emgBr)direct;
11 Coll alias "collision"requires (nColl)
13 mishap;}
From Hazard Analysis to Hazard MitigationPlanning: The Automated Driving Case
Mario Gleirscher(B) and Stefan Kugele
Technische Universitat Munchen, Munich, Germany{mario.gleirscher,stefan.kugele}@tum.de
Abstract. Vehicle safety depends on (a) the range of identified hazardsand (b) the operational situations for which mitigations of these hazardsare acceptably decreasing risk. Moreover, with an increasing degree ofautonomy, risk ownership is likely to increase for vendors towards regula-tory certification. Hence, highly automated vehicles have to be equippedwith verified controllers capable of reliably identifying and mitigatinghazards in all possible operational situations. To this end, available meth-ods for the design and verification of automated vehicle controllers haveto be supported by models for hazard analysis and mitigation.
In this paper, we describe (1) a framework for the analysis and designof planners (i.e., high-level controllers) capable of run-time hazard iden-tification and mitigation, (2) an incremental algorithm for constructingplanning models from hazard analysis, and (3) an exemplary applicationto the design of a fail-operational controller based on a given control sys-tem architecture. Our approach equips the safety engineer with conceptsand steps to (2a) elaborate scenarios of endangerment and (2b) designoperational strategies for mitigating such scenarios.
Keywords: Risk analysis · Hazard mitigation · Safe state · Controllerdesign · Autonomous vehicle · Automotive system · Modeling · Planning
1 Challenges, Background, and Contribution
Automated and autonomous vehicles (AV) are responsible for avoiding mishapsand even for mitigating hazardous situations in as many operational situationsas possible. Hence, AVs are examples of systems where the identification (2a)and mitigation (2b) of hazards have to be highly automated. This circumstancemakes these systems even more complex and difficult to design. Thus, safetyengineers require specific models and methods for risk analysis and mitigation.
As an example, we consider manned road vehicles in road traffic with anautopilot (AP) feature. Such vehicles are able to automatically conduct a rideonly given some valid target and minimizing human intervention. The followingAV-level (S)afety (G)oal specifies the problem we want to focus on in this paper:
SG: The AV can always reach a safest possible state σ wrt. the hazardsidentified and present in a specific operational situation os .
c© Springer International Publishing AG 2017C. Barrett et al. (Eds.): NFM 2017, LNCS 10227, pp. 310–326, 2017.DOI: 10.1007/978-3-319-57288-8 23
L. Bulwahn, M. Kamali, S. Linker (Eds.): First Workshop onFormal Verification of Autonomous Vehicles (FVAV 2017).EPTCS 257, 2017, pp. 75–90, doi:10.4204/EPTCS.257.8
c© M. GleirscherThis work is licensed under theCreative Commons Attribution License.
Run-Time Risk Mitigation in Automated Vehicles:A Model for Studying Preparatory Steps
Mario Gleirscher
Faculty of InformaticsTechnical University of Munich
Munich, [email protected]
We assume that autonomous or highly automated driving (AD) will be accompanied by tough assur-ance obligations exceeding the requirements of even recent revisions of ISO 26262 or SOTIF. Hence,automotive control and safety engineers have to (i) comprehensively analyze the driving process andits control loop, (ii) identify relevant hazards stemming from this loop, (iii) establish feasible auto-mated measures for the effective mitigation of these hazards or the alleviation of their consequences.
By studying an example, this article investigates some achievements in the modeling for thesteps (i), (ii), and (iii), amenable to formal verification of desired properties derived from potentialassurance obligations such as the global existence of an effective mitigation strategy. In addition, theproposed approach is meant for step-wise refinement towards the automated synthesis of AD safetycontrollers implementing such properties.
1 Introduction
For many people, driving a car is a difficult task even after guided training and many years of drivingpractice: This statement gets more tangible when driving in dense urban traffic, complex road settings,road construction zones, unknown areas, with hard-to-predict traffic co-participants (i.e., cars, trucks,cyclists, pedestrians), bothersome congestion, or driving a defective car. Consequently, hazards such asdrivers misjudging situations and making poor decisions have had a long tradition. Hence, road vehicleshave been equipped with many sorts of safety mechanisms, most recently, with functions for “safetydecision support” [7], driving assistance, and monitor-actuator designs, all aiming at reducing operationalrisk or making a driver’s role less critical. In AD, more highly automated mechanisms will have tocontribute to risk mitigation at run-time and, thus, constitute even stronger assurance obligations [7].
In Sec. 1.1, we introduce basic terms for risk analysis and (run-time) mitigation (RAM) for automatedvehicles (AV) as discussed in this work. Sec. 1.2 elaborates some of these assurance obligations.
1.1 Background and Terminology
According to control theory, a control loop L comprises a (physical) process P to be controlled anda controller C, i.e., a system in charge of controlling this process according to some laws defined byan application and an operator [20]. The engineering of safety-critical control loops typically involvesreducing hazards by making controllers safe in their intended function (SOTIF), resilient (i.e., toleratedisturbances), dependable (i.e., tolerate faults), and secure (i.e., tolerate misuse).
In automated driving, the process under consideration is the driving process which we decomposeinto a set of driving situations S , see Sec. 2. Taxonomies of such situations have been published in, e.g.
154
8.4 Strukturen für die Gefahren-erkennung und -behandlung in autonomen Maschinen
Dr. Mario GleirscherDepartment of Computer Science, University of York und Fakultät für Informatik, Technische Universität München
In diesem Abschnitt werden hochautomatisierte – insbesondere autonom handelnde – Maschinen (AM) wie zum Beispiel autono-me mobile Roboter betrachtet. Man erwartet von solchen Syste-men, dass deren Regelungen in Gefahrensituationen ebenso nützliche Handlungsalternativen bieten wie im Normalbetrieb. Ausgehend von dieser Problemstellung wird nun eine werkzeug-gestützte Herangehensweise
i. für die Modellierung von Gefahrensituationen sowie
ii. für die Bewertung der Plausibilität und Vollständigkeit solcher Modelle
anhand eines Beispiels aus dem Bereich des automatisierten Fahrens (AF) diskutiert.
1 Motivation
Im Rahmen der gefahrenreduzierenden Absicherung von Syste-men ist es eine Herausforderung, möglichst starke Sicherheits-eigenschaften für autonome, hochautomatisierte Maschinen schon zum Entwurfszeitpunkt festzulegen und Regler solcher Maschinen für die Einhaltung dieser Eigenschaften zur Laufzeit zu entwerfen. Im Folgenden werden formale Strukturen, welche für die Modellbildung und später für die detaillierte Entwicklung von Reglern hilfreich sind, besprochen. Die Motivation, solche Strukturen zu nutzen, resultiert aus
§ Bestrebungen, die Risikobewertung und den Verlässlich-keitsnachweis für allgemeine Systemklassen durch speziali-sierte Modellbildung zu unterstützen (siehe Kapitel 4/Bertsche et al.),
1 | Vgl. Kugele et al. 2017.2 | Vgl. Alexander et al. 2009.3 | Vgl. McDermid 2001 und Kumamoto 2007 diskutieren „As Low As Reasonably Practicable“ (ALARP).4 | Vgl. Schnieder/Schnieder 2009, Schnieder/Schnieder 2008, Schnieder/Drewes 2008.5 | Vgl. Lund et al. 2011.
§ Erfahrungen in der Architekturabsicherung komplexer einge-betteter Systeme1 und
§ der Beobachtung, dass einige Empfehlungen für die Gewähr-leistung von AM-Sicherheit nicht eindeutig und vollständig sind.2
2 Hintergrund
In diesem Abschnitt werden einige Begriffe aus der Literatur und Vorarbeiten des Autors dargestellt, auf denen die spätere Diskus-sion aufbaut.
2.1 Allgemeine Grundlagen
In der Risikoanalyse wird bewertet, inwiefern eine Gefahrenquel-le (Aggressor) ein Risiko für ein Schutzziel (zum Beispiel Safety, Security, körperliche Unversehrtheit) eines Schutzobjekts (im Englischen: asset) darstellt und inwieweit ein Schutzmechanis-mus (auch Beschützer, Sicherheitsfunktion) dieses Risiko wenigs-tens auf ein akzeptables Restrisiko3 reduzieren kann. Häufig wird dazu eine Menge wahrscheinlicher Szenarien in Form von poten-ziell unendlichen Ursache-Wirkungs-Ereignisketten betrachtet, wobei die ultimativen und unerwünschten Auswirkungen als Un-glücks-, Unfalls- oder Schadensereignis und alle Ereignisse auf diesem Wege als Zusammensetzung von potenziell verursachen-den Faktoren beschrieben werden. Hierzu wird weiter unten von Kausalfaktoren und -strukturen gesprochen. Ein kompatibler Be-griffsrahmen wird in den Kapiteln 3/Schnieder und Schnieder, 5/Beyerer und Geisler sowie 7.1/Vieweg ausführlicher behandelt.
Dieser vielfach diskutierte Begriffsrahmen lässt sich auf ganz un-terschiedliche Bereiche anwenden, zum Beispiel technische An-lagen, IT-Systeme, in der Patientenbehandlung im Krankenhaus, im unternehmerischen Projektmanagement, in Arbeitsprozessen im Hochbau oder an Flughäfen (siehe Kapitel 8.1/Wolf und Lichte sowie 8.2/Deutschmann et al.). Je nach Art des Schutz-ziels und -objekts gibt es spezifische Herangehensweisen zur RA sowie verschiedene Bezeichnungen, wie zum Beispiel funktiona-le Sicherheit in der Mechatronik und Automatisierungstechnik4 oder Cyber Security für stark vernetzte IT-Systeme.5 Regelmäßig wird versucht, Erkenntnisse aus verschiedenen Arbeitsfeldern
arX
iv:s
ubm
it/26
6338
0 [
cs.S
E]
23
Apr
201
9
RISK STRUCTURES: TOWARDS ENGINEERING RISK-AWARE
AUTONOMOUS SYSTEMS
A PREPRINT
Mario GleirscherComputer Science Department, University of York, York, UK∗
April 23, 2019
ABSTRACT
Inspired by widely-used techniques of causal modelling in risk, failure, and accident analysis, thiswork discusses a com positional framework for risk modelling. Risk models capture fragments ofthe space of risky events likely to occur when operating a machine in a given environment. More-over, one can build such models into machines such as autonomous robots, to equip them with theability of risk-aware perception, monitoring, decision making, and control. With the notion of arisk factor as the modelling primitive, the framework provides several means to construct and shaperisk models. Relational and algebraic properties are investigated and proofs support the validity andconsistency of these properties over the corresponding models. Several examples throughout thediscussion illustrate the applicability of the concepts. Overall, this work focuses on the qualitativetreatment of risk with the outlook of transferring these results to probabilistic refinements of thediscussed framework.
Keywords Causal modelling · risk · analysis · modelling · safety monitoring · risk mitigation · robots · autonomoussystems
Contents
1 Introduction 2
1.1 Abstractions for Machine Safety and Risk Awareness . . . . . . . . . . . . . . . . . . . . . . . . . . 3
1.2 Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
1.3 Contributions and Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
2 Background 7
2.1 Notions of Risk . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
2.2 The Risk of Undesired Events . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
2.3 Formal Preliminaries . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
3 Risk Elements 9
∗Correspondence and offprint requests to: Mario Gleirscher, Univerity of York, Deramore Lane, Heslington, York YO10 5GH,UK. e-mail: [email protected] work is supported by the Deutsche Forschungsgemeinschaft (DFG) under Grants no. GL 915/1-1 and GL 915/1-2. c© 2019. This manuscript is made available under the CC-BY-NC-ND 4.0 licensehttp://creativecommons.org/licenses/by-nc-nd/4.0/.Reference Format: Gleirscher, M.. Risk Structures: Towards Engineering Risk-aware Autonomous Systems (April 23, 2019).Unpublished working paper. Department of Computer Science, University of York, United Kingdom.
4.0 / Gleirscher / Shonan, JP/ June 26, 2019 67
Risk Factors (Basic Phase Model)
0f
f
fef …endangermentofn
nominaloperation…
mf …mitigationrecovery… mfr
mfd …direct mitigation
endanger-ment … ef
ofe…endangeredoperation
ofm …mitigated operation
Purposes:• Modelling primitive for risk space exploration• Semantics of basic events in DFTs or DFRTs• Synthesis of local enforcement monitors
4.0 / Gleirscher / Shonan, JP/ June 26, 2019 68
SAV: Situational Risk Space
CCOCICSCR
CollCR
CCOCCR
CR
OCICSColl
OCCR
CCICSCR CCOCCollCCCollCR
CCOCICSColl
CCICSColl
ICS
ICSColl OCColl
OCICSCR
CCOC
0
OCICSCollCR
CCICS
ICSCollCR CCOCICS
ICSCR
CC
OCCollCR
OCICSCCCR
CCOCCollCR
Coll
CCColl
CCOCICSCollCR
CCICSCollCR
OC
eVehicle
ICS eVehicle
OC
eVehicle
OC
eVehicle
CC
eVehicle
CC
eVehicle
Coll
eVehicle
CR
eVehicle
ICS
eVehicle
CC
eVehicle
ICS
eVehicle
CR
eVehicle
OC
eVehicle
Coll
eVehicle
CR
eVehicle
Coll
eVehicle
CR
eVehicle
CC
eVehicle
ICS
eVehicle
Coll
eVehicle
Coll
eVehicle
CR
eVehicle
ICS
eVehicle
ICS
eVehicle
OC
eVehicle
CReVehicle
OC
eVehicle
Coll eVehicle
ICS
eVehicle
Coll eVehicle
ICSeVehicle
CR
eVehicle
OC eVehicle
CR
eVehicle
OC
eVehicle
OC
eVehicle
ICSeVehicle
CC
eVehicle
CC eVehicle
ICS
eVehicle
OC
eVehicle
ICS
eVehicle
CC
eVehicle
ColleVehicle
ICS
eVehicle
OC
eVehicle
Coll
eVehicle
ICS
eVehicle
CC
eVehicle
Coll eVehicle
CR
eVehicle
OC
eVehicle
CR
eVehicle
Coll
eVehicle
CC
eVehicle
CC
eVehicle
OC
eVehicle
ICS eVehicle
CReVehicle
Coll
eVehicle
Coll
eVehicle
OC
eVehicle
OC
eVehicle
CR eVehicle
CC
eVehicle
Coll
eVehicle
CR
eVehicle
CC
eVehicle
OC
eVehicle
CR
eVehicle
CC
eVehicle
Coll eVehicle
CR
eVehicle
CC
eVehicle
CR
eVehicle
ICS
eVehicle
ICS
eVehicle
OC
eVehicle
CC
eVehicle
CC
eVehicle
Coll
4.0 / Gleirscher / Shonan, JP/ June 26, 2019 69
SAV: Situational Risk Space (zoomed)
CCOCICSCR
CollCR
CCOCCR
CR
OCICSColl
OCCR
CCICSCR CCOCCollCCCollCR
CCOCICSColl
CCICSColl
ICS
ICSColl OCColl
OCICSCR
CCOC
0
OCICSCollCR
CCICS
ICSCollCR CCOCICS
ICSCR
CC
OCCollCR
OCICSCCCR
CCOCCollCR
Coll
CCColl
CCOCICSCollCR
CCICSCollCR
OC
eVehicle
ICS eVehicle
OC
eVehicle
OC
eVehicle
CC
eVehicle
CC
eVehicle
Coll
eVehicle
CR
eVehicle
ICS
eVehicle
CC
eVehicle
ICS
eVehicle
CR
eVehicle
OC
eVehicle
Coll
eVehicle
CR
eVehicle
Coll
eVehicle
CR
eVehicle
CC
eVehicle
ICS
eVehicle
Coll
eVehicle
Coll
eVehicle
CR
eVehicle
ICS
eVehicle
ICS
eVehicle
OC
eVehicle
CReVehicle
OC
eVehicle
Coll eVehicle
ICS
eVehicle
Coll eVehicle
ICSeVehicle
CR
eVehicle
OC eVehicle
CR
eVehicle
OC
eVehicle
OC
eVehicle
ICSeVehicle
CC
eVehicle
CC eVehicle
ICS
eVehicle
OC
eVehicle
ICS
eVehicle
CC
eVehicle
ColleVehicle
ICS
eVehicle
OC
eVehicle
Coll
eVehicle
ICS
eVehicle
CC
eVehicle
Coll eVehicle
CR
eVehicle
OC
eVehicle
CR
eVehicle
Coll
eVehicle
CC
eVehicle
CC
eVehicle
OC
eVehicle
ICS eVehicle
CReVehicle
Coll
eVehicle
Coll
eVehicle
OC
eVehicle
OC
eVehicle
CR eVehicle
CC
eVehicle
Coll
eVehicle
CR
eVehicle
CC
eVehicle
OC
eVehicle
CR
eVehicle
CC
eVehicle
Coll eVehicle
CR
eVehicle
CC
eVehicle
CR
eVehicle
ICS
eVehicle
ICS
eVehicle
OC
eVehicle
CC
eVehicle
CC
eVehicle
Coll
4.0 / Gleirscher / Shonan, JP/ June 26, 2019 69
Causality (Lewis 1973)
Definition (Counterfactual Conditional)A 2Ñ C is nonvacuously true iff C holds at all the closestA-worlds.
What are the closest A-worlds?
4.0 / Gleirscher / Shonan, JP/ June 26, 2019 70
SAV: Risk Identification and Assessment Yap script
OperationalSituation "drive"2 {
include "envPerc";4 }
6 ControlLoop "Vehicle" for "drive"{
8 replan alias "Slow-down || re-planroute";
brake alias "Standard brake";10 swerve alias "Short-term circumvention
of obstacle";EB alias "Emergency brake";
12 accel alias "Accelerate";airbag alias "Front airbag";
14 }
16 HazardModel for "drive"{
18 OC alias "on occupied course"mitigatedBy (PREVENT_CRASH.replan)
20 direct;
22 CR alias "increased collision risk"requires (OC)
24 deniesMit (OC)
excludes (OC)26 mitigatedBy (PREVENT_CRASH.EB)
;28 CC alias "on collision course"
requires (CR)30 deniesMit (CR,OC)
excludes (CR,OC)32 mitigatedBy (PREVENT_CRASH.swerve)
;34 ICS alias "inevitable collision state"
requires (CC)36 excludes (CC,CR,OC)
causes (Coll)38 mitigatedBy (PREVENT_CRASH.EB)
;40 Coll alias "actual collision"
requires (ICS)42 excludes (CC,CR,OC,ICS,ES)
mitigatedBy (ALLEVIATE.airbag)44 mishap
;46 ES alias "perception system fault"
excludes (CC,CR,OC,ICS)48 deniesMit (OC,CC)
;50 }
4.0 / Gleirscher / Shonan, JP/ June 26, 2019 71
SAV: Situational Risk Space
0
ES
CC
OC
CR
Coll
eVehicle
ES
eVehicle
CollICS
eVehicle
CR
eVehicle
CC
eVehicle
OC
eVehicle
ES
eVehicle
ES
eVehicle
ES
Approach: LOPA/BA tocreate chain of possibleinterventions
4.0 / Gleirscher / Shonan, JP/ June 26, 2019 72
SAV: Situational Risk Space
0
ES
CC
CC
ES
CR
Coll
OC
Coll
CR
eES
eES
eES
mCCe
eCR
prvCCc
mColle
eES
eCC
eOC prvOCc
prvCRc
attCollalv
eES
rES
eCollICS
mCRe
fopESfb
eES
Approach: LOPA/BA tocreate chain of possibleinterventions
Layered interventionpattern for SAV obstacleavoidance
Nice side-effect: Usepattern as enhancedphase model for similarrisk factors
4.0 / Gleirscher / Shonan, JP/ June 26, 2019 72
Taxonomy of Mitigations
OverallSafety
SafetyConstraints
FaultContainment
Fail-Safe
FaultDetection
FaultAvoidance
PreventiveSafety
PassiveSafety
Error-CorrectingCodes
RedundancyFail-Operational
Limp-Home
Fail-Silent
Fail-Secure
ConditionMonitoring
Checking
Pre-CrashAssistance
ObstacleAvoidance
Limiter
Protection
partiallyrealizes
Override
fullyrealizes
Non-Interference
SanityCheck
VigilanceCheck
Warning
Masking
VotingDegradation
Recovery
Rollback
Repair
SimplicitySubstitution
Replication Diversity
Barrier
Comparison
Shutdown
Interlocking
Separation
Stabilization
Reconfiguration
Notification
Filter
4.0 / Gleirscher / Shonan, JP/ June 26, 2019 73
Taxonomy of MitigationsOverallSafety
SafetyConstraints
FaultContainment
Fail-Safe
FaultDetection
FaultAvoidance
PreventiveSafety
PassiveSafety
Error-CorrectingCodes
RedundancyFail-Operational
Limp-Home
Fail-Silent
Fail-Secure
ConditionMonitoring
Checking
Pre-CrashAssistance
ObstacleAvoidance
Limiter
Protection
partiallyrealizes
Override
fullyrealizes
Non-Interference
SanityCheck
VigilanceCheck
Warning
Masking
VotingDegradation
Recovery
Rollback
Repair
SimplicitySubstitution
Replication Diversity
Barrier
Comparison
Shutdown
Interlocking
Separation
Stabilization
Reconfiguration
Notification
Filter
4.0 / Gleirscher / Shonan, JP/ June 26, 2019 73
SAV: Situational Risk Space (Monitor Candidate)
0
ES
CC
CC
ES
CR
Coll
OC
Coll
CR
eES
eES
eES
mCCe
eCR
prvCCc
mColle
eES
eCC
eOC prvOCc
prvCRc
attCollalv
eES
rES
eCollICS
mCRe
fopESfb
eES
Approach: LOPA/BA tocreate chain of possibleinterventions
= Specification of a validsafe system
expected order violatedÑ lack of observability,incomplete monitor,context mismatch?
4.0 / Gleirscher / Shonan, JP/ June 26, 2019 74
Risk Structures: Templates for Causal Reasoning
A risk structure R is valid iff for
s, s1, s2, s3 P RpFq, s3 P Mishaps, e,m, e1 P Σ˚
@s, s3 P RpFq, t P R Ds1, s2 P RpFq,m P Σ˚ : t “ eme1^
s s’ s” s”’e m e’
Definition (Mitigation from Counterfactual Perspective)m is mitigation of cause c ‰ s2 of a mishap s3 iff e1 gets unlikely.(s2, e1 form the counterfactual.)
Proof obligations for each t: Check that, from s,
1. s2 is actual cause of s3,
2. s1 is recognisable,
3. from s1, m reduces s2.4.0 / Gleirscher / Shonan, JP/ June 26, 2019 75
Risk Mitigation / Intervention / Enforcement
Approach: Activesafety monitors,enforcementmonitors
System/ProcessModel
MDP,LTS, HA
RiskStructure
ActiveMonitor
Testmodel
ActiveMonitor
Implementation
MonitoredProcess
abstracts
fromabstracts
from
synthesisedfrom conform
sto
recognises/
enforces
property
RQ: How to build a mitigation monitor? / Which model to use?/ Which abstraction? / What do we need to verify?
4.0 / Gleirscher / Shonan, JP/ June 26, 2019 76