Experiences Verifying Autonomous RoboticSystems

Clare DixonDepartment of Computer ScienceUniversity of Liverpool

1 University of Liverpool (UoL)2 University of Hertfordshire (UoH)3 Bristol Robotics Lab (BRL)

www.robosafe.org

[Images from UoH, BRL]

Farshid Amirabdollahian2 Anthony Pipe3

Kerstin Dautenhahn2 Jin Sa3

Kerstin Eder3 Maha Salem2

Michael Fisher1 Joe Saunders2

Paul Gainer1 Richard Stocker1

Dejanira Araiza Illan3 Matt Webster1

M.-Carmen Fernandez Gago1 David Western3

Kheng Lee Koay2 Alan Winfield 3

Savas Konur1Clare Dixon Verification of Autonomous Robotic Systems 1 / 1

Autonomous Robots

Autonomous robots are being developed inmany different domains across society e.g.robot assistants, UAVs, driverless carsThese robots will need to be able to actautonomously and make decisions tochoose between a range of activities.Some of these will need to help, workclosely with or in collaboration with humansin industrial, domestic and health careenvironments.How do we make sure they are trustworthy,safe, reliable and do what they aresupposed to?

Spraying Sugar Cane image [CC0 https://pixabay.com/en/spraying-sugar-cane-sugar-cane-2746350/]Google driverless car at intersection by Grendelkhan [CC BY 4.0https://commons.wikimedia.org/wiki/File:Google_driverless_car_at_intersection.gkjpg]

Clare Dixon Verification of Autonomous Robotic Systems 2 / 1

Trustworthy Robotic Assistants Project

The EPSRC funded Trustworthy Robotic Assistants Projectdeveloped and applied three different approaches to verificationand validation of robotic assistants in a domestic environmentand for collaborative manufacture.Each approach is aimed at increasing trust in robotic assistants.

Formal Verification (Liverpool)Simulation-based Testing (Bristol Robotics Laboratory)End-user Validation (Hertfordshire)

Clare Dixon Verification of Autonomous Robotic Systems 3 / 1

Verification and Validation of Robot Swarms

A robot swarm is a collection of simple (often identical) robotsworking together to carry out some task where each robot istypically able to interact with nearby robots and its environment.

It is challenging to formulate individual robot behaviours so thatthe emergent behaviour of the swarm as a whole is guaranteedto achieve the task of the swarm.

Formal Verification (Liverpool)Simulation of Swarms (Bristol Robotics Laboratory)Real robot experiments (Bristol Robotics Laboratory)

Clare Dixon Verification of Autonomous Robotic Systems 4 / 1

Formal Verification

A mathematical analysis of all behaviours using logics, andtools such as theorem provers or model checkers.We focus on temporal verification using automatic toolsand techniques that do not require user interaction.Model checking is a fully automatic, algorithmic techniquefor verifying the temporal properties of systems.Input to the model checker is a model of the system and aproperty to be checked on that model.Output is that the property is satisfied or a counterexample is given.

Model Checker

Property holds

or

counter example

Property eg

"always p"

Clare Dixon Verification of Autonomous Robotic Systems 5 / 1

Use Cases: Robot Assistants and Robot Swarms

8 Journal Title XX(X)

• System model inaccuracies. All the verificationtechniques use models of the real-world. The modelsmight have been constructed erroneously, or may beinconsistent with the real world, or relative to oneanother.

• Requirement model inaccuracies. In our approach,the real-world requirements of the system are con-verted into textual requirements, assertions and prop-erties for verification. These requirements modelsmay not have been correctly formulated.

• Tool inaccuracies. It is possible that numericalapproximations affect the verification results. Inaddition, third party tools can contain bugs that areunknown to us.

We could now proceed to perform “Experiments.” Asbefore, we may find a problem with the textual require-ments or the physical system during experimentation. Atthe same time, the assurances from formal verificationand/or simulation-based testing can be compared againstthe experiment results. We may also discover that one of theassurances holds during simulation-based testing or formalverification, but not during the experiments. In this case wemay need to refine any of the other assets, as explainedbefore.

Careful comparisons must be made between the dif-ferent representations in order to discover the cause ofthe assurance conflicts. Such comparisons are indicatedby the bi-directional arrows between “Formal Verification”and “Simulation-based Testing”, “Simulation-based Test-ing” and “Experiments”, and “Formal Verification” and“Experiments”, respectively, in Figure 1.

4 The BERT Handover Task: A Case StudyIn this section, we present a case study to demonstratethe application of assurance-based verification to an HRIscenario considering the following research question: canassurance-based verification provide a higher degree ofconfidence in the resulting assurances than when usingverification techniques in isolation?

BERT 2 is an upper-body humanoid robot designed tofacilitate research into complex human-robot interactions,including verbal and non-verbal communication, such asgaze and physical gestures (Lenz et al. 2010) (see Figure 2).BERT 2’s software architecture was originally developedusing YARP¶. More recently, this system has been wrappedwith a ROS interface.

We verify an object handover to exemplify our approach,in the context of a broader collaborative manufacturescenario where BERT 2 and a person work together toassemble a table (Lenz et al. 2012). In the handover, thefirst step is an activation signal from the human to the

Figure 2. BERT 2 engaged in the handover task.

robot. BERT 2 then picks up a nearby object, and holdsit out to the human. The robot announces that it is readyto handover. The human responds verbally to indicate thatthey are ready to receive. (For practical reasons, human-to-robot verbal signals were relayed to the robot by a humanoperator pressing a key.) Then, the human is expected topull gently on the object while looking at it. BERT 2 thencalculates three binary sensor conditions:

• Gaze: The human’s head position and orientationrelative to the object are tracked using the Vicon R�

motion-tracking system for an approximate measureof whether he/she is looking at the object.

• Pressure: Changes in the robot’s finger positionsare sensed to detect whether the human is applyingpressure to take the weight of the object.

• Location: The Vicon R� motion-tracking system isused to determine whether the human’s hand islocated on the object.

The sensor conditions must be calculated within a timethreshold for BERT 2 to determine if the human “is ready”.The robot should release its grip on the object if allthree conditions are satisfied. Otherwise, the robot shouldterminate the handover and not release the object. Thehuman may disengage and the robot can timeout, whichwould cancel the remainder of the handover task. Thesensors are not completely accurate and may sometimesgive incorrect readings.

A safety requirement ensures that “nothing badhappens”, whereas a liveness requirement ensures that“something good happens eventually” or inside a threshold

¶http://www.yarp.it

Prepared using sagej.cls

Clare Dixon Verification of Autonomous Robotic Systems 6 / 1

What to Verify?

We assume an architecture where there is a separationbetween the high level decision making layer and the low levelcontrol layer.

etc

Control System

Sense and act

High level choices

Rational Agent

Low level control

Decision making

Avoidance

Reactive

Goal selection

Plan selection

Predictionetc

We aim to represent and verify the decision making layer andwe don’t deal with low level control such as movement etc.

Clare Dixon Verification of Autonomous Robotic Systems 7 / 1

A Domestic Robot AssistantHere we apply model checking to the high level behaviourscontrolling the Care-O-bot R©, developed as a mobile roboticassistant to support people in domestic environments.This is deployed in a domestic-type house (the robothouse) at the University of Hertfordshire.The robot house is equipped with sensors which provideinformation on the state of the house and its occupants,Low-level robot actions such as movement, speech, lightdisplay, etc., are controlled by groups of high-level rulesthat together define particular behaviours.

3

Fig. 2. A plan view of the ground floor of the University of Hertfordshire Robot House. Numbered boxes show the locations of sensors.

models, and their formal verification, are described inSection IV.

• Figs. 2 and 3 have been added to provide additionalinformation on the Robot House and the user activitywithin it.

• Section V on related work has been updated, and Sec-tion VI on conclusions and future work has been ex-tended.

II. MODELLING THE CARE-O-BOT USING BRAHMS

The autonomous decision making within the Robot Houseand Care-O-bot R� at the University of Hertfordshire is carriedout by a high-level planning/scheduling system described inthe previous section. The code base includes a database of 31default rules for the Robot House and Care-O-bot to follow.Careful examination of these rules revealed that they aresimilar in structure to the various constructs within the Brahmsmulti-agent workflow programming language.

The first step in modelling was to convert the full set ofCare-O-bot rules into a more convenient if-then rule repre-sentation. For example, the rule in the previous section wasrewritten as:

IF tray_is_raised AND tray_is_emptyTHEN set_light(yellow)

move_tray_and_wait(lowered_position)set_light(white)wait()set(tray_is_raised,false)set(tray_is_lowered,true)

Once translated into this format, these rules could then bestraightforwardly translated into Brahms. A key concept inBrahms is the ‘workframe’, which specifies a sequence ofthings to be done when a given condition holds. The RobotHouse rules were translated into Brahms workframes withinthe Care-O-bot agent, with the IF a THEN b rules trans-lated into the when a do { b } construct in Brahms.For example, the rule above was translated into a Brahmsworkframe called wf_lowerTray:

workframe wf_lowerTray {repeat: true;priority: 10;

when(knownval(current.trayIsRaised = true)andknownval(current.trayIsEmpty = true))

do{conclude((current.lightColour =

current.colourYellow));lowerTrayAndWait();conclude((current.lightColour =

current.colourWhite));waitForLightColourChange();

Clare Dixon Verification of Autonomous Robotic Systems 8 / 1

Care-O-bot Decision Making: Behaviours

The Care-O-bot’s high-level decision making is determinedby a set of behaviours of the form precondition→ action(each a sequence of rules).Examples of high-level rules can take the form “lower tray”,“move to sofa area of the living room”, “say ‘The fridgedoor is open’ ”, set a flag, check a sensor etc.Only one behaviour executes at once.Each behaviour has a priority (integer between 0 and 90).Higher priority behaviours are executed in preference tolower priority behaviours.Each behaviour is flagged as interruptible or not.Once it has started executing, a behaviour will execute tocompletion, if it is not interruptible.Users can add new behaviours.

Clare Dixon Verification of Autonomous Robotic Systems 9 / 1

The S1-watchTV Behaviour

IF the living room sofa is occupied ANDthe television is on ANDGOAL-watchTV is false ANDhas been false for more than 3600 seconds

THENturn light on CoB to yellowexecute sequence ’lowerTray’move CoB to Sofa Area in the Living Room

and wait for completionturn light on CoB to whiteCoB says ’Shall we watch TV together?’

and wait for completionset GOAL-watchTV TO truedisplay on CoB GUI: S1-Set-Watch-TV, S1-Set-Continue,

S1-Set-ReturnHome

Its priority is 30 and it is interruptible.Clare Dixon Verification of Autonomous Robotic Systems 10 / 1

The Alpha Algorithm

We consider Nembrini’s Alpha Algorithm.

This makes use of local wireless connectivity information aloneaiming to achieve swarm aggregation.

Every cycle the robot checks how many nearby robots it candetect.

α is a parameter stating the number of other robots a robotmust be able to detect to be connected.

Clare Dixon Verification of Autonomous Robotic Systems 11 / 1

The Alpha Algorithm

Every cycle the robot checks how many nearby robots it candetect.

Each robot can be in exactly one of the forward or coherentmodes and is connected or not connected.

In the forward mode, when connected, move forward andthe motion mode remains forward.In the forward mode, but not connected, turn 180◦ andchange the motion mode to coherent.In the coherent mode, but not connected, move forwardand the motion mode remains as coherent.In the coherent mode, when connected, perform a randomturn and change the motion mode to forward.

Clare Dixon Verification of Autonomous Robotic Systems 12 / 1

Abstractions

We need to abstract away from many details to obtain adiscrete and finite representation that is not too large.

Things to consider include:

timing details and “wait for completion” (CoB);planning to move from room to room (CoB);exact location, wheel movements, low level control (both);direction, wireless range, step size, avoidance (swarms);choice of concurrency (swarms);the need to model uncertain information (both).

Clare Dixon Verification of Autonomous Robotic Systems 13 / 1

CoB Models

We developed a (by hand) model in the input language forthe model checker NuSMV and later developed a tool(CRutoN) to automatically translate from behaviours toNuSMV input.CRutoN allowed us to translate from different databases ofbehaviours into input for a model checker, settingparameters to control particular aspects of the translation.CRutoN uses an intermediate representation so that inputto different model checkers can potentially be generated.We also modelled the scenario in Brahms, a language formodel human-robot-agent teamwork that has beendeveloped and used at NASA.We then used an existing tool to translate from Brahmsmodels into the model checker Spin.

Clare Dixon Verification of Autonomous Robotic Systems 14 / 1

Swarms: Models–Location and Movement

We assumed a number of identical robots moving about ann × n square grid with at most one robot in each square.We assumed a step size of one grid square and a robotcan detect other robots for avoidance in the adjacentsquares.Each robot is moving in some direction (North, South,East, West).The random turn is a turn 90◦ right or left.We assume the grid wraps round. Thismeans a robot could move in one directionforever.We consider different types ofconcurrency: synchrony, strict andnon-strict turn taking and fair asynchrony.

Clare Dixon Verification of Autonomous Robotic Systems 15 / 1

CoB and Swarm: Properties

Ideally properties would come from a specification orstandards documents about what is expected of therobot(s) with respect to functionality, safety etc.For the handover task we considered the probability ofsuccessful handovers.For the CoB we focus on issues relating to the schedulingof behaviours, priorities and interruptions (which at leastprovide a sanity check).For the swarm coni is a derived proposition which is truefor robot i if there are at least α robots within its wirelessrange and false otherwise.We aim to verify ♦coni for each of the i robots (notethis is not exactly what we want as it may result indisconnected subgroups groups).

Clare Dixon Verification of Autonomous Robotic Systems 16 / 1

CoB: Results

((sofa_occupied ∧ tv_on ∧ ¬goal_watch_tv ∧ goal_watch_tv_time = m60)⇒♦(location = sofa ∧ say = shall_we_watch_tv))

Property Output Time (sec)5 FALSE 11.6

Even if we strengthen the left hand side of the formula toassume the behaviour is also scheduled this property shouldfail as it is interruptible.

The model had 130,593 reachable states.We did find a small bug in the behaviours (a flag waswrongly set) but this was by inspection of the behaviours.It would be better to try properties relating to therequirements of the robot.

Clare Dixon Verification of Autonomous Robotic Systems 17 / 1

CoB: Discussion

CRutoN allowed us to translate from different sets ofbehaviours (of a similar format), control particular aspectsof the translation and potentially provide input to differentmodel checkers (e.g. relating to uncertainty or timings).Modelling using Brahms is potentially more general but thedecision making of the robot still has to be encoded as aBrahms model.Understanding the semantics of the robot execution cycletook a lot of close work and interaction with UoH.The state explosion problem means we have to find abalance between the level of detail/abstraction andverification times (timing details were not well represented).The model of a person in the robot house was notrepresented but this could be incorporated showing theirlocation for example.

Clare Dixon Verification of Autonomous Robotic Systems 18 / 1

Swarms: Results

We applied model checking to a variety of grid sizes, numbersof robots, wireless range, and alpha parameters but can onlydeal with a small number of robots and small grid sizes.

Certain (grid independent) types of failing trace can be unfoldedinto a larger or infinite grid (so no need to check larger grids).

Using model checking we obtain the following failing traces.

These results confirm a known problem with the alphaalgorithm, when a robot or group of robots is linked to the restof the swarm by a single link (known as a bridge or cutvertex).

Clare Dixon Verification of Autonomous Robotic Systems 19 / 1

Swarms: Discussion

Due to the state explosion problem we can only considersmall number of robots and small grid sizes.

Techniques such as symmetry may help to verify largernumber of robots (e.g. Kerstin considered the relativepositions of the robots).Techniques that ensure once a property has been shownfor n robots it holds for all m ≥ n are also useful.We ignored the relative size of parameters such as stepsize, avoidance detection and wireless detection ranges.For the foraging robots we used a population model andprobabilistic verification with the model checker PRISM.

Clare Dixon Verification of Autonomous Robotic Systems 20 / 1

Concluding Remarks

We discussed experiences applying formal verification to two totwo main types of system robot assistants and swarm robots.

We considered issues such as what to verify, the developmentof models, properties, and results.

Formal verification of such systems involves abstractions of thereal world to develop models and properties but verificationover these models is exhaustive.

The state explosion problem is an issue for such formalverification.

We advocate the use of a suite of verification and validationtechniques at different levels of abstraction and coverability tohelp gain assurance about safety, reliability and functionalcorrectness.

Clare Dixon Verification of Autonomous Robotic Systems 21 / 1

Overall Approach

Clare Dixon Verification of Autonomous Robotic Systems 22 / 1