Verifying Distributed Adaptive Real-Time (DART) Systems · 2015. 10. 16. · Summary. Distributed...

19
© 2015 Carnegie Mellon University Software Engineering Institute Carnegie Mellon University Pittsburgh, PA 15213 Distribution Statement A: Approved for Public Release; Distribution is Unlimited Verifying Distributed Adaptive Real-Time (DART) Systems Sagar Chaki, Dionisio de Niz October 8, 2015

Transcript of Verifying Distributed Adaptive Real-Time (DART) Systems · 2015. 10. 16. · Summary. Distributed...

Page 1: Verifying Distributed Adaptive Real-Time (DART) Systems · 2015. 10. 16. · Summary. Distributed Adaptive Real -Time (DART) systems promise to revolutionize several areas of DoD

© 2015 Carnegie Mellon University

Software Engineering InstituteCarnegie Mellon UniversityPittsburgh, PA 15213

Distribution Statement A: Approved for Public Release; Distribution is Unlimited

Verifying Distributed Adaptive Real-Time (DART) SystemsSagar Chaki, Dionisio de NizOctober 8, 2015

Page 2: Verifying Distributed Adaptive Real-Time (DART) Systems · 2015. 10. 16. · Summary. Distributed Adaptive Real -Time (DART) systems promise to revolutionize several areas of DoD

2SEI Research Review 2015October 7–8, 2015© 2015 Carnegie Mellon UniversityDistribution Statement A: Approved for Public Release; Distribution is Unlimited

Copyright 2015 Carnegie Mellon University

This material is based upon work funded and supported by the Department of Defense under Contract No. FA8721-05-C-0003 with Carnegie Mellon University for the operation of the Software Engineering Institute, a federally funded research and development center.

Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the United States Department of Defense.

NO WARRANTY. THIS CARNEGIE MELLON UNIVERSITY AND SOFTWARE ENGINEERING INSTITUTE MATERIAL IS FURNISHED ON AN “AS-IS” BASIS. CARNEGIE MELLON UNIVERSITY MAKES NO WARRANTIES OF ANY KIND, EITHER EXPRESSED OR IMPLIED, AS TO ANY MATTER INCLUDING, BUT NOT LIMITED TO, WARRANTY OF FITNESS FOR PURPOSE OR MERCHANTABILITY, EXCLUSIVITY, OR RESULTS OBTAINED FROM USE OF THE MATERIAL. CARNEGIE MELLON UNIVERSITY DOES NOT MAKE ANY WARRANTY OF ANY KIND WITH RESPECT TO FREEDOM FROM PATENT, TRADEMARK, OR COPYRIGHT INFRINGEMENT.

This material has been approved for public release and unlimited distribution except as restricted below.

This material may be reproduced in its entirety, without modification, and freely distributed in written or electronic form without requesting formal permission. Permission is required for any other use. Requests for permission should be directed to the Software Engineering Institute at [email protected].

DM-0002760

Page 3: Verifying Distributed Adaptive Real-Time (DART) Systems · 2015. 10. 16. · Summary. Distributed Adaptive Real -Time (DART) systems promise to revolutionize several areas of DoD

3SEI Research Review 2015October 7–8, 2015© 2015 Carnegie Mellon UniversityDistribution Statement A: Approved for Public Release; Distribution is Unlimited

BackgroundDistributed Adaptive Real-Time (DART) systems are key to many areas of DoD capability (e.g., autonomous multi-UAS missions) with civilian benefits.

However achieving high assurance DART software is very difficult • Concurrency is inherently difficult to reason about.• Uncertainty in the physical environment.• Autonomous capability leads to unpredictable behavior.• Assure both guaranteed and probabilistic properties.• Verification results on models must be carried over to source code.

High assurance unachievable via testing or ad-hoc formal verification

Goal: Create a sound engineering approach for producing high-assurance software for Distributed Adaptive Real-Time (DART)

Page 4: Verifying Distributed Adaptive Real-Time (DART) Systems · 2015. 10. 16. · Summary. Distributed Adaptive Real -Time (DART) systems promise to revolutionize several areas of DoD

4SEI Research Review 2015October 7–8, 2015© 2015 Carnegie Mellon UniversityDistribution Statement A: Approved for Public Release; Distribution is Unlimited

DART Approach

System + Properties

(AADL + DMPL)Verification Code

Generation

1. Enables compositional and requirement specific verification

2. Use proactive self-adaptation and mixed criticality to cope with uncertainty and changing context

Demonstrate on DoD-relevant model problem (DART prototype)

• Engaged stakeholders

• Technical and operational validity

1. ZSRM Schedulability (Timing)

2. Software Model Checking (Functional)

3. Statistical Model Checking (Probabilistic)

Brings Assurance to Code

1. Middleware for communication

2. Scheduler for ZSRM

3. Monitor for runtime assurance

Page 5: Verifying Distributed Adaptive Real-Time (DART) Systems · 2015. 10. 16. · Summary. Distributed Adaptive Real -Time (DART) systems promise to revolutionize several areas of DoD

5SEI Research Review 2015October 7–8, 2015© 2015 Carnegie Mellon UniversityDistribution Statement A: Approved for Public Release; Distribution is Unlimited

Example: Self-Adaptive and Coordinated UAS Protection

High Hazard

Area

Adaptation: Formation change (loose ⇔ tight)

Loose: fast but high leader exposure

Tight: slow but low leader exposure

Low Hazard

Area

Loose Formation

Tight Formation

Challenge: compute the probability of reaching end of mission in time 𝑻𝑻 while never

reducing protection to less than 𝑿𝑿.

Challenge: compare between different adaptation strategies.

Solution: Statistical model checking (SMC)

Page 6: Verifying Distributed Adaptive Real-Time (DART) Systems · 2015. 10. 16. · Summary. Distributed Adaptive Real -Time (DART) systems promise to revolutionize several areas of DoD

6SEI Research Review 2015October 7–8, 2015© 2015 Carnegie Mellon UniversityDistribution Statement A: Approved for Public Release; Distribution is Unlimited

Example: Self-Adaptive and Coordinated UAS Protection

High Hazard

Area

Adaptation: Formation change (loose ⇔ tight)

Loose: fast but high leader exposure

Tight: slow but low leader exposure

Low Hazard

Area

Loose Formation

Tight Formation

Video

Page 7: Verifying Distributed Adaptive Real-Time (DART) Systems · 2015. 10. 16. · Summary. Distributed Adaptive Real -Time (DART) systems promise to revolutionize several areas of DoD

7SEI Research Review 2015October 7–8, 2015© 2015 Carnegie Mellon UniversityDistribution Statement A: Approved for Public Release; Distribution is Unlimited

Architecture

DMPL

Proactive Self-

Adaptation

Statistical Model

Checking

MADARA

ZSRM Scheduling

Functional Verification

Constrain the system structure and behavior to facilitate tractable analysis and code generation

Program DART systems and specify properties in a precise manner

Use probabilistic model checker to repeatedly compute optimal adaptation strategies with bounded lookahead

Evaluate adaptation strategy quality over

mission lifetime

Efficient middleware provides distributed shared variables with

well-defined data consistency

Ensures high-critical tasks meet their

deadlines despite CPU overload

Combine model checking & hybrid analysis to

ensure end-to-end CPS correctness

Page 8: Verifying Distributed Adaptive Real-Time (DART) Systems · 2015. 10. 16. · Summary. Distributed Adaptive Real -Time (DART) systems promise to revolutionize several areas of DoD

8SEI Research Review 2015October 7–8, 2015© 2015 Carnegie Mellon UniversityDistribution Statement A: Approved for Public Release; Distribution is Unlimited

𝑁𝑁𝑁𝑁𝑁𝑁𝑒𝑒1 𝑁𝑁𝑁𝑁𝑁𝑁𝑒𝑒𝑘𝑘

Environment – network,

sensors, atmosphere, ground etc.

Low-Critical Threads (LCTs)

High-Critical Threads (HCTs)

HCT

LCT

Software for guaranteed requirements, e.g., collision

avoidance protocol must ensure absence of collisions

ZSRM Mixed-Criticality SchedulerOS/Hardware

SchedOS/HW

MADARA Middleware MADARA

Software for probabilistic requirements, e.g., adaptive path-

planner to maximize area coverage within deadline

Sensors & Actuators

Distributed Shared MemoryBaked into the

programming languages used

Design constraint enables analysis tractability

Page 9: Verifying Distributed Adaptive Real-Time (DART) Systems · 2015. 10. 16. · Summary. Distributed Adaptive Real -Time (DART) systems promise to revolutionize several areas of DoD

9SEI Research Review 2015October 7–8, 2015© 2015 Carnegie Mellon UniversityDistribution Statement A: Approved for Public Release; Distribution is Unlimited

DART Modeling and Programming Language (DMPL)

C-like language that can express distributed, real-time systems• Semantics are precise• Supports formal assertions usable for model checking and

probabilistic model checking• Physical and logical concurrency can be expressed in sufficient

detail to perform timing analysis• Can call external libraries• Generates compilable C++

Developed syntax, semantics, and compiler (dmplc)

DMPL supports the right level of abstraction to formally reason about DART systems

Open Source Release on Github

Page 10: Verifying Distributed Adaptive Real-Time (DART) Systems · 2015. 10. 16. · Summary. Distributed Adaptive Real -Time (DART) systems promise to revolutionize several areas of DoD

10SEI Research Review 2015October 7–8, 2015© 2015 Carnegie Mellon UniversityDistribution Statement A: Approved for Public Release; Distribution is Unlimited

t=0

t=1

p3p2

T1

p1

T2 T1 T2 T1 T2

p3p2

T1

p1

T2 T1 T2 T1 T2

T1 T2

p3p2

T1

p1

T1 T2

T2

system

t=1

environment

t=0

non-deterministic

probabilistic

deterministic

PRISMstrategy synthesis Resolves

nondeterministic choices to maximize expected

value of objective function

First choice independent of subsequent

environment transitions Ongoing work: replace probabilistic model checking with dynamic programming for speed.

Page 11: Verifying Distributed Adaptive Real-Time (DART) Systems · 2015. 10. 16. · Summary. Distributed Adaptive Real -Time (DART) systems promise to revolutionize several areas of DoD

11SEI Research Review 2015October 7–8, 2015© 2015 Carnegie Mellon UniversityDistribution Statement A: Approved for Public Release; Distribution is Unlimited

𝑅𝑅𝑒𝑒𝑅𝑅𝑅𝑅𝑅𝑅𝑡𝑡1log-

analyzelog-genlog-genlog-genlog-

analyzelog-analyze

𝑅𝑅𝑒𝑒𝑅𝑅𝑅𝑅𝑅𝑅𝑡𝑡1𝑅𝑅𝑒𝑒𝑅𝑅𝑅𝑅𝑅𝑅𝑡𝑡𝑛𝑛

Update 𝑅𝑅𝑒𝑒𝑅𝑅𝑅𝑅𝑅𝑅𝑡𝑡and 𝑅𝑅𝑅𝑅

𝑅𝑅𝑅𝑅acceptable?

𝑁𝑁𝑁𝑁

𝑌𝑌𝑒𝑒𝑅𝑅

𝑅𝑅𝑒𝑒𝑅𝑅𝑅𝑅𝑅𝑅𝑡𝑡

Batch Log and Analyze

Each run of log-generator and log-analyzer occurs on a Virtual Machine. Multiple such VMs run in parallel on

HPC platform. Clients can be added and removed on-the-fly.

Future Work: Importance Sampling to reduce

number of simulations needed for “rare” events.

SMC Client

SMC Aggregator

Statistical Model Checking of Distributed Adaptive Real-Time Software. David Kyle, Jeffery Hansen, Sagar Chaki. In Proc. of Runtime Verifcation 2015 (to appear)

Page 12: Verifying Distributed Adaptive Real-Time (DART) Systems · 2015. 10. 16. · Summary. Distributed Adaptive Real -Time (DART) systems promise to revolutionize several areas of DoD

12SEI Research Review 2015October 7–8, 2015© 2015 Carnegie Mellon UniversityDistribution Statement A: Approved for Public Release; Distribution is Unlimited

Zero-Slack Rate Monotonic (ZSRM) software stack

• ZSRM SchedulabilityAnalysis as AADL/OSATE Plugin

• ZSRM Scheduler as Linux Kernel Module

• ZSRM Priority & Criticality Ceiling Mutexes

CPU1

CPU2

task1

task2

task1

task2

Parallelexecution

End-to-end Zero-Slack Scheduling• Based on pipelines that allows

parallel execution of multiple tasks in different stages.

• Avoids assuming all tasks start together in all stages

• Reduces the end-to-end response time and improves utilization

• Working on submission to RTSS’15

Page 13: Verifying Distributed Adaptive Real-Time (DART) Systems · 2015. 10. 16. · Summary. Distributed Adaptive Real -Time (DART) systems promise to revolutionize several areas of DoD

13SEI Research Review 2015October 7–8, 2015© 2015 Carnegie Mellon UniversityDistribution Statement A: Approved for Public Release; Distribution is Unlimited

Combining model checking of collision-avoidance protocol with reachability analysis of control algorithms via assume-guarantee reasoning

Application

Controller

Platform

𝑰𝑰𝑨𝑨𝑨𝑨

𝑰𝑰𝑨𝑨𝑪𝑪

Prove application-controller controller contract for unbounded time

• Previously limited to bounded verification only

Prove controller-platform contract via hybrid reachability analysis

• Done by AFRL

Working on automation and asynchronous model of computation

Proof of collision avoidance

DART Node

Assume-Guarantee Contract

Assume-Guarantee Contract

Page 14: Verifying Distributed Adaptive Real-Time (DART) Systems · 2015. 10. 16. · Summary. Distributed Adaptive Real -Time (DART) systems promise to revolutionize several areas of DoD

14SEI Research Review 2015October 7–8, 2015© 2015 Carnegie Mellon UniversityDistribution Statement A: Approved for Public Release; Distribution is Unlimited

Transition and application to realistic systems

Logical Isolation between Verified and Unverified Code

Big Trusted Computing Base (Compilers)

Discovered more complexity and nuances about mixed-criticality scheduling (end-to-end)

Importance sampling for distributed systems

Longer term: Fault-Tolerance, Runtime Assurance, Security

Page 15: Verifying Distributed Adaptive Real-Time (DART) Systems · 2015. 10. 16. · Summary. Distributed Adaptive Real -Time (DART) systems promise to revolutionize several areas of DoD

15SEI Research Review 2015October 7–8, 2015© 2015 Carnegie Mellon UniversityDistribution Statement A: Approved for Public Release; Distribution is Unlimited

TeamBjorn Andersson Mark KleinBud Hammons Arie GurfinkelGabriel Moreno David KyleJeffery Hansen James EdmondsonScott Hissam Dionisio de NizSagar Chaki

QUESTIONS?

https://github.com/cps-sei/dart

SummaryDistributed Adaptive Real-Time (DART) systems promise to revolutionize several areas of DoD capability (e.g., autonomous systems). We want to create a sound engineering approach for producing high-assurance software for DART Systems, and demonstrate on stakeholder guided examples.

Page 16: Verifying Distributed Adaptive Real-Time (DART) Systems · 2015. 10. 16. · Summary. Distributed Adaptive Real -Time (DART) systems promise to revolutionize several areas of DoD

16SEI Research Review 2015October 7–8, 2015© 2015 Carnegie Mellon UniversityDistribution Statement A: Approved for Public Release; Distribution is Unlimited

Contact Information

Sagar ChakiSenior MTSSSD/CSCTelephone: +1 412-268-1436Email: [email protected]

U.S. MailSoftware Engineering InstituteCustomer Relations4500 Fifth AvenuePittsburgh, PA 15213-2612USA

Webwww.sei.cmu.eduwww.sei.cmu.edu/contact.cfm

Customer RelationsEmail: [email protected]: +1 412-268-5800SEI Phone: +1 412-268-5800SEI Fax: +1 412-268-6257

Page 17: Verifying Distributed Adaptive Real-Time (DART) Systems · 2015. 10. 16. · Summary. Distributed Adaptive Real -Time (DART) systems promise to revolutionize several areas of DoD

17SEI Research Review 2015October 7–8, 2015© 2015 Carnegie Mellon UniversityDistribution Statement A: Approved for Public Release; Distribution is Unlimited

© 2015 Carnegie Mellon UniversityDistribution Statement A: Approved for Public Release; Distribution is Unlimited

Backup Slides

Page 18: Verifying Distributed Adaptive Real-Time (DART) Systems · 2015. 10. 16. · Summary. Distributed Adaptive Real -Time (DART) systems promise to revolutionize several areas of DoD

18SEI Research Review 2015October 7–8, 2015© 2015 Carnegie Mellon UniversityDistribution Statement A: Approved for Public Release; Distribution is Unlimited

Monitor ExecuteKnowledge

Analyze Plan

Target systemMAPE-K [Kephart 2003]

Proactive Self-Adaptation

Implemented proactive self-adaptation manager in a multi-UAS coordinated protection DART example. Manager adapts by changing system formation to tradeoff between energy consumption and protection provided to a mothership.

Paper presented at ACM/SIGSoft FSE’15: Gabriel Moreno, Javier Camara, David Garlan and Bradley Schmerl, "Proactive Self-Adaptation under Uncertainty: a Probabilistic Model Checking Approach".

Adaptation Decision

Page 19: Verifying Distributed Adaptive Real-Time (DART) Systems · 2015. 10. 16. · Summary. Distributed Adaptive Real -Time (DART) systems promise to revolutionize several areas of DoD

19SEI Research Review 2015October 7–8, 2015© 2015 Carnegie Mellon UniversityDistribution Statement A: Approved for Public Release; Distribution is Unlimited

dmplc Log Generator

𝓜𝓜

(DMPL)

Executable

Log (1 per

node)

Log Analyzer Result

SMC Client

SMC Aggregator

One Bernoulli

Trial

𝝓𝝓

(DMPL)