Cyrille Comar

Integrating Proof & Testing in Verification

Strategies

Agenda

• Place of Testing in Safety Standards

• Programming with executable Contracts

• Executable Contracts vs Formal Contracts

• Reducing Testing with Formal Proofs

DO-178 define mainly 6 sets of objectives related to

•  Planning (7)

•  Development (7)

•  Verification (43)

•  Configuration Management (6)

•  Quality Assurance (5)

•  Certification Liaison (3)

Place of Testing in DO-178

ü  Various reviews & analysis

ü  Requirements-Based Testing

SLR HLR LLR

Requirements in DO-178

• There are 3 levels requirements

• System Requirements (SLR)

• High-Level Requirements (HLR)

• Low-Level Requirements (LLR)

Allocated to

Software

Software Architecture

SLRs HLRs

LLRs

Requirements- Based (RB) Testing in DO-178

RB Test Generation

Low-Level Tests

Software Integration

Tests

Hardware / Software

Integration Tests

RBTest Coverage Analysis

Structural Coverage Analysis

Statements Decisions

MCDC

LLR HLR

Integrated software

Robustness vs functional test cases in DO-178

• 2 kinds of test cases

• Normal Range Test Cases

• Robustness Test Cases

 “  …  ability  of  the  software  to  respond  to                              inputs  and  conditions  …  ”  

normal  

abnormal  

From outside

(response should be specified in SLR/HLR)

From inside

(Software error)

comp1 comp2 comp3

Low-Level Testing is long & difficult

•  Is it as relevant as HLR based testing ?

• LLR verification is important but is “testing”

the most appropriate one?

subp1  

subp2  

subp4  subp3  

subp5  

subp6  Test harness Subp4

stub

Subp3

Programming by Contract

• A CONTRACT is

• A logical property

• Associated with a program entity

•  It can be verified

•  It can be relied upon

•  Precondition    •  Postcondition    

•  Invariant    •  Predicate    

•  Assertion  

•  subprogram  

•  type  

•  statement  

Programming By Contract ( Ada examples)

procedure Square_Root (Input : Integer; Res : out Natural) with pre => Input >= 0, post => Res * Res <= Input and (Res+1)*(Res+1) > Input;

type Positive_Stack is private with Invariant => Is_Empty (Stack) or else Top (Stack) > 0;

subtype Odd_Positive is Natural with Predicate => (Odd_Positive/2)*2 /= Odd_Positive;

Implicit contract

Explicit contract

What can Contracts be used for?

• Adding user-defined dynamic checks at runtime

• A way of expressing constraints, properties

LLRs

“When a low level requirement can be expressed as a contract, it can be tested much more easily”

Robustness Testing & Contracts

• Internal Robustness issues are related to Run-Time Errors

• Ada guarantees (most cases of) RTEs è run-time exception

• Preconditions are better than “defensive coding”

• The more runtime checks the better Robustness Testing is

Requirement Testing & Contracts

• LLR is expressed as a subprogram contract

• Successful execution of Postcondition è Test successful

• No need for collecting & verifying output

Executable Contracts vs Formal Contracts

• Execution of “executable” contracts can be disabled

• The syntactic difference is not relevant

• The same contracts can be interpreted in 2 different world

procedure  Square_Root        (Input  :  Integer;  Res  :  out  Natural)  with        pre    =>  Input  >=  0,      post  =>    Res      *  Res        <=  Input                and  (Res+1)*(Res+1)  >    Input;      (Ada  2012)  

procedure  Square_Root        (Input  :  Integer;  Res  :  out  Natural)  -­‐-­‐#  pre    Input  >=  0;  -­‐-­‐#  post        Res      *  Res        <=  Input    -­‐-­‐#        and  (Res+1)*(Res+1)  >    Input;        (SPARK  2005)  

Executable boolean expression First order logic formula

Requirement based Verification & Formal Contracts

•  LLR is expressed as a subprogram formal contract

•  Successful proof of Postcondition è LLR verified for any input

•  Approach allowed by DO-333

•  Proof of absence of RTEs provides for some Robustness verification

Hybrid Verification (1)

• Some LLRs are tested

• Some LLRs are proved

•  Is the combination as “strong” as all LLRs are tested?

Hybrid Verification (2) Modular Formal verification is based on assumptions:

• My Precondition is never violated by Callers

•  Postconditions of callees are true after the call

•  all “outputs” are known •  Direct or Indirect •  Explicit (out parameters) or Implicit (globals)

•  No surprises (e.g. unexpected aliasing)

Hybrid Verification (3)

•  Executable contracts “protect” assumptions of formal verification

•  Globals can be computed by static analysis

•  Surprises can be eliminated by subsetting

Tested_Function   Proved_Function  calls

Precondition must be executed

Proved_Function   Tested_Function  calls

Postcondition must be executed

Conclusion: Hybrid Verification Benefits

•  Helps with gradual introduction of Formal proofs

• The traditional 80 / 20 % rule is true for both formal

verification and testing

80%

easily proved

80% remaining

easily tested

• Open-DO Initiative

• www.open-do.org

• http://www.open-do.org/projects/hi-lite/

Extra information

Cyrille Comar +33-1-49-70-67-16

comar@adacore.com 46 Rue d’Amsterdam

Paris 75009