Languages
Pages
Legal
Cyrille Comar
Integrating Proof & Testing in Verification
Strategies
Agenda
• Place of Testing in Safety Standards
• Programming with executable Contracts
• Executable Contracts vs Formal Contracts
• Reducing Testing with Formal Proofs
DO-178 define mainly 6 sets of objectives related to
• Planning (7)
• Development (7)
• Verification (43)
• Configuration Management (6)
• Quality Assurance (5)
• Certification Liaison (3)
Place of Testing in DO-178
ü Various reviews & analysis
ü Requirements-Based Testing
SLR HLR LLR
Requirements in DO-178
• There are 3 levels requirements
• System Requirements (SLR)
• High-Level Requirements (HLR)
• Low-Level Requirements (LLR)
Allocated to
Software
Software Architecture
SLRs HLRs
LLRs
Requirements- Based (RB) Testing in DO-178
RB Test Generation
Low-Level Tests
Software Integration
Tests
Hardware / Software
Integration Tests
RBTest Coverage Analysis
Structural Coverage Analysis
Statements Decisions
MCDC
LLR HLR
Integrated software
Robustness vs functional test cases in DO-178
• 2 kinds of test cases
• Normal Range Test Cases
• Robustness Test Cases
“ … ability of the software to respond to inputs and conditions … ”
normal
abnormal
From outside
(response should be specified in SLR/HLR)
From inside
(Software error)
comp1 comp2 comp3
Low-Level Testing is long & difficult
• Is it as relevant as HLR based testing ?
• LLR verification is important but is “testing”
the most appropriate one?
subp1
subp2
subp4 subp3
subp5
subp6 Test harness Subp4
stub
Subp3
Programming by Contract
• A CONTRACT is
• A logical property
• Associated with a program entity
• It can be verified
• It can be relied upon
• Precondition • Postcondition
• Invariant • Predicate
• Assertion
• subprogram
• type
• statement
Programming By Contract ( Ada examples)
procedure Square_Root (Input : Integer; Res : out Natural) with pre => Input >= 0, post => Res * Res <= Input and (Res+1)*(Res+1) > Input;
type Positive_Stack is private with Invariant => Is_Empty (Stack) or else Top (Stack) > 0;
subtype Odd_Positive is Natural with Predicate => (Odd_Positive/2)*2 /= Odd_Positive;
Implicit contract
Explicit contract
What can Contracts be used for?
• Adding user-defined dynamic checks at runtime
• A way of expressing constraints, properties
LLRs
“When a low level requirement can be expressed as a contract, it can be tested much more easily”
Robustness Testing & Contracts
• Internal Robustness issues are related to Run-Time Errors
• Ada guarantees (most cases of) RTEs è run-time exception
• Preconditions are better than “defensive coding”
• The more runtime checks the better Robustness Testing is
Requirement Testing & Contracts
• LLR is expressed as a subprogram contract
• Successful execution of Postcondition è Test successful
• No need for collecting & verifying output
Executable Contracts vs Formal Contracts
• Execution of “executable” contracts can be disabled
• The syntactic difference is not relevant
• The same contracts can be interpreted in 2 different world
procedure Square_Root (Input : Integer; Res : out Natural) with pre => Input >= 0, post => Res * Res <= Input and (Res+1)*(Res+1) > Input; (Ada 2012)
procedure Square_Root (Input : Integer; Res : out Natural) -‐-‐# pre Input >= 0; -‐-‐# post Res * Res <= Input -‐-‐# and (Res+1)*(Res+1) > Input; (SPARK 2005)
Executable boolean expression First order logic formula
Requirement based Verification & Formal Contracts
• LLR is expressed as a subprogram formal contract
• Successful proof of Postcondition è LLR verified for any input
• Approach allowed by DO-333
• Proof of absence of RTEs provides for some Robustness verification
Hybrid Verification (1)
• Some LLRs are tested
• Some LLRs are proved
• Is the combination as “strong” as all LLRs are tested?
Hybrid Verification (2) Modular Formal verification is based on assumptions:
• My Precondition is never violated by Callers
• Postconditions of callees are true after the call
• all “outputs” are known • Direct or Indirect • Explicit (out parameters) or Implicit (globals)
• No surprises (e.g. unexpected aliasing)
Hybrid Verification (3)
• Executable contracts “protect” assumptions of formal verification
• Globals can be computed by static analysis
• Surprises can be eliminated by subsetting
Tested_Function Proved_Function calls
Precondition must be executed
Proved_Function Tested_Function calls
Postcondition must be executed
Conclusion: Hybrid Verification Benefits
• Helps with gradual introduction of Formal proofs
• The traditional 80 / 20 % rule is true for both formal
verification and testing
80%
easily proved
80% remaining
easily tested
• Open-DO Initiative
• www.open-do.org
• http://www.open-do.org/projects/hi-lite/
Extra information
Top Related