Balancing Practices: Inspections, Testing, and Others

JAXA scenario (formal method)

Masa KatahiraJapanese Space Agency

Strategy to select methods

• Methods Compliment– Inspection/Review– Testing– Formal Method

• Theorem Proving • Auto Model Checking

• Inspections/Reviews– Hard to cover all aspects

• Testing– Not complete, too late in some case

• Formal Method– TP: Complex for practical use– MC: State explosion possible

• We realize that – the correct use of particular methods, – the combination of several methods

are very important.• But how?

– Quality Goals

– Budget Limitation

– System Characteristics

– Data availability, Development Phase

Selection and Scalability of Methodologies(sample)

Completeness/Consistency

Selection (Depth)Selection (Depth)Light

Full set Phases

Modeling/Model CheckingInspection/Review

(Check List)

Simulation

Interface Validation

Design Coverage & Timing

Verification Coverage

Auto Test Case Generation &

Robustness EvaluationTest Case & Test

Test Result Review

Compliance/Traceability

Risk Analysis (Robustness)

Process & Quality

Static analysis(Problem Reports)

In line Process Monitor (SMIP)

Manual Check(Tools Support)

Auto Equivalency Check

Hazard Analysis/SFMEA

REA

Ass

essm

ent A

ttri

bute

s A

sses

smen

t Att

ribu

tes

(Sam

ple)

(Sam

ple)

Still we don’t know how to decide methods’ selection!

0% 20% 40% 60% 80% 100%

A

B

C

D

E

Review Model Checking Review with Checklist

Fig.1 Each methods’ effectiveness among all significant issues

A

B

C

D

E

A

B

C

D

E

Fig.2 Each methods’ effectiveness among all Editorial Errors

Fig.3 Each methods’ effectiveness among all Significant issues and Editorial errors

ConsiderationConsideration

For projects A,B,C, there is not enough time to perform model checking. Review with check list instead. For Project D, a checklist is useful for the data correctness and consistency of data handling system.For Project E, the effective of model checking is confirmed due to having enough time.

Review also shows important role in erroneous description in the specificationFor design phase such as C,E, model checking and a checklist help finding errors which can not be found by review.

For projects A,B,C, there is not enough time, and the review with check list shows efficiency.For Project D, Review as well as review with checklist shows the efficiency for data handling system.

For such as Project E case, when enough time is assigned, the model checking shows good results.

Lessons Learned Summaryin JAXA case study

Method Advantage Disadvantage

Formal Model/Model Checking

●It is useful to find the problem concerning the complicated state/mode transition and processing timing issues which is hard to be found by manual.●Erroneous description in the spec. at modeling which is more effective than normal review.

●A certain mount of time are necessary to modeling and model checking.●Need to know modeling and model checking knowledge.●Low cost effectiveness for software which does not have complicated logic such as data handing, or transformation

Review withChecklist (Inspection)

●High cost effectiveness even if there is very short time to access it.●Erroneous descriptions are covered by check items in the list.●It does not depend on the skill of evaluators than model methods.

●There are limited based on the items in the checklist. ●It is hard to check the detailed behaviors in the complex system and to cover the possible combination.

Boundary of Formal Method Application

Available man-month

Safe

ty C

riti

cal

Syst

em C

hara

cter

istics

Compl

exity

Important Border

JAXA formal method activity

Needs and remaining issues of formal method

[Problem Statement]• Need to assure the high reliability of spacecraft• Facing to the difficulty to prove the goodness only by test and inspection

because of system complexity and safety requirements such “must not work”

• Large number of defects are introduced mostly at the Requirement Phase or the Early Design Phase. Unintended or Unexpected system/software behavior is difficult to be found at the inspection/review by manual.

[Challenge]• Knowledge base inspection/review is still very important, but model

checking gives a chance to detect important findings which are easy to miss by the reviewer.

• Modeling task itself gives a chance to think enough deeply what the specification really says as if the reviewers build the software by themselves.

[Remaining Issues]• Quality of Model and model checking task

– Large amount of time is spent to correct the erroneous model• Abstraction and partitioning techniques

– To avoid state explosion, and missing the scenarios• Better Productivity

– Hard to find real problems from thousands of auto checking results• Personnel skill

Modeling and model checking in JAXA

Requirement Model

(SpecTRM*1, Uppaal)

DesignModel

(SpecTRM, Uppaal)

Req. Spec.ICDHazard Report

Design Spec.ICDHazard Report

Modeling

Natural Language Input Tools

Natural Language Input Tools

UppaalUppaal

Flow DiagramToolFlow DiagramTool

Model Checking(Static Analysis)

CompletenessConsistencyReachability

CompletenessConsistencyReachability

EquivalencyEquivalency

SMV, SPINSMV, SPIN

Executable Code

Simulation(Dynamic Analysis)

RobustnessRobustness

Behavior AnalysisBehavior Analysis

FindingsFindings

Findings

Test Case Proposal

Operation Model

ProcedureOpe. Scenario

ConsistencyTask Model ToolTask Model Tool

*1:SpecTRM: Specification Tools and Requirements Methodology

Direct Modeling

Productivity improvement

0

2

4

6

8

10

12

14

DirectModeling

NaturalLanguage

Model

Flow ChartModeling

Modeling Task Cost

Real issues from the results of the modeling and model checking

0 0.2 0.4 0.6 0.8 1

Proj A/ Req.

Proj A/ Preli Design

Proj A/ Detail Design

Proj B/ Req.

Proj B/ Preli Design

Proj B/ Detail Design

Proj C Design

Modeling Task

Consistency Analysis

Completeness Analysis

SPIN

Identified issues in the specification

• Modeling – Can organize information and execute the modeling in the brain – Identify lots of basic problems in the specification (ambiguous

descriptions, inconsistency of the contents, unclear data definition) as to make the accurate model of the specification in the formal language

• Automated Consistency analysis– Effective to identify the inconsistency in the requirement specification– Identify the inconsistency among the procedures in case that multiple

tasks executions are allowed simultaneously in the design level. • Automated Completeness analysis

– Check whether the nodes after the transition at the branch in the flowchart meet the number of the transition conditions and its contents, and whether all error handling and exceptional procedure are covered in the design level.

• Formal Validation of the functional behavior using SPIN – Effective to validate whether the procedures are executed without

stagnation and those behavior meet the requirements for the procedure flow in the detailed design

– Effective to verify whether hazard control function/failure recovery functions are working without unintended stop in the real time

Lessons Learned from industry use of modeling and model checking

Questions? (Formal Method)A) What is the role of formal method (Theorem Proving, Model

checking etc.) in many quality practices?

B) When is a Formal Method necessary or efficient?

C) What is a Formal Method useful for? Specific Aspects?

D) What are most important research issues to deploy the method into real projects? Industry Needs?

E) What empirical data gathered at the industry will be useful to future research?

F) What is an expected benefit from use of formal method?

Backup Slide

Findings from each methods(Spacecraft’s Projects)

Significant: Signification Issues to be modified such as incorrect or missing functions/logic/data

Editorial: Editorial Errors in the specification

No Issue: Non real problem (misunderstanding/modeling mistake)

Type of system

Findings type and number from each methods

Review Formal Modeling and Model Checking

Review with Checklist

Significant Editorial No Issue Sum

System A(Controller) /Req.

0 1 0 1 1 1 0 2 2 2 1 5

System B(Controller) /Req.

8 4 4 16 4 2 1 7 9 2 1 12

System C(Controller)/Design

0 0 2 2 1 0 1 2 1 5 4 10

System D(DataHandling), Design

0 24 34 58 2 0 3 5 4 0 5 9

System E(Controller)/Design

2 0 27 29 3 13 8 24 2 0 32 34

Significant Editorial No Issue Sum Significant Editorial No Issue Sum

SpecTRM (Model) Based Robustness Test Environment (SpecRobusT)

Outline:• By using specification models, the important test cases are generated for full software simulation during

development contractor’s test phase automatically and comparing results.• Especially, all inputs are verified in the model to generate the test cases.• Auto tests are performed at 10,000 – 100,000 cases / sec.Results of Project application:• # of Test Case ： 550,870,000,000 • Benefits:

– Verification at very early phase – Introduction to automated test environment – Introduce “Test Before Development” paradigm into development process

Implementation Procedure