Requirements Based Testing (Powerpoint Talk)

Advanced Technology Center Slide 1

Requirements-Based TestingRequirements-Based Testing

Dr. Mats P. E. HeimdahlUniversity of Minnesota Software Engineering Center

Dr. Steven P. Miller

Dr. Michael W. WhalenAdvanced Computing Systems

Rockwell Collins

400 Collins Road NE, MS 108-206

Cedar Rapids, Iowa 52498

[email protected]


Outline of PresentationOutline of Presentation

Motivation

Validation Testing

Conformance Testing

What’s Next


How We Develop SoftwareHow We Develop Software

SW High-Level Reqs. Development

SW Design Description Dev. (SW Low-Level

Reqs. & SW Arch.

SW Source Code Dev.

SW Integration (Executable Code Production)

SW Low-Level Testing

SW Integration Testing

HW/SW Integration Testing


How we Will Develop SoftwareHow we Will Develop Software(From V to a Y)(From V to a Y)


Software Model




Can we trust the code

generator?

How do we know our model is

correct?

Validation TestingFormal

Verification

Conformance Testing



Motivation

Validation Testing

Conformance Testing

What’s Next




Software Model




How do we know our model is

correct?


Modeling ProcessModeling Process


Software Model


Desired Model Properties

High-Level Requirements

Low-Level Requirements


Problem—Modeling FrenzyProblem—Modeling Frenzy


Software Model



Headfirst into m

odeling

How do we know the model is

“right”?How do we test the

model?


One Solution: Redefine RequirementsOne Solution: Redefine Requirements

System Reqs. Development

Software Model




Software Development

Processes(DO-178B)


Processes(DO-178B)

System Development

Processes(ARP4754)

System Development

Processes(ARP 4754)

The model is the requirements

Use Engineering Judgment when

Testing


One Solution: Redefine RequirementsOne Solution: Redefine Requirements


Software Model





Processes(DO-178B)


Processes(DO-178B)

System Development

Processes(ARP4754)

System Development

Processes(ARP 4754)

The model is the requirements

Use Engineering Judgment when

Testing

My Com

ment


Testing Does not go AwayTesting Does not go Away


Software Model




Extensive Testing (MC/DC)


It Simply MovesIt Simply Moves


Software Model




Extensive Testing (MC/DC)


Do it Right!Do it Right!


Software Model



Analysis (Model Checking,Theorem Proving)Specification Test –

Is the Model Right?


How Much to Test?How Much to Test?

State Coverage

Masking MC/DC?

Transition

Coverage?

Decision Coverage

?

Def-Use Coverage

?

Something New??

MC/DC

Where Do the TestsCome From?


Properties are Requirements…

Requirements Based TestingRequirements Based Testing


Software Model



Cover the Properties!


Properties are RequirementsProperties are Requirements


Requirements Based Testing Requirements Based Testing AdvantagesAdvantages

Objective Measurement of Model Validation Efforts– Requirements Coverage in Model-based Development

– Help Identify Missing Requirements• Measure converge of model

Basis for Automated Generation of Requirements-based Tests– Even If Properties Are Not Used for Verification, They Can Be

Used for Test Automation

How Are Properties “Covered” with Requirements-based Tests?


Property CoverageProperty Coverage

“If the onside FD cues are off, the onside FD cues shall be displayed when the AP is engaged”– G(((!Onside_FD_On & !Is_AP_Engaged) -> X(Is_AP_Engaged -> Onside_FD_On))

Property Automata Coverage– Cover a Synchronous Observer Representing the

Requirement (Property)

Structural Property Coverage– Demonstrate Structurally “Interesting” Ways in Which

the Requirement (Property) Is Met


Property Automata CoverageProperty Automata Coverage

Cover Accepting State Machine As Opposed to Structure of Property

Büchi Coverage– State Coverage, Transition Coverage, Lasso

Coverage…

S1S0

Is_AP_Engaged vOnside_FD_On

not Is_AP_Engaged ^not Onside_FD_On

not Is_AP_Engaged ^not Onside_FD_On

Onside_FD_On

1

2

3

4


Alternative MachineAlternative Machine

Different synthesis algorithms give different automata– Will affect the test cases

required for coverageS0 S1

S4 S2 S5

S3

S6 S7

Init

a

a

b

a

bb

b

a, b

a, b

a, b

abb

a

!a, b

!a, b

!a, b

b

b

b

!a

!a

!a

b

b

b

a

!a, !b

!a, !b

!a, !b !a, !b

!a, !b

!a, !b


Structural Property CoverageStructural Property Coverage

Define Structural Coverage Criteria for the Property Specification– Traditional Condition-based Criteria such as MC/DC

Prime Candidates

Property Coverage Different than Code Coverage– Coverage of Code and Models

• Evaluate a decision with a specific combination of truth values in the decision

– Coverage of Properties• Run an execution scenario that illustrates a specific way a

requirement (temporal property) is satisfied


ExampleExample

– G(((!Onside_FD_On & !Is_AP_Engaged) -> X(Is_AP_Engaged -> Onside_FD_On))

Demonstrate That Somewhere Along Some Execution Trace Each MC/DC Case Is Met– Only the “positive” MC/DC cases

• The negative cases should have no traces

In the Case of G(p)—Globally p Holds—we Need to Find a Test Where– in the prefix the requirement p is met

– we reach a state of the trace where the requirement p holds because of the specific MC/DC case of interest – let us call this case a

– then the requirement p keeps on holding through the remainder of the trace

p U ( a U X(G p))

p p a p p p


SummarySummary

Objective Measurement of Model Validation Efforts– Requirements Coverage in Model-based Development– Help Identify Missing Requirements

Basis for Automated Generation of Requirements-based Tests– Even If Properties Are Not Used for Verification, They Can Be Used for

Test Automation and Test Measurement

Challenges – How Are Properties Specified?

• Combination of Observers and Temporal Properties

– What Coverage Criteria Are Suitable?– How Is Automation Achieved?– How Do We Eliminate “Obviously” Bad Tests? Should We?– How Do We Generate “Realistic” Test-cases?– Rigorous Empirical Studies Badly Needed



Motivation

Validation Testing

Conformance Testing

What’s Next




Software Model




Can we trust the code

generator?


““Correct” Code Generation—How?Correct” Code Generation—How?

Provably Correct Compilers– Very Hard (and Often Not

Convincing)

Proof Carrying Code

Generate Test Suites From Model– Compare Model Behavior

With Generated Code

– Unit Testing Is Now Not Eliminated, but Largely Automated

Specification/Model

Implementation

Output

Output

Specification Based Tests

Generate


Existing CapabilitiesExisting Capabilities

Several Commercial and Research Tools for Test-Case Generation– TVEC

• Theorem Proving and Constraint Solving techniques

– Reactis from Reactive Systems Inc. • Random, Heuristic, and Guided Search

– University of Minnesota• Bounded Model Checking

– NASA Langley• Bounded Model Checking/Decision Procedures/Constraint Solving

Tools Applicable to Relevant Notations– In Our Case Simulink


An Initial ExperimentAn Initial Experiment

Used a Model of the Mode Logic of a Flight Guidance System As a Case Example

Fault Seeding– Representative Faults– Generated 100 Faulty Specifications

Generate Test Suites– Selection of Common (and Not So Common) Criteria

Fault Detection– Ran the Test Suites Against the Faulty Specifications– Recorded the Total Number of Faults Detected


Fault Finding ResultsFault Finding Results

0

10

20

30

40

50

60

70

80

90

100

Variable Domain

Transition

Decision

Decision (use)

Masking MC/DC

Masking MC/DC (use)

MC/DC

MC/DC (use)

Random

Same Effort


Model “Cheats” Test Generator Model “Cheats” Test Generator

FGSR

ModeLogic

ControlLaws

FGSL

ModeLogic

ControlLaws

AutopilotPFDRPFDL

Air DataL

FMSL

Air DataR

FMSR

FCP

ControlSurfacesFCS

Architecture


Effect of Test Set SizeEffect of Test Set Size

0102030405060708090

100

Variable Domain

Transition

Decision

Decision (use)

MC/DC

MC/DC (use)

Full Reduced


SummarySummary

Automated Generation of Conformance Tests– Current Technology Largely Allows This Automation

Challenges – Development of Suitable Coverage Criteria

– Effect of Test Set Size on Test Set Effectiveness

– Effect of Model Structure on Coverage Criteria Effectiveness

– Traceability of Tests to Constructs Tested

– Empirical Studies of Great Importance



Motivation

Conformance Testing

Validation Testing

What’s Next


New Challenges for TestingNew Challenges for Testing

Model Validation – Requirements-based Testing– How Do We Best Formalize the Requirements?– What Coverage Criteria Are Feasible?– Which Coverage Criteria Are Effective (If Any)?– How Do We Generate “Realistic” Tests?– Will This Be a Practical (Tractable) Solution?

Conformance Testing– What Coverage Criteria Are Effective?

• Detecting Faults From Manual Coding

• Detecting Faults From Code Generation

– Relationship Between Model Structure and Criteria Effectiveness

– Traceability From Tests to Model– Relationship Between Model Coverage and Code Coverage

• Optimizations in Code Generator Will Compromise Coverage


DiscussionDiscussion


Perfection is Perfection is NotNot Necessary Necessary

Tools and Models Only Need To Be Better Than Manual Processes…– How Do We Demonstrate This?

• Empirical Studies Are of Great Importance

≥Missed Faults

I Think Many Already Are


DO-178B Test ObjectivesDO-178B Test Objectives

1. The executable code complies with the high-level requirements.

2. The executable code complies with the specification (low-level requirements).

3. Test coverage of high-level requirements is achieved

4. Test coverage of specification (low-level requirements) is achieved

5. Test coverage of the executable code is achieved

Requirements-Based Testing

Conformance Testing

Requirements Based Testing (Powerpoint Talk)

Documents

Transcript of Requirements Based Testing (Powerpoint Talk)