Verification “Auditability” Can Be “Free”

James H. Jones

Optants Documented Decision Support Company (ODDSCO)

www.optants.com

2

Verification “Auditability” Can Be “Free”1. Definitions of Key Words

• Auditability: Requirements clarity and traceability from elicitation through V&V.

• Permutation: A distinct and meaningfully separate instance of a requirement that must be part of a comprehensive verification.

• Requirement: A statement specifying a mandatory characteristic or some functional behavior to be provided by product/process.

3

Verification “Auditability” Can Be “Free”1. Definitions of Key Words (Continued)

• Validation: Determining if “the right system was built” (for customer/user).

• Verification: Determining that “the system was built right” (according to specification).

• Verification Audit Support Standards: (VASS) Use of Requirements Management methods that automate the “auditability” of requirements verification.

4

Verification “Auditability” Can Be “Free”2. Benefits of Audit Support Standards

• Assists proper system allocations.

• Supports % Complete project metrics, from each requirement to entire specification.

• Assists detailed technical review of plans and procedures for requirements V&V.

• Supports audit of V&V development tasks at any time during the project.

5

Verification “Auditability” Can Be “Free”3. System Engineering 101 (FRAT)

• Functions are product “needs” from the user or customer perspective.

• Requirements specify mandatory function, performance, and qualification objectives.

• Architecture is an implementation concept for a specification fulfilling product.

• Testing (V&V) confirms (or denies) that the system fulfills all of its requirements.

6

Verification “Auditability” Can Be “Free”4. Typical RVTM Format

[SYSTEM] REQUIREMENTS VERiCATION TRACE MATRIX

PS ID System Requirement SS ID# Allocated Requirement VP ID# Comment

3 Functional Requirements S_3 Functional Requirements N/A Header Label

3.1 Subsection 1 Title N/A N/A N/A Header Label

3.1.1 Category 1 Title N/A N/A N/A Header Label

3.1.1.1 System shall (requirement 1) E_3.4.7.3 CPU shall (provide function) E_3.1.2

3.1.1.1 S_3.3.2.1 SW shall (provide function) S_3.3.14

3.1.1.2 System shall (requirement 2) M_3.3.2.1 SW shall (provide function) S_3.2.1

... ... ... ... ... ...

4 Design Qualification S_3 Design Qualification N/A Header Label

4.1 Subsection 1 Title N/A N/A N/A Header Label

4.1.1 Category 1 Title N/A N/A N/A Header Label

... ... ... ... ... ...

7

Verification “Auditability” Can Be “Free”5. Basic Requirements Writing Rules

• Although they may swap designation with priority change, differentiate requirements and “desirements” (non-mandatory items).

• Limit requirements to one function, unless listing closely affiliated items.

• Avoid implicit requirements.

• Avoid “shall not” wording that requires test of universe to prove a negative.

8

Verification “Auditability” Can Be “Free”5. Requirement Writing Rules (Continued)

• Avoid redundancies of “be capable of”or “be able to” (use direct action verb).

• Until fully verifiable by instrumented test, demonstration, inspection, or analysis, a requirement is not valid. (So, be cautious with adverbs and, when possible, replace adjectives with measurable values, ranges, or limits.)

9

Verification “Auditability” Can Be “Free”5. Requirement Writing Rules (Continued)

• A Formal Specification Language approach uses the following standard form for all the requirement statements:

[Optional qualifier,] The (system/subsys-tem/process name) shall [use “should” or “will” if a desirement] (direct action verb statement of function) [, alternate position for the optional qualifier].

10

Verification “Auditability” Can Be “Free”5. Requirement Writing Rules (Continued)

• The foregoing basic “rules” are a small subset of many commonly listed constraints and suggestions in requirements writing training courses, but they are sufficient to provide most users a consistently effective approach.

11

Verification “Auditability” Can Be “Free”6. SRVTM (Tracking Permutations)

SUBSIDIARY REQUIREMENTS VERIFICATION TRACE MATRIX (Electrical, Mechanical, or Software Subsystem)

SS ID SS Requirement VP ID# Permutations List Comment 3 Functional Requirements N/A Header Label 3.1 Subsection 1 Title N/A Header Label 3.1.1 Category 1 Title N/A Header Label 3.1.1.1 The [subsystem] shall

(requirement 1) Permutation 1,

Permutation 2, Permutation 3 (To do)

(Probably all in ?_3.3.2)

3.1.1.2 The [subsystem] shall (requirement 2)

?_3.3.1.1/4/7/14

Permutation 1, Permutation 2, Permutation 3 (Done) Permutation 4, Permutation 5, Permutation 6, Permutation 7 (To do)

(4 VPs necessary for coverage)

... ... ... ... ...

12

Verification “Auditability” Can Be “Free”6. SRVTM (Hazard Abatement Trace)

• When a requirement in a subsidiary hardware or software specification has been assigned to accomplish necessary reduction of risk level for an item in system Hazard Analysis, insert a column for showing that hazard identifier. [Such a column is put in the System RVTM, because assignments in Hazard Analysis are to the System Spec.]

13

Verification “Auditability” Can Be “Free”7. Test Procedure Suggestions (VASS)

• Identify all Test Procedures (TPs) used for Verification of the permutations of each requirement in the Subsidiary RVTM.

• To support reviewers and auditors (when more than one TP used), list in order every requirement tested therein [and number of instances, when multiple permutations are tested] in the introduction section of each.

14

Verification “Auditability” Can Be “Free”7. Test Procedure Suggestions (Continued)

• In the Expected Results column (or in the automated test scripts or test program code comment statements [or output to the results file]), annotate each test step with both the requirement identifier and extent of testing (such as which permutation was exercised and identification of a Hazard depending on its fulfillment for risk level reduction).

15

Verification “Auditability” Can Be “Free”7. Test Procedure Suggestions (Continued)

• To ensure test step will confirm risk level reduction for an assigned Hazard, annotate the requirement identifier with that Hazard identifier.

• Reported verification success for that test step then becomes evidence that the hazard was abated by the identified action.

16

Verification “Auditability” Can Be “Free”8. Percent Complete Metric (PD/TP * 100)

SUBSIDIARY SPECIFICATION REQUIREMENTS VERIFICATION TRACE MATRIX SS ID SS Requirement VP ID# TP PD Permutations List % Complete 3 Functional Requirements N/A 7362 1106 15 % 3.1 Subsection 1 Title N/A 184 157 85 % 3.1.1 Category 1 Title N/A 33 12 36 % 3.1.1.1 The [subsystem] shall

(requirement 1) 3 0 Permutation 1,

Permutation 2, Permutation 3 (To do)

(Probably all in ?_3.3.2)

3.1.1.2 The [subsystem] shall (requirement 2)

?_3.3.1.1 /4/7/14

7 3 Permutation 1, Permutation 2, Permutation 3 (Done) Permutation 4, Permutation 5, Permutation 6, Permutation 7 (To do)

(4 VPs necessary for coverage)

... ... ... ... ...

17

Verification “Auditability” Can Be “Free”9. Related Metrics Expansion

• Because each Subsidiary RVTM is separate, spreadsheets can be used to automate the results of updates computation.

• RVTMs can include a new column for each Verification development milestone. (Such as Design Done, Dry Run, Verified, and Reported.)

18

Verification “Auditability” Can Be “Free”9. Related Metrics Expansion (Cont.)

• The verification procedure development milestone completion entry for each requirement is summed into its requirements specification subsection.

• Each subsection development milestone completion total is summed into its requirements specification section.

19

Verification “Auditability” Can Be “Free”9. Related Metrics Expansion (Cont.)

• Each verification development milestone completion section total is summed into its requirements specification document total.

• Using requirements permutations totals for each subsection, section, and specification, compute actual verification development milestones % complete values for reporting to self, supervisor, and project management.

20

Verification “Auditability” Can Be “Free”10. Requirements Management Tools

• For simpler projects, the RVTM and metrics computations are appropriately developed with an ordinary spreadsheet.

• For medium to large projects, requirements management tools such as DOORS support for producing the RVTMs as reports and permitting the recommended % complete metrics computation.

21

Verification “Auditability” Can Be “Free”11. Conclusions

• VASS can provide comprehensive rigor at less cost than previous practices would, so “auditability” can be considered “free”.

• Requirements Verification development metrics that reflect reality and help you know and report your part of the project status are superior to traditionally estimated percent complete values.

22

Optants Documented Decision Support Co.www.optants.com

• Defense/Medical Systems Engineering

• Requirements Management

• Decision Modeling (incorporating project technical/schedule/cost risk)

• Process Improvement

• Training/Consulting