Verification and Validation: Verification and Validation: A Quick IntroductionA Quick Introduction

1-2 Lectures1-2 Lectures

Learning ObjectivesLearning Objectives1.1. The student will be able to define both verification and The student will be able to define both verification and

validation, and to know the difference between them.validation, and to know the difference between them.

2.2. The student will be able to identify the appropriate activities The student will be able to identify the appropriate activities and their classification as either verification or validation.and their classification as either verification or validation.

3.3. The student will be able to distinguish between verification The student will be able to distinguish between verification and validation, given a set of activities.and validation, given a set of activities.

4.4. Given a set of software artifacts, the student will understand Given a set of software artifacts, the student will understand the appropriate verification or validation activity to that the appropriate verification or validation activity to that artifact.artifact.

5.5. Given a set of projects in different categories, the student will Given a set of projects in different categories, the student will be able to determine which category of projects would be able to determine which category of projects would require more V&V activity, and which would require less.require more V&V activity, and which would require less.

6.6. Given a set of defects and the phases of development in Given a set of defects and the phases of development in which they were found, the student will be able to estimate which they were found, the student will be able to estimate the relative cost of correcting errors early or late in the the relative cost of correcting errors early or late in the lifecycle.lifecycle.

Why Invest in Quality?Why Invest in Quality? Cost effectiveCost effective Provides competitive edgeProvides competitive edge Essential for business survivalEssential for business survival Essential for international marketingEssential for international marketing Helps to retain customers, and Helps to retain customers, and

increase profitsincrease profits Hallmark of world class businessHallmark of world class business

Ways to Improve QualityWays to Improve Quality Prevention of DefectsPrevention of Defects

• Process ImprovementProcess Improvement• Complexity ReductionComplexity Reduction• Risk ManagementRisk Management• Causal AnalysisCausal Analysis

Detection and Correction of DefectsDetection and Correction of Defects• VerificationVerification• ValidationValidation• ReworkRework• Causal AnalysisCausal Analysis

VerificationVerification

Verification – Are we building the product Verification – Are we building the product right?right?

Verification is any Verification is any checking process checking process conducted on software artifacts in an conducted on software artifacts in an attempt to determine if they work as attempt to determine if they work as specified by the specified by the designers of the systemdesigners of the system..

Includes reviews, inspections, Includes reviews, inspections, walkthroughs, unit testing and integration walkthroughs, unit testing and integration testing.testing.

ValidationValidation

Validation – Are we building the right Validation – Are we building the right product?product?

Validation is the process of evaluating Validation is the process of evaluating software artifacts during the software software artifacts during the software development process in an attempt to development process in an attempt to determine if the system works as required by determine if the system works as required by the the customerscustomers. Any evaluation activity that . Any evaluation activity that involves the customer can be used for involves the customer can be used for validation purposes.validation purposes.

Includes program reviews, system testing, Includes program reviews, system testing, customer acceptance testing.customer acceptance testing.

Verification vs. ValidationVerification vs. Validation VerificationVerification

• Main purpose is to detect defects in the Main purpose is to detect defects in the artifacts of the system under artifacts of the system under development.development.

ValidationValidation• Main purpose is to show that the system Main purpose is to show that the system

under development meets user needs, under development meets user needs, requirements, and expectations.requirements, and expectations.

Verification & Validation Verification & Validation TechniquesTechniques

Static MethodsStatic Methods• Techniques applied to artifacts without Techniques applied to artifacts without

execution.execution. Dynamic MethodsDynamic Methods

• Techniques applied to artifacts through Techniques applied to artifacts through execution.execution.

Mathematically Based MethodsMathematically Based Methods

Static: ReviewsStatic: Reviews WalkthroughsWalkthroughs

• CodeCode verificationverification• DocumentDocument

ConOps, SRSConOps, SRS validationvalidation STEP, SAD, SDDSTEP, SAD, SDD verificationverification

InspectionsInspections• CodeCode verificationverification• Document AuditsDocument Audits verificationverification

Program ReviewsProgram Reviews• Customer involvedCustomer involved validationvalidation• No customerNo customer verificationverification

Effectiveness of Static VerificationEffectiveness of Static VerificationEffectiveness of Static VerificationEffectiveness of Static Verification

More than 60% of program defects can be More than 60% of program defects can be detected by program inspections.detected by program inspections.

More than 90% of program defects may be More than 90% of program defects may be detectable using more rigorous detectable using more rigorous mathematical program verification.mathematical program verification.

The defect detection process is not The defect detection process is not confused by the existence of previous confused by the existence of previous defects.defects.

Dynamic: Testing (Verification)Dynamic: Testing (Verification)

Unit Test (Detailed Design): • Testing the individual software modules,

components, or units. Integration Testing (Architectural Design):

• After unit test, the system is put together in increments. Integration testing focuses on the interfaces between software components (OO thread-based, cluster-based testing)

System Testing (Requirements Spec): • One goal of system testing is to ensure that

the system functions as specified in the specification.

Dynamic: Testing (Validation)Dynamic: Testing (Validation)

System Testing (Requirements Spec):• Another goal of system testing is to ensure

that the system functions as the client expected in a controlled environment.

User Acceptance Test (ConOps):• A set of formal tests run for the client, and

specified by the client. When the system passes these tests, the software has been accepted by the client as meeting the requirements.

Verification and Validation in the Verification and Validation in the Development LifecycleDevelopment Lifecycle

CodeExecute Unit

Tests

White-Box &Black-Box

Testing

Black-BoxTesting

RequirementsAnalysis

ExecuteSystem Tests

Execute Integration

TestsDesign

Validate the System

Verify Design

Verify Implementation

Review

s

Mathematics-Based VerificationMathematics-Based Verification

Verification is based on mathematical Verification is based on mathematical arguments which demonstrate that a arguments which demonstrate that a program is consistent with its program is consistent with its specification.specification.

Programming language semantics Programming language semantics must be formally defined.must be formally defined.

The program must be formally The program must be formally specified.specified.

Why Inspection is important?Why Inspection is important? Relative cost to fix a defectRelative cost to fix a defect Phase in which found Cost ratio Phase in which found Cost ratio

(hours) (hours)– – Requirement Requirement 1 1– – DesignDesign 3-6 3-6– – CodingCoding 10 10– – TestingTesting 15-70 15-70– – OperationOperation 40-100040-1000

Data derived from Capers Jones.

Costs of Finding and Fixing LateCosts of Finding and Fixing Late

Delays in identifying and fixingdefects gets geometrically more

expensive as the lifecycle progresses!

$1

$10

$100

$1000

Cost

of

Fix

ing

a D

efe

ct

Defe

cts

Time in (Phase of) Development

Cost of QualityCost of Quality

Includes all costs of quality-related Includes all costs of quality-related activities.activities.

Quality costs =Quality costs =+Prevention costsPrevention costs+Detection and Appraisal costsDetection and Appraisal costs+Failure costsFailure costs

• Internal failure costsInternal failure costs• External failure costsExternal failure costs

Quality Cost ComponentsQuality Cost Components Direct CostDirect Cost

– – Reviews/inspectionsReviews/inspections

– – Unit testingUnit testing

– – System testingSystem testing

– – Acceptance testingAcceptance testing

– – Test planning and Test planning and

designdesign

– – Computer timeComputer time

– – Resources (terminals,Resources (terminals,

staffs, etc.)staffs, etc.)

Indirect CostIndirect Cost

– – ReworkRework

– – RecoveryRecovery

– – Corrective action costCorrective action cost

– – FailuresFailures

– – Analysis meetingAnalysis meeting

– – DebuggingDebugging

– – RetestingRetesting

– – Legal feesLegal fees

Verification or… Validation?Verification or… Validation?

ReviewsReviews EitherEither Unit testingUnit testing

VerificationVerification Integration TestingIntegration Testing

VerificationVerification System testingSystem testing ValidationValidation Acceptance testingAcceptance testing ValidationValidation

Balancing Trade-OffsBalancing Trade-Offs What is the real requirement?What is the real requirement? What are the expected benefits of What are the expected benefits of

V & V vs.V & V vs.– – cost of increasing V&V activitycost of increasing V&V activity– – additional time requiredadditional time required– – technical feasibilitytechnical feasibility– – risk of not finding defectsrisk of not finding defects

» delivery» delivery» operation» operation

““When is it good enough?”When is it good enough?”(Bach, 1997)

SummarySummary Verification and Validation are both Verification and Validation are both

important to the quality of software.important to the quality of software. Each are different:Each are different:

• Verification determines that what we are doing Verification determines that what we are doing is done correctly.is done correctly.

• Validation determines whether we are doing Validation determines whether we are doing the right thing.the right thing.

Sometimes, less quality is okay. Some Sometimes, less quality is okay. Some software systems require less V&V, as the software systems require less V&V, as the cost is not supportable. Some require cost is not supportable. Some require more no matter what it costs.more no matter what it costs.

The earlier defects are discovered, the less The earlier defects are discovered, the less it will cost to correct them.it will cost to correct them.

Backup SlidesBackup Slides

•Design • Coding

• System Requirements• Contractor Management

• Evaluate Requirements• Evaluate Design

• Monitor Tests• Integration

TestedProducts

Discrepancies

Subs

Project Manager

IV&V

• Testing

Independent Verification and Independent Verification and Validation (IV&V)Validation (IV&V)

Static and Dynamic Static and Dynamic Verification & ValidationVerification & Validation

Formalspecification

High-leveldesign

Requirementsspecification

Detaileddesign

Program

PrototypeDynamicvalidation

Staticverification

Sommerville, p. 421

Test Case Development in the Test Case Development in the V ModelV Model

CodeExecute Unit

Tests

Test cases designed and executed by developers

Test cases designedand executed by

independent testers

RequirementsAnalysis

ExecuteSystem Tests

Execute Integration

TestsDesign

Validate Requirements

Verify Design

Verify Implementation