Kerim Cakmak, Moshe Cohen -- Continuous Verification and Validation
-
Upload
anatoly-levenchuk -
Category
Technology
-
view
3.319 -
download
0
description
Transcript of Kerim Cakmak, Moshe Cohen -- Continuous Verification and Validation
© Copyright IBM Corporation 2013
Applying Continuous Verification and Validationto achieve the right Quality in Systems delivery
TechnicalRisk
Business Risk
Kerim Cakmak, PhD
IBM Rational Tiger Team CEE
Moshe S. Cohen
QM Market Manager and Evangelist
© 2012 IBM Corporation
Software and Systems Engineering | Rational
The V is dead! Long live the new V with continuous Verification and Validation across the full V!
Abstract:
While we know how to reduce technical risks in Systems delivery through Verification activities, our business risk is on a steep rise. This is largely due to the fact that "Validation" is performed at the end of the development lifecycle, rather than throughout, and it is done against customers requirements rather than end-users needs. In this session we will discuss some new best practices around Verification and Validation that were observed through recent successful and even more unsuccessful projects, and show you how to continuously verify and validate your designs throughout the whole lifecycle and not only at the tail end of it. Not only will this improve the quality of your products and systems, but it will also allow you to converge earlier on the right requirements, compressing the time to end users feedback, and improve your time to market predictability. While it doesn't change the V model, it is a new V.
2
© 2012 IBM Corporation
Software and Systems Engineering | Rational
Bottom Line is…
While delivering Quality is already a major challenge, the definition of Quality is changing…
– Low tolerance for bad quality (real and perceived)
– Expected short cycles with continuous improvements
To deliver Quality, we need a holistic approach to Quality– Continuously verify and validate our designs, throughout the whole lifecycle.
Business results:
– Deliver the right system
• Compressing time to customer’s feedback
– Better time to market predictability
• Earlier convergence on the right Requirements
– Lower cost of Quality
• Defect* Avoidance, Lower defect density, Higher defect removal efficiency
3
* Defect, as defined in ISO 9000 (2008): The non-fulfillment of intended usage requirements. The term Defect will be used here in the general sense, regardless whether it is a product defect, design defect, manufacturing defect, an error, a fault or a failure.
© 2012 IBM Corporation
Software and Systems Engineering | Rational
Agenda
4
© 2012 IBM Corporation
Software and Systems Engineering | Rational
5
In the News… Business Implications of a Failed missile testCould result in major financial consequences, unless Raytheon succeeds in demonstrating that it is not its fault.
© 2012 IBM Corporation
Software and Systems Engineering | Rational
Business Implications of RecallsNot only recalls are very expensive, but they also drive corporate behaviors.
Sony recalls: “Sony-related recalls are following one another, and that may ruin the company’s brand image,” said Keita Wakabayashi, an analyst at Mito Securities Co. with a “neutral plus” rating on the stock. Bloomberg, Oct 2011.
Toyota Prius gas pedal recall, 2010: ~$3B (CNNMoney.com)
– Repair cost (quality related expenses): $1.12B
– Legal cost (class-action lawsuit): $1.1B (WSJ, Dec 27, 2012)
– Image cost (sales losses): $770M-$880M
– “Toyota said Thursday that a software problem is causing problems with the brakes”…
CNNMoney, Chris Isidore, senior writer, Feb 4, 2010.
GM presses suppliers for future recall costs (Automotive News, Aug 5, 2013)
– GM has adopted a new purchasing contract that would allow it to recover the cost of safety recalls from its suppliers.
– The new GM contract has open-ended implications, stating that the supplier's components "will not, at any time (including after expiration or termination of this contract), pose an unreasonable risk to consumer or vehicle safety."
6
© 2012 IBM Corporation
Software and Systems Engineering | Rational
So… What is Quality? Exceeding end users needs, even when they keep changing.
No single definition, most definition are very detailed and verbose… but the common theme is most definitions is:
Yesterday: Meeting the Spec.
Today: Meeting or exceeding customers and end-users needs and expectations
– “End users needs” ≠ “Initial Requirements”
7
“The test of innovation lies not in its novelty, its scientific content or its cleverness.
It lies in success in the marketplace.”Peter Drucker.
(Business Management Philosopher)
© 2012 IBM Corporation
Software and Systems Engineering | Rational
Agenda
8
© 2012 IBM Corporation
Software and Systems Engineering | Rational
Continuous Verification & Validation ProcessEnd users needs, at least initially, are imprecise and incomplete – This is Normal!
Starts with early elicitation of end-users needs Helping the customer/end user to flash out their needs (Requirements Elicitation)
– Imagine you want to build a new home, do you know the exact size of the house?
ContinuouslyVerify…
ContinuouslyValidate…
Continuously validate with the customer, as the design is getting more concrete, that you are building the right system.
• As an architect and/or the builder validating assumptions they have made as your house is being built.
Continuously verify, throughout Development, that you are building the system right.
• As the builder checking with the architect.• As the city inspections, making sure the house is
being built in compliance with regulations.
Delivery
Initial Elicitationof end-users needs
Notice that this results in Initial understanding of the needs, still partial and incomplete
Continuously improving understanding of customer/end-users needs
© 2012 IBM Corporation
Software and Systems Engineering | Rational
What all the stakeholders need
What the customer needs
What the customer thinks they need
Different perspectives
What the customer says they want
Given thisHow do we get to this?
© 2012 IBM Corporation
Software and Systems Engineering | Rational
Verification vs. Validation ExamplesWhile Verification is “Internal”, Validation is “External”
Verification: The evaluation of whether or not a product, service or system complies with a regulation requirement, specification, or imposed condition. It is often an internal process. (IEEE)
• Code should never run thru a divide by zero as it may cause a crash or unexpected results.
• Did you implement the design pattern properly?
• Does this block diagram represent the best architecture for the system?
Validation: The assurance that a product, service, or system meets the needs of the customer and other identified stakeholders. It often involves acceptance and suitability with external customers. (IEEE)
• On a cycling computer… The customer asked for certain data fields to be displayed… but wouldn't the end user want to customize them?
• Is this {…} the right sequence of voice commands for making a phone call while driving your car?
• What's the right behavior of the wifi enabled home dialysis machine when the connection is intermittent?
11
© 2012 IBM Corporation
Software and Systems Engineering | Rational
Verification vs. ValidationWhile Verification is “internal”, Validation is “External”
Verification: Building the system right
– Testing against requirements
• “Internal” requirements
– Utilizes Test, Simulations, Analysis, Inspections and Demonstrations
– Assuring robust design and implementation
– Compliance with industry regulations
– We “know” how to do that… often done well!
Validation: Building the right system
– Validating against your customers needs
• As expected, imprecise and incomplete
• Often negotiated with several teams
• Often negotiated with people who may not be the end users
– Do we know how to do that? Do we do it?
12
SystemTesting
RequirementsAnalysis
Systems AnalysisSystems Design
Detailed Design&
Implementation
IntegrationTesting
CustomersNeeds
Note that the V does not represent a development process but rather a flow of information as we decompose and recompose a system.
© 2012 IBM Corporation
Software and Systems Engineering | Rational
Verification vs. ValidationWhile Verification detects defects, Validation is about avoiding them.
13
Left side of the V
Right side of the V
Verification • Early defect removal
• Expensive defect removal
Validation • Defect Avoidance
• Early detection of potentially business or mission failures
• Early Acceptance tests
• Final Acceptance test
SystemTesting
RequirementsAnalysis
Systems AnalysisSystems Design
Detailed Design&
Implementation
IntegrationTesting
CustomersNeeds
© 2012 IBM Corporation
Software and Systems Engineering | Rational
Verification vs. ValidationWhile Verification reduces Technical risk, Validation reduces Business or Mission risk
Verification: Reduces Technical risks
– It is a Testing process…
– We “know” how to do that… often done well!
Validation: Reduces Business risks
– Focuses on closing the loop with the end-user as often as possible
• Can’t just blame the Requirements being wrong…
• “Meeting the spec” is not a guarantee for business or mission success…
– It is NOT a Testing process… but rather an on-going Discovery process
• Important to “developers”
• Critical to End Users – Helping users figuring out their own Requirements!
– Do we know how to do that? Do we do it?
14
TechnicalRisk
Business Risk
© 2012 IBM Corporation
Software and Systems Engineering | Rational
Continuous Verification and Validation (V&V)Compressing time to customers feedback!
CustomersRequirements
Late Validation (Re-Evaluate)
SystemTesting
RequirementsAnalysis
IntegrationTesting
Detailed Design&
Implementation
Systems AnalysisSystems Design
SystemTesting
RequirementsAnalysis
IntegrationTesting
Detailed Design&
Implementation
Systems AnalysisSystems Design
Very Common:
SystemTesting
RequirementsAnalysis
IntegrationTesting
Detailed Design&
Implementation
Systems AnalysisSystems Design
Long/CostlyProcess
Example:
AA and United approached Boeing for a new flight entertainment system
Job has been sub’ed to Sony Trans Com., who designed & delivered a state-of-the-art system.
End result:
– Premium paying passengers had trouble operating the system
– Flight attendants had trouble supporting the passengers
End-End result: Flight attendants shutting down the system before take off, claiming malfunction.
© 2012 IBM Corporation
Software and Systems Engineering | Rational
Continuous Verification and Validation (V&V)Compressing time to customers feedback!
16
CustomersRequirements
Late Validation (Re-Evaluate)
SystemTesting
RequirementsAnalysis
IntegrationTesting
Detailed Design&
Implementation
Systems AnalysisSystems Design
SystemTesting
RequirementsAnalysis
IntegrationTesting
Detailed Design&
Implementation
Systems AnalysisSystems Design
Validation
CustomersNeeds
Verification
Validation
Verification
Validation
Verification
Validation
Verification
Validation
Verification
Continuous V&V per Feature:
Very Common:
SystemTesting
RequirementsAnalysis
IntegrationTesting
Detailed Design&
Implementation
Systems AnalysisSystems Design
Long/CostlyProcess
Use
Scenarios?Operational Scenarios? Prototypes? HIL/SIL? Final?
Compressing time to customers feedback!
© 2012 IBM Corporation
Software and Systems Engineering | Rational
Agenda
19
© 2012 IBM Corporation
Software and Systems Engineering | Rational
Collaboration through ReviewsTesting for early feedback from Internal and External stakeholders
Review process where peers and other stakeholders collaborate in order to identify potential errors and deviations from the spec.
Objectives include:
– Detect wrong understanding, bad assumptions and defects close to when introduced (Functional review)
– Identify potential improvements and/or risks (Functional review)
– Verify compliance with regulations and conformance to standards (Quality review)
Audience:
– External Reviews: With customers and/or end-users,often focused on Validation
– Internal Reviews: With internal stakeholders (peers, management), often focused on Verification
Formal inspections (with defined roles & process) to ad-hoc team reviews and walkthroughs.
FDA Example: On Going Reviews / Late Validation
20
© 2012 IBM Corporation
Software and Systems Engineering | Rational
Reviews are collaboration pointsInclude the voice of the customer for on-going validation
Applies to the full development life cycle and to all artifacts
Reviews are short, and per feature (best practice: focus on end user feature)
Examples:
21
Review Objective Internal/External
Requirements elicitation Identifying Use Scenarios
External Review What are the main functions/activities?
Requirements Review Defining Operational Scenarios
External Review How does the system operate?
Architecture Design Review
Is this the “best” architecture?
Internal Review Design documents
Initial Prototype Review Early feedback based on feature execution
External review Low Fidelity prototype, early feedback.
Advanced Prototype Review
Production code in HIL/SIL environment
External Review Hi Fidelity prototype, still early feedback.
Test Plan Review Agreement across all stakeholders
Internal and/or External Review
Sunny & Rainy days scenarios
Test Execution Review Discuss Quality status and next actions
Internal and/or External Review
Focused on high level operational scenarios
© 2012 IBM Corporation
Software and Systems Engineering | Rational
Reviews ImpactReviews are ~2x more efficient in removing defects than Testing(Not including Defect Avoidance)
22
Reviews
Testing
Source: Capers Jones, 2012
© 2012 IBM Corporation
Software and Systems Engineering | Rational
23
Reviews ImpactReviews are ~2x more efficient in removing defects than Testing(Not including Defect Avoidance)
Reviews
Testing
© 2012 IBM Corporation
Software and Systems Engineering | Rational
End Users Operational ScenariosUse them as top level test cases to continuously drive all V&V activities
Operational Scenarios
– Quasi formal but yet intuitive
– Can be extended with sketches and Interaction Diagrams
Operational Scenarios are a Testable versionof the requirements!
– Can be converted into System level test cases
System level test cases:
– Should be Validated (reviewed and approved) with the customer and/or end-user
– To be used as a Golden Reference model throughout design and implementation for Verification purposes.
Although diagrams shown here are SysML diagrams, concept applies to any system.
24
© 2012 IBM Corporation
Software and Systems Engineering | Rational
25
Validate your understanding of the needs by reviewing the expected scenarios with the stakeholders and getting them approved.
A Modeling environment that support Use Case analysis
A Quality Management process that support the conversion of Operational scenarios into test cases
– Test cases for either Manual Testing or Automated Testing
– May require internal review and approval
Traceability: Needs Requirements Operational Scenarios Test Cases
Virtual
Prototype
RealSystemLevel Test
Cases
OperationalScenarios
To leverage System Level Test Cases, you need…Use Operational scenarios as Tests throughout the whole development Lifecycle
© 2012 IBM Corporation
Software and Systems Engineering | Rational
Example User Scenario
To have sailedand
survived
Overall
goal
Sub-goals
Ready to sail
Sailed
Returnedhome
Boatloaded
Boatunloaded
Boatrigged
sequential
Boat lifted
Boat on car
Mast rigged
Center-plate rigged
Rudder riggedparallel
Maneuvered
periodic
Ashore
Gybed
Tacked
Cruisedalternate
alternateRecover
from Capsize
Sailednormally
exception
Righted
Coast guard
contacted
© 2012 IBM Corporation
Software and Systems Engineering | Rational
Requirements from Scenarios
Goal hierarchy
Stakeholder requirements
Two people shall be able to lift the boat onto the
roof of the average saloon car.
The sailor shall be able to perform a gybe.
The sailor shall be able to contact the coastguard
when the boat is capsized.
traceability
To have sailedand
survived
Overallgoal
Sub-goals
Ready to sail
Sailed
Returnedhome
Boatloaded
Boatunloaded
Boatrigged
sequential
Boat lifted
Boat on car
Mast rigged
Center-plate rigged
Rudder riggedparallel
Maneuvered
periodic
Ashore
Gybed
Tacked
Cruisedalternate
alternateRecover
from Capsize
Sailednormally
exception
Righted
Coast guard
contacted
© 2012 IBM Corporation
Software and Systems Engineering | Rational
Virtual ModelsEnabling early Verification and Validation… before building anything real.
Virtual models that can be used for early analysis and trade studies (functionality, behavior, architecture, structure, performance, reliability, safety, etc.)
– Models abstract mechanics, electronics and/or software entities, operating independently or integrated
– Different domains:
• Mechanical/Control models (such as Mathworks, NI)
• Graphical languages (such as SysML)
• Textual language (such as SystemC for HW design)
Can be used for
– The system being designed AND its TestBench (Plant models)
Use System level test cases to drive analysis
– Execution, Simulation or Prototyping
Various configurations driving on going system analysis:• Eg. 1: Virtual sensors driving virtual HW+SW models
• Eg. 2: Mechanical prototype driving virtual HW+SW models
• Eg 3.: Mechanical prototype driving virtual HW+production SW
28
© 2012 IBM Corporation
Software and Systems Engineering | Rational
Models are Executable
– You can ask Questions and get consistent answers!
Models can be Analyzed
– Tradeoff studies
– Performance
– Behaviors
– Etc.
Leverage correct-by-construction Synthesis capabilities to guide the design and implementation process
– Eg. 3D printers, C/C++ for SW, SystemC for HW
– Enabling Co-Execution for V&V purposes
29
Source: Capers Jones, 2012
Model Based:Low Defect Potential,
Best Removal Efficiency
Modeling has low potential of introducing defectsModels are best in removing defects
© 2012 IBM Corporation
Software and Systems Engineering | Rational
Effective and Efficient TestingSingle source of truth for all Quality related artifacts and processes
Central Quality Management Hub, acting as a single source of truth, enabling:
– Collaborative Test Planning
• Development and approval for Verification and Validation test plans– As well as specific plans as needed, such as Safety.
• Traceability back to Requirements (why we need which test?)
– Managing test execution (manual and automated) according to Test Plan
• Who is running which test, when, why, pre-conditions, test lab/equipment reservation etc.
– Integrated Defect Management
• What went wrong, how to reproduce, defect resolution, reporting, etc.
– Traceability across the full lifecycle
• User Needs Requirements Test cases Test Results Defects Resolutions
Other key capabilities
– Collaboration around Rainy days scenarios
– Test assets ReUsability
– Reconciliation process between Requirements and Test Plans/Test Cases
– Actionable Analytics to drive process improvements
30
© 2012 IBM Corporation
Software and Systems Engineering | Rational
Overall Impact on Testing EffectivenessCollaborative Test Planning, Traceability, Test Execution and Analytics
Impact of Quality Management on Process Efficiency
15%
30%
60%
75%
85%
32%
58%
76%
85% 87%
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
1 2 3 4 5
CMMI Levels W/O QM W QM
20% 40% 40% 40% 10%QM Impact:
W/O QM: Percentage of defects detected during Requirements review, design reviews, unit testing and Functional testing – Current practice
W QM: Percentage of defects detected thru Requirements review, design reviews, unit testing and Functional testing – Improved practice
Source: Analysis of over 850 GBS projects and Industry data
Detect 3, release 7
Detect 3, release 2
31
© 2012 IBM Corporation
Software and Systems Engineering | Rational
Agenda
32
© 2012 IBM Corporation
Software and Systems Engineering | Rational
Key Take-Aways
Verification in Inbound; Validation is Outbound. Both needed.
Verification is a Testing process. Validation is a Discovery process, helping the customer refine their understanding of their own needs.
Compressing time to feedback allows for change… and therefore reduces surprises.
Reduce business risk with Early and Frequent feedback from all stakeholders
– Use Scenarios and Operational Scenarios
– Validation Test Plan
– Low Fidelity and Hi Fidelity prototypes
Operational scenarios = Testable Requirements = Golden reference
– Use them for your Testing throughout the whole development process and not only at the end!
Question:
Unit Testing and Integration Testing are both very important.
Which is more important?
Which one would you do first?
34
© 2012 IBM Corporation
Software and Systems Engineering | Rational
Continuous V&V – Reducing Technical and Business Risks
35
Quality can be delivered by compressing the time to customer feedback with
continuous Verification and Validation.
Both Technical and Business risks can be lowered
Business Implications:
– Lower cost of Quality
• Defect Avoidance, Lower defect density, higher defect removal efficiency
– Deliver the right system
• Compressing time to customer feedback
– Better time to market predictability
• Earlier convergence on the right Requirements
TechnicalRisk
Business Risk
© 2012 IBM Corporation
Software and Systems Engineering | Rational
3636
© 2012 IBM Corporation
Software and Systems Engineering | Rational
© Copyright IBM Corporation 2013. All rights reserved. The information contained in these materials is provided for informational purposes only, and is provided AS IS without warranty of any kind, express or implied. IBM shall not be responsible for any damages arising out of the use of, or
otherwise related to, these materials. Nothing contained in these materials is intended to, nor shall have the effect of, creating any warranties or representations from IBM or its
suppliers or licensors, or altering the terms and conditions of the applicable license agreement governing the use of IBM
software. References in these materials to IBM products, programs, or services do not imply that they will be available in all countries in which IBM operates. Product release dates and/or capabilities referenced in these materials may change
at any time at IBM’s sole discretion based on market opportunities or other factors, and are not intended to be a commitment to future product or feature availability in any way. IBM, the IBM logo, Rational, and other IBM products and services are trademarks of the International Business
Machines Corporation, in the United States, other countries or both. Other company, product, or service names may be
trademarks or service marks of others.
www.ibm.com/software/rational
37