Pittsburgh, PA 15213-3890 The ROI of CMMI: Using Process … · 2017. 5. 19. · page 1 Pittsburgh,...
Transcript of Pittsburgh, PA 15213-3890 The ROI of CMMI: Using Process … · 2017. 5. 19. · page 1 Pittsburgh,...
![Page 1: Pittsburgh, PA 15213-3890 The ROI of CMMI: Using Process … · 2017. 5. 19. · page 1 Pittsburgh, PA 15213-3890 Portland State University The ROI of CMMI: Using Process Simulation](https://reader034.fdocuments.net/reader034/viewer/2022051605/6013924b9aea010f0f6554eb/html5/thumbnails/1.jpg)
page 1
Pittsburgh, PA 15213-3890
Portland StateUniversity
The ROI of CMMI:Using Process Simulation to Support Better Management Decisions
David M. Raffo, Ph.D.Visiting Scientist, Software Engineering Institute Associate Professor, Portland State University
Sponsored by the U.S. Department of Defense© 2005 by Carnegie Mellon University
![Page 2: Pittsburgh, PA 15213-3890 The ROI of CMMI: Using Process … · 2017. 5. 19. · page 1 Pittsburgh, PA 15213-3890 Portland State University The ROI of CMMI: Using Process Simulation](https://reader034.fdocuments.net/reader034/viewer/2022051605/6013924b9aea010f0f6554eb/html5/thumbnails/2.jpg)
page 2
Portland StateUniversity
AcknowledgementsPortions of this tutorial are reprinted and presented with the permission of Portland State University.
![Page 3: Pittsburgh, PA 15213-3890 The ROI of CMMI: Using Process … · 2017. 5. 19. · page 1 Pittsburgh, PA 15213-3890 Portland State University The ROI of CMMI: Using Process Simulation](https://reader034.fdocuments.net/reader034/viewer/2022051605/6013924b9aea010f0f6554eb/html5/thumbnails/3.jpg)
page 3
Pittsburgh, PA 15213-3890
Portland StateUniversity
Introductions and Logistics
![Page 4: Pittsburgh, PA 15213-3890 The ROI of CMMI: Using Process … · 2017. 5. 19. · page 1 Pittsburgh, PA 15213-3890 Portland State University The ROI of CMMI: Using Process Simulation](https://reader034.fdocuments.net/reader034/viewer/2022051605/6013924b9aea010f0f6554eb/html5/thumbnails/4.jpg)
page 4
Portland StateUniversity
IntroductionsWorkshop leader introductions
Participant introductions• Name• Position• Expectations
- What do you want to get out of the workshop?- Do your expectations match the workshop agenda?
![Page 5: Pittsburgh, PA 15213-3890 The ROI of CMMI: Using Process … · 2017. 5. 19. · page 1 Pittsburgh, PA 15213-3890 Portland State University The ROI of CMMI: Using Process Simulation](https://reader034.fdocuments.net/reader034/viewer/2022051605/6013924b9aea010f0f6554eb/html5/thumbnails/5.jpg)
page 5
Portland StateUniversity
LogisticsWorkshop time/duration
Rest Rooms
Breaks
Smoking Rules
Phones
Messages
![Page 6: Pittsburgh, PA 15213-3890 The ROI of CMMI: Using Process … · 2017. 5. 19. · page 1 Pittsburgh, PA 15213-3890 Portland State University The ROI of CMMI: Using Process Simulation](https://reader034.fdocuments.net/reader034/viewer/2022051605/6013924b9aea010f0f6554eb/html5/thumbnails/6.jpg)
page 6
Portland StateUniversity
Workshop ApproachLecture/presentation
Examples
Ask questions
Participate!!!
![Page 7: Pittsburgh, PA 15213-3890 The ROI of CMMI: Using Process … · 2017. 5. 19. · page 1 Pittsburgh, PA 15213-3890 Portland State University The ROI of CMMI: Using Process Simulation](https://reader034.fdocuments.net/reader034/viewer/2022051605/6013924b9aea010f0f6554eb/html5/thumbnails/7.jpg)
page 7
Portland StateUniversity
Audience• Executive/leaders of organizations seeking to understand
- The costs/benefits of CMMI-based process improvement- How to quantify them- How simulation can help them achieve higher CMMI levels
• Executives/leaders seeking to benchmark their processes and performance with industry
• Process improvement/EPG personnel seeking ways to communicate more effectively to senior management about the costs/benefits of CMMI-based process improvement
• Personnel seeking to transition to the CMMI, or implement higher-maturity process areas
• Personnel working to define process and estimate performance based upon quantitative measurements.
![Page 8: Pittsburgh, PA 15213-3890 The ROI of CMMI: Using Process … · 2017. 5. 19. · page 1 Pittsburgh, PA 15213-3890 Portland State University The ROI of CMMI: Using Process Simulation](https://reader034.fdocuments.net/reader034/viewer/2022051605/6013924b9aea010f0f6554eb/html5/thumbnails/8.jpg)
page 8
Portland StateUniversity
OverviewProcess simulation is a high leverage way to determine which process improvement opportunities are likely to have the best outcome
Goals of the tutorial:• Familiarize participants with Process Simulation –
What, Why, How• Show participants how to utilize simulation results
to support process improvement decisions
This tutorial will focus on one simulation method – the Process Tradeoff Analysis Method (PTAM) and will briefly touch on others
![Page 9: Pittsburgh, PA 15213-3890 The ROI of CMMI: Using Process … · 2017. 5. 19. · page 1 Pittsburgh, PA 15213-3890 Portland State University The ROI of CMMI: Using Process Simulation](https://reader034.fdocuments.net/reader034/viewer/2022051605/6013924b9aea010f0f6554eb/html5/thumbnails/9.jpg)
page 9
Portland StateUniversity
OverviewThe tutorial is not intended to be comprehensive, some topics are presented at a high-level only
No knowledge of simulation or finance is assumed
![Page 10: Pittsburgh, PA 15213-3890 The ROI of CMMI: Using Process … · 2017. 5. 19. · page 1 Pittsburgh, PA 15213-3890 Portland State University The ROI of CMMI: Using Process Simulation](https://reader034.fdocuments.net/reader034/viewer/2022051605/6013924b9aea010f0f6554eb/html5/thumbnails/10.jpg)
page 10
Portland StateUniversity
Agenda1. Introduction: What is Process Simulation?
2. Motivation: Why do Process Simulation?
3. Overview of Process Simulation Alternatives
4. How do we build process simulation models?
5. Process Tradeoff Analysis Method (PTAM)
6. Examples of Process Simulation Applications in Industry and Government.
7. Wrap-Up/ Conclusions
![Page 11: Pittsburgh, PA 15213-3890 The ROI of CMMI: Using Process … · 2017. 5. 19. · page 1 Pittsburgh, PA 15213-3890 Portland State University The ROI of CMMI: Using Process Simulation](https://reader034.fdocuments.net/reader034/viewer/2022051605/6013924b9aea010f0f6554eb/html5/thumbnails/11.jpg)
page 11
Pittsburgh, PA 15213-3890
Portland StateUniversity
1 – Introduction: What is Process Simulation?
![Page 12: Pittsburgh, PA 15213-3890 The ROI of CMMI: Using Process … · 2017. 5. 19. · page 1 Pittsburgh, PA 15213-3890 Portland State University The ROI of CMMI: Using Process Simulation](https://reader034.fdocuments.net/reader034/viewer/2022051605/6013924b9aea010f0f6554eb/html5/thumbnails/12.jpg)
page 12
Portland StateUniversity
What Is a Simulation Model?• A simulation model is a computerized model (not a
maturity model) designed to display significant features of the dynamic system it represents.
• Simulations are generally employed when- behavior over time is of particular interest or
significance, and- the economics or logistics of manipulating the
system being modeled are prohibitive
• Common purposes of simulation models are:- to provide a basis for experimentation, - to predict behavior, - to answer “what if” questions, - to teach about the system being modeled.
![Page 13: Pittsburgh, PA 15213-3890 The ROI of CMMI: Using Process … · 2017. 5. 19. · page 1 Pittsburgh, PA 15213-3890 Portland State University The ROI of CMMI: Using Process Simulation](https://reader034.fdocuments.net/reader034/viewer/2022051605/6013924b9aea010f0f6554eb/html5/thumbnails/13.jpg)
page 13
Portland StateUniversity
Process Simulation Models• Process simulation models focus on the
dynamics of software and systemsdevelopment, maintenance and acquisition.
• They represent the process- as currently implemented (as-is, as-
practiced, as-documented), or- as planned for future implementation (to-
be)• The models represent only selected relevant
aspects of a defined process.
![Page 14: Pittsburgh, PA 15213-3890 The ROI of CMMI: Using Process … · 2017. 5. 19. · page 1 Pittsburgh, PA 15213-3890 Portland State University The ROI of CMMI: Using Process Simulation](https://reader034.fdocuments.net/reader034/viewer/2022051605/6013924b9aea010f0f6554eb/html5/thumbnails/14.jpg)
page 14
Portland StateUniversity
Simulation Features• Use Graphical interfaces• Utilizes actual data/ metrics • Predict performance • Supports “What if” Analyses• Support business case analyses • Reduces risk
![Page 15: Pittsburgh, PA 15213-3890 The ROI of CMMI: Using Process … · 2017. 5. 19. · page 1 Pittsburgh, PA 15213-3890 Portland State University The ROI of CMMI: Using Process Simulation](https://reader034.fdocuments.net/reader034/viewer/2022051605/6013924b9aea010f0f6554eb/html5/thumbnails/15.jpg)
page 15
Portland StateUniversity
Company StrategyCompetitive Advantage
Customer Value
Improving OperationsIndustry Standards
CMMI, Six Sigma, ISO
Process SimulationEvaluate Impact on
Process Performance
Performance MeasuresCost, Quality, Schedule
Financial Benefits - NPV, ROI
Many choices.Which one(s) to choose?
Which change will provide the greatest improvement?
Need to focus effortsto be successful.
What is the financial impact?
Set of Potential Process Changes
![Page 16: Pittsburgh, PA 15213-3890 The ROI of CMMI: Using Process … · 2017. 5. 19. · page 1 Pittsburgh, PA 15213-3890 Portland State University The ROI of CMMI: Using Process Simulation](https://reader034.fdocuments.net/reader034/viewer/2022051605/6013924b9aea010f0f6554eb/html5/thumbnails/16.jpg)
page 16
Pittsburgh, PA 15213-3890
Portland StateUniversity
2 – Motivation: Why Do Process Simulation?
![Page 17: Pittsburgh, PA 15213-3890 The ROI of CMMI: Using Process … · 2017. 5. 19. · page 1 Pittsburgh, PA 15213-3890 Portland State University The ROI of CMMI: Using Process Simulation](https://reader034.fdocuments.net/reader034/viewer/2022051605/6013924b9aea010f0f6554eb/html5/thumbnails/17.jpg)
page 17
Portland StateUniversity
Benefits of Process Simulation
Option
Total Effort
(PM) Dev Eff + Dev
Rwk
Rework Effort Devel
Defects (PM)
Project Duration (Calendar Months)
Projected Cost or
Revenue delta due
to Duration Change
Total Injected Defects
Corrected Defects
Escapted Defects
Rework Effort for
Field Defects
(PM)
Implementation
Costs ($) NPV ROI0
200 90 18 $0.00 1150 990 160 40 $0.00 n.a. n.a.
1 190 75 17.5 $0.00 1150 1020 130 30 $100,000 $165,145 15%
2 185 75 17 $ 100,000 1150 1050 100 20 $120,000 $185,231 29%
3 175 65 16 $ 300,000 1150 1090 60 10 $ 80,000 $289,674 88%
4 230 130 22 $(400,000) 1150 900 250 80 $0.00 -$378,043 -129%
5
Add QuARS Tool
Eliminate
Additional Process
Project
Base Case
Implement QFD
Implement VOC
![Page 18: Pittsburgh, PA 15213-3890 The ROI of CMMI: Using Process … · 2017. 5. 19. · page 1 Pittsburgh, PA 15213-3890 Portland State University The ROI of CMMI: Using Process Simulation](https://reader034.fdocuments.net/reader034/viewer/2022051605/6013924b9aea010f0f6554eb/html5/thumbnails/18.jpg)
page 18
Portland StateUniversity
Benefits of Process Simulation• Decision Support and Tradeoff Analysis• Sensitivity Analysis – “What if”• Supports Industry Certification and process
improvement programs including CMMI, Six Sigma, and others
• Benchmarking• Design and Define Processes/Metrics• Bring Lessons Learned Repositories Alive• Can save cost, effort, and expertise• Can be used to address project manager
concerns such as….
![Page 19: Pittsburgh, PA 15213-3890 The ROI of CMMI: Using Process … · 2017. 5. 19. · page 1 Pittsburgh, PA 15213-3890 Portland State University The ROI of CMMI: Using Process Simulation](https://reader034.fdocuments.net/reader034/viewer/2022051605/6013924b9aea010f0f6554eb/html5/thumbnails/19.jpg)
page 19
Portland StateUniversity
Software Project Manager Concerns• What development phases are essential? • Which phases could be skipped or
minimized to shorten cycle time and reduce costs without sacrificing quality?
• Are inspections worthwhile?• What is the value of applying automated
tools to support development activities? • How do we predict the benefit associated
with implementing a process change?• How do we prioritize process changes?• How to achieve higher levels of the
CMMI?• What is the level of Risk associated with a
change?
![Page 20: Pittsburgh, PA 15213-3890 The ROI of CMMI: Using Process … · 2017. 5. 19. · page 1 Pittsburgh, PA 15213-3890 Portland State University The ROI of CMMI: Using Process Simulation](https://reader034.fdocuments.net/reader034/viewer/2022051605/6013924b9aea010f0f6554eb/html5/thumbnails/20.jpg)
page 20
Pittsburgh, PA 15213-3890
Portland StateUniversity
3 – Overview of Alternative Process Simulation Approaches
![Page 21: Pittsburgh, PA 15213-3890 The ROI of CMMI: Using Process … · 2017. 5. 19. · page 1 Pittsburgh, PA 15213-3890 Portland State University The ROI of CMMI: Using Process Simulation](https://reader034.fdocuments.net/reader034/viewer/2022051605/6013924b9aea010f0f6554eb/html5/thumbnails/21.jpg)
page 21
Portland StateUniversity
Alternative Process Simulation ApproachesModeling Paradigms• Knowledge-Based
Systems• Agent Based• State-Based • Discrete Event• System Dynamics• Hybrid
Research Outlets• Software Process:
Improvement and Practice
• Journal of Systems and Software
• Tools– Arena– ProModel– Extend– Stella– VenSim– Research tools
• Conferences– Winter Simulation
Conference– ProSim– SEPG– SSTC
![Page 22: Pittsburgh, PA 15213-3890 The ROI of CMMI: Using Process … · 2017. 5. 19. · page 1 Pittsburgh, PA 15213-3890 Portland State University The ROI of CMMI: Using Process Simulation](https://reader034.fdocuments.net/reader034/viewer/2022051605/6013924b9aea010f0f6554eb/html5/thumbnails/22.jpg)
page 22
Portland StateUniversity
Alternative Process Simulation ApproachesKnowledge Based Systems• Person-in-the loop• Fine level of granularity• Supports process enactment
Agent Based Systems• Fine level of granularity• Supports detailed work interactions
State Based Systems• Captures flow of control (work activities,
parallelism) well• Multi-view graphical representations• Difficult to capture task, work package and
resource details
![Page 23: Pittsburgh, PA 15213-3890 The ROI of CMMI: Using Process … · 2017. 5. 19. · page 1 Pittsburgh, PA 15213-3890 Portland State University The ROI of CMMI: Using Process Simulation](https://reader034.fdocuments.net/reader034/viewer/2022051605/6013924b9aea010f0f6554eb/html5/thumbnails/23.jpg)
page 23
Portland StateUniversity
Alternative Process Simulation ApproachesDiscrete Event Simulation• Able to represent richness of processes, work packages
and resources• Good for modeling quantitative process performance• Good tool support
System Dynamics• Captures feedback well• Often used for high level qualitative issues
Hybrid• Captures best aspects of Discrete Event and System
Dynamics• Models are complex• Being used to predict performance of multi-site
development
![Page 24: Pittsburgh, PA 15213-3890 The ROI of CMMI: Using Process … · 2017. 5. 19. · page 1 Pittsburgh, PA 15213-3890 Portland State University The ROI of CMMI: Using Process Simulation](https://reader034.fdocuments.net/reader034/viewer/2022051605/6013924b9aea010f0f6554eb/html5/thumbnails/24.jpg)
page 24
Portland StateUniversity
Common Applications of Each Approach
STRAT PLAN MGMT IMPR UNDR TRAIN
KBS X X
Agent Based X X
State-Based X X X X
Discrete Event x X X X X X
System Dynamic X x x X X
Hybrid X X X X X X
![Page 25: Pittsburgh, PA 15213-3890 The ROI of CMMI: Using Process … · 2017. 5. 19. · page 1 Pittsburgh, PA 15213-3890 Portland State University The ROI of CMMI: Using Process Simulation](https://reader034.fdocuments.net/reader034/viewer/2022051605/6013924b9aea010f0f6554eb/html5/thumbnails/25.jpg)
page 25
Pittsburgh, PA 15213-3890
Portland StateUniversity
4 –How to Build Process Simulation Models
![Page 26: Pittsburgh, PA 15213-3890 The ROI of CMMI: Using Process … · 2017. 5. 19. · page 1 Pittsburgh, PA 15213-3890 Portland State University The ROI of CMMI: Using Process Simulation](https://reader034.fdocuments.net/reader034/viewer/2022051605/6013924b9aea010f0f6554eb/html5/thumbnails/26.jpg)
page 26
Portland StateUniversity
How it works
Process PerformanceCost, Quality, Schedule
Code DevCodeInsp
Unit Test Functional
Test
System Test
FieldSupportandMain-tenance
H Lev DesignHLD Insp
L Lev Design LLDInsp
Func SpecFSInsp
Project is Approved Development
CompleteUnit Test Complete
Release to Customers
InspUTPlan
Follow UT Pln
ProposedProcessChange
CreateUTPlan
Project DataProcess and
Product
BetterProcess
DecisionsSoftware Development Process
SW Process Simulation Model
Code DevCodeInsp
Unit Test Functional
Test
System Test
FieldSupportandMain-tenance
H Lev DesignHLD Insp
L Lev Design LLDInsp
Func SpecFSInsp
Project is Approved Development
CompleteUnit Test Complete
Release to Customers
InspUTPlan
Follow UT Pln
ProposedProcessChange
CreateUTPlanModel
Parameters
![Page 27: Pittsburgh, PA 15213-3890 The ROI of CMMI: Using Process … · 2017. 5. 19. · page 1 Pittsburgh, PA 15213-3890 Portland State University The ROI of CMMI: Using Process Simulation](https://reader034.fdocuments.net/reader034/viewer/2022051605/6013924b9aea010f0f6554eb/html5/thumbnails/27.jpg)
page 27
Portland StateUniversity
Process Tradeoff Analysis Method (PTAM)• Based on extensive research into Software Process
Modeling conducted in academia, SEI and industry.
• Graphical user interface and models software processes
• Integrates SEI methods to define processes and supports CMMI PAs
• Integrates metrics related to cost, quality, and schedule into understandable project performance picture.
• Predicts project-level impacts of process improvements in terms of cost, quality and cycle time
• Support business case analysis of process decisions -ROI, NPV and quantitatively assessing risk.
![Page 28: Pittsburgh, PA 15213-3890 The ROI of CMMI: Using Process … · 2017. 5. 19. · page 1 Pittsburgh, PA 15213-3890 Portland State University The ROI of CMMI: Using Process Simulation](https://reader034.fdocuments.net/reader034/viewer/2022051605/6013924b9aea010f0f6554eb/html5/thumbnails/28.jpg)
page 28
Portland StateUniversity
Process Tradeoff Analysis Method (PTAM)
• Reduces risk associated with process changes by predicting the probability of improvement
• Saves time, effort and expertise over other methods
![Page 29: Pittsburgh, PA 15213-3890 The ROI of CMMI: Using Process … · 2017. 5. 19. · page 1 Pittsburgh, PA 15213-3890 Portland State University The ROI of CMMI: Using Process Simulation](https://reader034.fdocuments.net/reader034/viewer/2022051605/6013924b9aea010f0f6554eb/html5/thumbnails/29.jpg)
page 29
Pittsburgh, PA 15213-3890
Portland StateUniversity
5 – The Process Tradeoff Analysis Method (PTAM)
![Page 30: Pittsburgh, PA 15213-3890 The ROI of CMMI: Using Process … · 2017. 5. 19. · page 1 Pittsburgh, PA 15213-3890 Portland State University The ROI of CMMI: Using Process Simulation](https://reader034.fdocuments.net/reader034/viewer/2022051605/6013924b9aea010f0f6554eb/html5/thumbnails/30.jpg)
page 30
Portland StateUniversity
Process Tradeoff Analysis (PTA) MethodCompany Strategy
Competitive AdvantageCustomer Value
Improving OperationsIndustry Standards
CMM, ISO 9000
Process SimulationEvaluate Impact on
Process Performance
Performance MeasuresCost, Quality, Schedule
Financial Benefits - NPV, ROI
Many choices.Which one(s) to choose?
Which change will provide the greatest improvement?
Need to focus effortsto be successful.
What is the financial impact?
Set of Potential Process Changes
![Page 31: Pittsburgh, PA 15213-3890 The ROI of CMMI: Using Process … · 2017. 5. 19. · page 1 Pittsburgh, PA 15213-3890 Portland State University The ROI of CMMI: Using Process Simulation](https://reader034.fdocuments.net/reader034/viewer/2022051605/6013924b9aea010f0f6554eb/html5/thumbnails/31.jpg)
page 31
Portland StateUniversity
Overview of PTAMSet-up Phase• Set the Goal of the Modeling Effort• Specify Questions for the Model to Address• Define Process Performance Measures • Identify Input Parameters
Gather InformationModeling PhaseAnalysis Phase
![Page 32: Pittsburgh, PA 15213-3890 The ROI of CMMI: Using Process … · 2017. 5. 19. · page 1 Pittsburgh, PA 15213-3890 Portland State University The ROI of CMMI: Using Process Simulation](https://reader034.fdocuments.net/reader034/viewer/2022051605/6013924b9aea010f0f6554eb/html5/thumbnails/32.jpg)
page 32
Portland StateUniversity
Set-up Phase GoalMajor Objective(s)
for model
Performance MeasuresMetrics/Model Outputs
designed to address key questions
Input DataData and information needed
to calibrate and estimate performance measures
QuestionsDefine key questions
to address
What information should we collect?
What decision(s) am I trying to make?
What questions does management have?
What do I need to know to answer the questions?
![Page 33: Pittsburgh, PA 15213-3890 The ROI of CMMI: Using Process … · 2017. 5. 19. · page 1 Pittsburgh, PA 15213-3890 Portland State University The ROI of CMMI: Using Process Simulation](https://reader034.fdocuments.net/reader034/viewer/2022051605/6013924b9aea010f0f6554eb/html5/thumbnails/33.jpg)
page 33
Portland StateUniversity
Overview of PTAM• Set-up Phase
- Set the Goal of the Modeling Effort- Specify Questions for the Model to Address- Define Process Performance Measures - Identify Input Data
• Gather Information• Modeling Phase• Analysis Phase
![Page 34: Pittsburgh, PA 15213-3890 The ROI of CMMI: Using Process … · 2017. 5. 19. · page 1 Pittsburgh, PA 15213-3890 Portland State University The ROI of CMMI: Using Process Simulation](https://reader034.fdocuments.net/reader034/viewer/2022051605/6013924b9aea010f0f6554eb/html5/thumbnails/34.jpg)
page 34
Portland StateUniversity
Why Simulate?• There are a variety of reasons / purposes for
undertaking process simulation.
• CMMI-Based Process Improvement- Strategic management- Planning- Control and operational management- Technology adoption- Understanding- Training and learning
![Page 35: Pittsburgh, PA 15213-3890 The ROI of CMMI: Using Process … · 2017. 5. 19. · page 1 Pittsburgh, PA 15213-3890 Portland State University The ROI of CMMI: Using Process Simulation](https://reader034.fdocuments.net/reader034/viewer/2022051605/6013924b9aea010f0f6554eb/html5/thumbnails/35.jpg)
page 35
Portland StateUniversity
CMMI Based Process ImprovementCMMI Levels 4 and 5• Process simulation helps to fulfill PAs (OID, CAR, OPP
and QPM - Sub Goals and Generic Goals)
CMMI Levels 2 and 3• Process simulation can be used to evaluate alternative
process choices (RD, TS, PI, V&V, RM, SAM, PPQA, and CM)
• Process simulation helps to fulfill PAs (OPF, OPD, OT, IPM, Risk, DAR, PP, PMA, MA, PPQA – Multiple Sub Goals and Generic Goals )
![Page 36: Pittsburgh, PA 15213-3890 The ROI of CMMI: Using Process … · 2017. 5. 19. · page 1 Pittsburgh, PA 15213-3890 Portland State University The ROI of CMMI: Using Process Simulation](https://reader034.fdocuments.net/reader034/viewer/2022051605/6013924b9aea010f0f6554eb/html5/thumbnails/36.jpg)
page 36
Portland StateUniversity
Case Study: Organizational Setting• Leading software development firm• Peak staffing of 60 developers on project• Assessed at strong Level 2 of CMM/CMMI• Experienced development staff• 5th release of commercial project• Data available in electronic and paper form:
quantitative and qualitative; professional estimates used to fill in gaps
• Active SEPG
![Page 37: Pittsburgh, PA 15213-3890 The ROI of CMMI: Using Process … · 2017. 5. 19. · page 1 Pittsburgh, PA 15213-3890 Portland State University The ROI of CMMI: Using Process Simulation](https://reader034.fdocuments.net/reader034/viewer/2022051605/6013924b9aea010f0f6554eb/html5/thumbnails/37.jpg)
page 37
Portland StateUniversity
Case Study: Validation and Verification• Problem: Releasing defective products,
had high schedule variance.• Why? Unit Test was main defect removal
stage. They did it unreliably.• Built a model of Large-Scale commercial
development process• Based on actual project data • Predicted project performance in terms of
effort, task duration and delivered defects. • Part of a full business case analysis -
determined financial performance of the process change
![Page 38: Pittsburgh, PA 15213-3890 The ROI of CMMI: Using Process … · 2017. 5. 19. · page 1 Pittsburgh, PA 15213-3890 Portland State University The ROI of CMMI: Using Process Simulation](https://reader034.fdocuments.net/reader034/viewer/2022051605/6013924b9aea010f0f6554eb/html5/thumbnails/38.jpg)
page 38
Portland StateUniversity
Process Overview - 1
Func SpecFS Insp
1
HL DesignHLD Insp
LL DesignLLD Insp
Code Code Insp
Unit Test Execution
Functional Test
1 Test PlanTP Insp
Test Case TC Insp
System Test
Field Support and Main-tenance
Project is Approved Development
Complete Unit Test Complete
Release to Customers
Change
Diagram of the Field Study Life Cycle AS-IS Process
Tasks AffectedBy ProcessChange
![Page 39: Pittsburgh, PA 15213-3890 The ROI of CMMI: Using Process … · 2017. 5. 19. · page 1 Pittsburgh, PA 15213-3890 Portland State University The ROI of CMMI: Using Process Simulation](https://reader034.fdocuments.net/reader034/viewer/2022051605/6013924b9aea010f0f6554eb/html5/thumbnails/39.jpg)
page 39
Portland StateUniversity
Process Overview - 2
Code DevCode Insp
Unit Test Execution
Begin Code Development
Conducted during Code Development
Create Unit Test Plans
Prep, Insp, and RWK UT Plans
Follow UT Plan
Conducted as part of regular Code Inspection
Followed while conducting Unit
Test
Unit Test Complete; Begin Functional Testing
Code Development is Complete
Code Inspection is Complete
![Page 40: Pittsburgh, PA 15213-3890 The ROI of CMMI: Using Process … · 2017. 5. 19. · page 1 Pittsburgh, PA 15213-3890 Portland State University The ROI of CMMI: Using Process Simulation](https://reader034.fdocuments.net/reader034/viewer/2022051605/6013924b9aea010f0f6554eb/html5/thumbnails/40.jpg)
page 40
Portland StateUniversity
Overview of PTAM• Set-up Phase
- Set the Goal of the Modeling Effort- Specify Questions for the Model to Address- Determine Organizational Scope- Define Process Performance Measures - Identify Input Data
• Gather Information• Modeling Phase• Analysis Phase
![Page 41: Pittsburgh, PA 15213-3890 The ROI of CMMI: Using Process … · 2017. 5. 19. · page 1 Pittsburgh, PA 15213-3890 Portland State University The ROI of CMMI: Using Process Simulation](https://reader034.fdocuments.net/reader034/viewer/2022051605/6013924b9aea010f0f6554eb/html5/thumbnails/41.jpg)
page 41
Portland StateUniversity
Specify Questions• Based upon the goal/ purpose of the
simulation model, specific management questions can be identified
• Questions should point to specific answers that management would like to obtain
• It should be recognized that the model may not be able to answer or even address all of the questions.
• The questions should document the full scope issues and information that need to be incorporated into the decision making process
• Questions document the use case
![Page 42: Pittsburgh, PA 15213-3890 The ROI of CMMI: Using Process … · 2017. 5. 19. · page 1 Pittsburgh, PA 15213-3890 Portland State University The ROI of CMMI: Using Process Simulation](https://reader034.fdocuments.net/reader034/viewer/2022051605/6013924b9aea010f0f6554eb/html5/thumbnails/42.jpg)
page 42
Portland StateUniversity
Example Questions• What is the optimal V&V strategy for a specific project?
For our organizational process?• Would it be better to use Requirements process “A” or “B” for
this new project?• What combination(s) of V&V techniques enable us to meet
or exceed the quality goals for the system? Which alternative is best?
• Given a budget of “X” dollars, what V&V activities should be conducted?
• What is the value of applying automated tools to support development activities?
• What is the level of Risk associated with a change?
![Page 43: Pittsburgh, PA 15213-3890 The ROI of CMMI: Using Process … · 2017. 5. 19. · page 1 Pittsburgh, PA 15213-3890 Portland State University The ROI of CMMI: Using Process Simulation](https://reader034.fdocuments.net/reader034/viewer/2022051605/6013924b9aea010f0f6554eb/html5/thumbnails/43.jpg)
page 43
Portland StateUniversity
Case Study: Questions Investigated• Will the process change improve project
performance?• What is the cost the firm is currently paying by
conducting Unit Tests incorrectly? • Is partial implementation of the proposed
process change possible? • How would potential learning curve effects
affect the performance of the process change?• Would alternative process changes offer a
greater improvement?• Can the project benefit from reusing process
artifacts?
![Page 44: Pittsburgh, PA 15213-3890 The ROI of CMMI: Using Process … · 2017. 5. 19. · page 1 Pittsburgh, PA 15213-3890 Portland State University The ROI of CMMI: Using Process Simulation](https://reader034.fdocuments.net/reader034/viewer/2022051605/6013924b9aea010f0f6554eb/html5/thumbnails/44.jpg)
page 44
Portland StateUniversity
Overview of PTAM• Set-up Phase
- Set the Goal of the Modeling Effort- Specify Questions for the Model to Address- Define Process Performance Measures - Identify Input Data
• Gather Information• Modeling Phase• Analysis Phase
![Page 45: Pittsburgh, PA 15213-3890 The ROI of CMMI: Using Process … · 2017. 5. 19. · page 1 Pittsburgh, PA 15213-3890 Portland State University The ROI of CMMI: Using Process Simulation](https://reader034.fdocuments.net/reader034/viewer/2022051605/6013924b9aea010f0f6554eb/html5/thumbnails/45.jpg)
page 45
Portland StateUniversity
Define Process Performance Measures
• Main output measures of the simulation• Should capture management interests and
interests of engineers responsible for implementing the process changes.
• Must enable the questions to be answered• Helps focus data collection and modeling
efforts. • Should be defined as early as possible on the
project
![Page 46: Pittsburgh, PA 15213-3890 The ROI of CMMI: Using Process … · 2017. 5. 19. · page 1 Pittsburgh, PA 15213-3890 Portland State University The ROI of CMMI: Using Process Simulation](https://reader034.fdocuments.net/reader034/viewer/2022051605/6013924b9aea010f0f6554eb/html5/thumbnails/46.jpg)
page 46
Portland StateUniversity
Examples of Common Performance Measures
Typical performance measures include the following:• effort / cost• cycle-time (a.k.a. interval, duration, schedule)• defect level• staffing requirements• staff utilization rate• cost / benefit, return on investment• throughput / productivity• queue lengths (backlogs)
![Page 47: Pittsburgh, PA 15213-3890 The ROI of CMMI: Using Process … · 2017. 5. 19. · page 1 Pittsburgh, PA 15213-3890 Portland State University The ROI of CMMI: Using Process Simulation](https://reader034.fdocuments.net/reader034/viewer/2022051605/6013924b9aea010f0f6554eb/html5/thumbnails/47.jpg)
page 47
Portland StateUniversity
Case Study: Performance MeasuresCost• Person-Months of Development, Inspection,
Testing and Rework effort• Equivalent Manpower (Staffing levels)• Implementation costs
Quality• Number of delivered defects by type
Schedule• Months of Effort
![Page 48: Pittsburgh, PA 15213-3890 The ROI of CMMI: Using Process … · 2017. 5. 19. · page 1 Pittsburgh, PA 15213-3890 Portland State University The ROI of CMMI: Using Process Simulation](https://reader034.fdocuments.net/reader034/viewer/2022051605/6013924b9aea010f0f6554eb/html5/thumbnails/48.jpg)
page 48
Portland StateUniversity
Overview of PTAM• Set-up Phase
- Set the Goal of the Modeling Effort- Specify Questions for the Model to Address- Define Process Performance Measures - Identify Input Data
• Gather Information• Modeling Phase• Analysis Phase
![Page 49: Pittsburgh, PA 15213-3890 The ROI of CMMI: Using Process … · 2017. 5. 19. · page 1 Pittsburgh, PA 15213-3890 Portland State University The ROI of CMMI: Using Process Simulation](https://reader034.fdocuments.net/reader034/viewer/2022051605/6013924b9aea010f0f6554eb/html5/thumbnails/49.jpg)
page 49
Portland StateUniversity
Input Data (1 of 2)
• Input data are used to predict the performance measures.
• Can be derived from the organization - Current baseline- Exemplary projects- Pilot data
• Can also be derived from- Expert opinion- Industry data from comparable organizations
• Best judgments to describe the state of your organization
![Page 50: Pittsburgh, PA 15213-3890 The ROI of CMMI: Using Process … · 2017. 5. 19. · page 1 Pittsburgh, PA 15213-3890 Portland State University The ROI of CMMI: Using Process Simulation](https://reader034.fdocuments.net/reader034/viewer/2022051605/6013924b9aea010f0f6554eb/html5/thumbnails/50.jpg)
page 50
Portland StateUniversity
Input Data (2 of 2)
Examples:• process documents and assessments• amount of incoming work• effort based on size (and/ or other factors)• defect detection efficiency• effort for rework based on size and number of defects• defect injection, detection and removal rates• decision point outcomes; number of rework cycles• hiring rate; staff turnover rate• personnel capability and motivation, over time• resource constraints• frequency of product version releases
![Page 51: Pittsburgh, PA 15213-3890 The ROI of CMMI: Using Process … · 2017. 5. 19. · page 1 Pittsburgh, PA 15213-3890 Portland State University The ROI of CMMI: Using Process Simulation](https://reader034.fdocuments.net/reader034/viewer/2022051605/6013924b9aea010f0f6554eb/html5/thumbnails/51.jpg)
page 51
Portland StateUniversity
Case Study: Input Data• CMM Level 2+ organization• Process documents and assessments• Project Size• Productivity• Earned Value by phase• Total number of defects removed• Defect injection, detection and correction rates• Effort and schedule data• Defect detection and rework costs
![Page 52: Pittsburgh, PA 15213-3890 The ROI of CMMI: Using Process … · 2017. 5. 19. · page 1 Pittsburgh, PA 15213-3890 Portland State University The ROI of CMMI: Using Process Simulation](https://reader034.fdocuments.net/reader034/viewer/2022051605/6013924b9aea010f0f6554eb/html5/thumbnails/52.jpg)
page 52
Portland StateUniversity
Overview of PTAM• Set-up Phase
- Set the Goal of the Modeling Effort- Specify Questions for the Model to Address- Define Process Performance Measures - Identify Input Data
• Gather Information- Gather qualitative and quantitative data about
processes and products from variety of sources in variety of forms
• Modeling Phase• Analysis Phase
![Page 53: Pittsburgh, PA 15213-3890 The ROI of CMMI: Using Process … · 2017. 5. 19. · page 1 Pittsburgh, PA 15213-3890 Portland State University The ROI of CMMI: Using Process Simulation](https://reader034.fdocuments.net/reader034/viewer/2022051605/6013924b9aea010f0f6554eb/html5/thumbnails/53.jpg)
page 53
Portland StateUniversity
Overview of PTAM• Set up phase
- Set the Goal of the Modeling Effort- Specify Questions for the Model to Address- Define Process Performance Measures - Identify Input Data
• Gather Information• Modeling Phase• Analysis Phase
![Page 54: Pittsburgh, PA 15213-3890 The ROI of CMMI: Using Process … · 2017. 5. 19. · page 1 Pittsburgh, PA 15213-3890 Portland State University The ROI of CMMI: Using Process Simulation](https://reader034.fdocuments.net/reader034/viewer/2022051605/6013924b9aea010f0f6554eb/html5/thumbnails/54.jpg)
page 54
Portland StateUniversity
Process Models• First, create the graphical model• Quantitative portion of the simulation model can be
theoretical or data driven- Data driven models analyze actual data from
past projects using statistical techniques such as correlation coefficients and regression.
- Theoretical models are independent of data (relationships)
• Process simulation can incorporate many kinds of analytical models (data driven or theoretical)- COCOMO, SLIM- Reliability- Other Regression, Queuing and others
![Page 55: Pittsburgh, PA 15213-3890 The ROI of CMMI: Using Process … · 2017. 5. 19. · page 1 Pittsburgh, PA 15213-3890 Portland State University The ROI of CMMI: Using Process Simulation](https://reader034.fdocuments.net/reader034/viewer/2022051605/6013924b9aea010f0f6554eb/html5/thumbnails/55.jpg)
page 55
Portland StateUniversity
Case Study: Build the Graphical Model
Func SpecFS Insp
1
HL DesignHLD Insp
LL DesignLLD Insp
Code Code Insp
Unit Test Execution
Functional Test
1 Test PlanTP Insp
Test Case TC Insp
System Test
Field Support and Main-tenance
Project is Approved Development
Complete Unit Test Complete
Release to Customers
Change
Diagram of the Field Study Life Cycle AS-IS Process
Tasks AffectedBy ProcessChange
![Page 56: Pittsburgh, PA 15213-3890 The ROI of CMMI: Using Process … · 2017. 5. 19. · page 1 Pittsburgh, PA 15213-3890 Portland State University The ROI of CMMI: Using Process Simulation](https://reader034.fdocuments.net/reader034/viewer/2022051605/6013924b9aea010f0f6554eb/html5/thumbnails/56.jpg)
page 56
Portland StateUniversity
Case Study: Simplified Error ModelErrors
injectedin thisphase
Errors detected(and removed)
PerformWork
VerifyWork
ReworkDetected
Errors
Undetectederrors from
previous phase
Undetectederrors to
next phaseVerificationEfficiency
Errors DetectedTotal Errors Present=
![Page 57: Pittsburgh, PA 15213-3890 The ROI of CMMI: Using Process … · 2017. 5. 19. · page 1 Pittsburgh, PA 15213-3890 Portland State University The ROI of CMMI: Using Process Simulation](https://reader034.fdocuments.net/reader034/viewer/2022051605/6013924b9aea010f0f6554eb/html5/thumbnails/57.jpg)
page 57
Portland StateUniversity
More Detailed Error Model
PreliminaryDesign
DevTA
TARev
Rwk
DevUA
UARev Rwk
Code CodeRev Rwk
RwkUT
RwkPT
RwkInt
Test
Errors injected
Errors detected
Errorsundetected
Errors injected
Errors injected
Errors detected
Errors detected
Errorsundetected
Errorsundetected
Errorsundetected
Errorsundetected
Errors detected
Errors detected
Errorsdetected
SW Req. Analysis& Preliminary Dsn SW Detailed Design Coding
UnitTest
ProcessTest
Integration& Formal Test
![Page 58: Pittsburgh, PA 15213-3890 The ROI of CMMI: Using Process … · 2017. 5. 19. · page 1 Pittsburgh, PA 15213-3890 Portland State University The ROI of CMMI: Using Process Simulation](https://reader034.fdocuments.net/reader034/viewer/2022051605/6013924b9aea010f0f6554eb/html5/thumbnails/58.jpg)
page 58
Portland StateUniversity
Linking Effort, Duration, and Staffing
Size(Input)
Productivity(Input)
Earned Value*(Input)
Effort*
Duration*(Input)
Task Offset*(Input)
Project and ActivityDuration*
Equivalent Manpower*
Staffing*(Input)
Detection and Rework Costs
![Page 59: Pittsburgh, PA 15213-3890 The ROI of CMMI: Using Process … · 2017. 5. 19. · page 1 Pittsburgh, PA 15213-3890 Portland State University The ROI of CMMI: Using Process Simulation](https://reader034.fdocuments.net/reader034/viewer/2022051605/6013924b9aea010f0f6554eb/html5/thumbnails/59.jpg)
page 59
Portland StateUniversity
Overview of PTAM• Set up Phase
- Set the Goal of the Modeling Effort- Specify Questions for the Model to Address- Define Process Performance Measures - Identify Input Data
• Gather Information• Modeling Phase• Analysis Phase
![Page 60: Pittsburgh, PA 15213-3890 The ROI of CMMI: Using Process … · 2017. 5. 19. · page 1 Pittsburgh, PA 15213-3890 Portland State University The ROI of CMMI: Using Process Simulation](https://reader034.fdocuments.net/reader034/viewer/2022051605/6013924b9aea010f0f6554eb/html5/thumbnails/60.jpg)
page 60
Portland StateUniversity
Analysis PhaseOnce the model results are validated and viewed as being credible, they can be used to support decisions.Major Steps• Evaluate Baseline Process Alternatives• Determine Tradeoff Rule(s)• Conduct Sensitivity Analyses• Select Alternative(s) for Implementation
![Page 61: Pittsburgh, PA 15213-3890 The ROI of CMMI: Using Process … · 2017. 5. 19. · page 1 Pittsburgh, PA 15213-3890 Portland State University The ROI of CMMI: Using Process Simulation](https://reader034.fdocuments.net/reader034/viewer/2022051605/6013924b9aea010f0f6554eb/html5/thumbnails/61.jpg)
page 61
Portland StateUniversity
Project Level Outputs –Which Alternative to Choose?
CONFIG DeliveredDefects
Life Cycle Effort
Project Duration
W W N N 13.4 51.72 17.81 F F N N 12.6 52.83 17.26
W W N W 9.1 48.79 14.92 W W W W 6.6 47.25 12.85
F F F F 3.3 48.60 12.11
![Page 62: Pittsburgh, PA 15213-3890 The ROI of CMMI: Using Process … · 2017. 5. 19. · page 1 Pittsburgh, PA 15213-3890 Portland State University The ROI of CMMI: Using Process Simulation](https://reader034.fdocuments.net/reader034/viewer/2022051605/6013924b9aea010f0f6554eb/html5/thumbnails/62.jpg)
page 62
Portland StateUniversity
Comparison by Mean Difference
CONFIG ReducedDefects
Reduced Effort
ReducedDuration
W W N N 0.00 0.00 0.00 F F N N 0.80 -1.11 0.55
W W N W 4.34 2.92 2.89 W W W W 6.82 4.47 4.96
F F F F 10.18 3.12 5.71
![Page 63: Pittsburgh, PA 15213-3890 The ROI of CMMI: Using Process … · 2017. 5. 19. · page 1 Pittsburgh, PA 15213-3890 Portland State University The ROI of CMMI: Using Process Simulation](https://reader034.fdocuments.net/reader034/viewer/2022051605/6013924b9aea010f0f6554eb/html5/thumbnails/63.jpg)
page 63
Portland StateUniversity
50 800
5
10
15
CUM
FIGURE 2 - PERFORMANCE MEASURE DISTRIBUTIONS
50 8020
30
40
50
60
*
*
*
CUM
50 8010
20
30
40
50
*
CUM
REM_ERR = Number of remaining errors; TOT_DUR = Total project duration (in days); TOT_EFF = Total staff effort (in days); CUM = Cumulative error detection capability
(% of initial errors detected); 50 = "AS-IS" No Inspection Baseline; 80 = "TO-BE" Inspection Baseline
Case Study: Baseline ComparisonQuality Schedule Effort
![Page 64: Pittsburgh, PA 15213-3890 The ROI of CMMI: Using Process … · 2017. 5. 19. · page 1 Pittsburgh, PA 15213-3890 Portland State University The ROI of CMMI: Using Process Simulation](https://reader034.fdocuments.net/reader034/viewer/2022051605/6013924b9aea010f0f6554eb/html5/thumbnails/64.jpg)
page 64
Portland StateUniversity
Analysis Phase• Evaluate Baseline Process Alternatives• Determine Tradeoff Rule(s)• Conduct Sensitivity Analyses• Select Alternative(s) for Implementation
![Page 65: Pittsburgh, PA 15213-3890 The ROI of CMMI: Using Process … · 2017. 5. 19. · page 1 Pittsburgh, PA 15213-3890 Portland State University The ROI of CMMI: Using Process Simulation](https://reader034.fdocuments.net/reader034/viewer/2022051605/6013924b9aea010f0f6554eb/html5/thumbnails/65.jpg)
page 65
Portland StateUniversity
Determine Tradeoff Rule(s)
Which alternative is best?
Need to reduce multiple performance measures to one decision statistic that can be used to rank process alternatives.Possible Options• Utility functions• Financial measures (e.g. Net Present Value (NPV), Internal
Rate of Return (IRR aka ROI), etc.)• Optimization techniques (e.g. Data Envelopment Analysis
(DEA))• Analytic Hierarchy Process (AHP)• Combination
![Page 66: Pittsburgh, PA 15213-3890 The ROI of CMMI: Using Process … · 2017. 5. 19. · page 1 Pittsburgh, PA 15213-3890 Portland State University The ROI of CMMI: Using Process Simulation](https://reader034.fdocuments.net/reader034/viewer/2022051605/6013924b9aea010f0f6554eb/html5/thumbnails/66.jpg)
page 66
Portland StateUniversity
Financial Measures of Performance
• Gets management interest (and excitement)• Supports building a business case• Trick is to convert performance measures to
cash equivalents• Examples:
- Net present value (NPV)- Internal rate of return (IRR aka ROI), etc.- Discounted Payback period
![Page 67: Pittsburgh, PA 15213-3890 The ROI of CMMI: Using Process … · 2017. 5. 19. · page 1 Pittsburgh, PA 15213-3890 Portland State University The ROI of CMMI: Using Process Simulation](https://reader034.fdocuments.net/reader034/viewer/2022051605/6013924b9aea010f0f6554eb/html5/thumbnails/67.jpg)
page 67
Portland StateUniversity
Determining Financial Benefits• Need to reduce all benefits to cash equivalents• Implementation costs are easy to include • Effort is a straight forward conversion• Some measures can be converted to effort (e.g.
number of customer defects are converted to the effort to correct them)
• Other measures (e.g. time to market) can be difficult to convert.
![Page 68: Pittsburgh, PA 15213-3890 The ROI of CMMI: Using Process … · 2017. 5. 19. · page 1 Pittsburgh, PA 15213-3890 Portland State University The ROI of CMMI: Using Process Simulation](https://reader034.fdocuments.net/reader034/viewer/2022051605/6013924b9aea010f0f6554eb/html5/thumbnails/68.jpg)
page 68
Portland StateUniversity
Ranking by Financial Performance
Rank CONFIG NPV(15%) Mean
NPV(15%) STDev
PR(NPV<0)(Risk)
1 F F F F $362,291.35 $118,344.45 0.11% 2 W W W W $253,041.92 $68,513.12 0.08% 3 W W N W $157,874.18 $44,518.84 0.09% 4 F F N N $27,836.80 $26,910.00 15.15% 5 W W N N $0.00 NA NA
![Page 69: Pittsburgh, PA 15213-3890 The ROI of CMMI: Using Process … · 2017. 5. 19. · page 1 Pittsburgh, PA 15213-3890 Portland State University The ROI of CMMI: Using Process Simulation](https://reader034.fdocuments.net/reader034/viewer/2022051605/6013924b9aea010f0f6554eb/html5/thumbnails/69.jpg)
page 69
Portland StateUniversity
Case Study: Cash Flows
time=0 time = 21 months
time = 33 months
- $31,650
- $4,591
+ $ 105,569
Implementation Costs
Life Cycle Effort
Savings
21 months = 16.44 TO-BE Component
Duration * 1.25
33 months = Project duration of 21 months + 12 months
Field Service Effort Savings
![Page 70: Pittsburgh, PA 15213-3890 The ROI of CMMI: Using Process … · 2017. 5. 19. · page 1 Pittsburgh, PA 15213-3890 Portland State University The ROI of CMMI: Using Process Simulation](https://reader034.fdocuments.net/reader034/viewer/2022051605/6013924b9aea010f0f6554eb/html5/thumbnails/70.jpg)
page 70
Portland StateUniversity
Case Study: Results• The process change offered significant
reductions in remaining defects, staff effort to correct field detected defects, and project duration. The expected ROI was 56% for a typical 30 KLOC release.
• Pilot implementations indicated that the process change provided a 37% ROI even under worst case conditions.
![Page 71: Pittsburgh, PA 15213-3890 The ROI of CMMI: Using Process … · 2017. 5. 19. · page 1 Pittsburgh, PA 15213-3890 Portland State University The ROI of CMMI: Using Process Simulation](https://reader034.fdocuments.net/reader034/viewer/2022051605/6013924b9aea010f0f6554eb/html5/thumbnails/71.jpg)
page 71
Portland StateUniversity
Analysis Phase• Evaluate Baseline Process Alternatives• Determine Tradeoff Rule(s)• Conduct Sensitivity Analyses• Select Alternative(s) for Implementation
![Page 72: Pittsburgh, PA 15213-3890 The ROI of CMMI: Using Process … · 2017. 5. 19. · page 1 Pittsburgh, PA 15213-3890 Portland State University The ROI of CMMI: Using Process Simulation](https://reader034.fdocuments.net/reader034/viewer/2022051605/6013924b9aea010f0f6554eb/html5/thumbnails/72.jpg)
page 72
Portland StateUniversity
Conduct Sensitivity Analyses• “What if” analyses allow managers to apply
the model to evaluate the proposed process change(s) under different business conditions and assumptions.
• Provides added insight and confidence into the potential process change
![Page 73: Pittsburgh, PA 15213-3890 The ROI of CMMI: Using Process … · 2017. 5. 19. · page 1 Pittsburgh, PA 15213-3890 Portland State University The ROI of CMMI: Using Process Simulation](https://reader034.fdocuments.net/reader034/viewer/2022051605/6013924b9aea010f0f6554eb/html5/thumbnails/73.jpg)
page 73
Portland StateUniversity
Case Study: Questions Investigated• Will the process change improve project performance?• What is the cost the firm is currently paying by
conducting Unit Tests incorrectly? • Is partial implementation of the proposed process
change possible? • How would potential learning curve effects affect the
performance of the process change?• Would alternative process changes offer a greater
improvement?• Can the project benefit from reusing process artifacts?
![Page 74: Pittsburgh, PA 15213-3890 The ROI of CMMI: Using Process … · 2017. 5. 19. · page 1 Pittsburgh, PA 15213-3890 Portland State University The ROI of CMMI: Using Process Simulation](https://reader034.fdocuments.net/reader034/viewer/2022051605/6013924b9aea010f0f6554eb/html5/thumbnails/74.jpg)
page 74
Portland StateUniversity
Case Study: Results Obtained• Compressing Unit Test causes significant
increases in schedule (+18%) and effort costs (+8%) during the later testing phases and reduces overall product quality(+48% increase in defects).
• Partial implementation of the process change is possible for complex portions of the code. Estimated ROI is 72%.
• Potential learning curve effects significantly enhance the performance of the process change. Expected ROI of 72% assuming only moderate improvements.
![Page 75: Pittsburgh, PA 15213-3890 The ROI of CMMI: Using Process … · 2017. 5. 19. · page 1 Pittsburgh, PA 15213-3890 Portland State University The ROI of CMMI: Using Process Simulation](https://reader034.fdocuments.net/reader034/viewer/2022051605/6013924b9aea010f0f6554eb/html5/thumbnails/75.jpg)
page 75
Portland StateUniversity
Case Study: Results Obtained• Improving inspections would be a more effective
process improvement than the Creating Unit Test Plans process change.
• Reusing the Unit Test Plans on the next development cycle provided an overall ROI of 73% (compared to 56% expected improvement without reuse)
![Page 76: Pittsburgh, PA 15213-3890 The ROI of CMMI: Using Process … · 2017. 5. 19. · page 1 Pittsburgh, PA 15213-3890 Portland State University The ROI of CMMI: Using Process Simulation](https://reader034.fdocuments.net/reader034/viewer/2022051605/6013924b9aea010f0f6554eb/html5/thumbnails/76.jpg)
page 76
Portland StateUniversity
Analysis Phase• Evaluate Baseline Process Alternatives• Select Evaluation Method and Criteria• Conduct Sensitivity Analyses• Select Alternative(s) for Implementation
![Page 77: Pittsburgh, PA 15213-3890 The ROI of CMMI: Using Process … · 2017. 5. 19. · page 1 Pittsburgh, PA 15213-3890 Portland State University The ROI of CMMI: Using Process Simulation](https://reader034.fdocuments.net/reader034/viewer/2022051605/6013924b9aea010f0f6554eb/html5/thumbnails/77.jpg)
page 77
Portland StateUniversity
Select Alternative(s) for Implementation
• Process simulation can be used to estimate the ROI and risk
• Results are traded-off with other factors not included in the model such as budget and political considerations
• Utilize all the information at hand (quantitative and qualitative) to choose the best alternative
![Page 78: Pittsburgh, PA 15213-3890 The ROI of CMMI: Using Process … · 2017. 5. 19. · page 1 Pittsburgh, PA 15213-3890 Portland State University The ROI of CMMI: Using Process Simulation](https://reader034.fdocuments.net/reader034/viewer/2022051605/6013924b9aea010f0f6554eb/html5/thumbnails/78.jpg)
page 78
Pittsburgh, PA 15213-3890
Portland StateUniversity
6 – Examples of Process Simulation Applications in Industry and Government
![Page 79: Pittsburgh, PA 15213-3890 The ROI of CMMI: Using Process … · 2017. 5. 19. · page 1 Pittsburgh, PA 15213-3890 Portland State University The ROI of CMMI: Using Process Simulation](https://reader034.fdocuments.net/reader034/viewer/2022051605/6013924b9aea010f0f6554eb/html5/thumbnails/79.jpg)
page 79
Portland StateUniversity
NASA IV&V
• Mission: To independently verify and validate software on all missions that are life critical or have significant vehicle cost involved.
• Problem: Limited resources to conduct IV&V. Critical need to deploy IV&V in most effective manner possible (biggest return on investment)
• Goal to optimize IV&V within a project and across projects.
![Page 80: Pittsburgh, PA 15213-3890 The ROI of CMMI: Using Process … · 2017. 5. 19. · page 1 Pittsburgh, PA 15213-3890 Portland State University The ROI of CMMI: Using Process Simulation](https://reader034.fdocuments.net/reader034/viewer/2022051605/6013924b9aea010f0f6554eb/html5/thumbnails/80.jpg)
page 80
Portland StateUniversity
NASA IV&V
Description of Model• Based on IEEE 12207 Software Development Process • Tuned for large-scale NASA projects (Size >100 KSLOC)
(uses actual data)• 8 major life cycle phases; 86 process steps • Includes IV&V Layer• Compares alternative IV&V configurations (ROI)
![Page 81: Pittsburgh, PA 15213-3890 The ROI of CMMI: Using Process … · 2017. 5. 19. · page 1 Pittsburgh, PA 15213-3890 Portland State University The ROI of CMMI: Using Process Simulation](https://reader034.fdocuments.net/reader034/viewer/2022051605/6013924b9aea010f0f6554eb/html5/thumbnails/81.jpg)
page 81
Portland StateUniversity
NASA IV&V• Mission: To independently verify and validate software
on all missions that are life critical or have significant vehicle cost involved.
• Problem: Limited resources to conduct IV&V. Critical need to deploy IV&V in most effective manner possible (biggest return on investment)
• Goal to optimize IV&V within a project and across projects.
Description of Model• Based on IEEE 12207 Software Development Process • Tuned for large-scale NASA projects (>100 KLOC) (Real data)• 8 major life cycle phases; 86 process steps • Includes IV&V Layer• Compares alternative IV&V configurations (ROI)
![Page 82: Pittsburgh, PA 15213-3890 The ROI of CMMI: Using Process … · 2017. 5. 19. · page 1 Pittsburgh, PA 15213-3890 Portland State University The ROI of CMMI: Using Process Simulation](https://reader034.fdocuments.net/reader034/viewer/2022051605/6013924b9aea010f0f6554eb/html5/thumbnails/82.jpg)
page 82
Portland StateUniversity
NASA Model – Includes IV&V Layer with IEEE 12207 Software Development Lifecycle
![Page 83: Pittsburgh, PA 15213-3890 The ROI of CMMI: Using Process … · 2017. 5. 19. · page 1 Pittsburgh, PA 15213-3890 Portland State University The ROI of CMMI: Using Process Simulation](https://reader034.fdocuments.net/reader034/viewer/2022051605/6013924b9aea010f0f6554eb/html5/thumbnails/83.jpg)
page 83
Portland StateUniversity
IV&V Layer – Select Criticality Levels for IV&V Techniques using pull-down menus
![Page 84: Pittsburgh, PA 15213-3890 The ROI of CMMI: Using Process … · 2017. 5. 19. · page 1 Pittsburgh, PA 15213-3890 Portland State University The ROI of CMMI: Using Process Simulation](https://reader034.fdocuments.net/reader034/viewer/2022051605/6013924b9aea010f0f6554eb/html5/thumbnails/84.jpg)
page 84
Portland StateUniversity
A Look Inside the Model…
![Page 85: Pittsburgh, PA 15213-3890 The ROI of CMMI: Using Process … · 2017. 5. 19. · page 1 Pittsburgh, PA 15213-3890 Portland State University The ROI of CMMI: Using Process Simulation](https://reader034.fdocuments.net/reader034/viewer/2022051605/6013924b9aea010f0f6554eb/html5/thumbnails/85.jpg)
page 85
Portland StateUniversity
What is IV&V?
• IV&V = Independent Verification and Validation
• Performed by one or more independent groups
• Can be employed at any phase with different levels of coverage
![Page 86: Pittsburgh, PA 15213-3890 The ROI of CMMI: Using Process … · 2017. 5. 19. · page 1 Pittsburgh, PA 15213-3890 Portland State University The ROI of CMMI: Using Process Simulation](https://reader034.fdocuments.net/reader034/viewer/2022051605/6013924b9aea010f0f6554eb/html5/thumbnails/86.jpg)
page 86
Portland StateUniversity
IV&V Techniques
• Traceability Analysis• Software Design Evaluation• Interface Analysis• Criticality Analysis• Component Test Plan Verification• V&V Test Design Verification• Hazard Analysis• And etc.
![Page 87: Pittsburgh, PA 15213-3890 The ROI of CMMI: Using Process … · 2017. 5. 19. · page 1 Pittsburgh, PA 15213-3890 Portland State University The ROI of CMMI: Using Process Simulation](https://reader034.fdocuments.net/reader034/viewer/2022051605/6013924b9aea010f0f6554eb/html5/thumbnails/87.jpg)
page 87
Portland StateUniversity
Importance/Benefits – Enduring NeedsIV&V Level• IV&V New Business Planning (Independent Bottoms-Up Cost
Estimation for NASA Projects and for IV&V)• IV&V Policy Research (IV&V strategies for alternative NASA
Project types)• IV&V Services Contract Bid Support • IV&V Services Replanning• Cost/Benefit Evaluation of new technologies and tools • Space Science Data Mining
![Page 88: Pittsburgh, PA 15213-3890 The ROI of CMMI: Using Process … · 2017. 5. 19. · page 1 Pittsburgh, PA 15213-3890 Portland State University The ROI of CMMI: Using Process Simulation](https://reader034.fdocuments.net/reader034/viewer/2022051605/6013924b9aea010f0f6554eb/html5/thumbnails/88.jpg)
page 88
Portland StateUniversity
Macro IV&V Questions• What is the optimal IV&V strategy for a given
NASA project or NASA project type?• What combination(s) of IV&V techniques enable
us to meet or exceed the quality assurance goals for the system?
• Given a budget of “X” dollars, what IV&V activities should be conducted?
• What if the complexity or defect profiles for a particular project were different than expected?
• How is the duration of the IV&V effort impacted by the overall staffing level for the project?
![Page 89: Pittsburgh, PA 15213-3890 The ROI of CMMI: Using Process … · 2017. 5. 19. · page 1 Pittsburgh, PA 15213-3890 Portland State University The ROI of CMMI: Using Process Simulation](https://reader034.fdocuments.net/reader034/viewer/2022051605/6013924b9aea010f0f6554eb/html5/thumbnails/89.jpg)
page 89
Portland StateUniversity
Preliminary Study• Use the model to quantitatively assess the
benefits of performing IV&V on software development projects
• Comparing benefit of applying IV&V activities at different phases and in combination
![Page 90: Pittsburgh, PA 15213-3890 The ROI of CMMI: Using Process … · 2017. 5. 19. · page 1 Pittsburgh, PA 15213-3890 Portland State University The ROI of CMMI: Using Process Simulation](https://reader034.fdocuments.net/reader034/viewer/2022051605/6013924b9aea010f0f6554eb/html5/thumbnails/90.jpg)
page 90
Portland StateUniversity
Impact of IV&V at Different Points in the Development Process
Case ConfigurationTotal Effort
Mean Rework Effort
Mean Duration Mean Corrected Defects
Mean Latent Defects
Mean (Person Months) (Person Months) (Months) (Number of Defects) (Number of Defects)
1 Baseline 346.26 201.65 58.42 6,038.26 629.48 2 IV&V at Validation 355.35 210.75 59.95 6,113.79 574.17 3 IV&V at Code 334.13 189.53 57.38 6,134.84 573.49 4 IV&V at Design 327.93 183.33 56.56 6,123.11 581.27 5 IV&V at Requirements 326.82 182.21 56.40 6,078.87 600.04
Result Comparison
% Improvement Compared to the Baseline
Case ConfigurationTotal Effort
MeanRework Effort
MeanDuration
MeanCorrected Defects
MeanLatent Defects
Mean1 Baseline2 IV&V at Validation -2.63%* -4.51%* -2.63%* +1.25% +8.79%*3 IV&V at Code +3.50%* +6.01%* +1.77% +1.60% +8.90%*4 IV&V at Design +5.29%* +9.09%* +3.17%* +1.41% +7.66%*5 IV&V at Requirements +5.62%* +9.64%* +3.46%* +0.67% +4.68%*
![Page 91: Pittsburgh, PA 15213-3890 The ROI of CMMI: Using Process … · 2017. 5. 19. · page 1 Pittsburgh, PA 15213-3890 Portland State University The ROI of CMMI: Using Process Simulation](https://reader034.fdocuments.net/reader034/viewer/2022051605/6013924b9aea010f0f6554eb/html5/thumbnails/91.jpg)
page 91
Portland StateUniversity
Impact of IV&V Techniques in Combination
Case ConfigurationTotal Effort
Mean Rework Effort
Mean Duration Mean Corrected Defects
Mean Latent Defects
Mean (Person Months) (Person Months) (Months) (Number of Defects) (Number of Defects)
1 Baseline 346.26 201.65 58.42 6,038.26 629.48 6 IV&V at Code and Validation 342.14 197.54 58.78 6,203.66 524.96 7 IV&V at Req and Code 316.15 171.55 54.41 6,170.94 547.74 8 Two IV&V Techniques at Code 327.10 182.50 57.54 6,180.22 540.60
Case ConfigurationTotal Effort
MeanRework Effort
MeanDuration
MeanCorrected Defects
MeanLatent Defects
Mean1 Baseline6 IV&V at Code and Validation +1.19% +2.04% -0.63% +2.74% +16.60%*7 IV&V at Req and Code +8.69%* +14.93%* +6.86%* +2.20% +12.99%*8 Two IV&V Techniques at Code +5.53%* +9.50%* +1.50% +2.35% +14.12%*
Result Comparison
% Improvement Compared to the Baseline
![Page 92: Pittsburgh, PA 15213-3890 The ROI of CMMI: Using Process … · 2017. 5. 19. · page 1 Pittsburgh, PA 15213-3890 Portland State University The ROI of CMMI: Using Process Simulation](https://reader034.fdocuments.net/reader034/viewer/2022051605/6013924b9aea010f0f6554eb/html5/thumbnails/92.jpg)
page 92
Portland StateUniversity
Rapidly Deployable Software Process Simulation Models and Training
• Goal: To create a flexible decision support tool that can be easily used to support better project management, planning and tracking by quantitatively assessing the economic benefit of proposed process alternatives.
• Motivation: Companies need to get useful results from simulation models quickly.
![Page 93: Pittsburgh, PA 15213-3890 The ROI of CMMI: Using Process … · 2017. 5. 19. · page 1 Pittsburgh, PA 15213-3890 Portland State University The ROI of CMMI: Using Process Simulation](https://reader034.fdocuments.net/reader034/viewer/2022051605/6013924b9aea010f0f6554eb/html5/thumbnails/93.jpg)
page 93
Portland StateUniversity
Rapidly Deployable Process Models
REQ DES IMP TEST CUST
TP TCG
Life Cycle Model Generic Process Blocks
Generalized Equations
Code DevCodeInsp
Unit Test Functional
Test
System Test
FieldSupportandMain-tenance
H Lev DesignHLD Insp
L Lev Design LLDInsp
Func SpecFSInsp
Project is Approved Development
CompleteUnit Test Complete
Release to Customers
InspUTPlan
Follow UT Pln
ProposedProcessChange
CreateUTPlan
Software Development Process
![Page 94: Pittsburgh, PA 15213-3890 The ROI of CMMI: Using Process … · 2017. 5. 19. · page 1 Pittsburgh, PA 15213-3890 Portland State University The ROI of CMMI: Using Process Simulation](https://reader034.fdocuments.net/reader034/viewer/2022051605/6013924b9aea010f0f6554eb/html5/thumbnails/94.jpg)
page 94
Portland StateUniversity
Simulation Dashboard
![Page 95: Pittsburgh, PA 15213-3890 The ROI of CMMI: Using Process … · 2017. 5. 19. · page 1 Pittsburgh, PA 15213-3890 Portland State University The ROI of CMMI: Using Process Simulation](https://reader034.fdocuments.net/reader034/viewer/2022051605/6013924b9aea010f0f6554eb/html5/thumbnails/95.jpg)
page 95
Portland StateUniversity
Demonstration
![Page 96: Pittsburgh, PA 15213-3890 The ROI of CMMI: Using Process … · 2017. 5. 19. · page 1 Pittsburgh, PA 15213-3890 Portland State University The ROI of CMMI: Using Process Simulation](https://reader034.fdocuments.net/reader034/viewer/2022051605/6013924b9aea010f0f6554eb/html5/thumbnails/96.jpg)
page 96
Pittsburgh, PA 15213-3890
Portland StateUniversity
7 – Wrap up/ Conclusions
![Page 97: Pittsburgh, PA 15213-3890 The ROI of CMMI: Using Process … · 2017. 5. 19. · page 1 Pittsburgh, PA 15213-3890 Portland State University The ROI of CMMI: Using Process Simulation](https://reader034.fdocuments.net/reader034/viewer/2022051605/6013924b9aea010f0f6554eb/html5/thumbnails/97.jpg)
page 97
Portland StateUniversity
ConclusionsProcess simulation modeling has been used successfully to quantitatively address a variety of issues from strategic management to process understanding.
Key benefits include:• Decision Support and Tradeoff Analysis• Sensitivity Analysis – “What if”• Supports Industry Certification and process improvement
programs including CMMI, Six Sigma, and others• Supports CMMI at all levels 2 through 5• Design and Define Processes• Benchmarking• Can address project manager concerns• Supports project management and control
![Page 98: Pittsburgh, PA 15213-3890 The ROI of CMMI: Using Process … · 2017. 5. 19. · page 1 Pittsburgh, PA 15213-3890 Portland State University The ROI of CMMI: Using Process Simulation](https://reader034.fdocuments.net/reader034/viewer/2022051605/6013924b9aea010f0f6554eb/html5/thumbnails/98.jpg)
page 98
Portland StateUniversity
ConclusionsProcess Tradeoff Analysis Method (PTAM) provides a tested approach for developing models and utilizing the results
Not a silver bullet
Focus on RAPID DEPLOYMENT• Reducing costs and time to develop models• Making models easier to use – No
simulation expert needed
![Page 99: Pittsburgh, PA 15213-3890 The ROI of CMMI: Using Process … · 2017. 5. 19. · page 1 Pittsburgh, PA 15213-3890 Portland State University The ROI of CMMI: Using Process Simulation](https://reader034.fdocuments.net/reader034/viewer/2022051605/6013924b9aea010f0f6554eb/html5/thumbnails/99.jpg)
page 99
Portland StateUniversity
The EndQuestions?
![Page 100: Pittsburgh, PA 15213-3890 The ROI of CMMI: Using Process … · 2017. 5. 19. · page 1 Pittsburgh, PA 15213-3890 Portland State University The ROI of CMMI: Using Process Simulation](https://reader034.fdocuments.net/reader034/viewer/2022051605/6013924b9aea010f0f6554eb/html5/thumbnails/100.jpg)
page 100
Portland StateUniversity
Contact InformationDavid M. Raffo, Ph.D.Associate Professor, Portland State UniversityVisiting Scientist, Software Engineering Institute
President, Vaiyu, Inc.
VVAIYUAIYU