A Mini-Tutorial for Building CMMI Process Performance Models_1

8/20/2019 A Mini-Tutorial for Building CMMI Process Performance Models_1

1/89


2/89

SEI Webinar (28 Apr 09)

Speaker Biographies

Robert W. Stoddard currently serves as a Senior Member of the Technical Staff atthe Software Engineering Institute. Robert architected and designed several leading

measurement and CMMI High Maturity courses including: “Understanding CMMI

” “ ” , ,

“Designing Products and Processes using Six Sigma.”

us y oung curren y serves as e ppra sa anager a e . us y as +years S&SW development, management, and consulting He has worked in large and

small organizations; government and private industry. His recent focus has been

CMMI High Maturity.

Dave Zubrow currently serves as the Manager of the Software Engineering

easurement an na ys s n t at ve w t n t e o tware ng neer ngInstitute (SEI). Prior to joining the SEI, Dave served as Assistant Director of

Analytic Studies for Carnegie Mellon University. He is a SEI certified instructor

and appraiser, member of several editorial boards of professional journals, and

2Kevin Schaaff, Robert StoddardRusty Young, Dave Zubrow

© 2009 Carnegie Mellon University

act ve n stan ar s eve opment.


3/89

NO WARRANTY

MATERIAL IS FURNISHED ON AN “ AS-IS" BASIS. CARNEGIE MELLON UNIVERSITYMAKES NO WARRANTIES OF ANY KIND, EITHER EXPRESSED OR IMPLIED, AS TO

ANY MATTER INCLUDING, BUT NOT LIMITED TO, WARRANTY OF FITNESS FORPURPOSE OR MERCHANTABILITY, EXCLUSIVITY, OR RESULTS OBTAINED FROM

.

WARRANTY OF ANY KIND WITH RESPECT TO FREEDOM FROM PATENT,TRADEMARK, OR COPYRIGHT INFRINGEMENT.

Use of any trademarks in this presentation is not intended in any way to infringe on the.

This Presentation may be reproduced in its entirety, without modification, and freelydistributed in written or electronic form without requesting formal permission. Permission is

required for any other use. Requests for permission should be directed to the SoftwareEngineering Institute at [email protected].

This work was created in the performance of Federal Government Contract Number FA8721-05-C-0003 with Carnegie Mellon University for the operation of the SoftwareEngineering Institute, a federally funded research and development center. TheGovernment of the United States has a royalty-free government-purpose license to use,

, , ,permit others to do so, for government purposes pursuant to the copyright license under the clause at 252.227-7013.



4/89


Permission to use Crystal Ball Screen Shots

Portions of the input and output contained in this modulemanual are rinted with ermission of Oracle formerl

Decisioneering). Crystal Ball 7.2.2 (Build 7.2.1333.0) is usedto capture screenshots in this module

The Web page for Crystal Ball is available athttp://www.crystalball.com




5/89


Topics

Introduction

Review of Process Performance Models (PPMs)

Technical Process of Building PPMs

Questions




6/89

Review of CMMI

Models (PPMs)



7/89


What is a PPM?

OPP SP 1.5

• PPMs are used to estimate or redict the value of a rocess-

performance measure from the values of other process, product,and service measurements

• PPMs t icall use rocess and roduct measurements collectedthroughout the life of the project to estimate progress towardachieving objectives that cannot be measured until later in the

project’s life

Glossary

• A description of the relationships among attributes of a process andits work roducts that is develo ed from historical rocess-

performance data and calibrated using collected process andproduct measures from the project and that is used to predict resultsto be achieved by following a process




8/89


Purpose and Usage of Process Performance

Software Coding Software Unit Testing

Design Systems

Testing

Requirements

Requirements

Management

Integration Testing

Customer

Project

Forecasting

cceptance

Testing

ProjectStart Project

Project

Planning




9/89


Healthy Ingredients of CMMI Process

Statistical, probabilistic or simulation in nature

Predict interim and/or final project outcomesUse controllable factors tied to sub-processes to conduct the prediction

Model the variation of factors and understand the predicted range or

variation of the outcomes

Enable “what-if” anal sis for ro ect lannin d namic re- lannin and

problem resolution during project execution

Connect “upstream” activity with “downstream” activity

Enable projects to achieve mid-course corrections to ensure projectsuccess




10/89


Interactive Question #1

Do you now feel comfortable with your knowledge of thehealth in redients of a CMMI Process Performance Model

that were just presented?

es




11/89

Technical Process of



12/89


Topics• ev ew o rocess er ormance o e s s

• Technical Process of Building PPMs

1. Identify or Reconfirm Business Goals

2. Identify the sub-processes/process

’.

4. Identify Controllable factors (x’s) to predict outcomes

5. Include Uncontrollable x factors

6. Collect Data

7. Assess Data Quality and Integrity

8. Identify data types of all y outcomes and x factors

9. Create PPBs

10.Select the proper analytical technique and/or type of regression equation

11.Create Predictions with both Confidence and Prediction Intervals

. -

13.Take Action Based on PPM Predictions

14.Maintain PPMs including calibration and reconfirming relationships

15.Use PPMs to assist in CAR and OID

• Questions




13/89


1) Business Goal Flowdown (Y-to-x)

YYY Y High Level Business Goals(Balanced Scorecard)

g n o s t i c

yyy y y yy Subordinate Business Goals(e.g., $ Buckets,

o c e s s

- A

yyyy y y yy er ormance

High Level Process

P r

e d

xxx x x xx

(e.g., Organizational Processes)

Subordinate s - O r i e n

xxxx x x xxProcesses

(e.g., Down to a Vital xsub-process to be P

r o c e




14/89


2) Identify the Sub-Process/Process

• Start with the Organization’s Business Objectives

•

Objectives (QPPOs)• For the QPPOs that can be Measured Quantitatively

• Perform Analysis to Determine which Sub-Process/Process Drivesthe Relevant Objective

• Determine if Sufficient Data is Available or can be Obtained toEstablish a Process Performance Baseline(s) and/or Build aProcess Performance Model(s)




15/89


Identify the Sub-Process/Process

• Given Organizational Business Objectives:

• Im rove ualit

• Improve cycle time• Improve productivity

• rans a e o measurea e s

• Post-delivery defect density of less than 0.5 Defects/KSLOC

• Achieve 85% defect detection before System testing

• Ensure requirements duration is within 15% of plan

• Achieve a 5 % software productivity improvement




16/89


Identify the Sub-Process/Process

• The QPPOs after analysis were determined to be driven bythe followin rocesses

• Peer Review• Test

• The processes were then decomposed to the followingrelated sub-processes to be statistically managed

• Inspection

•

• Requirements Analysis




17/89


3) Identify Outcomes to Predict

DesignTransitionto Customer

Defectsinjected

Defectsinjected

SoftwareRequirements

Defectsinjected

Fielded

Defectsinjected

Implementation Integration

DesignReview

CodeReview

Defectsremoved

DefectsremovedReq.Review

Defectsremoved

Defectsremoved

Defectsremoved

Defectsremoved

Across business functions and disciplines!

Outcomes relatedto key handoffs of work durin ro ect

Outcomes relatedto interim milestonesand mgt reviews

Outcomes relatedto risks during the

to project completion




18/89


Examples of Outcomes

Rework

Progress*




19/89


4) Identify Controllable factors (x’s) to Predict

-

“ ”

influence over the factor prior to or during the projectexecution

A common misconception is that factors are not controllable.Requires out-of-the-box thinking to overcome this. Someorganizations employ individuals known as “assumption

”




20/89


Identify Controllable factors (x’s) to Predict

-

As we view process holistically, controllable factors may bere ate , ut not m te , to any o t e o ow ng:

• People attributes

• Environmental factors

• Technology factors

• Tools (physical or software)

• Customers

• Suppliers

• Other Stakeholders




21/89


5) Identify Uncontrollable Factors

• Normally these are constraints placed by the customer orconcrete terms of a contract or government regulation

• Can also be factors for which the project team truly has no

• Can be factors that are unchan in for a iven ro ect butcan be changed for future projects

• Often includes external factors or factors related to otherteams outside of the project

21

Kevin Schaaff, Robert StoddardRusty Young, Dave Zubrow



22/89



Of the steps presented so far, which step do you believewould be the most challen in for our or anization?

1) Business Goal formulation to drive PPMs

2) Identification of critical subprocesses supporting goals

3) Identify outcomes to predict with PPMs en y con ro a e ac ors o ma e pre c ons w

5) Identify uncontrollable factors to make predictions with

22




23/89


6) Measurement and Analysis in Action

5 Decision-making

Corrective actionsto improve process

1

Data collection

4 Data & interpretationsreported

2Data stored

400

500

600

700

800

,interpreted, & stored

MeasurementReport

PrototypeRepository

DataRepository

0

100

200

300

1997 1998 1999 2000 2001E

23



easuremen s co ec e

23David Zubrow , October 2008© 2008 Carnegie Mellon University


24/89


Objective

Indicator Name/TitleDate

EstablishMeasurement

Documenting Measurement

Questions

Visual Display

80

100

Communicate

ect ves

Data Storage

, ,Measures

20

40

60

Perspective

Results

Algorithm

WhereHow

Security

Store Data &

Results

Data Elements

Definitions

Data Collection Specify

pec yMeasures

Interpretation

Assumptions

Probing Questions

Specify Analysis

Procedures

Form(s)

When/How Often

By Whom CollectData

DataCollectionProcedures Evolution

Analysis

Feedback Guidelines

Data

Responsibilityfor Reporting

Data Reporting

By/To Whom

How Often

CommunicateResults

-re erence

24



kaz6


25/89

Slide 24

kaz6 This slide seems out of place to me ... since the slides that follow cover some of the things noted in the template. the template.

Quite honestly, I would remove the slide since it is only repeating some of the criteria that is already being listed.Mark Kasunic, 10/7/2008


26/89


7) Cost of Poor Data Quality to an Enterprise –

Typical Issues

• Inaccurate data [1-5% of data fields are erred]

• Inconsistencies across databases

•

Typical Impacts

• Lowered customer

satisfaction

• Increased cost

• Poorer decision making &

decisions take longer

• More difficult to im lement

• More difficult to set strategy

• More difficult to execute strategy

• Contribute to issues of data

• Lowered employeesatisfaction

data warehouses• More difficult to engineer

• Increased organizational

ownership• Compromise ability to align

organization

25



•ource: e man,


27/89


Impacts of Poor Data Quality

Inability to• manage the quality and performance of software or

•

Estimate and plan realisticallyIneffective

• process change instead of process improvement• and inefficient testing causing issues with time to

market, field quality and development costsProducts that are painful and costly to use within real-lifeusage profiles

Bad Information leading to Bad Decisions

26




28/89


Where do Measurement Errors come From1

Data Entry Errors

– Manual data entr

– Lack of integrity checks

Differing Operational Definitions

– , , ,

completion

Not a priority for those generating or collecting data

– omp e e e e or me s ee a e en o e mon

– Inaccurate measurement at the source

Double Duty

– Effort data collection is for Accounting not Project Management• Overtime is not tracked

• Effort is tracked only to highest level of WBS

27




29/89


8) Types of Data

Examples

Defect types

Labor types

Categorical data where the order of thecategories is arbitraryNominal

Attribute(aka categorized or

anguages

A B C

Examples

Severity levelsNominal data with an ordering; may

have une ual intervalsSurvey choices 1-5

Experience categories< <Increasinginformation

content

ExamplesContinuous data that has equal intervals;

Continuous(aka variables

Ratio

Defect densities

Labor rates

Productivity

Variance %’s0

A B

may have decimal values

Interval data setthat also has

28



data) Code size SLOC


30/89


Appropriate Analysis: Types of Hypothesis Tests

29

Kevin Schaaff, Robert Stoddard

Rusty Young, Dave Zubrow© 2009 Carnegie Mellon University


31/89


9) Creating Process Performance Baselines

• Definition: A Process Performance Baselines (PPB) is adocumented characterization of the actual results achieved byfollowin a rocess

• Therefore a PPB needs to reflect actual project performance• CMMI-DEV OPP PA informative material:

• Establish a quantitative understanding of the performance of theorganization’s set of standard processes in support of objectives

• Select the processes that summarize the actual performance ofprocesses in projects in the organization

• Alternatively Practical Software and Systems MeasurementPSM recommends an or anization follow three basic ste s:

• Identify organization needs

• Select appropriate measures

30



• n egra e measuremen n o e process

SEI W bi (28 A 09)


32/89


Creating Process Performance Baselines

• We only need one baseline

•

• One data point constitutes a baseline• We can’t use the baseline until it is stable

• If the initial baseline is unstable we just remove the datapoints outside of the control limits and recompute the control

31



SEI W bi (28 A 09)


33/89


10) Select the Proper Analytical Model

Statistical Modeling and Regression Equations

Monte Carlo Simulation

Probabilistic Modeling including Bayesian Belief Networks

Discrete Event Process Simulation

er vance o e ng ec n quesMarkov, Petri-net, Neural Nets, Systems Dynamics

32





34/89


Statistical Regression Analysis

Y

ANOVA Chi-Square,

e t e

an ummy

Variable

og

Logistic D i s c

egress on egress on

Correlation i n u o u

& LinearRegression

Regression C o n

33





35/89


Why Use Monte Carlo Simulation?

Use Monte Carlo simulation to do the following:

•

values instead of single value)

• Enable more accurate sensitivity analysis

•

(e.g., more realistic)

•

Aid buy-in and acceptance of modeling because user-provided

• Provide a basis for confidence in a model output (e.g., supports risk

management)

•

Increase the usefulness of the model in predicting outcomes

34




36/89



37/89


Example: Adding Reality to Schedules-1

Process Durations

Step Best Expected Worst

2 45 50 1253 72 80 200

4 45 50 125

5 81 90 225

6 23 25 63

7 32 35 88

8 41 45 113

10 23 25 63

500

What would you forecast theschedule duration to be?

36





38/89

( p )

Adding Reality to Schedules-2

With 90% confidence,The ro ect is almost

under 817 daysduration.

guaranteed to missthe 500 days duration

100% of the time.

37





39/89

( p )

Adding Reality to Schedules-3

With only 50%conf idence, the projectwil l be under 731 daysduration.

38




40/89



41/89

Create Predictions with Both Confidence and

-

Confidence Intervals: The statistical range of behavior of aan average va ue compu e rom a samp e o u ure a a

points

Prediction Intervals: The statistical range of behavior ofindividual future data points

Note: Prediction Intervals are almost always much wider thanconfidence intervals because averages don’t experience the wide

swings that individual data points can experience (similar to howindividual grades in college compared to your grade point average)

40





42/89


Based on what you now know, which analytical modelingtechni ue do ou believe would be most ractical and usefulin your organization?

tat st ca regress on equat ons

2) Monte Carlo simulation

4) Discrete Event Simulation

5 Other

41




12) St ti t i ll M S b /PPM


43/89

12) Statistically Manage Subprocesses w/PPMs

Review performance and capabili ty ofstatistically managed subprocesses for

progress to achieving objectives

QPM SP 2.2

QPM SP 2.3

OnDetermine actions needed totrack

Yes

No

Meet interimobjectives

address deficiencies such as change objectives

CAR (improve processreduce common cause,change mean etc.)

ado t new rocess ando

Suppliers on

technology(OID)

Revise PPM ,predict ifchanged wil l meet objectives

Yes

Yes

Updated PPM prediction

Yes

oMeetobjectives

No

Note: This isnot meant to be

anim lementat ion

trackYes

No

Identify and manage issuesand risks

YesTerminate No

End of

objectives

No

flowchart

42





44/89

13) Take Action Based on Results of PPM

If a PPM model predicts an unacceptable range of values,

more desirable range of outcome

Once a PPM model predicts a range of values for aparticular outcome, then actual values can be compared to

. ,be treated similarly to a point on a control chart fallingoutside of the control limits

Use PPM predictions to help inform process composition

43





45/89

What is Sub-optimization and how can PPMs

Sub-optimization is where one parameter is optimized at theex ense of other s

• Reduce delivered defects, but are late and over budget

• Meet the cost goal but don’t deliver desired functionality

• Gage the trade-offs amongst multiple goals

• Gage the effects of changes to multiple parameters

44





46/89

14) Validating and Maintaining PPMs

Initial estimation of a PPM typically yields

independent variables (x’s) and the dependent variable (y)

• An indication of the goodness-of-fit of the model to the data (e.g.,- -,

These do not necessarily indicate whether the modelprov es su c en prac ca va ue

• Track and compare predictions with actual results

• Failure to meet business criteria e. ., +/- 10% indicates need to

recalibrate (i.e, same variables with different data) or remodel (newvariables and data)

45





47/89

15) How PPMs Assist CAR

• Aid impact, benefit, and ROI predictions for

• Selectin defects for anal sis

• Selecting action proposals for implementation

• Use PPMs to identify potential sources of the problem or

• Use PPMs to understand the interactions among selected

im rovements; and the combined redicted im acts, costs,and benefits of the improvements (considered as a set)

• Compare the result versus the original PPM-based prediction

46



*The result of using a PPM to make a predict ion often takes

the form of a prediction interval as well as a single point .



48/89

How PPMs Assist OID

• Select process improvement proposals for implementationb aidin im act, benefit, and ROI redictions

• Identify opportunities for improvement

• Use PPMs to understand the interactions among selectedmprovements; an t e com ne pre cte mpacts, costs,and benefits of the improvements (considered as a set)

• • Confirm the prediction (provides input to maintaining PPMs)

47




Q #


49/89


Based on the information you saw in this mini-tutorial, wouldou be interested in attendin a full da tutorial on this

subject?

es

2) No

48





50/89

Questions?49




Contact Information


51/89

Kevin Schaaff U.S. mail:

Email: [email protected]

Robert W. Stoddard


Software Engineering Institute

Customer Relations

Rawdon Young


Dave Zubrow

Pittsburgh, PA 15213-2612

USA


World Wide Web: Customer Relationswww.sei.cmu.edu

www.sei.cmu.edu/contact.html


Telephone: +1 412-268-5800. . .

sema/presentations.html SEI Phone: +1 412-268-5800SEI Fax: +1 412-268-6257

50




52/89

Backup Slides


All Models (Qualitative and Quantitative)


53/89

, ,

Statist ical or Probabilist ic Models

Anecdotal

Controllable x factors involved

samples

No

uncertaintyor variationmodeledOnly final

QQual

Model -With controllable x

factors tied to

outcomesaremodeledOnly

uncontrollable

factors are

Processes and/orSub-processes

modeledOnly phasesor lifecyclesare modeled



How are PPM Used? (OPP SP 1 5)


54/89

How are PPM Used? (OPP SP 1.5)

The PPMs are used as follows:

• The or anization uses them for estimatin anal zin and redictin the process performance associated with the processes in the

organization’s set of standard processes• The or anization uses them to assess the otential return on

investment for process improvement activities

• Projects use them for estimating, analyzing, and predicting the

process performance for their defined processes• Projects use them for selecting processes or subprocesses for use

•

more information about the use of PPMs

53




How are PPMs Used? (QPM)


55/89

How are PPMs Used? (QPM)

SP 1.4

• PPMs calibrated with obtained measures of critical attributes to estimate progress toward achieving the project’s quality and

process-performance objectives• PPMs are used to estimate ro ress toward achievin ob ectives

that cannot be measured until a future phase in the project lifecycle

SP 2.2

,establishing trial natural bounds are sometimes available from priorinstances of the subprocess or comparable subprocesses, process-erformance baselines, or PPMs

54




How are PPMs used? (OID)


56/89

How are PPMs used? (OID)

PPMs are used to quantify the impact and benefits ofinnovations.

SP 1.1

• PPMs provide insight into the effect of process changes on process

SP 1.2

•

PPMs can provide a basis for analyzing possible effects of changesto process elements

55




Examples of Controllable People x factors


57/89

Examples of Controllable People x factors

TraitsInterruptions

Communication Mechanisms

Nature of Leadership

56




Example of Controllable Environmental x


58/89

Example of Controllable Environmental x

Nature of work facilities

Accomodations for specific needs

Degree of Security Classification

57




Example of Controllable Technology x Factors


59/89

Example of Controllable Technology x Factors

Mature tools

Availability of equipment, test stations Availabilit of Technolo

Newness of TechnoloProgramming Language Used

Technology TrendsTechnology Roadmap

58




Example of Controllable Process x Factors


60/89

p

Efficiency of a work task

Com liance of a work task

ua y o ar ac s(Input to or Output from

a work task)

Resolution time of technical inquiries

Quality of a work task

Timeliness of a work task

Timeliness of Artifacts

Task Interdependence

Difficult of a work task

Readability of Arti facts

An of the criteria for

Measures of bureaucracy

Resource contention between tasks

Number of people involved with a work task

Degree of Job Aids, Templates, Instructions

good reqts statements

Any of the criteria for goo es gns

Code measuresStatic and D namic

Peer Review Measures

Test Coverage

Choices of subprocesses

Modifications to how work

59



Measures Tasks are performed


Example of Controllable Customer, Supplier


61/89

Example of Controllable Customer, Supplier

“ Maturity” assessment

Health of relationship

Degree of communication

Degree of Documentation

of Expectations

Speed of feedback loops

Trust

mage an ercep ons

Complexity of relationship

such as simultaneousl a

Degree of partnership, collaboration

competitor and partner and supplier Style

Culture

60



ra eo s, omprom ses, p m za on


Criteria for Evaluation: Measurement Planning


62/89

Criteria for Evaluation: Measurement Planning

1

Measurement Objectives and Alignment

• business and project objectives

• prioritized information needs and how they link to the business,organizational, regulatory, product and/or project objectives

• necessary organizational and/or software process changes to implement

the measurement plan

•

criteria for the evaluation of the measurement process and qualityassurance activities

• schedule and responsibilities for the implementation of measurement plan

including pilots and organizational unit wide implementation

Adapted from ISO 15939.

61




Measurement Planning Criteria2


63/89

g 2

Measurement Process

•

• Responsibility for data collection and sources of data

• Schedule for data collection (e.g., at the end of each inspection, monthly)

• Tools and procedures for data collection

• Data storage

•

Requirements for data validation and verification procedures• Confidentiality constraints on the data and information products, and

actions/precautions necessary to ensure confidentiality

• Procedures for configuration management of data, measurement experiencease, an a a e n ons

• Data analysis plan including frequency of analysis and reporting

Adapted from ISO 15939.

62




Criteria for Evaluation: Measurement Processes


64/89

•

Availability and accessibility of the measurement process andrelated procedures

• e ne respons y or per ormance

• Expected outputs

• Interfaces to other processes

– a a co ec on may e n egra e n o o er processes

• Are resources for implementation provided and appropriate

• Is training and help available?

• s e p an sync ron ze w e pro ec p an or o er organ za ona

plans?

63




65/89


Criteria for Evaluation: Data Collection


66/89

Is implementation of data collection consistent with definitions?

Reliabilit of data collection actual behavior of collectors

Reliability of instrumentation (manual/automated)

Ease/cost of collecting data

torage

- Raw or summarized

-

- Ease of retrieval

65




Criteria for Evaluation: Data


67/89

Criteria for Evaluation: Data

Quality

• Data integrity and consistency

• u

– Performance variables

– Contextual variables

• ccuracy an va y o co ec e a a

• Timeliness of collected data

•

Precision and reliability (repeatability and reproducibility) ofcollected data

• Are values traceable to their source (meta data collected)

Audits of Collected Data

66




Criteria for Evaluation: Data Analysis


68/89

y

Data used for analysis vs. data collected but not used

Appropriateness of analytical techniques used

- For data type

- For hypothesis or model

Analyses performed vs. reporting requirements

Data checks performed

Assumptions made explicit

67




69/89


Interquartile Range: Example


70/89

333

5012

Upper outlier boundary*

4

40

30

27Q3

Interquartile Range30 – 16 = 14

.

22

20

18

Procedure

1. Determine 1st and 3rd

quarti les of data set: Q1,16

16

13

Q1

3

Q3

2. Calculate the difference:interquartile range or IQR

3. Lower outlier boundar =ower ou er oun ary

16 – 1.5*14 = -5

Q1 – 1.5*IQR

4. Upper outl ier boundary =Q3 + 1.5*IQR

69




Tips About Outliers


71/89

Outliers can be a clue to process understanding

,

•

repair the erroneous data if possible• if it cannot be repaired, delete it

Charts that are particularly effective to flag possible outliersinclude: box plots, distributions, scatter plots, and control

Rescale charts when an outlier reduces visibilit into

variation.Be wary of influence of outliers on linear relationships

70




Modeling and Outliers


72/89

Even if outliers are valid they can distort “typical”

So you might• delete the observations

• recode outliers to be more in line with the expected distribution

• for more information, research robust techniques

In any case, do so with caution

Run our models usin different techni ues to see if the

converge

71




73/89

Programmatic Aspects



Topics


74/89

• ev ew o rocess er ormance o e s s• Technical Process of Building PPMs

• Programmatic Aspects of Building PPMs

• Skills needed to develop PPMs

•

Forming the PPM Development Team• Up front Critical Thinking Needed

• Barriers to Building PPMs

• Documentation needed when building PPMs

• Evidence from the building and usage of PPMs that may help SCAMPI teams

•

73




Skil ls Needed to Develop PPMs


75/89

• Business Acumen

•

• Process Expertise

• Understandin of Measurement and Anal sisTechni ues

• Understanding of Advanced Statistical Techniques

• Understanding of Quantitative Management

74




76/89


Forming the PPM Development Team


77/89

Basic statistical methods

• Using basic statistics to predict outcomes

• ANOVA; regression; multiple regression; chi-square; logistic regression;

•

Hypothesis testing• Design of experiments;

Monte Carlo simulation and optimization

• Using automated “what-if” analysis of uncertain factors and decisions in aspreadsheet

Process simulation

• Modeling process activities with a computer

• Discrete event process simulation

Probabilistic networks

• Using the laws of probability instead of statistics to predict outcomes

76




Topics


78/89




•





•

77




Considerations for Developing Models - 1


79/89

Think Process – what are the x’s and y’s

Be sensitive to scope of application – what works in onese ng may no wor n ano er

,

improvement or risk

78




Considerations for Developing Models - 2


80/89

Three Criteria for Evaluating Models

• Truth : correctly explains and predicts behavior

• Beauty : simple in terms of underlying assumptions and number of

variables, and more broadly applicable

• Justice : implications for action lead to “a better world”

79




Topics


81/89




•





•

80




Barriers to Building PPMs


82/89

Lack of compelling outcomes to predict due to misalignment,

management sponsorship and involvement

- that direct changes in that process or sub-process can helpcause changes in predicted outcomes

Insufficient process and domain knowledge which isnecessary to identify the probable x factors to predict the

Insufficient training and practice with modeling techniques

81




Documentation Needed when Building PPMs-1


83/89

Similar to the existing SEI Indicator Template but with someadditional information content:

1.Identity of associated processes and subprocesses

2.Identity of the outcome measure (y) and the x factors

3.Data type of all outcome (y) and x factors

4.Statistical evidence that the x factors are significant (e.g. p

5.Statistical evidence of the strength of the model (e.g. theadjusted R-squared value)

6.The actual prediction equation for the outcome (y)7.The performance baselines of the x factors

82




Documentation Needed when Building PPMs-2


84/89

Similar to the existing SEI Indicator Template but with someadditional information content continued :

8.The resulting confidence interval of the predicted outcome

9.The resulting prediction interval of the predicted outcome

10.Use case scenarios of how the PPM is intended to be usedby different audiences for specific decisions

. , ,and calibrated

12.Description of how often the PPM is used to makepredictions with results shown to decision-makers

13.Description of which organizational segment of projects the

83




Topics•


85/89

ev ew o rocess er ormance o e s s• Technical Process of Building PPMs



•





•

84




Its not the Data/Chart, it is How it is Used


86/89

wa u o con ro c ar s oes no ma e a eve• Who is using them for management

• How are they using them

• How timely is the use

– Retrospective vs. real-time

– As the events occur vs. end-of-phase

s ng na ura oun s

• Natural bounds vs. trial bounds

• Natural bounds vs. specification limits

( ----

-

--- )

LimitsInterval

PPM

85




Evidence for Establish


87/89

Statistical analysis leading to which controllable anduncontrollable factors are selected

• ANOVA or any of the other basic statistical methods discussed

• Monte Carlo/Discrete Event simulation calibration vs past

• Hypothesis tests

• p values, R-squared, etc.

ags• Apparent p value chasing

• Inordinatel hi h R-s uared

Awareness• When unstable data used

86



• ource o a a


Evidence for Maintain


88/89

Similar to establish, but analyzing changes to recalibratemodels

Rules for when models are recalibrated

• Process changes

• rocess r t

• New platforms, domains, etc.

• Voice of the Customer

• Changing business needs

87




Evidence for Usage


89/89

Composing projects defined process

behavior

Usage for evaluating alternative solutions whenprocess pro ect per ormance na equate to meetgoals/objectives

opportunities/causes

Usage to check if predicted performance is being achievedan no , w y

88



A Mini-Tutorial for Building CMMI Process Performance Models_1

Documents

Transcript of A Mini-Tutorial for Building CMMI Process Performance Models_1