Strengthening CMMI Maturity Levels with a Quantitative Approach to Root-Cause Analysis

www.eng.it SMEF 2008 – L.Buglione © 2008

Strengthening CMMI Maturity Levels with a Quantitative Approach to Root-Cause Analysis

5° Software Measurement European Forum:

Milan, 28-30 May 2008

Luigi BuglioneEngineering.it / ETS Montréal

[email protected]

mailto:[email protected]

SMEF 2008 – L.Buglione © 2008 2www.eng.it

GoalsGoals: G1. Discuss and analyze the opportunity to approach RCA in a quantitative manner G2. Using Q-RCA in a visual, more appealing manner using Mind Maps, in order to overcome possible communication problems G3. Show some possible usages of the “visual Q-RCA” with a discussion of expected benefits


◆ Introduction

A short summary about RCA

◆ Root-Cause Analysis (RCA) and Process Improvement

Causal Analysis & Resolution and CMMI: high or low maturity process?

Impacts on Process Rating and subsequent improvement actions

◆ Joining RCA and ‘Open’ Measurement

Measuring RCA by ODC: some limitations

Extending the measurable RCA scope of application

From GQM to ISO 15939 Measurement Information Model (MIM)

◆ Visualizing and Communicating RCA

SPI models and the Communication process

Fishbone diagrams: some problems and limitations

Mind Maps: origins & perspectives

Possible Applications of Q-RCA using Mind Maps

◆ Conclusions & Prospects

AgendaAgenda


IntroductionIntroductionSome short notes on RCA

• From Staged towards Continuous representations in Maturity Models broader awareness about the path towards a ‘real’ process improvement appraisal of each single process of interest for an organization

• Most known and applied MM and RCA CMMI has both representations ISO/IEC 15504 (aka SPICE) has only the continuous representation

• Root-Cause Analysis (RCA) and Maturity Levels (ML) CMMI has a ML5 process area (CAR – Causal Analysis & Resolution) ISO 9001:2000 asks for RCA as a basic requirement to be satisfied Approx. equivalence between ISO 9001 certified companies and CMMI ML2-3 Discrepancy among the RCA’s positioning in CMMI and ISO 9001

• Possible solutions Adopt a continuous representation in MM (no influence of staged view on PI) Shift down CAR-related assets/issues across levels:

− Move CAR at ML2 in MM (re-alignment of assumptions for proper appraisals)− Move GP5.2 to GP2.x (being a basic mechanism to evolve towards higher CL)


IntroductionIntroductionSome short notes on RCA

• Q: RCA is a typical qualitative TQM tool. Is it possible to evolve it as a quantitative tool?

Make RCA more quantitative, linking it to organizational PAL’s− PAL (Process Asset Libraries)− Those ones typically demanded in CMMI’s OPD/OPF process areas

Provide some tips for spreading its use in organizations− Communication− Visual aids− …

• A: possible solutions


◆ Introduction















AgendaAgenda


RCA & Process ImprovementRCA & Process ImprovementCAR and CMMI: high or low maturity process?

• RCA and Total Quality Management (TQM) RCA is one of the ‘old’ TQM tools (more qualitative) aka ‘Fishbone’ or ‘Ishikawa’ diagram Objective: to derive, in a back-forward manner, related causes for an effect Visually represented as a ‘fishbone’

Elab

orat

ion

fro

m:

Lesz

ak M

., P

erry

D.E

. &

Sto

ll D

., A

Cas

e St

udy

in R

oot

Caus

e D

efec

t An

alys

is,

Proc

eedi

ngs

of t

he 2

2nd

Inte

rnat

iona

l Con

fere

nce

on S

oftw

are

Engi

neer

ing

(ICS

E 20

00),

Lim

eric

k (I

rela

nd),

Jun

e 4-

11 2

000,

ISB

N 1

5811

3206

9, p

p.42

8-43

7


RCA & Process ImprovementRCA & Process ImprovementCAR and CMMI: high or low maturity process?

• Nature of the tool Core analysis technique within an improvement path It summarizes more TQM techniques into one

− brainstorming, affinity diagram, Pareto analysis Foundation mechanism for establishing any action plan

• RCA and ML CMMI has a ML5 process area (CAR – Causal Analysis & Resolution) RCA is as a basic requirement to be satisfied for ISO 9001:2000 certification Approx. equivalence between ISO 9001 certified companies and CMMI ML2-3 Discrepancy among the RCA’s positioning in CMMI and ISO 9001 Opened a CR to SEI on it in 2007 partly received in the CMMI-SVC ‘Problem Management’ PA draft (ML2)


RCA & Process ImprovementRCA & Process ImprovementImpacts on Process Rating and subsequent improvement actions

• A: the apparent impact on process and organizational ratings is less than the real one, because:

the current CMMI ‘staged’ path does not reflect a company’s priorities, and CAR is relegated where it will be evaluated by few ones

• Q: Being RCA executed in any project (even if not in a graphical notation) and evincible from any meeting report, shouldn’t it be at ML2?

• Q: if CAR is a basic mechanism for process improvement, shouldn’t it be positioned before statistical control mechanisms (ML4/GP4.x)?

• Q: how can CAR and DAR be two ML apart, being so intimately related?


◆ Introduction















AgendaAgenda


Joining RCA and ‘Open’ MeasurementJoining RCA and ‘Open’ MeasurementMeasuring RCA by ODC: some limitations

• Measuring RCA by ODC ODC: IBM’s Orthogonal Defect Classification (R.Chillarege, 1992) ODC proposed a standard taxonomy of causes for software testing process by the number of defects detected Pros: quantitative usage of RCA Limitations:

− allow benchmarks but applying such taxonomy− not directly covered other SLC phases

http://www.ibm.com/developerworks/rational/library/aug06/gu/index.html


Joining RCA and ‘Open’ MeasurementJoining RCA and ‘Open’ MeasurementExtending the measurable RCA scope of application

• RCA, QM and GQM By analogy, here a comparison with a three-tier schema Addition of a third layer to RCA with measures will allow to quantify causes Q-RCA (Quantitative RCA)

Metric (Measure)(Measure)Measure3Sub-questionSub-CauseSub-Characteristic2

QuestionCauseCharacteristic1

GoalEffectProduct0

GQMRoot-Cause Analysis (RCA)

Quality Model (QM) Level

• Possible advantages with Q-RCA spread the ODC lesson to all the facts of interest in a project less subjectivity in assigning priorities to causes for tasks in action plans stimulate an easier data gathering across projects and in the organization reduce CONQ (Cost of non Quality) in m/l term and optimize COQ (Cost of Quality) easy applicable to all processes


Joining RCA and ‘Open’ MeasurementJoining RCA and ‘Open’ MeasurementExtending the measurable RCA scope of application

• Q: as an intermediate goal, can Q-RCA be made automatic (and easy to be used)?


Joining RCA and ‘Open’ MeasurementJoining RCA and ‘Open’ MeasurementFrom GQM to ISO 15939 MIM

Goal

Question

Metric

Implicitmodels

Q3 Q4Q1 Q2

M1 M2 M3 M4 M5 M6 M7

Def

initi

on

Inte

rpre

tatio

n

• GQM (Goal-Question-Metric)• MIM (Measurement Information Model)


◆ Introduction















AgendaAgenda


Visualizing & Communicating RCAVisualizing & Communicating RCASPI models and the Communication Process: some drawbacks…

• Communication issues in MM Partly addressed, even if fundamental

− CMMI GP2.7 on stakeholders’ involvement

Heavily documented and discussed how communication pitfalls represent a major problem which may make a process improvement initiative unsuccessful

http://www.youtube.com/watch?v=XZKWCuEFze0


Visualizing & Communicating RCAVisualizing & Communicating RCA… and possible solutions

• Possible solutions to reduce resistance to change introduction of organizational improvement models (i.e. IDEAL) and related analysis tools such as RCA, using it also as a communication tool a daily/more current use of RCA will necessarily increase the quality of communication among the different stakeholders Q-RCA can improve the quantitative view on the effects to be analyzed


Visualizing & Communicating RCAVisualizing & Communicating RCAFishbone diagrams: some problems & limitations

• Difficulties to connect with other org. Information Systems• Increased effort for maintenance of RCA charts reduce its adoption

• …which solution to satisfy all these requirements at a time?

• Appraisal of appropriate resources for each process of interest CMMI GP2.3 (Provide Resources)

• Available types free-hand drawing with a presentation tool

− i.e. MS-Powerpoint; OpenOffice Impress, … specific drawing tools

− i.e. RCAXPress; Freemind, …Generic drawing tools with RCA templates

− i.e. MS-Visio; SmartDraw, …


Visualizing & Communicating RCAVisualizing & Communicating RCAMind Maps: origins & perspectives

• Definition: “A mind map is a diagram used to represent words, ideas, tasks or other items linked to and arranged radially around a central key word or idea […]” (from Wikipedia)

• Origins at least from the 300BC on (i.e. Porphyry of Tyros, a noted thinker) more recently proposed by psychologist Tony Buzan

• Perspectives several possible usages, with the common aim to represent concepts grouping them by some criteria brainstorming, creating a hierarchy of concepts, note-taking, etc…

http://en.wikipedia.org/wiki/Mind_map


Visualizing & Communicating RCAVisualizing & Communicating RCAFrom RCA to Mind Maps (1/5)


Visualizing & Communicating RCAVisualizing & Communicating RCAPossible applications of Q-RCA using Mind Maps

Meeting Reports (3 pages)



Within a wiki



At the project closure (‘post-mortem analysis’)

Example taken from: www.scatolepensanti.it

The ‘measure’ bone level

http://www.scatolepensanti.it/


Visualizing & Communicating RCAVisualizing & Communicating RCAMeasurement construct with MIM: an example

Results outside the established threshold require further investigation Decision Criteria

Compute thresholds using values of reworked URs per project and customer Model

UR completeness level Indicator

Divide the number of URs to be reworked per iteration by the total number of URs for the iteration

Measurement Function

Percentage of URs to be reworked per iteration Derived Measure

1. URs2. URs

Unit of Measurement

1. Ratio2. Ratio

Type of Scale

1. Integers from zero to infinity2. Integers from zero to infinity

Scale

1. Objective2. Objective

Type of Measurement Method

1. Count the number of URs to be re-worked in an iteration2. Count the overall number of URs formulated in an iteration

Measurement Method

1. Number of UR to be re-worked per iteration2. Number of URs written per iteration

Base Measures

1. Text of URs2. List of changes to insert in the new, refined version of the URs

Attributes

1. URs produced within the current iteration Relevant Entities

Completeness of URs Measurable Concept

Evaluate the completeness level for a project’s User Requirements (UR) Information Need

See ISO/IEC 15939:2007, Appendix A


◆ Introduction















AgendaAgenda


• One of the foundation processes for continuous improvement is cause-effect analysis

Aka Root-Cause Analysis (RCA) or Ishikawa diagram• Well-known SPI models (i.e. CMMI) positioned RCA at the highest level

ML5 from a process viewpoint (staged view) GP5.2 from an appraisal viewpoint

• Direct effects from appraisal results are… RCA will be not often evaluated An underevaluation of the real capability of an organization It is not aligned with the ISO 9000/9001 principles & requirements

• Possible solutions Place RCA at ML2/GP2.x in SPI models Evolve RCA as a quantitative tool (Q-RCA), based on GQM principles Adopt RCA using a visual approach, using mind maps

• Q-RCA possible advantages More objective evaluation of resources for tasks in the action plan Reduce project stakeholders’ resistance, providing more information Direct linkage of Q-RCA with the adopted information systems

Conclusions Conclusions && Prospects Prospects


……and a bit of ‘and a bit of ‘murphologymurphology’’

• COMMERCE LAW: Excess of analysis causes paralysis

• LAW OF COMMUNICATIONS: The inevitable result of improved and enlarged communications between different

levels in a hierarchy is a vastly increased area of misunderstanding

• FIFTH LAW OF DECISION MAKING: Decisions are justified by the benefits to the organization, but they are MADE by

considering the benefits to the decision-makers

• SECOND LAW OF COMMUNICATION: The information conveyed is less important than the impression


Q & A

Thanks for your attention!Thanks for your attention!

www.eng.it SMEF 2008 – L.Buglione © 2008

Engineering.it S.p.A.Via Riccardo Morandi, 32

I-00148 Rome (Italy)www.eng-it.it

LuigiBuglionet +39 06 83074472

m +39 335 [email protected]

http://www.eng-it.it/

http://www.eng-it.it/

Strengthening CMMI Maturity Levels with a Quantitative Approach to Root-Cause Analysis

Documents

Transcript of Strengthening CMMI Maturity Levels with a Quantitative Approach to Root-Cause Analysis