Performance Effect of CMMI Based Process Improvement in ... · Definition Project Process Product...

Motorola Document Classification, File Name, Rev NumberAdd additional legal text here if required by your local Legal Counsel.MOTOROLA and the Stylized M Logo are registered in the US Patent & Trademark Office. All other product or service names are the property of their respective owners. © Motorola, Inc. 2005

Motorola Confidential Proprietary MOTOROLA and the Stylized M Logo are registered in the US Patent & Trademark Office.All other product or service names are the property of their respective owners. © Motorola, Inc. 2005.

Performance Effect of CMMI Based Process Improvement in Motorola

Angel Liu

Motorola Software Group, China

Abstract Reference: Abstract ID: #3745)


Why to Measure

How to Measure

What to Measure

What to Be Measured

What Changes to Lead the Result

Performance Result & Benefit

Performance Effect of CMMI Based Process Improvement


Why it is important to measure the CMMI Performance Result?

We are often questioned about the cost & return of CMMI-based process improvement?

Sponsor - Why sponsor? What is the value to business?

Project Manager – Why process? My customer only care about the schedule

Engineer - Why measure? Does it measure my performance?

Why to Measure

How to Measure

What to Measure

What to Be Measured




Why it is important to measure CMMI Performance Result?

Building continuous commitment and sponsorship

Understanding the value of CMMI to Motorola business

Process Improvement is oriented by business and is expected to benefit businessBenchmarking with other

organizationsSet up the baseline for continuous

improvement

Why to Measure

How to Measure

What to Measure

What to Be Measured





Why to Measure

How to Measure

What to Measure

What to Be Measured




Collect GSG Business

Needs in 2006

Investigate GSG Site

PCB & 2005 YBL

How to Measure

Measure Performance in Business Context

Understand what are the BUSINESS NEEDSAnalyze what are the business goal of CUSTOMERAnalyze what’s PERFORMANCE & Quality TrendBenchmark and know what’s our Position Calibrate the analysis result and setup business goalKnow our performance RESULT

Measure

Set BusinessGoal

Performance Result


Project Level FocusGlobal Metrics-CCB & Site Metrics-CCB

Process Assets and Standards

Measurement System Overview

GSG

Process Data Repository EIQMEn

Q Data & SQAP Template

SW Metrics Spec & Data Collection Guideline

MA/QPM/OPP process &MA/QPM guideline

Process Model Usage Guideline

Metrics Automation Tools

M-CCB Mechanism

Org. Data Audit Team

PQ Monthly Review

Site Level Focus

Org Data Audit Mechanism(Org Audit + SQE Audit +

Tool Validation)

Goal Setting (SQAP)

Post-mortem/Causal Analysis

Monthly Project Review

Quality Trend Analysis & POR Mechanism

Lessons Learned , POR

Yearly Baseline Mechanism(YBL/PCB)

Metrics Core Team

Yearly Goal Setting & Monthly Tracking Mechanism


Metrics Dictionary

Review/InspectionTesting

Risk ManagementRM

PP/PMC

Question

Measure

Goal

Measurem

ent and analysis requirem

ents

Project MetricsOrganizational Metrics

SW Metrics Specification

Description

Question

Formulas

Chart

Purpose

Definition

ProjectProcess

Product

View2

View3

Project Data Collection Guideline

Metrics Analysis Template & Guideline

QPM Guideline

Supporting Artifacts

View1


Risk ManagementRM

PP/PMC


Risk ManagementRM

PP/PMC

Question

Measure

GoalQuestion

Measure

Goal

Measurem

ent and analysis requirem

ents




Description

Question

Formulas

Chart

Purpose

Definition


Description

Question

Formulas

Chart

Purpose

Definition

Description

Question

Formulas

Chart

Purpose

Definition

ProjectProcess

Product

ProjectProcess

Product

View2

View3

Project Data Collection Guideline

Metrics Analysis Template & Guideline

QPM Guideline

Supporting Artifacts

View1

Define Common Measures of Performance

Business Needs &Value

Measures of CMMI PAProductProject


GSG Process Measurement Environment

Global Central Metrics Database

Project Metrics Data

Business Partner Environment

Beijing Team

••••

Manage By Data

Australia Dev Team

PMBOOK

Defect Tracking

EPMS Proj Mgmt

Inspection/Review DB

Change Mgmt

Asset Characterization

Center Goals/Baselines

Project Metrics Database Center Metrics Database GSG Metrics Database

Project Record

SMSC SMSCSite Metrics Database

Why Measure How Measure What Measure Result of Measure


Why to Measure

How to Measure

What to Be Measured





Process Initiatives & Benefit Analysis

BG1 BG2 BG3 BG4 BG5 BG6 BG7 BG8

PI-1 (PP)

PI-2 (RSK)

PI-3 (RM)

PI-4 (CAR)

PI-5 (MA)

PI-6 (OID)

PI-7 (VER-PR)

PI-8 (VER-Test)

PI-9 (OPP)

PI-10(OID-2)

Process Initiatives : All major initiatives will have ROI analysis in POR System

JProductivityJScheduleJCost of QualityJCost of Poor QualityJCycle TimeJROIJOn Time DeliveryJCustomer Satisfaction

BusinessGoal & Benefit


Organization Performance TrackingChart 9: Productivity

0.001.002.003.004.005.006.007.008.009.00

10.00

1Q02 2Q02 3Q02 4Q02 1Q03 2Q03 3Q03 4Q03 1Q04 2Q04 3Q04 4Q04

KA

ELO

C /

staf

f-mon

thProductivity

Chart 3: Cost of Quality

0%

10%

20%

30%

40%

50%

60%

1Q01

3Q01

1Q02

3Q02

1Q03

3Q03

1Q04

3Q04

%of

Tot

al E

ffort

External Failure Ef f ort % Internal Failure Ef f ort %Prevention Ef f ort % Appraisal Ef f ort %% COQ Goal % COPQ Goal

Chart 4: In-Process Faults for Development & Test

0.001

0.01

0.1

1

10

100

1000

1Q98 1Q99 1Q00 1Q01 1Q02 1Q03 1Q04

FDR

C D

evel

opm

ent &

Test

Test nIPF/Test DKNCSS FDRC Test Goal Dev nIPF/Prod DKNCSS FDRC Prod Goal IPF - KAELOC

↓ Down Indicates Improvement

Chart 2: Post-Release Defects forDevelopment & Test

0.000

0.001

0.010

0.100

1.000

10.000

100.000

1Q98

3Q98

1Q99

3Q99

1Q00

3Q00

1Q01

3Q01

1Q02

3Q02

1Q03

3Q03

1Q04

3Q04

DD

RC

Dev

elop

men

t & T

est

Test nPRD/Test DKNCSS DDRC Test Goal Dev nPRD/Prod DKNCSS DDRC Prod Goal PRD - KAELOC

Chart 7: Project Estimation, Planning &Control

0

20

40

60

80

100

1Q02

2Q02

3Q02

4Q02

1Q03

2Q03

3Q03

4Q03

1Q04

2Q04

3Q04

4Q04

Perc

enta

ge

% Projects On-Time Projects-On-Time Goal LinePEEA PSEAPSEA Outbreak PEEA OutbreakPZEA Outbreak

GSG Software Metrics Summary Chart (SMSC)



Project Performance Dashboard

Measure the Performance at different level

Phase-wise Fault Distribution

16%

35%

49%

Req Design Code

Req Fault Distribution

0

1

2

3

4

5

6

A5 A3 A2 A4 A1 A60.00%

20.00%

40.00%

60.00%

80.00%

100.00%

120.00%

Design Fault Distribution

0123456789

10

B4 B3 B6 B2 B5 B10.00%

20.00%

40.00%

60.00%

80.00%

100.00%

120.00%

Code Fault Distribution

0

2

4

6

8

10

12

C1 C3 C4 C2 C5 C6 C9 C10 C70.00%

20.00%

40.00%

60.00%

80.00%

100.00%

120.00%

Cumulative PR 1-2 Severity**

0

5

10

15

20

25

30

35

40

45

8/10/2001 8/17/2001 8/24/2001 8/31/2001 9/7/2001 9/14/2001

# of

cum

ulat

ive

PR

0

1

2

3

4

5

6

7

8

# of

ope

n P

R

# of Open PR # of Cumulative Arrival PR # of Cumulative Closed PR

Cumulative PR 3-4 Severity**

0

10

20

30

40

50

60

70

8/10/2001 8/17/2001 8/24/2001 8/31/2001 9/7/2001 9/14/2001

# of

cum

ulat

ive

PR

0

2

4

6

8

10

12

14

16

18

20

# of

ope

n P

R

# of Open PR # of Cumulative Arrival PR # of Cumulative Closed PR

% Surveys achieving High Importance factors

77%86%

86%81%

0%10%20%30%40%50%60%70%80%90%

100%

Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec

% Pass Cumulative % Pass GSG Goal

Customer Overall Satisfaction Trend

9.15 9.04 9.09

0.00

2.00

4.00

6.00

8.00

10.00

1Q20

052Q

2005

3Q20

054Q

2005

YTD

Scor

e

Average Max Min

Current Phase: [ ] Key Metrics Summary Chart As up to: [ mm/dd/yyyy ]Project Category: [ DEV/CLC/POR-DEL/VSP ]

Actual Est.latest Est.lnitial IPF - Total(nIPF/DKNCSS) IPF - MajorEffort(Stf Wks): CurrentCycle Time(Wks): Goal Prod. Size(DKNCSS): Baseline UCLProd. Size(KAELOC): Avg. LCLSPR1(LOC/Staff Hour) COQCurrent Goal Current Goal Baseline UCL Baseline UCLAvg. LCL Avg. LCLSPR4(LOC/Staff Hour) COPQCurrent Goal Current Goal Baseline UCL Baseline UCLAvg. LCL Avg. LCL

Req Design CUT IT STProjected Faults

Actual Faults

PCE ActualPCE Goal

PSE ActualPSE Goal

Phase Detected Statistics

Req

Design

CUT

Phas

e In

trod

uced



-50

0

50

100

150

200

250

300

Pro

duct

ivity

(LO

C/S

taff

Day

)

6 12 18 24 30 36 42 48 54 60 66 72 78

Avg=24.64

LCL=-37.85

UCL=87.12

0.00

0.05

0.10

0.15

0.20

RV

(%)

6 12 18 24 30 36 42 48 54 60 66 72 78

Avg=0.0060

LCL=-0.0243

UCL=0.0364

Process Performance Baseline

-0.4

-0.3

-0.2

-0.1

-0.0

0.1

0.2

EE

A L

ates

t

3 6 9 12 15 18 21 24 27 30 33

Avg=-0.106

LCL=-0.368

UCL=0.156

-10

0

10

20

30

40

50

60

70

80

Det

ectio

n R

ate

9 18 27 36 45 54 63 72 81 90 99 108 126

Avg=17.45

LCL=-6.17

UCL=41.08

Software Development -Productivity

Project Planning –Estimation Accuracy

Verification –Inspection Process Control

Requirement Management –Requirement Volatility



What new and changed process led to the performance result in GSG?

Why to Measure

How to Measure

What to Be Measured




Process Architecture and Integrated Project Process

GSG Process Hierarchy StructureAdapted to accommodate process changes due to CMMI.

Tailoring FlexibilityOrganized to assist the projects on process model selection, and provides guidance and flexibility to select and tailor the processes at a detailed sub-process level.

Easy AccessThe process asset library can be accessed by several criteria, including CMMI process areas, phase, process templates, checklists, and other types of documents.


• Understand Scorecard

• COQ,COPQ, Productivity ,Defect Leakage ,Customer Satisfaction, On-Time

Delivery…

• Process Opportunity

Estimation Improvement, Test Process Improvement,etc…

Critical Sub-Process

• Analyze Process Performance

Process Capability Baseline, Yearly Baseline, Benchmark Result….

Identify Project Planning

Project Monitoring &Control

Verification

Requirement Management

Define the Measures Measure Performance Monitor Performance Stabilize Proces

Focus more on critical processes aligned with business objectives, particularly in the areas of requirements management and verification.

Quantitative Project Management Process

sControl Process Improve Process Performance Measure the Benefit Result


Using of High Level Process Area

OIDOPP

CARQPM

SPC method and tool to support baseline and model establishment

1. Provide analysis techniques2. Provide statistical method to understand variation

1. Provide analysis techniques2. Provide statistical method to understand variation

1. Quantitatively identify improvement opportunities2. Cost-benefit analysis3. Measure improvement effect by performance analysis

Techniques &Tools in Digital Six Sigma

POR

Project Management & Engineering

Defects and other problems

Goals, baselines, models

POR

PORProject Data

Using of S

upport Process A

rea

Using of H

igh Level Process A

rea


Support Verification Process Improvement

CAR and OID-based processes were used to improve verification process effectiveness without compromising quality in specific classes of cases.

The formal process rules the Center previously used were less effective for smaller documents and project teams.

Faul

t Den

sity

(Fau

lts/1

00LO

C)

25 50 75 100125150175200225250275300325350

Mean

LCL

UCL Effo

rt Tr

end

(SH

/siz

e)

0Normal Small

Material Size Category

All PairsTukey-Kramer0.05

Normal SizeSmall Size


Inspection Process Improvement

Mini-reviews and mini-inspections were introduced; effort and cost were reduced without affecting defect density. The results were used to establish new baselines for planning and quantitative project management.

O b s e r v a t i o n

Ind

ivid

ua

l V

alu

e

_M e a n

U C L

L C L

A f t e r I m p r o v e m e n t

O b s e r v a t i o n

Mo

vin

g R

an

ge

_ _M R

U C L

L C L

A f t e r I m p r o v e m e n t

11

11

1

11

1

1

1

111

1

1

111

1

11

1

1

1

1

11

1

1

11

1

11

1

F o r m a l I n s p e c t i o n M e e t i n g P e r f o r m a n c e

Before Improvement After Improvement


Enterprise Project Management System

Use Enterprises Project Management System (EPMS) as a common project management platform to manage global team

Use Project Pamphlet as a local supply tool of EPMS for PM performance tracking

Staffing

00.5

11.5

22.5

33.5

44.5

Staffin

g

0

5

10

15

20

25

Total S

taffi

ng

Plan Re-plan ActualAcc Plan Acc Re-plan Acc Actual

Effort

0102030405060708090

Month

Staff d

ays

050100150200250300350400450

Total S

taff

days

Plan Re-plan ActualAcc Plan Acc Actual Acc Re-plan

COQ/COPQ

0.00%

5.00%

10.00%

15.00%

20.00%

25.00%

30.00%

35.00%

40.00%

45.00%

50.00%

Nov-03 Dec-03 Jan-04 Feb-04 Mar-04 Apr-04 May-04 Work Week

Plan COQ

Actual COQ

Plan COPQ

Actual COPQ

COQ Goal

COPQ Goal


RM Improvement is based on guidance in CMMI

Requirement VolatilityLate additions and changes to requirements are measured.

RM Process Performance and ManagementF The project level results then are rolled up to the organizational level for senior management insight FSet control limits for future use in quantitative process management of requirements volatility.

Use of RM Measure ResultThe measured and estimated impact of the requirements changes FLed to new project processes for replanning, estimation, and FA factual basis for negotiation with customers.

Requirement Management Process Improvement

Better Manage the Change Better Project Management Delight the Customer


2003 – Before CMMI 2004/2005 – After CMMI

Improvement Action:

1. RM process

2. IAD template

3. RM TRMX template

4. RV data sheet

R e q u i r e m e n t V o l a t i l i t y - E f f o r t

0 %

1 %

2 %

3 %

4 %

5 %

6 %

7 %

8 %

9 %

1 0 %

1 1 %

1 2 %

1 3 5 7 9 11 13 15 17 19 21 23 25 27 29

R V E f f o r t - C u m u la t iv e %R V T r ig g e rR V E f f o r t - E s t im a t e d %

R e q u i r e m e n t V o l a t i l i t y - E f f o r t

0 %

5 %

1 0 %

1 5 %

1 2 3 4 5 6 7 8 9 10 11 12 13 14

R V E f f o r t - T o t a l % R V T r ig g e r

Requirement Management Process Improvement


Decision Analysis and Resolution

New practices based on the CMMI Decision Analysis and Resolution (DAR) process area were used to improve the Center’s managerial and engineering processes, particularly those that map to Technical Solution, Configuration Management, and OID process areas.

These DAR-based practices have been applied to improve decision making effectiveness at both the organization and project level. Decision ROI Analysis

Project Name: xxx

Criteria >>> CostSchedule Impact Benefit

Customer Satisfaction Performance

Relative Importance >>> 6 10 10 6 4

AlternativesWeightedAverage

A1 3 2 4 7 3 3.7A2 2 6 4 5 3 4.3A3 6 3 3 3 5 3.7A4 1 8 3 2 5 4.1

0 0 0 0 0 0.00 0 0 0 0 0.00 0 0 0 0 0.00 0 0 0 0 0.0

Weighted-Average

Ranking

Weighted-Average

Decision Tree


Causal Analysis Improvement

Shifted its focus on causal analysis processes from the relatively narrow defect prevention perspective in the SW-CMM to the broader problem prevention focus in CMMI


Improve Training Measurement ->Improve Training Cost Effectiveness

RTP Course Effectiveness

3.0

3.5

4.0

4.5

5.0

1 2 3 4 5 6 7 8 9 10 11 12 13 14

Effectiveness USL LSL

5-1. ITP attendance status by roundMar~Jul, 2005

0%10%20%30%40%50%60%70%80%90%

100%

Round1(BJ)

Round2(BJ)

Round2(CD)

Round3(NJ)

Round4(NJ)

Round6

Round7

Round8

Round9

Round10

Round11

Round12

Round13

Round14

attend(section staff) postpone(section staff) waiver(section staff) absent(section staff)

ITP Pass v.s Fail by OperationMar~Jul, 2005

0%10%20%30%40%50%60%70%80%90%

100%

AW452

AW455

AW512

AW513

AW516

AW517

AW518

AW519

AW520

AW521

AW531AF4

55AF5

15AF5

16AF5

17AF5

18AF5

19AF5

20AF5

21AF5

22LG

512LG

516LG

520LG

521LG

603

Fully pass Pass with some absence before Pass with postpone Pass with waiver Not finish

Training - Satisfaction

3.82 3.583.69 3.91 3.723.49

0123456789

10

RequirementGathering

ITP Course DesignedWell

RTP CoursesDesigned Well

Category

Rat

ings

2003Q3 2004Q4

Training - Importance

4 4.01 4.044.15 4.21 4.12

0123456789

10

Requirement Gathering ITP Course DesignedWell

RTP Courses DesignedWell

Category

Rat

ings

2003Q3 2004Q4


Why to Measure

How to Measure

What to Be Measured



What Are the Overall Quantitative Results?

( Before & After using CMMI )


COQ Reduction by one-third from pre-CMMI baseline due to improvement in verification, peer review and software test process.


COQIPFEAPCETCS

Reduction of Cost of Quality


Improvement in Fault Density

IPF Reduction by 40% due to improved Casual Analysis and Resolution (CAR) procedures by implementing real time RCA and by improved verification process


COQIPFEAPCETCS


Improvement in Estimation Accuracies

Improvement in Effort Estimation accuracy by 31% and Schedule duration by 84% - due to improved Project Planning and tracking practices.


COQIPFEAPCETCS


Requirement Phase Containment

Requirements PCE sustained at 90%

in spite of enormous organizational growth and constant requirement churn in number of projects.


COQIPFEAPCETCS


Design Phase Containment

Design PCE improved by 8%

Improvements were due to improved Verification, Technical Solution and Organization Training process areas


COQIPFEAPCETCS


Code Phase Containment

Code PCE improved by 12%

Improvements were due to improved Verification, Technical Solution and Organization Training process areas


COQIPFEAPCETCS


Customer Satisfaction

J Consistently received high TCS scores (9/10) with a sustained 100% on-time delivery rate

J Continued to exceed customers’ expectations to show the ability to deliver cost effective, high quality products on time by providing excellent technical solutions and project execution.


COQIPFEAPCETCS


Business BenefitWhy to Measure

How to Measure

What to Be Measured



F Customer Satisfaction Sustain to exceed customer expectation (+9/10)

F Business Growth GSG Population 2225 (Y2000) - > 6840 (2006)

FEngagement CapabilityModel Level->System Level->Product Level


Why to Measure

How to Measure

What to Be Measured



Thanks

Performance Effect of CMMI Based Process Improvement in ... · Definition Project Process Product...

Documents

Transcript of Performance Effect of CMMI Based Process Improvement in ... · Definition Project Process Product...