Identifying and Using a Project’s Key Subprocess Metrics Jeff S. Holmes BTS Fort Worth.
-
Upload
edwin-hutchinson -
Category
Documents
-
view
216 -
download
0
Transcript of Identifying and Using a Project’s Key Subprocess Metrics Jeff S. Holmes BTS Fort Worth.
Identifying and Using a Project’s Key Subprocess
Metrics
Jeff S. HolmesBTS Fort Worth
Everyone Loves a Hero
Heroes Come Through!
Firemen – Saves Baby in Burning House Policemen – Catches Bad Guy Athlete – Hits Game Winning Homerun
Software Engineering Heroes!
All night coding! Debugging over the weekend!
THIS SHOULD NOT BE THE NORM!
How To Minimize “Fire Drills”?
Preventative Actions Proper wiring can prevent fires. Locking your car can prevent theft. Don’t get behind in the ball game. Understand project status earlier.
Metrics, metrics, everywhere…
But What is Really But What is Really Important?Important? Customer WantsCustomer Wants
– FunctionalityFunctionality– Zero DefectsZero Defects– On TimeOn Time
What software What software metrics map to metrics map to these?these?
How can we optimize How can we optimize these outputs?these outputs?
BTS Fort Worth Approach
Selected DMAIC to Improve Process Identified Project with Two Years of Data Performed Statistical Analysis Conducted Pilot Currently in “Control” Phase
DMAIC : Define
Identify “what is important” BTS FW Monitors
Productivity (KLOC/Hour)*
Quality (Post Release Defects/KLOC)Schedule Adherence
These are BTS FW “Big X’s”
DMAIC : Measure
Software Development Life CycleRequirementsResources
Perfect Software!
The “Simple” View
DMAIC : More DetailsRequirements
RequirementsResources
Perfect Software!
Design
Code
Test
Perfect RequirementsResources
Perfect Design Perfect Models
Perfect DesignPerfect ModelsResources
Perfect Code
Perfect CodeResources
DMAIC : Subprocess Identification
BTS FW Identified Following Subprocesses Planning Phase Requirements Phase Design Phase Code Phase Test Phase Release Phase Code Inspections
DMAIC : Measured Data BTS FW Uses Following Data:
# Requirements # Developers on the project (Resources) % Time in Planning % Time in Requirements % Time in Design % Time in Code % Time in Test % Time in Release Requirements Churn Actual Size (KLOC) Avg Defect Detection Rate (DDR) in Code Inspection
DMAIC : Data Sources BTS FW Data
DOORs for Requirements Project Plans # Developers and % Times ClearCase for Code Size Inspection Database for DDR
DMAIC : BTS Subprocess Metrics Planning
Design
Code
Test
Req CountTime in Phase
Req ChurnTime in Phase
Requirements
ResourcesTime in Phase
Req ChurnKLOC
Time in Phase
Time in Phase
Release Time in Phase
Code Inspections DDR
DMAIC : BTS Subprocess Metrics Planning
Design
Code
Test
Req CountTime in Phase
Req ChurnTime in Phase
Requirements
ResourcesTime in Phase
Req ChurnKLOC
Time in Phase
Time in Phase
Release Time in Phase
Code Inspections DDR
Productivity (KLOC/Hour)
Quality (PR Defects/KLOC)
Schedule Adherence
?
?
?
?
?
DMAIC : BTS FW Analysis
Project Data• 8 Releases since 2002
• Similar work
• “Stable” team
Used Step-wise Linear Regression to • Identify statistically significant factors
• Develop prediction formulas for “Big X’s”
CAUTION !!
The following slides contain statistics that could be hazardous to your health!
Persons who suffer from narcolepsy or “statisticitis” should consider leaving the room.
DMAIC : Stepwise Linear Regression
Describes the relationship between one 'predicted' variable and 'predictor' variables
Goal – get the simplest equation with the best predictive power for• Productivity – KLOC/Hour
• Quality – Post Release Defect/KLOC
DMAIC : Standard Least Squares
Model accounts for99.82% of variance.
DMAIC : Significant Effects
Most significant effects• % Time in Req
• Average DDR
• Interaction between % Time in Code and Average DDR
< 0.05 is significant
DMAIC : Standard Least Squares
Model accounts for90.62% of variance.
DMAIC : Significant Effects
Most significant effects• % Time in Requirements
• Interaction between % Time in Requirements and Requirements Churn
< 0.05 is significant
DMAIC : Statistically Significant
# Requirements # Developers on the project % Time in Planning % Time in Requirements % Time in Design % Time in Code % Time in Test % Time in Release Requirements Churn Actual Size (KLOC) Avg Defect Detection Rate (DDR) in Code Inspection
DMAIC : Key Subprocess Metrics
Planning
Design
Code
Test
Req CountTime in Phase
Req ChurnTime in Phase
Requirements
ResourcesTime in Phase
Req ChurnKLOC
Time in Phase
Time in Phase
Release Time in Phase
Code Inspections DDR
Productivity (KLOC/Hour)
Quality (PR Defects/KLOC)
Schedule Adherence
?
?
?
?
?
DMAIC : Variation Analysis
Prediction formulas generated to identify:• Good and bad variance
• Most significant factors
NOTE: Prediction formula uses all effects from the models, not just the significant ones. Formula added in Percent Planning.
DMAIC : Factor WeightingMetric 10% Deviation in Each Subprocess Area
% Plan 10% 0 0 0 0
% Req 0 10% 0 0 0
% Code 0 0 10% 0 0
Req Churn 0 0 0 10% 0
Avg DDR 0 0 0 0 10%
Productivity 0 3% 31% 0 9%
Quality 14% 40% 0 6% 0
Cycle Time 0 3% 24% 0 9%
DMAIC : Factors’ Effects
Metric LOC/Hr Effect PR Defect/KLOC Effect
% Planning Time N/A More time = Less defects/KLOC
% Req Time More Time = Less KLOC/Hr
Less time = More defects/KLOC
% Code Time More Time = Less KLOC/Hr
N/A
Code Insp. DDR Higher DDR = Less KLOC/Hr
N/A
2SR Req. Churn N/A Less Churn = Less defects/KLOC
DMAIC : BTS FW Limits
Green limit indicates direction a metric can deviatefrom the average and have desired results.Red indicates direction of undesired results.
Metric Average Std Dev Lower Limit
Upper Limit
% Planning Time 18 % 8 % 10 % 26 %
% Req Time 15 % 8 % 7 % 23 %
% Code Time 21 % 12 % 11 % 33 %
Code Insp. DDR 1.89 .5 1.39 2.39
2SR Req. Churn 1.76 1.89 0 3.65
Subprocess Metrics Notes
Initial Data Left Much to be “Desired” Despite Poor Data, the Analysis
Identified:• Which Metrics and Processes Are Significant
• Prediction Formulas Based on Project’s Data
• Insight into Factors’ Effects
• Limits for Monitoring the Factors
DMAIC : Pilot Confirmation
Used prediction formulas on other projects Compared project actuals vs. predicted. Used historical data from 5 projects. Unable to compare predicted quality versus
actual. (Predicted LOC/Hr)• These projects have not been in field long enough for
CRUD to stabilize.
Interesting results found on predicted LOC/Hr.
Project A Project B Project C Project D Project E% Plan 0% 18% -65% 10% 17%% Req -28% 10% -97% 4% 17%% Code -348% 20% 0% 43% 25%Inspection DDR 1.2 1.2 0 1.51 1.89Requirements Churn-2SR 1 1.81 0 0 0Pred LOC/Hr 92.673688 4.789660 25.540537 2.288774 2.418825Pred PR Def/KLOC 7.7196696 1.33169549 28.1019536 3.39281 0.4010827
Actual LOC/Hr 1.21 1.79 0.829 2.039 1.945LOC/Hr Delta 99% 63% 97% 11% 20%
DMAIC : Predicted vs. Actual LOC/Hr
•Projects A, B, and C projects had huge deviations.• Projects D and E were within 20%.
DMAIC : Improve Performance?
So what? How do you use this information? Does Project Management have confidence
in this analysis?
More emphasis on statistically significant activities
Resulting in Increased Productivity
On-Time Delivery Desired Functionality Delivered
Improved Quality
DMAIC : Applying Analysis
DMAIC : Agile Processes
BTS FW Adopted Agile Practices Iterative Development
Prioritizes Requirements Negates Requirements Churn
Pair Programming Optimizes Coding and Inspection Time
Minimal Documentation Moves effort from non-statistical activities.
DMAIC : Agile Pilot Results
Productivity 0.00291 KLOC/Hr 20% improvement from 0.002399
Inspection Defect Detection Rate 1.18 Defects/Hr Detected 48% improvement from 0.8
Quality 0 Post Release Defects!
DMAIC : Agile Pilot Results
Customer Wants Functionality – All functionality delivered Zero Defects – No customer found defects On Time – Product delivered 6 months early!
DMAIC : Agile Monitoring
Monitor Iterations Not Phases Refactoring Subprocess Monitoring Two Agile projects in-work now
DMAIC : Agile Monitoring
Monitoring LOC per week Defects caught per week by inspection Defects caught per week by test Time spent per week Ratio of new work to correction work.
DMAIC : Agile Metrics% Fault Work
0%
2%
4%
6%
8%
10%
12%
14%
16%
18%
20%
7/31/2005 8/7/2005 8/14/2005 8/21/2005 8/28/2005
SummarySummary
Save your “heroes” for real crises. Save your “heroes” for real crises. Understand subprocesses Understand subprocesses Monitor subprocessesMonitor subprocesses Seek to optimize key subprocessesSeek to optimize key subprocesses
RecommendationsRecommendations
Examine current project data, it Examine current project data, it could prove to be very valuable!could prove to be very valuable!
Improve data capture on important Improve data capture on important data.data.
Use the data as a guideline, but Use the data as a guideline, but experience can never be discounted.experience can never be discounted.
THANK YOU!THANK YOU!
Jeff S. HolmesJeff S. HolmesPrincipal Staff Software EngineerPrincipal Staff Software Engineer
Motorola Six Sigma Black BeltMotorola Six Sigma Black Belt
Fort Worth BTS Development TeamFort Worth BTS Development Team
Fort Worth, TXFort Worth, TX
817-245-7053817-245-7053
[email protected]@Motorola.com