Lessons Learned from Forty-five Years of Software Measurement

23
BT4 Concurrent Session 11/8/2012 10:15 AM "Lessons Learned from Forty-five Years of Software Measurement" Presented by: Edward Weller Integrated Productivity Solutions, LLC Brought to you by: 340 Corporate Way, Suite 300, Orange Park, FL 32073 8882688770 9042780524 [email protected] www.sqe.com

description

Counting is easy. However, what makes measurement really valuable-and really hard to get right-is knowing what to count and what to do with the results. If your organization is mostly tracking resource usage, costs, and schedule data, it is making a big mistake. What about the users? The customers? The overall business strategy? Sharing the lessons he has learned from fighting-and surviving-many software measurement battles, Ed Weller offers a step-by-step approach for implementing a practical and valuable metrics program. After understanding what measures are most important to the business strategy and all stakeholders, the next step is to decide what data supports those measures and how to capture it. With data in hand, you can create simple and informative ways to make the resulting metrics visible and easy to digest. The biggest challenges-avoidance, disbelief, and rationalization-come next. To overcome these challenges, Ed advises you be pro-active, patient, and persistent. In the end, your organization can have a metrics program that really works.

Transcript of Lessons Learned from Forty-five Years of Software Measurement

Page 1: Lessons Learned from Forty-five Years of Software Measurement

 

    

BT4 Concurrent Session 11/8/2012 10:15 AM 

       

"Lessons Learned from Forty-five Years of Software Measurement"

   

Presented by:

Edward Weller Integrated Productivity Solutions, LLC

       

Brought to you by:  

  

340 Corporate Way, Suite 300, Orange Park, FL 32073 888‐268‐8770 ∙ 904‐278‐0524 ∙ [email protected] ∙ www.sqe.com

Page 2: Lessons Learned from Forty-five Years of Software Measurement

Ed Weller Integrated Productivity Solutions, LLC

Ed Weller is the principal in Integrated Productivity Solutions, providing solutions to companies seeking to improve productivity. Ed is internationally recognized as an expert in software engineering and in particular software measurement. His focus on quality started with his work on the Apollo program with General Electric; was reinforced during his work as a hardware, test, software, and systems engineer and manager on mainframe systems at Honeywell and Groupe Bull; and continued as the process group manager on the Motorola Iridium Project. During the past fourteen years, Ed has worked to improve the quality and productivity in small to very large companies that develop engineering and IT applications.

Page 3: Lessons Learned from Forty-five Years of Software Measurement

Ed WellerIntegrated Productivity Solutions

Improving Business ResultsImproving Business Results

1

First Job - Apollo Program ◦ Measured signal strength of the Control Center to g g

Crawler Transporter communications link◦ Ditto for Fueling Teams for Lunar ModuleAs a hardware design engineer, measuring and counting was a normal part of our daily workThese useful habits carried over to myThese useful habits carried over to my software test and development work

2

Page 4: Lessons Learned from Forty-five Years of Software Measurement

I‘ve learned a lotI haven’t learned enougha e t ea ed e ougI keep learningWe are in competitive businesses◦ Survival of the fittest

Profitability driven by products customers want withOn time deliveryOn budgetOn budgetSatisfactory or superior quality

◦ We do not know the answers to the three characteristics unless we measure them

3

Curiosity◦ You like to know “why”Continuous Learning◦ How many measurement books have you read?◦ How much time do you spend with a mentor?Critical Thinking◦ Can you relate the data to the underlying software

engineering processes in use?engineering processes in use?Basic skills in arithmeticYou like to work with data

4

Page 5: Lessons Learned from Forty-five Years of Software Measurement

A Baker’s Dozen

5

It’s nice to know where you’ve been◦ “A map won’t help you if you don’t know where you

are.”Accurate prediction requires sufficient historical data (with context)◦ Similar projects with similar team make-up◦ Accurate data collection by the providersThe ultimate trailing indicator – decliningThe ultimate trailing indicator – declining customer base

6

Page 6: Lessons Learned from Forty-five Years of Software Measurement

A short list◦ Burn rate◦ Velocity in story points◦ Defect removal effectiveness◦ Problem backlog profilesNote a common characteristic – when used as predictors, they require (multiple instances of) reasonably accurate historical dataof) reasonably accurate historical dataLeading indicators require sufficient trailing indicator data to be useful

7

Historical data says we remove 70% of defects prior to entering system testp g yHow do we take a trailing indicator and use it as a leading indicator?Defects found to date = 700◦ 70% * X = 700◦ What is X?D ’ f hi h ld b dDon’t forget this should not be used as a point estimate!

8

Page 7: Lessons Learned from Forty-five Years of Software Measurement

When using Leading Indicators, always provide rangesp g◦ Value */-◦ With sufficient data, range estimation can be quite

sophisticated◦ With few data points, I use the min and max and

clearly state the confidence I have in the estimate (guess)

Just be sure your min or max isn’t the only value that is carried forward!

9

PSI = Problem Saturation Index◦ Same as being in the swamp with alligatorsg g◦ Management doesn’t want to hear about future

problems when today is filled with problemsPredictions of Doom are not well received◦ “Not a team player”◦ “Pessimist”◦ RationalizationRationalization

10

Page 8: Lessons Learned from Forty-five Years of Software Measurement

Don’t ignore the problem solution when identifying problems using leading indicators y g p g gUse risk analysis and identify cost vs return of mitigation or contingency◦ Avoid overstating the Doom◦ Don’t promise the world with your solutionKnow when your predictions are career threateningthreatening

11

“Managing a company by means of the monthly report is like trying to drive a car by watching the

ll l h ” byellow line in the rear view mirror.” Myron TribusWe ignore trends (see Leading Indicator)We over-react to random variationNo variation when there should be randomness or change

12

Page 9: Lessons Learned from Forty-five Years of Software Measurement

From a Monthly “Quality Report”12

0

2

4

6

8

10

Unfavorable trend in March

Jan Feb Mar

13

Run ChartsLook at the vertical scale AND the horizontal oo at t e e t ca sca e t e o o tascale◦ How may or how much you are behind is interesting◦ How long it will take to recover is more interesting

PlannedPlannedActual

14

Page 10: Lessons Learned from Forty-five Years of Software Measurement

Backlog was about 20,000One week we lowered the backlog for the first O e ee e o e ed t e bac og o t e sttime by 200Everyone in the meeting cheered – until I noted 20,000/200 = 100 weeksI was told “Never come to my status meeting again”

15

Understand variationDo not make a trend out of a single data o ot a e a t e d out o a s g e datapointUnderstand why the data has changedDon’t ignore trends

16

Page 11: Lessons Learned from Forty-five Years of Software Measurement

What’s wrong with this data?CPI

0.9

0.92

0.94

0.96

0.98

1

1.02

Jan Feb Mar Apr May Jun Jul Aug Sep Oct

CPI

CPI

Is it likely that any measure on a project shows no variation over 10 months?

Jan Feb Mar Apr May Jun Jul Aug Sep Oct

CPI is Cost Performance Index, but substitute any measure and the story is the same

17

Why did this happen?◦ When CPI fell to .99, project managers were “beaten j g

up” in the weekly project review◦ Recovery plans required◦ If CPI exceeded 1.00 project budgets were reducedWhat were the results?◦ Project managers learned to “fix” the charts◦ Management refused to believe the data was phonyManagement refused to believe the data was phony

18

Page 12: Lessons Learned from Forty-five Years of Software Measurement

Do not punish or reward “bad” or “good” data◦ Data providers are very sensitive to potential career

threatening responses to data they provide◦ Data ceases to be an asset when people are

threatened by data not matching expectationsTrends are often ignored with optimistic assumptions that “things will get better”◦ If you haven’t changed something, why will it getIf you haven t changed something, why will it get

better?◦ Is this week/month’s change really significant

(random vs assignable cause)?

19

Attempts to stabilize an inspection process were unsuccessful; defect density was out of ; ycontrolFeeding process was at fault◦ “Draft Review” step was uncontrolled

Effort spent varied from 5 minutes to 40 hours (!)Would this affect the number of defects found in the formal inspection? (This is a rhetorical question!)formal inspection? (This is a rhetorical question!)

20

Page 13: Lessons Learned from Forty-five Years of Software Measurement

When the data looks crazy, it may be the (unmeasured) feeding process that is at faultg p

21

Most data comes “from the trenches” Most decisions are made by project managers ost dec s o s a e ade by p oject a age sand higher levels of managementWhat happens if the providers have no clue about how or if the data is used?◦ “You can’t use that data for planning, we made it

up!”Random number generator used to input effort data◦ Random number generator used to input effort data◦ 95%+ of data records missing key values in

inspection data records

22

Page 14: Lessons Learned from Forty-five Years of Software Measurement

It you want good data, be sure the providers understand what decisions are made using gthe data they provideSee Lesson Learned #2

23

When the data looks funny, do not jump to conclusions◦ Inspection data

Modified modules: Prep rate 138, defect rate 22/KLOCNew modules Prep Rate 500, defect rate 10/KLOC

◦ Inspection theory would say the 2nd group of inspections were poorly conducted

Discussions with the people doing the work p p grevealed significant differences in the work products being inspected

24

Page 15: Lessons Learned from Forty-five Years of Software Measurement

Fix times recorded as zero hours◦ “Joe” was one of the most knowledgeable g

developers in the organization◦ Analysis of defect repair time revealed Joe was the

best – his effort was zero◦ So, we wanted to duplicate Joe across the

organization◦ When I asked how he did this, his answer was “After

d b h bl k h h fdebugging the problem, I knew what the fix was so the fix time was 0. If you want the analysis, fix, and retest time, ask for that”

25

Product factors may override process factorsAlways talk to the providers when the data ays ta to t e p o de s e t e datalooks funnySometimes the providers are playing games with the system (Joe)As a side effect, when you do this, the providers will recognize the data is being used and (usually) do a better job of collecting and recording it

26

Page 16: Lessons Learned from Forty-five Years of Software Measurement

Above was written on a cubicle white board◦ Discussion revealed frustration with data recordingg◦ When recording defect types, some categories were

viewed ad overlapping or poorly definedWe simplified the recording requirements◦ Data wasn’t being used◦ Irritation put other data elements at risk

27

Classic Apples and Oranges situation◦ Common understanding of sizeg

“Too hard to count”“I want to get credit for all the code I write”

First problem was easy to solve – write a utility to count, or even easier – estimate the size in about 1 minuteSecond problem was harder as it meantSecond problem was harder as it meant educating developers on how code counts were used

28

Page 17: Lessons Learned from Forty-five Years of Software Measurement

Always allow for an “other” or “N/A” selection◦ Encourage use of it when unsure of the selectiong◦ Too many “other” selections say the data definitions

are not clearCheck to see if your operational definitions of the data are clearly understood and how they are used

29

Intuitive-Sensor – N or S?◦ How many have been tested?

H N? S?◦ How many are N? S? Extremes on either end probably not for the best◦ A “full N” will tend to the abstract and theoretical◦ A “full S” will tend to only see what is there (or twist the

data to support their position)Be careful when you give data to an “S”Ask what they want the data for!

M t l th t li htl “N i h”Measurement people that are slightly “N-ish” seem better suited to the work (so I was told –true or false?)

30

Page 18: Lessons Learned from Forty-five Years of Software Measurement

Know your strengths or weaknesses relative to N or SKnow the tendency of anyone you give data to (especially if it is your boss!)

31

Initial attempts at using measurements may not be acceptedp◦ If predictions do not match expectations

“Interesting, but not believable”“Yeah, but we did X and Y, so things must be better”“Guess you are not a team player”

By the third cycle of prediction, learning organizations will be receptiveorganizations will be receptive

32

Page 19: Lessons Learned from Forty-five Years of Software Measurement

Perseverance will pay off in the long run or…You may need to look for greener pasturesou ay eed to oo o g ee e pastu es◦ But, if the grass is greener on the other side of the

fence, beware of rug-burn on astro turf

33

“Test cost is too high”◦ Engineering management viewed test as separate g g g

from development◦ Kept trying to reduce cost with single digit

reductions in test budget◦ Last 2 major releases had nearly 20,000 defects

found in testCompletely overlooked where the true cost was◦ Development was spinning its wheels fixing defects◦ Interesting cultural issues created a blind spot

34

Page 20: Lessons Learned from Forty-five Years of Software Measurement

Recognize software development is an end-to-end systemyDo not focus on the small stuff (Amdahl’s Law)In this case they overlooked the potential gain from reducing the number of defects –their cost accounting did not recognize d l kdeveloper rework costs◦ “It’s a complex product, of course there will be a lot

of defects”

35

There was once a large corporation that gained acclaim for their SW processes and g pmeasurement program◦ Measurement became the end goal◦ “The monthly metrics report is due, we better create

it”Root Cause analysis program failed◦ Punitive◦ Punitive◦ Only metric collected was Problem Closure to RCA

report interval

36

Page 21: Lessons Learned from Forty-five Years of Software Measurement

Measurement is a means to an end, not the endOnce the report becomes more important than information needed to manage, clever people will create a satisfactory report

37

Find a mentorBuy and read books on measurement and uy a d ead boo s o easu e e t a dsoftware engineering◦ “Software Metrics” by Fenton and Pfleeger◦ “Software Metrics” by Grady and CaswellTalk to measurement expertsListen to measurement experts

38

Page 22: Lessons Learned from Forty-five Years of Software Measurement

CuriosityContinuous Learning Co t uous ea gCritical Thinking

39

Each day is a learning experienceIn a baker’s dozen, the last Krispy Kreme may a ba e s do e , t e ast spy e e aybe the best

40

Page 23: Lessons Learned from Forty-five Years of Software Measurement

Ed WellerIntegrated Productivity Solutions, LLC

[email protected]

41