Software Metrics – So Simple Yet So Dangerous

Shrini KulkarniTest Consultant

HCL TechnologiesBangalore, India

http://shrinik.blogspot.com

Seek Simplicity, on achieving it, learn to distrust it - Alfred North Whitehead

A EuroSTAR 2009 Presentation

2

Why measure at all….

To understand subjective things objectively - Objectivity Software (is this software ready to ship?) Performance of group of people (how well is this vendor doing? How well is

this company performing) Performance of an individual (how good is this developer or tester)

As a tool to support making decisions – Decision Support High defect rate = Application quality is poor High BP = Heart problems High inflation = Regulate (reduce) money supply in the market High number of Road accidents = Driving under the influence / helmet/seat

belts, better roads

To make improvements (Anything that cannot be measured cannot be confirmed

as improved/managed)

3

What kinds of measurements do we do in software?

Classic Metrics - related to software code Code size - KLOC Cyclomatic complexity Code coverage

Software Resource requirements Humans (effort – man months) Time Money Project Size estimation – Function Points Programmer productivity (KLOC/month)

Software Quality Defects – Density (Defects/KLOC), Leakage Test cases

4

Long ago, Scientists and Philosophers made measurements …

Tell me, O my God, what am I measuring, when I say either, with no aim at precision, that one period is longer than the other, or, precisely, that one is double the other? That I measure time, I know. But I do not measure the future, for it is not yet; nor the present, for it is not extended in space, nor the past, which no longer exists. So what do I measure? St. Augustine (354-430 A.D) , Confessions, A.D. 398

For the measurement of time, we employed a large vessel of water placed in an elevated position; to the bottom of this vessel was soldered a pipe of small diameter giving a thin jet of water, which we collected in a small glass during the time of each descent, whether for the whole length of the channel or for a part of its length; the water thus collected was weighed, after each descent, on a very accurate balance. -- Galileo Galilei, Dialogues Concerning Two New Sciences

When you can measure what you are speaking about, and can express it in numbers, you know something about it; but when you cannot measure it, when you cannot express it in numbers, your knowledge is of a meagre and unsatisfactory kind: It may be the beginning of knowledge, but you have scarcely in your thoughts advanced to the stage of science - Lord Kelvin

5

Measurement and Metrics … in Software World - I

“Measurement is the process by which numbers or symbols are assigned to attributes of entities in the real world in such a way as to describe them according to clearly defined rules” – Norman E Fenton

Tom DeMarco said something that still echoes in software WorldYou can’t control what you can’t measure

Victor Basili gave a model for defining metrics – GQM (1988)“Whatever the product, whether it’s software or something else, process is important, because it is the only way to optimize what you are doing. And measurement is important because it’s the only way you can observe and get feedback about what is really happening”. Victor Basili

6

Then came, Jerry Weinberg with “orders measurement*

Weinberg’s Law of Metrics: “That which gets measured gets fudged.”

First Order Measurement: that help “just enough” to build things in the first place. Qualitative, inexpensive and fast. Experience based

Second Order Measurement: that help to optimize, tune and make things cheaper. More Quantitative, based on formal and mathematical models.

Third Order Measurement: Highly quantitative measurements that support the physicist's search for new natural laws.

What seems to be happening? What should I do now?

What is really happening? How can

I manipulate it?

Why it is happening? What will happen

next?

* Weinberg, Gerald M. Quality Software Management, Volume 2: First-Order Measurement. Dorset House Publishing, New York, 1993.

3 Categories of Measurement

7

Measurement and Metrics … in Software World - II

Capers JonesWorld’s Record Keeper of IT

MetricsCem Kaner’s

Construct Validity

James Bach could just make up any metric

Michael Bolton’s Appeal to look for stories behind metrics

Mathew Heusser’s Hair Cutting Metaphor

8

Not so good news now … Metrics can be dangerous

9

What makes metrics “dangerous”

Challenges with Construct Validity

Goal Displacement – Need to measure overrides everything else

Software is reified – Do you care?

Very act of measurement alters the system – in a big way

Metrics are usually gamed – To show only “good” things

Information loss – Metrics represent rich multi-dimensional data

It is tempting to measure what is easy to measure

10

But there are compelling reasons to use metrics regardless of ….

Software development/Testing is not an art (at least completely) hence abandoning all kinds of quantitative measurements does not appear right !!!

Just because people misuse metrics should we abandon all metrics ? Gun in the hands of kid, kills someone – would you blame the gun? Blame the

victim? You can get into accidents so stop driving the car? Metrics can be gamed so what? Dysfunctional metrics are not unique to our field Metrics are key to process improvement - how else we can know how good is

the process? Metrics are just numbers – nothing to fear about or to avoid The lack of metrics and the ability to equate them to our business value is our

own fault It is impractical to do “management by walking around” Executives do not make decisions by qualitative inputs – they need hard facts,

numbers

Two kinds of people use statistics - liars and politicians - Anonymous

11

OK, Let us accept there are problems with metrics … so what?

Deal with Software as you deal with any other human constructs in social and cultural contexts Family and Trust in human Relationships Teenager upbringing Economic models of demand and supply

When using metrics – go beyond metrics – seek qualitative explanations or “stories” being told about metrics. Seek many alternative (contrasting) stories. Ask “ What would be a contrasting interpretation ?”

Do not measure humans and their work in numbers.

Use broad based attributes to minimize harmful side effects of one dimensional metrics

Use metrics to gain more information – use them as clues to solve and uncover deeper issues.

Many important things that must be managed can’t be measured – W. Edward Deming

12

What else we can do ….

Do not use metrics to judge the work of a human – Reward or punish individual‘s work. Programmer/Tester productivity metric is something that you need to avoid – at least in isolation

Use your metrics as tool to “inquire” not to take actions

Key to your success is your distinction between inquiry metrics (metrics that are used to prompt questions) and control metrics (questions that are used to trigger some action other than the asking of questions)

Be skeptical with good looking metrics – 100% statement /branch coverage can still mean untested code

Create an environment where metrics misuse can be minimized

Understand the context of the measurement. Do not blindly chase industry standard numbers

Metrics should never be an end unto themselves. Metrics are meant to help you think, not to do the thinking for you - Alberto Savio

Looking for Engineering and technical solution in lab/enterprise setup is probably the mistake that we are making in dealing with metrics….

13

If Metrics are innocent, people use them wrongly …

Software Metrics related problems are in the realm of

“social science” and should be dealt in that way.

Are we treating them like that ?

14

Closing thought …

The main role of models is not so much to explain and to predict – though

ultimately these are main functions of science – as to polarize thinking and to

pose sharp questions

Mark Kac on mathematical models, adopted from Introduction to General systems

thinking – Gerald Weinberg

Now replace word “model” by “metric” – How does it read?

Good Metrics do not give answers but make important questions apparent

15

Thank You!

shrinik@gmail.comhttp://shrinik.blogspot.com

http://twitter.com/shrinik