Error reports as a source for SPI

Tor Stålhane

Jingyue Li, Jan M.N. Kristiansen

IDI / NTNU

Goals

Business:

How can we reduce the cost of corrective maintenance?

Research:

What are the main cost drivers for corrective maintenance

Company A – 1 Company A is a software line company with

only one product.

The product is deployed on more than 50

operating systems and hardware platforms.

The company has 700 employees and of

these 400 are developers and testers.

Company A – 2Via the company’s DTS – Defect TrackingSystem – the company collected:• Defect ID, priority, severity and report

creation date• Defect summary and detailed description• Who found the defect and when• Estimated correction time• When was the defect corrected• Detailed comments and work log of the

person who corrected the defect

What we did in company A

We improved the company’s DTS based on

the IBM concept of orthogonal defect

classification – ODC.

Based on a study of earlier reported defects,

the classification scheme was modified to fit

the company’s needs.

DTS system enhancements at company A

Added or revisedattributes

Value of the attributes

Effort to fixTime-consuming: more than one person-day effort Easy: less than one person-day effort

Qualifier Missing; Incorrect; or Extraneous

Fixing type Extended the ODC “type” attributes to reflect the actualdefect correction activities of the company.

Root causeProject entities, such as requirement,design, and documentation, which should be done betterto prevent the defect from occurring earlier.

Company B – 1 Company B is a software house that

develops business critical systems, primarily

for the bank and finance sector.

Most of their projects have a fixed price and

a fixed delivery date.

The company has 800 developers and

testers.

Company B – 2

Via the company’s DTS the company

collected among other things:

• Defect ID, priority, severity and report creation date

• Defect summary and detailed description

• Who found the defect and when

• Who tested the correction and how

What we did in company B

Just as for company A, we improved the

company’s DTS based on a study of earlier

reported defects.

In addition, the changes also enabled us to

collect data that should later be used for

software process improvement.

DTS system enhancements at company ¨B

Added or revised attributes

Value of the attributes

Effort to fixClassify effort to reproduce and fix defects as “simple”– less than 20 minutes, “medium” – between 20 minutesand 4 hours, and “extensive” – more than 4 hours.

Defect typeIntroduced a new set of attributes that were adapted tothe way the developers and testers wanted to classifydefects.

Root causeThe attributes here are project entities, such asrequirement, design, development, and documentation,which can be done better to prevent the defect earlier.

Data collection

In both companies, we collected defect data

when the companies had used the new DTS

for six months. Only defect reports that had

all their fields filled in were used for later

analysis. This gave us:

• Company A – 810 defects

• Company B – 688 defects

Data analysis – 1 Our goal was to identify cost drivers for themost expensive defects. Thus, we split thedefects into categories depending onreported “time to fix”. • Company A – two groups: “easy to fix” and

“time consuming”• Company B – three groups: “simple”,

“medium” and “extensive”. “Simple” and “medium” defects was combined into one group – “simple” to be compatible with company A

Data analysis – 2

We identified the most important root causes

for the costly fixes through qualitative

analysis.

• For both companies we had “correction type” and “cause”.

• We also found important information in the– developer discussions (company A)– test descriptions (company B)

Root causes for high effort fixes - A

Reasons for the costly debugging Numbers of high effort defects in each business unit

Core B2C B2B

Hard to determine the location ofthe defect

20 37 4

Implemented functionality was newor needed a heavy rewrite

13 29 2

Long clarification (discussion) ofdefect

19 5 0

The original fix introduces newdefects / multiple fixes

13 9 0

Others (documentation is incorrector communication is bad)

2 0 0

Reasons not clear 3 16 0

Root causes for company ABased on the collected data, we identifiedthe most important root causes for costlydefects:• It was hard to determine the defect’s

location• Implemented functionality was new or

needed a heavy rewrite• Long, costly discussion on whether the

defect report really was a defect or just misuse of the system.

Root causes for high effort fixes – B

Root cause attribute

Number of defects

SimpleMedium Extensive Sum

Functional defect 9 2 11

Wrong test data in build 77 4 81

Bad specification 89 12 101

Bad test environment 9 1 10

Development problems 317 57 374

Sum 501 76 577

Root causes for company B Bad specifications and development

problems account for 91% of the high effort

defects.

If we consider the sub-categories defined for

these two root causes, we find that 70% of

all correction costs are due to:

• Errors in business logic

• Unclear requirements

• Problems in graphical interface

Important maintenance factors – 1

Several published studies have identified the following important maintenance cost factors:

• Maintainers’ experience with system and application domain

• System size and complexity

• The development activity where the defect is discovered

• Tool and process support

Important maintenance factors – 2

• System size and complexity – large complex systems makes it difficult to– Analyze the system to decide where the

defect stems from - A – Deiced how to fix the defect - A – Find the right solution to implement – A, B

Important maintenance factors – 3

• Low maintainers’ experience with system and application domain cause – Need for heavy rewrite of functionality - A– Development problems, e.g. with business

logic and user interface – B

Important maintenance factors – 3 ISO 9126 defines maintainability as: • Analyzability• Changeability• Stability • Testability

The high maintenance cost factors size and

complexity fit well into this model. However, the

model focuses on software characteristics and

ignore the influence of the developers’ knowledge

and experience

Conclusions - 1 Important data sources when analyzing

maintainability are:

• Resources needed for fixing

• Defect typology, e.g. ODC

• Qualitative data such as test description and developer discussions

The effort model should be updated

regularly, based on the defect profile and

project environment.

Conclusions – 2

There is no “best estimator” for corrective

maintenance. Important factors are

• Software characteristics – e.g. as defined in ISO 9126

• Staff characteristics – e.g. knowledge and experience