Measurement Program

8/6/2019 Measurement Program

1/9

A Software Metrics ProgramD. DeToma

J. PerryGTE Government Systems Corporation

Needham, MAA B S T R A C T

Software metrics play a central role insoftware management and technology, aswell as, in support of general businessgoals. Metrics are utilized at all levels ofa corporation. This paper focuses onsoftware metrics at the multi-&visionallevel. It views metrics as an integral partof software process. The ideas presentedare based on experiences and lessonslearned with the metrics and processprogram within GTE GovernmentSystems Corporation(GSC). Thesoftware metrics program at GTE GSC isdescribed, including metrics for softwareestimation, tracking, and control ofsoftware project progress and quality.Trends in software metrics are identified.

QRGAN IZATIONAL CONTEXT

Software activities are performed byorganizational units as part of a softwareprocess, for example, a softwaredevelopment or software maintenanceprocess. In this paper, the organizationalunit which performs the software

activities is considered part of a softwareproject, which, in turn, is part of a largerorganizational unit, such as a group ordepartment. At a higher level, these arewithin a division, which has market andproduct responsibility for a corporation.The term organization will apply to allthese levels, project, division,corporation.

The project software process is anintegrated collection of softwareactivities, roles, procedures, methods and

tools, for producing or enhancing asoftware product. The commonalityacross the project software processeswithin an organizational unit comprisesthe software process for that level of thecorporation, for example, a divisionsoftware process.

SOFTWARE MASTERA N N I N G

Organizational software processessupport organization business,management, and technical goals.Ideally, software goals are documented ina software master plan as subgoals of thebusiness goals.

A software master plan identifies andprioritizes the major software goals whichsupport the business goals of theorganization. The goals should bequantitative. Examples of such goalsinclude, reduction of errors in softwareproducts, increase of productivity of

software processes, and increase ofpredictability of software jobs. Inaddition, a master plan presents thestrategy for achieving the goals and theactions necessary to implement thestrategy.

RATIONALE FOR SOFTW AR EM ET R IC S

Software metrics are used to obtain datato support the software goals for theproject, for the divisions and, ultimately,

the business goals of the corporation. Itis important that these technical andbusiness goals be defined for each level,from corporate to division to project, elsethe software metrics could lack purposeand effectiveness. The data obtainedfrom software metrics comes from thesoftware projects, i.e. project softwareprocess and products and is aggregatedfor the divisions. For the metrics anddata collection to be effective andefficient, the software processes for theprojects and the divisions must be welldefined in termsof breadth and level ofde ai1.

SOFTWARE P ROCE SSFRAMEWORK

A software process framework helpsaddress the completeness and level ofdetail necessary for a software process.

230


2/9

An organization which has softwarerelated jobs has the elements of asoftware process, including the softwareactivities, software roles andresponsibilities, software products andthe artifacts which support them,However, the degree to which these areintegrated, formally defined, complete,supported, and practiced, varies. Asoftware process framework helps indescribing, defining, benchmarking, andimproving software processes.

The Capability Maturity Model(CMM)1,2 is a popular industry andgovernment software process frameworkused to assess/evaluate** and guide theimprovement of software processes. It isa collection of descriptions of softwarepractices. The software practices aregrouped into 5 levels, each level having a

set of subprocesses called key processareas(KPAs). Each KPA is described interms of goals and features. Thefeatures, include commitment to perform,ability to perform, activities performed,measurement and analysis, andverification. The features cover theissues of policy, document standards andprocedures re sources, training ,responsibilities, mechanisms,compliance, etc. The descriptionsemphasize what is required, rather thanhow the software practices are

implemented. The CMM is useful as aguide to an organization for addressingthe completeness and level of detail of itssoftware processes. A s such, i t isimportant for an organization to relate thegoals of the CMM KPAs to the technicaland business goals of the projects, thedivisions, and the corporation.

The CMM distinguishes the softwareprocess of an organization and that of aproject. The organizational softwareprocess comprises the software practicesused for all projects and is tailored for useby a project. A project software processis said to be an instance of theorganizational software process.

*Note: When used by a customer to determine thesoftware capabilities of an organization, the term'evaluation' is used; when used by an organization toexamine its own capabilities, the term 'assessment' isused.

Tailoring should be done according to adefined procedure with appropriatemanagement review and approval.

A software process framework identifiesthe activities and products of a processand these elements are useful inidentifying data and measurement pointsfor collecting metrics during a project.

SOFTWARE METR ICsFRAMEWORK

Metrics and data collection for a largeorganization require significant resourcesand, therefore, must be necessary andsufficient. This means that they must bedriven by and justified by a specificorganizational purpose. In science, forexample, data comes from experiments,designed to answer specific questions

about a hypothesis related to a theory. InBasili's GoaVQuestiodMetric paradigmfor software experimentation3, metricsare defined to provide data for analysis toanswer questions to satisfy specifiedgoals. A software metric framework,ensures that the metrics and data areeffective and efficient, by justifying themon the basis of the project, division, andcorporate technical and business goals.

If the CMM is used as the softwareprocess framework for an organization,

then it can also be used as a softwaremetrics framework for the organizationaland project metrics definition andcollection.

The CMM, fundamentally, addressessoftware costs, schedule, functionality,and quality goals. These can beconsidered the goals of the softwareprocess, in support of the overall goals ofthe organization at all levels. Forexample, production of a softwareproduct within a specified budget andschedule, with a specified limit on thenumber of defects fromintegration/system test, is a softwareprocess goal. Improvement of the ROIfor the software process is an example ofa higher organizational goal for softwareprocess. The software process goalsaresupported by the goals of the KPAs. Thegoals of the KPAs are satisfied if thefeatures have been successfully

23


3/9

implemented in the software processesfor the project and for the divisions.

To satisfy the variety of technical andbusiness goals for the projects anddivisions, memcs must supportestimation, tracking, and controlfunctions of the software process andproducts. Division estimates of cost,schedule, functionality, and quality aremade for classes of projects across thedivisions. Project estimates are made atproject proposal and project inception andare updated throughout the project.Metrics data collected during the projecttrack the status and progress of theproject. These actuals are compared withthe estimates to indicate and avoidproblems. During a project, a myriad oftechnical and management decisions aremade; metrics help identify these decisionpoints and help in the selection ofalternatives. For example, they supportthe decision to stop an activity, such astesting.

In addition to the CMM, it is necessary tohave more detailed frameworks or modelsfor each KPA. For example, for thesoftware product engineering KPA, aspecific life-cycle model andmethodology are necessary to identifyparticular engineering activities, roles,and artifacts, in sufficient detail for more

accurate estimation, tracking, and control.The goal hierarchy is depicted in Figure 1 below. A specific metric, for example,defects per module, can support severalfunctions, several KPA goals, etc. Thenumber of defects indicate projectprogress, product quality, whetherreview and testing completion criteriahave been met, whether estimates wereaccurate, whether the organization iscompetitive, etc. Other examples areshown in Figure 2.

Corporate Goals

n

n

... n

Division Goals

Project Goals

Process Goals

n... Key Process Area

Metric

Figure 1. Goal Hiera rchy for Metrics

CM hiWAS

Sollware Sollware Project Quanlitslive ProcssProjecl Tracking & ManagementPlanning Oversighl h

ActualEstimtr mt. schedule, si=. effort, m u , fea

"cs

Figure 2. Example CM MK P A goals and associated melrics whichsupport (he goals

The coverage and granularity of the data,as well as the time or frequency of itscollection, depend on the type of metricand the goals. Data is collectedperiodically, when an event occurs, fori l l or somiactivities, for all or somesoftware artifacts, for all or someprojects. The data must be validated,then analyzed, and used to support allthe goals. A basis or core set of

23 2

of


4/9

fundamental metrics, and therelationships/correlation which holdamong them, can be used to derive newmetrics, validate other metrics, andsupport most management and softwareprocess goals.

GTE GOVERNMENT SYSTEMS

-GTE GSC's metric program formeasuring, evaluating, and reporting onsoftware process and products is thefoundation for estimating, monitoringprogress, managing problems and risks,and ensures that GTE management andcustomers have insight into the softwareprocess and software products.

Metrics are a key part of the softwareprocess and program management at GTEGSC for all divisions. Projects benefitfrom a long-standing metrics program,because by collecting cost, schedule, andresource and quality measurements forprojects across all divisions, moreaccurate estimation, tracking, and controlcan be achieved.

Features of the GTE GSC Governmentmetrics collection program include:

An existing knowledge base whichsupports prediction of projectperformance with confidence usingmanagement, quality and cost/scheduleestimating data

A well-established, recognized metricsprogram which supports engineering,finance, and business areas

A means of allowing all levels ofmanagement to monitor softwaredevelopment progress to the necessarylevel of granularity and visibility

A means for quantifying results ofprocess improvement activities.

G TE GSC METRICS POLICYND METHODOLOG Y

GTE has a mature, proven SoftwareMetrics Program. This program wasformally instituted in 1986, and hasbecome an integral part of thedocumented GTE Software DevelopmentProcessand is mandated by GTE GSC

Software Engineering Policy. Thispolicy states:

"Consistent measures whichshow trends, assign earned value oridentify activity completion shall becollected, analyzed and applied to supportproject management and to improve the

software process.Metrics on products and processshall be established and trackedthroughout the software life cycle.

At a minimum, technical baselineshall be established during the proposalphase using approved cost and schedulemodels; tracking and analysis of softwaredevelopment shall be performed; andproject lessons learned shall begenerated. "2

A standard set of metrics for projectsacross all divisions addressing the fullprogram life cycle is described in theSoftware Metrics Handbook. Themetrics fit into the following categories:

Software SizeSoftware PersonnelSoftware HoursDevelop men t ProgressProblem MetricsProblem PreventionRequirementsDesign StabilitySoftware ComplexityComputer Resource Utilization

From these, important ratios, forexample, (errorshize), and othercombinations can be produced.To identify the scope of the SoftwareMetrics Program from bid board throughproject completion, GTE GSCestablished a Software MetricsMethodology (see Figure 3), whichemphasizes a constant feedback loop tothe projects.

A centralized data storage facility servingall divisions was established in 1986 fortimely dissemination of project andorganization information. This data isused to support management decisionsand to pinpoint process improvementopportunities in order to improve thequality and timeliness of delivery of ourproducts. The advantages provided toprojects by this database include better

23 3


5/9

software sizing estimates and moreappropriate schedules for biddingpurposes.

GTE GSC's long-standing commitmentto metrics will benefit projects by betterpredictability of future performance andbetter control of the software

development process, thus ensuring ahigher quality product.

BETTER PREDICTABILITY OFPROCESS PERFO RMANC F

Predicting future performance based onpast performance is another goal of theGTE GSC Software Metrics Program.All software metrics are stored in theGTE GSC metrics data base so that agiven project's performance statistics canbe analyzed and used for bids and forother projects. The advantages provided

to projects by this data base include bettersoftware sizing estimates, moreappropriate schedules for biddingpurposes, and lessons learned.The use of cost/schedule estimatingmodels is mandated by GTE GSCSoftware Engineering Policy.

BID BOARD PRE-PROPOSAL PROPOSAL COSTIFMR PROJECT AWARD

182 18 2 18 2

.. . ...

COST/SCHEDULEESTlMATlNGAND MANAGMENTMETRCSPLAN

HLSTORYE X E R NAL DATABASE

SEOSEPG

I UPGRADING I

IECTORRESOURCES 8P R E E S SI-ESlirnadng Mod&

2 - M e t i c s C d l e c t i macmrding lo p n

TIME

Figure 3: GTE GSC Software Metrics Methodology

At GTE GSC, SEER and COSTAR arethe primary software estimating toolsused, implementing the Jensen andCOCOMO models respectively.Software cost/schedule estimating modelsare required for use on all proposals,cost-to-complete exercises, and in theformulation of lessons learned.Involvement in all of these activities will

help to ensure furtherrefinemendcalibration of the models sothat they accurately portray the way GSCdoes business. These model estimates arecompared to the engineering "bottomsup " estimates as a sanity check. Themodels are used to do what-if analysisregarding development environmentselection, resource experience and

234


6/9

allocation, and so forth. This includessensitivity analysis on the parameters ofthe model to identify and control the mostinfluential factors which affect the modeloutput. At the completion of the project,the models are run using the actual size ofthe software system to address estimatedcost(mode1 outputs derived from project

parameters) and actual cost(actual datacollected from the project onexpenditures, etc.). This finalinformation is captured in the LessonsLearned report. By tracking and refiningour estimating process, our goals ofimproved predictability will be moreeasily met. Note: The models are notintended to replace good engineeringjudgment, but to supplement it withindustry experience.

B E T T E R C O N T R O L O FWARE DEVELOPMENT

GTE recognizes the benefits ofdeveloping plans and managing cost,schedule, and quality performance. Agoal of software metrics for the projectsis to provide data to management forcontrolling software development. Whensubcontractors are used as teammembers, they provide metrics data on amonthly basis in order for GTE toeffectively manage and report progress atscheduled reviews. GTE will ensure thatmilestones are met by scheduling reviewsthroughout the project life cycle, anddocumenting action items. These metricsor indicators will identify potentialproblems early in the life cycle and helpminimize schedule slips. Software sizewill be monitored in temis of estimatedlines of code, especially during designphases, to ensure that the project isstaffed properly.

HIGHER OUALI TY PRODUCT

Quality metrics are collected for thepurpose of producing reliable andmaintainable software. Quality can beassessed throughout the life cycle. A keymeans of measuring quality during thedevelopment process is conducting peerreviews. Peer reviews are held andmetrics on the product and process arecollected and reported. Specifically, the

quality of the software design ismeasured by the amount and severity ofaction items and changes generated atpreliminary and detailed design reviews.Lines of code and complexity aremonitored by the software developmentenvironment on a CSU (computersoftware unit) basis. Code is examined

to ensure that guidelines are followed.Software Development Files are inspectedto determine, among other things,completeness of unit testing. Defects arecaptured in Software Problem Reportsand categorized by type and priority. Thenumber of problem reports byCS C(compu er softwarecomponent)/CSU are also monitored. Astudy is underway which will help us inpredicting the number of defects in thesoftware. Data is being used to validatean existing model which has been used at

another GSC location. Preliminaryresults show that minor tuning isnecessary for the model to represent theway GSC does business.

METRICS SE T UP AND TRAININ(ACTIVITIES

Prior to the start of a project, the softwaremanager/technical lead developsschedules, etc. and contacts the memcsgroup who will provide them with afolder on the file server for their project,and with id's for each person responsiblefor entering data. A standard softwaretemplate, which has been developed inconjunction with management reportingcriteria, is tailored to reflect the number ofCSCI's (computer software configurationitems) for the project. At the time ofcompletion, each team member is trainedin file server log on, accessing their files,data to be filled in, macros to be run andhelpful hints derived to date.Additionally, a formal metria trainingclass is conducted in conjunction with theGTE GSC Software EngineeringTraining Initiative. Software Metria isalso included as a topic of the SoftwareProcess Course.

TOOL A N D ENVIRONMENTSUPPORT OF METRICSCOLLECTION

235


7/9

Data collection should not be a significantoverhead burden to a project. It shouldbe as natural a byproduct of projectactivities as possible. For example, peerreview minutes usually alreadyincorporate the important defect data.

Data collection and analysis should also

be automated as much as possible.Software size and complexity data, forexample, can be obtained from softwaretools.

ProcessImmaturc to Heavily

"tooled" Mature Process

SOF TW ARE METRIC A NALYSIS

210%/630%

Analysis is done on a per project basis aswell as in a composite manner. Areas ofanalysis include productivity, problemdensity, effort allocation to phases,completion on time, software staffing(plan vs. actual vs. models), softwaresize (plan vs. actual), effort required forproject (plan vs. actual vs. models), anddefect occurrence. Specific topics ofanalysis have included the productivityand quality associated with differentdevelopment processes and with use ofthe generic architecture (GTE STEPgeneric software architecture and genericexecutive).

SOFTWARE PROCESS ANDPRODUC T INTE RACTION

Immature to MatureProcess

The effectiveness of a software processand associated software metrics dependon the accuracy and precision of themodel of the software to be produced.With a richer model, the softwareprocess and metrics can be moreeffective. For example, the GTE STEPgeneric architecture models the softwareas a 4 evel structure constrained by dataand control scope, access, andoperational rules. This architecture is

supported by generic infrastructuresubsystems and services.

Defect Density Reduced50%-70-%

Ch4M KPAs tailored and implemented todirectly support this architecture will bemore effective. For example, softwareconfiguration control of software buildscan be automated because of the knownstructure of the software. For example,software project planning and softwaretracking and oversight will be more

Productivity (SLOChr)Quality (FTRKSLOC)Predictability (effort hrs)

accurate and precise because of past dataon the generic architecture, its supportinginfrastructure software, and its use onsimilar applications.

135% increase23 1 ?6 decrease

within 9% actuals

RESULTSEXP E RIE N C E / O U A NTIF IED

Measured results have shown higherproductivity using more matureprocesses:

I new/newplusreused I

Measured results have also shown thatrisk is reduced by using more matureprocesses:

I I newhew ulus reused I

Table 2. Quality ImprovementRelating to Process Used

A s in industry, GTE GSC has focusedmore attention on Process Improvement.The table below illustrates the resultingincreases in productivity, quality andpredictability.

I I Immovemen I

Table 3. ImprovementsExperienced

236


8/9

cROGRAM

When considering the cost of the metricsprogram, a few areas must be considered.There is a metrics administrator, who is

responsible for the coordination of themetrics program for the entireorganization, and project contacts. Theproject contacts are the ones who actuallycollect and report the data to thecentralized function. The actual activitiesperformed by the central metrics functionhave been previously discussed. There isan investment associated with having amemcs function. It requires up frontinvestment while planning is being done,data is being collected, validated andanalyzed. Being able to predict cost,

schedule and quality of a productaccurately based on historical data isnecessary in today's market. It is alsonecessary to be able to effectively managecost, schedule and quality of a product.One of the key initiatives in probablyevery business is Process Improvement.An effective metrics program will providethe data needed to identify to determineROI(return on investment) andopportunities for Process Improvement,including improvement of the metricsprogram itself. The need for a metric

may change, the effort to collect data maybe inefficient, or additional metrics maybe needed. The payoff of having aneffective metrics program outweighs theinvestment.

LESSONS LEARNED. TRENDS,A N D R E C O M W N D AT IONS

In order to have a successful metricsprogram, it is necessary to have top-down support. There has to be buy-inand commitment at all levels. A centralmemcs function is essential to insureconsistency, including standards formeaics definitions, collection, andvalidation, for data to be used, distributedand stored for future use. It is necessaryfor the central function to get involvedwith the projects, train them, work withthem, get them interested. This helps tofacilitate the buy-in that is so necessaryfor success.

A metrics program need not be complex.It should be planned, monitored,reviewed, supported, and improved. Thememcs collected must be justified on thebasis of organizational and process goals.The effectiveness of the memcs isdetermined by their support of these

goals, including estimating, monitoring,and control.

Once a standard set of basic metrics isidentified and defined, data collected, andvalidated, simple derivations, such asratios, can be made. Statistical modelsincorporate other relationships which canbe used to support estimation andplanning. Software architecture modelsprovide a richer structure for improvingestimation and supporting tracking andcontrol.

A myriad of decisions are involved inplanning and conducting a software job.Metrics and their underlying models giveadvance indication of many of thesedecisions, alternative choices andconsequences of the choices. Data on thethese choices and consequences can thenbe taken into consideration for cost andschedule bids, minimizing risks, andavoiding problems, and preventingerrors.

A metric program should be focused onsupporting all levels of management andtechnical decision making. For example,performance consequences of softwaredesign decisions will impact cost andschedule. These should be known atproposal time and appropriate actiontaken throughout the job to ensure theplanned final system performance. Datashould be available to support the cost,schedule, and resource estimates andactuals for all types of requirements.

Thus, software metrics should be usednot only by software personnel andmanagement, but by systems, finance,and business management.

A metrics programmustbe flexible andresponsive to the change in goals.Technology is changing, prompting theneed for new metrics, for example newsoftware size measures. Business is also

237


9/9

changing. Today, many corporations areexperiencing a change in job type, to lesssoftware development, to moremaintenance, reengineering , ntegra ion,or outsourcing jobs. For example,annual software costs for U.S. industryare estimated to be $100 billion with $30billion spent on new development and

$70 billion spent on maintenance6.Maintenance costs could grow to $1trillion in 10 years. These changes entailchanges in issues which affect cost,schedule, and performance. A metricsprogram must evolve to meet thechanging needs.

REFERENCES

1. Capability Maturity Model forSoftware Version 1.1, CMU/SEI-93-TR-24, February 1993, SoftwareEngineering Institute, Carnegie MellonUniversity, Pittsburgh, PA 15213.

2. Key Practices of the CapabilityMaturity Model, Version 1.1, CMU/SEI-93-TR-25, February 1993, SoftwareEngineering Institute, Carnegie MellonUniversity, Pittsburgh, PA 1521 3.

3. Basili,V., Weiss, D., A Methodologyfor Collecting Valid SoftwareEngineering Data, IEEE Transactions onSoftware Engineering, Vol. SE-10, No.6, November 1984.

4. GTE Government Systems Sector,Communications Systems DivisionSoftware Engineering Policy, EN 5.2,GTE Government Systems Corporation,

Needham, MA 02194.5 . GTE GSC Metrics Handbook, V. 6,1992, GTE Government SystemsCorporation, Needham, M A 02 194.

6. Edelstein, V., "New Born IEEEStandard for Software Maintenance",Proceedings of Strategic Solutions forExisting Systems Conference,Washinton, D.C., November 1993.

238

Measurement Program

Documents

Transcript of Measurement Program