Navair Software Estimation Improvement Program

Rick Holcomb AIR-4.2 Cost Department

(301) 342-2450 [email protected]

Dan GalorathGalorath Incorporated


30 January, 2003 36th ADoDCAS1

NAVAIR SW Estimation Improvement Program

30 January, 2003 36th ADoDCAS 2

Agenda

Whats Necessary for Accurate Software Estimation?

NAVAIR Initiatives Galoraths Phase II Implementation of

SBIR N01-020


Whats Necessary for Accurate SW

Estimation?


What is Accuracy?

How close is the initial estimate to the final estimate?

How has functionality changed from the initial technical baseline to the final product? What impact has this had on the estimate?


Teamwork & Discipline are Required

Improvement in SW Estimation requires participation by both NAVAIR & its Industry Partners, not just the Cost Department.

Requires mature, disciplined software processes for all phases of the life-cycle for both the Acquisition and Development organizations to produce an accurate complete Technical & Programmatic Baseline.

the major factor in successful software development is disciplined execution. Report of the Defense Science Board Taskforce on Defense Software, November 2000.


A Mature Disciplined Development Environment is Important

Generally accurate software estimates can only be produced when high quality data about the development organization(s) is available.

High quality software data can only be generated by mature, disciplined Acquisition and Development organizations.

Even if data is available for an immature, undisciplined organization, attempting to estimate for this organization is problematic.

Having good data on an immature undisciplined organization is anoxymoron.Once data starts to become available on an organization, it indicates at least the start of an effort to increase the maturity and discipline of the software development process.


SW-CMM Summary

INITIAL

REPEATABLE

DEFINED

MANAGED

OPTIMIZING

Ad hoc; occasionally chaotic

Disciplined

Standard, consistent process

Predictable process

Continuouslyimproving process

1

2

3

4

5LOW

R I

S K

R I

S K

R I

S K

HIGHHIGH

Levels of Process Maturity


Probability of Success

(1) Capers Jones, Becoming Best In Class, Software Productivity Research, 1995 briefing(2) 1 Function Point = 50 SLOC ADA, 40 SLOC C++, 320 SLOC Assembly

0%

20%

40%

60%

80%

100%

120%

0 1 10 100 1,000 10,000 100,000

SW Size (Function Points)

Prod

uct S

ucce

ss

SEI 1 & 2 SuccessSEI 3,4, & 5 Success


With & Without Accurate Data

.

0 %

140%

-140%

...

.

.

..

..

..

..

. .

. . . .

.. . .

. .

.

.

. . . .. . . . . . . .... . . .. ..

. .. ..

.

.

.

. .. .. ...... . .. . ... . .. . .. ..

Without Historical Data With Historical DataVariance between + 20% to - 145% Variance between - 20% to + 20%

(Efforts = Labor Hours)

(Mostly Level 1 & 2) (Level 3)

Ove

r/Und

er P

erce

ntag

e

.

(Based on 120 projects in Boeing Information Systems)

.. . .

.

.

. .

..

.

. .

..

..

. .

..

.. .. . .. . . . . .. . . . . .. .

... . .. . .

. . . .. . . .. . . . .. . . . .

. . . . .. . . . . .. . . . .. . . . . .

. . . . . .. . . . . . . .

. . .. . . . .

. . . . .. . . .

. . . . . .. . . . . .

. . . . . .

John Vu, Boeing, keynote talk at SEPG 97, Software Process Improvement Journey (From Level 1 to Level 5)


NAVAIR Initiatives


Process Improvement

Developers of ACAT I, II, III, & IV Software Intensive Systems will achieve a SEI SW-CMM Level 3 or equivalent.

NAVAIRINST 5234.1 Policy on Software Evaluations for Naval Air Systems Command Programs

Achieve a corporate SW-CMM or CMMI Level III by 2005, and Level V by 2008.

NAVAIRINST 5234.2 Requirements for Process Improvement Actions for Naval Air Systems Command Software Acquisition, Development and Life-Cycle Support

Implement Corporate Requirements for Software Metrics & Measurement Collection.

NAVAIRINST 5234.5 Naval Air Systems Command Metrics for Software Intensive Systems


Software Resource Center

The NAVAIR Software Resource Center provides expert training & assistance to programs involved in the development of software intensive systems in areas such as:

SW-CMM & CMMITeam Software Process (TSP) / Personnel Software Process (PSP)Practical Software & Systems Measurement (PSM)Independent Expert Program Reviews (IEPR)Software EstimationSoftware Engineering Acquisition, Management & DevelopmentLessons Learned


Communications

The best results are achieved when all levels of the organization agree with the need for change rather then having it forced upon them.

A major aspect of NAVAIRs initiatives is the training and communications with the workforce.


Progress

F/A-18 Weapons Software Support Activity (WSSA), SW-CMM Level V.

Longest history of process improvement in NAVAIR. Prime contractor, Boeing, is also a Level IV. Recent developments have been very successful at meeting cost and schedule objectives.One of Top 5 DoD Software projects in 2001, http://www.stsc.hill.af.mil/top5projects/

AV8B WSSA, SW-SMM Level IV and TSP/PSP adopted.AV-8Bs Experience Using the TSP to Accelerate SW-CMM Adoption, Crosstalk Sep 2002, http://www.stsc.hill.af.mil/crosstalk/2002/09/hefley.html

1 SW-CMM Level III Group 3 SW-CMM Level II Groups 10 Groups adopting TSP/PSP 18 Groups started SW-CMM Process Improvement Efforts 8 Groups investigating CMMI


Improved Tools

SBIR N01-020, Software Cost & Schedule Estimation:

Demonstrate processes and tools that will estimate software cost and schedule with a standard deviation of 10% or less. Make best possible use of limited data sample sets for developing software cost and schedule estimates.Enhance update of estimates based on actual project data.Enhance collection and archival of project software metrics and measurements in order to support calibration and improve accuracy for both current and future projects


Galoraths Phase II Implementation of

SBIR N01-020


Project Plan

ID Task Name Finish

1 ProjectMiner Mon 4/19/042 ProjectMiner Release 1 Thu 7/3/03

106 ProjectMiner Release 1B Tue 9/23/03160 SEER-SEM Integration Tue 10/21/03161 ProjectMiner Release 2 Mon 1/26/04246 ProjectMiner Release 3 Wed 2/18/04273 ProjectMiner Release 4 Tue 3/23/04300 Transition - non-iterated tasks Mon 4/19/04

2/10 4/14 6/16 8/18 10/20 12/22 2/23 4/27 6/29 8/31 11/2 1/4 3/7anuary 21 June 1 October 11 February 21 July 1 Nov ember 11 March


Data Rights

Government has standard SBIR data rights for application.

Can view all requirements, design and source code of product.After 5 years, Government may go to another source to modify application, IF:

Galorath Incorporated has not copyrighted the application; however, it probably will.Government has not contracted with Galorath for further modifications of the application.


Plan to Meet NAVAIR Goals

SizingSizing made easier, and more possible earlier in the estimation process

EstimatingDecision makers will better understand the origins of an estimateEstimate will be more accurate, more reliable, and better integrated into the project life cycleImpact of new technologies and specific circumstances will be captured

PlanningEstimate will be integrated into earned value and other project management methodsAn integrated estimating methodology will be supported

Data WarehousingPast project acutals will be sorted and accessed for more accurate, more defensible estimates


Accurate Estimating RequiresAttention to Sizing

COTS entrySizing based on characteristics, a completely different approach to scope.

Size by ComparisonHighly accurate method of sizing new pieces of software against known past items.

AnalogiesSize new jobs against standardized benchmarks.

COTS Estimating By Characteristic

+ Characteristic Rating - Rigor of requirements Very High - Number of interfaces 70-100 - Complexity Very High - Application Domain Aero - Application Type Signal processing - >



Characteristic Rating

Specify COTS Characteristic

Rigor of requirements High

OK Cancel

Next >>

Help


Detailed Sizing Workups

Rather than one or two major entries, many separate sizing sub-entries can be made within any particular program being estimated. A wide range of sizing methods can be combined or used alongside each other.

Result Estimates can be top-down (general)OR bottom-up (detailed).

COTS Estimating By Characteristic+ SIZE Metric Quantity Legacy - Component A SLOC 1700 Pre-existing - Component B - part of A SLOC 120 Pre-existing - Component C ELOC 450 Pre-existing - Component D Classes 12... New - Component E SLOC 60 New - Component F - part of A Use Cases 2... Pre-existing - Component G COTS Sizing select for info COTS - Component H SLOC 15000 Pre-existing - Component I SLOC 300 Pre-existing - Component J - part of A Function Based select for info Pre-existing - Component K Detailed Objects 45... Pre-existing - Component L SLOC 1200 Pre-existing

Sizing View


SEER NexGens ProjectMinerPreliminary Proven Performance

Within 12%

Within10%

Within 20%

These are actual results on data that would be used in a production system. All data and tests are available for independent review.Please note that final results should be better, as the ProjectMiner methodology is still being prototyped.*

Using extremelylimited, relevant

past data.Results obtained using 5 aviation data points and

Least Absolute Error estimation.

Using extremely limited, relevant

past data, with good assumptions about

predicted project.Results obtained using

maximum information pair-wise comparisons, with accurate

conjecture on probable range.Using sufficientrelevant past data.

Results obtained using neural network.

With a modest amountof NAVAIR data.

Using a preliminary version of ProjectMiner, mixing Least Absolute Error, Weighted

Least Squares and Maximum Likelihood .

With NO relevant past data.Results regularly obtained using SEER-SEM,

in the future also using ProjectMinerconsensus estimates coupled with the

Development Factors model.


ProjectMiner Concept Many Approaches Make a Better

Estimate

Functional Forms

Estim

atin

g M

etho

ds

Data

Set

BestModels,Methods& Data

SettingsFor

CurrentEstimate

CombinedResults of

VariousStatisticalMethods

CombinedResults of

VariousStatisticalMethods

y=a*xb, etc

Multiplestatisticalmethods,

etc

NAVAIR,contractors,

industry,etc


ProjectMiner How It Works

Unlimited numberof core models

PastProject Data

DetermineRelevant

Data

DevelopPotentiallyRelevant

Equations

Apply Data andEquations To

EstimatingMethods

Adjust BestEstimate With

Empirically DerivedAdd-On Models

AssessPerformance

Of EachApproach

IntelligentlyCombine

BestApproaches

Unlimitednumber of add-

on models


Key Leveraging Technologies: Voting and Conformal Estimating

Voting finds the most accurate estimating

approaches.

Voting finds the most accurate estimating

approaches.

Conformal estimating keeps unique info about past projects and then uses that to improve

future estimates.

Conformal estimating keeps unique info about past projects and then uses that to improve

future estimates.oth

ercom

pariso

n

regres

sion

NAVAIR data

Contractor data

Other data

Combines the best methods, paying particular attention to:

The most relevant data sources.

Embedded intelligence about performance of specific methods and models against

specific data.

StandardInfo:

Size, Platform,Application, etc.

RESULT:A much more accurate estimate.

Invaluable additional info

UniqueInfo:

Specific Tools,Circumstances, etc.


Resolving Concerns Overa Data-Driven Approach

We dont know the sizeSize by comparison against 10-100s of similar projects an accurate, proven method.A large number of sizing methods offers many solutions to project scoping.

Theres not enough past dataNumerous estimating methods (least absolute error, etc) are specialized for small samples.

The data is very noisyModels & methods are carefully combined to produce a more robust estimate.Weighted least squares and other techniques help.

We dont have any dataBut the model does: several thousand, many from NAVAIRs own contractors.

Nothing may be relevantEstimating methods dont work that way they can find relevance even in different sorts of data.SEER parametric modeling is the backup for when nothing else works!

30 January, 2003 36th ADoDCAS 27Potential variation in duration

Design

Prototype

Spec

Code

Unit Test

______________________________________________________________________________________Month 3 6 9 12 15 18 21 24Midcourse Calibration

Quick EstimateCalibration Dialog

Activity Duration Chart

Midcourse Calibration

Phase duration estimates are

updated on the chart, with risk ranges (in red)

Dialog lets estimator input earned value

information for auto updating MS

Project


wInsight Earned Value Will Integrate With NexGen


Custom Lifecycles:Data-Driven or User-Defined

Lifecycle DesignerLifecycle Designer

Activities and labor allocations can be formulaic, fixed, or automatically determined by ProjectMiner.

Example - add FQT or systems integration.


Accurate Scheduling Feature

A set of estimated WBS elements

Determine which tasks should follow which, how critical the relations are

Determine which tasks should follow which, how critical the relations are

A set of properly scheduled elements

Automatic optimal scheduling


Software Estimation & Project Management

MS Project

S/W Estimation

MS Project w/ Software Estimation


Estimation and Transformation of Microsoft Project

Saves time by automatically laying out highly tailored plans in Microsoft Project

Automatically connects with estimation tools to obtain sophisticated software project resource & schedule estimates

Encourages best practices and uniform estimating by connecting project planners with enterprise information

Perfectly complements Project 2002s enterprise and usability focus

Arrow in the Quiverfor ProcessImprovement


Data-Driven Estimates Against Past Projects

..\..\..\business\01-016 NAVAIR SBIR N01-020 Software Estimating\Deliverables and Briefings\August 28th Briefing\scatterplot_run.avi


Data Collection Activities

NAVAIR is soliciting additional software data Will benefit the entire community To participate contact:

Rick Holcomb AIR-4.2 Cost Department


Navair Software Estimation Improvement Program

Documents

Transcript of Navair Software Estimation Improvement Program