Local Bias and its Impacts on the Performance of Parametric
Estimation Models
Accepted by PROMISE2011 (Best paper award)Ye Yang, Lang Xie, Zhimin He (iTechs)Qi Li, Vu Nguyen, Barry Boehm (USC)
Ricardo Valerdi (MIT)
2
Agenda
Background Research questions Measuring local bias Measuring the impacts of local bias Handling Local Bias Conclusions and future work
3
Background
COCOMO II model Proposed by Dr. Barry Boehm; one of the most accurate cost
estimation models; widely adopted by industry. Typical parametric estimation model, need tune parameters
against local data (local calibration)
5
1
170.01
1
ii
B SF
jj
Effort A Size EM
Organization 1
Organization 2
General Model
4
Background (Cont.)
Model usage circle Local calibration relies on local historical data and domain
knowledge, i.e. with local assumptions. In most cases, such local assumptions vary from the general
model assumptions. It is possible that the mismatches between “general assumptions” and “local assumptions” will result in surprising calibration results.
Model Localization
Model Usage
Model Calibration
Model Building
General assumptions
Underlying model
Local data
Calibrationdata
Local assumptions
Model updates
Historical data
E.g., counter-intuitive calibration results: negative values of regression coefficients for level of programmer capability (PCAP), indicating higher PCAP leads to higher effort.
5
Research questions
Research questions: Is there a way to measure the local bias introduced in the
model localization (local calibration) stage? As the historical data accumulates from multiple companies,
how will the associated local bias impact the performance of the general parametric estimation model?
Are there any correlation patterns between local bias and model performance variation after incorporating local dataset into the calibration dataset?
Assumptions: The general parametric model follows a similar structure as the
COCOMO II. In model localization stage, constant A and constant B are
tuned with local data. In model usage stage, locally calibrated A and B are used for
project estimation.
6
Measuring local bias
Definition of local bias:
where A’and B’are model parameters calibrated from local data of each organization, A and B are default constant values of COCOMO II model (A=2.94, B=0.91), and in our study we set Size=100KLOC.
' '| ln( ) | | ln( ) ( ' ) ln( ) |
Effort Alocalbias B B Size
Effort A
7
Measuring local bias (cont.)
Data sets CII 2010 data set; contains two subsets: the CII2000 subset
(161 data points from 16 organizations) and the After2000 subset (92 additional data points newly collected from 10 different organizations since year 2000)
Characteristics CII2000 Subset
CII 2010 Dataset
# data points 161 253# organizations 16 23
Size(KSLOC)min 2.6 1.68max 1292.8 2505.2median 46.92 45.28
Effort(PM)min 6 3.5max 11400 11400median 192.5 170
8
Measuring local bias (cont.)
Analysis procedure Divide After2000 subset into 10 groups according to their
corresponding organization. For each group, we conduct a representative local calibration
using data in that group only and produce its local A’ and B’. Calculate the corresponding local bias value of each group. Compare local bias values among all groups.
CII 2000 SubsetAfter2000 Subset
Subset1
…
A, BA1’, B1’ A2’, B2’ An’, Bn’
local_bias1 local_bias2 local_biasn
CII 2010Dataset
Subset2
Subsetn
Group by Organization_IDDefault Constants: A, B
9
Measuring local bias (cont.)
Parameters of local models: Local bias of each group:
Different local A and B in each group, indicating local bias introduced when adopting local calibration;
Local bias varies in different group, ranging from 0.06 to 2.25; the local bias measures how much relative error the corresponding local model will produce.
10
Measuring the impacts of local bias
Analysis procedure First, for each group ssi in the After2000 subset:
1. combine ssi with CII 2000 data set to produce a new data set dsi ;
2. Assessing model performance on data set dsi , record values of performance indicators;
Then conduct correlation analysis between local bias and model performance
CII 2000 subsetI SS1 Performance Local bias
CII 2000 subsetI SS2 Performance Local bias
…… …… ……
Correlation analysis
11
Measuring the impacts of local bias
Performance assessment Basic performance indicators: MMRE (mean MRE), stdMRE (the
variance of MRE) Assessment procedure:
In our study, we employ Average MMRE, Range of MMRE, Average stdMRE, and Range of stdMRE to assess the performance of an estimation model.
Spliting data set into training set
and test set
Tuning model parameters on
training set
Evaluating model
performance on test set
MMRE, stdMRE
Average MMRERange of MMREAverage stdMRERange of stdMRE
Repeat the above steps for 2000 times
2000 (MMRE, stdMRE) pairs
12
Measuring the impacts of local bias(cont.)
Model performance
13
Measuring the impacts of local bias(cont.) Spearman correlation coefficients between local
bias and model performance:
At the significant level of p-value less than 0.05, the range of stdMRE is significantly positive correlated with local bias and local_bias*num. Both the average stdMRE and the average MMRE are significantly positive correlated with local_bias*num.
Range of stdMRE reflects the uncertainty of model performance. Hence, the bigger the local bias is, the weaker the performance is.
Range of stdMRE
Average stdMRE
Range of MMRE
Average MMRE
Local bias
Correlation Coefficient 0.7787 0.1677 0.4731 0.1671
p-value 0.0080 0.6435 0.1673 0.6455
Local bias *num
Correlation Coefficient 0.6120 0.8085 0.4731 0.6777
p-value 0.0508 0.0046 0.1673 0.0313
14
Handling Local Bias
Motivation Performance of the general COCOMO II model seriously
decrease on the After2000 subset! Need to calibrate a new version of COCOMO II model on the CII
2010 data set.
CII2000 After2010 CII2010
Pred(20) 0.6211 0.2393 0.4822
Pred(25) 0.6957 0.3152 0.5573
Pred(30) 0.7516 0.3696 0.6126Pred(20) Pred(25) Pred(30)
CE/ 通用格式
CE/ 通用格式
CE/ 通用格式
CII2000After2010CII2010
15
Handling Local Bias (cont.)
Local bias handling approach Assumption : local historical data set with higher local bias
presents more different pattern for cost estimation, and it should be assigned a lower weight when being used for model calibration.
Constraints for weight distribution function Weight=F ( LocalBias )
IF LocalBias =0, THEN Weight =1; IF LocalBias → +∞, THEN Weight → 0; The F should be a decreasing function on interval [0, +∞).
Three functions
11
1F Weight
LocalBias
:
12
1 ln( )F Weight
LocalBias
:
13F Weight LocalBiase
: CE/通用格式
CE/通用格式
CE/通用格式
CE/通用格式
CE/通用格式
CE/通用格式
CE/ 通用格式
CE/ 通用格式
CE/ 通用格式1/(X+1)1/ln(X+1)+11/E^x
16
Handling Local Bias (cont.)
OID LocalBias F1 F2 F3
46 0.03952465 0.96197815 0.962682998 0.961246259
595 0.20628288 0.828992947 0.842074323 0.813602892
179 0.276339409 0.783490655 0.803861012 0.758555426
14 0.302222798 0.767917749 0.791093772 0.73917336
106 0.302948518 0.767490032 0.790745252 0.738637122
99 0.568446596 0.637573509 0.689614414 0.566404611
93 0.726123018 0.579332985 0.646881635 0.483780969
590 1.009427633 0.49765415 0.588980208 0.364427506
599 1.820976691 0.354487154 0.490897974 0.161867579
597 2.190999501 0.313381434 0.462891345 0.1118049441 2 3 4 5 6 7 8 9 10
CE/ 通用格式
CE/ 通用格式
CE/ 通用格式
LocalBias1/(X+1)1/[ln(X+1)+1]1/e^X
Weight assigned to each organization
17
Handling Local Bias (cont.) Model performance on the CII2000 subset
COCOMOEqual
Weights F1 F2 F3
Pred(20) 0.6211 0.4907 0.5093 0.5031 0.5342
Pred(25) 0.6957 0.5963 0.6149 0.6149 0.6273
Pred(30) 0.7516 0.677 0.7205 0.7143 0.7081
Model calibrated with equal weights performs worst on the CII2000 subset;
The general COCOMO II model performs best;
Pred(20) Pred(25) Pred(30)CE/ 通用格式
CE/ 通用格式
CE/ 通用格式
COCOMO IIEqual weightsFunction-1Function-2Function-3
18
Handling Local Bias (cont.) Model performance on the After2000 subset
COCOMOEqual
Weights F1 F2 F3
Pred(20) 0.2393 0.25 0.2609 0.2717 0.2609
Pred(25) 0.3152 0.3261 0.3261 0.3261 0.3152
Pred(30) 0.3696 0.3804 0.4022 0.4022 0.4022
The general COCOMO II model performans worst on the After 2000 subset
Models calibrated with weights exhibit better performance than models calibrated without weights.
Pred(20) Pred(25) Pred(30)CE/ 通用格式
CE/ 通用格式
COCOMO IIEqual weightsFunction-1Function-2Function-3
19
Handling Local Bias (cont.) Model performance on the whole CII 2010 data set
COCOMOEqual
Weights F1 F2 F3
Pred(20) 0.4822 0.4032 0.419 0.419 0.4348
Pred(25) 0.5573 0.498 0.5099 0.5099 0.5138
Pred(30) 0.6126 0.5692 0.6047 0.6008 0.5968
The general COCOMO II model works better on the whole CII 2010 data set than calibrated models;
Models calibrated with weights exhibit better performance than models calibrated without weights.
Pred(20) Pred(25) Pred(30)CE/ 通用格式
CE/ 通用格式
CE/ 通用格式
COCOMO IIEqual weightsFunction-1Function-2Function-3
20
Conclusions
The proposed LocalBias measure can be used to quantitatively measure and analyze potential local bias associated with individual organization data subset in the overall dataset.
As historical data accumulates from multiple companies, the associated local bias will cause the range of stdMRE increase.
The correlation analysis verifies that the model performance is significantly correlated by the degree of local bias and the number of data points associated with each additional group.
Weight calibration helps to reduce impact of local bias and thus improve the usability of cross-company data for model calibration.
21
Future work
More empirical studies on other public dataset to future validate and refine results.
Develop more effective methods for reducing local bias and improving general calibration outcomes.
Thanks!Q&A
Top Related