Understanding the Human Estimator Gary D. Boetticher [email protected] Univ. of Houston - Clear...
-
Upload
keegan-delany -
Category
Documents
-
view
214 -
download
0
Transcript of Understanding the Human Estimator Gary D. Boetticher [email protected] Univ. of Houston - Clear...
Understanding the Human Estimator
Gary D. Boetticher [email protected]. of Houston - Clear Lake, Houston, TX, USA
http://nas.cl.uh.edu/boetticher/publications.html The 2nd International Predictor Models in Software Engineering (PROMISE) Workshop
Nazim Lokhandwala [email protected]. of Houston - Clear Lake, Houston, TX, USA
James C. Helm [email protected]. of Houston - Clear Lake, Houston, TX, USA
Introduction
Chaos Chronicles [Standish03]
300 billion dollars
250,000 new projects
1.2 million dollars per project
http://nas.cl.uh.edu/boetticher/publications.html The 2nd International Predictor Models in Software Engineering (PROMISE) Workshop
Boehm’s 4X
http://nas.cl.uh.edu/boetticher/publications.html The 2nd International Predictor Models in Software Engineering (PROMISE) Workshop
Types of Estimation [Jorgenson04]
63 - 86% Human-Based
7 - 16% Algorithmic and Machine
Learners
http://nas.cl.uh.edu/boetticher/publications.html The 2nd International Predictor Models in Software Engineering (PROMISE) Workshop
Research Focus
Number of Papers On Software Estimation in IEEE [Jorgenson02]
Human-Based Estimation (17%)
Other (83%)
http://nas.cl.uh.edu/boetticher/publications.html The 2nd International Predictor Models in Software Engineering (PROMISE) Workshop
Statement of Problem
How do human demographics affect human-based estimation?
Can predictive models be constructed using human demographics?
http://nas.cl.uh.edu/boetticher/publications.html The 2nd International Predictor Models in Software Engineering (PROMISE) Workshop
Investigation Procedure
Collect demographics from participants
Request participants to estimate software components
Build models (Estimates vs. Actuals)
Su
rvey
http://nas.cl.uh.edu/boetticher/publications.html The 2nd International Predictor Models in Software Engineering (PROMISE) Workshop
Which Demographics?
Basic Demographics
Academic Background
Work Experience
Domain Experience
http://nas.cl.uh.edu/boetticher/publications.html The 2nd International Predictor Models in Software Engineering (PROMISE) Workshop
The Survey
http://nas.cl.uh.edu/boetticher/EffortEstimationSurvey.html
http://nas.cl.uh.edu/boetticher/publications.html The 2nd International Predictor Models in Software Engineering (PROMISE) Workshop
Competitive Procurement Software
Buyer Admin
Buyer1
Buyern
...
Buyer Software
DistributionServer
Supplier1
Supplier2
Suppliern
:
SupplierSoftware
http://nas.cl.uh.edu/boetticher/publications.html The 2nd International Predictor Models in Software Engineering (PROMISE) Workshop
Sample Estimation Screenshots
http://nas.cl.uh.edu/boetticher/publications.html The 2nd International Predictor Models in Software Engineering (PROMISE) Workshop
Survey Results Screenshots
http://nas.cl.uh.edu/boetticher/publications.html The 2nd International Predictor Models in Software Engineering (PROMISE) Workshop
Data Collection
Invitations
Filtered Incomplete Records
122 Final Records
http://nas.cl.uh.edu/boetticher/publications.html The 2nd International Predictor Models in Software Engineering (PROMISE) Workshop
Participant Educational Background
Most of the
participants hold
Bachelors or
Masters Degrees
Mean MaximumStandard Deviation
Computer Science
Undergrad Courses 8.8525 70 11.6326
Grad Courses 2.4262 15 3.2293
Hardware
Undergrad Courses 3.5246 64 8.0209
Grad Courses 0.5000 10 1.3252
Management Information Systems
Undergrad Courses 0.7705 12 1.5892
Grad Courses 0.4918 9 1.3742
Project Management
Undergrad Courses 0.2951 4 0.6886
Grad Courses 0.8115 6 1.1806
Software Engineering
Undergrad Courses 0.9180 7 1.2958
Grad Courses 2.1557 21 3.1202
http://nas.cl.uh.edu/boetticher/publications.html The 2nd International Predictor Models in Software Engineering (PROMISE) Workshop
Participant Work Experience
Mean Maximum
Standard Deviation(Years) (Years)
Years of Experience As
Hardware Project Manager 0.6557 15 1.9251
Software Project Manager 1.3443 10 2.0811
No of Projects estimated
Hardware Projects 0.8279 20 2.6307
Software Projects 2.9508 28 4.4848
http://nas.cl.uh.edu/boetticher/publications.html The 2nd International Predictor Models in Software Engineering (PROMISE) Workshop
Participant Domain Experience
2.2512200.7274 Process Industry
1.3818100.6209 Procurement and Billing
Domain Experience
Standard Deviation
Maximum(Years)
Mean
(Years)
http://nas.cl.uh.edu/boetticher/publications.html The 2nd International Predictor Models in Software Engineering (PROMISE) Workshop
Data Preparation
INPUT= 69% zeros…Needs Consolidation
Courses, Workshops, Conferences, Programming Exp.
45 attributed reduced to 14 attributes Highest Degree Achieved…Need Transformation
9 4 2 0 0 0 1 0 0 0 9 1 0 0 0 0 0 0 0 0 0 0 1 0 1 1 0 9 0 0 2 0 0 0 0 0 0 3 0 7 4 2 4 0 0
0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 1 0 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 5 0 5 0 5 2 0
0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 1 2 0 0 0 0 0 0 0 0 0 0 0 0 4 0 0 0 0 0 0 1
0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 2 0 1 1 1 0 2 0 0 1 0 0 1 0 0 0 4 0 0 0 0 5 1 0
OUTPUT=MRE=Abs (Total Actual – Total Est.)/(Total
Actual)
http://nas.cl.uh.edu/boetticher/publications.html The 2nd International Predictor Models in Software Engineering (PROMISE) Workshop
Build Models
Linear Regression (Excel)
Non-Linear Regression (DataFit)
Genetic Programming (GDB_GP)
http://nas.cl.uh.edu/boetticher/publications.html The 2nd International Predictor Models in Software Engineering (PROMISE) Workshop
GP Configuration
3 Settings 1000 Chromosomes 50 Generations 512 Chromosomes 128 Generations 1000 Chromosomes 128 Generations
http://nas.cl.uh.edu/boetticher/publications.html The 2nd International Predictor Models in Software Engineering (PROMISE) Workshop
20Trialseach
Results: All Demographic Factors
1.87E-15 3.45E-17T-test
0.88470.55920.1550Mean
Non-Linear Regression
Genetic Programming
Linear Regression
1.6470
0.8847
Non-Linear Regression
Std. Error
R Squared
1.38754.4580
0.9174 0.1550
Genetic Programming
Linear Regression
Best Values of R Squared with Min. Std. Error
T-Test between Average R Square Values
http://nas.cl.uh.edu/boetticher/publications.html The 2nd International Predictor Models in Software Engineering (PROMISE) Workshop
Results: Educational Factors
0.0486 2.74E-13T-test
0.21360.19730.0373Mean
Non-Linear Regression
Genetic Programming
Linear Regression
4.1667
0.2136
Non-Linear Regression
Std. Error
R Squared
3.97384.6101
0.27840.0373
Genetic Programming
Linear Regression
Best Values of R Squared with Min. Std. Error
T-Test between Average R Square Values
http://nas.cl.uh.edu/boetticher/publications.html The 2nd International Predictor Models in Software Engineering (PROMISE) Workshop
Results: Work Experience
1.54E-11 2.73E-19T-test
0.36980.55640.0596Mean
Non-Linear Regression
Genetic Programming
Linear Regression
4.0644
0.3698
Non-Linear Regression
Std. Error
R Squared
2.28554.5169
0.75720.0596
Genetic Programming
Linear Regression
Best Values of R Squared with Min. Std. Error
T-Test between Average R Square Values
http://nas.cl.uh.edu/boetticher/publications.html The 2nd International Predictor Models in Software Engineering (PROMISE) Workshop
Results: Domain Experience
4.55E-16 3.27E-23T-test
0.32600.54050.0243Mean
Non-Linear Regression
Genetic Programming
Linear Regression
3.9091
0.3260
Non-Linear Regression
Std. Error
R Squared
2.92834.5425
0.59110.0243
Genetic Programming
Linear Regression
Best Values of R Squared with Min. Std. Error
T-Test between Average R Square Values
http://nas.cl.uh.edu/boetticher/publications.html The 2nd International Predictor Models in Software Engineering (PROMISE) Workshop
Summary of All Experiments
R Square Values
Linear Regression
Best Case Genetic Prog.
Avg. Case Genetic Prog.
Non-Linear Regression
All Factors 0.1550 0.9174 0.5592 0.8847
Education Only 0.0373 0.2784 0.1973 0.2136
Work Experience Only 0.0596 0.7572 0.5564 0.3698
Domain Experience Only
0.0243 0.5911 0.5405 0.3260
http://nas.cl.uh.edu/boetticher/publications.html The 2nd International Predictor Models in Software Engineering (PROMISE) Workshop
Best Equation: All Factors. r2 = 0.9174((Log (TechGradCourses + (TechGradCourses ^ ((Log TotWShops)/(Cos (TechGradCourses ^ ((ProcIndExp + (Cos (TechGradCourses ^ ((ProcIndExp + (Log (Log (TechGradCourses ^ (TechGradCourses ^ (Cos (Log (Log (TechGradCourses ^ (Cos (Log (Log (Log SWProjEstExp))))))))))))) / (TechGradCourses ^ (Log SWProjEstExp)))))) / (((Cos (TechGradCourses ^ ((ProcIndExp + (Cos (TechGradCourses ^ ((ProcIndExp + (Log (Log (TechGradCourses ^ (TechGradCourses ^ (Cos (Log (Log (TechGradCourses ^ (Cos (TechGradCourses ^ ((ProcIndExp + (((ProcIndExp + (Log (Sin MgmtGradCourses)))/(Sin SWPMExp)) + (Sin ((Cos (TechGradCourses ^ ((ProcIndExp + (Cos (TechGradCourses ^ ((ProcIndExp + (Log (Log (TechGradCourses ^ (TechGradCourses ^ (Cos (Log (Log (TechGradCourses ^ (Sin SWPMExp)))))))))) / (TechGradCourses ^ (Log SWProjEstExp)))))) / (((Cos (TechGradCourses ^ ((Log SWProjEstExp) / (((Log (ProcIndExp + (Log (TechGradCourses ^ ((Log SWProjEstExp) / (Log SWProjEstExp)))))) - 3) / (ProcIndExp + (TechGradCourses ^ (Cos (TechGradCourses ^ ((ProcIndExp + (Log (Log (TechGradCourses ^ (TechGradCourses ^ (Cos (Log (Log (TechGradCourses ^ (Cos ((((Log SWProjEstExp) / ((ProcIndExp + (Log (TechGradCourses ^ (TechGradCourses ^ (Log SWProjEstExp))))) / (Log (Log (TechGradCourses ^ (TechGradCourses ^ (Cos (Log (Log (TechGradCourses ^ (Cos (Log (Log (Log SWProjEstExp)))))))))))))) / (Sin SWPMExp)) / (Sin SWPMExp)))))))))))) / (TechGradCourses ^ (Log SWProjEstExp))))))))))) - 3) / (TechGradCourses ^ (Log SWProjEstExp)))))) + ((Log SWProjEstExp) / (Log SWProjEstExp)))))) / (Log (Log (Log (TechGradCourses + (Cos (Log (Log (TechGradCourses ^ (Cos (((((Log SWProjEstExp) / (TechGradCourses ^ (Log SWProjEstExp))) / ((ProcIndExp + (Log (Sin MgmtGradCourses))) / ((Log SWProjEstExp) / (Log SWProjEstExp)))) / (Sin SWPMExp)) / (Sin SWPMExp))))))))))))))))))))))) / (TechGradCourses ^ (Log SWProjEstExp)))))) / (((Log ((((Log TotLangExp) / (Log SWProjEstExp)) / (Log SWProjEstExp)) / (Sin SWPMExp))) - 3) / (TechGradCourses ^ (Log SWProjEstExp)))))) - 3) / (TechGradCourses ^ (Log SWProjEstExp)))))))))) + (((((ProcIndExp + (Log (TechGradCourses ^ (Log (TechGradCourses + ((TechGradCourses ^ (TechGradCourses ^ (Cos (TechGradCourses ^ ((ProcIndExp + (Log (Log (TechGradCourses ^ (TechGradCourses ^ (Cos (Log (Log (TechGradCourses ^ (Cos ((((Log SWProjEstExp) / ((ProcIndExp + (Log (TechGradCourses ^ (Log (TechGradCourses + (Cos (Log (Log (TechGradCourses ^ (Cos (((((Log SWProjEstExp) / (TechGradCourses ^ (Log SWProjEstExp))) / ((ProcIndExp + (Log (Sin MgmtGradCourses))) / ((Log SWProjEstExp) / (Log SWProjEstExp)))) / (Sin SWPMExp)) / (Sin SWPMExp)))))))))))) / ((Log SWProjEstExp) / (Log SWProjEstExp)))) / (Sin SWPMExp)) / (Sin SWPMExp)))))))))))) / (TechGradCourses ^ (Log SWProjEstExp))))))) / (Sin SWPMExp))))))) / (TechGradCourses ^ (Log SWProjEstExp))) / (TechGradCourses ^ (Log SWProjEstExp))) / (TechGradCourses ^ (Log SWProjEstExp))) / (Sin SWPMExp)))
http://nas.cl.uh.edu/boetticher/publications.html The 2nd International Predictor Models in Software Engineering (PROMISE) Workshop
Too Much of a Good Thing?
Conclusions
Viability of a human-based est. model Model assessment
Non-linear GP Impact on Human Based Estimation
1) All Factors
2) Domain Experience Work Experience
3) Education
http://nas.cl.uh.edu/boetticher/publications.html The 2nd International Predictor Models in Software Engineering (PROMISE) Workshop
Future Directions
Equation Optimizer for GP Collect More Data
Further analysis without consolidation Detailed Effect of Educational Factors
Use other statistical indicators Build other models
Hybrid (Non-linear and GP) Classifiers
Impact of process on estimation
http://nas.cl.uh.edu/boetticher/publications.html The 2nd International Predictor Models in Software Engineering (PROMISE) Workshop
http://nas.cl.uh.edu/boetticher/publications.html The 2nd International Predictor Models in Software
Engineering (PROMISE) Workshop
Questions?
http://nas.cl.uh.edu/boetticher/publications.html The 2nd International Predictor Models in Software
Engineering (PROMISE) Workshop
Thank You !