of Gun” Software and Systemc-spin.net/2010/cspin201009Top_Gun_Presentation_Numetrics.pdf ·...
46
1 Best Practices of “Top-Gun” Software and System Developers Alex Silbey VP, Client Engagements Numetrics Chicago SPIN Sept. 15, 2010
Transcript of of Gun” Software and Systemc-spin.net/2010/cspin201009Top_Gun_Presentation_Numetrics.pdf ·...
1
Best Practices of
“Top-Gun” Software and System Developers
Alex Silbey
VP, Client Engagements
Numetrics Chicago SPIN Sept. 15, 2010
2
―When realistic opinions are suppressed, while optimistic ones are rewarded, an organization’s ability to think critically is undermined‖
―In planning major initiatives, executives routinely exaggerate the benefits and discount the costs, setting themselves up for failure.‖
Source: Lovallo & Kahneman ―Delusions of Success,‖ Harvard Business Review
Thoughts About Planning
About Numetrics
A software company (headquartered in Cupertino, Calif.,)
specializing in tools that calculate SW & HW project
schedules and resource plans for embedded systems
projects
Productivity and predictability for embedded SW & HW
team performance for 10 years
Built an industry project database for SW & HW products
– Time, effort, product descriptions, team environment….
Developed models on this data to measure SW & HW
complexity & productivity
– For estimation & benchmarking purposes
About Today
Share with you:
– Best-in-class performance – what a ―Top Gun‖ looks like
– Common root causes preventing "Top-Gun" performance
– Management practices to address root causes
– Impact of those practices
Invite you participate in an experiment
– EE Times’ Productivity Initiative
– Measure effectiveness of "Top-Gun" practices
Outline
What it takes to be a ―Top Gun‖
Why care?
The eight "Top-Gun" best practices
Q & A
Data source: >2,000 Embedded HW/SW Projectsin Numetrics’ Industry Databases
Wireless
– Antenna interface/RF
– Broadcast satellite
– Voice and data
Wired
– Networking telecom
– Line interface
Computers
– CPU peripherals
– Processors
– Graphics controllers
– Bus/interface controllers
– Memory/programmable products
Entertainment
– Home/professional entertainment
– Displays
– Portable
– Photography
Military
– Avionics/satellite
– Weapons systems
– Communications
– Transportation
Computer System Peripherals
– Peripheral controllers
– Storage products
Transportation
– Automotive infotainment
– Automotive powertrain/chassis
– Automotive body/convenience systems
Industrial Products
– Embedded control
– Security/ID & electronic currency
– Power devices
– Medical electronics
– Sensors
Schedule Predictability—a Candid View
60% of projects slip at least one quarter
16% of projects slip more than one year-2
5
-10 0 5
10
20
30
40
50
60
70
80
90
100
More
Schedule Slip (Weeks)
% o
f pro
jects
“Top-Gun” Zone
10%
Profile of ―Top-Gun‖ HW Engineering Manager
―Top-Gun‖ HW managers have
– 119% better project productivity
– 40% shorter development cycle times
– 50% fewer defects
– 50% less schedule slip
Better project performance translates
directly into increased competitiveness
Bar
s: P
erce
nta
ge o
f p
roje
cts
Blu
e cu
rve:
Pro
du
ctiv
ity
Schedule slip in months
15%
10%
5%
0%
1000
1150
1300
1450
1600
850
13.1%
0 4 8 12 16
“Top Guns”
Profile of a "Top-Gun" Embedded SW Manager
Residual Defects
The “Top Guns”:
- Deliver on time (< 1 month slip)
- Achieve 38% higher productivity
- 6x fewer residual defects
Outline
What it takes to be a ―Top Gun‖
Case Study
– Can better management practices make us ―Top
Guns?‖
The eight "Top-Gun" best practices
Q & A
The Case Study:
Selected 200+ new embedded system projects
covering a range of end equipment
– Half of the projects applied "Top-Gun" methods
– Half applied ―traditional‖ methods
Compare
Impact of ―Top-Gun‖ Methods on Schedule Slip
Schedule Slip of Projects using
“traditional” methods
Schedule Slip of Projects using
“Top-Gun” methods
Schedule slip on projects applying ―Top-Gun‖ methods was reduced by more than 50%
Impact of ―Top-Gun‖ Methods on Cycle Time
Projects applying ―Top-Gun‖ methods exhibit 5%-10% shorter cycle times (for equal complexity)
Cycle Time of Projects using
“traditional” methods
Cycle Time of Projects using
“Top-Gun” methods
Impact of "Top-Gun" Methods on Productivity
Projects managed using ―Top-Gun‖ methods achieved 38% lower cost due to substantially higher productivity
Productivity of Projects using
“traditional” methods
Productivity of Projects using
“Top-Gun” methods
Outline
What it takes to be a ―Top Gun‖
Case Study
– Can better management practices make us ―Top
Guns?‖
The eight ―Top-Gun‖ best practices
Q & A
Origins of the ―Top-Gun‖ Practices
Numetrics has ―benchmarked‖ embedded
systems projects for over 10 years.
• Schedule slip, productivity, cycle time…
Identified the four ―root causes‖ of poor project
performance
– Underestimation of project complexity
– Overestimation of productivity
– Resources not available as planned
– Changes during the project
―Top-Gun‖ practices prevent or minimize impact
of these root causes.
Where ―Top Guns‖ Operate
―Top-Gun‖ managers use best practices
throughout the project life cycle
We define project duration from requirements
definition to release-to-volume
Spec Implementation Verification Validation Productization
A systematic method for calculating project
complexity is essential
ProjectDescription
Complexity Calculation
Project ComplexityRating
“Top-Gun” Practice #1: Compute project complexity statistically
Tackles the ―complexity is underestimated‖ problem
Software Complexity
Test PlanCustomer
Requirements
Project Type
ProgrammingLanguage
ProjectScope/Activities
EndEquipment
OpenSource
Object CodeFootprint
Lines of Code
Function Points
SW Components& Architecture
Functional Requirements
Software Complexity Calculation
Test Reuse
Code Reuse
HW PlatformAttributes
Sizing Software Complexity
System-Level Test Cases
Unit/Regress.Test Cases
Defects
Number ofVariants
Release Strategy
Many size measures are available. Numetrics includes all the above in its complexity rating.
Key Requirements for Rating Complexity
Complexity inputs must be objective &
measurable
Complexity inputs must be knowable early-on
Impact of inputs must be calibrated using real-
world data
– Implies need of a project database
―We have far too many metrics in our industry but not much reliable data‖ Capers Jones, June 2010
Accurately compute
project complexity
1
―Top-Gun” practice #1:Compute project complexity statistically
Spec Implementation Verification Validation Productization
ProjectDatabase
EstimationModels
Estimated Plan Staffing Profile & Timeline
“Top-Gun” Practice #2:Estimate resources and schedule—based on models
Models embody real-world history
Models help guide intuition and experience
Simple models: LoC/day, LoC/feature, effort/defect…
Complex models include: impact of # of sites, team size, experience…
Constraints&
―ProductivityFactors‖
Complexity Description
Project &
resource
planning
Prior projects database
2
“Top-Gun” Practice #2:
Resource planning using models & data
Compute
project
complexity
1
Spec Implementation Verification Validation Productization
“Top-Gun” Practice #3:Rigorously perform ―what-if‖ scenarios for schedule/resource optimization
Requires ability
to accurately
trade off
between project
constraints
Schedule
StaffingFeatures
(Complexity)
Quality
- Explore options, pick the best
- Calculates impact of uncertainties & unknown
- Leads to ―balance‖ among all 4 constraints
Project planning & many “what-if”
scenarios
Prior projects database
3
2
"Top-Gun" Practice #3:Do several ―what-if‖ scenarios
Compute
project
complexity
1
Spec Implementation Verification Validation Productization
Test your project execution assumptions against
your own history and industry data
Manager’s Staffing & Schedule
Product Description
Inputs
Test Assumptions
Against Reality
Your Assumptions
And Targets
"Top-Gun" Practice #4:Benchmark project execution assumptions
Industry benchmarking
database
User’s projects database
Cycle Time
Team Size
Productivity
Complexity
# Tests
Spec-Change Rate
Defect Rate
ExampleExecution
Assumptions
What are Your Team’s Current Capabilities?
Actual measured
productivity levels
of teams on
similar projects
Productivity vs. Team Size
Team Size
Productivity vs. Team Size
How Realistic are Your Targets?
New project in
planning
Benchmark assumed productivity in your
bottom-up plans
Team Size
How Risky (Realistic) is the Target?
Benchmark assumed productivity in your
bottom-up plans
Productivity vs. Team Size
Schedule Risk
Team Size
Benchmark planning
assumptions
Project planning &
many what-if
scenarios
Prior projects database
4
32
"Top-Gun" practice #4:- Benchmark project plan assumptions
- Measure project risk
Compute
project
complexity
1
Spec Implementation Verification Validation Productization
Manager must know what is the highest
achievable productivity of his/her team
– Schedule
– Productivity
– Complexity
Industry norm
Aggressive
Unrealistic!
"Top-Gun" Practice #5:Set the most aggressive, yet achievable, targets
Your Project Plan
How much risk can you accept?
Benchmark assumed productivity in your
bottom-up plans
Productivity vs. Team Size
Schedule Risk
Team Size
Optimal target: Stretch But Don’t Break
New project in
planning
Achieve productivity excellence
Optimal
productivity target
Team Size
Target setting
Benchmark planning
assumptions
Project planning &
many what-if
scenarios
Prior projects database
4
32
"Top-Gun" Practice #5: Set aggressive yet achievable targets based ondatabase & models
5
Compute
project
complexity
1
Spec Implementation Verification Validation Productization
Compute cost of each request in terms of
schedule slips or resources by project phase
"Top-Gun" Practice #6Quantitatively assess schedule/resource impact of each feature request
Changes are inevitable … but
―Top-Gun‖ managers carefully assess impact
―Top-Gun‖ managers carefully also track ―rate
of changes‖
Top Down Planning
Staying on Track ―Top Guns‖ monitor progress & assess impact of changes
"Top-Gun" Project Manager
Initial
Project Planning
Assumptions
MonitorPlanning Assumptions
& Product Maturity
Defect Rate
Spec Change
Spec-Change Rate
LOC Count
Test Count
Object Code Size
Staffing Level
Assumptions
Still Valid?No
Update
Project Planning
Assumptions
Estimate Resource &
Schedule Estimates
Benchmark Execution
Assumptions
37
Company SW Projects
27-Apr-2010. Information confidential and proprietary to Numetrics Management Systems, Inc.
Analysis Table
Number of
Observations10
X-Mean 0.93
Y-Mean 1,360
R2 0.16
Example: Effect of Defect Rate on Productivity
Monitoring defect rate enables productivity (effort) estimates to be refined
Benchmark planning
assumptions
Project planning &
many what-if
scenarios
Prior projects database
4
3
"Top-Gun" Practice #6Monitor assumptions; assess impact of changes
Compute
project
complexity
1
Re-planning
62
Target setting5
Spec Implementation Verification Validation Productization
"Top-Gun" Practice #7:Diagnose root causes of poor performance
Store and organize data from prior projects
Mine data and extract insight
Schedule slip vs. spec stability
Identify & eliminate barriers that lower team productivity
Schedule
slip (
% o
verr
un o
f ori
gin
al pla
n d
ura
tion
Assessing schedule risk
Project planning &
many what-if
scenarios
Prior projects database
4
3
Best Practice: Rigorous Root-Cause Analysis
Compute
project
complexity
1
Re-planning
62
Root-cause
analysis
7
Target setting5
Spec Implementation Verification Validation Productization
“Top-Gun” Practice #8:Foresee resource shortfall across pipeline
Total resources
today
Resource contention is the third-leading cause of schedule slip
Assessing schedule risk
Project planning &
many what-if
scenarios
Prior projects database
4
3
"Top-Gun" Practice #8:
Foresee resource shortage/conflict across pipeline
5
Compute
project
complexity
1
Re-planning
62
Root-cause
analysis
7
Monitor project pipeline for
resource conflicts8
Target setting5
Spec Implementation Verification Validation Productization
The "Top-Gun" Practices: Operational Disciplines
Compute project complexity
statistically
Resource planning using
models and data
Generate multiple scenarios
using ―what-if‖ analysis
Benchmark project
execution assumptions4
3
2
7
6
5
8
Generate most aggressive,
yet achievable, project plan
Quantitatively assess impact
of changes
Perform root-cause analysis
when project finishes
Analyze multi-project
execution pipeline to identify
resource shortfalls/conflicts
1
Summary
Research into root causes of poor project performance led to
the identification of eight best practices ("Top-Gun" practices).
A case study involving 200+ Embedded System projects
determined the "Top-Gun" practices are strongly correlated to
superior project performance
– 38% higher productivity, 55% lower slip, 5-10% shorter duration
Next step: An experiment to apply "Top-Gun" practices to
another community of embedded SW projects
You’re invited to participate!
How You Can Participate
1. Complete a survey with questions about a recently finished
(or nearly finished) embedded SW project
– Duration, effort, complexity & management methods
– ~ 30 minutes
– Establishes a ―productivity baseline‖
– You will receive
• $25 Amazon gift card
• Access to aggregated industry results ($1,295 value)
2. Apply the "Top-Gun" practices on your next SW project &
benchmark your results
Contact [email protected] to find out how
