© G. Ruhe 2008 1 Günther Ruhe University of Calgary & Expert Decisions Inc. @ Software Engineering...
-
Upload
brittney-hadwin -
Category
Documents
-
view
214 -
download
0
Transcript of © G. Ruhe 2008 1 Günther Ruhe University of Calgary & Expert Decisions Inc. @ Software Engineering...
© G. Ruhe 2008 1
Günther Ruhe
University of Calgary &
Expert Decisions Inc.
@
Software Engineering Institute
Pittsburgh,
January 15, 2008
The Art and Science ofSoftware Release Planning
© G. Ruhe 2008 2
Agenda
Decision Support in Requirements Engineering Software Release Planning EVOLVE* Decision Support System ReleasePlanner® Case Studies Future Research
© G. Ruhe 2008 3
Laboratory for Software Engineering Decision Support
Created in July 2001 at the University of Calgary
Research team of 12 researchers (2 undergraduate, 3 graduate, 5 PhD students, and 2 profs)
Research topics: Decision support (systems) forSoftware release planningProject portfolio planningSoftware resource managementCOTS selectionVerification and validation
Research approach: Interdisciplinary Both fundamental and applied researchEmpirical validation of results
University spin-off company: Expert Decisions
© G. Ruhe 2008 4
Problemdescription
Userrequirements
Developer’srequirements
Systemdesign
Componentrequirements
Component-leveldesign
Unitcode
Executablecomponents
Executablesystem
Usablesystem
Usedsystem
is verified against
is developed into
is integrated into
is validated against
How to allocate test effort?
When to terminate testing ?
Inspection?Re-inspection?
Reuse of components?
Classes? Objects?Methods?
Which technique?Which artifacts?
Decisions, Decisions, Decisions ….
Which requirements?
© G. Ruhe 2008 5
Description of a Decision Problem
Set A = {a1, a2, …} of alternatives (these alternatives are not necessarily described explicitly);
Set G = {g1, g2, …,gn} of criteria to evaluate each alternative a A from different perspectives, and
Preference structure (partial order defined on A)
Main categories of decision problems: Selection (P): Select one alternative a* A or a subset A* A; Triage (P): Assign each alternative a A to one of the
classes C1, C2, …, Ck.
Ranking (P): Arrange all alternatives in A according to an order a1 ≥ a2 ≥ …( a ≥ b means “alternative
a is at least as good as b”). Other classification: Structured versus unstructured decisions Strategic, tactical, and operational decisions
© G. Ruhe 2008 6
Decisions in Requirements Engineering
Activity Decision Level
Identification of business goals StrategicIdentification of stakeholders Tactical Level of involvement of stakeholders Tactical Selection of elicitation techniques Tactical Requirements elicitation Operational Requirements reuse Operational Necessity check Operational Feasibility check OperationalConsistency and completeness check OperationalRequirements negotiation OperationalRequirements prioritization Operational
Requirements documentation Requirements document standard Strategic Identification of problems Operational Actions to address the identified problems Operational Acceptance of requirements Strategic
Elicitation
Requirements analysis and negotiation
Requirements validation
© G. Ruhe 2008 7
Executableprogram
Changerequirement
SoftwareEvolution
Programvariants
PlanningProject goals
Project plan
SoftwareDevelopment
Specification
Information is uncertain inconsistent incomplete fuzzy
Decision space of large size of high complexity is dynamically
changing
Multiple objectives Usability Security Reliability Maintainability Portability
Hard and softconstraints on Time Effort Quality Resources
Why are RE Decisions so Hard?
© G. Ruhe 2008 8
Expected Benefits from Decision Support
Basic assumption
The more qualified computer-based/intelligent decision support is provided based on well-defined processes and roles, the more likely it is to find an appropriate decision.
Decision support systems combine the intellectual resources of individuals and organizations with the capabilities of the computer
– to generate and evaluate (new) solution alternatives;
– to allow more transparent decisions (to be better understood by involved people);
– to analyse decisions pro-actively;
– to better react on changes in the problem parameters (re-planning);
– to make more effective (trade-off) decisions (improved quality);
– to make more efficient decisions (improved cost-benefit).
© G. Ruhe 2008 9
Agenda
Decision Support in Requirements Engineering Software Release Planning EVOLVE* ReleasePlanner® Case Studies Future Research
© G. Ruhe 2008 10
Planning for Software Releases
Incremental (not monolithic) software development
Set of competing (to be implemented) features
Stakeholder prioritize objects in terms of urgency, business value, risk, …
Release cycle time can be predefined or can be open
Limited resources and budgets need to considered
Release: Set of features constituting the next version of the product
Release can be refined into increments (strategic versus operational planning)
Hypothesis: Comprehensive and systematic release planning results in more reliable plans, better customer satisfaction, and more efficient usage of short resource ( = maximum value)
© G. Ruhe 200811
Decision Support for Product Release Planning (1/2)
Uncertainty is pervasive and unavoidable in software engineering. Uncertain software engineering decision problems are unlikely to be explicitly
modeled and completely formalized.
Set
of F
eatu
res/
Obj
ects
Interation 1
Interation 2
Interation 3
Release k
Release k+2
Release k+1
! Stakeholder involvement
! Unclear objectives ! Effectiveness &efficiency
! Resource, risk, andtechnologicalconstraints
© G. Ruhe 2008 12
Decision Support for Product Release Planning (2/2)
Decisions in software engineering are made by humans. They are based on both explicitly formulated and implicitly known objectives and constraints.
The goal of decision support is not to replace human judgment and expertise, but to assist in making better decisions.
Based on uncertain models, any formalized computational technique (subsumed as computational intelligence) in isolation is unlikely to determine meaningful results.
The advantage of the human intelligence based approach is that it is able to better handle soft and implicit objectives and constraints.
Support to solve
− to solve the right problem and
− to solve the problem right.
© G. Ruhe 2008 13
Maturity Factors for Product Release Planning
Processes rigour Level 1 Level 5
Degree of integration into product lifecycle management Level 1 Level 5
Planning based on measurement and defined criteria Level 1 Level 5
Type and degree of stakeholder involvement Level 1 Level 5
Continuity of planning and re-planning Level 1 Level 5
© G. Ruhe 2008 14
Art and Science of Release Planning
Art: Focus on the human intuition and communication– Pro: Flexible, easy to perform, ability to handle soft and implicit
concerns– Con: Effectiveness and efficiency for mid-size to large scale problems,
not transparent and not repeatable, re-planning is not supported, stakeholder involvement limited, resource consumption hard to take into account
Science: Emphasis on formalization of the problem and application of computational algorithms to generate best solutions.– Pro: Effective and efficient, easy re-planning, transparent, repeatable,
comprehensive stakeholder involvement possible, resource consumption part of the solution procedure
– Con: Needs formal description of the problem, needs computer support
© G. Ruhe 2008 15
Analysis of Existing Approaches RP MethodsDimensions
[Greer 2004]
[Penny 2002] [Bagnall et al. 2001]
[Jung 1998] [Karlsson & Ryan 1997]
[Denne & Cleland-
Huang 2004]
[Ruhe & Ngo-The 2004]
Scope 1 release 1 release 1 release 1 release 1 release Chunks of small releases
2 releases planned ahead
Objectives Based on benefit of system changes
Based on benefit of features to customers
Based on weight of customers
Based on value of requirements
Based on customer satisfaction
Based on return on investment
Based on value, urgency, stakeholders weights and satisfaction
Stakeholders involvement
Project manager
Developers, project management
Project manager, customer
Involvement of project manager is implied
Project manager, customer, users
All major stakeholders
All major stakeholders
Prioritization mechanism
Optimization-based
No defined prioritization scheme
Optimization-based
Optimization AHP IFM Heuristics Stakeholders prioritize features; Optimization heuristics used to balance conflicts
Technological constraints
Not available
Not available precedence Not available Not available Precedence(precursor)
Coupling and precedence
Resource constraints
Cost, risk Effort Cost Cost Cost Cost, time-to-market
Effort, risk, schedule
System constraints Operational risks
Not available Not available Not available Not available Not available Not available
Character and quality of solutions
One solution plan
One solution plan
One solution plan by any chosen search algorithm
One solution plan generated
One solution plan provided
One plan spanning many release periods
Several alternative solution plans are provided. These plans are diversified and fulfill some target quality level
Tool support Not available
Time-tracking system
Not available Not available Not available Partially available
ReleasePlanner®
© G. Ruhe 2008 16
Agenda
Decision Support in Requirements Engineering Software Release Planning EVOLVE* ReleasePlanner® Case Studies Future Research
© G. Ruhe 2008 17
Evolutionary Solution Approach EVOLVE*
Computational Intelligence
Interation 1 Release 1
Release 2Interation 2
Interation 3 Release 3
Human Intelligence
© G. Ruhe 2008 18
Phase 1 - Modeling: Definition/update of
– Decision variables– Technological and resource constraints– Objectives– Stakeholder voting
Phase 2 - Exploration: Generate alternative solutions from the application of
− Voting analysis− Specialized linear integer programming− Branch and bound algorithms− Heuristics
Phase 3 - Consolidation: Human decision maker studies solution alternatives
− Maximally diversified solution alternatives− Absolute and relative degrees of stakeholder satisfaction− Comparison between solutions− Sorting capabilities− Reporting capabilities− Definition of re-planning scenarios
The Three Phases of EVOLVE*
© G. Ruhe 2008 19
Diversification Principle
Set of qualified &maximally
diversified
alternative solutions
Value
Urg
en
cy
95% Optimality alternatives
A portfolio of qualified solutions being structurally diversified is more appropriate than a single solution that is formally optimal but unlikely to solve the “right problem”.
© G. Ruhe 2008 20
Agenda
Decision Support in Requirements Engineering Software Release Planning EVOLVE* ReleasePlanner® Case Studies Future Research
21
© G. Ruhe 2008 22
Business objects &objectives
Resource
types
&
their ca
pacitie
sResource
consumptions
Ran
king
of
obje
cts
Stakeholder &
their importance
Key Components
© G. Ruhe 2008 23
ReleasePlanner® for Elicitation of Customer Priorities
© G. Ruhe 2008 24
Feature elicitation
Problemspecification (2)
Feature repository (1)
Resource utilizationanalysis (3)
Generation of releaseplan alternatives (7)
Evaluation of planalternatives (8)
Selection of mostattractive planning scenarios
and alternatives New
/ev
olvi
ng fe
atur
es
Sho
rtag
e of
res
ourc
es
Stakeholder voting (4)
Stakeholdervoting analysis (5)
Product Manager Stakeholders ReleasePlanner®
Resource Estimation
What-if analysis (9)
Stakeholderaccess definition
Reporting (10)
Corporate Business IT
Business data,products and
processes
(6)
© G. Ruhe 2008 25
Agenda
Decision Support in Requirements Engineering Software Release Planning EVOLVE* ReleasePlanner® Case Studies Future Research
© G. Ruhe 2008 26
Case Studies and Lessons Learned (LL)
Corel iGrafx (2003/04): Product release planning
LL: Structured process Broader stakeholder involvement Siemens CT SE 3 (2004 - ): Service portfolio planning
LL: Explicit process and objectives Facilitator for understanding & improvement. Involvement of key account managers.
City of Calgary (2005 - ): Project portfolio planning
LL: Integration of release planning into business processes WestJet (2005/06): Project portfolio planning
LL: Risk mitigation from running what-if scenarios Solid Technology (2005): Product release planning
LL: Greater likelihood of feasible and stable plans Chartwell Technology (2005 - ): Product release planning
LL: More conscious release decisions more competitive products Siemens Audiologische Technik (2007): better understanding of
(world-wide) customer needs
© G. Ruhe 2008 27
Case Study at Trema Laboratories
Case study based on release planning process at Trema Laboratories, Inc.
Used both current ad hoc planning methodology and ReleasePlanner®– Determination of the operational feasibility of the strategic release plan– Assessment of the value of using ReleasePlanner® compared to ad
hoc planning Study used one of the projects for the delivery of the next release:
– Releases delivered every three months – Team size of 14 with varying levels of skills, experience and availability– Roles vary from product specialist, analyst/designer, developer, and/or
tester– Adopted a 2-weekly increment– Six stakeholders with varying interests and input and geographically
dispersed.
© G. Ruhe 2008 28
Comparison of Solutions
PS AD DV TSCMM-1019A CFF - Phase II 4 3 25 6CMM-1090A Report Manager Infrastructure for Forecast Reports - Phase II 2 2 13 3CMM-1118A Overall CMM Integration 5 4 16 5CMM-1076 Transaction Message Integration 2 4 14 2CMM-1092 CMM-ACM Integration 9 6 32 3CMM-1028 Target Balance 3 2 35 5CMM-1049 Transaction / Cash Record Reporting 1 2 10 2CMM-1043 Bank Records for Internal Transactions - Creation and Reconciliation 5 3 26 6CMM-1034 Handling Bank Transactions originated by the Bank 1 2 11 1CMM-1119 Menu Cleanup 1 0 8 1CMM-1047 Security and Auditing for MIHB 5 5 15 5CMM-1016 Automation Performance and Coverage 1 2 15 2CMM-1022 Parsing and Enriching 6 3 29 2CMM-1120 Flexible Parameters 1 2 20 2CMM-0638 Context Sensitive Help 0 0 6 29
TOTAL EFFORT 46 40 275 74
Size (Mandays)CMM Functionality Description
CMM 7.1.0 - R1 Deliverables
ReleasePlanner® Adhoc Release Plan
Comparing Results from ad hoc and ReleasePlanner®
© G. Ruhe 2008 29
Conclusions
PS AD DV TSCMM-1019A CFF - Phase II 4 3 25 6CMM-1090A Report Manager Infrastructure for Forecast Reports - Phase II 2 2 13 3CMM-1118A Overall CMM Integration 5 4 16 5CMM-1076 Transaction Message Integration 2 4 14 2CMM-1092 CMM-ACM Integration 9 6 32 3CMM-1028 Target Balance 3 2 35 5CMM-1049 Transaction / Cash Record Reporting 1 2 10 2CMM-1043 Bank Records for Internal Transactions - Creation and Reconciliation 5 3 26 6CMM-1034 Handling Bank Transactions originated by the Bank 1 2 11 1CMM-1119 Menu Cleanup 1 0 8 1CMM-1047 Security and Auditing for MIHB 5 5 15 5CMM-1016 Automation Performance and Coverage 1 2 15 2CMM-1022 Parsing and Enriching 6 3 29 2CMM-1120 Flexible Parameters 1 2 20 2CMM-0638 Context Sensitive Help 0 0 6 29
TOTAL EFFORT 46 40 275 74
Size (Mandays)CMM Functionality Description
CMM 7.1.0 - R1 Deliverables
ReleasePlanner®
Adhoc Release Plan
Comparing Results from ad hoc and ReleasePlanner®
One of the five alternative plans from ReleasePlanner® is similar to the plan generated using the ad hoc method
Alternative 1 has an additional requirement implemented
Using ReleasePlanner® has taken less than a day
Ad hoc approach to arrive at same conclusions took weeks of work and meetings
ReleasePlanner® allows easy re-planning in case of changed requirements, capacities or effort estimates.
Substantial time savings
Higher likelihood to understand and address customer priorities
© G. Ruhe 2008 30
Added Value Quantitative
49%
11%
31%
9%
Initial Planning
Replanning
Stakeholder Savings
Sales Revenue Increase
Breakdown of Value Added by
ReleasePlannerTM
V(T) = $187,250.00 for T = 1 year
© G. Ruhe 2008 31
Agenda
Decision Support in Requirements Engineering Software Release Planning EVOLVE* ReleasePlanner® Case Studies Future Research
© G. Ruhe 2008 32
Summary and Conclusions
Requirements engineering is a decision-centric process
Decision support for product release planning aims to move from reactive proactive mode of performance
Good release decisions are based on explicit and systematic processes aimed to balance intuition and rationality
Ongoing and future research:
– Release planning for product lines
– Non-linear release planning
– Operational resource allocation
– Release planning with flexible release dates
– Learning and explanation for release planning
– Light-weight re-planning
– Empirical evaluation