Systems Engineering Technical Debt Bob Epps USC-CSSE COCOMO II Workshop October 16, 2012.

Systems Engineering Technical Debt

Bob Epps

USC-CSSE COCOMO II Workshop

October 16, 2012

• Software Technical Debt has been an area of increasing focus and discussion, but we also believe it has application to Systems Engineering, as well.

• The intent of the workshop is to understand the scope of Technical Debt and what it means with respect to Systems Engineering.

What Does Technical Debt Mean at a System Level

Workshop Format

• Techniques that will be used– We would look at the terms and concepts

from Software Technical Debt and then compare and contrast as needed to ensure consistency, when adopted for Systems Engineering. The intent is to build from the existing knowledge rather than re-invent the wheel.

Workshop Outline

• System Lifecycle • Definition of Technical Debt

– Management of Technical Debt- Steve McConnell– Technical Debt Observations- Jim Highsmith– Types of Debt- Chris Sterling

• Workshop Exercise # 1- Identifying Technical Debt• Workshop Exercise # 2- Architecture Technical Debt• Workshop Exercise # 3- Operations & Maintenance

(O&M) Technical Debt• Workshop Summary/Action Plan

The Defense Acquisition Management System

Operations & Support

Decision points: 6 Phases: 5 Milestone documents: 40+

IOC

Engineering & ManufacturingDevelopment

Production & Deployment

Pre-Systems Acquisition Systems Acquisition


Sustainment

Technology Opportunities & Resources

MaterielSolutionAnalysis

TechnologyDevelopment

Post CDRAssessment

FRPDecisionReview

FOC

Materiel DevelopmentDecision

User Needs

CDR

Disposal

• The Materiel Development Decision precedes entry into any phase of the acquisition framework• Entrance criteria met before entering phases• Evolutionary Acquisition or Single Step to Full

Capability

PDR

BA C

ICD CDD CPD

Relationship to JCIDS

Initial Capabilities Document (ICD)

Capability DevelopmentDocument (CDD)

Capability ProductionDocument (CPD)

Source: Defense Acquisition University briefing, “The Defense Acquisition System” ,August 2010

ISO/IEC 15288, IEEE 1220 and ISO/IEC 26702 Stages

Source: Systems Engineering Leading Indicator Guide, INCOSE Technical Product Number: INCOSE-TP-2005-001-03, January 2010

What Does Technical Debt Mean at a System Level

2 August 2012

Bob Epps, Lockheed Martin

PSMPSM

Practical Software and Systems MeasurementObjective Information for Decision Makers

http://www.psmsc.com/UG2012/Workshops/w8-SE%20Technical%20Debt%20Workshop%202012%20plus%20Outbrief.pdf

Objectives of the Workshop

• Identification and Management of Technical Debt in System Development and Operation & Maintenance (O&M) phases of the System Lifecycle.

• Discuss how the Decisions made during the System Lifecycle influence the level of Technical Debt within the System.

• Evaluation of Technical Debt’s impact to the Operation & Maintenance (O&M) phase.

• Discussion on the relationship of Systems Engineering Technical Debt to System Affordability.

Source: PSM Workshop Outbrief on Systems Eng Technical Debt, August 2012

Workshop Background• PSM history in this area

– 2011 Keynote Address: Cheryl McIntyre, Lockheed Martin– 2011 Workshop: Bob Epps/Garry Roedler, Lockheed Martin– 2012 Workshop: Bob Epps, Lockheed Martin

• Where we’re heading– Defining Technical Debt influence throughout the System

Lifecycle– Identification and Management of Technical Debt

• Issues, questions, and topics– Impact of Technical Debt on System Development phase,

O&M phase & System Affordability


Intended Output

• Collaboration with other industry associations(NDIA, INCOSE, etc.) on a white paper on the topic of Systems Engineering Technical Debt.


When is it Technical Debt?

• Discussion converged on the following:– 3 I’s – Intention, Impact, and Interest– Results from intentional decisions

• Commits the program in a certain direction• Decision trade space has specific future impacts that

are understood• Impacts have “interest payments” associated• Deferred decisions are only Technical Debt when the

deferral causes accrued interest

– Measurable – can identify and assess– Manageable – within the control of the program

team Source: PSM Workshop Outbrief on Systems Eng Technical Debt, August 2012

Difference of System and Software

• Software Technical Debt focuses more on:– Software product quality

• System Technical Debt includes:– System definition quality (requirements, architecture,

interfaces, trades, analysis)– System operational/performance issues – both internal

and interoperability – System life management issues (e.g., adaptability,

extensibility, …)– System maintainability/sustainment issues


Other key points

• What can create greatest impact?– Top level architectural issues

• Dependency analysis may help highlight• Consideration of future needs and interoperability

– Top system definition that touches or drives the most lower level elements

• Technical debt was not viewed as being a factor of System Complexity

• Identified additional boundaries for System Technical Debt, examples include:– Use of NDI is not necessarily TD– Expected iteration between processes is not TD


Assessing the Total System Technical Debt?

• Technical debt is composed of impact costs from various categories of effort– Rework– Quality impacts other than rework– Performance impact– Maintainability/sustainment impacts– Adaptability/flexibility/evolution impacts (e.g., cost of change)– …

• More analysis needed to fully identify and define this set

• The total TD is the aggregate of these elementsSource: PSM Workshop Outbrief on Systems Eng Technical Debt,

August 2012

Technical Debt and Risk

• Technical debt involves both risk management and problem management

• Need to investigate more with Risk Management experts

• After decisions are made, TD has occurred and problem management is needed– Catalog and prioritize to pay down as ROI

shows it is advantageous


The Cost of Undetected Defects

Time

70%

85%

95%Committed Costs

Cost to E

xtract

Defects

3X-6X

20X-100X

500X-1,000X

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Cu

mu

lati

ve P

erce

nta

ge

of

Lif

e-C

ycle

Co

st

15%

Design

20%

Development 50%

Production/Test

100%

Operation Disposal

Concept

8%

Reference: Defense Systems Management College (DAU)

Technical Debt “Management of Technical Debt”. Steve McConnell

– “ ‘Technical Debt’ refers to the delayed technical work that is incurred when technical short cuts are taken, usually in pursuit of calendar driven software schedules. Just like financial debt, some technical debt can serve valuable business purposes. Other technical debts are simply counter productive. The ability to take on debt safely, track their debt, manage their debt and pay down their debt varies among organizations. Explicit decision making before taking on debt and more explicit tracking of debt are advised”

Technical Debt “Management of Technical Debt” Steve McConnell

– “ The term ‘technical debt’ was coined by Ward Cunningham to describe the obligation that a software organization incurs when it chooses a design or construction approach that’s expedient in the short term but that increases complexity and is most costly in the long term”

– There are two kinds of technical debt• Type I, Debt incurred unintentionally

– Inexperienced individuals produces error prone results

• Type II, Debt incurred intentionally– Conscious decision to optimize for the “present” rather than

the “future”

Technical Debt “Management of Technical Debt”, Steve McConnell

• Type II, Debt incurred intentionally– “Short-Term” Debt (Type II.A)

» A company takes on a short term debt when it has the money; it just does not have it now.

» Short term debt is expected to be paid off frequently» Focused Short-Term Debt(Type II.A.1)» Unfocused Short-Term Debt (Type II.A.2)

» Should be avoided

– “Long-Term” Debt(Type II.B)» A company takes on strategically and proactively» Primary rationale is that the development work “today” is seen as more expensive than

the cost in the future.» Example:

» Responding to “Time to Market” pressures» Preservation of Startup capital» Delaying Development expense

– Debt Service» The “interest” charged for incurring the debt


Summary of Kinds of Debt

Non Debt

Features backlog, deferred features, cut features, and so on. Not all incomplete work is debt. These are not debt because they do not require interest payments

DebtI. Unintentional Debt. Debt incurred unintentionally due to low quality

II. Intentional Debt. Debt incurred intentionally

II.A Short-Term Debt. Short Term Debt, usually incurred reactively, for tactical reasons

II.A.1 Focused Short Term Debt. Individually identifiable shortcuts(like a car loan)

II.A.2 Unfocused Short-Term Debt. Numerous tiny shortcuts(like a credit card)

II.B Long-Term Debt. Long-term debt, usually incurred proactively, for strategic reasons


Communicating about Technical Debt

• Shift from Technical vocabulary to a Financial vocabulary

• Use a projects Maintenance budget as a rough proxy for its technical debt service load

• Discuss Debt in terms of “money” instead of “features”

• Be sure you’re taking the right kind of debt• Treat the discussion of Debt as an ongoing dialog

rather than a single discussion

Technical Debt Observations“Agile Project Management”, Jim Highsmith, second edition

Cos

t of

Cha

nge

(C

oC

)

Years

1 2 3 4 5 6 7 8

ProductRelease

Customer Responsiveness

Actual CoC

Optimal CoC

Technical Debt


– “When product development teams give lip service to technical excellence, when project and product managers push teams beyond quickness into hurrying, technical debt is incurred”(pp 216)

– “Technical debt can arise during initial development, ongoing maintenance (keeping a product at its original state), or enhancements (adding functionality).”(pp 216)When does the insertion of Technical Debt have its greatest impact?


– “Without a firm dedication to long-term technical debt management, development groups are pressured into increasing technical debt trap. As the debt gets worst, the delays become greater. As the delays lengthen, the pressure increases, usually leading to another hurried implementation, which increases the technical debt yet again.”(pp 217)


– “It must be noted that managing technical debt does not keep products from becoming obsolete. A technical debt strategy does not attempt to stave off eventual obsolescence, but keeps the cost of change low so that customer responsiveness remains as high as possible during a product life.”(pp 217)

Types Debt“Managing Software Debt: Building for Inevitable Change”, Chris Sterling

• Software Debt– Composed of the following forms of “Debt”

• Technical Debt, Quality Debt, Configuration Management Debt, Design Debt & Platform Debt

– Indicators of Software debt are the following:• Do you have “like-to-like” migration?• Do you have “limited expertise” available?• Do you have “expensive release stabilization” phases?• Do you have “increased cost for regression testing”

your software assets?

Types Debt“Managing Software Debt: Building for Inevitable Change”, Chris Sterling

• Technical Debt– “..the decay of component and inter-component behavior when the

application functionality meets a minimum standard of satisfaction for its users”

– Produced by work patterns:• Schedule pressure

– “Excessive pressure causes team to take short cuts to meet expectation of management and customer”

• Duplication– “..because of cut-and-paste tactics results in teams making even simple changes in

more than one place.”

• Mentality of getting it right the first time– “..incorrect assumptions about what we can know about the future.”

– “..payoff Technical Debt immediately, insert strategically placed runtime exceptions, and add technical debt to the Product Backlog.”

Workshop Exercise # 1Identifying Technical Debt

• Development profile (Perfect World)• Defect Density profile• Development profile(Real World)• Mapping Development Profiles

Development Cost(Perfect World)

Analysis Design Implementation Test Integration

% E

ffort

per

Pha

se


Defect Insertion Defect detection & Removal

DesignDefects

IntegrationDefects

Typical Defect Profiles

ImplementationDefects

Classification of Defects

Development Cost(Real World)


% E

ffort

per

Pha

se

Development Cost


% E

ffort

per

Pha

se

Development Cost


% E

ffort

per

Pha

se

TechnicalDebt?

TechnicalDebt?

TechnicalDebt?

TechnicalDebt?

Development Cost


% E

ffort

per

Pha

se

TechnicalDebt?

TechnicalDebt?

Better or Worst?

COTS Integration


% E

ffort

per

Pha

se

TechnicalDebt?

Technical Debt “Enabling Agility by Strategically Managing Architectural Technical Debt”, Ipek Ozkaya

– “Practices intended to speed up the delivery of value to users, however, often result in high rework costs that ultimately offset the benefits of faster delivery, especially when good engineering practices are forgotten along the way. The rework and degrading quality often is referred to as technical debt”

– “For example, through our work on architecture-centric engineering, we often encounter projects that defer modifiability requirements, specifically portability.”


– “Our current work focuses on architectural technical debt, which involves architectural decisions made to defer necessary work during the planning or execution of software projects, such as short-cuts taken in designing the structure of the system that may require rework.”

– “We are particularly interested in identifying the measureable aspects of architectural technical debt by exploring dependency analysis”


– “By the end of this project, we will produce a model for managing technical debt that will allow the incurrence of some debt to increase delivery tempo when needed, but prevent too much accumulation, which would impede the ability to deliver.”

Reference: “Enabling Agility through Architecture”, Nanette Brown, Robert Nord, Ipek Ozkaya; CrossTalk-Nov/Dec 2010

Workshop Exercise #2Architecture & Technical Debt




IOC





Sustainment




Post CDRAssessment

FRPDecisionReview

FOC


User Needs

CDR

Disposal


Capability

PDR

BA C

ICD CDD CPD






Levels of Architecture(Conceptual to Logical to Physical Mapping)

Conceptual Level

Logical Level

Physical Level

System

Elements

Components

Cost EstimateLow Prob High

Seg ASeg BSeg DSeg HSeg I

WBS

ArchitectureSystem Integration Plan

A1 Spec

B5Spec

B5Spec

B5Spec

SOW

System Requirements

Notional

Architecture Process

NDI

COTS GFSReuse

NDI Products

System Segment Schedule

System SegmentComplexity

(Domain Knowledge)

DTC RisksSchd TechCost

Risk ARisk BRisk CRisk DRisk E

L

MM

H

H

BusinessRequirements

Where can Technical Debtbe incurred in this process?

Declare Architecture

Review and Refine

System Level Requirements

Identify, Review& Select NDI

Products

RefineArchitecture

& WBS

EstablishIntegration Strategy

Define SystemLoad Distributionon Infrastructure

RefineArchitecture

Notional Architecture Process

Step 1 Step 2 Step 3a & 3b

Step 4 Step 5a & 5b Step 6

Step 7 Step 8

Apply BusinessRequirements & Evaluate System

Complexity

Estimate CostFor Each Segment

Estimate Cost for CompleteSystem

DetermineSystem Level

Risks(Cost, Schedule, Technical)

Risks withinAcceptable

Limits

Refine IntegrationStrategy

Refine Requirements

Refine Architecture &Implementation

No

Complete and Submit Basis of Estimates and

Associated Risks

Yes

Notional Architecture Process(continuation)

Step 9 Step 10 Step 11

Step 13Step 14Step 15

Step 16

Step 12

Step 1 Declare Architecture

Steps 2 & 14 Review & Refine System Requirements

A1 Spec

B5Spec

B5Spec

B5Spec

SOWDesign

to Cost(DTC)Objectives

Are “Derived” requirements a source or a mitigation for Technical Debt?

Step 3a Business Requirements

BusinessRequirements

Product Line ManagementStrategies

Technology Discriminators

Application ofFrameworks

ArchitectureReference Models

ProductReuse

New DevelopmentMethodologies

Is this a Source of Technical Debt?

"The Challenge"

$

Technology

Value

Low High

Low

High

Low

High

Process/Procedures(Methodology, Standards, Practices)

Products/Assets(IR&D, Prototypes, Labs, Conferences)

Proposals/Reviews

(Contracts, Projects, RFCs)




IOC





Sustainment




Post CDRAssessment

FRPDecisionReview

FOC


User Needs

CDR

Disposal


Capability

PDR

BA C

ICD CDD CPD






"The Challenge"

$

Technology

Value

Low High

Low

High

Low

High



Proposals/Reviews


Identification of Trade-space basedon Stakeholders definition of Affordabilityduring the pre- Systems Acquisition phase

Step 3b Evaluate System Complexity

Low

Moderate

Medium

High

Fidelity

Response Time

Size

Factor X

Reliability

ResourceUtilization

Maintainability

Factor Z

Required

Proposed

Is Technical Debt a System complexity factor?

NDI

COTS GOTSReuse

Step 4 Identification, Review & Selection of NDI Products

Does selection of NDIincur Technical Debt?

NDI: Non-Developmental ItemCOTS: Commercial-Off-The-ShelfGOTS: Government-Off-The-Shelf)/GFE/GFS

Steps 5a, 8 & 15 Refine Architecture

WBS

Step 5b Refine WBS

System Segment Schedule

System SegmentComplexity

Segment A Segment C

Segment B

Segment D

Segment ESegment F

Segment G

Segment HSegment I

Steps 7 & 13 Integration Strategy/Plan

What type of Technical Debt is incurred if the Integration Strategy is ill-defined?


Analysis

Design

Integration

Total

Step 9 Segment Level Cost Estimate

Estimate

Estimate = High - Low + 4(Prob)

6


Segment A

Segment B

Segment E

Integration

Step 10 System Level Cost Estimate

Estimate

Estimate = High - Low + 4(Prob)

6

RisksSch TechCost

Risk A

Risk B

Risk C

Risk D

Risk E

L

H

H

L

H

Step 11 Risk Identification/ Mitigation

How is Technical Debt related toSchedule, Cost and Technical Risks ?

Risks withinAcceptable

Limits

No

Yes

Step 12: Assessment of Risks against Acceptance Criteria

Is this where “intentional” or “unintentional” level of Technical Debt is determined?

Operations & Maintenance(O&M) Technical Debt

O&M phase occurs after the formal release of the system tothe Customer.

For DoD contracts, the O&M phase may be awarded as a separate Contract. Potentially resulting in a different team executing this phase, then the team that executed the System Development phase. O&M phase is cited as the phase where significant costs occur and therefore is a target for Affordability activities to address.

Commercial products have a similar phase which may determine how much of the market share that a company captures with the release of new versions of their system, based on competitive pressures and consumer demands.

SystemAvailability

System Training/Orientation

SystemOperations

SystemMaintenanceUnplanned

Time

ProductiveTime

UnproductiveTime

Task Performance System

Configuration

ErrorCorrections

DefectCorrection

Preventive Maintenance

Notional Definition of System Availability

SystemAvailability

System Training/Orientation

SystemOperations

SystemMaintenanceUnplanned

Time

ProductiveTime

UnproductiveTime

Task Performance System

Configuration

ErrorCorrections

DefectCorrection

Preventive Maintenance

Mission Requirements

Requests for Change(RFCs)

Potential Consequences ofTechnical Debt

Activities during Operations & Maintenance(O&M)

Analysis Design Imp Test Integ

% E

ffort

per

Pha

se


% E

ffort

per

Pha

se


% E

ffort

per

Pha

se

RFC #1 RFC #2 RFC #3

Sys

tem

C

om

ple

xity

Time

If each RFC has equal complexity you might expect this


AnalysisDesign Imp Test Integ

% E

ffort

per

Pha

se


% E

ffort

per

Pha

se

AnalysisDesign Imp Test Integ

% E

ffort

per

Pha

se

RFC #1

RFC #2

RFC #3

Sys

tem

C

om

ple

xity

Time

Even if each RFC has different complexity , it still would be manageable.If maintainability is measured by the number of systems components changedfor an RFC what is the ideal number of components changed? What dictates this number?



% E

ffort

per

Pha

se

Analysis Design Imp Test Integ%

Effo

rt p

er P

hase


% E

ffort

per

Pha

se

RFC #1

RFC #2

RFC #3

Sys

tem

Com

plex

ity

Time

However, experience shows that evenwhen the RFCs have the same complexitythe “aggregate” complexity of the system causesside effects which complicate future RFCs.

How is the phenomenon managed?




IOC





Sustainment




Post CDRAssessment

FRPDecisionReview

FOC


User Needs

CDR

Disposal


Capability

PDR

BA C

ICD CDD CPD






"The Challenge"

$

Technology

Value

Low High

Low

High

Low

High



Proposals/Reviews


Identification of Trade-space basedon Stakeholders definition of Affordabilityduring Sustainment phase?

Revisit prior conclusions and add to and/or revise, as required

When is it Technical Debt?

• Discussion converged on the following:– 3 I’s – Intention, Impact, and Interest– Results from intentional decisions

• Commits the program in a certain direction• Decision trade space has specific future impacts that

are understood• Impacts have “interest payments” associated• Deferred decisions are only Technical Debt when the

deferral causes accrued interest

– Measurable – can identify and assess– Manageable – within the control of the program

team Source: PSM Workshop Outbrief on Systems Eng Technical Debt, August 2012

Difference of System and Software

• Software Technical Debt focuses more on:– Software product quality

• System Technical Debt includes:– System definition quality (requirements, architecture,

interfaces, trades, analysis)– System operational/performance issues – both internal

and interoperability – System life management issues (e.g., adaptability,

extensibility, …)– System maintainability/sustainment issues


Other key points

• What can create greatest impact?– Top level architectural issues

• Dependency analysis may help highlight• Consideration of future needs and interoperability

– Top system definition that touches or drives the most lower level elements

• Technical debt was not viewed as being a factor of System Complexity

• Identified additional boundaries for System Technical Debt, examples include:– Use of NDI is not necessarily TD– Expected iteration between processes is not TD


Assessing the Total System Technical Debt?

• Technical debt is composed of impact costs from various categories of effort– Rework– Quality impacts other than rework– Performance impact– Maintainability/sustainment impacts– Adaptability/flexibility/evolution impacts (e.g., cost of change)– …

• More analysis needed to fully identify and define this set

• The total TD is the aggregate of these elementsSource: PSM Workshop Outbrief on Systems Eng Technical Debt,

August 2012

Additional Observations/Conclusions?

Technical References&

Background Material

Technical Debt References• Managing Technical Debt

– Tom McConnell– http://www.construx.com/Page.aspx?cid=2801

• The Financial Implications of Technical Debt– Jim Highsmith– http://www.jimhighsmith.com/2010/10/19/the-financial-implications-of-technical-debt /

• Third International Workshop on Managing Technical Debt– Ipek Ozkaya, Rod Nord, Philippe Kruchten, Joost Visser– http://www.sei.cmu.edu/community/td2012/

• Assessing and Avoiding Technical Debt– Barry Boehm– http://www.sei.cmu.edu/library/assets/presentations/boehm-mtd2012.pdf

• SEI blog on the Management of Technical Debt– Douglas C. Schmidt– http://blog.sei.cmu.edu/post.cfm/strategic-management-of-architectural-technical-debt

• Managing Technical Debt– Philippe Kruchten– http://pkruchten.files.wordpress.com/2011/10/kruchten-111027-techdebt.pdf

• Enabling Agility by Strategically Managing Architectural Technical Debt– Ipek Ozkaya– http://blog.sei.cmu.edu/post.cfm/enabling-agility-by-strategically-managing-architectural-technical-debt

http://www.construx.com/Page.aspx?cid=2801

http://www.construx.com/Page.aspx?cid=2801

http://www.jimhighsmith.com/2010/10/19/the-financial-implications-of-technical-debt/

http://www.jimhighsmith.com/2010/10/19/the-financial-implications-of-technical-debt/

http://www.sei.cmu.edu/community/td2012/

http://www.sei.cmu.edu/library/assets/presentations/boehm-mtd2012.pdf

http://www.sei.cmu.edu/library/assets/presentations/boehm-mtd2012.pdf

http://blog.sei.cmu.edu/post.cfm/strategic-management-of-architectural-technical-debt

http://pkruchten.files.wordpress.com/2011/10/kruchten-111027-techdebt.pdf

http://pkruchten.files.wordpress.com/2011/10/kruchten-111027-techdebt.pdf

http://blog.sei.cmu.edu/post.cfm/enabling-agility-by-strategically-managing-architectural-technical-debt

Systems Engineering Technical Debt Bob Epps USC-CSSE COCOMO II Workshop October 16, 2012.

Documents

Transcript of Systems Engineering Technical Debt Bob Epps USC-CSSE COCOMO II Workshop October 16, 2012.