Death by technical debt:lessons learned to get unburied

QA & Developer Joint Forum Event

SERIES SPONSORS

JAMES SHOREConsultant, AuthorThe Art of Agile@jamesshore

1800GOAGILEby Becky James

Debt Counseling

#surelytheremustbeoneanswer

#managersbringconstraints, #debtfreewithoutmanagers

#oldnews, #boldstatement#noteamcohesion

#itsalltheirfault

#unempowered, #overthewallthinking#lacksownership, #actualquotes

I shouldn't have to spend any of

my resource pool on technical

debt that the Development

teams introduced.

Surely, we don’t need that much

time to fully test the

feature – let’s cut that

estimate in half.

What do you mean my

feedback is too late…I just got pulled into the

project yesterday.

Product Owner Developer QA Engineer

#completetransformation, #declaredbankruptcy

#empowered, #teamsownsdecisions

#selforganizingteams, #nobystanders#deliberatedecisions, #ownership

I own quality and I own

keeping our technical debt under control.

I own quality and I own

keeping our technical debt under control.

Product Owner Developer QA Engineer

I own quality and I own

keeping our technical debt under control.

#theybringdonuts

Technical DebtDeath by Technical Debt:

Lessons Learned to get you Unburied

Product Owner Point of View

Todd Whitaker, Product Owner

Tripwire, Inc.

What is technical debt?

14

• Technical work that should have been completed, but for whatever reason was not and thus it is being “carried” in the product/codebase.

• Analogies:

Technical Debt – This PO’s POV

If you are a product owner you should care about technical debt.

If you work with a product owner, you need to help him/her know why they should care.

15

How does it happen?

• “Startup mode”• Time to market pressure• Inexperience• A natural side-affect of an imperfect median

16

Technical Debt – Points to consider

• +1 for not incurring it in the first place• If it is being incurred, make it visible!• In reality it happens and should be tracked

– Tracking too much becomes ineffective

• Developers are usually the most aware• Others are impacted in many ways:

– Dev teams, Support, Prof Services, etc. and customers!

17

Motivating your PO• Make the PO aware of tangible effects and

benefits of addressing• Help him/her understand and quantify ROI

– Performance (speed/throughput)– Improved dev team velocity– Faster time to market– More features in the release– Increased sales– Benefit to Prof. Services and System Engineers (sales)

• Be willing to let go if you cannot justify it.

18

assign $

Questions?

19

ADP Dealer ServicesADP Dealer ServicesTechnical Debt Technical Debt DiscussionDiscussion

November 2013November 2013

21

ADP Dealer ServicesADP Dealer Services

FOCUS

22

ADP Dealer ServicesADP Dealer Services

PRODUCTS

LEAD MANAGER

CRM

DESKING

CREDIT

COMPLIANCE

F&I

MENU

STOCKING

MERCHANDISING

23

ADP Dealer ServicesADP Dealer Services

PEOPLE

37 SPRINT TEAMS250+ ASSOCIATES

24

ADP Dealer ServicesADP Dealer Services

PROCESS

25

ADP Dealer ServicesADP Dealer Services

OUR TECH DEBT STORY…

26

ADP Dealer ServicesADP Dealer Services

FIRST, THE OLD…

My Top 5 Favorites

27

ADP Dealer ServicesADP Dealer Services

1.Our application tips over every Saturday

2.We need to plan a hot fix after every release

3.We are scared to change ‘X’

4.Slow Releases

5.We Had to Roll Back the Release Because of xyz.dll

28

ADP Dealer ServicesADP Dealer Services

1.Our application tips over every Saturday

2.We need to plan a hot fix after every release

3.We are scared to change ‘X’

4.We Can’t Release That for Another 10 Days

5.We Had to Roll Back the Release Because of xyz.dll

29

ADP Dealer ServicesADP Dealer Services

1.Our application tips over every Saturday

2.We need to plan a hot fix after every release

3.We are scared to change ‘X’

4.We Can’t Release That for Another 10 Days

5.We Had to Roll Back the Release Because of xyz.dll

30

ADP Dealer ServicesADP Dealer Services

1.Our application tips over every Saturday

2.We need to plan a hot fix after every release

3.We are scared to change ‘X’

4.We Can’t Release That for Another 10 Days

5.We Had to Roll Back the Release Because of xyz.dll

31

ADP Dealer ServicesADP Dealer Services

1.Our application tips over every Saturday

2.We need to plan a hot fix after every release

3.We are scared to change ‘X’

4.We Can’t Release That for Another 10 Days

5.We Had to Roll Back the Release Because of xyz.dll

32

ADP Dealer ServicesADP Dealer Services

Now the New…

33

Culture ChangeCulture Change

34

EndEnd

© 2012 Portland General Electric. All rights reserved.

Technical Debt Management

TAO Panel Discussion

Date: November 7, 2013Presenter: Mark Menger

3611/13/13

15 application delivery teams1 large scale project teamExperience ranging from months to

decades

People, …

3711/13/13

ScrumKanbanWaterfall

…Process, …

3811/13/13

.Net & T-SQLJava / WebSphereDatapowerDataStageOracle Forms & PL/SQLAnd others

…and Technology …

3911/13/13

Unacknowledged technical debtUnmanaged technical debtVendor technical debtKnowledge debt

Technical Debt Challenges

4011/13/13

Unplanned and planned workCommitted workCone of uncertainty

The importance of language

4111/13/13

Introduce the concept at all levelsCommon term used by IT senior

managementDon’t create more technical debtScrumMaster / Product Owner

partnershipOccasional topic at business

sponsor groups

First steps out of the pit

4211/13/13

Improve fluency with the concept Informally capture technical debt

backlogsExpand upon success of SM/PO

partnershipDocument project end technical

debtAdditional process measures (e.g.

code coverage)

Next steps out of the pit

Tech DebtThe NWEA Product Engineering Perspective

Recommendations

1. Change the language• “Technical Debt” is “Business Debt”

1. Create owner and advocate for technical debt

2. Make the cost transparent (Get your CFO’s partnership!)

3. Create both initiatives and pre-allocated capacity

4. Invest in good design early• Good design early beats Great design late

1. Be proactive

NWEA

• Mission - “Partnering to Help All Kids Learn”• $100 Million Revenue• 500+ Employees• Serve 7+ Million kids World Wide• 42+ Million Assessments per year • 100+ Engineers• Based in Portland

Our Challenge

• Scaling Web Applications• High Capacity, High Transaction Volume• High Concurrency• > 1,000 moving to > 100,000 Concurrent Users

• Moving from Product to Platform• Moving from Single Product to Suite• Wrong Architecture• Both Hardware and Software• Built for one purpose, deployed for a different one

Technical Debt Definition

• “A confusing combination of black magic, voodoo, and artifacts generated in the daily activities of Software Developers, Technical Architects, System Engineers and Operations of Technology Platforms, which is not readily apparent to, or valued by, customers and business partners.” • The Evil Twin of product “Features”• The “Buzz-Kill” of Product Managers customer focus groups

and initiative funding meetings• The “Hard Sell”

Examples

• The Obvious• Hardware and OS Infrastructure EOL• Legacy or EOL Software Frameworks• De-supported Software and Hardware Platforms

• The Non-Obvious• Short-Cut and/or Wrong Architecture and Design Decisions• Lowest Bid Technology Acquisition (a.k.a the cheapest route is the

most expensive long term solution)• The Prototype gone wild (a.k.a “Fast is Good”)• The “Tent City” – Bad Urban Planning in Technology Architecture

Strategies to Manage Technical Debt

1. Create an Initiative

2. Hide Capacity

3. Pass it to your Successor

Create an Initiative

• The Good• Full Organization Sign-up and Support• Focus• Resources • Targets

• The Bad• Need strong executive sponsor• Usually driven by pain not typical positive drivers• Seen as “Necessary Evil”• Must compete with customer facing, revenue generating, projects

Hide Capacity

• The Good• No negotiation required• Give engineers license to do this work• Self Protection• Less likely to be caught “out”

• The Bad• Tough to keep in the priority work• Needs internal governance and oversight• Requires discipline

Pass it to your Successor

• The Good• No management required• No architecture and design required• No negotiation required

• The Bad• Can’t predict the timing of “The Crisis Point”• Your tenure has limited time horizon• Your infrastructure will fail (i.e. What color is your parachute?)

Recommendations

1. Change the language• “Technical Debt” is “Business Debt”

1. Create owner and advocate for technical debt

2. Make the cost transparent (Get your CFO’s partnership!)

3. Create both initiatives and pre-allocated capacity

4. Invest in good design early• Good design early beats Great design late

1. Be proactive

Q&A

SERIES SPONSORS