Applying 'Kanban' in Enterprise-Class Products Sustaining Engineering - An Experience Report

1

www.scrumindia.in

Date: 7-9 Dec 2012Name: TV

ScrumIndia.In Proprietary Information

Applying ‘Kanban’ in Enterprise-Class Products Sustaining Engineering

~ An Experience Report ~

Tathagat Varma@tathagatvarma

http://managewell.net http://slideshare.net/managewell

2

www.scrumindia.in



Discussion Topics

• Enterprise-class Software Products• Our Product • Old Process• Problems with Old Process• New Process• Did this move the needle?• What is Kanban?• Kanban in Software Engineering• What did we learn?

3

www.scrumindia.in



What characterizes Enterprise Software?

4

www.scrumindia.in



Microsoft Windows timeline

https://en.wikipedia.org/wiki/Timeline_of_Microsoft_Windows

5

www.scrumindia.in



Other major OS and tools timeline

https://en.wikipedia.org/wiki/Solaris_(operating_system) https://en.wikipedia.org/wiki/Bugzilla

6

www.scrumindia.in



As a contrast, what are consumer internet companies doing?

• Continuous Integration -> Continuous Delivery -> Continuous Deployment

• On ‘good days’, Flickr releases a new version every half an hour (Jun 20, 2005)

• IMVU pushes a revision of code to the website every nine minutes (Feb 10, 2009)

• The other day we passed product release number 25,000 for WordPress. That means we’ve averaged about 16 product releases a day, every day for the last four and a half years! (May 19, 2010)

• A new version of Google Chrome now due every six weeks (Jul 22, 2010)

• Facebook does code push twice a day (Aug 4, 2012)

7

www.scrumindia.in



Desktop OS Market share (Nov 2012)

http://marketshare.hitslink.com/operating-system-market-share.aspx?qprid=10&qpcustomd=1

8

www.scrumindia.in



Our Product

• Network Management domain• Windows-based specialized hardware (“Appliances”)• Installed in data centers for traffic monitoring,

analysis and network troubleshooting

– but not generally on production network• Typical users are technical folks – CIO, Network

Manager, Network Engineers• Selling cycles typically align with quarterly or annual

budget cycles• Many sales require implementing customer specials

9

www.scrumindia.in



Old Process, circa 2003

• Customer Bugs prioritized based on multiple business parameters, including (partial list) -

– Severity

– Impact on Revenue, Volume, Competitive, etc.

– Case age• PMO would prepare Maintenance Release Plan of Record (MR

POR) and get buy-in for various types of MRs -

– Service Packs – bunch up ~50-60 bugs typically every quarter

– Hot Fixes – 1-2 high-urgency bugs that can’t wait until next SP

– Patches – workaround for customer-specific issues• SPs would have

– Above The Line (ATL) requirements – must fix

– Below The Line (BTL) requirements – fix if time permits

10

www.scrumindia.in



A typical support timeline

http://itconvergence.blogspot.in/2012/10/oow-12-elison-extradata-oracle-r122.html

11

www.scrumindia.in



Problems with Old Process

• Dev team had no bandwidth to take on maintenance releases• Huge pile of customer escalations without “home”• Compounded by high incoming field rate• Low closure rate (largely due to no dedicated resources)• Large wait for customers to get bug fixes• Tech Support often tasked team directly and broke the process• Hot fixes not always available to all customers• Sometimes, a new bug fix might break a hot fix• If a hot fix failed in the field, rollbacks would be very difficult• Difficult to estimate time to resolve a bug and give an ETA• High-priority bugs could arrive at any time• Customer specials could arrive anytime with top priority• High internal rejection rate of bug fixes by Tech Support

12

www.scrumindia.in



New Process, 2004-05

13

www.scrumindia.in



Product C

Product D

Product C

Product D

ProtocolsProtocols

Decide DriversDecide Drivers

Backend

Product B

Backend

Product B

GUI

Product A

GUI

Product A

Our Kanban Process in action…

PMO

CST Manager

QA TeamQA Team

Dev Team = 15

WIP = 15

WIP = 3

WIP = 4

WIP

= 2

WIP

= 3

WIP

= 3

WIP = 1wk

WIP = 1wk

Tech Support

Qu

eue

= 0

Queue = 0

14

www.scrumindia.in



Process improvement…the beginnings…

15

www.scrumindia.in



Cumulative Hot Fix Process

16

www.scrumindia.in



Weekly Build Process

17

www.scrumindia.in



So, what is happening?

• Though not an originally stated vision or goal, the “Work in Progress” (WIP) is being limited to # of team members

• At any time, one developer is assigned only one piece of work, thereby achieving “One-Piece Flow”

• New work is only assigned when current work is completed (or cancelled/stalled), and a team member is available

• No wait state or switching costs at an individual level• Smaller lead time for bugs (in contract to lead time for SP)• The process is allowing ‘continuous deployment’ of each

of the hot fixes – even though it is only being practiced in a limited manner

• Finally, the flexibility gained is not a zero-sum game – there is no penalty on performance in rest of the process

18

www.scrumindia.in



Did this move the needle?

• Bugs addressed each quarter• Quality of bug fixes• “Homes” for bugs• Total bugs open• Open days open• People motivation

19

www.scrumindia.in



Shift from SPs to Cumulative Hot Fixes while maintaining High Quality

Maintenance Releases (Service Packs, Patches, Hot Fixes) Q3 2003 Through Q4 2006 (Fiscal Year)

0 2 3 4 4 4 42 2 2 1 2 3

15 2 1

53 1 0

0 1 0 0 01

0

28

11

1618 20

62

28 2726 26 24

28

21

100

92

87

96

88

92

85

80

97 9693

100

9491

7

12

15

25 25 25

30 3028 27 26

32

22

66

0

10

20

30

40

50

60

70

Q3 03 Q4 03 Q1 04 Q2 04 Q3 04 Q4 04 Q1 05 Q2 05 Q3 05 Q4 05 Q1 05 Q2 06 Q3 06 Q4 06

0

10

20

30

40

50

60

70

80

90

100

Service Packs Patches Hot Fixes % Successful

Percentage of released Maintenance Releases (Service Packs, Patches, Hots Fixes) that addressed customer reported

Total Maintenance Releases (Service Packs, Patches, Hot Fixes) for this quarter.

ServicePacks

Patches

Hot Fixes

20

www.scrumindia.in



Increase in bugs with “homes”

Customer Direct and Indirect % w/Homes - HistoricalWeek Ending February 03, 2006

55

46 45 4744 45 45 44

5552 54 56

35

4539 38

41

5146

31

40 40

5963

60 61 61

68 6974 72

69 69

55

73

61

535655

4854 55

52 51 51 51 50 4954 55

47 47

1115

18 16

4753 55

43 42 43

75

8276 76 75

82 83 82 81

71 72

65

8782

7580

55

4751 51 50 48 48 48 48 48

53 5451 51

2127 27 25

44

52 51

43 41 42

6772

68 69 68

75 76 78 7670 71

60

80

71

6468

4447

0

10

20

30

40

50

60

70

80

90

100

Direct Indirect Goal (80% w/Homes) Combined

Percent with Releases Identified: 56% (Driect), 80% Indirect, 68% (Combined)

21

www.scrumindia.in



Total Bugs Open

Severity Across Products (Historical)

Week Ending March 31, 2006

222

191

243

213

224230

236

201199

210211

196197195

161

147146146146

132

122

131

121127

111

127125

148150

65605857

5357

626466676768717475756866

4947465049495052

606369707069

747368

61605555

4749514944

3538323332343536

0

10

20

30

40

50

60

70

80

90

100

110

120

130

0

20

40

60

80

100

120

140

160

180

200

220

240

260

Total Severity 1 Severity 2 Severity 3 Severity 4 Severity 5

22

www.scrumindia.in



Average Days Open

Severity: Average Days By Severity (Historical)

Week Ending March 31, 2006

203

155

136

244242246246244248

234235226223

232

244

260268

275275

299

232

201210207

179

165172

156162

180

168163

156152147

142

132135139143

148140

144147146144

155

135

145152

148157156156

149

133

144140

144147154155157

163166

177179183182

164162165

154146

141149

123117

134134

123

0

25

50

75

100

125

150

175

200

225

250

275

300

325

350

375

400

425

450

475

500

0

25

50

75

100

125

150

175

200

225

250

275

300

Avg Days Open Severity 1 Severity 2 Severity 3 Severity 4 Severity 5

23

www.scrumindia.in



People motivation

• Started with 16 people dev team• We had zero attrition in the team• Once the backlog started coming down, engineers were

ramped off the team to do new features• Eventually dismantled the team and rolled-up engineers into

dev teams when backlog came down to single digits

24

www.scrumindia.in



What is Kanban?

• Kanban (literally signboard or billboard) is a scheduling system for lean and just-in-time (JIT) production. According to its creator, Taiichi Ohno, kanban is one means through which JIT is achieved.

• Kanban is not an inventory control system; it is a scheduling system that helps determine what to produce, when to produce it, and how much to produce.

• The need to maintain a high rate of improvement led Toyota to devise the kanban system. Kanban became an effective tool to support the running of the production system as a whole.

• In addition, it proved to be an excellent way for promoting improvements because reducing the number of kanban in circulation highlighted problem areas.

https://en.wikipedia.org/wiki/Kanban

25

www.scrumindia.in



A Kanban System at my Toyota dealership

https://twitpic.com/het3u

26

www.scrumindia.in



How does it work?

http://www.toyota-global.com/company/vision_philosophy/toyota_production_system/just-in-time.html

27

www.scrumindia.in



How Kanban helps achieve “Just-in-Time”

• For example, to efficiently produce a large number of automobiles, which can consist of around 30,000 parts, it is necessary to create a detailed production plan that includes parts procurement. Supplying "what is needed, when it is needed, and in the amount needed" according to this production plan can eliminate waste, inconsistencies, and unreasonable requirements, resulting in improved productivity.

http://www.toyota-global.com/company/vision_philosophy/toyota_production_system/just-in-time.html

28

www.scrumindia.in



Kanban for Software

• Visualize the Workflow: Represent the work items and the workflow on a card wall or electronic board

• Limit Work-in-Progress (WIP): Set agreed upon limits on how many work items are in progress at a time

• Measure and Manage Flow: Track work items to see if they are proceeding at a steady, even pace

• Make Process Policies Explicit: Agree upon and post policies about how work will be handled

• Use Models to Evaluate Improvement Opportunities: Adapt the process using ideas from Systems Thinking, Deming, etc.

Kanban: Successful Evolutionary Change for your Technology Business – David Anderson

29

www.scrumindia.in



Why Kanban in Software Engineering?

https://leanandkanban.files.wordpress.com/2009/04/kanban-for-software-engineering-apr-242.pdf

30

www.scrumindia.in



What did We learn?

• Process improvement should be

driven by business needs – and NOT because some process looks sexy!

• Don’t let a process limit your potential – think beyond gurus!

• Don’t let absence of a process limit your potential – do whatever it takes to

serve customer better!

31

www.scrumindia.in



Thanks guys!

• To the wonderful Customer Sustaining Team (Dev and QA), Program Management and Tech Support at Network Associates / Network General in Bangalore and San Jose for some great stuff !!

32

www.scrumindia.in



Resources

• Was that Kanban? - http://finance.groups.yahoo.com/group/kanbandev/message/4131 and http://finance.dir.groups.yahoo.com/group/kanbandev/message/4166

• http://refcardz.dzone.com/refcardz/getting-started-kanban• Ship early and ship twice as often, https://www.facebook.com/notes/facebook-

engineering/ship-early-and-ship-twice-as-often/10150985860363920• How we built Flickr,

http://www.plasticbag.org/archives/2005/06/cal_henderson_on_how_we_built_flickr/• Continuous Deployment at IMVU: Doing the impossible fifty times a day,

http://timothyfitz.com/2009/02/10/continuous-deployment-at-imvu-doing-the-impossible-fifty-times-a-day/

• A New Version of Google Chrome now due every six weeks, http://techcrunch.com/2010/07/22/google-chrome-versions/

• In praise of continuous deployment: The WordPress.com story, http://toni.org/2010/05/19/in-praise-of-continuous-deployment-the-wordpress-com-story/

• CONWIP, https://en.wikipedia.org/wiki/CONWIP • Kanban applied to Software Development: from Agile to Lean,

http://www.infoq.com/articles/hiranabe-lean-agile-kanban

33

www.scrumindia.in



Q & A

Questions and Answers?

Applying 'Kanban' in Enterprise-Class Products Sustaining Engineering - An Experience Report

Technology

Transcript of Applying 'Kanban' in Enterprise-Class Products Sustaining Engineering - An Experience Report