Why Agile? Why Now?

79
Why Agile? Why Now? Jorj Bauer < [email protected] > @bozoskeleton Mike Toppa <[email protected]> @mtoppa Wharton Web Conference – July 14, 2011 Thursday, July 14, 2011

description

An overview of Agile and Scrum practices, and a comparison of 2 different, real-life Agile adoption experiences in a University setting. Presented at the Wharton Web Conference, July 2011

Transcript of Why Agile? Why Now?

Page 1: Why Agile? Why Now?

Why Agile? Why Now?

Jorj Bauer <[email protected]> @bozoskeleton

Mike Toppa <[email protected]> @mtoppa

Wharton Web Conference – July 14, 2011Thursday, July 14, 2011

Page 2: Why Agile? Why Now?

Overview

We’re going to talk through two perspectives on the adoption of Agile methodologies

What drove us

Why we approached Agile

How we approached adoption

Our perspective from where we are now

Thursday, July 14, 2011

Page 3: Why Agile? Why Now?

First: who are we?

Thursday, July 14, 2011

Page 4: Why Agile? Why Now?

Jorj Bauer

Manager of Engineering, Research and Development for Penn/ISC

Professional software developer since 1994

Co-owner of DejaVu Software, Inc.

2004-2008: Director of Networking, Penn/SEAS

2008-present: Manager of ERD in Penn/ISC

Thursday, July 14, 2011

Page 5: Why Agile? Why Now?

Jorj: The Past

Most of my software development background has been ad-hoc, with little to no supporting infrastructure

Goals and requirements have been very fluid, with small teams

... until my current position

Thursday, July 14, 2011

Page 6: Why Agile? Why Now?

Mike Toppa

Director, Web Applications for Penn/SOMIS

15 years of experience in web programming, IT project management, and functional management

Georgetown University; Stanford University; University of Pennsylvania

E*Trade; Ask Jeeves

2 failed start-ups (Finexa, Kai’s Candy Co.)

Thursday, July 14, 2011

Page 7: Why Agile? Why Now?

Mike: The Past

None of the places I’ve worked followed any formal development methodology

Thursday, July 14, 2011

Page 8: Why Agile? Why Now?

Mike: Becoming Director

My team was overwhelmed, but didn’t fully realize it

Like boiling frogs

Thursday, July 14, 2011

Page 9: Why Agile? Why Now?

Mike: The Clients

Clients come from across the School’s administrative, academic, and research organizations

Hybrid funding model

Clients were working directly with developers

Usually no project managers

Reactive planning

Thursday, July 14, 2011

Page 10: Why Agile? Why Now?

Mike: The Projects

A large number of projects

Little to no documentation

No one in a position to prioritize

Thursday, July 14, 2011

Page 11: Why Agile? Why Now?

Mike: The Team

Talented

Independent

Customer service oriented

Focused on fast delivery

Thursday, July 14, 2011

Page 12: Why Agile? Why Now?

Mike: The Dev Environment

No specific development procedures

An aging development framework

No automated testing

Non-critical bugs and missed requirements common

Daily firefighting

Difficulty planning

Thursday, July 14, 2011

Page 13: Why Agile? Why Now?

Jorj’s challenges

Thursday, July 14, 2011

Page 14: Why Agile? Why Now?

Jorj: The New Department

In 2008, I became manager of 5 direct reports, in a department of ~30, full of dotted-lines and large projects

Part of a reorganization of the department

Two large projects-in-progress with campus-wide scope were instantly “mine”

Thursday, July 14, 2011

Page 15: Why Agile? Why Now?

Jorj: The Projects

Rapid shifting of priorities

Large, with no distinct requirements

Disconnect between technical requirements for a good service, level of effort to learn new services, and management expectations of level of effort required

Everything is behind schedule

A lot of fire-fighting related to the reorg as well as project management culture in the department

Thursday, July 14, 2011

Page 16: Why Agile? Why Now?

Jorj: The Team

Very talented and conscientious

Not a lot of project focus; rapid shifting of priorities

A lot of unrest due to uncertainty and perceived political stupidity

Thursday, July 14, 2011

Page 17: Why Agile? Why Now?

Jorj: The Dev Environment

Everyone For Themselves

Central RCS and CVS repositories, so some people chose to set up their own SVN repos

No specific development procedures

No automated testing

Projects begin and take priority based on political demand

Thursday, July 14, 2011

Page 18: Why Agile? Why Now?

Jorj: The Biggest Challenge

“Project Management” happened between developers and their direct management – no higher, and no wider

The culture of “how project management works” was dysfunctional

Thursday, July 14, 2011

Page 19: Why Agile? Why Now?

Prioritizing Challenges

Both of us approached our situations similarly:

1.Get a handle on existing and upcoming projects

2. Improve predictability

3. Improve quality

Thursday, July 14, 2011

Page 20: Why Agile? Why Now?

Three Approaches...

In software development, you’re looking at one of these three general approaches:

Ad-hoc (informal)

Waterfall (anticipation)

Agile (adaptation)

Thursday, July 14, 2011

Page 21: Why Agile? Why Now?

Waterfall

Thursday, July 14, 2011

Page 22: Why Agile? Why Now?

Waterfall: why so pervasive?

“Nobody ever got fired for buying IBM”

Many people understand Waterfall

It seems sensible to wait for one thing to be complete before working on the next

The customer gets a whole solution, all at once

Thursday, July 14, 2011

Page 23: Why Agile? Why Now?

Waterfall: low success rates

http://www.ambysoft.com/surveys/Thursday, July 14, 2011

Page 24: Why Agile? Why Now?

Waterfall: why does it often fail?

Customers never know what they want up front; requirements always change

Customers and developers speak two different languages

Progress reports are often wrong, or set up a false expectation

If I’m 100% done the first task of 4... am I 25% done now?

Thursday, July 14, 2011

Page 25: Why Agile? Why Now?

Ad-hoc

Home grown workflows, if any

Prioritization is arbitrary or politicized

Often hard to do long term planning

Success dependent on personalities and circumstances

Thursday, July 14, 2011

Page 26: Why Agile? Why Now?

Agile

We are uncovering better ways of developing software by doing it and helping others do it. Through this work we have come to value.

Individuals and interactions over processes and tools

Working software over comprehensive documentation

Customer collaboration over contract negotiation

Responding to change over following a plan

That is, while there is value in the items on the right, we value the items on the left more.

©2001 authors of The Agile Manifestoagilemanifesto.org

Thursday, July 14, 2011

Page 27: Why Agile? Why Now?

Agile is incremental and iterative

Graphic by Jeff Patton

Thursday, July 14, 2011

Page 28: Why Agile? Why Now?

Agile: “inspect and adapt”

Thursday, July 14, 2011

Page 29: Why Agile? Why Now?

The Agile Umbrella

“Agile” applies to a large swath of development methodologies, many of which overlap

Scrum; Kanban; Crystal; Extreme Programming (XP); Lean...

And their hybrids

Scrum-Ban; ADROIT; WetAgile/Agile Waterfall...

Disclaimer: not all of these may be “good”

Thursday, July 14, 2011

Page 30: Why Agile? Why Now?

Scrum: overview

Graphic by Skip Angel

Thursday, July 14, 2011

Page 31: Why Agile? Why Now?

Slice vertically,not horizontally

Thursday, July 14, 2011

Page 32: Why Agile? Why Now?

Thursday, July 14, 2011

Page 33: Why Agile? Why Now?

Mike: Why Agile? Why Scrum?

Maintain frequent interactions with clients

Provides quick feedback

Existing good relationships

But have them take more ownership of their business process

Thursday, July 14, 2011

Page 34: Why Agile? Why Now?

Mike: Why Agile? Why Scrum?

Put structure around our work

Enable planning

Make commitments, and meet them

Reduce need for firefighting

Reduce risk

Have more than one person know a project

Thursday, July 14, 2011

Page 35: Why Agile? Why Now?

Mike: Why Agile? Why Scrum?

Improve quality

Reduce misunderstandings

Reduce missed requirements

Have fewer bugs

Increase technical knowledge sharing

Thursday, July 14, 2011

Page 36: Why Agile? Why Now?

Authority vs. Responsibility

Cartoon by Mike Lynch

Used with permission

Thursday, July 14, 2011

Page 37: Why Agile? Why Now?

What makes a job enjoyable?

Autonomy

Reward for effort

Challenging/complex work

“Work that fulfills these three criteria is meaningful.”

– Malcolm Gladwell, “Outliers: The Story of Success”

Thursday, July 14, 2011

Page 38: Why Agile? Why Now?

Scrum has 3 roles

Product owner

Authority and responsibility for “what”

Programming team

Authority and responsibility for “how”

Scrum Master

Authority over the Scrum process

Responsibility for driving continuous improvement

Thursday, July 14, 2011

Page 39: Why Agile? Why Now?

Scrum role: Product Owner

Responsible for what the team will work on, but now how the work is done

Thursday, July 14, 2011

Page 40: Why Agile? Why Now?

Scrum role: The Team

Takes collective responsibility for doing the work

Thursday, July 14, 2011

Page 41: Why Agile? Why Now?

Scrum role: Scrum Master

A “servant-leader” for the team

Thursday, July 14, 2011

Page 42: Why Agile? Why Now?

Agile Adoption Patterns

Top-down vs. Bottom-up

“All in” vs. incremental

Thursday, July 14, 2011

Page 43: Why Agile? Why Now?

Mike: top-down

Upper management: supportive

Team: uncertain

They recognized certain problems

But learning and conforming to a process

feels like a loss of autonomy

and yet another thing that has to be done

Thursday, July 14, 2011

Page 44: Why Agile? Why Now?

Mike: all-in

Team learned core scrum concepts

I explained the new process to clients

A small team did a pilot project first

Then the whole group switched

Thursday, July 14, 2011

Page 45: Why Agile? Why Now?

Mike: initial transition

It did not go very well

Team not used to working together

I misunderstood the roles

I overemphasized doing only one project at a time

Too many “scrum-buts”

Thursday, July 14, 2011

Page 46: Why Agile? Why Now?

Mike: a visualization

Cartoon by Esther Derby

Used with permission

Thursday, July 14, 2011

Page 47: Why Agile? Why Now?

Mike: using scrum coaches

I brought in coaches

They provided good training

Led candid discussion of problems

Most long pre-dated my introduction of scrum

Thursday, July 14, 2011

Page 48: Why Agile? Why Now?

Should you use a coach?

A skilled, external coach is often key for driving change

If you haven’t done it before, it’s surprisingly easy to introduce Agile the wrong way

You need at least enough management support to pay them

You need to make sure you’re bringing in someone good

Thursday, July 14, 2011

Page 49: Why Agile? Why Now?

But it’s still difficult

“In my Scrum classes I warn attendees of what I call the Scrum Promise: If you adopt Scrum, there will be a day you come into the office nearly in tears over how hard the change can be. This is because Scrum doesn’t solve problems, it uncovers them and puts them in our face. Then, through hard work we address them.”

– Mike Cohn, Agile Trainer

Thursday, July 14, 2011

Page 50: Why Agile? Why Now?

Jorj: Bottom-up, Incremental

Thursday, July 14, 2011

Page 51: Why Agile? Why Now?

Jorj: Bottom-up, Incremental

Ad-hoc

The “Cloak of Confusion” surrounded project management; most staff couldn’t see the problems, just the symptoms

The staff perceived control in their current methodology; nobody wants to give up control

Thursday, July 14, 2011

Page 52: Why Agile? Why Now?

Why Incremental?

Convincing staff to change their culture is hard

Resources completely overcommitted

It’s difficult to “sell” a change without having data on how it will work

It’s impossible to show results for a system without using it

Thursday, July 14, 2011

Page 53: Why Agile? Why Now?

NES’ Transition

Phase 1: adopt formal project workflow

Built a formal Iterative Waterfall process within the existing framework

A lot of resistance and much more project planning but improved results for existing projects

Increased awareness around product ownership, sources of delay, and available resources

Warning: this will not make you new friends

Thursday, July 14, 2011

Page 54: Why Agile? Why Now?

Incrementally Improving

“Iterative Waterfall” is single-looped: allowed the team to abandon the waterfall process at many points (causing the project manager lots of additional work)

Continue working on the process itself

Thursday, July 14, 2011

Page 55: Why Agile? Why Now?

Showing Results

This slowed everything down

3-4x the project management work (for me)

Allowed us to find and address deep-seated problems in workflow

Yields an inventory of projects

Gave me the ability to say “No” to new work

Thursday, July 14, 2011

Page 56: Why Agile? Why Now?

Stepping back a moment...

Thursday, July 14, 2011

Page 57: Why Agile? Why Now?

Saying “No” Is Important

If you take on an infinite number of projects, then your staff are constantly task-swapping (unless you have infinite staff)

Context switching between two projects eats about 20% of a full-time worker’s schedule

Cutting back on the number of active projects is key to getting back lost productivity from a bottlenecked staff

Thursday, July 14, 2011

Page 58: Why Agile? Why Now?

Danger Signs of Meetings

In the meeting but doing other work while present

Being unavailable, or arriving late, for regularly scheduled

meetings

Suggesting radical departures from the current plan, at a stage that’s too

late

Never speaking in a meeting you attend regularly

Thursday, July 14, 2011

Page 59: Why Agile? Why Now?

Reduce Meeting Complexity

Agile methodologies adopt more frequent meetings (generally daily). Staff specifically talk about:

What I did yesterday

What I’m doing today

What I’m waiting for

This is a more effective use of tech staff time

(But meetings may be a band-aid for other problems)

Thursday, July 14, 2011

Page 60: Why Agile? Why Now?

Time/Project Management

Brooks’ Law

adding manpower to a late project will make it later

Eliyahu Goldratt’s Theory of Constraints

Identify the bottlenecks and use them to predict and control the workflow

Thursday, July 14, 2011

Page 61: Why Agile? Why Now?

Thursday, July 14, 2011

Page 62: Why Agile? Why Now?

Thursday, July 14, 2011

Page 63: Why Agile? Why Now?

Answer: both have to be done as soon as possible.

Better answer: look at the costs, benefits, and staff resources for both and be able to shelve one

NES’ Turning Point

Two services: one brings in 2/3 of your department’s budget, and one brings in almost nothing. How do you prioritize the work?

Answer: both have to be done as soon as possible.

Thursday, July 14, 2011

Page 64: Why Agile? Why Now?

No Looking Back

From here, big gains were made fairly quickly on teams that adopted these Agile practices:

Daily stand-ups; reduced queue sizes; more direct interaction with people “outside” the team

Better results on a shorter timeline within 4 months

Managers discussing and inspecting the process itself and its results, and are improving upon it

Not Done Yet!

Thursday, July 14, 2011

Page 65: Why Agile? Why Now?

Mike: taking inventory

Thursday, July 14, 2011

Page 66: Why Agile? Why Now?

Mike: taking inventory

9 developers, 2 product owners, and me supporting

22 clients with

124 applications

3 designers and 1 product owner supporting

about 200 static content web sites

Thursday, July 14, 2011

Page 67: Why Agile? Why Now?

Mike: our maintenance curve

Thursday, July 14, 2011

Page 68: Why Agile? Why Now?

Mike: project commitments for first 6 months... oh crap

Thursday, July 14, 2011

Page 69: Why Agile? Why Now?

Mike: the first 6 months

Working through our over-commitment

While learning a new way to work

The continuous 2 week delivery cycle highlighted process weaknesses

Staff changes

Thursday, July 14, 2011

Page 70: Why Agile? Why Now?

Planning and estimating

How did we come up with that chart?

Agile requirements gathering

Agile estimating techniques

Thursday, July 14, 2011

Page 71: Why Agile? Why Now?

Requirements gathering

Breaking down a project into feature areas

Epics

Breaking down feature areas into individual features

User stories

How do you know when you’re done?

Conditions of satisfaction

Thursday, July 14, 2011

Page 72: Why Agile? Why Now?

Estimating: story points and planning poker

Photo by Kelly Hirano

Used with permission

Thursday, July 14, 2011

Page 73: Why Agile? Why Now?

Velocity enables scheduling

Thursday, July 14, 2011

Page 74: Why Agile? Why Now?

Mike: one year later

We can plan and estimate better

Our programmers work together

Our next challenges

A project prioritization and intake process

Improving our technical practices

Thursday, July 14, 2011

Page 75: Why Agile? Why Now?

Manifesto for Half-Arsed Agile Software Development

We have heard about new ways of developing software bypaying consultants and reading Gartner reports. Through

this we have been told to value:

Individuals and interactions over processes and toolsand we have mandatory processes and tools to control how those

individuals (we prefer the term ‘resources’) interact

Working software over comprehensive documentationas long as that software is comprehensively documented

Customer collaboration over contract negotiationwithin the boundaries of strict contracts, of course, and subject to rigorous change control

Responding to change over following a planprovided a detailed plan is in place to respond to the change, and it is followed precisely

That is, while the items on the left sound nicein theory, we’re an enterprise company, and there’s

no way we’re letting go of the items on the right.http://www.halfarsedagilemanifesto.org/

Thursday, July 14, 2011

Page 76: Why Agile? Why Now?

Technical practices:ridiculously brief overview

“Specification by Example” by Gojko Adzic

“Clean Code” and “Agile Software Development, Principles, Patterns, and Practices” by Bob Martin

“Refactoring” by Martin Fowler

“Agile Database Techniques” by Scott Ambler

“Growing Object Oriented Software, Guided by Tests” by Steve Freeman and Nat Pryce

Thursday, July 14, 2011

Page 77: Why Agile? Why Now?

Agile is relevant beyond software development

“Angry Dinosaurs: Accelerating Change and Institutional Incompetence”

Cory Ondrejka, Wharton Web Conference, 2010

http://bit.ly/bY4E15

“Let it Roll: Why more companies are abandoning budgets in favor of rolling forecasts”

CFO Magazine, May 1, 2011 http://bit.ly/k94Sfc

Thursday, July 14, 2011

Page 78: Why Agile? Why Now?

Other References

Eliyahu M. Goldratt, “The Goal” (Theory of Constraints)

Fred Brooks, “The Mythical Man-Month”

Gerald Weinberg, “Quality Software Management: Systems Thinking”

“Convincing Management That Context Switching Is a Bad Idea”, Johanna Rothman http://www.jrothman.com/Papers/context-switching.html

“In Praise of Bad Programmers” http://corvusintl.com/

Thursday, July 14, 2011

Page 79: Why Agile? Why Now?

Any Questions?

Jorj Bauer <[email protected]> @bozoskeleton

Mike Toppa <[email protected]> @mtoppa

Wharton Web Conference – July 14, 2011Thursday, July 14, 2011