Aot Version 2

8/12/2019 Aot Version 2

1/26

IBM Academy of Technology

Second Agile Methods and Practices Conference

McKimmon Center at NC State UnivJune 23-25, 2008Agile@IBM

Agile DevelopmentA System Test Perspective

Dave [email protected]

WPLC (Lotus)
mailto:[email protected]:[email protected]


2/26

IBM Academy of Technology: Second Agile Methods and Practices Conference

McKimmon Center at NC State Univ

June 23-25, 2008 Agile@IBM2

Premise of the Talk

Agile development may be one of the keys to the future success of IBMsoftware development

Much focus has been put into the development, unit test, and function testaspects of the Agile development process

Very little documented focus in the Agile community on system test

System Test is usually the phase where IBM products are first tested usingCustomer-like configurations, loads, data, etc.

Lack of System Test involvement leads to products that may be theoreticallysound but the products may not meet Customers expectations

Involving System Test increases the length of the cycle times needed forcomplex products


3/26



June 23-25, 2008 Agile@IBM

Outline of Talk

Review of the agile process

How the agile process is being implemented inside WPLC products

What is role of system test in the product cycle

What effect does system test have on the agile process

Conclusions

3


4/26




Review: The Agile Development Execution Cycle for a Team

4

Decide onUse Cases

DesignSolution

Code

Unit Test

Two main goals of the agile methodology are to minimize development risk and to

increase customer satisfaction by rapidly and continuously delivering software thatcustomers want and need

Time t


5/26


6/26




Agile Development and Large Projects

All WPLC products are moving to agile development processes

Many of these products are made up of 100s of developers

The developers are split into smaller teams, each focused on an area(e.g., mail router, portlets, base WAS integration, installation, security,etc.)

Each of these individual teams is implementing the agile methodology team meetings, goals, use cases, iterations, etc.

6


7/26




Agile Development for a Single Team Time Bound Iteration

7

Decide onUse Cases

DesignSolution

Code

Unit Test

Time t


8/26




Agile for Large Numbers of Teams

Each individual team is doing their own work, during a time boundediteration

Several sets of teams are doing their work in parallel during the sametime bounded iteration

The output from each of these individual teams must be mergedtogether during the same time bounded iteration

NOTE: when time gets tight, unit test is often given less attention

8


9/26




The Development Execution Cycle for a Product - Time Bound Iteration

9

Decide on

Use Cases

DesignSolution

Code

Unit Test

Decide onUse Cases

DesignSolution

Code

Unit Test

Decide on

Use Cases

DesignSolution

Code

Unit Test

Decide onUse Cases

DesignSolution

Code

Unit Test

Decide on

Use Cases

DesignSolution

Code

Unit Test

Decide onUse Cases

DesignSolution

Code

Unit Test

Time t

Team 1 Team 2 Team 3

Team 4 Team 5 Team n


10/26




The Agile Process on a Large Product

teams doiterative

development

Team Integrateindividual output

10

Decideon

UseCase

s DesignSolut

ion

Code

UnitTest

Decideon

UseCase

s DesignSolut

ion

Code

UnitTest

Decideon

UseCase

s DesignSolut

ion

Code

UnitTest

Decideon

UseCase

s DesignSolut

ion

Code

UnitTest

Decideon

UseCase

s DesignSolut

ion

Code

UnitTest

Decideon

UseCase

s DesignSolut

ion

Code

UnitTest

Decide onUse Cases

Integratepieces

Functiontest

Time t


11/26




What Am I Trying to Say

The one constant in the previous charts is the iteration time t

During an iteration of length t the individual teams develop their pieces

During an iteration of length t the whole TEAM must integrate all thepieces together to form a usable product

It is often the case that the pieces cannot be integrated until after all theindividual teams are complete

With complex/large products, the number of pieces may be 10-20

Bottom line: there is a lot of work to do during the iteration of length t , just to develop and unit test the pieces and integrate them together; Thesmaller the value for t , the less you can actually do during an iteration

11


12/26




Outline of Talk


How the agile process is being implemented for large projects

What is role of system test in the agile cycle

What has WPLC tried

Conclusions

12


13/26




What are the Goals of System Test?

The Mission of the System Test organization is to efficiently andeffectively validate that Lotus products meet our customers' highexpectations in the areas of consumability, reliability, andinteroperability.

We accomplish our mission by:Test using real, Customer- representative, complex environments

Test the product using interactions that simulate Customer traffic

Test the product using real Customer -like loads, volume of data, numberof transactions per second, number of users, number of logins, etc., and bykeeping the systems running for long periods of time

Test with the same enterprise vendor products our Customers use

13


14/26




What does this really mean

Domino approximately 26 configurations, 9+ different operatingsystems or versions, 12+ additional products used in integrationscenarios, terabytes of data, over 1000 defects opened by the systemtest team during the 8.0 release

Portal approximately 25 configurations, 12+ different operatingsystems or versions, 7 different LDAPs or versions, 9 different DBs orversions, 9+ additional products used in integration scenarios, 1000 s ofusers, 100 s of GB of data, over 1100 defects opened by the systemtest team during the 6.1 release

System Test finds defects that our Customers are likely to find whenthey deploy and exercise our solutions

14


15/26




The System Test Execution Cycle

15

Install thebuild

Migratethe data

Ramp upthe test

Discovera defect

Debugthe defect

Fix thedefect

Time t


16/26




Example: Configuration for ST/Portal

16

Decideon

UseCase

s DesignSolut

ion

Code

UnitTest

Decideon

UseCase

sDesign

Solution

Code

UnitTest

Decideon

UseCase

s DesignSolut

ion

Code

UnitTest

Decideon

UseCase

sDesi

gnSolut

ion

Code

UnitTest

Decideon

UseCase

s DesignSolut

ion

Code

UnitTest

Decideon

UseCase

sDesign

Solution

Code

UnitTest

Decide on

Use Cases

Integratepieces

Functiontest


17/26




Focus

Focus on load testing, duration testing, and product integration testing

Focus on edge conditions e.g, when 1000 s of users all login at thesame time, when multiple applications make LDAP requests at thesame time, etc.

Focus on multiple product integration (under load) scenarios thingsoften work well in isolation it is often interactions that cause problems

Focus on volumes of data things work well with 10 users, or whenthere are 10 content elements it is when the numbers get big thatthings get interesting

17


18/26


19/26




The Agile Process on a Large Product

teams doiterative

development

Team Integrateindividual output System Test

19

Decideon

UseCase

s DesignSolut

ion

Code

UnitTest

Decideon

UseCase

sDesignSolution

Code

UnitTest

Decideon

UseCase

s DesignSolut

ion

Code

UnitTest

Decideon

UseCase

sDesignSolution

Code

UnitTest

Decideon

UseCase

s DesignSolut

ion

Code

UnitTest

Decideon

UseCase

sDesignSolution

Code

UnitTest

Decide onUse Cases

Integratepieces

Functiontest

Install thebuild

Migratethe data

Ramp upthe test

Discover adefect

Debug thedefect

Fix thedefect

Time t


20/26


21/26




Outline of Talk


How the agile process is being implemented for large projects

What is role of system test in the agile cycle

What has WPLC tried

Conclusions

21

d f h l d l h d d f


22/26




The Hannover Agile Process

22

Team 1

Team 2

Team n

Team 3

FVT

FVT

FVT

FVT

SVT

Timeline

End of Iteration- Outputgiven to customer

Iteration 2 Iteration 3

Team 1

Team 2

Team n

Team 3

FVT

FVT

FVT

FVT

SVT


..

SVT

IBM A d f T h l S d A il M h d d P i C f


23/26




System Test Lessons from Hannover

Early involvement essential need system test folks on each teamThe output of the iteration was usable by a Customer

Iterations lasted 10-12 weeks

The perceived quality of the final product is better

This would never have worked with 4-6 week iterations

There was one iteration solely focused on bug fixes

23

IBM A d f T h l S d A il M h d d P i C f


24/26




The Portal Agile Process

24

Team 1

Team 2

Team n

Team 3

FVT

FVT

FVT

FVT

PVT

Timeline


Iteration 1 Iteration 2

Team 1

Team 2

Team n

Team 3

FVT

FVT

FVT

FVT

PVT


..

Team n+1

SVTeFVT

IBM A d f T h l S d A il M th d d P ti C f


25/26




System Test Lessons from Portal

Early involvement essential need system test folks on each teamThe output of an iteration was NOT consumable by a Customer

Integration was attempted on last day(s) of an iteration

Cycle time was 4-6 weeks

PVT was a good educational experience for system test team, BUT itdidn t drive the types of bugs we had hoped to find

We modified our approach to system test as the project evolved

Many improvements could be suggested, including that we should havechosen longer iteration cycles

25

IBM A d f T h l g S d Agil M th d d P ti C f


26/26


McKimmon Center at NC State UnivAgile@IBM

Conclusions

Agile is important to IBM; it allows us to get products to Customers morequicklyCustomers expect the products we deliver, regardless of the cycle time,to properly deploy and operate in their complex environmentsThe system test discipline drives out a set of problems that other phasesof test cannot discoverSystem test teams must engage early in the development processSystem test teams must be flexible in their approach to testingIteration lengths, t , need to be well thought out and planned.

Agility is NOT code for no planning; more planning, includingintegration planning, is needed for agile development to be successful

26

Aot Version 2

Documents

Transcript of Aot Version 2