W16

Agile Testing

5/7/2014 3:00:00 PM

Meet Big Agile: Testing on

Large-Scale Projects

Presented by:

Geoff Meyer

Dell, Inc.

Brought to you by:

340 Corporate Way, Suite 300, Orange Park, FL 32073

888-268-8770 ∙ 904-278-0524 ∙ sqeinfo@sqe.com ∙ www.sqe.com

Geoff Meyer

Dell, Inc.

A test architect in the Dell Enterprise Solutions Group, Geoff Meyer has more than twenty-eight years of experience as a software developer, manager, business analyst and test architect. Geoff helps serve the software development practices of more than 800 development, test, and UX engineers across four Global Design Centers. Geoff is an active member in and contributor to the Agile Austin community.

4/26/2014

1

Agile Testing @ LargeGeoff Meyer, geoffrey_meyer@dell.com

May 2014Last updated: 25 February, 2014

Courtesy: Gulliver's Travels (2010)

Session Objectives

• Challenges to expect with Agile practices in

large-scale, multi-team projects

• Brief overview of Industry Models

• Real-world examples − from a large organization

perspective

• Organizational strategies & considerations

• Test automation considerations

• An In-depth Tools Discussion

• Focused on how to develop Test Automation

IS

IS

NOT

4/26/2014

2

Geoff Meyer

• Dell Inc, 1998 – present

– Test Architect

› Agile Test & Automation Strategy

– Agile Steering Committee Co-Chair

– Global Projects

› 15 Scrum teams (2 regions)

› 11 Scrum teams (4 regions)

› 9 scrum teams (2 regions)

› 7 Scrum teams (4 regions)

• NCR Corp. 1984 – 1998

– SW developer, Project Lead, SW Manager

• B.S. Computer Science, San Diego State University

• Masters Engineering Management - NTU

3

4

Agenda

• The Dell Context and

• Early Agile Testing Challenges

• In Search of… Solutions for Scale

• Adaptations for Testing at Scale

• Automating in an Agile world

• Lessons of Large

Agile @

4/26/2014

3

5

The Context at Dell ESG

http://amazngwallpapers.blogspot.com

Dell Enterprise Solutions GroupGlobal Design and Development using Agile Scrum

6

Noida Design

Center

Bangalore

Design Center

Austin

Design Center

Silicon Valley

Design Center

4/26/2014

4

Dell Enterprise SolutionsSoftware Products

• Server Systems Management

• Converged Infrastructure Systems Management

• Private Cloud Systems Management

• Console Plug-ins (i.e. for SCCM, vCenter…)

7

Common Product Characteristics:

• Embedded Software

• Large hardware support test matrix

• Software is installed in the Data Center

• Enterprise update cycles ~6 months

• Products must function even as underlying HW, FW,

BIOS, and Drivers are upgraded - SUSTAINING

Storage

Network

Compute

The Early Challenges

8

?

http://cleaneatingchelsey.com/

4/26/2014

5

Culture Transition

• Early commitment on Requirements

• Insufficient Detail in Requirements

• Development vs. Test

• A different “School” of Test1

• Fail Early, Inspect & Adapt

91 – Scott Barber “Approaches to Software Testing: An Introduction “

“ “Everybody, every team, every platform, every division, every component is there not for individual competitive profit or recognition, but for contribution to the system as a whole”

W. Edwards Deming

Signs of Trouble• Thursday’s Build

• Planning more stories into sprint than team’s velocity

• High % of carry-overs

• Inability to Automate Acceptance Tests within Sprint

• “Engineering” Stories

Large-Scale

• Cross-team interdependencies block sprint progress

• Large HW configuration validation consumes test resources

• No single team had visibility to the entire product

• Insufficient Product Owner bandwidth

http://www.huffingtonpost.com/2012/03/21/truck-stuck-thomas-

circle_n_1369545.html?p=truck+and+underpass&vm=r

4/26/2014

6

Test Automation Pitfalls

• Minimal guidance to Test beyond “Go forth and automate”

• Black Box Test Mentality

• Automation was interpreted by many as “Automate the UI”

11

• Insufficient SW engineering skills across Test teams

• Architecture(s) not optimized for Automatability

12

In Search of…

Solutions For Scale

4/26/2014

7

Leverage the Community

13

Books

Conferences

Professional Groups

ConsultantsDallas Agile Leaders’ roundtable

Scaled Agile Framework™ Big Picture

© 2008- 2013 Scaled Agile, Inc and Leffingwell, All rights reserved

4/26/2014

8

Agile Testing Quadrants

15

4 - http://lisacrispin.com/wordpress/2011/11/08/using-the-agile-testing-quadrants/

4

ALM Tooling

16

• Multi-team, geographically distributed projects create a necessity for ALM Tooling

–Requirements Management

–User Story Management

–Test Case Management

–Defect Management

–Test Script Repository

4/26/2014

9

Automation Tooling

17

• IDE

• UI Tools

• Scripting Languages

• Services

• Frameworks

18

Agile @ DellAdaptations

4/26/2014

10

Agile Testing @ PG Enterprise Solutions Group How did we go about it?

19

Process/Project

Culture/Organization

X-Large

project (15)

2011 2014

UI Automation

“Automate-First”

Service-level

automation ALM using

TFS

In-sprint

Automation

Accelerate

Automation

Skillset

Maximize

UtilizationAutomation

Architects

Automated

Setup & Config

4 Geography

project (11)

2012

TDD &

Automated

UT

Non-Functional

Scale, Longevity

2010 2013

Test into

Scrum

2009

Collaborative

Seating

Rally

Agile @ DellCore Activities

• Pre-Sprint activities:

– Staffing

– Training - Project tools and processes

– CI/Build environment

– Automation Framework and BVT

• Establish Project-wide ‘Done’ criteria

• End-to-end, short duration User Stories

• Test Automation is included User Story acceptance criteria

• Refresher Workshops for new projects

20

4/26/2014

11

Week 1 Week 2

2 week Sprint Lifecycle Example

Manual Regression

Automated

Regression

Automated

Regression

Automated

Regression

Sprint

Planning

Automated

Regression

Automated

Regression

Sprint

Review

Retro

spective

Demo

Accept

User

Story

Demo

Accept

User

Story

Demo

Accept

User

Story

Demo

Accept

User

Story

Demo

Accept

User

Story

Product Owner (or Proxy)

Scrum Master

Development

Test

Tech Pubs

UI

22

In-Sprint: Testing

• Design/Develop Test cases and Review with team

• Develop test automation and maintain in TFS

• Execute test cases (manual and automated)

• Immediate Feedback (IM, email, in-person) with team of

sightings… do not automatically enter defects

• Verify fixes for defects; re-test if needed

• Maintain test case status (in MTM)

• Cross-team coordination including Weekly SoS

Confidential

4/26/2014

12

Agile @ Dell Roles/Responsibilities

23

Product

Owner

Product Owner Proxy1 per Scrum

Scrum Master1 per Scrum

Development4-5 per Scrum

Test3:1

Tech Pubs1 per 3 scrum team

UI1 per Scrum

Scrum teams

• Small teams

• Co-located

• Multi-disciplined

• Feature-based

• Distributed Projects

• Scrum of Scrums

• Teams formed from

functional silos

…

Release Manager

SW Architect

Test Architect

Automation Architect

Cross-team Test roles

24

• Test Architect

– Defines Test Strategy across all Scrum teams

– Establishes non-Functional Test approach

– Influences consistent Application Architectures

• Automation Architect

– Leads Analysis and POC activities for new automation needs

– Ensures automation consistency for BVT and nightly regression suites

– Safeguards against automation tool proliferation

4/26/2014

13

EXAMPLE - Project-wide “Done” criteria

25

Code

– Junit Tests

– SONAR

– Code Reviews completed

– All User Story tasks completed

– Demo and accepted by Product Owner

– All known error conditions caught and handled appropriately

– Code is Globalized

Test

– Acceptance Tests Reviewed with scrum team

– Acceptance Tests documented in Tool

– All planned Acceptance Tests for User Story has been attempted

– Acceptance Tests executed against Test system configuration

– Defects that are “must fix” before releasing must be fixed prior to story acceptance

– Service-level automated tests

– Service-level automated Tests are globalized

Large-Scale Complexities

26

Characteristic Description

Large-Scale Project consists of more

than 4 Scrum teams

Interdependency Requirements

implemented across

multiple Scrum teams

Extensive

Configuration

Matrix

Extensive HW or SW

configurations

Waterfall

Intersection

One or more

components are

managed using Waterfall

Geography Project members are

Geographically dispersed

4/26/2014

14

Development

Usability

Test

Wireframes

Architecture

Automation FW/Tools

Release Planning

Pre-sprints

Pre-sprints

HW resources

HW resources

Pre-sprints

: Agile with Adaptations

27

Hardening

1 2 3 N-1 N

1 2 3 N-1 N

1 2 3 N-1 N

… Sprints

Define Plan Develop Launch

OLP

…

Software System Test

Extended Sprint Test

Stability

Agile @

Extended Sprint TestExtended Hardware Configurations

• Optimizes HW utilization and cost in support of Scrum teams

• Owns Test Execution against extendedconfigurations

• Scrum team:

– Establishes Acceptance criteria

– Identifies all Test scenarios

– Owns Test Execution against reference configuration(s)

– Develops in-sprint automation tests

28

Configurations

Positive X X X X

Negative X X X

Boundary X X X

Stress X X

Scalability X X X

Concurrency X X X

Globalization X X

Candidate for

Extended Sprint Test

4/26/2014

15

Pre-SST

Software System Test“Alpha Customer”

29

Hardening

• Workflow Design

• Test Analysis

• Test Design

• Test Development

• Test Automation

Development

• Test Execution

• Regression Test

• Test Execution

• Regression Test

SST

IS IS NOT

Requirements-based Functional testing

Customer-usage based First time integration test of workflows

Interoperability coverage of HCL in

specific customer environments

Exhaustive interoperability matrix

coverage of the HCL

Three phased approach Performance baseline testing

SAFe1 &

30

Agile @

1- Courtesy of Dean Leffingwell and Al Goerner of Valtech

Scrum teams

System team

4/26/2014

16

PG Enterprise Validation

Agile & Waterfall Converged Model

Dell Confidential31

SST

Hardware Product Test

EST

FinalEST NEST 1

Sprint1 Sprint2Sprint N-1

�

SprintN

Sprint1 Sprint2Sprint N-1

SprintN

Sprint1 Sprint2Sprint N-1

SprintN

Hardening

Stability

Sprint

Extended Sprints

SW Regression

Test

Solution stack with

software focus

�

SST

EST Exit

CCB Review

Code Freeze

A-Rev

Final

Extended Regression

F

V

PT

Stability

SST

A-Rev

Candidate

SSI

�Stability

SprintRC-0

Regression

Regression

HW-Specific User Stories Complete

HW Enablement Features Complete

Vendor Deliverables Required

100% PT Attempted

All Stories Accepted

Feature Complete

X-on-A Stories

Accepted

PG Enterprise Validation

Agile @ DellGeography

32

• Co-location of a Scrum team is always preferred

• Limit geographical distribution of Scrum team to no more than two time-zones

• Scrum teams within a project can be distributed

4/26/2014

17

33

Automating in an Agile World

Opportunities for Automation

• It’s not just about automating test cases

• Can also provide efficiencies to:

–Test Preparation, Setup and Configuration

–Non-Functional Testing (“ility”)

› Longevity, Scale and Performance Characterization

–Compatibility Testing (Solution and Device)

34

4/26/2014

18

Test Preparation

• Bare-metal Deployment

• Setup & Configuration

– OS Provisioning and Configuration

• Test case staging

• Environment Cleanup/Baseline

• Virtualization-based test environments

35

Test Content Development

• UI automation on Customer Usage workflows

36

CAUTION: Application Architecture can be an enabler or inhibitor

• Unit Test Automation

• In-Sprint, Automated Acceptance Tests

• Web-services (or CLI) Test Automation for Functional Testing

• Automated CI (UT, Build, BVT)

2 - Agile Testing: A Practical Guide for Testers and Agile teams

2

4/26/2014

19

Application Architecture Matters

• Does it facilitate testing at the API/Services level?

• Does the Business/Error Logic reside below the services level?

• Does the UI architecture support test automation other than record-playback?

37

Large Scale Organizations:• Standardize UI architectures across the product

portfolio• HTML5, Flex, Silverlight…

• Standardize Service architectures across the

product portfolio• SOAP, REST, CLI, API

Non-Functional Testing

• Performance Characterization

• Longevity

• Stress

• Scale

• Concurrency

38

Often times analysis of these areas are simply too cost-prohibitive to be done manually

3- http://lisacrispin.com/wordpress/2011/11/08/using-the-agile-testing-quadrants/

3

4/26/2014

20

Confidential39

Functional

Validation

Performance-

Characterization and

non-Functional Validation

Longevity

11 M2 1.0

SST

Baseline

M1

Scale

Concurrency

Simulation

L

a

r

g

e

-

S

c

a

l

e

L

a

r

g

e

-

S

c

a

l

e

12 13 17 18 19�

Sprint 15

Performance Characterization Timeline

14 16

Sprint 21

Key Dependency:

• 3rd party HW available

• 3rd party user stories

completed

The Lessons of ‘Large’

40

Courtesy: Gulliver's Travels (2010)

4/26/2014

21

The Lessons of ‘Large’

1. Co-located Teams

2. Project-wide ‘Done’ Criteria

3. Culture of Automation

4. System Test Team

5. Test Architect

6. ALM Tooling

Questions?

42

4/26/2014

22

Resources• Agile Manifesto - http://agilemanifesto.org/

• Articles/Blogs:

– http://support.smartbear.com/articles/testcomplete/automated-testing-agile-environment/ -SmartBear

– http://lisacrispin.com/wordpress/2011/11/08/using-the-agile-testing-quadrants/ - Lisa Crispin

– http://www.stickyminds.com/sitewide.asp?Function=edetail&ObjectType=COL&ObjectId=17793&tth=DYN&tt=siteemail&iDyn=2 – Rajini Padmanaban

– Beyond Agile Testing: http://www.utest.com/int-v1/a/beyond-agile-testing - uTest

– Testing and Checking Refined: http://www.satisfice.com/blog/archives/856 - James Bach and Michael Bolton

• Books:

– Scaling Software Agility: Best Practices for Large Enterprises – Dean Leffingwell

– How Google Tests Software – James Whittaker, Jason Arbon, Jeff Carollo

– Agile Testing: A Practical Guide for Testers and Agile teams – Lisa Crispin, Janet Gregory

– http://www.dorothygraham.co.uk/automationExperiences/index.html - Dorothy Graham

• Presentations:

– Approaches to Software Testing: An Introduction – Scott Barber

– Scaled Agile Framework Big Picture – Dean Leffingwell43

Backup

44