Automated Acceptance testing by Developers & Automated...

19
Gowrishankar Sundararajan QA Manager Tata Consultancy Services, Canada Automated Acceptance testing by Developers & Automated Functional Testing by Testers

Transcript of Automated Acceptance testing by Developers & Automated...

Page 1: Automated Acceptance testing by Developers & Automated ...nordictestingdays.eu/files/files/gowrishankar-automated-acceptance.pdf · Automated Acceptance testing by Developers & Automated

Gowrishankar Sundararajan

QA Manager

Tata Consultancy Services, Canada

Automated Acceptance testing by Developers &

Automated Functional Testing by Testers

Page 2: Automated Acceptance testing by Developers & Automated ...nordictestingdays.eu/files/files/gowrishankar-automated-acceptance.pdf · Automated Acceptance testing by Developers & Automated

Overview on Traditional Agile Testing Approach

Comparison of Traditional Vs New Proposed Approach

Recommended New Agile Approach with Benefits

Executive Summary

2Gowrishankar Sundararajan Nordic Testing Days 2016

Case Study – An implementation overview with KPI’s

Page 3: Automated Acceptance testing by Developers & Automated ...nordictestingdays.eu/files/files/gowrishankar-automated-acceptance.pdf · Automated Acceptance testing by Developers & Automated

Introduction to Agile

Introduction

Frequent delivery of deployable business value

Frequent changes in requirements

Close collaboration between entities

Prioritization of work is done by the

stakeholders on the basis of requirements and

focus is then paid on the highest risk issues as

the work progresses

Customers/business owners make business

decisions, developers make technical decisions

Agile Flow

3Gowrishankar Sundararajan Nordic Testing Days 2016

Page 4: Automated Acceptance testing by Developers & Automated ...nordictestingdays.eu/files/files/gowrishankar-automated-acceptance.pdf · Automated Acceptance testing by Developers & Automated

Traditional Agile – An Overview

Test Design

Day 1 Day 2 - 8 Day 9- 13 Day 14-15 Day 1

Coding Defect Fix ClosurePlanning

Test Execution ClosurePlanning

QA Plan day wise

Day 1 – Previous Sprint Handover to UAT/Business and

Scope finalization for new Sprint

Day 2-8 – Test Design phase in parallel with Coding

Day 9-13 – Test Execution – Functional Testing

Day 14-15 – Test Execution – Regression Testing for the

current sprint

In a normal 3 weeks Sprint window, below is the QA plan/duration

for each testing Phase80%

Design

Execution

Sprint Duration

Pla

nn

ing

Week1 Week2 Week3

Test

Co

mp

leti

on

%

100%

Closure

4Gowrishankar Sundararajan Nordic Testing Days 2016

Page 5: Automated Acceptance testing by Developers & Automated ...nordictestingdays.eu/files/files/gowrishankar-automated-acceptance.pdf · Automated Acceptance testing by Developers & Automated

Drawbacks/Key Issues faced in Traditional Approach

Comparison

Issues

Satisfaction

• On day 9, when testing team starts

with Smoke testing there is always

greater chance to identify

‘Blocker’ defects.

• Due to blocker defects, a day or

two will be lost which makes

shorter duration for testing team

to complete test execution in 5

days.

• Sometimes, even regression for

the current sprint release

cannot be completed.

• In Many Cases, regression of

prior releases are ignored which

will have higher impact in UAT.

• Finally, test suite preparation of

regression & End-End testing

involves considerable effort,

cost.

• Due to time constraint, few items needs to be removed

from the current sprint.

• Ultimately, Product team is unhappy with the

items not getting delivered which was promised.

5Gowrishankar Sundararajan Nordic Testing Days 2016

Page 6: Automated Acceptance testing by Developers & Automated ...nordictestingdays.eu/files/files/gowrishankar-automated-acceptance.pdf · Automated Acceptance testing by Developers & Automated

The Solution – New Best in Class Approach

QA Plan day wise

Day 2 – 4 : Testers develop the core automated script and test data for Critical path to satisfy Smoke/Sanity Criteria and

hands it to Dev team to run the script before sending the new build to QA

Day 5 – 8 : Testers add scripts to cover all possible test scenarios

Day 9 – 11 : Functional Test Execution using Automation Scripts

Day 12 – 13: In-sprint Defect Resting & Regression Test Execution

Day 14 – 15: Regression for Prior sprints

6

Day 1 Day 2 - 4Day

9 - 11 Day 14-15 Day1

Coding Defect Fix ClosurePlanning

Test Design Test Execution ClosurePlanning

Day 5 - 8

Core ScriptAll possible

Combination

Day

12 - 13

Automated Script Exec

In-sprint Regression

Prior sprint Regression

Sanity Test

Gowrishankar Sundararajan Nordic Testing Days 2016

Page 7: Automated Acceptance testing by Developers & Automated ...nordictestingdays.eu/files/files/gowrishankar-automated-acceptance.pdf · Automated Acceptance testing by Developers & Automated

New Best in Class Approach vs. Traditional Approach – Comparison Chart

Week1 Week2 Week3

Test

Co

mp

leti

on

%

80%

Pla

nn

ing Sprint

DesignSprint

ExecutionClosure

Sprint Duration

Week1 Week2 Week3

Test

Co

mp

leti

on

%

Pla

nn

ing Sprint

DesignSprint

Execution

Closure

Core Regression

Traditional Approach New Approach

Acceptance test

7Gowrishankar Sundararajan Nordic Testing Days 2016

Sprint Duration

Page 8: Automated Acceptance testing by Developers & Automated ...nordictestingdays.eu/files/files/gowrishankar-automated-acceptance.pdf · Automated Acceptance testing by Developers & Automated

8

The Solution – New Best in Class Approach

• Early Automation – Start automation early in SDLC from smoke/Sanity testing rather

than waiting for functional testing

• Testers will no longer perform Smoke/Sanity testing

• Testing team develops automated smoke/sanity test Scripts

• Developers executes automated smoke test on new build before handover to QA

Automation

Smoke/

Sanity Test

• Functional testing is no longer manual

• Maximized Automation

• Scripts reuse for Regression/End-End testing

Functional

Required Change in Approach

Gowrishankar Sundararajan Nordic Testing Days 2016

Page 9: Automated Acceptance testing by Developers & Automated ...nordictestingdays.eu/files/files/gowrishankar-automated-acceptance.pdf · Automated Acceptance testing by Developers & Automated

Benefits with New Approach

9

Defect Leakage

Cost Saving

Customer Satisfaction

Coverage

• Cost saving due to

automated smoke,

functional testing

• Automation ROI for

regression and end to

end testing is much

faster

• Zero Showstopper

Leakage from Dev

to SIT to UAT

• Reduced Major

defects leakage

from SIT to UAT

• Increased coverage

by ensuing room for

in-sprint regression

• Ensure coverage for

core regression

• Deliver what is

promised for each

print

• More confidence on

delivered product

• Improved overall

customer satisfaction

Key

Benefits

Gowrishankar Sundararajan Nordic Testing Days 2016

Page 10: Automated Acceptance testing by Developers & Automated ...nordictestingdays.eu/files/files/gowrishankar-automated-acceptance.pdf · Automated Acceptance testing by Developers & Automated

Case Study – Testing Support for a Leading Canadian Money Transfer Payment Network Provider

Client Profile and Background

Canada's largest money transfer Payment network and Gateway provider.

The client does the services similar to VISA, PAYPAL etc. and carries out 98% of money

transfer across people belonging to same/different FI’s with in Canada.

Key Features

Sending Money across FI’s/Credit Unions

Requesting Money across FI’s/Credit Unions

Project Information

No of resources

7 (1 On : 6 Off)

Product Usage

Money Transfer across FI’s

Technologies

Java, Xml, Batches, Web

Automation Tool

SOAP SONAR

Duration

Jan 2013 – Dec 2015

Tracking / Listing the transactions

Consolidated Reports

10Gowrishankar Sundararajan Nordic Testing Days 2016

Page 11: Automated Acceptance testing by Developers & Automated ...nordictestingdays.eu/files/files/gowrishankar-automated-acceptance.pdf · Automated Acceptance testing by Developers & Automated

Case Study – Testing Support for a Leading Canadian Money Transfer Payment Network Provider

11Gowrishankar Sundararajan Nordic Testing Days 2016

Scope of Work

• Current feature is built on Java Technology (Struts/JDBC) -> System heavy and affecting its performance.

• Rewrite of existing application + new features on Spring/Hibernate feature.

• No change in the XML Structure that is being used across FI’s for Sending Money.

Regression Testing of Old Features Functional Testing of New Features Integration Testing of Both Features

Project Intro

• Web banking customers of participating FI -> electronic payments to anyone else -> web banking

customer of a participating FI.

• Current functionality one-time, immediate money transfer.

• Future Release includes Requesting Money, Auto-deposit etc.

• The system uses an application service provider model (XML’s) with an interface to the FI’s banking site

Page 12: Automated Acceptance testing by Developers & Automated ...nordictestingdays.eu/files/files/gowrishankar-automated-acceptance.pdf · Automated Acceptance testing by Developers & Automated

Case Study – Testing Support for a Leading Canadian Money Transfer Payment Network Provider

Types of Testing

Smoke/Acceptance Testing

System/Functional Testing

System Integration Testing

(End to End)

Regression Testing

Test Automation

User Acceptance Testing

Beta Testing

Performance Testing

12Gowrishankar Sundararajan Nordic Testing Days 2016

Customer

Banks /

Credit

Union

Gateway

Banks /

Credit

Union

Customer

Send Money Receive Money

Send

Request

Receive Request

Receive

Money

Fulfill Request

Customer

Info update

Flow Diagram

Page 13: Automated Acceptance testing by Developers & Automated ...nordictestingdays.eu/files/files/gowrishankar-automated-acceptance.pdf · Automated Acceptance testing by Developers & Automated

Work Load Split-Up

13

MODULES USAGE SPLIT % FUNCTIONAL TESTING EFFORT SPLIT %

Though our application uses web services only by 50%, actual functional testing effort to test other modules

in turn requires to execute web services again. Overall Functional testing effort alone was more than 70%

with web services module

Gowrishankar Sundararajan Nordic Testing Days 2016

Page 14: Automated Acceptance testing by Developers & Automated ...nordictestingdays.eu/files/files/gowrishankar-automated-acceptance.pdf · Automated Acceptance testing by Developers & Automated

Case Study – Key KPI’s to Prove

ProductivityDEFECT LEAKAGE TO UAT Productivity ProductivityDEFECT LEAKAGE BY SPRINT

On an average, we had 4 – 5 ‘Blocker’ and 24

‘Major’ Defects leakage to UAT from SIT before

implementing new approach. Later, we only had

5 ‘Major’ and Max of 1 ‘Blocker’ defect.

From the above graph, it is clear that there is drastic

reduction in the number of both ‘Blocker’ and ‘Major’

defects from Sprint 9 (after implementing new approach).

**First 3 Sprints count is ignored to understand application and taking

into consideration as ‘Learning Curve’

14Gowrishankar Sundararajan Nordic Testing Days 2016

Page 15: Automated Acceptance testing by Developers & Automated ...nordictestingdays.eu/files/files/gowrishankar-automated-acceptance.pdf · Automated Acceptance testing by Developers & Automated

Case Study – Key KPI’s to Prove

ProductivityProductivityREGRESSION DEFECTS LEAKAGE TO UAT ProductivityREGRESSION DEFECTS IDENTIFIED IN SIT

From above graph, it is clear that there is Zero

defect leakage (both Blocker & Major) to UAT from

Sprint 11. The reason we had it in Sprint 9 & 10 is

that regression coverage was done gradually.

From the above graph, it is clear that

a) Regression Defects are caught in SIT phase.

b) Considerable reduction in number of defects

leaked from Dev to SIT.

15Gowrishankar Sundararajan Nordic Testing Days 2016

Page 16: Automated Acceptance testing by Developers & Automated ...nordictestingdays.eu/files/files/gowrishankar-automated-acceptance.pdf · Automated Acceptance testing by Developers & Automated

Case Study – Key KPI’s to Prove

ProductivityProductivityProductivityTEST EXECUTION COVERAGE GAPS IN EXISTING SYSTEMProductivityQA DAYS LOST

QA days lost due to

showstopper issuesAfter implementing new approach,

all the possible negative

flows/paths is covered. This helped

us in identifying few gaps in

existing system, which helped

business to fix this issues before

rolling out new features. This will in

turn save number of tickets opened

to support team and their effort in

future.

Approach

Type

Days

Lost

Traditional 4 PD’s

New 0 PD’s

Exit criteria of any testing team is to ensure 100% coverage is

covered. But due to ‘Blocker’ defects, we always try to cover High

and Medium Priority test cases. Hence, even some Low priority test

cases will result in ‘Major’ UAT defect.

16Gowrishankar Sundararajan Nordic Testing Days 2016

Page 17: Automated Acceptance testing by Developers & Automated ...nordictestingdays.eu/files/files/gowrishankar-automated-acceptance.pdf · Automated Acceptance testing by Developers & Automated

How easy to get this approach implemented in other Testing Model

No Hassle

Most of the interactions across

applications happens thru web

services, interfaces/ESB. And in

recent days, after introduction of

Component Integration

testing this approach is easier

to implement and get the

benefits out of it. Though we

implemented specifically for

XML’s, same approach can be

used for web and other

technologies.

17Gowrishankar Sundararajan Nordic Testing Days 2016

Page 18: Automated Acceptance testing by Developers & Automated ...nordictestingdays.eu/files/files/gowrishankar-automated-acceptance.pdf · Automated Acceptance testing by Developers & Automated

Questions?

18

Page 19: Automated Acceptance testing by Developers & Automated ...nordictestingdays.eu/files/files/gowrishankar-automated-acceptance.pdf · Automated Acceptance testing by Developers & Automated

Thank You !

Contact @ [email protected]