Introduction to Back End Automation Testing - Nguyen Vu Hoang, Hoang Phi

19
1 Introduction to Back End Automation Testing 16 | 07 | 2016 Nguyen Vu HOANG Hoang PHI FPT Software

Transcript of Introduction to Back End Automation Testing - Nguyen Vu Hoang, Hoang Phi

Page 1: Introduction to Back End Automation Testing - Nguyen Vu Hoang, Hoang Phi

1

Introduction to Back End

Automation Testing

16 | 07 | 2016

Nguyen Vu HOANG

Hoang PHI

FPT Software

Page 2: Introduction to Back End Automation Testing - Nguyen Vu Hoang, Hoang Phi

WHO we are

2

HOANG Nguyen PHI Hoang

Education and Certification Education and Certification

Master of Computer Sciences

ISTQB (Advanced Level Technical Test Analyst)

Scrum Master, Product Owner and ITIL (FL)

CEH, LPIC-1, Novell CLA, CCNA, MCSE, MCDBA, MCSA, MCSD.NET and MCAD.NET.

Bachelor of Sciences

ISTQB (FL)

IBM Certified (Plan, Estimation, Design and Negotiation)

Page 3: Introduction to Back End Automation Testing - Nguyen Vu Hoang, Hoang Phi

Agenda

3

WHAT is back end testing

WHY Test Automation

Testing Problems and Solutions

WHAT we do

HOW we do

Measurable Results

Demo

Page 4: Introduction to Back End Automation Testing - Nguyen Vu Hoang, Hoang Phi

WHAT is back end testing

4

Service layer (80%) – perform testing for services WITHOUT User Interface (UI), AND

WHAT in the scope

Enterprise Services at API layer.

RESTful Web Service architectural style.

SOAP Web Service used for general enterprise integration.

Connect-based middleware stack (i.e.: JMS – Java Message Service)

WHAT are tested…

No UI or Used by back-office operators Web Service, Database, File System, Web Application

Backend application (20%) –perform testing for application/system with UI which is used to monitor and control services.

Enterprise system with UI used by back-office operators.

Page 5: Introduction to Back End Automation Testing - Nguyen Vu Hoang, Hoang Phi

WHAT is back end testing (Cont.)

5

Web Application(Service Monitor and Control)

Core Business Services

End-User Platforms

External Services or Apps

Client Platforms

Operators

Applications and Services Backend Web

Application can be

tested via user

interface.

Platforms can be

any if it follows

service protocols.

Delivered services

are NOT for

specific platform.

1

1

2

2

Not Available

3

Platforms are not

available when

testing.

3

3rd party Services DBMSXML

Page 6: Introduction to Back End Automation Testing - Nguyen Vu Hoang, Hoang Phi

WHY Test Automation

6

Saves time and money

Increases test coverage

Improves accuracy and quality

Huge data verification achievable only by automation. (i.e. verify thousands of XML attributes)

Better coverage, more executions, higher number of test cycles.

Reliability – ensure correct test at protocol layer and eliminate human error.

Auditability – provide comprehensive results.

Regression testing – gain confidence that nothing is broken.

Reusability – same tests on different environmentsand configurations.

Reduce execution time.

Page 7: Introduction to Back End Automation Testing - Nguyen Vu Hoang, Hoang Phi

Testing Problems and Solutions

7

Testing Problems High Level Solutions

Different technology platforms under

test (Operating System, Database, Web

Server and application protocols and

development).

High data rates and responsiveness

Defragmented testing result so it is

difficult to make decision.

Must align testing with agile

development methodologies.

Integrate automation testing to Continuous

Integration.

Select proper technology stack to centralize

the testing result from both manual and

automation teams

Design automation test framework to deal

with Shell, Web Service, Web Application,

gigabytes of XML Data on Windows and

Linux.

1

2

3

Page 8: Introduction to Back End Automation Testing - Nguyen Vu Hoang, Hoang Phi

WHAT we do - Standardize CI/CD

8

Continuous Integration

Build Automation

Static Code QualityAnalysis

VersionControl

Collaboration Wikis, Forums

Manage Develop Build Test Deploy

Test Automation, Performance Testing

Code Review and Quality Analysis

Unit Testing

Cloud Deployment

Containerize

Improve the way we develop and test applications by standardizing Continuous Integration (CI) and

Continuous Deployment (CD) tool set.

Reduce test and deployment cycles through automation.

Issue TrackingTest / ReqManagement

Artifact Management

1

Page 9: Introduction to Back End Automation Testing - Nguyen Vu Hoang, Hoang Phi

WHAT we do – Do Continuous Deployment

9

By saving time in deployment, testers will be able

to spend more time on feature testing.

Prevent mistakes in installation while there is lots

of configuration defined in release note.

New and old application/services are

deployed side by side to eliminate delay

impact on testing team in case testing is

blocked on the new version.

Testing team starts testing cycle on the new

version after smoke testing passed.

Ant executes test cases

Automation Test Engine

VIP Cut Over

Application1.0

Virtual IP

Application1.1

Dependent ComponentDB 1 DB 2

1

Page 10: Introduction to Back End Automation Testing - Nguyen Vu Hoang, Hoang Phi

WHAT we do – Centralize Test Result

10

Full cycle from test design to test

execution and reporting.

LISA facilitates test result to

SpiraTeam.

Integration with different tools –

extensibility mechanisms to

support Web GUI and bash shell.

Simple maintenance – applied

keyword and data driven

automation, conventions and best

practices.

2

Reporting Database

Test Case ManagementRequirement Management

Page 11: Introduction to Back End Automation Testing - Nguyen Vu Hoang, Hoang Phi

HOW we do – LISA as Core

11

Deliver end-to-end parallel app development solutions to help develop and test simultaneously [1].

Powerful at testing middleware using Service-Oriented Architecture and Database.

Allow high level of customization. Functionality can be extended through its SDK (software development kit) [2].

Enable in-process, bi-directional communication between the Java VM and the CLR (.NET)

Both desktop or server versions provide integrated environment to build all in a code-less manner.

3

Page 12: Introduction to Back End Automation Testing - Nguyen Vu Hoang, Hoang Phi

HOW we do – Test Web Service and Database

12

Use Service Virtualization to

break dependency of testingfrom development schedule [2].

Use consistent approach to

design test cases for web

services.

Use powerful built-in

functionalities of LISA to

automate Web Services and

Database testing.

Extend LISA’s capabilities to

test JSON.

Client Server

Web Server Database Server

Development in progress

Business process analysis

Transaction analysis (loss and duplication)

Performance analysis

Path coverage analysis

Error Handling analysis

Profiling …

Service Mocking, Service Virtualized

Test Case Design

Approach

3

Page 13: Introduction to Back End Automation Testing - Nguyen Vu Hoang, Hoang Phi

HOW we do – Consideration in Scripting

13

Identification

Implementation

Execution

Report

Assessment

3

Page 14: Introduction to Back End Automation Testing - Nguyen Vu Hoang, Hoang Phi

HOW we do – Test Web GUI and Shell

14

Integrate Selenium into LISA so

that it can test Web GUI.

A wrapper layer is built on top of

core Selenium in order to provide

strong GUI testing capability for

LISA.

Integrate JCraft into LISA so that

it can test Shell script.

Use Awk/Nawk/Gawk [3]language at remote host instead

bash script to have fast text

processing with low memory

usage. Performance is 10+ times

faster than bash script.

JCraft

Extended Selenium (Java base)

Core Selenium

LISA

SSH

3

Page 15: Introduction to Back End Automation Testing - Nguyen Vu Hoang, Hoang Phi

HOW we do – Test Big XML files

15

LISA

Xstream JAXB

Process large files (2GB to 3GB for each).

Traverse nodes to smaller chunk.

Cost less memory resource but takes time to read XML file.

Xstream

Process delta (small) data.

Binding Java-to-XML makes it easy to incorporate XML data and processing Java functions quickly.

Cost more memory resource but it’s fast.

JAXB

Combine Xstream and JAXB

3

~3 min: XML (~3GB) and XSD Schema.

~6 min: 2 XML files (~3GB per each)

Results

Page 16: Introduction to Back End Automation Testing - Nguyen Vu Hoang, Hoang Phi

Measurable Results

16

Unified automation test approach for 60+ projects.

Automated tests that cannot be done by manual.

100 000+ executed tests

12 000+ automated tests

92% saving execution time

Application1(547TCs)

Manual4 MDs

Automated50min

97%

Application2(590TCs)

Manual23.5MDs

Automated90min

99%

Application3(211TCs)

Manual5.1MDs

Automated1 MD

80%

Application4(2000TCs)

Manual1755hrs

Automated155hrs

91%

Application5(221TCs)

Manual117hrs

Automated7 hrs

94%

Application6(30TCs)

Manual30hrs

Automated13min

99%

Tool Cost ($) Similar Tool

Spira 3-Users ($79.99 / month) Testlink (free)

LISA 5-Users (~$1,666 / month) SoapUI (free/license)

Tool price and similar tool

Page 17: Introduction to Back End Automation Testing - Nguyen Vu Hoang, Hoang Phi

Demo

17

1) Trigger test from Jenkins

2) Call LISA script

3) Call Shell script

4) Call Selenium script

5) Update result to SpiraTeam

(Duration: 1.5 minutes)

Page 18: Introduction to Back End Automation Testing - Nguyen Vu Hoang, Hoang Phi

References

18

[1]. Develop & Test (former named CA LISA), http://www.ca.com/us/products/develop-test.html

[2]. Documentation of DevTest Solutions - 8.0, https://docops.ca.com/devtest-solutions/8-0/en

[3]. Awk/Nawk/Gawk tutorial, http://www.grymoire.com/Unix/Awk.html

Page 19: Introduction to Back End Automation Testing - Nguyen Vu Hoang, Hoang Phi

© 2016 HCMC Software Testing Club

THANK YOU