ZERO cost Performance Test Automation Design and Implementation– CI/CD

Abstract

There is increasing concern among agile framework worlds in identifying the best and cost effective approach for performance testing which is independent of operating system. This paper demonstrates an approach and implementation of a zero cost performance test automation framework that can be implemented in any operating system. This framework can be used for performance testing for web applications using the protocols such as Web-HTTP, HTTPS, SOAP, REST, FTP, Database via JDBC, LDAP, Message-oriented middleware

PardhaSaradhi DSenior Technology Architect

Infosys Limited

(MOM) via JMS, Mail - SMTP(S), POP3(S) and IMAP(S), Native commands or shell scripts, TCP.

1. Introduction

This document will define the Design and implementation framework for Zero cost Performance test automation for web application using various protocols such as Web-HTTP, HTTPS, SOAP, REST, FTP, Database via JDBC, LDAP, Message-oriented middleware (MOM) via JMS, Mail - SMTP(S), POP3(S) and IMAP(S), Native commands or shell scripts, TCP. This framework can be largely used for Continuous Integration and Continuous Delivery streams in the Agile and Devops world. This firework was developed on open source tools such as Java, apache ant, Jmeter and not limited to Jmeter. It has the capability to extend the framework for various licensed load simulation tools such as BlazeMeter, NeoLoad, HP load runner, IBM RPT etc.

The scope of application / system level performance testing is to verify that the application continues to function and meets its performance targets according to defined non-functional requirements. The framework shall be independent of other integrated systems. The Interfaces with these systems must be accurately stubbed to allow a full simulation of end-to-end transaction flow.

The prime objective automated performance test execution the process to run automated tests and checks whether the latest software/code change influences any application capacity thresholds or SLAs.

Business objectives:

- Run performance tests automatically on iteration basis.- Establish a trend for performance at early stages.- Developers to find code and architecture issues at early stages.- Performance teams to focus E2E performance and infrastructure testing.- Minimize performance defects escape.- Reduce TCO and Increase ROI to client / customer.

2. Automated Performance Testing Framework lifecycle

An Automated performance test lifecycle should follow the generic process mentioned below.

Lifecycle Stage 1: Restore or Reset Database:- The framework should have the capability to reset or restore the application database as

a precursor to the test execution.- To ensure the repeatability of tests on same size of database.

Lifecycle Stage 2: Deploy Code: - The framework should have the capability to fetch the application latest code, Scripts,

Configuration files from version controller repository.- The framework should have the necessary scripts to deploy in the application code in

environment.

Lifecycle Stage 3: Create Test Data:- It is essential that framework to have capability to generate the test data required for the

performance test.

- Example: Generating test data in the database using the application through test data scripts or database scripts.

Lifecycle Stage 4: Reset Environment:Environment Reset has to be performed by framework scripts. Below are the list of activities that has to be performed:

- Application server restarts.- Truncate\Delete Application logs.- Check Disk space availability.- Delete Temp folders.- Verify Application is up and running.

Lifecycle Stage 5: Smoke Test- Framework to have feature to perform the application, DB and network smoke test

before executing the actual test to ensure application APIs and interfaces were initialized.

- In case of smoke test failure the test will be aborted.- Notification will be sent out in case smoke fails.

Lifecycle Stage 6: Execute Test- Framework to have the capability to dynamically select the Work Load modal

predefined for a particular type of performance test (Peak volume, Soak/Endurance or Stress test).

- The concurrency, loops and ramp up will be dynamically passed based the type of test selection.

Lifecycle Stage 7: Collate results:- Once the test is complete, the framework should collate the results and publish the

results to a Dashboard or repository.

Lifecycle Stage 8: Analyse results:- The Framework should have the capability to analyze the results at high level and send

email to the required stake holders.

3. Automated Performance Testing Scope - The scope of application / system level performance testing is to verify that the application

continues to function and meets its performance targets according to defined non-functional requirements independent of other integrated systems.

- The scope includes all application, database and network components up to the point of integration with other systems. Interfaces with these systems must be accurately stubbed to allow a full simulation of end-to-end transaction flow.

- Automated application performance testing needs to be considered within the wider performance test strategy.

- An automated application performance test suite as part of a Continuous Delivery process will typically be configured to simulate load test scenario.

- Automated performance test suites must be reusable, configurable and can be executed on any environment such as performance test environment, Production support servers, etc.

- Follows the 8-step Performance Testing framework. Each stage is clear and distinct in its implementation.

Best practice criteria for implementation:- Modularity - Decoupled/independent of stubs & data tools. i.e. changes to external interface

stub implementation should not require significant SVT tool changes.- Maintainability – Script comments; consistent implementation patterns; clear & concise

documentation.- Portability - Parameterisation of scripts to facilitate environment independence.- Usability - Industry standard tooling & paradigms; Implementation simplicity.- Repeatability – Consistent and manageable volume generation to ensure test repeatability.- Scalability – Ensure scripts can be easily configured to scale to various volumes (future

volumes, or lower environment capacity).

4. Performance Test automation Implementation and Overview

Tools employed for implementing automation.

Software\Tool Version Description

Apache Jmeter 3.1 Apache JMeter – An open source load testing tool considered for Performance testing and measuring the performance of a variety of services

Apache Ant 3.1 An Open source tool - To create build scripts- Compile java classes- Executing SSH commands- Passing Jmeter runtime parameters such as threads, virtual

user ramp-up, throughput, etc. - Passing transaction Work load for performance testing,

etc.

DB scripts SQL Restoring database to an earlier flashback point.

Restore Database- Custom DB SQL scripts written to restore the application’s database to a baseline date

with production-like copy. To ensure repeatability of test on the same size of database.

Deploy Code: - Custom scripts written to fetch latest code base from version controller -GIT repository

and deploy in the application environment.- Custom scripts written to fetch latest Jmeter performance scripts from version controller

-GIT repository. - Custom scripts written to fetch latest Ant build scripts

Create Test Data:

- This step is essential for applications, which needs and consumes data for every business transaction and the input data cannot be reused. It is advised to create the required test data in the persistent storage area (such as Database) through application.

- Create performance test scripts using Jmeter for Test data creation which will be consumed in actual performance test execution.

- Properties files to be created to pass the work load as Jmeter variables such as throughput, Thread count, rampup configuration, etc.

- Naming convention has to be followed for loopCount, pacing, Rampup, Thread and Throughput configuration.For Example, the content of properties files is as below: PROJECTXXX_CreateSI_appointment.Rampup=1PROJECTXXX_CreateSI_appointment.Loop=10PROJECTXXX_CreateSI_appointment.Pace=2

PROJECTXXX_CreateSI_appointment.Threads=4PROJECTXXX_CreateSI_appointment.ThroughPut=3

- Work load attributes from property files, can be passed to Jmeter scripts through ant build file in the following way.For Example as shown for a sample ant script:

<target name="run"><property file="${conf.dir}/ThreadConfiguration.properties"/><property file="${conf.dir}/LoopCount.properties"/><property file="${conf.dir}/PacingConfiguration.properties"/><property file="${conf.dir}/RampupConfiguration.properties"/><property file="${conf.dir}/ThrouputConfiguration.properties"/>

<jmeter jmeterhome="${jmeter.home}" testplan ="${scriptpath}/${Test_Data_Generation}.jmx" resultlog="${finalresult}/${Test_Data_Generation}.jtl">

<property name="jmeter.save.saveservice.assertion_results" value="all"/><property name="jmeter.save.saveservice.output_format" value="xml"/><property name="PROJECTXXX_CreateSI_appointment.Threads" Value="${PROJECTXXX_CreateSI_appointment.Threads}" /><property name="PROJECTXXX_CreateSI_appointment.Loop" Value="${PROJECTXXX_CreateSI_appointment.Loop}" /><property name="PROJECTXXX_CreateSI_appointment.Pace" Value="${PROJECTXXX_CreateSI_appointment.Pace}" /><property name="PROJECTXXX_CreateSI_appointment.Rampup" Value="${PROJECTXXX_CreateSI_appointment.Rampup}" /><property name="PROJECTXXX_CreateSI_appointment.ThroughPut" Value="${PROJECTXXX_CreateSI_appointment.ThroughPut}" />

</jmeter></target>

- Work load attributes from property files through Apache ant build xml, can be accessed from Jmeter scripts as shown in the below figure.

- The Test Data generation test suite can be executed through an Ant command.

Example: ant -buildfile=../../../src/xml/TestData_Generation.xml -DTest_Data_Generation=XXX_JMX_File

- The Sample apache ant snippet will execute the Test and the output of the test execution can be generated in html format as shown below.

Apache Ant Snippets:<jmeter jmeterhome="${jmeter.home}" testplan ="${scriptpath}/${Test_Data_Generation}.jmx" resultlog="${finalresult}/${Test_Data_Generation}.jtl">

<property name="jmeter.save.saveservice.assertion_results" value="all"/>

<property name="jmeter.save.saveservice.output_format" value="xml"/> </Jmeter><target name="report" depends="run">

<echo message="Results Jmeter report generation - Starts" /> <java jar="${jmeter.home}/bin/ApacheJMeter.jar" fork="true">

<arg value="-g" /> <arg value="${resultspath}/${TestData_Generation}.jtl" /> <arg value="-o" /> <arg value="${finalresult}" />

</java><echo message="Results Jmeter report generation - Ends" />

</target>HTML report:

Reset Environment:

- Apache Ant script can be created to restart, Delete application logs/temp files etc., the application servers remotely

Smoke and Performance test Executions

- Create smoke test scripts using Jmeter to execute Smoke Test to ensure all the interfaces and API are up and running.

- Create Performance test suite scripts using Jmeter to execute Peak/Stress/Endurance Tests to measure the performance of the application.

- Create new copy of Ant build file and set of properties files for Peak/Soak/Endurance workloads for performance scripts.

- The implementation process for smoke and performance test execution is same as test data step, however the number of scripts and workload may vary based on the scope of test coverage.

Collate and analyse the results

- Post-performance test execution, the transactions which are based on synchronous response can be collated and analysed through Jmeter.

- The performance metrics of Asynchronous transaction can be extracted through Jmeter rest interface to E2E monitoring tools such as Dynatrace, AppDynamics,Splunk, etc.

The Sample Ant build script is mentioned below for implementation:

How to Run:

ant -buildfile=../../../src/xml/TestData_Generation.xml -DTest_Data_Generation=XXX_JMX_File -DSplunk_Result=YYY_Splunk_Result

Sample Ant file:

<project name="TestData_Generation" default="main">

<property name="test.dir" value="../../"/><property name="jmeter.home" value="../../Installable/apache-jmeter-3.1"/><property name="resultspath" value="${test.dir}/JTLFile"/><property name="finalresult" value="${test.dir}/Result"/><property name="scriptpath" value="${test.dir}/script/"/><property name="conf.dir" value="${test.dir}/src/conf"/><property name="Dynamic_conf.dir" value="${test.dir}/conf"/>

<property name="xmlpath" value="${test.dir}/src/xml"/><delete dir="${resultspath}"/><delete dir="${finalresult}"/><mkdir dir="${resultspath}"/><mkdir dir="${finalresult}"/><mkdir dir="${Dynamic_conf.dir}"/><!-- Defining $Values here : Starts--><property name="DFetch_Prerequisites_DB" value="${DFetch_Prerequisites_DB}"/><property name="Test_Data_Generation" value="${Test_Data_Generation}"/><property name="Splunk_Result" value="${Splunk_Result}"/><!-- Defining $Values here : Ends--><tstamp>

<format pattern="yyyy-MM-dd'T'HH:mm:ss.SSSXXX" property="earliest_time_T1"/>

</tstamp><echo message="earliest_time_T1:${earliest_time_T1}"/>

<target name="main" depends="FetchDBData,report,result"/><target name="FetchDBData">

<echo message="Genrate csv files by fetching data from database : Starts"/><jmeter

jmeterhome="${jmeter.home}"testplan ="${scriptpath}/${DFetch_Prerequisites_DB}.jmx"resultlog="${finalresult}/${DFetch_Prerequisites_DB}.jtl"><property name="jmeter.save.saveservice.assertion_results"

value="all"/><property name="jmeter.save.saveservice.output_format"

value="xml"/></jmeter><echo message="Genrate csv files by fetching data from database : Ends"/>

</target><target name="report" depends="run">

<echo message="Results Jmeter report generation - Starts" /> <java jar="${jmeter.home}/bin/ApacheJMeter.jar" fork="true">

<arg value="-g" /> <arg value="${resultspath}/${TestData_Generation}.jtl" /> <arg value="-o" /> <arg value="${finalresult}" />

</java><echo message="Results Jmeter report generation - Ends" />

</target><target name="run">

<echo message="Initialize and Run Jmeter for Test Data Creation : Starts"/><property file="${conf.dir}/ThreadConfiguration.properties"/><property file="${conf.dir}/LoopCount.properties"/><property file="${conf.dir}/PacingConfiguration.properties"/><property file="${conf.dir}/RampupConfiguration.properties"/>

<property file="${conf.dir}/ThrouputConfiguration.properties"/><jmeter

jmeterhome="${jmeter.home}"

testplan ="${scriptpath}/${Test_Data_Generation}.jmx" resultlog="${finalresult}/${Test_Data_Generation}.jtl">

<property name="jmeter.save.saveservice.assertion_results" value="all"/>

<property name="jmeter.save.saveservice.output_format" value="xml"/>

<property name="PROJECTXXX_CreateSI_appointment.Threads" Value="${PROJECTXXX_CreateSI_appointment.Threads}" /><property name="PROJECTXXX_CreateSI_appointment.Loop" Value="${PROJECTXXX_CreateSI_appointment.Loop}" /><property name="PROJECTXXX_CreateSI_appointment.Pace" Value="${PROJECTXXX_CreateSI_appointment.Pace}" /><property name="PROJECTXXX_CreateSI_appointment.Rampup" Value="${PROJECTXXX_CreateSI_appointment.Rampup}" /><property name="PROJECTXXX_CreateSI_appointment.ThroughPut" Value="${PROJECTXXX_CreateSI_appointment.ThroughPut}" />

</jmeter><echo message="Initialize and Run Jmeter for Test Data Creation : Ends"/>

</target><target name="result">

<tstamp><format pattern="yyyy-MM-dd'T'HH:mm:ss.SSSXXX"

property="earliest_time_T2"/></tstamp><echo message="earliest_time_T1:${earliest_time_T2}"/><echo message="Splunk results for Test Data Creation : Starts"/><jmeter

jmeterhome="${jmeter.home}"testplan ="${scriptpath}/${Splunk_Result}.jmx"resultlog="${finalresult}/${Splunk_Result}.jtl"><property name="jmeter.save.saveservice.assertion_results"

value="all"/><property name="jmeter.save.saveservice.output_format"

value="xml"/><property name="earliest_time_T1" Value="${earliest_time_T1}" /><property name="earliest_time_T2" Value="${earliest_time_T2}" />

</jmeter><echo message="Splunk results for Test Data Creation : Ends"/>

</target><!-- ant-jmeter.jar comes with jmeter, be sure this is the release you have --> <path id="ant.jmeter.classpath"> <pathelement location="${jmeter.home}/extras/ant-jmeter-1.1.1.jar" /> <fileset dir="${jmeter.home}"> <include name="ant-jmeter*.jar"/> </fileset> </path> <taskdef name="jmeter"

classpathref="ant.jmeter.classpath" classname="org.programmerplanet.ant.taskdefs.jmeter.JMeterTask" /></project>

References & Appendix

1. http://sub-second.blogspot.com.au/2012/03/how-to-integrate-jmeter-into-ant-build.html 2. http://testingfreak.com/step-by-step-process-to-run-jmx-file-through-command-prompt-

using-ant-and-generate-html-report/ 3. https://www.blazemeter.com/blog/5-ways-launch-jmeter-test-without-using-jmeter-gui 4. http://ant.apache.org/

Author Biography

PardhaSaradhi did masters and a Bachelor of Engineering in Electrical and Electronics from India. He has started perusing his career in IT industry from 2004 and been working with top IT clients all over the globe from last thirteen years in performance engineering & testing domain. Pardha has been analysing performance issues raised by various banking and telecom clients. He has been providing technical solutions for Java based applications with performance improvements. Combining his expertise in Performance engineering and interest in exploring new methodologies to improve the productivity of his team, Pardha has created many frameworks to analyse and identify application bottlenecks quickly.

THANK YOU!