Evolution of Test Automation

EVOLUTION OFTEST AUTOMATION

A Cygnet Infotech Whitepaper

As technology gets more complex, quality expectations get more

exigent and development iterations get shorter. Software testing within

a limited budget and under time pressure has become a huge challenge.

Testing automation is the strategic investment for businesses to lower

costs while reducing the efforts to maintain the quality and time to

market.

Conventional automation testing is based on “scripting”, which

requires the testing team to have knowledge about native scripting

languages supported by automation tools. However, an extensive

usage and adoption of agile development has imposed many

challenges on the automation team. Issues like unclear project scope,

multiple iterations, sketchy documentation, early and frequent

automation needs and proactive stakeholder involvement has made it

more tedious in creating and maintaining automation scripts in sync

with the application under test. This has resulted into inadequate

testing and/or delayed releases.

Considering the current trends of application development

methodologies and delivery requirements, test automation using the

automation methodologies has raised several questions:

In addition to these questions, a few technical challenges like high

maintenance and technical skillset requirements for maintaining the test

scripts developed, need sensible answers.

Organizations are still struggling with their test automation initiative

either due to budget issues, poor skillset or lack of a suitable

automation strategy, says World Quality Report published by

Capgemini Group (Capgemini & Sogeti) and HP Software. A few of the

key discoveries of this report include:

Almost 13% of the organizations surveyed indicate that they don’t

have dedicated budget for quality assurance team and they are still

struggling with estimating their testing efforts

25% of the organizations surveyed indicate that their dedicated QA

teams run almost 81% to 100% of the tests manually

“Is it really practical and/or possible to use automated testing

in agile methodologies?”

“Is it possible to use traditional tools?”

“Should I opt for open source tools?

“What are the challenges in implementing test automation in

an agile environment?”

INTRODUCTION

A Cygnet Infotech Whitepaper: Code Less Test Automation

According to the “Risk Investment Test Automation?” by Konrad

Schlude in December 2008 edition of “Testing Experience”, almost

63% of the test automations fail to deliver the promises made by

automation. Such reports and failures have led the organizations to

search for more powerful and user-friendly alternatives of automation

testing.

Enter Code less Automation Testing! It is an innovative approach that

helps users automate their tests without even writing code or scripts

using programming languages. It addresses all the major issues of

current automation testing industry while improving the productivity &

reducing the costs. In this whitepaper, we will discuss about the

evolution of test automation, what Code less Test Automation is and

how it addresses the major issues discussed above.

EVOLUTION OF TEST AUTOMATION

The test automation process is a succession of generations, which have

become extensively advanced and sophisticated over a time. In order

to understand the advanced test automation, it is important to

understand the evolution of test automation approaches. The table

below showcases the evolution of test automation approaches:

Record &

Playback

Use/Reuse

of Functions

Data Driven

Scripts

Action

Keyword

Scripts

Code less

Automation

First

Generation

Second

Generation

Third

Generation

Fourth

Generation

Fifth

Generation

Figure 1: Evolution of Test Automation Tools


The first generation automation tools could run unattended for a few

hours each and every day, which was the major benefit. Moreover, it

made automation of regression tests feasible. But unfortunately it was

deceitful. The scripts were inherently unstable: any minor change or

error in the application or the data caused the automation to break.

Even the playback timing was not as reliable as the automation runs

became quite slower with time.

Moreover, it was almost impossible to maintain the scripts as they

were not documented well. For example, the record script included

both the steps and the data embedded within the test, so that if a

hundred accounts were added as a part of the test, it led to the same

step being repeated hundreds of times. After that when changes or

alterations were made to the application, they too had to be repeated

hundreds of times. Look at the example below.

Some of these first generation tools were built as DOS batch files or

UNIX shell scripts.

FIRST GENERATION TEST AUTOMATION

Record and Playback

Figure 2: Record & Playback Code


The second generation of the test automation tools brought a breath

of fresh air by bringing dozens of new features to the test tools. These

brought up similarities between the test tool IDE and developer’s IDE.

These tools offered a fully-grown scripting or even some support for

object oriented languages for the automation professionals. There were

tons of APIs available to simplify the very common window based

tasks like application launch, generating elementary test reports, file

operations and so on.

However, the script based approach still required huge investments in

terms of the amount of time spent in learning and gaining expertise in

the scripting language and the time required to build a suitable

framework that not only embraces the resource code but also simplifies

the data driving scripts.

This made automated testing similar to a software development

project as it demanded a whole new test process to create a new script.

It took a considerable amount of technical skills and support.

Moreover, managing the ever-rising number of automation scripts

became a great challenge to cope up with.

The Selenium IDE is a good example of a second generation tool.

SECOND GENERATION TEST AUTOMATION

Use / Reuse of

Functions

Compared with the earlier approaches, Data Driven frameworks

address the two major pain points: Test Coverage and Maintainability.

In this simple but effective approach, automation scripts were

separated from the input and output data from the software or

application under test.

In this approach, the test data is created separately and the test library

is comprised of the test scripts. Capture replay may or may not be used

as an input to the script-generating process but more important now

are the driver scripts.

THIRD GENERATION TEST AUTOMATION

Separation of Data &

Scripts


Figure 3: Separation of Script and Test Data

Companies have to hire programmers or form automation teams who

were quite comfortable doing the programming for the driver scripts.

This meant externalizing the data into spreadsheets or text files and

adding logic to process the data, tackle timing synchronization, error

handling, recovery and result logging and reporting.

Each and every script had to be programmed and maintained by the

test specialists but could be used frequently with various kinds of data

sets to expand the test coverage. But testing is not all about just

entering data. It’s about simulating real life business scenarios to test

the application under test rigorously.

While this approach succeeded in empowering analysts to input test

data and reduce the length of code, it increased the complexity of the

code that needed to be developed and maintained. In the end,

companies realized that the test automation code is much like any

application software code: it is painful if not impossible for one

programmer to maintain the code written by another. Therefore,

scripted code had to be modified not only when the application

changed, but often rewritten when there was staff turnover.


The fourth generation further breaks down the test automation

approach in an advanced, structured and elegant approach. This

approach has reduced the cost and time of test design, automation and

execution by empowering an entire testing team to focus on what they

do best! Smart frameworks are used in this scenario to overcome the

inherent scripting issues of earlier generation tools by replacing

scripting for the major part. In addition, the maintenance required

because of the revisions in the test is greatly reduced.

Below is one example of a key/action word test, where the Test Case

column contains a list of keywords and the following columns provide

the data:

Figure 4: Keyword / Action Driven Automation

FOURTH GENERATION TEST AUTOMATION

Action Keyword

Scripts

It is not the script now that directs the testing but it is the test itself. The

test data using keywords provides a sequence for the actions that need

to be carried out. When the automated test case runs, it will go

through the test data and call for related scripts specified by the specific

keyword. With this approach, the testers have total control over what

to do and in which manner. But the development of automation code

is still specific to the application under test.

In the example above, “add product” and “check product” are the

names of scripts that contain the detailed level code for adding a

product to the inventory application that is being tested.


Below is the example of how that key or action word might be

expressed:

Key/action word frameworks succeeded in enabling non-programmers

to design and execute automated test cases more easily. But it still

required some amount of custom code. Moreover, as the rate and the

amount at which the UI of the application under test undergoes

change, it will determine the increase in the maintenance of the scripts.

Thus, for large test automation projects, the need for finding ways to

reduce maintenance of the scripts has increased.

Even though the automation feasibility issues get resolved with more

robust object recognition technologies, the issue of poor ROI due to

the productivity problem is still a major headache. Some other

examples in this generation of tools include Selenium Web Driver and

HP QTP.

Figure 5: Script for the Keyword Action

A Revolutionary Leap

Automation professionals have started looking for better and reliable

solutions to address all the issues in the previous generations of

automation approaches. Meanwhile, some automation experts have

noticed that Microsoft Visual Studio and Foundation classes have

brought a huge shift in productivity with which Windows GUI could be

created-instead of manual coding with the Win32 API. Developers can

now use the wizard to “Visually design” the GUI in a “studio”.

That’s where the concept of CODE LESS test automation came into

existence. Instead of conventional code editors, automation experts

now work in a graphical environment where they can easily compose

automated tests by choosing objects and actions from the drop down

menus. They can even visually create conditions, iterations and tons of

other stuff on the go! Let’s see what Code less test automation is all

about!


With the rapidly emerging trends in software quality and testing,

condition of the testing services providers, budgets and their

relationship with the business, many organizations are on the move to

modernize or modify their thought process. Code less test automation

is currently a “work in progress” in majority of the firms as most of

them are transitioning from the record and playback approach to the

Code less approach!

CODE LESS TEST AUTOMATION

automation is an approach to build

a test automation engine by

testing team to and

easily build automated test cases by simply

ordering but code assets to

make sure 100% test coverage is achieved

while increasing the test automation ROI.

Let’s start digging the statement...

Code less means no scripting/programming within the test automation

tool’s native language. But this doesn’t mean that there are no scripts

at all! It is inconceivable to think of automation without any piece of

software. Explicitly, this means that while automating the test cases,

you won’t need to program scripts for each and every test case. Now,

each and every stakeholder in the testing ecosystem can understand

and contribute towards the surety of the application under test.

Well-optimized means optimizing the existing test automation assets

like testers, automation tools, infrastructure and much more to

enhance the productivity. It involves operation optimization, financial

optimization and strategic optimization, which at the end maximizes

the test automation tool throughput and boosts up quality in sync with

the production.

Empowering means to elevate each and every stakeholder within the

testing ecosystem. Code less test automation is a layer that rides on the

top of the test automation tools to hide the complexities associated

with scripting from the stakeholders. It has become very easy for the

QC team to collaborate efficiently in order to build quality in the

product.


Quickly means to automate more number of test cases per unit of time.

With the Code less approach, automating test cases is like writing test

cases in MS Excel. No matter which test automation tool you are using,

with Code less approach, your testing team can easily and effectively

automate almost any kind of test cases up to 5 times faster than before.

Ready means nuggets of scripts already incorporated in the system,

which resides on the top of the test automation tool.

Reusable means it is possible to reuse the test specific components like

object repository and application components across the test

automation tools and technology.

Test automation has always been an attractive proposition as it reduces

dull and repetitive testing efforts and improves regression testing. But

whenever the product does change with each new release, the

automated test suit should also change, which is completely a waste of

time and money spent on creating the previous test automation.

Code less Test automation is going to provide what it promises:

reduced repetitive testing efforts and improved regression testing.

Going Code less will ensure that you leverage benefits from test

automation much faster than the script-based test automation. Here are

some solid benefits of code less automation.

Shorten the time to automate by almost 80%

The faster you automate, the faster your product will go to market.

With Code less test automation approach; your testing team can

automate giant test suites almost 5 times faster than before, which

means time to automate reduces almost by 80%. And this will

safeguard the true value of the product that undergoes alteration.

Visibility into test automation for the domain champions

It is one of the biggest benefits of going code less. Up till now, business

analysts and subject matter experts were unable to participate in the

test automation process as the coding world was like French for them.

But with this approach, they can easily establish the working of the

entire test automation suite or even an individual test case to make

sure the product is tested for its quality – not only for one or two

times, but with each new release or major update.

GO FOR CODE LESS AUTOMATION


Enable the testing team to do what they can do best

Going Code less will save loads of time of your testing team in creating

test automation. As there is no coding required, they can now spend

their time in achieving highest test coverage, which ensures that your

testing team can identify bugs at the very beginning of the product

development and help you deliver a high quality product at the speed

that you have never seen before.

Helps you save money on developing in-house scripting skills

Yes, that is now possible. Now no need to go for those lengthy and

boring training sessions on how to use the test automation tools! Code

less test automation will make sure that creating test automation is

only a matter of a few clicks.

While conventional test automation makes

you work hard, Code less test automation

will work hard for you!

So, if you really wish to revolutionize the test automation experience

within your existing testing team, then going Code less is one of the

best and only ways.

BENEFITS OF CODE LESS AUTOMATION

No more resource specific coding

Code less test automation makes the test automation tool almost

invisible by acting as mediator between the user and automation tool.

The end user can now create automation using the English language.

The Code less framework compiles this into the automation tool

compatible language. It frees the automation tool technology but

collaborates with it seamlessly.

Less time needed to automate

Going Code less empowers users with the ability to translate test cases

into test scripts, and that also in plain English language. This eliminates

the need of any kind of complicated programming and thus, you can

easily automate test cases faster with a significant time saving!

Rigorously tested and examined end product

As Code less framework compiles automated test cases into what the

test tool will understand on run time, there is no requirement to hard

code test cases to achieve the desired success.


Optimized investment

As Code less framework takes the responsibility to create automation,

the testers can now concentrate on increasing throughput – showcasing

the regression testing. The total number of regression cycles that earlier

automation tools were doing can now be achieved by less number of

testers. Even a manual tester can carry out automation, which reduces

the need for experts.

Zero complexity and reduced maintenance

Code less will help you cope up with the changes easily without having

a need to depend on any resources. It will also provide guidance on

critical dependencies of the change throughout the automation life

cycle, which at the end supports your decision at every stage to

improve the quality of your automation.

Highly automated code

“Creating automation that is not specific to the application under test

but UI object class specific” – this is the philosophy on which Code less

approach is built. Therefore, for an application with 500 textboxes in

its UI all actions are performed by a single script that is specific to the

object class of the test box. The more number of times that script is

used, the more it will tested.

More strategic bandwidth in hand

With Code less approach, you can easily:

Create and architect test cases that can easily cope up with the

changes in application, complexity and technology change

Spare more time in interpreting automation execution results

Draw test cases towards the specific areas of change where there are

maximum chances to find bugs

Utilize time to create a test suite to achieve 100% test coverage with

each successive release of the product

Get more time to analyze and identify which is a more suitable and

better automation tool based on their features, reliability and

capability to tackle large test suites

In addition, all the components of a test automation script - the steps

in each keyword, the sequence of keywords, data per step, UI

definition and a lot more other requirements are available in a single

and easy to use interface – and that too without any programming.

Code less test automation approach enables updating the backend

automation engine on a regular basis by the Automation engineer to

support new operations or technologies taking place in the industry.


Some of the key features of a Code less framework, which aid in

automation creation are:

Moreover, such frameworks have a client-server based approach where

automation once created is readily available to a whole team.

Code less framework also provides great features for test execution or

re-execution, some of which are

ROI from Code less framework is also mind-blowing as it is productive

virtually from day one, as you don’t have to spend time at the

beginning to write the framework.


One example of a code less test automation tool is shown below. This

tool, TestingWhiz, captures the web object details during the recording

process and then automatically writes the back-end code. With an

intuitive test editor, even a novice tester can modify the scripts as the

application under test undergoes changes.


From the above discussion, it is clear that Code less automation has the

potential to be the next generation automation technology. It not only

addresses test automation’s productivity issue, but also does it in a way

that never forces you differentiate manual tester and automation

professional. However, it is also necessary to keep in mind that not all

the Code less framework or tools are truly Codeless. They may

provide many features or all the features in a limited manner. It is best

to evaluate the tool to find out whether it suits your test automation

needs.

CLOSING REMARKS

Cygnet Infotech is a global IT services and IT solutions provider with

clients in 24 countries. Software testing and QA is our forte and we

have provided various functional testing, performance testing and

automated testing services to clients in various industries.

Talk to us about QA, Testing and Test Automation

Tel: (India) +91-79-30487400 | (USA) +1-646-915-0021

Email: [email protected]

Web: www.cygnet-infotech.com

All images in this whitepaper have been used for illustrative purposes. Cygnet does not claim rights to any image except the logo.

ABOUT CYGNET INFOTECH

mailto:[email protected]

http://www.cygnet-infotech.com/

Evolution of Test Automation

Software

Transcript of Evolution of Test Automation