The Magazine for Professional Testers

June 2011

ISSN

1866

-570

5

w

ww

.test

inge

xper

ienc

e.co

m

fr

ee d

igita

l ver

sion

prin

t ver

sion

8,0

0 €

prin

ted

in G

erm

any

14

© iStockphoto.com/ jgroup

Improving the Test Process

112 The Magazine for Professional Testers www.testingexperience.com

Automation responsibilityby Zbyszek Moćkun

© iStockphoto.com/carlosphotos

We are in the project meeting and discuss how to speed up the time of software releasing. Suddenly someone from the end of the room shouted: “Let’s automate our regression tests”. The idea looks very interesting, but during further discussion about auto-mation we found several questions that need to be answered. Be-fore we start to work on automation we definitely have to answer one of these questions: Who should be responsible for automa-tion and why?

The first step is the same as for any project - at the beginning one should define a goal. We need to know what we want to achieve. Do we want to automate only smoke tests (few tests that verify if everything is ok), check all important features (about 20% of cov-erage), or do we need to automate almost all of the functionality of our application (about 80%). Please note that we can automate even 100%, but the costs of automation for the last 20% will be very high and in most of cases it’s not worth to do this (manual tests are cheaper). Below I will concentrate mostly on the most popular and the best approachinvolving 20 to 80 percent of au-tomated tests.

We must not forget about two resources that are always limited: time and budget. If we sum up those (refer to the goal from the paragraph above) we realize something very important – auto-mation is a project. It’s unique, and we move within a triangle (scope, budget and time).

Another question is - What kind of project is it? Is it external or internal in accordance with the application (main project) that we want to test? Before I answer this question, let’s go back to au-tomation. The task of automating can be divided into two phases: phase 1 - create new test scripts, and phase 2- maintenance (run and keep up to date). The question is - should we treat both of them in the same way?

Of course not - Writing new test scripts can be outsourced to an external project. The author (person who writes automated test scripts) does not need to be involved in the application project since he rewrites regression tests. If manual test scripts are well made, the author does not even need to know anything about the application. He knows the steps and the expected results, which should be enough to created automated test scripts.

On the other of the hand, we have maintenance, which contains activities like: run tests, report results, debug failed tests and up-date automated scriptsif there were changes in the application. There is no way to do this outside the project, so maintenance can be considered as a type of sub-project (internal project) which should be conducted within the budget and resources of the pro-ject. It will be very difficult to make quick changes to scripts (con-nected with new features/change requests to the application) as an external project. So the changes should be done as soon as they were identified by one of the project team members.

Let’s summarize: Writing automated scripts is an external pro-ject, and maintenance is internal.

It’s time to go back to the second part of the question - who should write the automated tests scripts, and who should be responsible for the maintenance? Should it be done by developers or testers? Does anyone know developers that like testing? I don’t think. It’s only the one reason but really important (if people do not do what they like, they just change a work or their performance slowly decrease). Another reason against developers is that they would have to learn about the test case management tools, where we keep test cases, learn about the environment and about the tool that we will be using for test automation. Learning new tools is a time consuming process and we have to be sure that our invest returned in long term. Developers should rather concentrate on unit tests, which is their level. So, why should automated tests be written by testers?

Testers are specialized in application testing; they are trained and should have a sixth sense. They know the testing technics and tools, so there is no need for additional training. In addition, when they write the automated tests, they gather knowledge, not only about the automation tool but also about the tested application. Thanks to this they can help us if there will be more work in the project (e.g., complicated features, need of manual regression, a lot of maintenance work in automated tests) or as a backup to testers to minimize the project risk. Learning auto-mation tools is not accomplished in just few days, sometimes it can take weeks. It’s crucial not to lose that knowledge. Developers go to other projects, where automation skills may not be desired, but testers’ knowledge will be needed again and again for writing

113The Magazine for Professional Testerswww.testingexperience.com

tests but also for maintenance.

If we summarize the above, we see that there are four different ways to write and maintain automated tests:

1. Developer Internal - This is a very interesting way, which I tried to introduce in one of my projects. The idea was that each developer delivers newly implemented features to-gether with the automated scripts. Unfortunately, it failed for several reasons. We had to invest a lot of time in automa-tion tool training (the cost was high as we had to train all developers in the team). Another issue that we faced is the decision on who would write the test scenarios that were to be automated. Developers, testers? In the case of developers, we would need to train them in how to write test scenarios and how to use the test case management tool. If we chose testers, we need to find a way for them to deliver scripts to developers (this idea requires them to create test scenarios at the beginning of a sprint, before requirements go into de-velopment). Another issue (I think, the most decisiveone for the failure) is whether automated tests are required for all features. There are always other important things to do, like features that should have been finished yesterday, so writing automated tests takes second placeafter a while and auto-mation dies. It is for the same reason that this approach can’t be used for maintenance.

2. Tester Internal - The idea is that the same team executes functional, regression and other types of tests and spends any free time on automation. From our experience, however, we know that there is no free time because there is always something to do. In this approach, a lot depends on the test-er’s personality. If he is “lazy”, he will automate the most fre-quently executed tests only. So if we just want to have really poor automated smoke test suites, this approach is for you. For bigger projects it will never finished with success.

As described above, this approach is best suited for mainte-nance of automated tests. Testers run test suites, report and debug failed test scenarios. As automated tests should be ru-nas often as possible (continuous integration approach) it is very important to make changes quickly. Testers in an inter-nal team can do this very quickly because there is no need to send anything to an external team, which always takes time. In addition,the external team does not lose time on updat-ing existing tests and can instead create new ones for new features.

3. Developer External – The idea is that we create an external team composed of developers with a separate budget, with a defined aim and resources. The use of developers takes over all the minuses mentioned above for the ‘developer internal’ project. The biggest factor is that developers are not for writ-ing test scripts, but for writing the application.

4. Tester External - This idea combines all benefits from inter-nal tester internal and external developer external. Testers do what they are specialized in, it’s very easy to manage the budget, and we have a backup for testers. I have managed several automation projects using this approach, and all of them finished with success. Of course, it is always nice to have a developer (it can be rotated role) assigned to an au-tomation team to resolve complicated programming tasks. Everyone knows that 80% of automated scripts can be done by copy & paste using already written code, so there is no need for special development skills (I believe that each tester should have basic development skills).

Why do we need two different approaches for creating and for maintaining automated test scripts?

Maintenance is connected with current application development where new features have an impact on existing scripts that are used for regression tests. As testers (internal) know the test suites and application best they are the ones that can recognize points that have to be changed and can introduce the changes quickly without the need to send them to an external team. It’s very irri-tating to run test suites that we know have failed, because we are waiting for changes to come from an external team. We should note that failed tests which are waiting for update do not test anything, and we should cover this area by manual tests. Please note that it is very hard (and needs a lot of time) to find new er-rors among several already known. This is why fixing or updating test scripts immediately is very important.

Employing on two teams (an external team which only writes new scripts and an internal team to test the application and maintain automated scripts – it can be even the same person who has divided task in time) results in a process for creating new ones, which is free of interference. We can achieve continu-ous gains of automated tests and coverage. If we sum this up, it gives us shorter regression time and lower budget for testers in the internal project.

An approach like this gives you clear division of responsibility, ensures continuous growth of coverage, and achieves the aim of a fixed budget without any risk to the application development project.

Figure: Automation responsibility square

DEVELOPERS

INTERN

AL

DEVELOPERS

EXTERNAL

TESTERS

INTERN

AL

TESTERS

EXTERNAL

MAINTANANCE

CREATE

Zbyszek Moćkunhas more than 6 years of ex-perience in QA engineering for big as well as for small com-panies. Currently he works as Head of QA Practice for Cognifide - digital technology agency. The author has been involved in several projects of different size and methodology where he could work on or introduce automation. His experience

was used to optimize the time spent on automation and getting maximum benefits. This is very important, espe-cially in Agile methodologies, which is the area in which he specializes.

> biography