Selenium - The Way Of Success

QA:news QA:news QA:news QA:news –––– N5 N5 N5 N5

Zbyszek Moćkun and Karol Kujawiak – Cognifide

June 2011

Selenium - the way of success

Intro

First, we would like to share with you when the idea of writing this article came to our mind.

A few months ago one of my colleagues sent us a link with a note - “it might be an

interesting conference for you”. We opened that link with curiosity - it was a link to the first

conference about Selenium in San Francisco (http://www.seleniumconf.com/). And it was

the moment when we realized how popular this tool is. It’s not only used by testers with

technical knowledge but by beginners too. In this article we will try to concentrate on

features and share our experience how to introduce and expand automation in projects

using Selenium.

About the conference - If you are using Selenium or want to learn more about the tool

please visit the webpage. You can find there videos from conference. On the webpage you

read in how many ways you can use it in automation.

Our experience with Selenium I and II

Figure 1 - Selenium

Authors have been using and extending Selenium for several years. A great influence on

how we use Selenium has our experience from testing tools that we had used before. What

we achieved is to moving good practices from them into Selenium - make this tool ideal :).

Trying to make automation easier and quicker we develop Selenium extensions and use

advice from Selenium community. Although we achieved a lot of using Selenium I about half

year ago we reached the point where further improvements were hard to introduce and time

consuming.

It was the moment when we had to decide about future of automation in our company. After

some research and labs we found it - the answer was quite easy - Selenium II. After less

than half of the year using Selenium II we made a big step in next level of automation.

“Exploratory Automated Test” - it’s the best description of the idea. The idea was to move

exploratory testing into automation (more flexible, lower costs,

quick to introduce, coverage most of functionalities without test

management costs, concentrate on aspects that are important for

users not on specific tests cases, concentrate on finding symptoms

of issues rather than verify specific functionality). Please notice that

automation in 99% of usage is use for regression tests.

Our aim is to achieve automated tests that can be written in “five”


minutes. Adding new component/functionality to tests should be as easy as writing one line

of simple code (example open a page with several version of the same component) that

shows where it is only. Scripts that are written by tester are run on that page and should

learn themselves about functionality and then use this knowledge in regression tests

(compare results with reference patterns).

Selenium against other automation tools

There is a lot of discussion on web which tool is better. Especially the war is between

Selenium and WebTest. Almost all blogs/discussions begin with or refer to Marc Gillemont

blog - and his comparison between WebtTest and Selenium

(http://mguillem.wordpress.com/II007/I0/II9/webtest-vs-selenium-webtest-wins-I3-5/).

We disagree with Marc opinion (please notice that he wrote it as WebTest committer) at

several points, its good place to start and compare your thoughts (as different features are

more important for you). Moreover, this blog post was written four years ago and

applications have changed a lot since then.

We don’t want to concentrate on that point in this article or try to compare Selenium with

other tools. We would rather concentrate on Selenium itself.

Selenium IDE - quick and easy to start

Selenium IDE is the most powerful part of Selenium. We think (and in our case it’s how it

was) IDE decided about Selenium popularity. Thanks this Selenium is not a tool for narrow

part of testers’ community who has developed skills but it can be used by all, even not a

testers. IDE provides a quick start which allows to show business side very quick results of

automation and gets money for further growing (for example develop programming skills).

From IDE to RC

For very beginners there is a record function, which has an ability to store actions like clicks

or typing. But only a very simple test could be recorded. When testing target is more

sophisticated, a tester has to raise his knowledge and learn some commands and their

usage. After that, designing even a complex test script isn’t so difficult.

Selenium IDE has many ready to use extensions which could extend its functionality such as

flow control, loops etc. Testers could also write extensions on their own. All that is needed is

more or less JavaScript knowledge.

Worth mentioning is that scripts written in Selenium IDE can be fully integrated with

continuous integration server, and run not only against Firefox but also against Internet

Explorer or other browsers. After the test run there is a detailed report generated which

provides information about test progress and possible errors. This report is not great but is

quite easy to read in compare with JUnit reports.

What more could we expect from testing tools? From our point of view many companies limit

their tests to which we have described previously. But if you look deeper there are some

areas which Selenium IDE or even RC with object oriented languages doesn’t cover. Here

comes Selenium II.


Selenium II

Selenium II in a new web testing framework which provides new opportunities, but also puts

new demands on the QA. In Selenium II test cases are written in high-level object oriented

programming languages such as C# or Java so HTML is no longer supported. Tester who

would like to write test scripts in this framework has to have knowledge of C# or Java

basics. But high-level object languages give an opportunity to make it fit your needs.

Functionality can be automated not only by simple step by step test case. These steps can

be concentrated in parameterized functions, which are simple to maintain.

Our goal is to provide high-quality products while keeping costs at an acceptable level. Apart

from functional tests we also wanted to run layout tests, check if site meets WC3 standards

and also if the analytics engine works as expected. We also wanted to have an ability to run

tests on collections of migrated sites (or after backend changes like new version of

Hibernate), to check if there are some differences. All of these automatically of course. Our

approach forced us to create our own framework, which gives us such possibilities. The main

advantage of our framework is that test cases written in it could catch issues that might be

unnoticed by manual tests.

Other reason why we created our framework was to have an ability to provide some

standalone testing tools for our clients. Especially when migration of huge collection of sites

comes into play, standalone tools are very helpful. In this case the only effort that clients

have to make is to create a set of original and migrated url’s.

Figure 2 - Four aspects

Testing mobile devices

If there is a client whose main targets are iPhone and Android phone users, website has to

meet some specific requirements. It’s easy to figure out that these requirements could be

tested only on native environment. Selenium II provides such functionality. Test scripts can


be run on the device or on the emulator. There is no need to prepare separate test suites to

run in mobile device environment. The same piece of code can be used for testing

standalone and mobile browser.

Current beta version is partly underdeveloped, so not everything works how it should, but I

am sure that final version will give us possibility to attract potential clients with such

opportunity.

Selenium II – imperfections

Reports

Reports are one of the most important parts for automation. Selenium I has a built- in

report generator for HTML tests, which by default provides detailed information about the

test progress and failures. Quick debug which give you enough information (no need to

rerun test, all results saved, all data for debug available). It’s very important, because we

want to run our test in Continuous Integration environment (each commit/daily) so we will

check them minimum once daily.

When we run Selenium tests in Java - we use JUnit framework which means that is case of

any errors we get only stack-trace. It’s very difficult to debug and it consumes a lot of time

to discover what was the cause of error - particularly when issue is not I00% reproductive or

without specific java knowledge. Bad reports as JUnit only complicates it, additionally in

most of cases require rerun test manually.

WebTest is an example of how reporting part should work. Fortunately, it was not difficult to

implement the same for Selenium and effort returned after a few weeks. The main

advantage of custom report is that it could be well fitted to our demands, and could contain

some customization like JSON files generated by firebug, screen-shots, WC3 validation

details, source code etc.

Lack of HTML support

As mentioned before Selenium II is based on high-level programming languages and no

longer supports html scripts. It could be hard for beginners to learn how to automate using

C# or Java, but you can look at it from the other side, and treat Selenium IDE as first

degree of initiation. Writing in Selenium II would for sure improve testers’ technical skills

and broaden their horizons.

Google contribution to the development

Worth mentioning is that the main developer of Selenium II framework is Google Inc. Google

launched wide accessible bug tracking system, which allows testers from all over the world

to notify about all of the issues and suggestions that they may have. It was also an impulse

to grow a huge community of Selenium users, which could communicate with themselves

and share their experience and know-how. That confirms our belief that the final version

Selenium II will be a high quality product with solid support from Google and community

side.


HTML against java

The most interesting survey which we have made during last few years was about Selenium

programming languages: java vs. HTML (by html we mean test created by Selenium IDE not

converted to other languages). The same teams in almost identical projects were trying to

achieve two goals: to cover by automated tests about 20% of functionality and maintain

them during the project life-cycle. The result was a little surprising. Coding in HTML and

maintenance these type of tests takes 50% less time than java. It shows a very important

fact and I think this one decided about Selenium popularity - ease of use, quickness of

creation, no need of technical knowledge so almost all team members can work on Selenium

HTML tests. Flexibility of Java wasn’t an argument even for Java developers. Of course

please notice that the aim was to automate about 20% of test cases (about 100-200 tests).

If we want to achieve better coverage probably java flexibility dominates html.

Continuous Integration

Integration with CI environment shows the strength of open source community. There is no

problem with configuration, because it was described on several blogs/pages and thanks this

was quick and easy. There is also a number of plug-ins which could improve functionality of

CI environment, which already have been written by open source community.

Developers could get test reports on demand and instantly. They can run our tests on

nightly builds, their development machines after each commit/change using one click

feature. Reports (rewrote from WebTests) was so clear that give them all information about

errors. Thanks this they send to QA high quality software with minimum of regression

issues. Less time spend on testing/retesting bugs allowed us to invest more in automation.

Please notice that Selenium suite contains Selenium Grid - a tool to run the same test in

parallel on different browsers/machines. We don’t want to describe it here, as our

experience is not big. We have just tried it and when realize that the same you can achieve

using CI tool (CruiseControl/Jenkins) stop researching. It’s better to minimize the number of

used tools.

Figure 3 - Continuous Integration Environment


Selenium II - future of automation and few our thoughts

Why have we called article - the way of success. It’s because Selenium allows us to

introduce automation successfully in long term and short term project, minimize automation

cost and gave us ability to automate almost all aspects that are important for users or

customers.

Thanks Selenium II we are able to think about future of testing. We called it - Exploratory

Automated Tests. Tests are learning using results from reference run and then only verify it

when rerun again. The idea is not to write test case for all functionalities separately - rather

to use pages created during manual (functionality) tests and run on them robots specified

for application. Robots are looking for symptoms of errors (changes on layouts, JavaScript

errors, performance issues, w3c validation, content comparison, link checker, errors in logs

...). Most of errors have some symptoms which can be found by above approach.

ABOUT THE AUTHORS

Karol Kujawiak and Zbyszek Mockun are working for Cognifide (http://www.cognifide.com)

- Digital Technology Agency where they was engaged in long or short term projects for

variety of clients (worldwide recognized brands). Authors gather their experience about

automation in previous companies too - thanks this they have wide recognition of

automation tools and approaches.

Last few years they have been working in Agile environment where clients demand

forced approach which is described in this article. If we summary our achievements this

approach is quicker, give better results and is cheaper for clients than standard one.

Karol Kujawiak Zbyszek Moćkun

Selenium - The Way Of Success

Technology

Transcript of Selenium - The Way Of Success