Selenium vs. Watir Evaluation

William MostellerTest Automation Specialist Copyright © 2009, Eloqua Inc.

The Initial Challenge

Who is Eloqua?Leading marketing automation platform (SaaS)Global operationsOver 200 employees, and 500+ customers

We’re testing a re-write of the web applicationEvaluate Selenium and Watir and decide which is better for our needs

Similarities

Both are open sourceBoth support browsers onlyWe drive both with Ruby (Selenium RC)

Ruby reports errors by line numberYou need an editor like NotePad++Or SCITE, which comes with Ruby -• C:\ruby\scite\SciTE.exe

Unlike SilkTest, no window definitionsUnlike SilkTest, tags may be fuzzy

Differences

Selenium RC requires that you start a server on your computer

Easily implemented with a .bat file in the Startup folder

Selenium supports methods with x, y offsetsSelenium supports more browsers and operating systemsSo we started with a Selenium bias

Our Strategy

• “Scientific Method” Comparison:– Multi Language Support (non-ascii

characters) – Docs – Maturity – Frames – Learning Curve – Existing team knowledge – Price/License – Community – Etc.

Initial Experiences

Early TrialsSelenium Scripting with preliminary versions of our applicationWatir Scripting with LinkedIn

Both looked promising

Scripting the Application

Developed in SproutCoreNew open source development framework

In both tools…Click doesn’t workDouble-Click doesn’t work

“No battle plan survives contact with the enemy!”My new thought:

We may have to settle for the one we can get to work!

More Issues

Handling Modal Dialog BoxesHandling Application All Actions ButtonEntering Text in the YUI EditorSelecting Text in the YUI EditorDrag ‘n Drop

Aside - Information Resources

Open source documentation is distributed, you need research skills

Watir - wiki.openqa.org/display/WTR/Project+HmeSelenium - seleniumhq.org/projects/core/ & wiki.openqa.org/display/SEL/Home

GoogleWatir group on GoogleSelenium Forum on Open QA

• clearspace.openqa.org/community/selenium

Selenium and Watir groups on LinkedInFriends

Click Resolution

In Selenium(unless otherwise noted, code examples here are Selenium)

mouse_down(locator)mouse_up(locator)

Double Click Resolution

SproutCore Changes RequiredMS-IE Implementation• fire_event(locator, "DblClick")

FireFox Implementation• paired MouseDown/MouseUp events

Dialog Box Resolution

Watir solutions are based on Click_No_WaitSelenium provides functions for “are you sure” caseUse AutoIt based scripts, running separately

www.autoitscript.com/autoit3/index.shtmlProgrammed in Ruby, but with different methodsMay need to register AutoitX3.dll using regsvr32Windows only

All Actions Resolution

MouseDown/MouseUp opens the drop-downClick chooses the actual selection

Aside - Tagging Tools

With JavaScript, View Source isn’t helpfulMS-IE

IE Developer Toolbar

FireFoxFirebugDOM InspectorXpather(Install these globally in FireFox)

YUI Text Entry Resolution

Proper Tagging//body[@class='gecko']Note terse tag coding style

YUI Text Selection

SilkTest and WatirTwo ways to put text in a field (Watir examples)• Set()

• Send_Keys()o Special syntax for control characterso "+{END}{DELETE}" + CannedText + "+

{HOME}"

Selenium doesn’t work this way

YUI Text Selection Resolution

.key_down_native("17") # cntl down.key_press_native("65") # “A” press.key_up_native("17") # cntl upNeed JavaScript Character Codes

www.cambiaresearch.com/c4/702b8cd1-e5b0-42e6-83ac-25f0306e3e25/Javascript-Char-Codes-Key-Codes.aspx

Drag ‘n Drop Resolution

drag_and_drop_to_object()Animation prettier with MS-IE

Have suggestions for Watir

We Chose Selenium – March, 2009

We solved all our issues thereProbably could also do so in Watir, but it looked more expensive• Drag ‘n drop worked intermittently

• Send_Keys() not available in FireWatir

And the initial Selenium plusses still matter

Selenium supports methods with x, y offsetsSelenium supports more browsers and operating systems

Since Selenium Selection

Now testing our legacy application with Watir

Selenium has trouble with modal browser windows, and legacy app has lots of themWatir handles MS-IE tables faster than Selenium

Frames in legacy app complicate Watir tags

Must mention every frame above your widget

Syntax Matters

Is a.b.c.d present on the screen?Selenium: …element?(a.b.c.d)Watir: …a.b.c.d.exists?• If c does not exist, or something weird

happens, you still get an exceptiono Use nested .exists? or o While/Begin/Break/Rescue/End around the code