Avoiding the Pitfalls of Automated Testing

Quality assurance engineers are being asked to take on more complex projects than ever before. Application complexity has increased to

the point that formerly simple test plans now encompass hundreds, if not thousands, of test cases that must be executed under stringent deadlines. These demands have resulted in quality assurance teams seeking higher effi ciencies than traditional manual testing provides.

by Allan McNaughton

Much of the focus of this search has been on automated testing tools. It is commonly thought that automated testing is the panacea for all quality assurance woes. But is that always the case? While a well-executed automated testing strategy can measurably improve productivity, in reality there are many pitfalls waiting to ensnare the unwary. Quality assurance teams need to carefully analyze the path before them. By choosing the right tool for their needs and using it wisely, they can reap the much heralded benefi ts of automated software testing.

Select the Right ToolNothing will sink an automated software testing project faster than adopting the wrong tool for the job. Testers are faced with a bewildering array of choices that range from loosely supported open source solutions to commercial tools that claim to solve most every automated testing problem.

Which tool is right for the job? The question is best answered by honestly evaluating your organization’s needs against the products that are available in today’s competitive marketplace. To succeed at automated testing, it is imperative that

the selected tool be a good fi t with your software technologies (MFC, .NET, Web, etc.) and your testing team’s technical capabilities.

Automated testing tools basically fall into two camps: those that rely on scripting languages and those that use a graphical approach. Code-based tools have complex scripting languages but can be customized in enumerable ways. While this fl exibility may seem desirable, it can cause more problems than it solves. These tools are extremely complex, and they can be daunting to all but the most technically sophisticated users. This complexity often results in the tool getting shelved after the

initial enthusiasm fades.

A better solution is to seek out a tool that does not require programming skills yet offers functionality comparable to those that do. Seapine Software’s QA Wizard fulfi lls this requirement by combining the power of a language-based tool with an easy-to-use graphical interface. This dramatically reduces the time it takes to learn the tool and lets all team members participate in automated testing projects, regardless of their technical skills.

QA Wizard’s simplicity also signifi cantly reduces long-term maintenance costs. This is a major benefi t as the only certainty in automated testing is that test scripts eventually break. Tests are created using assumptions about the application under test. When the application changes in a meaningful way it is quite likely scripts will fail. Users of language-based tools will fi nd themselves laboriously poring over lengthy scripts trying to rectify the problem. Users of QA Wizard can see at a glance why tests failed and can update out-of-date scripts without touching a single line of code.

Plan for SuccessAutomated testing projects must be carefully planned to ensure success. The fi rst step is to set realistic goals. Many testers think the goal should be to automate 100% of the test plan. Experience has shown that just reaching the 75% level is a noteworthy accomplishment. The fact is that manual testing is still required even in the most heavily automated environments. However, by using QA Wizard you can reach higher levels of automation faster than what is possible with language-based tools.

Determining which parts of the application are right for automated testing and which parts should continue to be tested manually is straightforward when the following guidelines are heeded:

1

■ Identify components of the application that require large amounts of data entry. Data entry is time consuming and can be easily automated. QA Wizard can retrieve these data values from a spreadsheet or database and enter them for you.

■ Identify areas of the application that require repetitive testing with different data values for each iteration. QA Wizard can retrieve these values from a spreadsheet or database and run through the test multiple times using new data values with each run.

■ Identify portions of the application that require data comparison or validation. Text comparison and data validation are areas that manual testing can easily overlook. QA Wizard can compare and validate the same information with total accuracy.

■ Identify which parts of the application deal with complex business rules. Although QA Wizard can easily direct the fl ow of a script using context checks and conditional statements, it is generally more effective to manually test complex business rules.

■ Avoid automating areas of the application that change constantly. Automated testing is better suited for more stable components (such as core functionality)

that can reuse test scripts.

Adopt Best PracticesAs with any tool there are many ways to approach a problem. To get the most out of automated testing it is recommended that a number of easy-to-follow best practices be followed. These simple steps will help you reap signifi cant rewards by streamlining the creation of automated test scripts and reducing maintenance of those scripts down the road.

Organize Your ProcessAd hoc scripting should not be permitted. Every test script should fi t into an overall architecture. The importance of properly organizing test scripts rises in

direct proportion to the number of tests that must be automated. When there are hundreds of automated tests it is easy to be overwhelmed without a well-defi ned system for managing them.

QA Wizard scales to support the largest test suites by allowing users to organize test scripts hierarchically into workspaces and

projects. Projects exist within workspaces and multiple workspaces can be created. Some teams create one workspace while others create separate workspaces for each application being tested. Likewise, some teams create projects for each component in an application while others create projects based on the type of testing taking place (See Figure 1).

Think SmallDevelop short maintainable scripts. Longer scripts are harder to debug and less reusable. With QA Wizard, you can record an individual script for each test in the plan or easily break down a longer script into multiple smaller scripts after it has been recorded.

The process of script decomposition highlights a key difference between QA Wizard and code-based tools. In a code-based tool you must locate the code that implements the test, cleanly extract it, and create a new script that seamlessly integrates this logic. As any tester knows, it is all too easy to make a mistake when doing this type of task. QA Wizard visually presents what each step does, not how it is accomplished. This makes moving a step from one script to another a simple matter of copying the step to its new location (See Figure 2).

Figure 1 Project Explorer

Figure 2 Step copy and paste

3

Plan for ReuseCreate reusable scripts to test common user interface components. For instance, an application often contains multiple occurrences of drop-down menus. While drop-downs always work the same, testers usually implement separate test logic for each one. A more reusable implementation is to develop a callable function that determines the number of items in the menu and iterates through them one by one until all choices have been tested.

Validate ResultsWhile automated testing tools are commonly used to control an application as if it were being used by a manual tester, they may not catch all the errors that a person can unless additional measures are taken. For example, validating a spreadsheet may require the tester to enter specifi c values into a worksheet and then verify the results of a calculation. If the script does not validate results, the test may pass even though the application generated an incorrect answer.

Testers can add validation steps to their test scripts by creating checkpoints using QA Wizard’s point-and-click interface (See Figure 3). QA Wizard supports the following types of checkpoints:

■ Value checkpoints verify that an object contains a certain value such as a button label or the result of a calculation.

■ Attribute checkpoints verify that an object has a specifi c attribute or property such as class or HTML ID.

■ Format checkpoints verify that an object has a specifi c format or property such as position, font, or text alignment.

■ Contains checkpoints verify that a parent object contains specifi c child objects.

■ Child count checkpoints verify that a parent object contains a specifi c number of child objects such as a drop-down list with 10 items.

■ Focus checkpoints verify that the focus is set to a specifi c object.

After each test run QA Wizard generates a report that shows whether a checkpoint or any other step in a test script failed. Testers can quickly ascertain the nature of failures by examining the actual and expected results of the script. Analysis is further simplifi ed as QA Wizard captures actual screenshots of failing script steps as they occur.

Use Data Driven ScriptsTest plans often require that the same test be executed using different parameters (user names, account codes, etc.). The work required to accomplish this can be dramatically reduced by having QA Wizard retrieve the test inputs from a data fi le. When a script is run, the input information is read from the external data fi le. Subsequent iterations of the script use a fresh set of records from the fi le. Data can be retrieved from a spreadsheet, database, or multitude of other data sources.

Test DeeplyThoroughly testing an application requires more than just clicking buttons using an automated testing tool. For example, if an application writes to the Windows registry during execution it would be useful to know that the correct value was written. This requires leaving the confi nes of the testing tool and delving into the operating system to retrieve the right value.

QA Wizard bridges this gap with its support for .NET assemblies. .NET assemblies can be used to integrate QA Wizard with external systems such as email servers or

SOAP servers or to access an operating system service such as reading and writing system registry keys. Assemblies can also be used for custom data manipulation such as generating unique login names. Methods can be called directly from QA Wizard so individual script steps can be customized as needed.

Capture KnowledgeTest scripts should be documented so they can be easily understood by others. QA Wizard’s graphical approach to automated testing relieves testers of much of this work. Every step of a QA Wizard test script has an image showing exactly what fi eld the script is testing (See Figure 4). This visual documentation can be quickly understood by those who are new to a project. Testers can further clarify scripts by inserting script steps that only contain descriptive comments.

Harness IntegrationAutomated testing tools are a key part of the software development process. To facilitate tester productivity they should integrate with source code control (SCC) and defect tracking tools.

QA Wizard makes life easier for test developers with its support for SCC tools such as Seapine’s Surround SCM and Microsoft Visual SourceSafe. This integration allows testers to benefi t from fi le versioning just as programmers do. Files can be checked in and out of repositories directly from QA Wizard. Testers should take full advantage of the robust branching capabilities of Surround SCM. Branches

Figure 3 Creating checkpoints

Figure 4 Previewer

let you manage multiple versions of a test suite so scripts for today’s version of an application can coexist with those for the next release (See Figure 5). If an incorrect script is checked in, Surround lets you correct it by rolling back the fi le to the last known good version.

QA Wizard also saves testers the drudgery of manually creating bug reports. It is fully integrated with TestTrack Pro, Seapine’s issue management system, and can be confi gured to automatically submit bug reports as

4

Figure 5 Script management in Surround SCM

tests fail or to allow the user to examine individual test failures and then submit reports at the push of a button. This integration further drives the development process as TestTrack Pro can send email notifi cations the moment new issues are received. This greatly increases communication between the quality assurance and development teams.

Developers can resolve problems found by QA Wizard more quickly than normal, as each issue fi led in TestTrack Pro contains the details necessary to duplicate the defect and a screen shot of the application at time of failure (See Figure 6).

Enhance ProductivityAutomated testing holds great promise. The tyranny of repetitiveness can be broken when

About the AuthorAllan McNaughton is a patent holding technologist and veteran writer with more than fi fteen years of industry experience. He is the president of Technical Insight, LLC, a fi rm specializing in the composition of high-technology white papers. Mr. McNaughton is a frequent contributor to leading technology publications and can be reached at allan@technical-insight.com.

Seapine Software™

Changing the World of Software DevelopmentSeapine SoftwareChanging the World of Software DevelopmentSeapine Software

Seapine Software, Inc.

Headquarters5412 Courseview Dr., Suite 200Mason, OH 45040Toll-free: (888) 683-6456

Seapine Software InternationalCanisiusgasse 6/391090 Vienna, AustriaTel: +43 676 930 63 93

Email: info@seapine.comWeb: www.seapine.com

the right tool is used intelligently. To reach this goal it is imperative that ease of use be seriously considered when selecting an automated testing tool. QA Wizard lets you rapidly create powerful scripts that can be easily understood by testers at all levels of technical skill. This simplicity lowers costs and enhances productivity by requiring less training and fewer resources than language-based tools.

Figure 6 TestTrack Pro integration