Automated Testing

Nathan Weiss

April 23, 2007

Overview

History of Testing Advantages to Automated Testing Types of Automated Testing Automated Testing Tools Disadvantages to Automated Testing

History of Testing

History of Testing

Four main phases in testing methodology: Debugging (1950’s) Exhaustive Testing/Prove it Works

(1960’s) Prove it Does Not Work (1970’s) Defect Prevention & Testing Process

(1980’s)

Debugging

No distinction between debugging and testing.

Consisted of making code compile and changing code so that it would not crash.

Exhaustive Testing/Prove it Works

Exhaustive testing in terms of code paths and given input. Found to be impossible

Write tests to prove the program works.

Testing defined as “what is done to demonstrate correctness of a program.”

Prove it Does Not Work

Testing not for proving correctness, but for finding defects.

Started testing to see if a program did what it was not suppose to do, not just test to prove that it does what it was suppose to do.

Defect Prevention & Testing Process

“Testing” traditionally referred to what was done to the system after working code was produced.

Testing moved towards an activity done during a phase, not a phase by itself.

Studies show that 50% of bugs are created in requirements and design stages.

Advantages to Automated Testing

Test scenarios impossible/unfeasible using manual testing.

Lowers testing time and costs in an iterative environment.

Remove the human error factor. Introduces stability in regression tests. Increase in testers’ morale.

Types of Automated Testing

Split into two categories Black box tests White box tests

PC applications Embedded Systems

Black Box Automated Testing

GUI testing Always been prime candidates for

manual testing. Need to automate has created many

commercial tools. Two ways to create the tests:

Scripts Recordings

Black Box Tests - Scripts

Less of a testing tool/environment and more of a testing library.

Flexible. Easy to run batch scripts to run

multiple tests.

Black Box Tests - Recordings

Full testing tool/environment. Tester performs a test once while tools

is recording his actions. Tester can add ending conditions to

test the success of a test. Very useful for testing GUIs of stand

alone applications.

White Box Automated Testing

Module testing. More APIs then environments. Embedded systems introduce more

complexity to white box testing.

White Box Tests – PC Apps

Tests usually written in the same language as the system being tested.

Tools usually come in the form of an API containing asserts and other testing utility methods.

Tests can be performed on any level of abstraction required by the project.

White Box Testing – Embedded Systems

Test code usually not stored/run on the same hardware as the system code.

Do not have direct access to system code or memory.

Requires an interface with an embedded system debugger.

Automated Testing Tools

xUnit SilkTest Selenium Custom

xUnit

Compilation of popular code driven testing frameworks.

Frameworks based on the design by Kent Beck.

Originally implemented for SmallTalk as SUnit.

xUnit

Partial List of Frameworks NUnit (.net) JUnit (java) CUnit ( C ) CPPUnit (C++) SUnit (SmallTalk)

xUnit

xUnit test execution Setup Test body Teardown

xUnit

SilkTest

Created by Borland. Works with:

Web Browsers(IE and Firefox) Java GUIs .Net GUIs Win32 GUIs

Price : 4-6k

Selenium

Testing tool for web applications. Comes in API and tool form. Supports:

Windows: IE, Firefox, Seamonkey, Opera Mac OS X: Safari, Firefox, Camino,

Seamonkey Linux: Firefox, Konqueror

Selenium

Can create tests using the Selenium library in many languages (Java, .NET, Perl, Python and Ruby)

Or you can create tests using their Selenium IDE which is implemented as a Firefox extension. Once created in Firefox, the tests can be

run in any of the suported browsers.

Selenium

Custom

Write your own custom testing environment or API.

Used when you have very specific testing needs.

Often used for embedded systems.

Disadvantages to Automated Testing

Initial cost.

Who tests the testers?

References

Borland. (2007). Borland SilkTest: An Automated Regression and Functional Software Testing Tool. Retrieved April 2, 2007, from http://www.borland.com/us/products/silk/silktest/index.html

[2] Dustin, E., Rashka, J., Paul, J. (1999). Automated Software Testing: Introduction, Management, and Performance. Addison-Wesley.

[3] Lewis, E., W., (2005). Software Testing and Continuous Quality Improvement. CRC Press LLC.

[4] NUnit. (2006). Nunit. Retrieved April 2, 2007, from http://www.nunit.org

[5] OpenQA. (2006). OpenQA: Selenium. Retrieved April 2, 2007, from http://www.openqa.org/selenium/index.html