A Sneaky Way to Introduce More Automated Testing

W11

Stockholm 02.12.2009

Agenda.

The Problem.

The “sneaky” Idea.

The Realization.

Conclusion and general discussion.

A Sneaky Way to Introduce More Automated Testing

Chapter 1.

The Problem.

The company.

The application.

The developer team.

Overview:

1 The Problem.

2 The Idea.

3 The Realization.

4 Conclusion.

1

The Company

Steria Mummert Consulting is one of the top ten IT companies in Germany (Counseling, IT-Service). In Germany and Austria about 1700 employees

Mummert ISS GmbH was established 2002 as a 100% subsidiary of Mummert Consulting. Develops standard software for insurance companies. About 50 employees

Steria Mummert

ISS

Steria Mummert Consulting

Steria Group

Steria Mummert ISS is a small and successful company that develops standard software for insurances

1XXXXXXXX.The Problem.

The Steria Group is one of the top ten IT-Service companies in Europe with worldwide about 18.000 employees.

The Application

The product is:

a complete software solution for insurance companies,

a standard software, that enables a flexible management of the products and processes of an insurance company and with which both partner and portfolio can easily be maintained.

System design:

Completely newly developed standard solution.

Customer oriented application structure.

…..

1Funktionsumfan.The Problem.

The application has a standardized core and offers great flexibility in modeling the individual processes of an insurance company.

The Developer Team

Test Automation

TeamNew Customer

New Customer

New Customer

Product team

1XXXXXXXX.

Some years ago the product team was able to support both new customers and the automation team, currently it has trouble to support even only the customers.

The Problem.

New Customer

New Customer

New Customer

X

Old Customer

Old Customer

Old Customer

Chapter 2.

The “sneaky” Idea.

What do we need in order to be able to automate?

Can manual testing deliver it?

Can we convince our colleagues to switch over to the “automation” framework for manual testing?

Can we extract and use the collected Information for automation?

Overview:

1 The Problem.

2 The Idea.

3 The Realization.

4 Conclusion.

2

Prerequisites for automation2Funktionsumfan.The Idea.

Testers that execute manual tests have the information needed to automate those same tests

By performing manual tests a tester can deliver at least:

Knowledge of the application under test.

Test specifications.

What do we need in order to be able to automate?

Knowledge of the application under test.

Test specifications.

Automation strategy.

Test-Tools.

Automation Strategy: Command-Driven Testing

Step 1

Step 3

Step 2

Prerequisites

2XXXXXXXX.

Capture

Tool Script

Tool Script

ISS

ISS Test Station

Driver Data

DriverData-

Template

Suite

Test cases

Registration ofGui Controls

1.

3.

Our Command-Driven framework supports the test automation team by (mostly) eliminating the need to program new keywords and by automating some of the steps

Automated test

Capture-ReplayTool

Suite

Test results

2.

Command Library

The Idea.

Tester

Tester

Supporting manual testing2Funktionsumfan.The Idea.

The original testing procedure was based on excel-test-sheets, that would be performed again and again for regression testing.

Original testing process

Delete old results from the template. Enter eventual corrections, improvements or just plain new test cases. Enter the new results. Calculate metrics. If defects are found, create a defect-item in the development control system.

Weaknesses of the manual procedure2XXXXXXXX.

Test case maintenance

3. Cumbersome defect reporting.

4. Test cases tend to be repeated in order to report the defects exactly.

Defect reporting

5. No automatic statistic of the executed test cases versus the defects found.

Summary reportingSummary reporting

In order to support manual testing with the automation framework, we had to find out how testers worked and devise better support.

1. Corrections or improvements to the test template can get lost.

2. Test-specifications are very “concise”.

The Idea.

Available features in the automation framework2XXXXXXXX.

Add or change test cases.

Overview of the available test cases.

Logging of test results.

Result reports or overviews.

Prioritization of the test cases.

Defect reporting.

Ava

ilab

le f

eatu

res

Must have Enhancement

X

X

X 1

X 5

X

X 3,4

The Idea.

Many of the available features in the automation framework could, with no change, be employed to support manual testing.

Features that have to be implemented2.

New features had to be implemented in the automation framework in order to be able to support manual testing adequately .

Test driven by sequential display of the selected test cases.

Paperless execution.

Support in:

Support for manual test execution

Import from the current excel sheets (csv-files) to the test suites in the framework format.

Import functionality driven by an external table.

Import test-template sheets

Capture-facility integrated into the framework.

Generation of a capture-replay script with simultaneous recording.

Starts or stops by pushing a button.

Recording automatically attached to defect tracking item.

Detail test execution for developers

Integration of SQL-scripts to extract data from the database.

Save application status to file

The Idea.

Performing tests.

Maintaining test cases.

Creating defect items.

Test recording.

Chapter 3.

Overview:

1 The problem.

2 The Idea.

3 The Realization.

4 Conclusion.

3The Realization.

Manual testing with the automation framework.

Automating the manual tests.

…

Manual test execution with the automation framework3

Test execution is driven by the framework, but the tester is free to decide what she wants to do next and how

Set preconditions and export them to file

Start recording

Perform test case

Export post- conditions to file

Check test case

display test case

OKDefect

The Realization.

Start test suite

Automating the manual tests

Step 1

Step 3

Step 2

Prerequisites

Capture

Tool Script

Tool Script

ISS

ISS Test Station

Driver Data

DriverData-

Template

SuiteRegistration ofGui Controls

1.

3.

After the testers started working with the framework, creating more automated tests has been much more rewarding because some steps have already been performed

Automated test

Capture-ReplayTool

Suite

Test results

2.

Command Library

Test cases

1

2

1

3.The Realization.

Automating the manual tests3

Call the framework functionality to transform the captured tool-scripts into Command-Driven scripts.

Step 1

Rework the command-Driven scripts.

Step 2

Create the new test procedures.

Step 3

Insert the test procedures to perform the test cases in the test suite structures.

Step 4

The input from manual testing reduced the effort to automate a test case to the following steps

The Realization.

Check the automated test suite

Chapter 4.

Overview:

1 The problem.

2 The Idea.

3 The Realization.

4 Conclusion.

4Conclusion.

Short summary of our experiences.

Next steps.

General discussion.

Short summary of our experiences4XXXXXXXX.

After some time to allow the testers to get acquainted with the new process, we started to get the first results.

Conclusion.

The testers had a hard time getting used to the new formats.

The import facility, test-sheet test suite was well accepted from the

very beginning.

The recording feature was very well received by both testers and

developers.

In the beginning the testers were recording either too much or too little.

Testers from other product families also started to use the framework.

Next4XXXXXXXX.

…and they lived happily ever after. Well, we still have lots of effort and work to do.Conclusion.

Partially automated tests.

Sophisticated metrics.

Integration with test planning and management tools .

It’s better to light a candle than to curse darkness

Confucius