A Test Automation Framework

Gregory SoloveyAlcatel–Lucent

www.qualitymanagementconference.com1

2www.qualitymanagementconference.com

Test related Processes

Provide decomposition/

abstraction

Design Tests

Build Harness

Automate Testing

Include in CI Framework

It is easier to

perceive error than

to find truth

Give me a lever and a

fulcrum and I shall move

the world

Divide and rule

He who lives must be

prepared for changes

Trust but verify

There is nothing new under the Sun

• 10 minutes to automate a test case• 10 minutes to identify an error• 10 minutes to update the testware

10/10/10 Goals for Test Automation Projects

To paraphrase a quote from Tolstoy’s Anna Karenina:

All successful test automation projects are alike; each unsuccessful automation project fails in its own way.

Test Automation

Test Tools ImplementationProgramming language Test language

Data structures

variables, arrays, hashes TS,UC,TC,TA, buffer

Control statements

if, case comparefor, while, until loop for TS,UC,TC,TA

Functions subroutine, procedure TC {set, test, compare, tear down}

Control flow

algorithm unconditional sequence

Users Developers prefer to use open source software which can be adapted

Testers prefer to use commercial tools, with technical support

The Three Pillars of a Framework

• testware hierarchy• testware organization• test language

Application Layer

Abstraction Layer

Middleware Layer

Driver Layer

Behavior Layer User documentsStructural Layer

Architectural documentsFunctional Layer

Requirements documentsInterface Layer

Design documents

System Hierarchies

Test Design Techniques:•Boundary analysis•Path sensitization

Derived Test Design Methods:•Arithmetic expressions •Relational expressions •Logical expressions •Algorithm•State machine•Instruction set•Syntax

Test Design Techniques

From test techniques to test methods, Professional Tester, issue 29, October 2014

• Test Set (TS) is a part of the testware that verifies a particular architectural unit of the system.

• Use Case (UC) is a collection of end-to-end scenarios to cover the functionality of a subsystem.

• Test Case (TC) is a single verification act that moves the object-to-test from an initial state, executes a test, compares the actual and expected results, and returns it back to its initial state; a UC consists of TCs.

• Test Action (TA) is the simplest act of communication with the object-to-test. A TA supports the interfaces of the object-to-test (CLI, GUI, SNTP, HTTP).

Testware Hierarchy

Test Script 2

Testware File Organization

Language Commands interface action words

Test Case callsbusiness action words

Test Script 1

Test Script 3

Config fileTS1

Driver Libraries

Test case Libraries

Testware File Organization

Language Commands interface action words

Test Case callsbusiness action words

Config file

Driver Libraries

Test case Libraries

TS: ‘verifyMidleware’, ‘verifyAudioDrivers’, ‘verifyMapApplication’

UC: ‘vrfBufferManagement’, ‘vrfProgramMonitor’, ‘vrfFaultNotifications’, ‘vrfNetworkManager’

TC: ‘createBuffer’, ‘FillBuffer’, ‘SendMessage’, ‘SendAlarm’’

TA: ‘set’, ‘send’, ‘push button’, ‘compare’, ‘repeat’, ‘connect’

Testware Scale

Number per unit Total Unique

TS 2-50 25 25scripts 1-4 60 60

UC 2-10 250 250TC 2-30 5,000 50TA 2-10 25,000 25

Test Language• A test is presented as an unconditional

sequence of TS/UC/TC/TA executions• The instruction set includes the following

command groups: – run TS/UC/TC commands (one or multiple

times)– connection commands– compare commands– variable manipulation commands – limited control commands-pause/stop/exit

Reliability

• returning the object to its initial state• handling multiple echoes• support for stimulus-response

synchronization• multiple connection attempts

Output Buffer Structure

Match Text

Test Case Structure

Test Case

Tear down phase

Test phase Action

Action

Compare

Action

Verdicts

Test object

Verdict Conditions

TA OKTIMEOUTERROR

‘MATCH’ detected in Output Buffer‘MATCH’ not found after predefined elapsed timePredefined error detected in Output Buffer

TC PASS

All TAs are ‘OK’ & expected result is found in Output BufferAt least one TA is not ‘OK’ or a compare failed

UC PASSFAIL

All TCs ‘PASS’edAt least one TC ‘FAIL’ed

TS PASSFAIL

All UCs ‘PASS’edAt least one UC ‘FAIL’ed

userFuncLib.pm

Framework Structure

tesToy.pl

ssh/ telnet/ ksh

Failed CasesResult Log

Debug Log

Capture Log

Config file

connectLib.pm

testCommLib.pm

fullPathib.pm

Test ScriptTest ScriptTest ScriptTest Script

Test ScriptTest ScriptTest ScriptTC Library

Object to test

The test framework is a data driven application that reads test data, provides syntax checking and compiles the data into internal test structures, creates the necessary connections to the object to test, executes the tests and finally creates the log and result files.

Framework Functionality

Debug Log Structure• Supports TS/UC/TC/TA hierarchy and IDs• TA: output buffer, rules to capture it, and

how verdict was determined, based on the buffer content.

• TC: expected and real results and how verdict was determined, based on the comparison constraints.

• UC/ TS: how verdict determined, based on the children verdicts

10/10/10 Goals• Test design in 10 minutes

A framework shifts the testers’ focus from writing new test scripts to reusing existing ones

• Error detection in 10 minutesThe debug file structure allows for easy error identification and creation of “defect tracker”

• Testware maintenance in 10 minutesTestware is organized in a fashion that allows to make unique changes for any single changes in the object-to-test.

Example: Triangle Algorithm

a< b+c and b<a+c and

a = b a = c

b = c b =c

Not a triangle (NT)

Equilateral(ET)

Isosceles(IT)

Regular(RT)

Enter a, b, c

TC # description a b c result1-14 less than 3 parameters , symbols, negative and 0 values for

each parameterNT

15-17 a<b+c 6611

NTNTRT

18-20 b<a+c 233

NTNTRT

21-23 c<a+b 233

NTNTRT

24-26 2: a=b 11113

RTITRT

27-29 3: b=c (a=b) 31111

ITETIT

30-32 4: a=c (a<>b) 31111

RTITRT

33-35 5: b=c (a<>b and a<>c) 11 9 9

RTITRT

Triangle Test Set

tesToy Configuration File

[tt_defaults] testwareDir = "/home/user/testware" swDesc="Triangle Test" prompt_base = "myServer!user> " time_out = 5 logDir = '/home/user/logs' keep_going = 1 expectEchoes = 1 debug_level = 3[test_files]TS "verify triangle application" "telnet" "myServer" run triangle.uc[libraries] /home/user/libs/general.lib[user_vars] myServer = "135.135.135.135"

tesToy Test Script

UC set test environment run general::setDir ("/home/user/bin")

UC verify out-boundary data run general::whichTriangle ( "", "", "", "error") run general::whichTriangle ("5", "", "", "error") run general::whichTriangle ("5", "5", "", "error") run general::whichTriangle ( "", "5", "5", "error") run general::whichTriangle ("5", "5", "a", "error")

UC verify that length of first side is less than other two run general::whichTriangle ( "6", "2", "3", "NT")

UC verify 'a=b' condition run general::whichTriangle ("11", "3", "9", "RT") run general::whichTriangle ("11","11", "9", "IT")

tesToy Library

# execute any command and compare results TC setDir send ("cd $P[0]") send ("pwd") compare ("$P[0]")reset_phaseend_of_case # execute the triangle application and verify output TC whichTriangle send ("bin/triangle.pl $P[0] $P[1] $P[2]") compare ($P[4])reset_phaseend_of_case

Further Reading• Embedding testability, Professional Tester, issue 27, August 2014,

8– 15. – how to build testability for an embedded system • From test techniques to test methods, Professional Tester, issue

29, October 2014, 4 – 14. – how to build the minimum number of tests that covers all implementation errors

• Automating Inside the Lines, Professional Tester, 2015, issue 30. – how to build a test automation framework

• A Continuous Integration Test Framework, Testing Experience, issue 28, December 2014, 30 – 34. – how to build a continuous integration test framework

• QA of Testing, Professional Tester, issue 28, October 2014, 9 – 12. - how to automatically monitor the test completeness

Thank you for attending this session

Send your comments or questions to gregory.solovey@alcatel-lucent.com

A Test Automation Framework

Software

Transcript of A Test Automation Framework

Magento Test Automation Framework

Open source Test Automation Tools for Mobile … Automation Tools.pdf · while choosing a test automation tool/framework for ... Open source Test Automation Tools for Mobile Applications

Ios ui automation test framework

Wso2 test automation framework internal training

Building a Test Automation Framework · ─Example “core” framework •Assumptions ─You understand what test automation is (and isn’t) ─You understand why a “framework”

HTAF 2.0 - A hybrid test automation framework.

Test Automation Framework Designs

Test Automation Framework Design |

Test Automation Framework User Manual

Test Automation Framework (RC TEAF)Test Automation Framework (RC TEAF) Royal Cyber This white paper is about Royal Cyber Test Automation Framework (RC TEAF). This paper provides an

Kamil Zieliński - Appium – mobile test automation framework

Designing a Test Automation Framework by Quontra Solutions

CTA Test Automation Framework v2.1

How-To Guide for Test Automation Framework (TAF)

Introducing TEAL - Integrated Test Automation Framework

WSO2 Test Automation Framework : Approach and Adoption

Test Data - Food for your Test Automation Framework

Test Automation - cs.virginia.eduup3f/cs3250/slides/Lec05A-TestAutomation.pdf · Test Automation Framework • A set of assumptions, concepts, and tools that support test automation

Web services test automation framework challenges-benefits

Example of Test Automation Marko Rintamäki. Why Test Automation Framework? There has to be common way to communicate with test environment. Test Automation.