Automation Framework eBook

8/8/2019 Automation Framework eBook

http://slidepdf.com/reader/full/automation-framework-ebook 1/17

Software Testing : Automation Framework’s eBook from www.OneStopTesting.com

World’s Largest Portal on Software Testing Information & Jobs -http://www.OneStopTesting.com

Join Software Testing Community athttp://groups.yahoo.com/group/OneStopTesting/

Over 5,000 Testing Interview Questions athttp://www.CoolInterview.com

Software Testing AutomationFramework

Free Study Material fromwww.OneStopTesting.com







Automation Framework

The Software Testing Automation Framework (STAF) is an open source project

that enables users to create cross-platform, distributed software test environments.The project is licensed using the Common Public License (CPL). STAF includes anumber of services that provide specific functionality. The most prominent of these isthe test execution engine called STAX. Other services provide cron, file system,inter-process communication, e-mail, and HTML support, among others.

STAF was originally developed for internal use by IBM in the 1990's. IBM supportsusers of STAF through extensive online documentation and user forums.







Keyword-driven Testing

Keyword driven testing is a software testing technique that separates much of the

programming work of test automation from the actual test design. This allows teststo be developed earlier and makes the tests easier to maintain. Some key conceptsin keyword driven testing include:

• Keywords, which are typically base level and describe generalized UIoperations such as "click", "enter", "select"

• Business templates which are typically high level such as "login", "entertransaction"

• Action Words, or short "Actions", which can be both base level and high leveland in their most general form allow earlier defined key words to be used todefine higher level action words

Keyword driven testing is a very powerful tool helping organizations to do moreautomated testing earlier in the testing process and making it easier to maintaintests over time. As with any complex undertaking, there are "success factors" thatcan determine whether or not a testing effort will be successful. This paper willoutline key success factors for keyword driven testing including base requirements,the vision for automation, success factors for automation, and how to measuresuccess.

Contents:

Overview Base Requirements The Framework

Methodology Vision for Automation Success Factors for Automation How to Measure Success Conclusion







Overview

Keyword-driven testing or Table-driven testing is a Software testingmethodology as part of the Test automation discipline that separates theprogramming work from the actual Test script so that the test scripts can bedeveloped without knowledge of programming. As a result the test scripts can bemaintained with only minor updates, even when the application or testing requiressignificant changes.

Base Requirements

There are numerous requirements that consider to be "base requirements" forsuccess with keyword driven testing. These include:

• Test development and automation must be fully separated – It is veryimportant to separate test development from test automation. The twodisciplines require very different skills. Fundamentally, testers are not andshould not be programmers. Testers must be adept at defining test casesindependent of the underlying technology to implement them. Individuals whoare skilled technically, the "automation people" (automation engineers), willimplement the action words and then test them.

• Test cases must have a clear and differentiated scope – It is important thattest cases have a clearly differentiated scope and that they not deviate fromthat scope.

• The tests must be written at the right level of abstraction – Tests must bewritten at the right level of abstraction such as the higher business level,lower user interface level, or both. It is also important that test tools providethis level of flexibility.







The Framework

The implementation of this methodology is framework dependent. This framework

requires the development of data tables and keywords, independent of the testautomation tool used to execute them and the test script code that "drives" theapplication-under-test and the data. In a keyword-driven test, the functionality of the application-under-test is documented in a table as well as in step-by-stepinstructions for each test.

If we were to map out the actions we perform with the mouse when we test ourWindows Calculator functions by hand, we could create the following table. The"Window" column contains the name of the application window where we'reperforming the mouse action .

Methodology

The keyword-driven testing methodology divides test creation into two stages:

• Planning Stage

Analyzing the application and determining which objects and operations areused by the set of business processes that need to be tested. Determining which operations require customized keywords to provideadditional functionality, to achieve business-level clarity, and/or to maximizeefficiency and maintainability.

• Implementation Stage

Building a collection of references that uniquely identify objects, sometimesknown as an "object repository", and ensuring that all such references haveclear names that follow any predetermined naming conventions. (This isprimarily for test automation.)Developing and documenting business-level keywords in function libraries.Creating function libraries involves developing customized functions for theapplication that needs to be tested.

Although this methodology requires more planning and a longer initial time-investment than going directly to the test creation stage and recording your steps,this methodology makes the test creation and test maintenance stages more efficient

and keeps the structure of individual tests more readable and easier to modify.







Vision for Automation

It is also important to have a clear vision for automation. Such a "vision" shouldinclude things such as:

• Having a good methodology – It is important to have a good integratedmethodology for testing and automation that places testers in the driver’sseat. It is also important to employ the best technology that supports themethodology, maximizes flexibility, minimizes technical efforts, andmaximizes maintainability.

• Have the right tool – Any tool that is employed should be specifically designedfor keyword based testing. It should be flexible enough to allow for the rightmix of high and low level testing. It should allow the testers to quickly buildkeyword tests, without difficulty. It should also not be overly complicated forautomation engineers to use when implementing the automation.

• Succeed in the three "success factors for automation" – There are threecritical success factors for automation that the vision should account for. Theyare:

1. test design2. automation solution3. organization







Success Factors for Automation

Test Design

Test design is more important than the automation technology. Design is the mostunderestimated part of testing. It is my belief that test design, not automation or atool, is the single most important factor for automation success. To understand moreabout test design see these previous articles:

• Key Principles of Test Design• The First Holy Grail of Test Design• The Second Holy Grail of Test Design• The Third Holy Grail of Test Design

Comprehensive Automation Architecture

• Scope, assumptions, risks• Methods, best practices• Tools, technologies, architecture• Stake holders, including roles and processes for input and approvals• …and more…

The "right" team must also be assembled. This team should include:

• Test management who is responsible for managing the test process.• Test development who is responsible for production of tests. Test

development should include test leads, test developers, end users, subjectmatter experts, and business analysts.

• Automation engineering who is responsible for creating the automationscheme for automatic execution. Members of this team include a leadengineer as well as one or more automation support engineers.

• Support functions, providing methods, techniques, know how, training, tools,and environments.

For the team there should be a clear division of tasks and responsibilities as well aswell defined processes for decision making and communication.

How to Measure Success

With any major undertaking, it is important to define and measure "success". Thereare two important areas of measurement for success – progress and quality.







Progress

You should measure test development against the test development plan. If goalsare not reached, act quickly to find the problems. Is the subject matter clear? Arestake holders providing enough input? Is it clear what to test (overall, per module)?Is the team right (enough, right skill set mix)?

You should measure automation and look at things such as implemented keywords(actions) and interface definitions (defined interface dialogs, pages, etc).

You should measure test execution looking at things such as how many modules areexecuted and how many executed correctly (without errors)?

Quality

Some of the key quality metrics include:

• Coverage of system and requirements• Assessments by peers, test leads, and by stake holders (recommended)• Effectiveness

o Are you finding bugs?o Are you missing bugs?o Can you find known bugs (or seeded bugs)?o After the system is released, what bugs still come up? You should

consider calculating the "Defect Detection Percentage" (DorothyGraham, Mark Fewster)

• Mine your bug base for additional insights

Conclusion

It is important to understand that keywords are not magic, but they can serve youwell. What is more important is to take the effort seriously and "do it right". Doing itright means that test design is essential, both global test design and the design of individual test cases. Automation should be done but it should not dominate theprocess. Automation should flow from the overall strategy, methodology, andarchitecture. It is also very important to pay attention to organization – the process,team, and project environment.







Data-driven Testing

Automated tests play back a recorded (or programmed) sequence of user actionsthat cover a certain area of the tested application. To get larger coverage, you canperform tests with different input data. Suppose, for example, you test an input formof an application. After you have recorded a script, it contains only values that youentered during the recording. Most likely, data you have specified does not causeerrors in the application, but other data may cause them. Data-driven testing allowsyou to test the form with a different set of input values to be sure that theapplication works as expected.

Contents:

Introduction Data Driven Automation Frameworks Data Driven Scripts TestComplete's Support for Data-Driven Testing







Verifying results

The easiest way to check whether your test actions were performed successfully is tostore the check value along with the set of input data and then compare the outputwith these values. If the output is an image or a file, you can use TestComplete’sfeatures to compare the images or the files' and object properties.







Data Driven Automation Frameworks

Over the past several years there have been numerous articles done on various

approaches to test automation. Anyone who has read a fair, unbiased sampling of these knows that we cannot and must not expect pure capture and replay of testscripts to be successful for the life of a product. We will find nothing but frustrationthere.

Sometimes this manifesto is hard to explain to people who have not yet performedsignificant test automation with these capture\replay tools. But it usually takes lessthan a week, often less than a day, to hear the most repeated phrase: "It workedwhen I recorded it, but now it fails when I play it back!"

Data Driven Scripts

Data driven scripts are those application-specific scripts captured or manually codedin the automation tool’s proprietary language and then modified to accommodatevariable data. Variables will be used for key application input fields and programselections allowing the script to drive the application with external data supplied bythe calling routine or the shell that invoked the test script.

Variable Data, Hard Coded Component Identification

These data driven scripts often still contain the hard coded and sometimes veryfragile recognition strings for the window components they navigate. When this isthe case, the scripts are easily broken when an application change or revision occurs.And when these scripts start breaking, we are not necessarily talking about just a

few. We are sometimes talking about a great many, if not all the scripts, for theentire application.

Figure 1 is an example of activating a server-side image map link in a webapplication with an automation tool scripting language:

This particular scenario of clicking on the image map might exist thousands of timesthroughout all the scripts that test this application. The preceding example identifiesthe image by the title given to the document and the index of the image on thepage. The hard coded image identification might work successfully all the waythrough the production release of that version of the application. Consequently,testers responsible for the automated test scripts may gain a false sense of security

and satisfaction with these results.







However, the next release cycle may find some or all of these scripts broken becauseeither the title of the document or the index of the image has changed. Sometimes,with the right tools, this might not be too hard to fix. Sometimes, no matter whattools, it will be frustratingly difficult.

Remember, we are potentially talking about thousands of broken lines of test scriptcode. And this is just one particular change. Where there is one, there will likely beothers.

Highly Technical or Duplicate Test Designs

Another common feature of data driven scripts is that virtually all of the test designeffort for the application is developed in the scripting language of the automationtool. Either that, or it is duplicated in both manual and automated script versions.This means that everyone involved with automated test development or automatedtest execution for the application must likely become proficient in the environmentand programming language of the automation tool.

Findings: A test automation framework relying on data driven scripts is definitely the easiestand quickest to implement if you have and keep the technical staff to maintain it. Butit is the hardest of the data driven approaches to maintain and perpetuate and veryoften leads to long-term failure.







TestComplete's Support for Data-Driven Testing

Since DDT itself is very straightforward, TestComplete’s direct support for the

technique is likewise quite straightforward. If you examine the TestComplete onlinehelp, under “data-driven testing”, you’ll find a fairly long help topic with links toexternal pages on the subject, but also links to five script routines. It’s worth notingthat while these methods are listed with the DDT topics, they’re not the only routinesyou’ll need to perform DDT, but for now we’ll look at those five. They areGetCSVCount, GetCSVItem, CallMethod, SetValue and GetValue. The first twomethods are ones that you’ll use quite a bit in DDT, the other three will see lessfrequent use unless you are getting into some fairly robust DDT scripting.

GetCSVCount and GetCSVItem work with comma-separated strings in order toprovide an easy way to get to the values contained within the strings. This is called

“parsing”. A comma-separated value (CSV) string might look something like this:

02,c,”John”,”Q”,”Public”,13,”Product 02”,”1234 Anywhere St.”,”San Antonio”,”TX”

You can see that each element of this string is separated by a comma. Eachindividual element of the string is a called a “field” or a “value”. The GetCSVCountand GetCSVItem methods work with strings like this. If we passed this string in tothe GetCSVCount function, we’d get a return value of 10. There are ten discreetvalues in this CSV string. In addition, if we wanted to the get the customer’s firstname, we could use GetCSVItem to do that.

Here’s an important thing to keep in mind: GetCSVItem is what’s known as “zero-based”. That means that the first item in the CSV string is considered number 0rather than number 1. Likewise, if we wanted the last item, it would be item number9, not number 10. Thus, if we wanted the customer’s first name, we’d have to askfor item number 2, not number 3, thus:

varCustFirstName = GetCSVItem(‘02,c,”John”,”Q”,”Public”,13,”Product 02”,”1234Anywhere St.”,”San Antonio”,”TX”’,

So, with these two functions, we’re able to easily retrieve values from CSV files,something that is quite useful if our external DDT data is in CSV text files orspreadsheets. Returning again to our customer/orders example, we might see thefollowing. (This is pseudo-code only.)







function ReadExternalData(aFileName){

var lsLinevar lsCustomerName, lsAddr, lsCity, lsRegion, lsPostalCode

openFile(aFileName)while not EOF(aFileName){

lsLine = readLine(aFileName)if (GetCSVCount(lsLine) == 5{

lsCustomerName = GetCSVItem(lsLine, 0)lsAddr = GetCSVItem(lsLine, 1)lsCity = GetCSVItem(lsLine, 2)lsRegion = GetCSVItem(lsLine, 3)lsPostalCode = GetCSVItem(lsLine, 4)

EnterCustomerRecord(lsCustomerName, lsAddr, lsCity, lsRegion, lsPostalCode)}

}closeFile(aFileName)

}function EnterCustomerRecord(aCustomerName, aAddr, aCity, aRegion,aPostalCode){

var p, wp = Sys.OrdersProcessw = p.MainFormw.Activate()w.ToolButton5.Click(6, 7)

w = p.OrderFrmw.Activate()w.CustomerName_Edit.wText = aCustomerNamew.Street_Edit.wText = aAddrw.City_Edit.wText = aCityw.State_Edit.wText = aRegionw.Zip_Edit.wText = aPostalCodew.OKButton.Click()validateRecordEntry()

}

Don’t worry about trying to make this code actually work – it’s rather abstracted andis for demonstration only, but your actual DDT code might be similar. What this smallamount of code does is very powerful and serves as a great example of a standardDDT scenario.







As for the other three methods covered in the help (CallMethod, SetValue andGetValue) I won’t talk about those now – they’re not necessary for this primer onDDT.

The part of this whole process that’s not clearly covered is that of actually readingthe external data. That is in part because there are so many options. TestCompletecan read your DDT data from text files, spreadsheets, database tables or frominternal data arrays. To access your data, you can use ADO, OLE automation orstandard text file access methods. Many of these storage options and accessmethods are shown in the DDT demo project included with TestComplete, which we’llcover shortly. For now, just plan on getting to your data whichever way is mostcomfortable.







For more Software Testing Resources, please visit

http://www.OneStopTesting.com

Join largest Software Testing Community at

http://groups.yahoo.com/group/OneStopTesting/

Over 5,000 Software Testing Interview Questions athttp://www.CoolInterview.com

and

http://www.TestingInterviewQuestions.com

and

http://www.NewInterviewQuestions.com

Automation Framework eBook

Documents

Transcript of Automation Framework eBook