Non-GUI Test Automation - RBCS, Inc · Non-GUI Test Automation Concepts and Case Studies in...

12
Non-GUI Test Automation Concepts and Case Studies in Maintainable Testing

Transcript of Non-GUI Test Automation - RBCS, Inc · Non-GUI Test Automation Concepts and Case Studies in...

Page 1: Non-GUI Test Automation - RBCS, Inc · Non-GUI Test Automation Concepts and Case Studies in Maintainable Testing

Non-GUI Test Automation Concepts and Case Studies in Maintainable Testing

Page 2: Non-GUI Test Automation - RBCS, Inc · Non-GUI Test Automation Concepts and Case Studies in Maintainable Testing

Non-GUI Test Automation

www.rbcs-us.com

Copyright (c) 2017-2018 RBCS Page 2

Introduction

Test automation is great, except when it’s not

Various reasons lead to test automation failure, some arising from automation via a GUI

You can run automated tests through other interfaces

Command lines

APIs

Data layers

Network services

We’ll look at two examples of sophisticated automated test platforms

One used the data layer

The other used the network services layer

Each using a flexible and maintainable keyword driven architecture. GUI focused test

automation: a wrong

turn whose time has

come and gone

Page 3: Non-GUI Test Automation - RBCS, Inc · Non-GUI Test Automation Concepts and Case Studies in Maintainable Testing

Test Automation Zombie Apocalypse

False positives False negatives Flaky tests Unreliable or incomplete test oracles Changing GUIs that break tests Disproportionate test maintenance costs Commercial tool licensing costs

Non-GUI Test Automation

www.rbcs-us.com

Copyright (c) 2017-2018 RBCS Page 3

Test automation failure is scary. Not this scary, maybe (Film: Dawn of the Dead. GIF: giphy.com)

Page 4: Non-GUI Test Automation - RBCS, Inc · Non-GUI Test Automation Concepts and Case Studies in Maintainable Testing

Test Automation Implementation

Best practices involve modularized automation, especially using keyword-driven design

Keywords usually are high-level business interactions with a system, covering a number of detailed interactions

Business-oriented testers can write tests in human language (albeit concrete tests) rather than using the tool’s programming language

This greatly improves maintainability over time as well as usability of the framework by the testers

Business process models, use cases, and other models can be used to create the keywords

The modelling can be manual or supported by tools

Non-GUI Test Automation Page 4

www.rbcs-us.com

Copyright (c) 2017-2018 RBCS

Page 5: Non-GUI Test Automation - RBCS, Inc · Non-GUI Test Automation Concepts and Case Studies in Maintainable Testing

Building a Keyword System

Technical testers/SDETs create a complex system which is designed to be easy to use

Typical components include Intelligent front-end (e.g., drop-down controls based on context)

Each keyword represents programmed functionality

Execution virtual machine • Executes keywords one-by-one

• Built in logging, error handling

Test suite execution selector

Long-term success requires a method for business-oriented testers to request more keywords

Non-GUI Test Automation Page 5

www.rbcs-us.com

Copyright (c) 2017-2018 RBCS

Page 6: Non-GUI Test Automation - RBCS, Inc · Non-GUI Test Automation Concepts and Case Studies in Maintainable Testing

Keyword-driven Automated Tests

Test cases built from sequences of keywords, instructions, test data, and expected results Advantages:

Keywords defined by business experts Business-focused testers automate tests Tests are maintainable Test specifications are independent of implementation

Technical testers/SDETs implement keywords via executable test scripts The creation of the test scripts must be integrated into the lifecycle These scripts cause the tools to read and run the test cases Business-focused testers execute the tests and analyze anomalies False positive rates can be higher, and must be isolated through manually repeating the test Sometimes, technical testers must help to isolate and resolve false positives

Non-GUI Test Automation

www.rbcs-us.com

Copyright (c) 2017-2018 RBCS Page 6

Page 7: Non-GUI Test Automation - RBCS, Inc · Non-GUI Test Automation Concepts and Case Studies in Maintainable Testing

Case Study 1: Environment Size, Diversity

System under test manages highly diverse systems

Agents running on each system interfaced with central workstation

Central workstation allows administration of accounts on all the systems with agents on the network

Non-GUI Test Automation

www.rbcs-us.com

Copyright (c) 2017-2018 RBCS Page 7

Page 8: Non-GUI Test Automation - RBCS, Inc · Non-GUI Test Automation Concepts and Case Studies in Maintainable Testing

Case Study 1: Tool Architecture

Non-GUI Test Automation

www.rbcs-us.com

Copyright (c) 2017-2018 RBCS Page 8

Page 9: Non-GUI Test Automation - RBCS, Inc · Non-GUI Test Automation Concepts and Case Studies in Maintainable Testing

Case Study 2: Environment, Data Diversity

System under test provides data analytics and extract capabilities from a diverse set of data repositories Provides command line interface (CLI) access to the data layer through English-like non-procedural commands CLI is highly stable and accessible via open-source scripting languages

Non-GUI Test Automation?

www.rbcs-us.com

Copyright (c) 2017-2018 RBCS Page 9

Page 10: Non-GUI Test Automation - RBCS, Inc · Non-GUI Test Automation Concepts and Case Studies in Maintainable Testing

Case Study 2: Data Analytics Tool

Scripts make the whole thing work Expected result location can vary, so search is DB-specific, then OS-specific, then generic Smart comparator (written in C for speed) allowed masking of changeable fields and record-order variations

Non-GUI Test Automation

www.rbcs-us.com

Copyright (c) 2017-2018 RBCS Page 10

Page 11: Non-GUI Test Automation - RBCS, Inc · Non-GUI Test Automation Concepts and Case Studies in Maintainable Testing

Conclusion

Test automation efforts sometimes come to grief

Trying to automate tests through the GUI has proven particularly challenging

Keyword-driven test automation is a best practice, but it’s not just for GUI testing

Keyword-driven test automation can be done at service layers and data layers, often using open-source tools

Time to defenestrate your automated tests!

Non-GUI Test Automation

www.rbcs-us.com

Copyright (c) 2017-2018 RBCS Page 11

Looking to sunset your GUI-based

automation? Maybe you should be…

Page 12: Non-GUI Test Automation - RBCS, Inc · Non-GUI Test Automation Concepts and Case Studies in Maintainable Testing

Non-GUI Test Automation

www.rbcs-us.com

Copyright (c) 2017-2018 RBCS Page 12

For over 20 years, RBCS has delivered consulting, training, and expert services to clients, helping them with software and hardware testing. Employing the industry’s most experienced and recognized consultants, RBCS advises its clients, trains their employees, conducts product testing, builds and improves testing groups, and hires testing staff for hundreds of clients worldwide. Ranging from Fortune 20 companies to start-ups, RBCS clients save time and money through improved product development, decreased tech support calls, improved corporate reputation and more. To learn more about RBCS, visit www.rbcs-us.com.

Address: RBCS, Inc. 31520 Beck Road Bulverde, TX 78163-3911 USA Phone: +1 (830) 438-4830 E-mail: [email protected] Web: www.rbcs-us.com Twitter: @RBCS, @MisterSDET, @LaikaTestDog Facebook: @TestingImprovedbyRBCS LinkedIn: https://www.linkedin.com/in/rex-black YouTube: https://www.youtube.com/user/RBCSINC

…Contact RBCS