Non-GUI Test Automation - RBCS, Inc · Non-GUI Test Automation Concepts and Case Studies in...
Transcript of Non-GUI Test Automation - RBCS, Inc · Non-GUI Test Automation Concepts and Case Studies in...
Non-GUI Test Automation Concepts and Case Studies in Maintainable Testing
Non-GUI Test Automation
www.rbcs-us.com
Copyright (c) 2017-2018 RBCS Page 2
Introduction
Test automation is great, except when it’s not
Various reasons lead to test automation failure, some arising from automation via a GUI
You can run automated tests through other interfaces
Command lines
APIs
Data layers
Network services
We’ll look at two examples of sophisticated automated test platforms
One used the data layer
The other used the network services layer
Each using a flexible and maintainable keyword driven architecture. GUI focused test
automation: a wrong
turn whose time has
come and gone
Test Automation Zombie Apocalypse
False positives False negatives Flaky tests Unreliable or incomplete test oracles Changing GUIs that break tests Disproportionate test maintenance costs Commercial tool licensing costs
Non-GUI Test Automation
www.rbcs-us.com
Copyright (c) 2017-2018 RBCS Page 3
Test automation failure is scary. Not this scary, maybe (Film: Dawn of the Dead. GIF: giphy.com)
Test Automation Implementation
Best practices involve modularized automation, especially using keyword-driven design
Keywords usually are high-level business interactions with a system, covering a number of detailed interactions
Business-oriented testers can write tests in human language (albeit concrete tests) rather than using the tool’s programming language
This greatly improves maintainability over time as well as usability of the framework by the testers
Business process models, use cases, and other models can be used to create the keywords
The modelling can be manual or supported by tools
Non-GUI Test Automation Page 4
www.rbcs-us.com
Copyright (c) 2017-2018 RBCS
Building a Keyword System
Technical testers/SDETs create a complex system which is designed to be easy to use
Typical components include Intelligent front-end (e.g., drop-down controls based on context)
Each keyword represents programmed functionality
Execution virtual machine • Executes keywords one-by-one
• Built in logging, error handling
Test suite execution selector
Long-term success requires a method for business-oriented testers to request more keywords
Non-GUI Test Automation Page 5
www.rbcs-us.com
Copyright (c) 2017-2018 RBCS
Keyword-driven Automated Tests
Test cases built from sequences of keywords, instructions, test data, and expected results Advantages:
Keywords defined by business experts Business-focused testers automate tests Tests are maintainable Test specifications are independent of implementation
Technical testers/SDETs implement keywords via executable test scripts The creation of the test scripts must be integrated into the lifecycle These scripts cause the tools to read and run the test cases Business-focused testers execute the tests and analyze anomalies False positive rates can be higher, and must be isolated through manually repeating the test Sometimes, technical testers must help to isolate and resolve false positives
Non-GUI Test Automation
www.rbcs-us.com
Copyright (c) 2017-2018 RBCS Page 6
Case Study 1: Environment Size, Diversity
System under test manages highly diverse systems
Agents running on each system interfaced with central workstation
Central workstation allows administration of accounts on all the systems with agents on the network
Non-GUI Test Automation
www.rbcs-us.com
Copyright (c) 2017-2018 RBCS Page 7
Case Study 1: Tool Architecture
Non-GUI Test Automation
www.rbcs-us.com
Copyright (c) 2017-2018 RBCS Page 8
Case Study 2: Environment, Data Diversity
System under test provides data analytics and extract capabilities from a diverse set of data repositories Provides command line interface (CLI) access to the data layer through English-like non-procedural commands CLI is highly stable and accessible via open-source scripting languages
Non-GUI Test Automation?
www.rbcs-us.com
Copyright (c) 2017-2018 RBCS Page 9
Case Study 2: Data Analytics Tool
Scripts make the whole thing work Expected result location can vary, so search is DB-specific, then OS-specific, then generic Smart comparator (written in C for speed) allowed masking of changeable fields and record-order variations
Non-GUI Test Automation
www.rbcs-us.com
Copyright (c) 2017-2018 RBCS Page 10
Conclusion
Test automation efforts sometimes come to grief
Trying to automate tests through the GUI has proven particularly challenging
Keyword-driven test automation is a best practice, but it’s not just for GUI testing
Keyword-driven test automation can be done at service layers and data layers, often using open-source tools
Time to defenestrate your automated tests!
Non-GUI Test Automation
www.rbcs-us.com
Copyright (c) 2017-2018 RBCS Page 11
Looking to sunset your GUI-based
automation? Maybe you should be…
Non-GUI Test Automation
www.rbcs-us.com
Copyright (c) 2017-2018 RBCS Page 12
For over 20 years, RBCS has delivered consulting, training, and expert services to clients, helping them with software and hardware testing. Employing the industry’s most experienced and recognized consultants, RBCS advises its clients, trains their employees, conducts product testing, builds and improves testing groups, and hires testing staff for hundreds of clients worldwide. Ranging from Fortune 20 companies to start-ups, RBCS clients save time and money through improved product development, decreased tech support calls, improved corporate reputation and more. To learn more about RBCS, visit www.rbcs-us.com.
Address: RBCS, Inc. 31520 Beck Road Bulverde, TX 78163-3911 USA Phone: +1 (830) 438-4830 E-mail: [email protected] Web: www.rbcs-us.com Twitter: @RBCS, @MisterSDET, @LaikaTestDog Facebook: @TestingImprovedbyRBCS LinkedIn: https://www.linkedin.com/in/rex-black YouTube: https://www.youtube.com/user/RBCSINC
…Contact RBCS