WSO2Con Asia 2014 - Effective Test Automation in an Agile Environment
-
Upload
wso2 -
Category
Technology
-
view
372 -
download
2
description
Transcript of WSO2Con Asia 2014 - Effective Test Automation in an Agile Environment
![Page 1: WSO2Con Asia 2014 - Effective Test Automation in an Agile Environment](https://reader034.fdocuments.net/reader034/viewer/2022051411/547997e8b37959892b8b47c8/html5/thumbnails/1.jpg)
Effective Test Automationin an Agile Environment
Krishantha Samaraweera
Senior Technical Lead – Test Automation
WSO2
![Page 2: WSO2Con Asia 2014 - Effective Test Automation in an Agile Environment](https://reader034.fdocuments.net/reader034/viewer/2022051411/547997e8b37959892b8b47c8/html5/thumbnails/2.jpg)
Agenda
● Automated testing● Success stories● Automation Concept – Test Pyramid● WSO2 Test Automation Framework● WSO2 Platform Automated Test Suite● Lessons learned ● Automation Best Practices● Continuous Integration and Delivery
![Page 3: WSO2Con Asia 2014 - Effective Test Automation in an Agile Environment](https://reader034.fdocuments.net/reader034/viewer/2022051411/547997e8b37959892b8b47c8/html5/thumbnails/3.jpg)
What is Test Automation
● Automation testing – Testing which can be done programmatically● Compare result – Actual and Expected● Generate reports● Goal is to reduce the number of test cases to run manually● Not to eliminate Manual testing all together
![Page 4: WSO2Con Asia 2014 - Effective Test Automation in an Agile Environment](https://reader034.fdocuments.net/reader034/viewer/2022051411/547997e8b37959892b8b47c8/html5/thumbnails/4.jpg)
Why Automated Testing
● To get early and instant feedback● Safety net to save time and which is reinvented in to manual
testing. ● Doesn't require human intervention.● Unattended (Overnight)● Manual testing has high risk of mission out on something
![Page 5: WSO2Con Asia 2014 - Effective Test Automation in an Agile Environment](https://reader034.fdocuments.net/reader034/viewer/2022051411/547997e8b37959892b8b47c8/html5/thumbnails/5.jpg)
Success Stories
● Google runs 100M+ automated test cases every day● Google customer surveys team can deliver a change to
production 8 mins after code is committed.● Ebay runs 35000 test cases in 5 hours● At Facebook, each of 5,000 engineers commits to trunk HEAD
at least once a day and the code at trunk HEAD is pushed to production once daily.
● Atlassian JIRA is protected by over 14,000 unit tests and 19,000 functional tests from regressions
![Page 6: WSO2Con Asia 2014 - Effective Test Automation in an Agile Environment](https://reader034.fdocuments.net/reader034/viewer/2022051411/547997e8b37959892b8b47c8/html5/thumbnails/6.jpg)
Success Stories Cont'd..
● Etsy used to take 14 hours and an army of people to deploy one release into production before CD kicked in. Now it takes 15 mins, 1 person. They did 729 deployments to production in Nov 2010, only 6 deployment related incidents.
● At Amazon, new code is deployed to production at a staggering rate of once every 11.6 seconds during a normal business day.
● That’s 3,000 production deployments per day.
![Page 7: WSO2Con Asia 2014 - Effective Test Automation in an Agile Environment](https://reader034.fdocuments.net/reader034/viewer/2022051411/547997e8b37959892b8b47c8/html5/thumbnails/7.jpg)
Test Automation Pyramid
![Page 8: WSO2Con Asia 2014 - Effective Test Automation in an Agile Environment](https://reader034.fdocuments.net/reader034/viewer/2022051411/547997e8b37959892b8b47c8/html5/thumbnails/8.jpg)
Test Automation Pyramid Cont'd..
![Page 9: WSO2Con Asia 2014 - Effective Test Automation in an Agile Environment](https://reader034.fdocuments.net/reader034/viewer/2022051411/547997e8b37959892b8b47c8/html5/thumbnails/9.jpg)
WSO2 Test Automation Framework ● Why our own framework
– Open source test scripts
– Stay lean
– Single framework for all products
– Facilitate backend testing + UI tests
![Page 10: WSO2Con Asia 2014 - Effective Test Automation in an Agile Environment](https://reader034.fdocuments.net/reader034/viewer/2022051411/547997e8b37959892b8b47c8/html5/thumbnails/10.jpg)
Technology outlineTechnology outline
![Page 11: WSO2Con Asia 2014 - Effective Test Automation in an Agile Environment](https://reader034.fdocuments.net/reader034/viewer/2022051411/547997e8b37959892b8b47c8/html5/thumbnails/11.jpg)
Test Automation Framework Architecture
![Page 12: WSO2Con Asia 2014 - Effective Test Automation in an Agile Environment](https://reader034.fdocuments.net/reader034/viewer/2022051411/547997e8b37959892b8b47c8/html5/thumbnails/12.jpg)
Glimpse of Test Automation Framework
● Simple automation context API.● Platform wide test execution support.● Annotation based test execution management support.● Test reports and coverage report generation.● Test tooling integration (Jmeter).● Selenium WebDriver integration and ability to run UI tests on
cross browser environments.● Framework extensibility capability through pluggable modules.● Easy retrieval of new configurations.
![Page 13: WSO2Con Asia 2014 - Effective Test Automation in an Agile Environment](https://reader034.fdocuments.net/reader034/viewer/2022051411/547997e8b37959892b8b47c8/html5/thumbnails/13.jpg)
WSO2 Platform Automated Test Suite
● Package tests for other people to use● Distribution contain all product integration tests and platform
tests jars.● Test script management tool● Ant based test executor● Test execution on distributed product clustered or cloud
deployment.● Cross platform test execution
![Page 14: WSO2Con Asia 2014 - Effective Test Automation in an Agile Environment](https://reader034.fdocuments.net/reader034/viewer/2022051411/547997e8b37959892b8b47c8/html5/thumbnails/14.jpg)
Lesson Learned
● Once you abandon the maintenance of the automation, it is likely to die. For a better chance of success, choose an automation approach that will require the least maintenance.
● Automation requiring multiple cycles and project releases for it to become fully effective and provide an acceptable ROI.
● Start by automating the most valuable tests.
![Page 15: WSO2Con Asia 2014 - Effective Test Automation in an Agile Environment](https://reader034.fdocuments.net/reader034/viewer/2022051411/547997e8b37959892b8b47c8/html5/thumbnails/15.jpg)
Lesson Learned Cont'd
● Plan your test automation at the beginning of the project.
● Unit Testing FIRST – Fix your test Pyramid.
● Staff should know how to diagnose failures. Minimize calling
automators for every issue
![Page 16: WSO2Con Asia 2014 - Effective Test Automation in an Agile Environment](https://reader034.fdocuments.net/reader034/viewer/2022051411/547997e8b37959892b8b47c8/html5/thumbnails/16.jpg)
Lesson Learned Cont'd..
● Automation development requires the same discipline as software development.
● Organization wide commitment is required for successful test automation methodology
● Everyone should know what is automated.
![Page 17: WSO2Con Asia 2014 - Effective Test Automation in an Agile Environment](https://reader034.fdocuments.net/reader034/viewer/2022051411/547997e8b37959892b8b47c8/html5/thumbnails/17.jpg)
Test Automation Benefits – Efficiency ROI
Time savings of test execution with automation vs test execution without automation
Product # of Tests Manual time per test case
Iterations Manual test execution time
Automated execution test
time
Saved time
ESB 922 8 min 5 384 hours 10 hours 77 man days
G-Reg 1557 8 min 5 1038 hours 15 hours 130 man days
![Page 18: WSO2Con Asia 2014 - Effective Test Automation in an Agile Environment](https://reader034.fdocuments.net/reader034/viewer/2022051411/547997e8b37959892b8b47c8/html5/thumbnails/18.jpg)
Continuous Integration
● Continuous Integration is a software development practice● Team integrate their work frequently● Each commit is verified by automated build● Rapid feedback● Ships better code faster● Fixing build breaks ASAP important. ● Should have a notification mechanism
![Page 19: WSO2Con Asia 2014 - Effective Test Automation in an Agile Environment](https://reader034.fdocuments.net/reader034/viewer/2022051411/547997e8b37959892b8b47c8/html5/thumbnails/19.jpg)
Continuous Integration Cont'd..
Source : http://mikeciblogs.wordpress.com/2010/06/15/continuous-integration-for-agile-project-managers-part-3/
![Page 20: WSO2Con Asia 2014 - Effective Test Automation in an Agile Environment](https://reader034.fdocuments.net/reader034/viewer/2022051411/547997e8b37959892b8b47c8/html5/thumbnails/20.jpg)
Continuous Delivery
How long would it take your organization to deploy a change [to production] that involves just one single line of code? Do you do this on a repeatable, reliable basis?”
- Mary and Tom Poppendieck,
Implementing Lean Software Development
“
![Page 21: WSO2Con Asia 2014 - Effective Test Automation in an Agile Environment](https://reader034.fdocuments.net/reader034/viewer/2022051411/547997e8b37959892b8b47c8/html5/thumbnails/21.jpg)
Continuous Delivery
Build software in such a way that it ready to be released at any time– Continuous feedback
– Reduce risk of failures in production system
– Rapid, reliable and repeatable automated processes
![Page 22: WSO2Con Asia 2014 - Effective Test Automation in an Agile Environment](https://reader034.fdocuments.net/reader034/viewer/2022051411/547997e8b37959892b8b47c8/html5/thumbnails/22.jpg)
Test Cases to be Automated
● HIGH RISK – Business critical test cases.● Features tend to break frequently● Repetitive tasks are good candidates for automation● Test cases that are tedious or difficult to perform manually● Test cases which are time consuming. ● Tests that use multiple data values for the same actions● Smoke Tests
![Page 23: WSO2Con Asia 2014 - Effective Test Automation in an Agile Environment](https://reader034.fdocuments.net/reader034/viewer/2022051411/547997e8b37959892b8b47c8/html5/thumbnails/23.jpg)
Test Cases not suitable for Automation
● Test cases that are newly designed – Not executed manually at least once.
● Test cases for which the requirements are changing frequently.● Usability testing – “How easy is the application to use?”● One-time testing
![Page 24: WSO2Con Asia 2014 - Effective Test Automation in an Agile Environment](https://reader034.fdocuments.net/reader034/viewer/2022051411/547997e8b37959892b8b47c8/html5/thumbnails/24.jpg)
Testing Philosophy
● You don't have to write 1,000 tests● The benefit of the tests scales very nicely.
– 1,000 tests is good
– 100 tests is also good
– 10 tests is also good
● And even one test is better than no tests at all
![Page 25: WSO2Con Asia 2014 - Effective Test Automation in an Agile Environment](https://reader034.fdocuments.net/reader034/viewer/2022051411/547997e8b37959892b8b47c8/html5/thumbnails/25.jpg)
Testing Philosophy
● So don't start out by saying "Oh, I have to write 1,000 tests for this."
● Then you might not write any at all.● Start by saying "I'll write one test".● Then write it● Then maybe write another one● You will immediately see the benefit of having one test
![Page 26: WSO2Con Asia 2014 - Effective Test Automation in an Agile Environment](https://reader034.fdocuments.net/reader034/viewer/2022051411/547997e8b37959892b8b47c8/html5/thumbnails/26.jpg)
Conclusion
● Fix your test pyramid right● Test maintenance is the king● Run you tests frequently● Importance players of successful test automation are right
selection of tools, testing process and team. ● Manual testing is also important
![Page 27: WSO2Con Asia 2014 - Effective Test Automation in an Agile Environment](https://reader034.fdocuments.net/reader034/viewer/2022051411/547997e8b37959892b8b47c8/html5/thumbnails/27.jpg)
References[1]“GTAC 2013: Presentations - Google Test Automation Conference — Google Developers.” [Online]. Available: https://developers.google.com/google-test-automation-conference/2013/presentations. [Accessed: 17-Mar-2014].
[2]“Continuous Integration for Agile Project Managers (Part 3) | Mike CI blogs.” [Online]. Available: http://mikeciblogs.wordpress.com/2010/06/15/continuous-integration-for-agile-project-managers-part-3/. [Accessed: 17-Mar-2014].
[3]“All About Automated Testing.” [Online]. Available: http://www.guru99.com/automation-testing.html. [Accessed: 17-Mar-2014].
[4]“Writing Automated Tests.” [Online]. Available: http://perl.plover.com/yak/testing/samples/. [Accessed: 17-Mar-2014].
[5]“Continuous Integration.” [Online]. Available: http://martinfowler.com/articles/continuousIntegration.html. [Accessed: 17-Mar-2014].
![Page 28: WSO2Con Asia 2014 - Effective Test Automation in an Agile Environment](https://reader034.fdocuments.net/reader034/viewer/2022051411/547997e8b37959892b8b47c8/html5/thumbnails/28.jpg)
Thank You