Building an effective test automation suite
As we grow an automated acceptance test suite which we can rely upon will become a valuable resource.
Building an effective test automation suite
The process of building an effective and sustainable test automation suite involves following a simple five step process.
Identify
Describe
Implement
Run
Integrate
Building an effective test automation suite
Identify test scenarios to include in our suite.
Identify
Describe
Implement
Run
Integrate
Building an effective test automation suite
Describe the test scenarios in a way that is easily understood by all product stakeholders.
Identify
Describe
Implement
Run
Integrate
Building an effective test automation suite
Implement the test steps using our chosen testing frameworks.
Identify
Describe
Implement
Run
Integrate
Building an effective test automation suite
Run our tests on all supported product development environments.
Identify
Describe
Implement
Run
Integrate
Building an effective test automation suite
Integrate our completed tests into our Continuous Integration and Testing system.
Identify
Describe
Implement
Run
Integrate
Building an effective test automation suite
Test Identification should take place as an ongoing process throughout the product development lifecycle.
Identify
Building an effective test automation suite
Test Identification should take place as an ongoing process throughout the product development lifecycle.
Identify Identify tests during Story creation. Stories should be described in Gherkin/BDD
style to develop an ubiquitous language and easily testable deliverables. Identify tests during Acceptance Criteria meetings. After reviewing the initial
requirements it may become clear that edge cases can also be defined. Identify tests during development. As the deliverables begin to become fleshed
out, additional test cases may become apparent, and should be defined. Identify tests during testing. When the testing process begins, more test
scenarios may be discovered, and should be defined and documented. Identify tests when errors appear. Errors/Bugs in any environment, including our
live environment, should result in a bug report and tests to prove when its fixed. Identify tests at any other time. Whenever a valuable test case is identified then
it should be documented and added to the test automation backlog.
Building an effective test automation suite
We describe our product requirements using the Gherkin syntax – Given, When, Then.
Describe Identify tests during Story creation. Stories should be described in Gherkin/BDD
style to develop an ubiquitous language and easily testable deliverables. Identify tests during Acceptance Criteria meetings. After reviewing the initial
requirements it may become clear that edge cases can also be defined. Identify tests during development. As the deliverables begin to become fleshed
out, additional test cases may become apparent, and should be defined. Identify tests during testing. When the testing process begins, more test
scenarios may be discovered, and should be defined and documented. Identify tests when errors appear. Errors/Bugs in any environment, including our
live environment, should result in a bug report and tests to prove when its fixed. Identify tests at any other time. Whenever a valuable test case is identified then
it should be documented and added to the test automation backlog.
Building an effective test automation suite
We describe our product requirements using the Gherkin syntax – Given, When, Then.
Describe Identify tests during Story creation. Stories should be described in Gherkin/BDD
style to develop an ubiquitous language and easily testable deliverables. Identify tests during Acceptance Criteria meetings. After reviewing the initial
requirements it may become clear that edge cases can also be defined. Identify tests during development. As the deliverables begin to become fleshed
out, additional test cases may become apparent, and should be defined. Identify tests during testing. When the testing process begins, more test
scenarios may be discovered, and should be defined and documented. Identify tests when errors appear. Errors/Bugs in any environment, including our
live environment, should result in a bug report and tests to prove when its fixed. Identify tests at any other time. Whenever a valuable test case is identified then
it should be documented and added to the test automation backlog.
Our BDD scenarios should be kept as short as possible – with a focus on making them easy to read and understand in as few sentences as possible.
The focus should be to explain the intended behavior of a particular piece of functionality – not to describe the steps required to make it happen.
GIVEN steps are used to describe the expected system state before our desired behavior occurs.
WHEN steps are used to describe the action that we would like to carry out on the system.
THEN steps describe the expected reaction or the state of the system after we have carried out the action in the When step.
Building an effective test automation suite
We describe our product requirements using the Gherkin syntax – Given, When, Then.
A Good BDD Example Identify tests during Story creation. Stories should be described in Gherkin/BDD
style to develop an ubiquitous language and easily testable deliverables. Identify tests during Acceptance Criteria meetings. After reviewing the initial
requirements it may become clear that edge cases can also be defined. Identify tests during development. As the deliverables begin to become fleshed
out, additional test cases may become apparent, and should be defined. Identify tests during testing. When the testing process begins, more test
scenarios may be discovered, and should be defined and documented. Identify tests when errors appear. Errors/Bugs in any environment, including our
live environment, should result in a bug report and tests to prove when its fixed. Identify tests at any other time. Whenever a valuable test case is identified then
it should be documented and added to the test automation backlog.
Story: Account Holder withdraws cash
As an Account HolderI want to withdraw cash from an ATMSo that I can get money when the bank is closed
Scenario: Card has been disabledGiven the card is disabledWhen the Account Holder requests $20Then the ATM should retain the cardAnd the ATM should say the card has been retained
http://dannorth.net/whats-in-a-story/
Building an effective test automation suite
We describe our product requirements using the Gherkin syntax – Given, When, Then.
A Good BDD Example Identify tests during Story creation. Stories should be described in Gherkin/BDD
style to develop an ubiquitous language and easily testable deliverables. Identify tests during Acceptance Criteria meetings. After reviewing the initial
requirements it may become clear that edge cases can also be defined. Identify tests during development. As the deliverables begin to become fleshed
out, additional test cases may become apparent, and should be defined. Identify tests during testing. When the testing process begins, more test
scenarios may be discovered, and should be defined and documented. Identify tests when errors appear. Errors/Bugs in any environment, including our
live environment, should result in a bug report and tests to prove when its fixed. Identify tests at any other time. Whenever a valuable test case is identified then
it should be documented and added to the test automation backlog.
Story: Account Holder withdraws cash
As an Account HolderI want to withdraw cash from an ATMSo that I can get money when the bank is closed
Scenario: Card has been disabledGiven the card is disabled <= Description of state before the actionWhen the Account Holder requests $20 <= The action to be carried outThen the ATM should retain the card <= How the system should react to the actionAnd the ATM should say the card has been retained <= The systems state afterward
http://dannorth.net/whats-in-a-story/
Building an effective test automation suite
Once our scenarios have been Identified and Described then they need to be implemented as automated tests in our chosen test automation tools - currently Cucumber & Capybara.
Implement Identify tests during Story creation. Stories should be described in Gherkin/BDD
style to develop an ubiquitous language and easily testable deliverables. Identify tests during Acceptance Criteria meetings. After reviewing the initial
requirements it may become clear that edge cases can also be defined. Identify tests during development. As the deliverables begin to become fleshed
out, additional test cases may become apparent, and should be defined. Identify tests during testing. When the testing process begins, more test
scenarios may be discovered, and should be defined and documented. Identify tests when errors appear. Errors/Bugs in any environment, including our
live environment, should result in a bug report and tests to prove when its fixed. Identify tests at any other time. Whenever a valuable test case is identified then
it should be documented and added to the test automation backlog.
Building an effective test automation suite
Once our scenarios have been Identified and Described then they need to be implemented as automated tests in our chosen test automation tools - currently Cucumber & Capybara.
Implement Identify tests during Story creation. Stories should be described in Gherkin/BDD
style to develop an ubiquitous language and easily testable deliverables. Identify tests during Acceptance Criteria meetings. After reviewing the initial
requirements it may become clear that edge cases can also be defined. Identify tests during development. As the deliverables begin to become fleshed
out, additional test cases may become apparent, and should be defined. Identify tests during testing. When the testing process begins, more test
scenarios may be discovered, and should be defined and documented. Identify tests when errors appear. Errors/Bugs in any environment, including our
live environment, should result in a bug report and tests to prove when its fixed. Identify tests at any other time. Whenever a valuable test case is identified then
it should be documented and added to the test automation backlog.
The Gherkin scenarios are included as feature files in our test automation project (PACT).
All defined tests are initially tagged as @not_started so they are not included in test runs.
When a test is being implemented then the tag changes from @not_started to @wip. Only one test should ever have a @wip tag.
When the test is successfully implemented then the @wip tag changes to a @complete tag so it is included in test runs.
Tests may also have other tags to describe what area they belong to. For example @messenger or @page_manager. This allows selective test runs.
Tests should be implemented to run on all our testable environments or should be tagged to indicate the environments on which they cannot be run. e.g. @not_staging or @not_production.
Building an effective test automation suite
Completed tests should be run as often as possible, by as many people as possible, on as many environments, operating systems and browsers as possible.
Run
Building an effective test automation suite
Completed tests should be run as often as possible, by as many people as possible, on as many environments, operating systems and browsers as possible.
Run Identify tests during Story creation. Stories should be described in Gherkin/BDD
style to develop an ubiquitous language and easily testable deliverables. Identify tests during Acceptance Criteria meetings. After reviewing the initial
requirements it may become clear that edge cases can also be defined. Identify tests during development. As the deliverables begin to become fleshed
out, additional test cases may become apparent, and should be defined. Identify tests during testing. When the testing process begins, more test
scenarios may be discovered, and should be defined and documented. Identify tests when errors appear. Errors/Bugs in any environment, including our
live environment, should result in a bug report and tests to prove when its fixed. Identify tests at any other time. Whenever a valuable test case is identified then
it should be documented and added to the test automation backlog.
During development of tests, they should be run on the local PC of the test implementer using their local OS and Browser configurations.
After completion of the tests then then should be run on a variety of supported OS and Browser combinations using our own machines or virtualized systems such as Saucelabs on a regular basis.
During our sprint testing cycles the test suites should be run by QA engineers to prove existing functionality and catch potential regressions.
As a regular ongoing process the test suites should be run against the latest code which is considered complete by the developers.
Building an effective test automation suite
In order to maximize the value of our test automation and to ensure any potential problems are caught as early as possible – they should be integrated in a CI environment.
Integrate
Building an effective test automation suite
In order to maximize the value of our test automation and to ensure any potential problems are caught as early as possible – they should be integrated in a CI environment.
Integrate Identify tests during Story creation. Stories should be described in Gherkin/BDD
style to develop an ubiquitous language and easily testable deliverables. Identify tests during Acceptance Criteria meetings. After reviewing the initial
requirements it may become clear that edge cases can also be defined. Identify tests during development. As the deliverables begin to become fleshed
out, additional test cases may become apparent, and should be defined. Identify tests during testing. When the testing process begins, more test
scenarios may be discovered, and should be defined and documented. Identify tests when errors appear. Errors/Bugs in any environment, including our
live environment, should result in a bug report and tests to prove when its fixed. Identify tests at any other time. Whenever a valuable test case is identified then
it should be documented and added to the test automation backlog.
Our tests are designed to be run regularly on a variety of environments without any manual setup or configuration steps. They are also designed to be quick and independent of each other.
To ensure that the tests are run as often as possible and are constantly proving the correctness of our systems, they will be integrated into a continuous testing environment, and run on a regular basis against all our testable environments.
We will use Jenkins, the continuous integration tool, to automate test runs. These will occur on both a scheduled basis and as new versions of our code are deployed onto the testable environments.
Some environments will receive more comprehensive testing on a more regular basis. For example, all tests would be run on the Test Automation environment whenever it is deployed (multiple times every day) but only a subset of tests would be run on Production (once every 2 weeks).
Building an effective test automation suite
Everyone on the team has a part to play when it comes to ensuring that we have a comprehensive test suite which describes the intended behavior of Wildfire’s software.
What does this mean to me?
PO’s Dev’s QA’s TA’s
Building an effective test automation suite
Everyone on the team has a part to play when it comes to ensuring that we have a comprehensive test suite which describes the intended behavior of Wildfire’s software.
Product Owner Ensure that all the sprints stories have enough Gherkin/BDD scenarios to get the
discussion started during planning sessions. The scenarios should be short but concise and written from the perspective of a
user of the system. Ideally these scenarios would be written in advance of any planning sessions and
circulated to the rest of the team for review. Follow up with the rest of the team both before, during and after work on the
story has been undertaken to ensure that your vision has been understood by everyone involved in the development process.
Ask to see the automated tests running when the story is marked as Done.
Building an effective test automation suite
Everyone on the team has a part to play when it comes to ensuring that we have a comprehensive test suite which describes the intended behavior of Wildfire’s software.
Developers Review the Gherkin/BDD scenarios of a story prior to the planning and AC
meetings to have a clear understanding of what is being asked. Give feedback and ask questions before, during and after the planning sessions. Work with the PM and QA’s to identify any other core scenarios and edge cases
which need to be added to the feature to ensure it is complete. Work with the QA’s to ensure that the feature is testable in both a manual and
automated fashion on all of our testable environments. When code complete – run the automated acceptance test suite on your
environment to ensure that nothing else got broken. Follow up with the test automators to ensure that all environments are tested.
Building an effective test automation suite
Everyone on the team has a part to play when it comes to ensuring that we have a comprehensive test suite which describes the intended behavior of Wildfire’s software.
Quality Assurance Engineers
Review the Gherkin/BDD scenarios of a story prior to the planning and AC meetings to have a clear understanding of what is being asked.
Give feedback and ask questions before, during and after the planning sessions. Work with the PM and Dev’s to identify any other core scenarios and edge cases
which need to be added to the feature to ensure it is complete. Work with the test automators to create a prioritized backlog of tests which
should be added to the automated test suite. Checkout the automated test suite and run it on a regular basis to help catch
regressions and identify environmental issues. Work with the automators to ensure your project has good test coverage.
Building an effective test automation suite
Everyone on the team has a part to play when it comes to ensuring that we have a comprehensive test suite which describes the intended behavior of Wildfire’s software.
Test Automators Review the Gherkin/BDD scenarios of a story prior to the planning and AC
meetings to have a clear understanding of what is being asked. Give feedback and ask questions before, during and after the planning sessions. Work with the PM and Dev’s to identify any other core scenarios and edge cases
which need to be added to the feature to ensure it is complete. Work with the QA’s to create a prioritized backlog of tests which should be
added to the automated test suite. Work with the Developers to ensure that automated tests can be run in any
environment and to ensure the test automation environment is suitable for test. Keep the test suite in a well maintained state so that it can be run at any time.
Building an effective test automation suite
Finally we have some goals which we should aim for with our Test Automation.
Building an effective test automation suite
Finally we have some goals which we should aim for with our Test Automation.
Our test automation suite should deliver wide ranging yet concise coverage of the existing functionality provided by Wildfire’s applications. This is to help identify and fix regressions at the earliest possible opportunity.
As new stories are implemented on a sprint by sprint basis we should identify every potentially automatable story and add it to the automation backlog.
The test suite should be deployable by anyone on the team and should require no manual intervention or manual setup tasks to run.
In collaboration with the developers and system administrators we must work towards having a completely isolated test automation environment which is free of dependencies on third party systems.
Our ultimate goal should be to have a self testing system which automatically deploys and tests itself on various environments and provides feedback on potential problems and tells us when it is ready to be deployed to our production environment – all without any manual intervention.
Building an effective test automation suite
One last thing…
Building an effective test automation suite
One last thing…
An automated test suite is not, and never will be, a replacement for manual testing activities.
The goal of an automated test suite is to provide fast and regular feedback on the known and predictable elements within our system.
We will always require manual testing in order to identify and understand the unknown or unpredictable elements within our system.
Our goal should be to find the right balance between manual and automated testing which enables us to deliver software in a fast, efficient and confident manner.
Building an effective test automation suite
This is still a work in progress. Please give your feedback to make this a more accurate representation of the process we want to achieve.Thanks..
Top Related