Getting your mobile test automation process in place - using Cucumber and Calabash
-
Upload
niels-frydenholm -
Category
Software
-
view
185 -
download
1
description
Transcript of Getting your mobile test automation process in place - using Cucumber and Calabash
![Page 2: Getting your mobile test automation process in place - using Cucumber and Calabash](https://reader033.fdocuments.net/reader033/viewer/2022051323/547e807ab4af9f9b158b570b/html5/thumbnails/2.jpg)
Get Your Testing Process in Place
Technical Lead,, Xamarin Test Cloud@karlkrukow
Karl Krukow
![Page 3: Getting your mobile test automation process in place - using Cucumber and Calabash](https://reader033.fdocuments.net/reader033/viewer/2022051323/547e807ab4af9f9b158b570b/html5/thumbnails/3.jpg)
![Page 4: Getting your mobile test automation process in place - using Cucumber and Calabash](https://reader033.fdocuments.net/reader033/viewer/2022051323/547e807ab4af9f9b158b570b/html5/thumbnails/4.jpg)
“Continuous Delivery is a software development discipline where you build software in such a way that the software can be released to production at any time.”
Martin FowlerThoughtWorks
![Page 5: Getting your mobile test automation process in place - using Cucumber and Calabash](https://reader033.fdocuments.net/reader033/viewer/2022051323/547e807ab4af9f9b158b570b/html5/thumbnails/5.jpg)
Continuous Delivery
![Page 6: Getting your mobile test automation process in place - using Cucumber and Calabash](https://reader033.fdocuments.net/reader033/viewer/2022051323/547e807ab4af9f9b158b570b/html5/thumbnails/6.jpg)
Mobile Continuous Delivery Challenges
• App Store Distribution Model
• Maturity - No Industry Standards (yet)
• Complicated Tool Setup
• Realistic tests are challenging
• Tested on your users’ devices
• Vary Device Conditions - Device settings, Network…
![Page 7: Getting your mobile test automation process in place - using Cucumber and Calabash](https://reader033.fdocuments.net/reader033/viewer/2022051323/547e807ab4af9f9b158b570b/html5/thumbnails/7.jpg)
Towards Continuous Delivery for Mobile
Tooling
People
Process
Aspects
![Page 8: Getting your mobile test automation process in place - using Cucumber and Calabash](https://reader033.fdocuments.net/reader033/viewer/2022051323/547e807ab4af9f9b158b570b/html5/thumbnails/8.jpg)
Towards Continuous Delivery for Mobile
Tooling
People
Process
• Xamarin.UITest, Calabash
• Jenkins, TeamCity, TFS, …
• Xamarin Test Cloud
ExamplesAspects
![Page 9: Getting your mobile test automation process in place - using Cucumber and Calabash](https://reader033.fdocuments.net/reader033/viewer/2022051323/547e807ab4af9f9b158b570b/html5/thumbnails/9.jpg)
Towards Continuous Delivery for Mobile
Tooling
People
Process
• Xamarin.UITest, Calabash
• Jenkins, TeamCity, TFS, …
• Xamarin Test Cloud
• Cross-functional Teams
• Behaviour Driven Development
• Shared responsibility
ExamplesAspects
![Page 10: Getting your mobile test automation process in place - using Cucumber and Calabash](https://reader033.fdocuments.net/reader033/viewer/2022051323/547e807ab4af9f9b158b570b/html5/thumbnails/10.jpg)
Xamarin.UITest & Calabash
![Page 11: Getting your mobile test automation process in place - using Cucumber and Calabash](https://reader033.fdocuments.net/reader033/viewer/2022051323/547e807ab4af9f9b158b570b/html5/thumbnails/11.jpg)
• Write Tests in C#
• Run with NUnit
• Xamrin/Visual Studio or CLI
Xamarin.UITest• Write Tests in Ruby
• Supports BDD, run with Cucumber
• RubyMine or CLI
Calabash
Xamarin.UITest & Calabash
![Page 12: Getting your mobile test automation process in place - using Cucumber and Calabash](https://reader033.fdocuments.net/reader033/viewer/2022051323/547e807ab4af9f9b158b570b/html5/thumbnails/12.jpg)
Xamarin.UITest - Architecture
Your App
Test Cloud Agent
![Page 13: Getting your mobile test automation process in place - using Cucumber and Calabash](https://reader033.fdocuments.net/reader033/viewer/2022051323/547e807ab4af9f9b158b570b/html5/thumbnails/13.jpg)
Calabash - Architecture
Your App
Test Cloud Agent
!
Scenario: Login
Given I am on the Login screen
When I login as "Nat"
Then I should go to the Assignments
![Page 14: Getting your mobile test automation process in place - using Cucumber and Calabash](https://reader033.fdocuments.net/reader033/viewer/2022051323/547e807ab4af9f9b158b570b/html5/thumbnails/14.jpg)
Calabash - Architecture
Your App
Test Cloud Agent
Cucumber
!
Scenario: Login
Given I am on the Login screen
When I login as "Nat"
Then I should go to the Assignments
![Page 15: Getting your mobile test automation process in place - using Cucumber and Calabash](https://reader033.fdocuments.net/reader033/viewer/2022051323/547e807ab4af9f9b158b570b/html5/thumbnails/15.jpg)
Calabash - Architecture
Your App
Test Cloud Agent
Cucumber
!
Scenario: Login
Given I am on the Login screen
When I login as "Nat"
Then I should go to the Assignments
![Page 17: Getting your mobile test automation process in place - using Cucumber and Calabash](https://reader033.fdocuments.net/reader033/viewer/2022051323/547e807ab4af9f9b158b570b/html5/thumbnails/17.jpg)
Agenda
■ From story to running tests ■ Improvements in development and release cycle ■ Demo ■ Best practices (based on our learnings) ■ Final words ■ Questions?
![Page 18: Getting your mobile test automation process in place - using Cucumber and Calabash](https://reader033.fdocuments.net/reader033/viewer/2022051323/547e807ab4af9f9b158b570b/html5/thumbnails/18.jpg)
eBay Classifieds, DenmarkTwo popular consumer apps and sites ■ DBA - DK’s #1 Classifieds site ■ BilBasen - DK’s #1 Online Vehicle Marketplace
A few numbers ■ Approx. 1/4 of the danish population is using the
DBA app ■ Mobile traffic share is 57% for DBA ■ Approx 1/8 of the danish population is using the
BilBasen app ■ Mobile traffic share is 54% for BilBasen
![Page 19: Getting your mobile test automation process in place - using Cucumber and Calabash](https://reader033.fdocuments.net/reader033/viewer/2022051323/547e807ab4af9f9b158b570b/html5/thumbnails/19.jpg)
How we are organised
■ 4 iOS developers / (backend) ■ 1 backend developer ■ 1 QA expert ■ 1 Scrum master
The team
![Page 20: Getting your mobile test automation process in place - using Cucumber and Calabash](https://reader033.fdocuments.net/reader033/viewer/2022051323/547e807ab4af9f9b158b570b/html5/thumbnails/20.jpg)
From Story to Running Tests
![Page 21: Getting your mobile test automation process in place - using Cucumber and Calabash](https://reader033.fdocuments.net/reader033/viewer/2022051323/547e807ab4af9f9b158b570b/html5/thumbnails/21.jpg)
Implementing a storyBefore sprint ■ Product owner (PO) introduces story ■ Team and PO agrees on accept criteria’s
New feature idea
Lots of questions
![Page 22: Getting your mobile test automation process in place - using Cucumber and Calabash](https://reader033.fdocuments.net/reader033/viewer/2022051323/547e807ab4af9f9b158b570b/html5/thumbnails/22.jpg)
Implementing a storyBefore sprint ■ Product owner (PO) introduces story ■ Team and PO agrees on accept criteria’s
New feature idea
Lots of questions
Implement featureWrite tests
ReviewTest
In Sprint ■ Developer(s) implements story and tests
■ Unit, Integration and UI tests ■ Done with code and tests are green
■ Review code (and tests) ■ Manuel test from other team member ■ Inspect code coverage to ensure test quality
![Page 23: Getting your mobile test automation process in place - using Cucumber and Calabash](https://reader033.fdocuments.net/reader033/viewer/2022051323/547e807ab4af9f9b158b570b/html5/thumbnails/23.jpg)
Continuous Integration■ Jenkins on a couple of Mac Mini’s
- Unit -> Integration -> UI tests
- UI tests only in Simulator ■ A subset after commit ■ A subset in Xamarin Cloud ■ All UI tests nightly
!
■ Keep status visible and Green!
![Page 24: Getting your mobile test automation process in place - using Cucumber and Calabash](https://reader033.fdocuments.net/reader033/viewer/2022051323/547e807ab4af9f9b158b570b/html5/thumbnails/24.jpg)
Focus on Automated Tests Has Improved ‘Everything’
![Page 25: Getting your mobile test automation process in place - using Cucumber and Calabash](https://reader033.fdocuments.net/reader033/viewer/2022051323/547e807ab4af9f9b158b570b/html5/thumbnails/25.jpg)
Releases 2013
JAN
DBA versions
BilBasen versions
Refactored tests to Page objects
FEB MAR APR MAY JUN JUL AUG SEP OCT NOV DEC
2.0 2.0.1 2.1 2.2 2.3 2.3.1 2.4 3.0
4.1.2
4.1.3
4.2 5.0 5.0.1 5.0.2
![Page 26: Getting your mobile test automation process in place - using Cucumber and Calabash](https://reader033.fdocuments.net/reader033/viewer/2022051323/547e807ab4af9f9b158b570b/html5/thumbnails/26.jpg)
5.2.1
3.53.3.1
3.2.3
3.2.23.1.1
Releases 2014
JAN
DBA versions
BilBasen versions
FEB MAR APR MAY JUN JUL AUG SEP OCT NOV DEC
3.0.1 3.1 3.2.1 3.4 3.4.2
5.0.3
3.2 3.3 3.4.1
5.0.4 5.0.5 5.0.6 5.1 5.2 5.2.2
![Page 27: Getting your mobile test automation process in place - using Cucumber and Calabash](https://reader033.fdocuments.net/reader033/viewer/2022051323/547e807ab4af9f9b158b570b/html5/thumbnails/27.jpg)
Development improvements
• Limited refactoring - risk too high
• All changes = huge test effort
• Many bugs were not found until later…
• Lots of “legacy code”
• See first bullet
In early 2013
![Page 28: Getting your mobile test automation process in place - using Cucumber and Calabash](https://reader033.fdocuments.net/reader033/viewer/2022051323/547e807ab4af9f9b158b570b/html5/thumbnails/28.jpg)
Development improvements
• Limited refactoring - risk too high
• All changes = huge test effort
• Many bugs were not found until later…
• Lots of “legacy code”
• See first bullet
• Planned refactoring of larger areas
• Boy scout clean up
• Adding tests to missing areas
• (Most) bugs are found immediately after commit
• QA can spend time on better things ■ Identify critical business areas ■ Find the crazy bugs
In early 2013 Now
![Page 29: Getting your mobile test automation process in place - using Cucumber and Calabash](https://reader033.fdocuments.net/reader033/viewer/2022051323/547e807ab4af9f9b158b570b/html5/thumbnails/29.jpg)
Project example
■ Search criteria pages and search logic was totally re-written ■ From hardcoded options in the app ■ To meta driven from the server ■ Server side meta data was build as
part of a project ■ Saved searches now in sync with web
BilBasen - new search
![Page 30: Getting your mobile test automation process in place - using Cucumber and Calabash](https://reader033.fdocuments.net/reader033/viewer/2022051323/547e807ab4af9f9b158b570b/html5/thumbnails/30.jpg)
BilBasen - new search
■ Most scenarios were already covered with UI tests
■ Manuel test effort ■ About one day of testing ■ Focused on network timings ■ Compare results with website
■ No serious bugs were found
Test results
![Page 31: Getting your mobile test automation process in place - using Cucumber and Calabash](https://reader033.fdocuments.net/reader033/viewer/2022051323/547e807ab4af9f9b158b570b/html5/thumbnails/31.jpg)
• Entire team spent 2-5 days on test runs and bug fixes
• Serious bugs could be found in last minute
• Big releases to save the overhead
• Releases were often delayed
Release improvementsIn early 2013
![Page 32: Getting your mobile test automation process in place - using Cucumber and Calabash](https://reader033.fdocuments.net/reader033/viewer/2022051323/547e807ab4af9f9b158b570b/html5/thumbnails/32.jpg)
• Entire team spent 2-5 days on test runs and bug fixes
• Serious bugs could be found in last minute
• Big releases to save the overhead
• Releases were often delayed
Release improvements
• Upload when all tests are green (all the time!)
• PO flexibility to change scope of release
• 1 hour test run for some releases ■ Bugs are very rarely found
In early 2013 Now
![Page 33: Getting your mobile test automation process in place - using Cucumber and Calabash](https://reader033.fdocuments.net/reader033/viewer/2022051323/547e807ab4af9f9b158b570b/html5/thumbnails/33.jpg)
Demo
![Page 34: Getting your mobile test automation process in place - using Cucumber and Calabash](https://reader033.fdocuments.net/reader033/viewer/2022051323/547e807ab4af9f9b158b570b/html5/thumbnails/34.jpg)
Best Practices (based on our learnings)
![Page 35: Getting your mobile test automation process in place - using Cucumber and Calabash](https://reader033.fdocuments.net/reader033/viewer/2022051323/547e807ab4af9f9b158b570b/html5/thumbnails/35.jpg)
Best Practices
• Structure your test code well
• Page objects
• Replace “sleeps” with “wait_for”
![Page 36: Getting your mobile test automation process in place - using Cucumber and Calabash](https://reader033.fdocuments.net/reader033/viewer/2022051323/547e807ab4af9f9b158b570b/html5/thumbnails/36.jpg)
Page objects and wait_for
Scenario: Check that change classification clears matrixdata with warning Given I am logged in as "Buyer" And I am on the SYI hub And I select classification "Hovedtelefoner" And I set "Type" to “In-ear” And I set price to "250" !
Before page objects (Ruby example)
![Page 37: Getting your mobile test automation process in place - using Cucumber and Calabash](https://reader033.fdocuments.net/reader033/viewer/2022051323/547e807ab4af9f9b158b570b/html5/thumbnails/37.jpg)
Page objects and wait_for
Scenario: Check that change classification clears matrixdata with warning Given I am logged in as "Buyer" And I am on the SYI hub And I select classification "Hovedtelefoner" And I set "Type" to “In-ear” And I set price to "250" !
Before page objects (Ruby example)
And /^I set price to "(.*?)"$/ do |price| macro 'I swipe up' sleep(1) touch("view marked:'Price'") sleep(0.5) set_text("view marked:'Price'", price) sleep(0.5) touch("view marked:'OK'") sleep(1) end
![Page 38: Getting your mobile test automation process in place - using Cucumber and Calabash](https://reader033.fdocuments.net/reader033/viewer/2022051323/547e807ab4af9f9b158b570b/html5/thumbnails/38.jpg)
Page object and wait_for
And /^I set price to "(.*?)"$/ do |price| @page.write_price(price) end
With page objects (Ruby example)
![Page 39: Getting your mobile test automation process in place - using Cucumber and Calabash](https://reader033.fdocuments.net/reader033/viewer/2022051323/547e807ab4af9f9b158b570b/html5/thumbnails/39.jpg)
Page object and wait_for
And /^I set price to "(.*?)"$/ do |price| @page.write_price(price) end
With page objects (Ruby example)
def write_price(price) scroll_and_wait_for_row_with_mark("priceCell") touch("view marked:'Price'") keyboard_enter_text price close_keyboard end
SellYourItemPage class (page object)
def select_bundle(bundle_name) ..logic here.. end !def write_headline(headline) ..logic here.. end
![Page 40: Getting your mobile test automation process in place - using Cucumber and Calabash](https://reader033.fdocuments.net/reader033/viewer/2022051323/547e807ab4af9f9b158b570b/html5/thumbnails/40.jpg)
Best Practices• Structure your test code well
• Page objects
• replace “sleeps” with “wait_for”
• Ensure stable test suite execution
• Give failed tests an extra run
![Page 41: Getting your mobile test automation process in place - using Cucumber and Calabash](https://reader033.fdocuments.net/reader033/viewer/2022051323/547e807ab4af9f9b158b570b/html5/thumbnails/41.jpg)
Stability in your CI setup• UIAutomation + Simulator can often fail when running
a big test suite
• Re-run failed tests an extra time
• Cucumber rerun formatter
• NUnit-retry
• Red builds should be trusted, investigated and fixed
• Only use this if test suite starts to be unstable
![Page 42: Getting your mobile test automation process in place - using Cucumber and Calabash](https://reader033.fdocuments.net/reader033/viewer/2022051323/547e807ab4af9f9b158b570b/html5/thumbnails/42.jpg)
Stability in your CI setup• UIAutomation + Simulator can often fail when running
a big test suite
• Re-run failed tests an extra time
• Cucumber rerun formatter
• NUnit-retry
• Red builds should be trusted, investigated and fixed
• Only use this if test suite starts to be unstable
![Page 43: Getting your mobile test automation process in place - using Cucumber and Calabash](https://reader033.fdocuments.net/reader033/viewer/2022051323/547e807ab4af9f9b158b570b/html5/thumbnails/43.jpg)
Best Practices• Structure your test code well
• Page objects
• replace “sleeps” with “wait_for”
• Ensure stable test suite execution
• Give failed tests an extra
• Run tests on real devices
• Xamarin Test Cloud
• Screen sizes, OS versions
![Page 44: Getting your mobile test automation process in place - using Cucumber and Calabash](https://reader033.fdocuments.net/reader033/viewer/2022051323/547e807ab4af9f9b158b570b/html5/thumbnails/44.jpg)
![Page 45: Getting your mobile test automation process in place - using Cucumber and Calabash](https://reader033.fdocuments.net/reader033/viewer/2022051323/547e807ab4af9f9b158b570b/html5/thumbnails/45.jpg)
Best Practices• Structure your test code well
• Page objects
• replace “sleeps” with “wait_for”
• Ensure stable test suite execution
• Give failed tests an extra run
• Run tests on real devices
• Xamarin Test Cloud
• Screen sizes, OS versions
• Strive for fast test execution
• Use different test suites
• Avoid too much UI interaction for setup/teardown operations
![Page 46: Getting your mobile test automation process in place - using Cucumber and Calabash](https://reader033.fdocuments.net/reader033/viewer/2022051323/547e807ab4af9f9b158b570b/html5/thumbnails/46.jpg)
Fast test execution
• Use the “Backdoor” to the app
• Xamarin.UITest.IApp.Invoke
• Invoke(“method”, param)
• Invoke logic - without using the UI
• Great for Setup / Teardown behaviour
• Backdoor logic should be tested with UI interactions in other tests
![Page 47: Getting your mobile test automation process in place - using Cucumber and Calabash](https://reader033.fdocuments.net/reader033/viewer/2022051323/547e807ab4af9f9b158b570b/html5/thumbnails/47.jpg)
Fast test execution
• Use the “Backdoor” to the app
• Xamarin.UITest.IApp.Invoke
• Invoke(“method”, param)
• Invoke logic - without using the UI
• Great for Setup / Teardown behaviour
• Backdoor logic should be tested with UI interactions in other tests
• We use it for things like
• Create a user / Login
• Create listings (things for sale)
• Cleaning up
• Delete favorites
• Delete users listings
![Page 48: Getting your mobile test automation process in place - using Cucumber and Calabash](https://reader033.fdocuments.net/reader033/viewer/2022051323/547e807ab4af9f9b158b570b/html5/thumbnails/48.jpg)
Fast test execution - backdoor optimised
Scenario: I can manage my listing and see the changes Given I am logged in as "UniqueSeller" using quick login And I have created a listing for "Hovedtelefoner" and is on the SYI VIP Then I see the VIP for "Her kommer en rimelig lang tekst" ! When I go back to my listings page And the created listing is in the list …more steps omitted…
![Page 49: Getting your mobile test automation process in place - using Cucumber and Calabash](https://reader033.fdocuments.net/reader033/viewer/2022051323/547e807ab4af9f9b158b570b/html5/thumbnails/49.jpg)
Fast test execution - backdoor optimised
Scenario: I can manage my listing and see the changes Given I am logged in as "UniqueSeller" using quick login And I have created a listing for "Hovedtelefoner" and is on the SYI VIP Then I see the VIP for "Her kommer en rimelig lang tekst" ! When I go back to my listings page And the created listing is in the list …more steps omitted…
1 scenario (1 passed) 19 steps (19 passed)
0m34.054s1 scenario (1 passed) 19 steps (19 passed)
1m16.610s
![Page 50: Getting your mobile test automation process in place - using Cucumber and Calabash](https://reader033.fdocuments.net/reader033/viewer/2022051323/547e807ab4af9f9b158b570b/html5/thumbnails/50.jpg)
Best Practices• Structure your test code well
• Page objects
• replace “sleeps” with “wait_for”
• Ensure stable test suite execution
• Give failed tests an extra run
• Run tests on real devices
• Xamarin Test Cloud
• Screen sizes, OS versions
• Strive for fast test execution
• Use different test suites
• Avoid too much UI interaction for setup/teardown operations
• Run tests for all active branches in CI
• Automate job creation
![Page 51: Getting your mobile test automation process in place - using Cucumber and Calabash](https://reader033.fdocuments.net/reader033/viewer/2022051323/547e807ab4af9f9b158b570b/html5/thumbnails/51.jpg)
Feature builds■ All jobs required for a branch ■ Red builds are accepted for a period ■ Only merge to master with green builds
![Page 52: Getting your mobile test automation process in place - using Cucumber and Calabash](https://reader033.fdocuments.net/reader033/viewer/2022051323/547e807ab4af9f9b158b570b/html5/thumbnails/52.jpg)
Final words
![Page 53: Getting your mobile test automation process in place - using Cucumber and Calabash](https://reader033.fdocuments.net/reader033/viewer/2022051323/547e807ab4af9f9b158b570b/html5/thumbnails/53.jpg)
Start today…• QA is a Team effort / responsibility
• Bugs/Crashes will still live in the app
• Do not expect it to be effortless to get started
• It will quickly pay off
![Page 54: Getting your mobile test automation process in place - using Cucumber and Calabash](https://reader033.fdocuments.net/reader033/viewer/2022051323/547e807ab4af9f9b158b570b/html5/thumbnails/54.jpg)
Start today…• QA is a Team effort / responsibility
• Bugs/Crashes will still live in the app
• Do not expect it to be effortless to get started
• It will quickly pay off
• Higher app quality (and code quality)
• Time to focus on new important features
• Faster time to market - with less bugs
• Happy customers - they like frequent updates of high quality
• Happy and calm QA
• Happy and brave developers
See it as an investment that gives you
![Page 55: Getting your mobile test automation process in place - using Cucumber and Calabash](https://reader033.fdocuments.net/reader033/viewer/2022051323/547e807ab4af9f9b158b570b/html5/thumbnails/55.jpg)
Questions?
Niels Frydenholm@nfrydenholm