COPYRIGHT, TECHTALK - WWW.TECHTALK.AT
ANDREAS WILLICH (@SABOTAGEANDI)
CHRISTIAN HASSA (@CHRISHASSA)
Mobile Quality Crew Meetup, Vienna, April 23rd 2015
Cross Mobile Testautomation withXamarin & SpecFlow
2
Topics
• Xamarin• Test automation in agile projects• Specification-By-Example• SpecFlow•UI automation on mobile devices•Mobile app automation with SpecFlow
3
Xamarin
4
Native development for each platform
5
Abstracting the platform
6
Xamarin approach
7
Compilation and run time
Android APK
Android/Linux kernelIL code for platform
specific profileVS solution with
projects
iOS APP
Android APK
Shared Code(C#,VB.NET)
Mono CLR for iOS (Monotouch)
Mono CLR for Android
(Monodroid)
Multi-Platformprofile assembly
Dalvik
Android.*Java.*
IL+JIT
AOT ARM Binary
iOS App (C#)
Android App(C#)
Xamarin.iOSprofile assembly
Xamarin.Androidprofile assembly
iOS/Cocoa Touch
iOS APP
ARMbinary
8
Native mobile apps with C#/Xamarin
1. Reuse existing .NET/C# Know-How
2. Powerful platform and tools(Linq, Xml, Events/Delegates, Parallel Programming, Visual Studio/TFS, …)
3. Shared code for multi-platform apps
9
Required Know-How
C#
10
Xamarin.iOSC#
11
Xamarin.Android
twitterListView.setOnItemClickListener (new OnItemClickListener () {public void onItemClick (AdapterView<?> parent, View view, int position, long id) {
Intent tweetDetails = new Intent (TwitterScreen.this, TweetDetailsScreen.class);tweetDetails.putExtra ("TweetID", TwitterFeed[position].ID);startActivity (tweetDetails);
}};
twitterListView.ItemClick += (s, e) => {var tweetDetails = new Intent (this, typeof (TweetDetailsScreen));tweetDetails.PutExtra ("TweetID", TwitterFeed[e.Position].ID);StartActivity (tweetDetails);
};
C#
12
Development environment
OS Mac OSX, Windows Windows
UI Designer Android, iOS
Build Android, iOS (via Xamarin Build Host on Mac OSX)
Debug Android (Device and ARM/x86 emulator)iOS (Device and emulator on Mac)
13
.NET libaries availabile on Xamarin
Assembly feature setmscorlib.dll SLOpenTK.dll OpenGL/OpenALSystem.dll SL+System.Core.dll SLSystem.Data.dll NET35-System.Data.Service.Client.dll vollständiger OData ClientSystem.Json.dll SLSystem.Runtime.Serialization.dll SLSystem.ServiceModel.dll SL (WCF)System.ServiceModel.Web.dll SL+System.Transactions.dll NET35System.Web.Services NET35-System.Xml.dll NET35System.Xml.Linq.dll NET35
Mono.CompilerServices.SymbolWriter.dll Für Compiler
Mono.Data.Sqlite.dll ADO.NET fürSQLite
Mono.Data.Tds.dll TDS ProtokollSupport
Mono.Security.dll Cryptographic API
monotouch.dll iOS BindingsMono.Android.dll Android Bindings
14
Xamarin.Forms
One UI definition for 3 platforms (iOS, Android, WP)in XAML (data binding)translates to native UI controls on each platformcan be mixed with custom UI
15
Test Automationin
agile projects
16
Test automation becomes expensive when …
• automating manual tests
• automation makes tests
unreadable
• automating only after
implementation
structure
readability
right moment
17
StructureManual tests
Asserts Multiple combined features
Structure ACT-ASSERT-ACT-ASSERT-ACT-ASSERT-…
Dependent featuresLong test path with high chance to breakCause and impact of error hard to trace
Automated Check
Single aspect of a single feature
ARRANGE –ACT –ASSERT
Independent featuresShort test path with lower chance to breakCause and impact of error easy to relate
18
What to automate
Userjourneys
Acceptance-criteria
Units
exploratory testing
Test automation pyramidSource: Mike Cohn many
few hard
easy
Automatability
19
// Go to web page 'http://localhost:40001/' using new browser instanceBrowserWindow localhostBrowser = BrowserWindow.Launch(
new System.Uri(this.RecordedMethod1Params.Url));
// Click 'Register found item' linkMouse.Click(uIFundstückerfassenHyperlink, new Point(56, 9));
// Click 'Save' buttonMouse.Click(uISpeichernButton, new Point(44, 14));
int fundNr1 = int.Parse(uIFundNr127Pane.InnerText.Substring(9));
// Click 'Register found item' linkMouse.Click(uIFundstückerfassenHyperlink, new Point(63, 7));
// Click 'Save' buttonMouse.Click(uISpeichernButton, new Point(34, 11));
int fundNr2 = int.Parse(uIFundNr128Pane.InnerText.Substring(9));
Assert.IsTrue(fundNr1 + 1 == fundNr2);
// Click 'Close' buttonMouse.Click(uICloseButton, new Point(26, 11));
Readability
20
A readable test case
Scenario: New found items should receive a consecutive number for the current year
Given the previous found item of thecurrent year had the number 145
When I register a new found item
Then the last found item of thecurrent year should have the number 146
21
When to test (right moment)
Acceptance criteria checks(ATDD, BDD)
Unit Tests(TDD)
business view
technical view
Exploratory tests User Journey tests
Performance, Scalability, Usability,Security, …
pro
du
ct d
efin
itio
n pro
du
ct critiqu
e
New dimension: defining the productSynergy: Specification of requirements and tests
Agile Testing Quadrants: Brian Marick
22
Automated checks don’t replace manual testing
Userjourneys
Acceptance-criteria
Units
exploratory testing
manual checkin definition of done for stories
Main success paths
Undiscovered acceptance criteria
No/(fewer) manualregression checks
Fewer pathsare enough
More timefor exploration
23
Specification-By-Example
24
Agile “requirements”
Specification-
By-ExampleImage from: 50 Quick ideas to improve your user stories: https://leanpub.com/50quickideas
Why?Outcome
How?Specification
25
Agile specifications
define upfront define just-in-time
Code
Reminderfor aconversation
Goals
Impacts
Epics
User Stories
Acceptance criteria
Why?Outcome
How?Specification
26
public void CalculateDiscount(Order order)
{
if (order.Customer.IsNew)
order.FinalAmount =
Math.Round(order.Total * 9/10);
}
Register as “bart_bookworm”Go to “/catalog/search”Enter “ISBN-0955683610”Click “Search”Click “Add to Cart”Click “View Cart”Verify “Subtotal” is “$33.75”
We’d like to encourage new customers to buy in our shop.Therefore we want offer 10% discount for their first order.
Original idea for the illustration: George Dinwiddie, http://blog.gdinwiddie.com
27
Acceptance criteria
Agile specifications
define upfront define just-in-time
Code
Goals
Impacts
Epics
User Stories
Bugs
Why?Outcome
How?Specification
28
Specification with examples
Examples …make abstract descriptions better understandable
are usually not formally documented or exchanged
Examples Tests
Requirements
describe validatefulfillment of
consist of
29
Register as “bart_bookworm”Go to “/catalog/search”Enter “ISBN-0955683610”Click “Search”Click “Add to Cart”Click “View Cart”Verify “Subtotal” is “$33.75”
public void CalculateDiscount(Order order)
{
if (order.Customer.IsNew)
order.FinalAmount =
Math.Round(order.Total * 9/10);
}
We’d like to encourage new customers to buy in our shop.Therefore we want offer 10% discount for their first order.
Original idea for the illustration: George Dinwiddie, http://blog.gdinwiddie.com
30Original idea for the illustration: George Dinwiddie, http://blog.gdinwiddie.com
New customers should receive 10% discount:Given the customer hasn‘t ordered yet
When the customer adds a book for EUR 37.50 to the shopping cart
Then the shopping cart shows a total price of EUR 33.75.
31
Actually, this is not quite right:Books on sale should be excluded!
Original idea for the illustration: George Dinwiddie, http://blog.gdinwiddie.com
32
“HappyPath”
Technical feasibility
Exceptions, border cases
3 Amigos
Original idea for the illustration: George Dinwiddie, http://blog.gdinwiddie.com
33
Collecting Acceptance Criteria
“I would try to put a book into the shopping cart …”
“I would try to remove a book from the shopping cart…”
“I’d check whether the shopping cart is empty, when I enter the shop …”
Books can be added to shopping cart.
Books can be removed from shopping cart.
Shopping cart should be empty when entering the shop.
... ? …
As a potential customerI want to collect books in a shopping cartSo that I can order several books at once.
“Imagine this story is already implemented:
How would you verify it?”
“I would try to add 1000 books to the shopping cart …”
34
Exploration through examples
Books in catalogue:
Title Author
Specification-By-Example Gojko Adzic
Impact Mapping Gojko Adzic
Explore It! Elisabeth Hendrickson
Competitive Engineering Tom Gilb
… I want to find books in the catalogue by author and title
Search for … Books found …
Spec Specification-By-Example
Hend Explore It!
et Explore It!, Competitive Engineering
Context
ActionAssertion
35
Key examples: Breaking the model
Books in catalogue:
Title Author
Specification-By-Example Gojko Adzic
Impact Mapping Gojko Adzic
Explore It! Elisabeth Hendrickson
Competitive Engineering Tom Gilb
… I want to find books in the catalogue by author and title
Search for … Books found …
Spec Specification-By-Example
Hend Explore It!
et Explore It!, Competitive Engineering
What happens, if I search for “Explore Specification”?
Can I search for single characters, e.g. “e”?
Is the number of search results limited, or paged?
Is the search also performed in the sub-title of a book?
36
Purpose of the examples
• Shared understanding:acceptance criteria
•Documentation:specification details
• Regression-tests:violated specifications
37
SpecFlow
38
SpecFlow – BDD for .NET
#62 in Visual Studio Gallery based on popularity (Apr 23 2015)
#303 most downloaded package on NuGetin the last 6 weeks (Apr 23 2015)
~1‘400 unique daily visitors on specflow.org
> 30’000 active users> 40 contributors
39
Feature: Description of feature or user story
Szenariogrundriss: Beschreibung des Akzeptanzkriteriums
Szenario: Beschreibung des Akzeptanzkriteriums
Gherkin Feature Files
Background: context for all scenarios in the feature file
Scenario: Description of acceptance criterion
Angenommen/Wenn/Dann: Automatisierte Szenario SchritteGiven/When/Then: automated scenario steps
Scenario Outline: Description of acceptance criterion
Angenommen/Wenn/Dann: Automatisierte Szenario SchritteGiven/When/Then: automated scenario steps with <place holder>
Examples: table with examples for <place holders>Examples: table with examples for <place holders>
Given: automated scenario steps
@tagname
@tagname
@tagname
@tagname
40
Abstract acceptance criteria
As a shop visitorI want to collect books in my shopping basketso that I can purchase multiple books at once.
Books can be added to the shopping basket
Books can be removed from the shopping basket
Shopping basket is initially empty
The same book can be added multiple times to the shopping basket
41
Scenarios: Examples in Gherkin
As a shop visitorI want to collect books in my shopping basketso that I can purchase multiple books at once.
Books can be added to the shopping basket
Given my shopping basket is empty
When I add the book “Harry Potter” to my shopping basket
Then my shopping basket should contain 1 copy of “Harry Potter”
42
As a shop visitorI want to collect books in my shopping basketso that I can purchase multiple books at once.
Books can be added to the shopping basket
Scenarios: Examples in Gherkin
Given my shopping basket contains 1 copy of “Harry Potter”
When I add the book “Harry Potter” to my shopping basket
Then my shopping basket should contain 2 copies of “Harry Potter”
The same book can be added multiple times to the shopping basket
43
The same book can be added multiple times to the shopping basket
Structure of Scenarios
Given my shopping basket contains 1 copy of “Harry Potter”
When I add the book “Harry Potter” to my shopping basket
Then my shopping basket should contain 2 copies of “Harry Potter”
Title: Describes intention/abstract acceptance criterion
Arrange: Context, describes state of the system
Act: Execution of the feature
Assert: Assertion of observable behaviour
And I should see the warning: “Book already existed in basket”
Triple-Aconstraint“Checks”
Chainingup steps
44
Automated continuous validation
Given my shopping basket contains 1 copy of “Harry Potter”
When I add the book “Harry Potter” to my shopping basket
Then my shopping basket should contain 2 copies of “Harry Potter”
System
„Step Definitions“ are binding individual steps to an automatable interface of the application.
Automatableinterface
UIAutomation
Automation does not necessarily have to bind to the UI.
Automatability of system is supported/evolving with development.
45
UI automationon mobile devices
46
Available tools/libraries
• Android• Android 2.3+: Instrumentation• Android 4.2+: UiAutomator• selendroid• Appium• Robotium• Calaba.sh• Xamarin.UITest
• iOS• UIAutomation• Appium• ios-driver• Calaba.sh• Xamarin.UITest• Frank• KIF (Keep it functional)
47
Selenium: cross platform, existing know how
•Android• selendroid• Appium
• iOS• Appium• ios-driver
48
Appium (www.appium.io)
• Provides WebDriver API (JSON Wire Protocol)•Uses native UI Automation library• Can test native & hybrid Apps, mobile web apps• Clients for C#, Java, Ruby, Python, PHP, JavaScript•Works with devices and emulators
49
Examples
50
Automation with SpecFlow and Appium
51
SpecFlow & Appium
•Write acceptance criteria independent of the platform•Write bindings against a Selenium- like- API• Run it on a devices or emulator
52
Demo
53
SpecFlow automationon the
controller layer
54
Different levels of automation
Controller
Business Layer
Data Layer
Model
View
Deviceautomation
Trigger behaviourthrough controller
Assert behaviouron model, db, ..
Setup pre-conditionsthrough serviceinterfaces
Out-of-process
In-process
55
How?
• Reuse the same feature files•Write new bindings against the
Controller/ViewModel/…
56
Demo
57
Faster feedback from tests
UI Automation Test Controller Test
58
Why is it faster?
•No overhead for deploying and starting the app•No need to wait for UI animations to finish•No delays for searching UI controls
59
Advantages Controller Testing
• Faster
• Better testable App architecture (Xamarin.Forms & MVVM)
60
Controller tests cannot fully replace UI tests
•UI tests are still needed• Diversity of mobile devices• Changes of manufacturer• Different Hardware Screen sizes Densities …
•Automated UI tests are still faster than manual testing
61
Conclusions
62
Excuses for avoiding mobile test automation
• “This is just a small app”• “Only complex end-end tests would make sense”• “Maintaining automated tests is too costly”
•Diversity of devices• Small apps also run into regressions• Bad feedback on app store cannot be undone
63
Automated testing in mobile App development
• Business readable automated checks• specification of acceptance criteria• test first development
• Testability is an ongoing architectural concern• every new feature needs to be testable• not every acceptance criterion needs to be tested through UI
• Automated tests don’t replace manual testing entirely• exploratory testing• assertions that are hard to automate• …
64
Questions and further information
Andreas Willich ([email protected])Christian Hassa ([email protected])
Top Related