automated acceptance testing of mobile...
Transcript of automated acceptance testing of mobile...
![Page 1: automated acceptance testing of mobile appsgotocon.com/dl/goto-aar-2012/slides/KarlKrukow_Calabash... · 2012-10-02 · The acceptance testing story for mobile apps 5 • As realistic](https://reader030.fdocuments.net/reader030/viewer/2022040610/5ed0b3cd146b1a6dad4b9fea/html5/thumbnails/1.jpg)
automated acceptance testing of mobile apps
Karl Krukow,CTO, LessPainful
Goto Aarhus, [email protected], @karlkrukow
1
tirsdag den 2. oktober 12
![Page 2: automated acceptance testing of mobile appsgotocon.com/dl/goto-aar-2012/slides/KarlKrukow_Calabash... · 2012-10-02 · The acceptance testing story for mobile apps 5 • As realistic](https://reader030.fdocuments.net/reader030/viewer/2022040610/5ed0b3cd146b1a6dad4b9fea/html5/thumbnails/2.jpg)
Agenda
• Automated testing for mobile
• desirable properties for an acceptance testing tool
• Introduce Calabash via Short Live Demos
• Focus on iOS
• LessPainful: test service and device cloud
2
tirsdag den 2. oktober 12
![Page 3: automated acceptance testing of mobile appsgotocon.com/dl/goto-aar-2012/slides/KarlKrukow_Calabash... · 2012-10-02 · The acceptance testing story for mobile apps 5 • As realistic](https://reader030.fdocuments.net/reader030/viewer/2022040610/5ed0b3cd146b1a6dad4b9fea/html5/thumbnails/3.jpg)
Professional practices?• Invest the time once! Get the practices going.
Examples:
• Unit testing
• Acceptance tests (see this talk!).
• Continuous build
• Continuous unit test, functional test (code coverage).
• Static analysis using clang/FindBugs.
• Automated deployment to users via in-app updates or link in email.
• Automatic Crash/Error reporting.
• ...
3
Go talk t
o the g
uys fro
m eBay
Classifi
eds G
roup
DenBlåA
vis+BilB
asen
tirsdag den 2. oktober 12
![Page 4: automated acceptance testing of mobile appsgotocon.com/dl/goto-aar-2012/slides/KarlKrukow_Calabash... · 2012-10-02 · The acceptance testing story for mobile apps 5 • As realistic](https://reader030.fdocuments.net/reader030/viewer/2022040610/5ed0b3cd146b1a6dad4b9fea/html5/thumbnails/4.jpg)
The QA story for mobile
4
tirsdag den 2. oktober 12
![Page 5: automated acceptance testing of mobile appsgotocon.com/dl/goto-aar-2012/slides/KarlKrukow_Calabash... · 2012-10-02 · The acceptance testing story for mobile apps 5 • As realistic](https://reader030.fdocuments.net/reader030/viewer/2022040610/5ed0b3cd146b1a6dad4b9fea/html5/thumbnails/5.jpg)
The acceptance testing story for mobile apps
5
• As realistic an environment as practically possible.
• simulators/emulators are good, but not enough!
• Many devices, screens, resolutions, OS’es, CPUs
• Varying Conditions (Settings, Network, Storage...)
• Often a manual process: repetitive, expensive
• Regressions, e.g., app crashes
• Visual appearance of screens matter (alot!)
• User experience, Design guidelines, branding,...
tirsdag den 2. oktober 12
![Page 6: automated acceptance testing of mobile appsgotocon.com/dl/goto-aar-2012/slides/KarlKrukow_Calabash... · 2012-10-02 · The acceptance testing story for mobile apps 5 • As realistic](https://reader030.fdocuments.net/reader030/viewer/2022040610/5ed0b3cd146b1a6dad4b9fea/html5/thumbnails/6.jpg)
Significant Differences
• Problems that arise from differences are significant
• Crashes
• Graphical Errors
• Wrong Computation
• UI Components
• Inaccessible text, buttons...
tirsdag den 2. oktober 12
![Page 7: automated acceptance testing of mobile appsgotocon.com/dl/goto-aar-2012/slides/KarlKrukow_Calabash... · 2012-10-02 · The acceptance testing story for mobile apps 5 • As realistic](https://reader030.fdocuments.net/reader030/viewer/2022040610/5ed0b3cd146b1a6dad4b9fea/html5/thumbnails/7.jpg)
• Minimize distance between use cases and actual test code (DSLs?).
• Expressive and efficient to write & maintain.
• Extensible (open vs closed source)
• High-level, declarative (robustness:“minor” UI changes).
• Support testing in realistic environments (real devices, on multiple OS versions, languages).
• Support Continuous integration.
7
Automated acceptance testing desiderata
tirsdag den 2. oktober 12
![Page 8: automated acceptance testing of mobile appsgotocon.com/dl/goto-aar-2012/slides/KarlKrukow_Calabash... · 2012-10-02 · The acceptance testing story for mobile apps 5 • As realistic](https://reader030.fdocuments.net/reader030/viewer/2022040610/5ed0b3cd146b1a6dad4b9fea/html5/thumbnails/8.jpg)
tirsdag den 2. oktober 12
![Page 9: automated acceptance testing of mobile appsgotocon.com/dl/goto-aar-2012/slides/KarlKrukow_Calabash... · 2012-10-02 · The acceptance testing story for mobile apps 5 • As realistic](https://reader030.fdocuments.net/reader030/viewer/2022040610/5ed0b3cd146b1a6dad4b9fea/html5/thumbnails/9.jpg)
Calabash• Vision: Cross-platform Automated Acceptance Testing
• One interface: Cucumber, for Android and iOS.
• BDD. Business understandable spec, executable as tests
• Reuse of Cucumber features across platforms possible.
• Runs on physical devices and simulators/emulators.
• Support for native and hybrid apps (embedded webviews)
• Free, open source (Eclipse Public License)
• with optional commercial extras support, training, consulting, device cloud, private device cloud
9
tirsdag den 2. oktober 12
![Page 10: automated acceptance testing of mobile appsgotocon.com/dl/goto-aar-2012/slides/KarlKrukow_Calabash... · 2012-10-02 · The acceptance testing story for mobile apps 5 • As realistic](https://reader030.fdocuments.net/reader030/viewer/2022040610/5ed0b3cd146b1a6dad4b9fea/html5/thumbnails/10.jpg)
LessPainfulTest Execution Service
• Execute Calabash tests in parallel on many devices, OS’es, languages.
• Visual test reports.
• Comparison across models and operating systems.
• Authentic: Not jailbroken/rooted, supports both iOS and Android devices.
• Continuous integration: lesspainful app.ipa KEY
tirsdag den 2. oktober 12
![Page 11: automated acceptance testing of mobile appsgotocon.com/dl/goto-aar-2012/slides/KarlKrukow_Calabash... · 2012-10-02 · The acceptance testing story for mobile apps 5 • As realistic](https://reader030.fdocuments.net/reader030/viewer/2022040610/5ed0b3cd146b1a6dad4b9fea/html5/thumbnails/11.jpg)
tirsdag den 2. oktober 12
![Page 12: automated acceptance testing of mobile appsgotocon.com/dl/goto-aar-2012/slides/KarlKrukow_Calabash... · 2012-10-02 · The acceptance testing story for mobile apps 5 • As realistic](https://reader030.fdocuments.net/reader030/viewer/2022040610/5ed0b3cd146b1a6dad4b9fea/html5/thumbnails/12.jpg)
tirsdag den 2. oktober 12
![Page 13: automated acceptance testing of mobile appsgotocon.com/dl/goto-aar-2012/slides/KarlKrukow_Calabash... · 2012-10-02 · The acceptance testing story for mobile apps 5 • As realistic](https://reader030.fdocuments.net/reader030/viewer/2022040610/5ed0b3cd146b1a6dad4b9fea/html5/thumbnails/13.jpg)
Calabash iOS: by way of Demo
13
tirsdag den 2. oktober 12
![Page 14: automated acceptance testing of mobile appsgotocon.com/dl/goto-aar-2012/slides/KarlKrukow_Calabash... · 2012-10-02 · The acceptance testing story for mobile apps 5 • As realistic](https://reader030.fdocuments.net/reader030/viewer/2022040610/5ed0b3cd146b1a6dad4b9fea/html5/thumbnails/14.jpg)
The Setup- install and run
- Cucumber Basics- predefined steps
14
tirsdag den 2. oktober 12
![Page 15: automated acceptance testing of mobile appsgotocon.com/dl/goto-aar-2012/slides/KarlKrukow_Calabash... · 2012-10-02 · The acceptance testing story for mobile apps 5 • As realistic](https://reader030.fdocuments.net/reader030/viewer/2022040610/5ed0b3cd146b1a6dad4b9fea/html5/thumbnails/15.jpg)
Calabash Architecture
JSON over HTTP
Your App
Calabash Servers (iOS, Android)Calabash Clients
Calabash Android
Calabash iOS
Calabash JVM
...
tirsdag den 2. oktober 12
![Page 16: automated acceptance testing of mobile appsgotocon.com/dl/goto-aar-2012/slides/KarlKrukow_Calabash... · 2012-10-02 · The acceptance testing story for mobile apps 5 • As realistic](https://reader030.fdocuments.net/reader030/viewer/2022040610/5ed0b3cd146b1a6dad4b9fea/html5/thumbnails/16.jpg)
Cucumber + Calabash
16
Step DefsSupport
FeaturesFeaturesFeatures
Gherkin Ruby
CalabashClients
Uses
Cucumber Tool
Executes
ProducesTest Results
tirsdag den 2. oktober 12
![Page 17: automated acceptance testing of mobile appsgotocon.com/dl/goto-aar-2012/slides/KarlKrukow_Calabash... · 2012-10-02 · The acceptance testing story for mobile apps 5 • As realistic](https://reader030.fdocuments.net/reader030/viewer/2022040610/5ed0b3cd146b1a6dad4b9fea/html5/thumbnails/17.jpg)
Calabash iOS Basics- interactive/exploratory development
- core API- screenshots
17
tirsdag den 2. oktober 12
![Page 18: automated acceptance testing of mobile appsgotocon.com/dl/goto-aar-2012/slides/KarlKrukow_Calabash... · 2012-10-02 · The acceptance testing story for mobile apps 5 • As realistic](https://reader030.fdocuments.net/reader030/viewer/2022040610/5ed0b3cd146b1a6dad4b9fea/html5/thumbnails/18.jpg)
Calabash iOS Advanced- record/transform/playback
- web view support- location support
18
tirsdag den 2. oktober 12
![Page 19: automated acceptance testing of mobile appsgotocon.com/dl/goto-aar-2012/slides/KarlKrukow_Calabash... · 2012-10-02 · The acceptance testing story for mobile apps 5 • As realistic](https://reader030.fdocuments.net/reader030/viewer/2022040610/5ed0b3cd146b1a6dad4b9fea/html5/thumbnails/19.jpg)
Roadmap for Calabash• Support for dynamic query and action on Android.
• Better support for reuse of test code across platforms
• More uniform APIs (old apis will not dissappear)
• Naming and conceptual model
• Protocol-level
• Support platform specific actions, where useful
• Support for JVM, Better WebView support, Better tooling.
• Clojure POC already exists - supports any JVM language
• Your inputs!
19
tirsdag den 2. oktober 12
![Page 20: automated acceptance testing of mobile appsgotocon.com/dl/goto-aar-2012/slides/KarlKrukow_Calabash... · 2012-10-02 · The acceptance testing story for mobile apps 5 • As realistic](https://reader030.fdocuments.net/reader030/viewer/2022040610/5ed0b3cd146b1a6dad4b9fea/html5/thumbnails/20.jpg)
References• https://github.com/calabash
• https://github.com/calabash/calabash-ios
• https://github.com/calabash/calabash-ios/wiki
• https://github.com/calabash/calabash-ios-server
• https://github.com/calabash/calabash-android
• http://blog.lesspainful.com/
• https://www.lesspainful.com/
20
tirsdag den 2. oktober 12
![Page 21: automated acceptance testing of mobile appsgotocon.com/dl/goto-aar-2012/slides/KarlKrukow_Calabash... · 2012-10-02 · The acceptance testing story for mobile apps 5 • As realistic](https://reader030.fdocuments.net/reader030/viewer/2022040610/5ed0b3cd146b1a6dad4b9fea/html5/thumbnails/21.jpg)
Making app testing less painful...Please contact us with any questions:
[email protected]@lesspainful.com - iOS
[email protected] - Android
http://www.lesspainful.com
Questions?
tirsdag den 2. oktober 12