Case studies for GUI test automation Alexandra...
Transcript of Case studies for GUI test automation Alexandra...
![Page 1: Case studies for GUI test automation Alexandra Schladebeckwiki.eclipse.org/images/0/00/Quality_assurance_for_mobile_applications.pdfApps are testable! Similar concepts, fewer components](https://reader035.fdocuments.net/reader035/viewer/2022081522/5fc4a7d2415d3c3d115ad6bd/html5/thumbnails/1.jpg)
Version 2.5 © Bredex GmbH
Quality assurance for mobile applications Case studies for GUI test automation
Alexandra Schladebeck
![Page 2: Case studies for GUI test automation Alexandra Schladebeckwiki.eclipse.org/images/0/00/Quality_assurance_for_mobile_applications.pdfApps are testable! Similar concepts, fewer components](https://reader035.fdocuments.net/reader035/viewer/2022081522/5fc4a7d2415d3c3d115ad6bd/html5/thumbnails/2.jpg)
Agenda
The history
The new questions
The candidates
Our experiences
Results and conclusion
23.09.2013 2 Eclipse Testing Day 2013
![Page 3: Case studies for GUI test automation Alexandra Schladebeckwiki.eclipse.org/images/0/00/Quality_assurance_for_mobile_applications.pdfApps are testable! Similar concepts, fewer components](https://reader035.fdocuments.net/reader035/viewer/2022081522/5fc4a7d2415d3c3d115ad6bd/html5/thumbnails/3.jpg)
The (hi)story
23.09.2013 Eclipse Testing Day 2013 3
![Page 4: Case studies for GUI test automation Alexandra Schladebeckwiki.eclipse.org/images/0/00/Quality_assurance_for_mobile_applications.pdfApps are testable! Similar concepts, fewer components](https://reader035.fdocuments.net/reader035/viewer/2022081522/5fc4a7d2415d3c3d115ad6bd/html5/thumbnails/4.jpg)
The (hi)story
23.09.2013 Eclipse Testing Day 2013 4
![Page 5: Case studies for GUI test automation Alexandra Schladebeckwiki.eclipse.org/images/0/00/Quality_assurance_for_mobile_applications.pdfApps are testable! Similar concepts, fewer components](https://reader035.fdocuments.net/reader035/viewer/2022081522/5fc4a7d2415d3c3d115ad6bd/html5/thumbnails/5.jpg)
The aims
Evaluate feasibility of mobile GUI test automation Functional and technical perspective
Find out how continuous integration and testing can be implemented
Analyse cross-platform testing in a mobile context Write once, run anywhere
Make informed decisions on mobile testing strategy
23.09.2013 Eclipse Testing Day 2013 5
![Page 6: Case studies for GUI test automation Alexandra Schladebeckwiki.eclipse.org/images/0/00/Quality_assurance_for_mobile_applications.pdfApps are testable! Similar concepts, fewer components](https://reader035.fdocuments.net/reader035/viewer/2022081522/5fc4a7d2415d3c3d115ad6bd/html5/thumbnails/6.jpg)
Candidate 1: Customer application
Datawarehouse for competition analysis Enterprise desktop application
Go mobile Conference catalog that uses data Extra comfort and features (bookmarking, quick navigation) Target group = managers at automobile congresses
iOS only Version 5 6 7
Release date *must* be kept
23.09.2013 Eclipse Testing Day 2013 6
![Page 7: Case studies for GUI test automation Alexandra Schladebeckwiki.eclipse.org/images/0/00/Quality_assurance_for_mobile_applications.pdfApps are testable! Similar concepts, fewer components](https://reader035.fdocuments.net/reader035/viewer/2022081522/5fc4a7d2415d3c3d115ad6bd/html5/thumbnails/7.jpg)
Candidate 2: Continuous Integration
We test our test tool with our test tool…
Testing support for widgets requires example applications Android, iOS, Windows 8 (modernUI) Perform actions and checks
Functionally uninteresting
Candidates for evaluating continuous
integration
23.09.2013 Eclipse Testing Day 2013 7
![Page 8: Case studies for GUI test automation Alexandra Schladebeckwiki.eclipse.org/images/0/00/Quality_assurance_for_mobile_applications.pdfApps are testable! Similar concepts, fewer components](https://reader035.fdocuments.net/reader035/viewer/2022081522/5fc4a7d2415d3c3d115ad6bd/html5/thumbnails/8.jpg)
Candidate 3: In-house application
In-house application Completely new development
For taking minutes in meetings
Cross-platform development (Xamarin)
Cross-platform automated testing
23.09.2013 Eclipse Testing Day 2013 8
![Page 9: Case studies for GUI test automation Alexandra Schladebeckwiki.eclipse.org/images/0/00/Quality_assurance_for_mobile_applications.pdfApps are testable! Similar concepts, fewer components](https://reader035.fdocuments.net/reader035/viewer/2022081522/5fc4a7d2415d3c3d115ad6bd/html5/thumbnails/9.jpg)
Experiences : Customer (iOS) project
Process Manual test scripts specified as automated tests Automated test started manually Testable version of the app provided manually New productive data available very late
Functional aspects Required features could be tested with automated test Regression testing for new versions (iOS 6)
Technical aspects Some modifications necessary due to custom components
23.09.2013 Eclipse Testing Day 2013 9
![Page 10: Case studies for GUI test automation Alexandra Schladebeckwiki.eclipse.org/images/0/00/Quality_assurance_for_mobile_applications.pdfApps are testable! Similar concepts, fewer components](https://reader035.fdocuments.net/reader035/viewer/2022081522/5fc4a7d2415d3c3d115ad6bd/html5/thumbnails/10.jpg)
Demo
23.09.2013 Eclipse Testing Day 2013 10
![Page 11: Case studies for GUI test automation Alexandra Schladebeckwiki.eclipse.org/images/0/00/Quality_assurance_for_mobile_applications.pdfApps are testable! Similar concepts, fewer components](https://reader035.fdocuments.net/reader035/viewer/2022081522/5fc4a7d2415d3c3d115ad6bd/html5/thumbnails/11.jpg)
Experiences: Continuous Integration
Hudson
Build
Deploy
setup
Linux
Mac OSX
Win7
Win Vista
Win8
…
23.09.2013 Eclipse Testing Day 2013 11
![Page 12: Case studies for GUI test automation Alexandra Schladebeckwiki.eclipse.org/images/0/00/Quality_assurance_for_mobile_applications.pdfApps are testable! Similar concepts, fewer components](https://reader035.fdocuments.net/reader035/viewer/2022081522/5fc4a7d2415d3c3d115ad6bd/html5/thumbnails/12.jpg)
Experiences: Continuous Integration
23.09.2013 Eclipse Testing Day 2013 12
Prepare Environment
“reset“ backend
setup test db build AUT
Prepare (virtual) test
instance start cleanup deploy
install configure
Run test(s) start AUT-Agent testexec
Finalize gather AUT logs
tear down AUT & env
& tests
![Page 13: Case studies for GUI test automation Alexandra Schladebeckwiki.eclipse.org/images/0/00/Quality_assurance_for_mobile_applications.pdfApps are testable! Similar concepts, fewer components](https://reader035.fdocuments.net/reader035/viewer/2022081522/5fc4a7d2415d3c3d115ad6bd/html5/thumbnails/13.jpg)
Experiences : Continuous Integration
iOS Mac machine XCode Simulator / Device Communication Deploy application (platform specific) Start AUT
Windows 8 (modernUI) Windows 8 machine Visual studio 2012 / 2013 Rights & signing AUT automatically startable
23.09.2013 Eclipse Testing Day 2013 13
![Page 14: Case studies for GUI test automation Alexandra Schladebeckwiki.eclipse.org/images/0/00/Quality_assurance_for_mobile_applications.pdfApps are testable! Similar concepts, fewer components](https://reader035.fdocuments.net/reader035/viewer/2022081522/5fc4a7d2415d3c3d115ad6bd/html5/thumbnails/14.jpg)
Experiences: Continuous Integration
Sandbox
Platform-dependent scripts necessary
Idiosyncrasies
Simulators alone not enough
Functional testing of Orientation changes GPS Battery Internet connection Access to external applications
23.09.2013 Eclipse Testing Day 2013 14
![Page 15: Case studies for GUI test automation Alexandra Schladebeckwiki.eclipse.org/images/0/00/Quality_assurance_for_mobile_applications.pdfApps are testable! Similar concepts, fewer components](https://reader035.fdocuments.net/reader035/viewer/2022081522/5fc4a7d2415d3c3d115ad6bd/html5/thumbnails/15.jpg)
Test results
23.09.2013 Eclipse Testing Day 2013 15
![Page 16: Case studies for GUI test automation Alexandra Schladebeckwiki.eclipse.org/images/0/00/Quality_assurance_for_mobile_applications.pdfApps are testable! Similar concepts, fewer components](https://reader035.fdocuments.net/reader035/viewer/2022081522/5fc4a7d2415d3c3d115ad6bd/html5/thumbnails/16.jpg)
Experiences: Cross-Platform Project
Cross-platform development
Test : write once, run anywhere
23.09.2013 Eclipse Testing Day 2013 16
![Page 17: Case studies for GUI test automation Alexandra Schladebeckwiki.eclipse.org/images/0/00/Quality_assurance_for_mobile_applications.pdfApps are testable! Similar concepts, fewer components](https://reader035.fdocuments.net/reader035/viewer/2022081522/5fc4a7d2415d3c3d115ad6bd/html5/thumbnails/17.jpg)
Cross-Platform
23.09.2013 Eclipse Testing Day 2013 17
RCP & GEF
SWT Swing mobile .NET HTML
iOS Android Windows Phone
![Page 18: Case studies for GUI test automation Alexandra Schladebeckwiki.eclipse.org/images/0/00/Quality_assurance_for_mobile_applications.pdfApps are testable! Similar concepts, fewer components](https://reader035.fdocuments.net/reader035/viewer/2022081522/5fc4a7d2415d3c3d115ad6bd/html5/thumbnails/18.jpg)
Cross-Platform
23.09.2013 Eclipse Testing Day 2013 18
SWT Swing mobile .NET HTML
iOS Android Windows Phone
concrete
abstract
RCP & GEF
![Page 19: Case studies for GUI test automation Alexandra Schladebeckwiki.eclipse.org/images/0/00/Quality_assurance_for_mobile_applications.pdfApps are testable! Similar concepts, fewer components](https://reader035.fdocuments.net/reader035/viewer/2022081522/5fc4a7d2415d3c3d115ad6bd/html5/thumbnails/19.jpg)
Cross-Platform
23.09.2013 19
RCP & GEF
SWT Swing mobile .NET HTML
concrete
abstract
iOS Android Windows Phone
Buttons
Labels
Lists
Text fields
Tables
…
Textual components
Gestures
Graphical components
Eclipse Testing Day 2013
![Page 20: Case studies for GUI test automation Alexandra Schladebeckwiki.eclipse.org/images/0/00/Quality_assurance_for_mobile_applications.pdfApps are testable! Similar concepts, fewer components](https://reader035.fdocuments.net/reader035/viewer/2022081522/5fc4a7d2415d3c3d115ad6bd/html5/thumbnails/20.jpg)
Cross-Platform
23.09.2013 20
RCP & GEF
SWT Swing mobile .NET HTML
concrete
abstract
iOS Android Windows Phone
Buttons
Labels
Lists
Text fields
Tables
…
Textual components
Gestures
Graphical components Test specification:
AUT-independent
Test execution: Correct UI-Driver
Eclipse Testing Day 2013
![Page 21: Case studies for GUI test automation Alexandra Schladebeckwiki.eclipse.org/images/0/00/Quality_assurance_for_mobile_applications.pdfApps are testable! Similar concepts, fewer components](https://reader035.fdocuments.net/reader035/viewer/2022081522/5fc4a7d2415d3c3d115ad6bd/html5/thumbnails/21.jpg)
The good news…
Different mobile UI toolkits use same/similar functional components Buttons, textfields, lists, tabbed components, combo
components
23.09.2013 Eclipse Testing Day 2013 21
![Page 22: Case studies for GUI test automation Alexandra Schladebeckwiki.eclipse.org/images/0/00/Quality_assurance_for_mobile_applications.pdfApps are testable! Similar concepts, fewer components](https://reader035.fdocuments.net/reader035/viewer/2022081522/5fc4a7d2415d3c3d115ad6bd/html5/thumbnails/22.jpg)
Example 1: Edit list
23.09.2013 Eclipse Testing Day 2013 22
![Page 23: Case studies for GUI test automation Alexandra Schladebeckwiki.eclipse.org/images/0/00/Quality_assurance_for_mobile_applications.pdfApps are testable! Similar concepts, fewer components](https://reader035.fdocuments.net/reader035/viewer/2022081522/5fc4a7d2415d3c3d115ad6bd/html5/thumbnails/23.jpg)
Example 2: Choose minute taker
23.09.2013 23 Eclipse Testing Day 2013
![Page 24: Case studies for GUI test automation Alexandra Schladebeckwiki.eclipse.org/images/0/00/Quality_assurance_for_mobile_applications.pdfApps are testable! Similar concepts, fewer components](https://reader035.fdocuments.net/reader035/viewer/2022081522/5fc4a7d2415d3c3d115ad6bd/html5/thumbnails/24.jpg)
Example 3: Enter participants
23.09.2013 24 Eclipse Testing Day 2013
![Page 25: Case studies for GUI test automation Alexandra Schladebeckwiki.eclipse.org/images/0/00/Quality_assurance_for_mobile_applications.pdfApps are testable! Similar concepts, fewer components](https://reader035.fdocuments.net/reader035/viewer/2022081522/5fc4a7d2415d3c3d115ad6bd/html5/thumbnails/25.jpg)
Cross-platform differences are desired
Differences between desktop versions = minimal
Differences between mobile clients = huge Features Workflows Use of hardware / software features Users are used to / expect these differences.
Differences between versions often large iOS 6/7 Android 2/4
23.09.2013 Eclipse Testing Day 2013 25
![Page 26: Case studies for GUI test automation Alexandra Schladebeckwiki.eclipse.org/images/0/00/Quality_assurance_for_mobile_applications.pdfApps are testable! Similar concepts, fewer components](https://reader035.fdocuments.net/reader035/viewer/2022081522/5fc4a7d2415d3c3d115ad6bd/html5/thumbnails/26.jpg)
Experiences: Cross-Platform Project
Cross-platform development
Test : write once, run anywhere
Technically possible
Unlikely to be used…
23.09.2013 Eclipse Testing Day 2013 26
![Page 27: Case studies for GUI test automation Alexandra Schladebeckwiki.eclipse.org/images/0/00/Quality_assurance_for_mobile_applications.pdfApps are testable! Similar concepts, fewer components](https://reader035.fdocuments.net/reader035/viewer/2022081522/5fc4a7d2415d3c3d115ad6bd/html5/thumbnails/27.jpg)
Results: GUI test automation y/n?
Should we automate mobile tests? Multiplication factor over devices New OS versions appear regularly No choice or control over runtime environment Quick feedback whether new version / feature is total fail Short release cycles for own software Changes happen that developers have no control over
23.09.2013 Eclipse Testing Day 2013 27
![Page 28: Case studies for GUI test automation Alexandra Schladebeckwiki.eclipse.org/images/0/00/Quality_assurance_for_mobile_applications.pdfApps are testable! Similar concepts, fewer components](https://reader035.fdocuments.net/reader035/viewer/2022081522/5fc4a7d2415d3c3d115ad6bd/html5/thumbnails/28.jpg)
Results: Writing automated tests for mobile
Apps are testable!
Similar concepts, fewer components
Extra hurdles More input from development team Additional tests necessary (GPS, Battery, Internet) Environment more complex
23.09.2013 Eclipse Testing Day 2013 28
![Page 29: Case studies for GUI test automation Alexandra Schladebeckwiki.eclipse.org/images/0/00/Quality_assurance_for_mobile_applications.pdfApps are testable! Similar concepts, fewer components](https://reader035.fdocuments.net/reader035/viewer/2022081522/5fc4a7d2415d3c3d115ad6bd/html5/thumbnails/29.jpg)
Results: Continuous integration for mobile
23.09.2013 Eclipse Testing Day 2013 29
Develop
Build
Test
Analyse
Feedback
![Page 30: Case studies for GUI test automation Alexandra Schladebeckwiki.eclipse.org/images/0/00/Quality_assurance_for_mobile_applications.pdfApps are testable! Similar concepts, fewer components](https://reader035.fdocuments.net/reader035/viewer/2022081522/5fc4a7d2415d3c3d115ad6bd/html5/thumbnails/30.jpg)
Results: Cross platform
Write once, run anywhere is possible
But seems unlikely
How much should / can development be influenced by test factors?
23.09.2013 Eclipse Testing Day 2013 30
![Page 31: Case studies for GUI test automation Alexandra Schladebeckwiki.eclipse.org/images/0/00/Quality_assurance_for_mobile_applications.pdfApps are testable! Similar concepts, fewer components](https://reader035.fdocuments.net/reader035/viewer/2022081522/5fc4a7d2415d3c3d115ad6bd/html5/thumbnails/31.jpg)
Results: Mobile test strategy
23.09.2013 Eclipse Testing Day 2013 31
![Page 32: Case studies for GUI test automation Alexandra Schladebeckwiki.eclipse.org/images/0/00/Quality_assurance_for_mobile_applications.pdfApps are testable! Similar concepts, fewer components](https://reader035.fdocuments.net/reader035/viewer/2022081522/5fc4a7d2415d3c3d115ad6bd/html5/thumbnails/32.jpg)
Thank you!
23.09.2013 32
Alexandra Schladebeck
@alex_schl
http://testing.bredex.de
blog.bredex.de
www.bredex.de
www.guidancer.de
www.eclipse.org/jubula
Eclipse Testing Day 2013