© mutualmobile

SYSTEM TESTING WITH ROBOTIUMWhy, What, When, Where and How

© mutualmobile

WHO AM I?

Elliott Chenger Android Engineer @ Mutual Mobile

@echenger erchenger• Likes

• Long walks on the beach

• Everything Android • Ice Hockey • Auburn • Cars

• Dislikes • the F word

(fragmentation) • when carriers delay

software updates • King Joffery from

GOT • I-35 traffic

© mutualmobile

AGENDA• Brief overview of different types

of Testing

• What value does system testing add?

• Different tools for system testing

• What we chose and why.

• Do’s and Don’ts to system testing

• How to write system tests

• Live Demo

• Q&A

© mutualmobile

DIFFERENT TYPES OF TESTINGWhite Box

• Tests code for precision and correctness

• Tester understands and is testing internal structure and design of software

• Generally Unit and Integration testing

Black Box

• Software internal structure is not known to the tester

• Mainly done through acceptance and system testing

• Tests are based on requirements

© mutualmobile

WHAT IS SYSTEM TESTING?

© mutualmobile

SYSTEM TESTING • Subset of Black Box Testing

• Tests software as a whole to make sure it meets specified requirements

• Defined by acceptance criteria

• Mimics the human

© mutualmobile

WHY SHOULD A DEVELOPER WRITE SYSTEM TESTS?

© mutualmobile

© mutualmobile

SYSTEM TESTING TOOLS

• Robotium

• Espresso

• Monkey Talk

• UIAutomator

© mutualmobile

WHY WE CHOSE ROBOTIUM?

• Written in Java and structured like JUnit tests

• Bootstrapping was easy

• Allows us to write tests that mimic what a human does

• Strong community support

© mutualmobile

PROCESS/ACTIVITIES• The project kicks off and a full audit of all current system testing tools is performed.

• After a month of reviewing various tools Robotium becomes the clear selection among all of the stakeholders.

• A machine was purchased to be used as our testing server.

• The script and boiler plate for testing was created.

• Through trial and error our script and the server became more stable.

• We switched from emulators to physical devices and we decided that Flow testing will be more beneficial instead of screen testing.

© mutualmobile

• Worked with QA to identify how we are going to capture tests and communicate what should and shouldn’t be tested.

• We started a pilot program for Robotium on two major projects.

• Much was learned about how we actually write these tests.

• Now we are going through the process of educating and advocating our department to add system testing into our normal development process

© mutualmobile

LESSONS LEARNED

© mutualmobile

AUTOMATELet Continuous Integration systems do

the heavy lifting for you. This will help you catch issues before it gets into QA’s

hands.

© mutualmobile

USE PHYSICAL DEVICES

We have found through testing that Android emulators are simply slow and unreliable. Another benefit is being able

to test on different device subsets.

© mutualmobile

WRITE TESTS THAT MIMIC A HUMAN

Put yourself in the user’s shoes, don’t get hung up testing pieces of the UI directly. Thinking this way makes your tests more stable and more valuable to QA and your

customer.

© mutualmobile

DON’T STRIVE FOR 100% COVERAGEStrive for good tests, not a certain

quantity of test coverage. Beware of writing tests just to have tests. Like most things System Testing has a time and a

place.

© mutualmobile

SCREEN TESTING VS

FLOW TESTING

© mutualmobile

SCREEN TESTING

© mutualmobile

FLOW TESTING

© mutualmobile

CODEThe fun part

© mutualmobile

OUR EXAMPLE APP

© mutualmobile

AGENDA• Setting up Robotium in a project

• Organizing your tests

• Boilerplate code

• Writing tests

• Running Tests

• Live Demo

© mutualmobile

SETTING UP ROBOTIUM WITH ECLIPSE• Download jar

https://code.google.com/p/robotium/

© mutualmobile

SETTING UP ROBOTIUM GENERIC

• Download jarhttps://code.google.com/p/robotium/

• Add the jar to your classpath

© mutualmobile

SETTING UP THE MANIFEST

© mutualmobile

BOILER PLATE

© mutualmobile

WRITING YOU FIRST TEST

© mutualmobile

BEFORE WE START DO’S AND DON’TS• DO write tests that mimic a human.

• DO write your tests to encompass the overall acceptance criteria

• Assert DO NOT assume (also follow assertions with detailed error messages)

• DO NOT depend on time.Example: Don’t assume a REST call will always take 2 seconds.

© mutualmobile

LET’S TEST A STORY• User Story: As a user I would like to be able to login to the app, so that I can see the next

screen

• Acceptance Criteria

• Button for logging in

• A place for the user to enter their name for logging in.

• Once they enter their name and click the log in button they go to the next screen.

• If they don’t enter a name they get an error dialog

© mutualmobile

SO FAR

• We have written tests for the happy path for testing login.

• We have an Activity that has visuals but nothing else in it.

© mutualmobile

LET’S NOW RUN THE TESTS

© mutualmobile

NOW CODE YOUR FEATURE TO MAKE THE TESTS GO GREEN

© mutualmobile

NOW WE CAN WRITE SOME SAD PATH TESTS

© mutualmobile

NOW

• We have written happy path tests and they are successful.

• We just finished writing our sad path tests and they should not pass.

• We have an Activity that has visuals and some of our interactions but we still need to code the rest of the sad path test.

© mutualmobile

NOW CODE YOUR TESTS TO GO GREEN AND YOU ARE DONE!

© mutualmobile

LIVE DEMO

© mutualmobile

WHATS NEXT?

• Go explore: https://code.google.com/p/robotium/

• Download the example app code for this presentation: http://goo.gl/WPkwNE

• Have questions or feedback? elliott.chenger@mutualmobile.com or @echenger

© mutualmobile

Q&A