Mobile Testing, That's Just a Smaller Screen, Right? Stephen Janaway

© Stephen Janaway 2014

Mobile Testing, That’s Just a Smaller Screen, Right?

Stephen Janaway


Who am I?Stephen Janaway

• 14 years experience in mobile software testing.

• Mobile devices.• Mobile networks.• Mobile applications.• Have worked for companies such as

Ericsson, Motorola and Nokia.

• Work for the NET-A-PORTER GROUP.

• www.stephenjanaway.co.uk• @stephenjanaway


www.testinginthepub.com@testinginthepubSearch for “Testing In The Pub’” on iTunes


Today

• An overview of mobile.• Why is mobile different?• What makes mobile testing different?• The future of mobile testing.• Your mobile testing future.


Mobile devices are

• Smart Phones• Feature Phones• Tablets• E-readers• Big screens, small screens, touch screens and

keypads.


And now…

• Fitness devices.• Smart watches.• Wearable devices.• Beacons.• Household gadgets.


Mobile devices are much more• Alarm Clocks• Maps• Link to loved ones• Hand held shops• Tickets• Music Players• Books• Games Players• Cameras• ……

• These are key experiences.

My alarm didn’t go off…

I’m lost


WHY IS MOBILE DIFFERENT?

MOBILE USERS ARE MORE EMOTIONALLY ATTACHED

MOBILE USERS ARE NOT SECOND CLASS CITIZENS


?

BUT QUALITY IS CURRENTLY STILL NOT GREAT…

AND USERS INCREASINGLY DON’T TOLERATE THIS

MOBILE USERS CAN LEAVE BAD FEEDBACK MUCH MORE QUICKLY


What Makes Mobile Testing Different?

• Many platforms.• Many OS versions in the field.• Many different hardware configurations.• Devices with limited memory and processing

capability.• Limited battery life.• Network connectivity.• Application stores.• Low maturity of automation solutions.

Usability Matters

• Usability is more important in the mobile world:– Smaller screen.– More difficult to interact with and typically used one

handed.


Brand Consistency Matters

• Think about what a mobile user would want.• Something familiar.• Taking advantage of mobile functionality:– Location.– Mobile search.– M-commerce.

• A design that can be viewed on a small screen and in bright light.


Mobile Introduces a Whole New Language


Mobile Introduces New Scenarios

• Download and install.• Un-install.• Upgrade.• Rotation.• Interaction with Sensors.• Location.• External media.

Mobiles Are Complicated

• While your application runs, the phone is:– Talking to the network– Working out where you are– Maybe receiving calls, texts, Facebook and Twitter

updates.

• What happens when you have your application running, and then get a call, and a text, whilst uploading video to YouTube? Test for it and find out.


Performance Matters

• Limited processor speeds.• Limited on-board memory.• Typically not turned off very often.• Don’t forget - most apps are just clients to

web services. Use your web performance testing skills.


Data Matters

• Check for excessive data consumption.• Using on device data measuring tools.• Using a proxy.


The Mobile World


Subscriptions vs. People


Different OS’s


Different Mobile Browsers


The FutureOf Mobile Testing


More Devices = More Testing (?)


OS’s

Source: gs.statcounter.com

Android versions

Source: developer.android.com


Screen Sizes


Deciding What to Test

• Know your predicated customers (not your current ones).

• Find out what OS updates are coming up.• Get as much data on device usage that you

can:– In app analytics.– Web searches.– Similar apps.

Options

• Don’t use them at all – use simulators or user agents.

• Buy all or some of them.• Borrow all or some of them.• Rent all or some of them.• Use a cloud based service (Perfecto Mobile,

DeviceAnywhere).



My Recommendation

• Buy 2 from the top, 2 from the middle and 2 from the bottom.

• Understand what your customers use or will use.

• Test first on simulators not only on simulators.• Use cloud services if needs be, to fill any gaps.

• Never just use browser emulation for mobile websites.


Open Device Labs

• Borrow devices, normally for free.

• http://opendevicelab.com • Open Device

Labs = shared community pools of internet connected devices for testing purposes.

http://opendevicelab.com/

http://opendevicelab.com/


Making More Use of Your Users


Dog fooding/ Alpha/ Beta

• Get an internal user group going.• Distribute app via TestFlight, TestFairy or

similar.• Alpha/ Beta test via app stores.


More Financial Transactions Will Happen On Mobile


Security

• Big recent rise in the amount of malware targeted at mobile devices.


Perception of Security


Smartphones Everywhere= Your App Everywhere


Accessibility

• How readable is the app/ website with a large font?

• Is there voice activation / control?• Are there text alternatives to images?• Tools:– TalkBack (Android)– Voice Over (iOS)


Voice Activation

• Siri – iOS• Google Now – Android• Also on Windows Phone and Blackberry.• Is the app voice activated? • What happens when you activate voice with

the app running?


Your Mobile Testing Future

You Can Avoid The Common Mobile Testing Mistakes


Only Testing On Simulators


Thinking a Mobile Device Lab Is a One Time Investment


Staying In The Office

Don’t Just Test Here…

On Here, and At Here…

Out Here…

In Here…


Forgetting About Battery Life


Battery Life


Ignoring OS Updates


OS Updates

• Always regression test your app or website before an OS update.

• Get on the developer program for early OS versions.

• Users update more quickly on iOS than Android.• Carriers often control update cycles. • Decide beforehand how many OS version you

will/ can support.


Thinking Mobile Test Automation Is Like Web Test Automation


Automation Suggestions

• Determine if you require cross platform support.

• Avoid tools that use embedded servers or agents – they alter the production code.

• Look for tools with active online communities.• Consider the future.


My Picks

• Appium:– Cross platform.– Well supported user community.– About to be more closely integrated in Selenium 3.

• iOS Driver:– Well supported user community.– About to be more closely integrated in Selenium 3.

• Calabash:– Well supported user community.– Heavily used at the BBC.

So……


Mobile Test Strategy – Top 5 Tips

1. Understand the potential user:– Which platforms and devices will they use?– How will they use the app or website?

2. Understand what OS versions are being used and when new ones are coming.

3. Test using simulators, real devices and users.4. Understand the cost of automation:

– It is where desktop automation was 6 years ago.– Best combined with extensive unit and integration testing.

5. Use an exploratory approach to manual testing:– Mobile device projects are typically short.

“Be proud, your work will soon be in the hands of millions”

Stephen Janaway

www.stephenjanaway.co.uk @stephenjanaway www.testinginthepub.com

Questions?

Mobile Testing, That's Just a Smaller Screen, Right? Stephen Janaway

Software

Transcript of Mobile Testing, That's Just a Smaller Screen, Right? Stephen Janaway