Why apps-succeed-wpr-mastercode.vn

18
WHY APPS SUCCEED Four Ways Enterprises Can Overcome Mobile and Web Testing Challenges to Achieve Digital Excellence Based on insights from 1,000 enterprise web and mobile professionals

Transcript of Why apps-succeed-wpr-mastercode.vn

Page 1: Why apps-succeed-wpr-mastercode.vn

WHY APPSSUCCEEDFour Ways Enterprises Can Overcome Mobile and Web Testing Challenges to Achieve Digital Excellence

Based on insights from 1,000 enterprise web and mobile professionals

Page 2: Why apps-succeed-wpr-mastercode.vn

2

“While users demand new mobile features and functionality, development and testing teams are struggling to keep up.”

How do successful brands drive revenue, attract and retain loyal customers, and get ahead of the competition?

Delivering exceptional digital experiences across web and mobile channels has become a critical path to achieving these goals, and separates the leaders from laggards. But creating a reliable and engaging digital experience for customers is also an increasing challenge for enterprises. While users demand new mobile features and functionality, development and testing teams are struggling to keep up with the growing proliferation of mobile devices, the speed at which new devices and operating system versions are launched, and the variety of user conditions that need to be tested.

To provide guidance and insight, we sent a survey about digital quality strategies to enterprise-level web and mobile professionals from various industries and geographies and received over 1,000 responses. The majority of respondents were in quality assurance, management and developer roles. Based on the survey results, we’ve outlined four key obstacles to digital excellence and our recommendations for overcoming them.

Executive Summary

Key takeaways from the survey findings:

• A unified digital development and testing strategy is imperative, driving a strong trend toward responsive websites and a single unified team for mobile and desktop web testing.

• Organizations still face significant challenges releasing new features and functionality in a timely fashion and delivering a consistent experience across devices and operating systems.

• Greater adoption of test automation, better testing of real user conditions, and having an efficient test lab strategy are major opportunities for organizations to improve the digital experience.

Page 3: Why apps-succeed-wpr-mastercode.vn

3

Digital Users Straddle Multiple Devices

Today’s digital consumer uses multiple devices to pay bills, shop, book flights, watch movies, and more. Forty percent of eCommerce transactions involve multiple devices along the path to purchase, according to research and advertising firm Criteo. While the desktop browser is still widely used for common online activities, the rise of mobile usage in terms of the time spent as part of the digital experience is undeniable.

Desktop is also the dominant choice for many important categories, including business/finance and entertainment/news.

0 20 40 60 80 100Source: comScore Media Matrix Multi-Platform, U.S., Dec 2015

69%

58%

51%

46%

44%

39%

38%

36%

23%

21%

12%

8%

6%

62%

64%

77%

79%

88%

92%

94%

56%

61%

54%

49%

42%

31%Portals

Desktop

Business/Finance

Entertainment - News

Health - Information

News/Information

Sports

Retail

Lifestyles

Personals

Social Networking

Online Gaming

Photos

Maps

Share of Content Category Time Spent by PlatformMobile

0

300,000

600,000

900,000

1,200,000

1,500,000

Growth in Digital Media Time Spent

Source: comScore's "2015 Mobile App Report", Dec. 2015

Jun-2013

476,553 480,967 550,522

118,299

409,847

621,410

778,954

97,440

77,081

Jun-2014 Jun-2015

Desktop

Mobile App

Mobile Browser

Tota

l Min

utes

Increase in time spent since 2013

Desktop (53.4%)

Mobile App (90%)

Mobile Browser(15.5%)

Page 4: Why apps-succeed-wpr-mastercode.vn

4

The Digital User Experience Is Critical

But no matter what screen a consumer is using, the ability to deliver a reliable, consistent and engaging user experience is becoming progressively important to any organization with a digital presence. Overall, 81% of the respondents view the digital experience as crucialto their business success—58% see it as a critical competitive issue, while 23% view it as a matter of “do or die.”

Critical to our competitive positioning

Have to do it if we want to survive

Important but not critical

It is a nice-to-have

Not important�����������58%

3% 1%

23%

15%

“Eighty-one percent of the respondents view the digital experience as crucial to their business success—58% see it as a critical competitive issue, while 23% view it as a matter of ‘do or die.’”

Page 5: Why apps-succeed-wpr-mastercode.vn

5

A Unified Strategy

The multi-device behavior of today’s digital user is reflected in the development and testing strategies of our survey respondents. It’s clear that organizations are preparing themselves to satisfy users no matter how or where they access digital information. Over three quarters (76%) of the respondents have both a mobile and web presence, while 15% are mobile-only and 9% have a web-only presence.

Company size also impacts decisions about multi-device experiences. On average, small organizations (1-24 employees) were twice as likely than enterprises (more than 1000 employees) to support only a single device experience.

Concurrently, organizations are adjusting how their test teams are structured. The majority of the survey respondents (59%) said they have a single unified team for both mobile and desktop web testing. Having one team is traditionally a smaller business approach, but larger companies are adopting a “one mobile and web team” approach due to the pressure to deliver digital products faster. This is an encouraging trend given that when mobile and web teams are united and using the same tools, it’s easier to spot differences across devices and platforms.

Mobile and Web Mobile only (native/hybridmobile app, m.dot)

Web only (dot.com)

�����76%

15%

9%

One team for both mobileand desktop web testing

59%

Separate mobile anddesktop web teams

41%

Page 6: Why apps-succeed-wpr-mastercode.vn

6

Digital Excellence: 4 Obstacles and Recommendations

While delivering a reliable and consistent experience is critical to most of the organizations in our survey, it is also their biggest challenge. Two-thirds of the respondents (67%) view the delivery of a reliable and consistent digital experience as a major challenge, and 30% see it as a minor challenge.

There are several obstacles that stand in the way of delivering a superb digital experience, including the desire to release new functionality faster, the challenge of doing testing that accurately represents real user conditions on an array of devices and operating systems, and the need to keep up with the pace of new technologies.

�������67%

30%

3%

Major challenge

Minor challenge

No challenge

Page 7: Why apps-succeed-wpr-mastercode.vn

7

Obstacle #1:Releasing new digital functionality faster

Almost every organization responding to our survey is struggling to keep up with users’ high expectations. As many as 95% of the organizations cited the timely releases of new digital functionality as either a major (65%) or minor (31%) challenge. This challenge is reflected in the slow release pace we’re seeing today: the median number of releases in 2015 was just five, and close to a quarter (24%) of the organizations had only two or fewer releases.

At the same time, organizations realize they need to accelerate the pace of mobile app and website releases to meet customer expectations. The median number of releases is expected to jump from five to eight in 2016.

Stepping up the pace of product releases requires organizations to adopt new strategies. When asked how they plan to increase their product release frequency, the key strategy cited by respondents was more automation and unit testing, followed by more parallel testing on multiple devices. (See graph on the next page)

Product releases 2015 Product releases 2016

Source: comScore Media Metrix Multi-Platform & Mobile Metrix, U.S., Total Audience

476,553

Mobile Browser

0

5

10

15

20

25

30

35

0

5

10

15

20

25

30

35

2 or fewer

24%

15%

31%

24%

16%15%

35%

20%

12%

9%

3-6 7-12 13-25 26 or more 2 or fewer 3-6 7-12 13-25 26 or more

Median5

Median8

Page 8: Why apps-succeed-wpr-mastercode.vn

8

While organizations see test automation as the primary strategy for faster product releases, most are not there yet. A majority of organizations (55%) have automated 40% or less of their mobile testing, and only 23% have automated over 61% of their mobile testing.

The picture is slightly brighter in web testing, where 37% of the organizations have automated 61% or more of their testing.

Mobile Browser

0 10 20 30 40 50 60 70 80

More automation and unit testing

Testing on more devices in parallel

Other

Using customer data to domore targeted testing

More proactive app/webmonitoring after release

66%

56%

40%

35%

8%

Strategies for Improving Frequency of Product Releases

Mobile testing Web testing

476,553

Mobile Browser

0

5

10

15

20

25

30

0

5

10

15

20

25

0-20%

30%

20%19%

24% 24%

13%

25%

22%

16%

7%

21-40% 41-60% 61-80% 81-100% 0-20% 21-40% 41-60% 61-80% 81-100%

What Percentage of Your Mobile/Web Testing Is Done Using Automation?

Perc

enta

ge o

f Res

pond

ents

Percentage of Automation Percentage of Automation

Page 9: Why apps-succeed-wpr-mastercode.vn

9

Recommendations

Automation is simply a MUST for your company to keep up with the mobile market. A big bank or airline can lose hundreds of thousands of dollars if their app is slow to market. Automation tests can be done quickly on many devices simultaneously through one script, and can be repeated at no additional cost.

It’s important that test automation be spread across the entire SDLC (software development lifecycle). For developers, that means automating unit tests and BATs (build acceptance tests); for QA, that means automating functional tests; and for IT ops, that means assuring performance and integration testing.

A good question to ask at the start is: What should I be automating? Organizations often do not choose the right tests to automate, resulting in lost development time and a reliance on manual testing. The key to mobile test automation is to start with a small number of test cases, automate them, and assure they can be executed in parallel and unattended. Only then should you invest more in your test automation suite. We recommend that 70% of your testing be automated.

The ultimate benefits of strong automation are that your team gets more tests executed earlier, meets release deadlines, and prevents defects from making it to customers’ screens.

To learn more, listen to the webinar, 7 Ways to Overcome Test Automation Obstacles

Sixty-six percent of respondents believe that doing more automation is the key to faster product releases.

Page 10: Why apps-succeed-wpr-mastercode.vn

10

Obstacle #2: Accurately testing for real user environments

It was encouraging to see that a majority of respondents (79%) are mimicking real user environments in their mobile testing.

The top two mobile user conditions that respondents are testing against are switching network conditions and apps running in the background. Other commonly tested conditions include call/text message interruptions and low memory.

Yet at the same time, the vast majority (96%) of the organizations are still challenged to guarantee that testing accurately represents real user environments – with 69% viewing it as a major challenge and another 27% calling it a minor challenge.

Mobile Browser

0 10 20 30 40 50 60 70 80

Apps running in the background

Call and text message interruptions

Low memory

Spikes in mobile traffic

Dying battery

Switch networks - from 3G to 4G to Wi-Fi etc.

Other

71%

71%

56%

51%

43%

40%

8%

What User Conditions Do You Test Against?

Page 11: Why apps-succeed-wpr-mastercode.vn

11

Recommendations

Step one in testing against real world user conditions is to identify your target users (aka “personas”) through customer data analytics. Getting consistent customer usage data relies on regular communication between technical and marketing teams.

The next step is integrating these user traits into testing.

Requirements for a lab that tests for user conditions include:

• Ability to insert tests for various network conditions, apps running in the background, phone call interruptions, and more

• Support for different geographies to test for relevant users in relevant countries

• Support for any device and operating system as they become available

Once the lab is ready, DevTesters can add the relevant user conditions to existing functional tests in various ways, including manual testing, data-driven testing or through automated test code. We recommend automation.

After defining target personas and implementing test code, you’ll need an actionable test report that provides valuable insights into the real user experience of your customers. The report should include: network logs, device and app memory consumption details, and timers showing the duration of certain user actions.

For more details on how to test for real users, download the white paper: 3 Keys for Mobile App Testing for Real User Conditions.

Page 12: Why apps-succeed-wpr-mastercode.vn

12

Obstacle #3: Staying up on the latest devices and operating systems

Organizations are finding it difficult to keep up with mobile market changes and the pace of technology. Ensuring that testing accurately delivers a reliable experience on new devices and operating systems is a major challenge for 64% of survey respondents.

Responsive websites play a pivotal role in addressing this challenge – 88% of the organizations either already have a responsive website or are in the process of developing one.

The trend toward responsive websites is driven by: the quest to provide a more consistent digital experience across devices (46% of respondents) as well as the need to guarantee mobile websites work on new devices as they become available (35% of respondents).

Visual testing and navigation testing are the most common strategies for responsive website testing (see second graph below).

Mobile Browser

0

10

20

30

40

50

60

70

80

Visual testing Navigationaltesting

Client-side performance

Testing

User conditiontesting

76%72%

61%57%

Mobile Browser

0

10

20

30

40

50

60

Yes, our websitesare responsive

We plan to in the next 6 months

We plan to in thenext 12 months

We have no plan to do

responsive design

We are currentlydeveloping a

responsive website

59%

14%

8% 7%

12%

Fifty-nine percent of organizations currently have responsive websites and 29% are adopting responsive web over the next 12 months.

Page 13: Why apps-succeed-wpr-mastercode.vn

13

Recommendations

A key way to prepare for new devices and operating systems is to create a market calendar —or use the mobile and web browser market calendars from our latest Digital Test Coverage Index report—and plan accordingly.

With all the new releases in a given year, you should keep a close eye on the market. Include beta testing of upcoming OS and browser versions and plan on a device lab refresh to accommodate new devices.

As we mentioned, responsive web design (RWD) is one great way to make sure your digital products will work on new devices. RWD done right will deliver a seamless digital experience on every screen, but must include rigorous testing.

Three RWD testing scenarios to keep in mind:

• Visual testing to make sure content looks right on every screen

• Client-side performance testing to measure the time it takes website objects to render on different screens

• Navigation testing to verify that the website performs expected functions correctly on all devices

Not all companies have multiple approaches to delivering content to a variety of devices —particularly with responsive websites—yet the need for a consistent user experience and supporting analytics is key to success. Dig in to customer usage data for clues on how to prioritize your testing strategy. For instance, if 80% of your mobile traffic is from iOS devices, start testing on the upcoming iOS 10 beta version the minute it’s available.

OS

PROJECTED TYPICAL

ADOPTION OF MAJOR OS

DEVICES

JANUARY FEBRUARY MARCH APRIL MAY JUNE JULY AUGUST SEPTEMBER OCTOBER NOVEMBER DECEMBER

Honor Holly 2Plus

Media Pad M2 P9

Nexus 7P

HTC One M10

LG K7

LG G5 LG V11

Nexus 6X

OnePlus 3

Surface Phone

Droid Turbo 3Xiaomi Redmi 3 Xiaomi Mi 5

iOS 10 release

iOS 10.0.1release

Android 6.1

Android 62% share

Android 610% share

Android 615% share

iOS 10.2release

Windows 10

iPad Air 3iPad Pro 9.7"

iPhone SE

iOS 10 announcement

Android 7release

iOS 10.1 release

iOS 9.2.1 iOS 9.3 Betapublic availability

iOS 9.3.2iOS 9.3.2 BetaiOS 9.3iOS 9.3.1

iOS 9 = 77%iOS 8 = 17%Other = 6%

iOS 10 = 35%iOS 9 = 60%Other = 5%

iPhone 7iPhone 7 Plus

iPad Mini 5Watch 2

iPad Pro 2

Live G500TG

Galaxy S7Galaxy S7 Edge

Galaxy Note 4Upgrade available

Android 6.0

Galaxy S4/S5Upgrade available

Android 6.0

Galaxy Note S6/Edge/Note 5

Upgrade availableAndroid 6.0

Sony Xperia X

Galaxy A9 Note 6

Sony Xperia X

HP Elite X3(Win 10)

HTC One X9

Mobile Market Calendar 2016Be sure to plan ahead to get maximum test coverage.

Our Ultimate Digital Test Coverage Toolkit offers three free tools to help you create a test coverage strategy.

Page 14: Why apps-succeed-wpr-mastercode.vn

14

Obstacle #4: Building a more efficient lab for mobile testing

Companies have options when it comes to mobile testing—from using homegrown mobile test labs to using cloud-based or on-premise commercial solutions to offload device ownership and improve reliability.

While commercial mobile testing solutions have become more robust and offer significant capabilities, many organizations – including 45% of our survey respondents – continue to run in-house mobile test labs despite challenges.

For companies that use their own lab, keeping up with frequent browser and device releases is the top challenge. Maintaining the test lab and keeping it up-to-date and secure is another significant issue, with some of the survey respondents reporting they actually devote more resources to lab maintenance than to their test codebase.

0 10 20 30 40 50

New browsers and devices are releasedto market faster than we can keep up

We don’t have a dedicated team to manage resourcesand update our lab quickly enough

We don’t have the expertise in-house to keep ourinternal lab up to date

We focus more on test labmaintenance than our test codebase

Other

44%

29%

11%

8%

8%

The Top Challenges Teams Face When Running In-House Testing Labs

Page 15: Why apps-succeed-wpr-mastercode.vn

15

Recommendations

It may seem convenient to buy and test manually on mobile devices in-house, but in reality it’s difficult and expensive. You will have to purchase and update the device, replace it if it’s lost, fix it if the screen cracks, ask IT to fix driver conflicts and allocate desktops to connect to the devices. The annual total-cost-of-ownership to manage a single mobile device in-house is $6,500!

We recommend testing on real devices, but for the reasons stated above we do not recommend buying them yourself and testing them in-house.

To test your apps or websites as your users would experience them on a mix of real devices, networks and locations, a cloud-based lab-as-a-service that’s available 24/7 and can scale as needed is the optimal choice.

A cloud-based lab gives teams in different geographies remote access to the same devices, makes new devices and OSes available immediately and allows unattended automation testing because devices are always accessible in a secure environment.

“It may seem convenient to buy and test manually on mobile devices in-house, but in reality it’s difficult and expensive.”

Read the new Forrest Wave report to see who’s leading the way in mobile front-end test automation tools.

Page 16: Why apps-succeed-wpr-mastercode.vn

16

ConclusionA seamless, cross-device digital experience is increasingly vital to companies that want to engage with customers effectively. This requires organizations of all sizes to quicken the pace of development and release without decreasing the quality of those experiences.

Most organizations still have a long way to go toward automating the testing of mobile and web applications, accurately representing the user environments, and adopting a test lab strategy that can scale to meet today’s needs. Yet by overcoming these challenges, an unprecedented opportunity will open up for companies to deliver memorable digital experiences to their customers.

Page 17: Why apps-succeed-wpr-mastercode.vn

17

Survey Methodology and DemographicsFor this report, Perfecto conducted a survey in April 2016 and received over 1,000 responses from enterprise-level web and mobile professionals representing a variety of roles, company sizes, industries and geographies.

���������58%

15%

20%

4%5%

Quality Assurance

Management

Development

DevOps

Other

���������46%

19%

30%

1%

4%

North America

Asia/Pacific

Europe/Middle

Central/South

Africa

High Tech

Financial S

ervice

s

Teleco

mmunicatio

ns

Healthca

re

Adverti

sing &

Marke

tingReta

il

Media & Enterta

inmet

Manufacturin

g

Public Secto

rOth

er

21%19%

10% 10% 9%8%

5%4% 4%

11%

0

5

10

15

20

25

Role The majority of survey respondents (56%) are in quality assurance roles, while another 20% hold management positions.

Geography46% of respondents are based in North America, 30% in Asia/Pacific, 19% in Europe/Middle East and 4% in Central/South America.

Company SizeClose to half of respondents come from large companies with over 1,000 employees.

IndustrySurvey respondents come from a wide range of industries, led by high tech and financial services.

58%

15%

20%

4%5%

1-24

14%12% 11%

13%

49%

25-100 101-250 251-1000 > 10000

10

20

30

40

50

Page 18: Why apps-succeed-wpr-mastercode.vn

18

See which devices should be covered in your test plan by visiting: www.perfectomobile.com/testoptimizer

© 2016 Perfecto Mobile Ltd.www.perfectomobile.com

781.205.4111

About Perfecto

Perfecto enables exceptional digital experiences. We help you transform your business and strengthen every digital interaction with a quality-first approach to creating web and native apps, through a cloud-based test environment called the Continuous Quality Lab. The CQ Lab is comprised of real devices and real end-user conditions, giving you the truest test environment available. More than 1,500 customers, including 50% of the Fortune 500 across the banking, insurance, retail, telecommunications and media industries rely on Perfecto Mobile to deliver optimal mobile app functionality and end user experiences, ensuring their brand’s reputation, establishing loyal customers, and continually attracting new users. For more information about Perfecto Mobile, visit perfectomobile.com, join our community, or follow us on Twitter at @PerfectoMobile.