Perform Realistic Web Performance Testing ToPerformance Testing To Achieve Peak User ExperiencesExperiencesMike GualtieriSenior Analyst

May 13, 2009

yForrester Research

Realistic web ea st c ebperformance testing is

critical to achieving c t ca to ac e gflawless user experiences.p

Agenda

• Why your customers won’t wait

Wh t i li ti f t ti• What is realistic performance testing

• Recommendations

4 Entire contents © 2008 Forrester Research, Inc. All rights reserved.

Agenda

• Why your customers won’t waitWh t i li ti f t ti• What is realistic performance testing

• Recommendations

5 Entire contents © 2008 Forrester Research, Inc. All rights reserved.

Performance means:• Raw page load speed Gomez Media Home

Page BenchmarkApril 10 ’09 – May 10 ’09

0:00 - 24:00 EST

6 Entire contents © 2008 Forrester Research, Inc. All rights reserved.

Gomez Media ArticlePerformance means:

Gomez Media Article Search Transaction

BenchmarkApril 10 ’09 – May 10 ’09

0:00 24:00 EST

• Customer workflow speed1

0:00 - 24:00 EST

2

33

7 Entire contents © 2008 Forrester Research, Inc. All rights reserved.

Realistic Web Performance Testing Is A F t B t P tiForrester Best Practice• Best Practices: Attaining And Maintaining Blazing Fast Web Sites

• Best Practices: Blazing Fast Web Site Infrastructure

• Case Study: How Orbitz.com Takes You Places Faster

• Case Study: How Msnbc.com Is First In News

8 Entire contents © 2008 Forrester Research, Inc. All rights reserved.

Blazing FastWeb Sites = CustomersWeb Sites Customers

Traditional web performance testing d ’t t itdoesn’t cut it• Traditional testing is expensive

• Cost of licenses, hardware and training

• Time consuming to implement

• Idiosyncrasies are often difficult to master

• Traditional testing is not realistic• Scope is limited to internal clean-room testing

• Not focused on end-user point of viewp

• Misses too many pieces in the web application delivery chain such as CDN, 3rd-party components,

11 Entire contents © 2008 Forrester Research, Inc. All rights reserved.

geography, etc…

Most Developers Are Building Web Apps.“What types of new custom applications are your firm’s developers building”

(multiple responses accepted)

12 Entire contents © 2008 Forrester Research, Inc. All rights reserved.

2008 Base: 977 platform software decision-makers at North American and European enterprises and SMBs

New trends make optimal user i i t texperiences even more important

• The Millennial generation expects more– 18 to 27 year olds– The tech-savvy generation

• Web site redesigns– RIA

• More Web sites are composed of 3rd-party componentsVi t li ti f i f t t i l di l d• Virtualization of infrastructure including cloud

• Mobile

13 Entire contents © 2008 Forrester Research, Inc. All rights reserved.

Agenda

• Why your customers won’t wait?

Wh t i li ti f t ti• What is realistic performance testing• Recommendations

14 Entire contents © 2008 Forrester Research, Inc. All rights reserved.

“Realistic testing” means testing from your customer’s point of viewcustomer s point of view

• Where they do it– LA, Edmonton, London, Bangkok, Des Moines,

Miami, Cannes, Dubrovnik, Osaka, Pusan

Wh h d• What they do– Key tasks, transactions, and business processes

• How they do it– Browsers galore, mobile devices

• When they do it (and how frequently)– Peak times, spikes

15 Entire contents © 2008 Forrester Research, Inc. All rights reserved.

p

The user’s distance from the content ttmatters

16 Entire contents © 2008 Forrester Research, Inc. All rights reserved.

Browsers render differently

17 Entire contents © 2008 Forrester Research, Inc. All rights reserved.

Source: Actual Experience XF Real-User Monitoring Data; Gomez, Inc.

What features should a realistic web performance testing platform have? It should:performance testing platform have? It should:

• Test from the end user’s point of view– Different locations

– Different browsers

– Different loads

– Different tasks

• Be easy for developers, testers and business people to use.

• Identify and resolve problems along the entire web application delivery chain.

18 Entire contents © 2008 Forrester Research, Inc. All rights reserved.

Agenda

• What do customers want?

Wh t i li ti f t ti• What is realistic performance testing

• Recommendations

19 Entire contents © 2008 Forrester Research, Inc. All rights reserved.

Recommendations (1 of 2)• Design a test plan based on how real users interact with your web application

• Use testing tools that allow you to test from your user’s point of view:

Wh Wh t H Wh– Where, What, How, When

• Test the entire Web application

• Democratize testing: – Build a test team that includes anyone who has a

stake in the outcome

20 Entire contents © 2008 Forrester Research, Inc. All rights reserved.

Recommendations (2 of 2)• Testing early to nip problems in the bud

• Test often to maintain a consistently supreme user experience

• Follow Blazing Fast Web Performance Best Practices to keep your Web site humming

21 Entire contents © 2008 Forrester Research, Inc. All rights reserved.

In this fiercely competitive environment, realistic web performance testing is the

right answer.

Thank you

Mike Gualtieri

1 617 613 6145+1 617.613.6145

mgualtieri@forrester.com

www.forrester.com

23 Entire contents © 2008 Forrester Research, Inc. All rights reserved.

Gomez Profile

▪ First SaaS Web Application Experience Management Suite▪ Headquarters pe e ce a age e t Su te

▪ Web Performance and Load Testing

▪ Cross-Browser Testing

▪ Lexington, MA

▪ Global Offices Cross Browser Testing

▪ Web Performance Monitoring

▪ 80 000+ internet experience

▪ UK

▪ Germany ▪ 80,000+ internet experience measurement points in 162 countries

Ge a y

▪ Switzerland

▪ China▪ Over 2,500 Customers Worldwide

▪ 12 out of 20 most visited US Websites tili G

24

utilize Gomez

▪ 300 Employees

Today’s Testing Challenges

Never Enough Time Small QA Teams

25Testing the End-User PerspectiveCatching All the Problems

When To Load Test

Launching New Apps/Features Deploying New Infrastructure

26Penetrating New MarketsMajor Marketing Campaigns

Web Application Delivery In A Web 2.0 World

▪ Web applications ppincreasingly assembled at browser level▪ 3rd party

performance from ads, videos, shopping cart vary widely cart vary widely under load

▪ Content Delivery Network configuration and effectiveness impact

d i

27

end-user experience

Load 2.0 Testing Solutions

Overcome Limitations of 1.0 Testing Approaches

▪ Find and fix problems across the entire web application delivery chain▪ Inside and outside the firewall▪ Inside and outside the firewall

▪ Flexible and easy to useFlexible and easy to use▪ On-demand – no installation required▪ Highly scalable to accommodate

peak loadspeak loads▪ Supports Agile development

28

▪ Test from the end-user perspective

Ensure Web Experiences Scale Under Load With Reality Load XF

Deploy web applications and ▪ Deploy web applications and infrastructure with confidence

▪ Combine high volume web application load testing with broad geographic experience testing

▪ Ensure on-demand testing gflexibility

▪ Detect failures in user experience not caught by traditional load not caught by traditional load testing

▪ Testing across the entire web li ti d li h i

29

application delivery chain

Critical To Test Across Entire Web Application Delivery Chain

Key Load Testing Metrics• Success rate of end-user workflow• Response time of end-user workflowResponse time of end user workflow• Quality of end-user workflow delivery

30

Load Testing 2.0Performance Testing For A Web 2.0 World

▪ Self-service testing gives you control to test as neededcontrol to test as needed

▪ Test internal & external web application components to ensure transactions perform ensure transactions perform under load

▪ Test cloud-based applications & servicesservices

▪ SaaS based Load Testing 2.0 solutions enable on-demand, scalable testing without

C bi hi h l g

associated hardware investment & maintenance costs

▪ Test the full web application

Combine high volume application load testing with broad geographic

31

ppdelivery chain all the way to your end users’ desktops

experience testing

Critical To Test Across Entire Web Application Delivery Chain

22 33

11

Ensuring Performance Of Login Process

Company• Online presence for a popular television show

11

p p p

Situation• Following episodes of the television show the web site

sees high traffic spikessees high traffic spikes

• Goal was to achieve 1500 logins per minute

• Load tested database optimizations aimed to improve performance in anticipation of another large traffic spikeperformance in anticipation of another large traffic spike

33

Application Bottleneck Causes Immediate Response Time Issue

• As users were added the• As users were added, the response time of step 3 (the login) climbed immediately

• The test bottlenecked at 160 logins per minute (Goal 1500)

• But quickly dropped off as users received server errors

• New login query was not optimized and was bottlenecking the database

’ CPU

34

servers’ CPUs

Ensuring Performance Of E-Commerce Transaction

22

Company• Online Retailer

Situation• Typical e-Commerce type application

S l 3 d P ti i l d i• Several 3rd Parties now involved in serving up key content

• Goal was to validate performance of entire application

Response Times Rise Due To Third-Party Object Error

The load increases throughout the test

The transaction rate increases and then falls off as response times climb

Errors are seen, all on a 3rd party object

3 d t h d i ffi i t f ll d d li ti• 3rd party hardware was insufficient for overall demands on application• Based on SLAs 3rd party had to improve performance to get paid

Ensuring Performance In Target Markets

33

Company• Regional Online News Source

Situation• Began testing for the 2008 election season• Goal was to validate performance of overall

application, and specifically for users in 2 key geographies

• Testing started with the homepage

No Performance Issues Detected From Data-Center

There was only Increase and hold load and not exceed response times of 4 seconds and Success Rate of 99%

Page response

1 page error and 11 errors total out of 60000+ transactions

Rate of 99%

Page response times stayed under 4 seconds, outside of one brief blip

transactions

p

By traditional test standards the test passed

Performance Issues Detected From Real User Desktops

Key geographies for this customer are New York and Pennsylvania

Last Mile data showing substantial number of measurements greater than 4 seconds

Performance Issues Detected For Target Market End Users

• Last Mile Users exceeding the 4-second goal – 29%

• Overall availability was 96.3%

• Availability for Last Mile UsersAvailability for Last Mile Users exceeding 4 seconds – 87%

• Last Mile Users reporting 10+ gsecond response times – 8%

B th t d d th t t f il dBy these standards the test failed

• Cutting 1 5 to 2 sec off the response time for the page would move• Cutting 1.5 to 2 sec. off the response time for the page would move another 16% of the Last Mile users into the 4 sec. response time range

Gomez Offers Full Lifecycle SolutionsFor Web Experience Management

“Gomez has made spectacular progress in the last 18 months…to

41

p p gbecome the leader in web experience management.”

Jean-Pierre Garbani, Vice President, Principal Analyst, Forrester ResearchTech Horizons: Evaluating Gomez’s Web Experience Management Services, August 2008

Find Out How Gomez Can Help You

Some of Our CustomersEnsuring great experiences in a Web 2.0 world requires a new approach to web application load testing Reality Load XFapplication load testing - Reality Load XF

Reality Load XF• Self-service or full turnkey solution with Gomez

P f i l S i ff iProfessional Services offerings • Tests outside-in, with drill down to web application

components• SaaS with no investment or maintenance costsSaaS with no investment or maintenance costs

and rapid payback• Full desktop browser testing across globally

distributed geographies

Learn More:Reality Load XF Solution Briefhttp://www.gomez.com/pdfs/SB_RealityLoad.pdf

42Speak To A Gomez Solutions ExpertToll Free: 877.372.6732