Edge 2016 can webpagetest be trusted
-
Upload
akamaidevrel -
Category
Technology
-
view
28 -
download
4
Transcript of Edge 2016 can webpagetest be trusted
© AKAMAI - EDGE 2016
Can Webpagetest be Trusted?Pierre Lermant
©2016 AKAMAI | FASTER FORWARDTM
WPT
It is THE synthetic performance testing tool
● Free to use public instanceThrough UI or APIs
● Feature rich
● Maintained to latest standards
● Can be hosted privately
©2016 AKAMAI | FASTER FORWARDTM
Goal:Explore WPT public instances repeatability and consistency
©2016 AKAMAI | FASTER FORWARDTM
● Thirty (30) URLs tested: Alexa top 10 from Shopping, Business, News
● Primarily focused on two metrics: onload time and speedIndex
● Chrome browser, 9 runs, medians, first view only, video on
● Tests ran between Sunday October 9 and Sunday October 16o All using APIs from python scriptso All URLs run in parallel from different agent locations
Experimental Setup
©2016 AKAMAI | FASTER FORWARDTM
Reliability
©2016 AKAMAI | FASTER FORWARDTM
Impact of location and retries on successful URLs (9 runs, 30 URLs)
Reliability
©2016 AKAMAI | FASTER FORWARDTM
Lesson # 1:● Check number of successful runs
Especially from non EC2 locationsIf 9 runs needed, plan for a few retries
©2016 AKAMAI | FASTER FORWARDTM
How many runs?
©2016 AKAMAI | FASTER FORWARDTM
Impact of run number on main metrics, 30 URLs - Oct 14 run
How many runs?
©2016 AKAMAI | FASTER FORWARDTM
Impact of run numbers on Coefficient of Variation (%) - Oct 14 run
How many runs?
©2016 AKAMAI | FASTER FORWARDTM
Lesson # 2:● First run yields slightly higher
onload and speedIndex values● Variability seems to settle
around 10-15%, except for TTFB
©2016 AKAMAI | FASTER FORWARDTM
Intra-Location consistency
©2016 AKAMAI | FASTER FORWARDTM
Impact of concurrent runs in same locations on median onload time
Intra location consistency
Oct 13 Night Oct 14 Day Oct 16 Night
©2016 AKAMAI | FASTER FORWARDTM
Impact of concurrent runs in same locations on median speedIndexIntra location consistency
Oct 13 Night Oct 14 Day Oct 16 Night
©2016 AKAMAI | FASTER FORWARDTM
Lesson # 3:● Intra-location results within ~10 %
©2016 AKAMAI | FASTER FORWARDTM
Inter-Location consistency
©2016 AKAMAI | FASTER FORWARDTM
Impact of location (desktop fios) on median onload times
Inter location consistency
Oct 10 Night Oct 13 Night Oct 14 Day Oct 16 Night
©2016 AKAMAI | FASTER FORWARDTM
Impact of location (desktop fios) on median speedIndex times
Inter location consistency
Oct 10 Night Oct 13 Night Oct 14 Day Oct 16 Night
©2016 AKAMAI | FASTER FORWARDTM
Lesson # 4:● Results vary greatly with location
EC2 instances slower than private ?
©2016 AKAMAI | FASTER FORWARDTM
RelativeInter-location consistency
©2016 AKAMAI | FASTER FORWARDTM
Relative site speed ranking per location (%)
Relative consistency
● Dulles:Fios v.s. ec2-us-east-1:FiosOnload: 14 %speedIndex: 24 %
● ec2-us-west-1:Fios v.s. SanJose:FiosOnload: 15 %speedIndex: 15 %
©2016 AKAMAI | FASTER FORWARDTM
Lesson # 5:● Relative speed ranking
(roughly) preserved
©2016 AKAMAI | FASTER FORWARDTM
Accuracy
©2016 AKAMAI | FASTER FORWARDTM
WPT results vs actual browser experience: onload times
Accuracy
Average Median Oct 16th run
©2016 AKAMAI | FASTER FORWARDTM
Lesson # 6:● Location Dulles:Cable
(=default) closest to Home!
©2016 AKAMAI | FASTER FORWARDTM
Emulation v.s. Real Device
©2016 AKAMAI | FASTER FORWARDTM
Emulation vs real device (Moto. G) for various agents: onload timesEmulation v.s. Real Device
Oct 10 Night Oct 13 Night Oct 14 Day Oct 14 Night Oct 16 Night
Emulation
Emulation
©2016 AKAMAI | FASTER FORWARDTM
Emulation vs real for various locations: speedIndex timesEmulation v.s. Real Device
Oct 10 Night Oct 13 Night Oct 14 Day Oct 14 Night Oct 16 Night
Emulation
Emulation
©2016 AKAMAI | FASTER FORWARDTM
Lesson # 7:● Large (> 20%) discrepancies
between real device and emulation (Dulles + Motorola G)
©2016 AKAMAI | FASTER FORWARDTM
More ...
©2016 AKAMAI | FASTER FORWARDTM
Compare WPT onload with Rum onload
Compare behavior of public instances with private ones
Find out what’s causing discrepancies between locations
Throttle emulation CPUs to better approximate real devices
More ...
©2016 AKAMAI | FASTER FORWARDTM
Summary
● Location has a huge impact on main performance metrics
● For a given location, allow for about 10-15% “repeatability margin”
● More testing is needed (surprise!)