Performance Related Changes and their User Impact Eric Schurman Principal Development Lead Bing Jake...
-
Upload
jacoby-messenger -
Category
Documents
-
view
219 -
download
2
Transcript of Performance Related Changes and their User Impact Eric Schurman Principal Development Lead Bing Jake...
![Page 1: Performance Related Changes and their User Impact Eric Schurman Principal Development Lead Bing Jake Brutlag Decision Support Engineering Analyst Google.](https://reader034.fdocuments.net/reader034/viewer/2022051819/55163c8f550346a2308b64bf/html5/thumbnails/1.jpg)
Performance Related Changes and their User
ImpactEric Schurman
Principal Development LeadBing
Jake BrutlagDecision Support Engineering Analyst
![Page 2: Performance Related Changes and their User Impact Eric Schurman Principal Development Lead Bing Jake Brutlag Decision Support Engineering Analyst Google.](https://reader034.fdocuments.net/reader034/viewer/2022051819/55163c8f550346a2308b64bf/html5/thumbnails/2.jpg)
Experiments• Server Delays (Microsoft and
Google)• Page Weight Increases• Progressive Rendering
![Page 3: Performance Related Changes and their User Impact Eric Schurman Principal Development Lead Bing Jake Brutlag Decision Support Engineering Analyst Google.](https://reader034.fdocuments.net/reader034/viewer/2022051819/55163c8f550346a2308b64bf/html5/thumbnails/3.jpg)
Server-side Delays Experiment
• Goal• Determine impact of server delays
• Methodology• Delay before sending results• Different experiments with different
delays• Small number of users• Monitor negative impact
![Page 4: Performance Related Changes and their User Impact Eric Schurman Principal Development Lead Bing Jake Brutlag Decision Support Engineering Analyst Google.](https://reader034.fdocuments.net/reader034/viewer/2022051819/55163c8f550346a2308b64bf/html5/thumbnails/4.jpg)
Server Delays Experiment: Results
• Strong negative impacts• Roughly linear changes with increasing delay• Time to Click changed by roughly double the delay
50ms - - - - - -200ms - - - -0.3% -0.4% 500500ms - -0.6% -1.2% -1.0% -0.9% 12001000ms -0.7% -0.9% -2.8% -1.9% -1.6% 19002000ms -1.8% -2.1% -4.3% -4.4% -3.8% 3100
- Means no statistically significant change
![Page 5: Performance Related Changes and their User Impact Eric Schurman Principal Development Lead Bing Jake Brutlag Decision Support Engineering Analyst Google.](https://reader034.fdocuments.net/reader034/viewer/2022051819/55163c8f550346a2308b64bf/html5/thumbnails/5.jpg)
Google Web Search Delay Experiments
• A series of experiments on a small % search traffic to measure the impact of latency on user behavior
• Randomly assign users to the experiment and control groups (A/B testing)
• Server-side delay:• Emulates additional
server processing time• May be partially masked
by network connection• Varied type of delay,
magnitude (in ms), and duration (number of weeks)
![Page 6: Performance Related Changes and their User Impact Eric Schurman Principal Development Lead Bing Jake Brutlag Decision Support Engineering Analyst Google.](https://reader034.fdocuments.net/reader034/viewer/2022051819/55163c8f550346a2308b64bf/html5/thumbnails/6.jpg)
Search Traffic Impact
Type of DelayDelay (ms)
ExperimentDuration (weeks)
Impact on Average Daily Searches Per User
Pre-header 50 4 Not measurablePre-header 100 4 -0.20%Post-header 200 6 -0.29%Post-header 400 6 -0.59%Post-ads 200 4 -0.30%
• Increase in abandonment heuristic = less satisfaction• Abandonment heuristic measures if a user stops
interacting with search engine before they find what they are looking for
• Active users (users that search more often a priori) are more sensitive
![Page 7: Performance Related Changes and their User Impact Eric Schurman Principal Development Lead Bing Jake Brutlag Decision Support Engineering Analyst Google.](https://reader034.fdocuments.net/reader034/viewer/2022051819/55163c8f550346a2308b64bf/html5/thumbnails/7.jpg)
da
ily
se
arc
he
s p
er
us
er
rela
tiv
e t
o c
on
tro
l
w k1 w k2 w k3 w k4 w k5 w k6
-1%
-0.8
%-0
.6%
-0.4
%-0
.2%
0%
0.2
%2 0 0 m s d e la y4 0 0 m s d e la y
a c tua ltre nd
Im p a c t o f P o s t-he a d e r D e la ys O ve r Tim e
-0.22%
-0.44%
-0.36%
-0.74%
![Page 8: Performance Related Changes and their User Impact Eric Schurman Principal Development Lead Bing Jake Brutlag Decision Support Engineering Analyst Google.](https://reader034.fdocuments.net/reader034/viewer/2022051819/55163c8f550346a2308b64bf/html5/thumbnails/8.jpg)
da
ily
se
arc
he
s p
er
us
er
rela
tiv
e t
o c
on
tro
l
w k3 w k4 w k5 w k6 w k7 w k8 w k9 w k1 0 w k1 1
-1%
-0.8
%-0
.6%
-0.4
%-0
.2%
0%
0.2
%
de
lay
rem
ov
ed
P e rs is te nt Im p a c t o f P o s t-he a d e r D e la y
2 0 0 m s d e la y4 0 0 m s d e la y
a c tua ltre nd
-0.08%
-0.21%
![Page 9: Performance Related Changes and their User Impact Eric Schurman Principal Development Lead Bing Jake Brutlag Decision Support Engineering Analyst Google.](https://reader034.fdocuments.net/reader034/viewer/2022051819/55163c8f550346a2308b64bf/html5/thumbnails/9.jpg)
Page Weight Experiment
• Goal• Determine impact of a heavier
page.
• Isolate bytes over the wire cost, not layout costs, etc.
• Methodology• Use incompressible HTML
comments
• Vary size (from 1.05x to 5x page size) and location of payload
• Experiment with payload in individual and multiple locations
• US-only test – mostly good broadband
<html>
<head>
<CSS />
<!--Payload-->
</head>
<body>
<answer>...
<!--Payload-->
</answer>
<results>
<result>...
<!--Payload-->
</result>
<result>...
<!--Payload-->
</result>
<result>...
<!--Payload-->
</result>
</results>
...
<!--Payload-->
<script />
<!--Payload-->
</body>
</html>
![Page 10: Performance Related Changes and their User Impact Eric Schurman Principal Development Lead Bing Jake Brutlag Decision Support Engineering Analyst Google.](https://reader034.fdocuments.net/reader034/viewer/2022051819/55163c8f550346a2308b64bf/html5/thumbnails/10.jpg)
Page Weight Experiment: Results• Minimal impact for small payloads• Payload at top of page had stronger effect• Performance suffered slightly – would have
been worse if tested in regions with poor connectivity
• Click metrics impacted more than Query metrics• Largest experiment (approx 5X control page
size)• Any Clicks: -0.55%
• No changes to query metrics
• Results only apply to one GET – not multiple
![Page 11: Performance Related Changes and their User Impact Eric Schurman Principal Development Lead Bing Jake Brutlag Decision Support Engineering Analyst Google.](https://reader034.fdocuments.net/reader034/viewer/2022051819/55163c8f550346a2308b64bf/html5/thumbnails/11.jpg)
Progressive Rendering Experiment
• Goal• Determine impact sending visual header before
results.• Methodology
• Build page in phases• Send using HTTP 1.1 Chunked Transfer Encoding
• Application design impacts
Visual Header - Fast to compute
Results - Slower to compute
![Page 12: Performance Related Changes and their User Impact Eric Schurman Principal Development Lead Bing Jake Brutlag Decision Support Engineering Analyst Google.](https://reader034.fdocuments.net/reader034/viewer/2022051819/55163c8f550346a2308b64bf/html5/thumbnails/12.jpg)
Progressive Rendering Experiment: ResultsMetric Change
Performance
Faster across all latency percentiles4-18% faster to download all HTMLRoughly halved time to see visible page change
Time to Click ~9% faster
Query refinement +2.2%
Clicks overall +0.7%
Pagination +2.3%
Satisfaction +0.7%
![Page 13: Performance Related Changes and their User Impact Eric Schurman Principal Development Lead Bing Jake Brutlag Decision Support Engineering Analyst Google.](https://reader034.fdocuments.net/reader034/viewer/2022051819/55163c8f550346a2308b64bf/html5/thumbnails/13.jpg)
Conclusion
• "Speed matters" is not just lip service
• Delays under half a second impact business
metrics
• The cost of delay increases over time and
persists
• Number of bytes in response is less
important than what they are and when they
are sent
• Use progressive rendering