Paruj Ratanaworabhan Kasetsart University, Thailand Ben Livshits and Ben Zorn Microsoft Research,...
-
Upload
jonah-foster -
Category
Documents
-
view
227 -
download
0
Transcript of Paruj Ratanaworabhan Kasetsart University, Thailand Ben Livshits and Ben Zorn Microsoft Research,...
Paruj RatanaworabhanKasetsart University, Thailand
Ben Livshits and Ben ZornMicrosoft Research, Redmond
JSMeter: Characterizing the Behavior of JavaScript Web Applications
1
in collaboration with
David Simmons, Corneliu Barsan, and Allen Wirfs-Brock
Background• Who is this guy?
– Current: Principal Researcher, Microsoft – Past: Professor, U of Colorado; PhD, UC Berkeley
• Research in Software Engineering (RiSE)– 30 researchers and developers– Notables
• Code Contracts (Visual Studio 2010)• Static Driver Verifier in Windows• Peli’s Video Blog at MSDN Channel 9
2
Why Measure JavaScript?
• Standardized, de facto language for the web
– Support in every browser, much existing code
• Browser and JavaScript performance is important
– Are current JavaScript benchmarks representative?
– Limited understanding of JavaScript behavior in real sites
• Who cares?
– Users, web application developers, JavaScript engine developers
3
4
ZDNet 29 May 2008
ghacks.net Dec. 2008
BrowserWars!
Artificial Benchmarks versus Real World Sites
5
Maps Maps
7 V8 programs:
• richards
• deltablue
• crypto
• raytrace
• earley-boyer
• regexp
• splay
8 SunSpider programs:• 3-draytrace
• access-nbody
• bitops-nsieve
• controlflow
• crypto-aes
• date-xparb
• math-cordic
• string-tagcloud
JSMeter 11 real sites:
Goals of JSMeter Project• Instrument JavaScript execution and measure behavior• Compare behavior of JavaScript benchmarks against real sites• Consider how benchmarks can mislead design decisions
6
How We Measured JavaScript
\ie\jscript\*.cpp
Source-level instrumentation
custom jscript.dll
custom trace fileswebsite visits
Offline analyzers
custom trace files0%
10%20%30%40%50%60%70%80%90%
100%
Constant
Other Str Ops
Concat Op
Visiting the Real Sites
• Getting past page load performance• Attempted to use each site in “normal” way:
7
amazon Search a book, add to shopping cart, sign in, and sign outbing Type in a search query and also look for images and newsbingmap Search for a direction from one city to anothercnn Read front page newsebay Search for a notebook, bid, sing in, and sign outeconomist Read front page news, view comments
facebookLog in, visit a friend pages, browse through photos and comments
gmail Sign in, check inbox, delete a mail, and sign outgoogle Type in a search query and also look for images and newsgooglemap Search for a direction from one city to anotherhotmail Sign in, check inbox, delete a mail, and sign out
Understanding JavaScript Behavior
Code
Events
JavaScript
Objects
8
Code Behavior
Code
Events
JavaScript
Objects
9
• Function size
• Instructions/call
• Code locality
• Instruction mix
Total Bytes of JavaScript Source
10
amaz
on
bingm
apeb
ay
face
book
hotm
ail
richar
ds
cryp
to
earle
y
splay
3d-rayt
race
bitops-
nsieve
cryp
to-a
es
mat
h-cor
dic
strin
g-tagclo
ud
0
500
1000
1500
2000
2500
So
urc
e s
ize
(k
ilo
byte
s)
Real Sites V8 SunSpider
Code|Objects|Events
Static Unique Functions Executed
11
amaz
on
bing
map
ebay
face
book
goog
le
hotm
ail
richa
rds
cryp
to
earle
ysp
lay
3d-ra
ytra
ce
bito
ps-n
sieve
cryp
to-a
es
mat
h-co
rdic
strin
g-ta
gcloud
0
500
1,000
1,500
2,000
2,500
3,000
3,500
4,000
Uniq
ue F
unct
ions E
xecu
ted
Real Sites V8 SunSpider
Code|Objects|Events
Bytecodes / Call
12am
azon
bingm
apeb
ay
face
book
hotm
ail
richar
ds
cryp
to
earle
y
splay
3d-rayt
race
bitops-
nsieve
cryp
to-a
es
mat
h-cor
dic
strin
g-tagclo
ud
0.00
50.00
100.00
150.00
200.00
250.00
300.00
Byte
co
de
s /
Ca
ll
function(a,b) { var i=0,elem,pos=a.length; if(D.browser.msie) { while(elem=b[i++]) if(elem.nodeType!=8) a[pos++]=elem; } else while(elem=b[i++]) a[pos++]=elem; return a}
Real Sites V8 SunSpider
Code|Objects|Events
Fraction of Code Executed
13
amaz
on
bingm
apeb
ay
face
book
hotm
ail
richar
ds
cryp
to
earle
y
splay
3d-rayt
race
bitops-
nsieve
cryp
to-a
es
mat
h-cor
dic
strin
g-tagclo
ud
0.00%
10.00%
20.00%
30.00%
40.00%
50.00%
60.00%
70.00%
80.00%
90.00%
100.00%
Fra
cti
on
of
Co
de
Exe
cu
ted
Most codenot executed
Real Sites V8 SunSpider
Code|Objects|Events
Hot Function Distribution
14
Real Sites V8 Benchmarks
80% of time in < 10 functions80% of time in 100+ functions
80% 80%
Code|Objects|Events
Object Allocation Behavior
Code
Events
JavaScript
Objects
15
• Allocation by types
• Live heap composition
• Lifetime distribution
Total Bytes Allocated
16
amaz
on
bingm
apeb
ay
face
book
hotm
ail
richar
ds
cryp
to
earle
y
splay
3d-rayt
race
bitops-
nsieve
cryp
to-a
es
mat
h-cor
dic
strin
g-tagclo
ud
0
5000
10000
15000
20000
25000
30000
35000
40000
45000Tota
l h
eap
data
(kilob
yte
s)
Real Sites V8 SunSpider
Code|Objects|Events
Heap Data by Type
17
amazon
bingmapebay
hotmail
richard
s
crypto
earley
splay
access-
nbody
contro
lflow
date-xparb
regexp-dna
aggregate0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
ObjectsEnumaratorRegExpMathErrorDateNative FuncStringArraysScript FuncH
eap
Dat
a by
Typ
e
Real Sites V8 SunSpider
Many functionsRest are strings
Few benchmarksallocate much data
Code|Objects|Events
Live Heap Over Time (gmail)
18
Functions growsteadily
Code|Objects|Events
GC reducessize of heap
Objects growsteadily too
Live Heap over Time (ebay)
19
Heaps repeatedly created, discarded
Heap containsmostly functions
Code|Objects|Events
Heap drops to 0 on page load
2 Search Websites, 2 Architectures
Bing Google
Code|Objects|Events
20
You stay on the same page during your entire visitCode loaded onceHeap is bigger
Every transition loads a new pageCode loaded repeatedlyHeap is smaller
Event Handlers in JavaScript
Code
Events
JavaScript
Objects
21
• Number of events• Sizes of handlers
Event-driven Programming Model• Single-threaded, non-preemptive event
handlers• Example handlers: onabort, onclick, etc.• Very different from batch processing of
benchmarks• Handler responsiveness critical to user
experience
22
Code|Objects|Events
Total Events Handled
23
amaz
onbin
g
bingm
ap cnn
ebay
econ
omist
face
book
gmail
map
hotm
ail
richar
ds
delta
blue
cryp
to
rayt
race
earle
y
regex
psp
lay
0
1,000
2,000
3,000
4,000
5,000
6,000
7,000T
ota
l E
ve
nts
Ha
nd
led
Real Sites V8
Almost no events
Code|Objects|Events
Median Bytecodes / Event Handled
24am
azon
bing
bing
map cn
neb
ay
econ
omist
face
book
gmail
goog
le
goog
lem
ap
hotm
ail
0
50
100
150
200
250
300
350
400
450
500M
ed
ian
Byte
co
de
s /
Eve
nt
Ha
nd
led
Code|Objects|Events
506 2137
Sure, this is all good, but…
• Everyone knows benchmarks are unrepresentative
• How much difference does it make, anyway?• Wouldn’t any benchmarks have similar issues?
25
Cold-code Experiment• Observation
– Real web apps have lots of code (much of it cold)– Benchmarks do not
• Question: What happens if the benchmarks have more code?– We added extra, unused to code to 7 SunSpider
benchmarks– We measured the impact on the benchmark
performance
26
Performance Impact of Cold Code
27
3d-raytr
ace
access
-nbody
bitops-n
sieve
contro
lflow
crypto-ae
s
math-co
rdic
string-t
agclo
ud0
100
200
300
400
500
600
700
800
900
0K200K400K800K1M2M
Tim
e (m
sec)
3d-raytr
ace
access
-nbody
bitops-n
sieve
contro
lflow
crypto-ae
s
math-co
rdic
string-t
agclo
ud0
100
200
300
400
500
600
700
800
900
0K200K400K800K1M2M
Tim
e (m
sec)
Chrome3.0.195.38
IE 88.0.601.18865
Cold code has non-uniform impacton execution time
Cold code makesSunSpider on Chromeup to 4.5x slower
Impact of Benchmarks
• What gets emphasis– Making tight loops fast– Optimizing small amounts of code
• Important issues ignored– Garbage collection (especially of strings)– Managing large amounts of code– Optimizing event handling – Considering JavaScript context between page loads
28
Conclusions• JSMeter is an instrumentation framework
– Used to measure and compare JavaScript applications– High-level views of behavior promote understanding
• Benchmarks differ significantly from real sites– Misleads designers, skews implementations
• Next steps– Develop and promote better benchmarks– Design and evaluate better JavaScript runtimes– Promote better performance tools for JavaScript
developers
29
Additional Resources
• Project: http://research.microsoft.com/en-us/projects/jsmeter/ • Video: Project JSMeter: JavaScript Performance Analysis in the Real World
" - MSDN Channel 9 interview with Erik Meier, Ben Livshits, and Ben Zorn
• Paper:– “JSMeter: Comparing the Behavior of JavaScript Benchmarks with Real Web
Applications”, Paruj Ratanaworabhan, Benjamin Livshits and Benjamin G. Zorn, USENIX 2010 Conference on Web Application Development (WebApps’10), June 2010.
• Additional measurements:– "JSMeter: Characterizing Real-World Behavior of JavaScript Programs", Paruj
Ratanaworabhan, Benjamin Livshits, David Simmons, and Benjamin Zorn, MSR-TR-2009-173, December 2009 (49 pages), November 2009.
30
Additional Slides
31
Related Work• JavaScript
– Richards, Lebresne, Burg, and Vitek (PLDI’10)– Draw similar conclusions
• Java– Doufour et al. (OOPSLA’03), Dieckmann and U.
Hölzle (ECOOP’99)• Other languages
– C++: Calder et al. (JPL’95)– Interpreted languages: Romer et al. (ASPLOS’96)
32