Memory Buddies: Exploiting Page Sharing for Smart Colocation in Virtualized Data Centers Timothy...
-
Upload
merryl-whitehead -
Category
Documents
-
view
225 -
download
0
Transcript of Memory Buddies: Exploiting Page Sharing for Smart Colocation in Virtualized Data Centers Timothy...
Memory Buddies: Exploiting Page Sharing for Smart Colocation in
Virtualized Data Centers
Timothy Wood, Gabriel Tarasuk-Levin, Prashant Shenoy, Peter Desnoyers*,
Emmanuel Cecchet, and Mark D. Corner
University of Massachusetts, Amherst
*Northeastern University
Server Placement in Data Centers
• Virtualization improves resource utilization by consolidating servers
• But how to determine which servers to place together?
• Must consider many resource constraints:– CPU, Disk, Network, Memory
Why Memory?• CPU scheduling is fine grain
– Easily share among many users– Work conserving, so no waste
• Memory is much less flexible– Allocated on a large time scales– Being wrong (paging) is disastrous
• Memory is an expensive resource
• Memory capacity is increasing slower than CPU power
HypervisorPhysicalRAM
FREE
D
FREE
Content Based Page Sharing
VM 1Page Table
ABCB
C
A
A
D
B
VM 2Page Table
ADB
• If two VMs have an identical pages in memory, just keep one copy
• Supported by VMware ESX platform– Experimental tests in Xen,
further support planned
• Potential benefits– 33% in VMware ESX paper– 65% with subpage sharing
(Difference Engine) 1) Hypervisor detects identical pages
2) Copy-on-Write references created for shared pages
But what if….
• Pages change over time, breaking sharing
• If memory is being overcommitted, this can lead to hotspots
PhysicalRAM
FREE
D
FREE
B
C
A
VM 1Page Table
ABC
VM 2Page Table
ADB
A*
A*
PhysicalRAM
What’s the problem?
VM 4Page Table
ABCB
C
A
FREE
D
FREE
E
D
F
VM 3Page Table
EDF
• Only get a benefit if VMs on a machine actually have pages to share!
PhysicalRAM
VM 2Page Table
ABCB
C
A
FREE
D
FREE
E
D
F
VM 1Page Table
EDF
Host 1 Host 2
Where to place a VM?• How do you figure out which
VMs to place together?– Meet resource constraints– Maximize sharing
• Why placement is hard in large data centers?– Many applications from
different clients– Many software stacks /
platforms– Workloads change over time
Here or there?
Or there or there or there…?
?
Memory Buddies Goals
• Efficiently analyze the memory contents of multiple VMs to determine sharing potential
• Find more compact VM placement schemes
• Respond quickly to changing conditions to prevent memory hotspots
Bonus! Traces released at traces.cs.umass.eduBonus! Traces released at traces.cs.umass.edu
Outline
• Motivation
• Memory Fingerprinting & Comparison
• Sharing-aware Colocation
• Hotspot Mitigation
• Implementation & Evaluation
• Related Work & Summary
Memory Fingerprints• Hypervisor creates hash for each page
– Check hash table to see if page is sharable– Record these hashes to create fingerprint
• Hash lists are big– 32bits per 4K page = 1MB per 1GB of RAM– Need to forward fingerprint to other hosts
• Comparisons of lists is relatively slow
VM 1
AB
0x11223344
0x55667788
Bloom Filter Fingerprints• Bloom filter is a probabilistic data structure
– Stores keys by setting some bits to 1– False positive chance at lookup from hash
collisions
• Very space efficient• Tradeoff between filter size and accuracy
0 1 1 0 1… 0 0 1
M bits
Insert(key) --> set h1(key)=1 and h2(key)=1VM 1
0x11223344
0x55667788
Fingerprint Comparison• Hash list comparison
– Sort each list and then step through
• Bloom Filter– Simple method: Dot product of bit vectors
– Bloom Sharing Equation• Corrects for the expected number of false
matches in each filter• Impressively accurate!
1 0 0 0 0
0 1 0 0 11
1 1 1
1 1 11
1 01 1 1
1 1 11
1= 4
Eval: Fingerprinting
• 4GB RAM VMs– Hash: 4 sec– Sorted: 0.3 sec– Bloom: 0.02 sec
• Bloom Fingerprint 10% the size, still < 1% error
Bloom filters are smaller and 10 to 100 times fasterBloom filters are smaller and 10 to 100 times faster
Outline
• Motivation
• Memory Fingerprinting & Comparison
• Sharing-aware Colocation
• Hotspot Mitigation
• Implementation & Evaluation
• Related Work & Summary
Sharing Aware Placement• Where to place a freshly started VM?• Use staging area to find initial placement• Find feasible hosts• Estimate sharing potential• Migrate VM• Done!
Staging Host
1100011
Host 1
1010101
Host 3 Host 2
1110000
1000010
1110010+=
Compare
Consolidation & Hotspot Mitigation
• Resource usage changes over time– Sharing may not last forever
• Periodically consolidate servers– Identify candidates (least loaded hosts)– Match to destinations (hosts with best sharing)– Migrate VMs– Disable unnecessary servers
• Hotspot Mitigation– Monitor memory usage to detect hotspots– VMs may run out of memory if sharing stops– Redistribute VMs to rebalance
Offline Planning Tool
Offline Planning Tool
Host resources Resource Traces Memory Fingerprints
Dynamic programming based bin-packing tool
Finds subsets of VMs that can be placed together and
maximize sharing
Host 1
Host N
Number of hosts required = XVM to host mapping
Estimated sharing per host = Y
Outline
• Motivation
• Memory Fingerprinting & Comparison
• Sharing-aware Colocation
• Hotspot Mitigation
• Implementation & Evaluation
• Related Work & Summary
Implementation
• Memory Tracer– Tool used to gather data for trace study– Runs on Linux, OS X, and Windows– Calculates 32bit hashes for each page in
memory– Sends either a hash list or Bloom filter to
control node
• Works on physical systems or in VMs
Implementation• Nucleus
– Collects memory fingerprints for each VM
– Sends data tocontrol plane
• Control Plane– Gathers VM statistics and makes migration decisions
based on sharing– Interacts with VMware Virtual Infrastructure to
manage VMs
Eval: Trace StudySystem Mix
Total MemoryMB
% SharableUsed Mem MB (with sharing)
Linux 2.6.9
Darwin 9.0
Windows XP
4223 13.2% 3666
Darwin 9.0
Darwin 9.4
Windows XP
5248 35.3% 3397
Darwin 9.0
Darwin 9.2
Darwin 9.4
Windows XP
6272 36.8% 3966
Darwin 9.4
(3 MacBook + iMac)8192 40.0% 4917
Eval: App Placement• Try to place as many VMs onto a set of 4 hosts• Sharing Oblivious: Place on first host with sufficient
capacity• Four app types -- data contents different for each VM
instance
1 2 3 4Host
1 2 3 4Host
Sharing Oblivious
Sharing Aware
TPC-W
OFBiz
RUBiS
SpecJBB
17 VMs 20 VMs
Outline
• Motivation
• Memory Fingerprinting & Comparison
• Sharing-aware Colocation
• Hotspot Mitigation
• Implementation & Evaluation
• Related Work & Summary
Related Work
• Waldspurger, OSDI 2002– CBPS in VMware ESX Server
• Gupta, et al., OSDI 2008– Increase sharing potential by looking at
parts of pages
• VM Memory provisioning– Zhao & Wang (yesterday) has a good list!
Summary• Hypervisors already support page sharing…
• Memory Buddies makes it more useful– Identifies sharing opportunities across data center– Migrates VMs to maximize sharing– Uses efficient memory fingerprinting techniques to
scale to large data centers
• Traces will be online (soon) at:– http://traces.cs.umass.edu– Macbooks, Linux servers, and more!
• Questions? [email protected]