Advances in Memory Management - Oracle › projects › tmem › dist › documentation › ... ·...
Transcript of Advances in Memory Management - Oracle › projects › tmem › dist › documentation › ... ·...
![Page 1: Advances in Memory Management - Oracle › projects › tmem › dist › documentation › ... · 2010-11-01 · Motivation •Memory is increasingly becoming a bottleneck in virtualized](https://reader034.fdocuments.net/reader034/viewer/2022050719/5ed73f12d37f9f58ca6a8aaa/html5/thumbnails/1.jpg)
<Insert Picture Here>
Advances in Memory Managementin a Virtual Environment
Speaker: Dan Magenheimer
Oracle CorporationLinux Plumbers
Conference 2010
![Page 2: Advances in Memory Management - Oracle › projects › tmem › dist › documentation › ... · 2010-11-01 · Motivation •Memory is increasingly becoming a bottleneck in virtualized](https://reader034.fdocuments.net/reader034/viewer/2022050719/5ed73f12d37f9f58ca6a8aaa/html5/thumbnails/2.jpg)
Agenda
• Motivation, “The Problem” and the Challenge
• Memory Optimization Solutions in a Virtual Environment
• Transcendent Memory (“tmem”) Overview
• Self-ballooning + Tmem Performance Analysis
Advances in Memory Management in a Virtualized Environment (LPC 2010) - Dan Magenheimer
NOTE: FOCUS IS ON
NOT ON:
AND
![Page 3: Advances in Memory Management - Oracle › projects › tmem › dist › documentation › ... · 2010-11-01 · Motivation •Memory is increasingly becoming a bottleneck in virtualized](https://reader034.fdocuments.net/reader034/viewer/2022050719/5ed73f12d37f9f58ca6a8aaa/html5/thumbnails/3.jpg)
Motivation
•Memory is increasingly becoming a
bottleneck in virtualized system
• Existing mechanisms have major holes
Four underutilized 2-cpu virtual servers
each with 1GB RAMballooning
Advances in Memory Management in a Virtualized Environment (LPC 2010) - Dan Magenheimer
One 4-CPU physical server w/4GB RAM
each with 1GB RAM
X
��������
����X
page sharing
memory overcommitment
![Page 4: Advances in Memory Management - Oracle › projects › tmem › dist › documentation › ... · 2010-11-01 · Motivation •Memory is increasingly becoming a bottleneck in virtualized](https://reader034.fdocuments.net/reader034/viewer/2022050719/5ed73f12d37f9f58ca6a8aaa/html5/thumbnails/4.jpg)
More motivation:
The memory capacity wall
100
1000
# Core
GB DRAM
Capacity Wall
� Memory capacity per core drop ~30% every 2 years
1
10
2003
2004
2005
2006
2007
2008
2009
2010
2011
2012
2013
2014
2015
2016
2017
Wall
http://isca09.cs.columbia.edu/pres/24.pptxSource: Disaggregated Memory for Expansion and Sharing in Blade Server
Advances in Memory Management in a Virtualized Environment (LPC 2010) - Dan Magenheimer
![Page 5: Advances in Memory Management - Oracle › projects › tmem › dist › documentation › ... · 2010-11-01 · Motivation •Memory is increasingly becoming a bottleneck in virtualized](https://reader034.fdocuments.net/reader034/viewer/2022050719/5ed73f12d37f9f58ca6a8aaa/html5/thumbnails/5.jpg)
More motivation: Energy Savings
“…several studies show the contribution of memory to the total cost and power consumption of future systems increasing
from its current value of about 25%...”
Go
og
le D
ata
Ce
nte
r in
Be
lgiu
m
Source: Disaggregated Memory Architectures for Blade Servers, Kevin Lim, Univ Michigan, PhD Thesis
Go
og
le D
ata
Ce
nte
r in
Be
lgiu
m
Advances in Memory Management in a Virtualized Environment (LPC 2010) - Dan Magenheimer
![Page 6: Advances in Memory Management - Oracle › projects › tmem › dist › documentation › ... · 2010-11-01 · Motivation •Memory is increasingly becoming a bottleneck in virtualized](https://reader034.fdocuments.net/reader034/viewer/2022050719/5ed73f12d37f9f58ca6a8aaa/html5/thumbnails/6.jpg)
<Insert Picture Here>
PSEUDO-RAM
Slide from: Linux kernel support to exploit phase change memory, Linux Symposium 2010, Youngwoo Park, EE KAIST
![Page 7: Advances in Memory Management - Oracle › projects › tmem › dist › documentation › ... · 2010-11-01 · Motivation •Memory is increasingly becoming a bottleneck in virtualized](https://reader034.fdocuments.net/reader034/viewer/2022050719/5ed73f12d37f9f58ca6a8aaa/html5/thumbnails/7.jpg)
Disaggregated memory concept
CPUsDIMM
DIMM
CPUsDIMM
DIMM
CPUsDIMM
DIMM
CPUsDIMM
DIMM
DIMM
DIMM
Ba
ckp
lan
e
DIMM
DIMM
DIMM
DIMM
� Break CPU-memory co-location
Leverage fast, shared communication fabrics
Memory blade
CPUsDIMM
CPUsDIMM
DIMM
Ba
ckp
lan
eDIMM
7
http://isca09.cs.columbia.edu/pres/24.pptx
Source: Disaggregated Memory for Expansion and Sharing in Blade Server
Advances in Memory Management in a Virtualized Environment (LPC 2010) - Dan Magenheimer
![Page 8: Advances in Memory Management - Oracle › projects › tmem › dist › documentation › ... · 2010-11-01 · Motivation •Memory is increasingly becoming a bottleneck in virtualized](https://reader034.fdocuments.net/reader034/viewer/2022050719/5ed73f12d37f9f58ca6a8aaa/html5/thumbnails/8.jpg)
1
10
100
1000
2003
2004
2005
2006
2007
2008
2009
2010
2011
2012
2013
2014
2015
2016
2017
# Core
GB DRAM
“HARD TO PREDICT THE FUTURE IS” --Yoda
Advances in Memory Management in a Virtualized Environment (LPC 2010) - Dan Magenheimer
2003
2004
2005
2006
2007
2008
2009
2010
2011
2012
2013
2014
2015
2016
2017
One 4-CPU physical server w/4GB RAM
Four underutilized 2-cpu virtual servers
each with 1GB RAM
X
��������
����X
![Page 9: Advances in Memory Management - Oracle › projects › tmem › dist › documentation › ... · 2010-11-01 · Motivation •Memory is increasingly becoming a bottleneck in virtualized](https://reader034.fdocuments.net/reader034/viewer/2022050719/5ed73f12d37f9f58ca6a8aaa/html5/thumbnails/9.jpg)
The “Meat” of the Problem
• Operating systems
are memory hogs!OS
Advances in Memory Management in a Virtualized Environment (LPC 2010) - Dan Magenheimer
Memory constraint
![Page 10: Advances in Memory Management - Oracle › projects › tmem › dist › documentation › ... · 2010-11-01 · Motivation •Memory is increasingly becoming a bottleneck in virtualized](https://reader034.fdocuments.net/reader034/viewer/2022050719/5ed73f12d37f9f58ca6a8aaa/html5/thumbnails/10.jpg)
• Operating systems are
memory hogs!
If you give an
OS
The “Meat” of the Problem
Advances in Memory Management in a Virtualized Environment (LPC 2010) - Dan Magenheimer
If you give an
operating system
more memory…..
New larger memory constraint
![Page 11: Advances in Memory Management - Oracle › projects › tmem › dist › documentation › ... · 2010-11-01 · Motivation •Memory is increasingly becoming a bottleneck in virtualized](https://reader034.fdocuments.net/reader034/viewer/2022050719/5ed73f12d37f9f58ca6a8aaa/html5/thumbnails/11.jpg)
• Operating systems are
memory hogs!
If you give an OS more
memory
My name is Linux and I
am a memory hog
The “Meat” of the Problem
Advances in Memory Management in a Virtualized Environment (LPC 2010) - Dan Magenheimer
memory
…it uses up any
memory you give it!
memory hog
Memory constraint
![Page 12: Advances in Memory Management - Oracle › projects › tmem › dist › documentation › ... · 2010-11-01 · Motivation •Memory is increasingly becoming a bottleneck in virtualized](https://reader034.fdocuments.net/reader034/viewer/2022050719/5ed73f12d37f9f58ca6a8aaa/html5/thumbnails/12.jpg)
The Virtualized Physical Memory
Resource Optimization Challenge
Optimize, across time, the distribution of RAM (and
future “pseudo-RAM”?) among a maximal set of
virtual machines by:
• measuring the current and future memory need of
each running VM and
Advances in Memory Management in a Virtualized Environment (LPC 2010) - Dan Magenheimer
each running VM and
• reclaiming memory from those VMs that have an
excess of memory and either:
• providing it to VMs that need more memory or
• using it to provision additional new VMs.
• without suffering a significant performance penalty
First step… put those pigs on a diet?
![Page 13: Advances in Memory Management - Oracle › projects › tmem › dist › documentation › ... · 2010-11-01 · Motivation •Memory is increasingly becoming a bottleneck in virtualized](https://reader034.fdocuments.net/reader034/viewer/2022050719/5ed73f12d37f9f58ca6a8aaa/html5/thumbnails/13.jpg)
OS Memory “Asceticism”
ASSUME that it is “a good thing” for the an OS
to use as little RAM as possible at any given moment
• motivation may be economic or power or virtualization or ???
SUPPOSE there is a mechanism for the OS to surrender RAM
that it doesn’t need at this moment, so it can “pursue goodness”
SUPPOSE there is a mechanism for the OS to ask for and obtain a
page (or more) of RAM when it needs more RAM than it currently has
THEN… HOW does the OS decide how much RAM it “needs”?
as-cet-i-cism, n. 1. extreme self-denial and austerity; rigorous self-discipline and active restraint; renunciation of material comforts so as to achieve a higher state
Advances in Memory Management in a Virtualized Environment (LPC 2010) - Dan Magenheimer
![Page 14: Advances in Memory Management - Oracle › projects › tmem › dist › documentation › ... · 2010-11-01 · Motivation •Memory is increasingly becoming a bottleneck in virtualized](https://reader034.fdocuments.net/reader034/viewer/2022050719/5ed73f12d37f9f58ca6a8aaa/html5/thumbnails/14.jpg)
Agenda
• Motivation and Challenge
• Memory Optimization Solutions in a Virtual Environment
• Transcendent Memory (“tmem”) Overview
• Self-ballooning + Tmem Performance Analysis
Advances in Memory Management in a Virtualized Environment (LPC 2010) - Dan Magenheimer
![Page 15: Advances in Memory Management - Oracle › projects › tmem › dist › documentation › ... · 2010-11-01 · Motivation •Memory is increasingly becoming a bottleneck in virtualized](https://reader034.fdocuments.net/reader034/viewer/2022050719/5ed73f12d37f9f58ca6a8aaa/html5/thumbnails/15.jpg)
VMM Physical Memory Management
Solutions
Solution Set A: Just let each guest hog all
memory given to it, but…
Solution Set B: Guest memory is dynamically
adjustable …somehowadjustable …somehow
Solution Set C: Total guest memory is dynamically
load-balanced across all guests …using some policy
Solution Set D: Host-provided “compensation” … to
correct for insufficiently omniscient policy
Advances in Memory Management in a Virtualized Environment (LPC 2010) - Dan Magenheimer
![Page 16: Advances in Memory Management - Oracle › projects › tmem › dist › documentation › ... · 2010-11-01 · Motivation •Memory is increasingly becoming a bottleneck in virtualized](https://reader034.fdocuments.net/reader034/viewer/2022050719/5ed73f12d37f9f58ca6a8aaa/html5/thumbnails/16.jpg)
VMM Physical Memory Management
Solution Set A
Solution Set A: Each guest hogs all memory given to it
• Partitioning
• Host swapping
• Transparent page sharing
Advances in Memory Management in a Virtualized Environment (LPC 2010) - Dan Magenheimer
![Page 17: Advances in Memory Management - Oracle › projects › tmem › dist › documentation › ... · 2010-11-01 · Motivation •Memory is increasingly becoming a bottleneck in virtualized](https://reader034.fdocuments.net/reader034/viewer/2022050719/5ed73f12d37f9f58ca6a8aaa/html5/thumbnails/17.jpg)
• By default, Xenpartitions memory• Xen memory
• dom0 memory
• guest 1 memory
guest
fallow
fallow
fallow
VMM Physical Memory Management
Partitioning (= NO overcommitment)
Advances in Memory Management in a Virtualized Environment (LPC 2010) - Dan Magenheimer
• guest 1 memory
• guest 2 memory
• whatever’s left over: “fallow” memory
guest
fallow
fallow, adj., land left without a crop for one or more years
![Page 18: Advances in Memory Management - Oracle › projects › tmem › dist › documentation › ... · 2010-11-01 · Motivation •Memory is increasingly becoming a bottleneck in virtualized](https://reader034.fdocuments.net/reader034/viewer/2022050719/5ed73f12d37f9f58ca6a8aaa/html5/thumbnails/18.jpg)
• Xen partitions memory among more guests• Xen memory
• dom0 memory
• guest 1 memory
• guest 2 memory
guest
guest
gues
tfallow
fallow
VMM Physical Memory Management
Partitioning (= NO overcommitment)
Advances in Memory Management in a Virtualized Environment (LPC 2010) - Dan Magenheimer
• guest 2 memory
• guest 3…
• BUT still fallow memoryleftover
guest
fallow
fallow, adj., land left without a crop for one or more years
![Page 19: Advances in Memory Management - Oracle › projects › tmem › dist › documentation › ... · 2010-11-01 · Motivation •Memory is increasingly becoming a bottleneck in virtualized](https://reader034.fdocuments.net/reader034/viewer/2022050719/5ed73f12d37f9f58ca6a8aaa/html5/thumbnails/19.jpg)
• Any page may be either in RAM or on disk
• Tricks like compression can reduce disk writes
• But still…
VMM Physical Memory Management
Host Swapping (SLOW overcommitment)
Advances in Memory Management in a Virtualized Environment (LPC 2010) - Dan Magenheimer
• But still…
Storage Technology Response
time (ns)
Typical disk (seek) 8000000
DDR3-1600 5
![Page 20: Advances in Memory Management - Oracle › projects › tmem › dist › documentation › ... · 2010-11-01 · Motivation •Memory is increasingly becoming a bottleneck in virtualized](https://reader034.fdocuments.net/reader034/viewer/2022050719/5ed73f12d37f9f58ca6a8aaa/html5/thumbnails/20.jpg)
VMM Physical Memory Management
Transparent Page Sharing (aka “KSM”)
(“FAUX” overcommitment)
• Keep one copy of identical pages
• Scan (huge swaths of memory) periodically for matches
• BUT…
Advances in Memory Management in a Virtualized Environment (LPC 2010) - Dan Magenheimer
• very workload dependent
• sometimes causes host swapping (resulting in unpredictable performance)
• poor match for 2MB pages
• BUT…
![Page 21: Advances in Memory Management - Oracle › projects › tmem › dist › documentation › ... · 2010-11-01 · Motivation •Memory is increasingly becoming a bottleneck in virtualized](https://reader034.fdocuments.net/reader034/viewer/2022050719/5ed73f12d37f9f58ca6a8aaa/html5/thumbnails/21.jpg)
VMM Physical Memory Management
Solution Set A Summary
Solution Set A: Each guest hogs all memory given to it
• Partitioning
• NO overcommitment
• Host swapping
• SLOW overcommitment• SLOW overcommitment
• like living in a swapstorm
• Transparent page sharing
• “FAUX” (fake) overcommitment, but
• advantage is very workload dependent
• inconsistent, variable performance, “cliffs”
• “semantic gap” between host and guest
Advances in Memory Management in a Virtualized Environment (LPC 2010) - Dan Magenheimer
![Page 22: Advances in Memory Management - Oracle › projects › tmem › dist › documentation › ... · 2010-11-01 · Motivation •Memory is increasingly becoming a bottleneck in virtualized](https://reader034.fdocuments.net/reader034/viewer/2022050719/5ed73f12d37f9f58ca6a8aaa/html5/thumbnails/22.jpg)
VMM Physical Memory Management
Solutions
Solution Set A: Each guest hogs all memory given to it,
but…
Solution Set B: Guest memory is dynamically
adjustable …somehowadjustable …somehow
Solution Set C: Total guest memory is dynamically
load-balanced across all guests …using some policy
Solution Set D: Host-provided “compensation” … to
correct for insufficiently omniscient policy
Advances in Memory Management in a Virtualized Environment (LPC 2010) - Dan Magenheimer
![Page 23: Advances in Memory Management - Oracle › projects › tmem › dist › documentation › ... · 2010-11-01 · Motivation •Memory is increasingly becoming a bottleneck in virtualized](https://reader034.fdocuments.net/reader034/viewer/2022050719/5ed73f12d37f9f58ca6a8aaa/html5/thumbnails/23.jpg)
VMM Physical Memory Management
Solution Set B
Solution Set B: Guest memory is dynamically adjustable
• Balloon driver
• “Virtual Hot plug” memory
Advances in Memory Management in a Virtualized Environment (LPC 2010) - Dan Magenheimer
![Page 24: Advances in Memory Management - Oracle › projects › tmem › dist › documentation › ... · 2010-11-01 · Motivation •Memory is increasingly becoming a bottleneck in virtualized](https://reader034.fdocuments.net/reader034/viewer/2022050719/5ed73f12d37f9f58ca6a8aaa/html5/thumbnails/24.jpg)
VMM Physical Memory Management
Balloon driver
• In-guest driver under the control of the host
• a “memory trojan horse”
Advances in Memory Management in a Virtualized Environment (LPC 2010) - Dan Magenheimer
![Page 25: Advances in Memory Management - Oracle › projects › tmem › dist › documentation › ... · 2010-11-01 · Motivation •Memory is increasingly becoming a bottleneck in virtualized](https://reader034.fdocuments.net/reader034/viewer/2022050719/5ed73f12d37f9f58ca6a8aaa/html5/thumbnails/25.jpg)
• very workload dependent
• sometimes causes host swapping (resulting in
VMM Physical Memory Management
Ballooning• In-guest driver under the control of the host
• a “memory trojan horse”
• BUT…
Advances in Memory Management in a Virtualized Environment (LPC 2010) - Dan Magenheimer
swapping (resulting in unpredictable performance)
• poor match for 2MB pages
![Page 26: Advances in Memory Management - Oracle › projects › tmem › dist › documentation › ... · 2010-11-01 · Motivation •Memory is increasingly becoming a bottleneck in virtualized](https://reader034.fdocuments.net/reader034/viewer/2022050719/5ed73f12d37f9f58ca6a8aaa/html5/thumbnails/26.jpg)
VMM Physical Memory Management
Virtual Hot Plug memory
• Fools the OS’s native hot-plug memory interface
• BUT…
Advances in Memory Management in a Virtualized Environment (LPC 2010) - Dan Magenheimer
• only useful for higher granularity
• hot-plug interface not designed for high frequency changes or mid-size granularity
• hot plug delete is problematic
• BUT…
![Page 27: Advances in Memory Management - Oracle › projects › tmem › dist › documentation › ... · 2010-11-01 · Motivation •Memory is increasingly becoming a bottleneck in virtualized](https://reader034.fdocuments.net/reader034/viewer/2022050719/5ed73f12d37f9f58ca6a8aaa/html5/thumbnails/27.jpg)
VMM Physical Memory Management
Solution Set B (Summary)
Solution Set B: Guest memory is dynamically adjustable
• Ballooning
• unpredictable side effects
• Hot plug memory
• Low granularity• Low granularity
Advances in Memory Management in a Virtualized Environment (LPC 2010) - Dan Magenheimer
![Page 28: Advances in Memory Management - Oracle › projects › tmem › dist › documentation › ... · 2010-11-01 · Motivation •Memory is increasingly becoming a bottleneck in virtualized](https://reader034.fdocuments.net/reader034/viewer/2022050719/5ed73f12d37f9f58ca6a8aaa/html5/thumbnails/28.jpg)
VMM Physical Memory Management
Solution Set B (Summary)
Solution Set B: Guest memory is dynamically adjustable
• Ballooning
• unpredictable side effects
• Hot plug memory
• Low granularity• Low granularity
These are mechanisms, not solutions!
Advances in Memory Management in a Virtualized Environment (LPC 2010) - Dan Magenheimer
![Page 29: Advances in Memory Management - Oracle › projects › tmem › dist › documentation › ... · 2010-11-01 · Motivation •Memory is increasingly becoming a bottleneck in virtualized](https://reader034.fdocuments.net/reader034/viewer/2022050719/5ed73f12d37f9f58ca6a8aaa/html5/thumbnails/29.jpg)
VMM Physical Memory Management
Solutions
Solution Set A: Each guest hogs all memory given to it,
but…
Solution Set B: Guest memory is dynamically
adjustable …somehow
Solution Set C: Total guest memory is dynamically
load-balanced across all guests …using some
policy
Solution Set D: Host-provided “compensation” … to
correct for insufficiently omniscient policy
Advances in Memory Management in a Virtualized Environment (LPC 2010) - Dan Magenheimer
![Page 30: Advances in Memory Management - Oracle › projects › tmem › dist › documentation › ... · 2010-11-01 · Motivation •Memory is increasingly becoming a bottleneck in virtualized](https://reader034.fdocuments.net/reader034/viewer/2022050719/5ed73f12d37f9f58ca6a8aaa/html5/thumbnails/30.jpg)
VMM Physical Memory Management
Solution Set C
Solution Set C: Guests are dynamically “load balanced”
using some policy
• Guest-quantity-based policy
• Guest-pressure-driven host-control policy
• Guest-pressure-driven guest-control policy• Guest-pressure-driven guest-control policy
Advances in Memory Management in a Virtualized Environment (LPC 2010) - Dan Magenheimer
![Page 31: Advances in Memory Management - Oracle › projects › tmem › dist › documentation › ... · 2010-11-01 · Motivation •Memory is increasingly becoming a bottleneck in virtualized](https://reader034.fdocuments.net/reader034/viewer/2022050719/5ed73f12d37f9f58ca6a8aaa/html5/thumbnails/31.jpg)
• administrator presets memory “range” for each guest
• balloons adjusted based on number of guests
• does NOT respond to
VMM Physical Memory ManagementCitrix Dynamic Memory Control (DMC)for Xen Cloud Platform (XCP)
Advances in Memory Management in a Virtualized Environment (LPC 2010) - Dan Magenheimer
• does NOT respond to individual guest memory pressure
http://wiki.xensource.com/xenwiki/Dynamic_Memory_Control
![Page 32: Advances in Memory Management - Oracle › projects › tmem › dist › documentation › ... · 2010-11-01 · Motivation •Memory is increasingly becoming a bottleneck in virtualized](https://reader034.fdocuments.net/reader034/viewer/2022050719/5ed73f12d37f9f58ca6a8aaa/html5/thumbnails/32.jpg)
• collects host and guest memory stats, sends to customizable policy engine
• controls all guest balloons, plus host page sharing (KSM)
VMM Physical Memory Management
KVM Memory Overcommitment Manager
Advances in Memory Management in a Virtualized Environment (LPC 2010) - Dan Magenheimer
• shrinks all guests “fairly” scaled by host memory pressure
BUT…• under-aggressive for idle guests
• issues due to lack of omniscience
http://wiki.github.com/aglitke/mom
![Page 33: Advances in Memory Management - Oracle › projects › tmem › dist › documentation › ... · 2010-11-01 · Motivation •Memory is increasingly becoming a bottleneck in virtualized](https://reader034.fdocuments.net/reader034/viewer/2022050719/5ed73f12d37f9f58ca6a8aaa/html5/thumbnails/33.jpg)
Ballooning works great for giving more memory TO a guest OS…
Look ma! No more
guest
guest
guest
VMM Physical Memory Managementin the presence of under-aggressive ballooning
Advances in Memory Management in a Virtualized Environment (LPC 2010) - Dan Magenheimer
fallow memory! (*burp*)guest
guest
guest
![Page 34: Advances in Memory Management - Oracle › projects › tmem › dist › documentation › ... · 2010-11-01 · Motivation •Memory is increasingly becoming a bottleneck in virtualized](https://reader034.fdocuments.net/reader034/viewer/2022050719/5ed73f12d37f9f58ca6a8aaa/html5/thumbnails/34.jpg)
• migration
• requires fallow memory
in the target machine
guest
guest
gue
stfallow
fallow
fallow
VMM Physical Memory Management
under-aggressive ballooning limits migration
Physical machine “B”
Advances in Memory Management in a Virtualized Environment (LPC 2010) - Dan Magenheimer
in the target machine
• leaves behind fallow
memory in the
originating machine
guest
guest
gue
st
fallow
fallow
fallow
Physical machine “A”
![Page 35: Advances in Memory Management - Oracle › projects › tmem › dist › documentation › ... · 2010-11-01 · Motivation •Memory is increasingly becoming a bottleneck in virtualized](https://reader034.fdocuments.net/reader034/viewer/2022050719/5ed73f12d37f9f58ca6a8aaa/html5/thumbnails/35.jpg)
• In Xen tree since mid-2008
• Use in-guest feedback to resize balloon• aggressively
• frequently
• independently
• configurably
VMM Physical Memory Management
Self-ballooning
• For Linux, size to maximum of:
• /proc/meminfo “CommittedAS”
• memory floor enforced by Xen balloon driver
• Userland daemon or patched kernel
Advances in Memory Management in a Virtualized Environment (LPC 2010) - Dan Magenheimer
guest
Committed_AS: An estimate of how much RAM you would need to make a 99.99% guarantee that there never is OOM (out of memory) for this workload. Normally the kernel will overcommit memory. The Committed_AS is a guesstimate of how much RAM/swap you would need worst-case. (From http://www.redhat.com/advice/tips/meminfo.html)
![Page 36: Advances in Memory Management - Oracle › projects › tmem › dist › documentation › ... · 2010-11-01 · Motivation •Memory is increasingly becoming a bottleneck in virtualized](https://reader034.fdocuments.net/reader034/viewer/2022050719/5ed73f12d37f9f58ca6a8aaa/html5/thumbnails/36.jpg)
• “enforced memory asceticism”
• ballooning does not work well to take memory away
VMM Physical Memory Management
over-aggressive ballooning
Advances in Memory Management in a Virtualized Environment (LPC 2010) - Dan Magenheimer
![Page 37: Advances in Memory Management - Oracle › projects › tmem › dist › documentation › ... · 2010-11-01 · Motivation •Memory is increasingly becoming a bottleneck in virtualized](https://reader034.fdocuments.net/reader034/viewer/2022050719/5ed73f12d37f9f58ca6a8aaa/html5/thumbnails/37.jpg)
Memory Asceticism / Aggressive Self-ballooning
ISSUES
ISSUE #1: Pages evicted due to memory pressure are most likely to be clean page cache pages. Eliminating these (without a crystal ball) results in refaults�additional disk reads
ISSUE #2: When no more clean pagecache pages can be evicted, dirty mapped pages get written … and rewritten… and rewritten to disk � additional disk writesand rewritten to disk � additional disk writes
ISSUE #3: Sudden large memory demands may occur unpredictably (e.g. from a new userland program launch) but the “ask for” mechanism can’t deliver enough memory fast enough� failed mallocs, swapping, and OOMs
Advances in Memory Management in a Virtualized Environment (LPC 2010) - Dan Magenheimer
![Page 38: Advances in Memory Management - Oracle › projects › tmem › dist › documentation › ... · 2010-11-01 · Motivation •Memory is increasingly becoming a bottleneck in virtualized](https://reader034.fdocuments.net/reader034/viewer/2022050719/5ed73f12d37f9f58ca6a8aaa/html5/thumbnails/38.jpg)
Memory Asceticism / Aggressive Self-ballooning
ISSUES
ISSUE #1: Pages evicted due to memory pressure are most likely to be clean pagecache pages. Eliminating these (without a crystal ball) results in refaults�additional disk reads
ISSUE #2: When no more clean pagecache pages can be evicted, dirty mapped pages get written … and rewritten… and rewritten to disk � additional disk writesand rewritten to disk � additional disk writes
ISSUE #3: Sudden large memory demands may occur unpredictably (e.g. from a new userland program launch) but the “ask for” mechanism can’t deliver enough memory fast enough� failed mallocs, swapping, and OOMs
Advances in Memory Management in a Virtualized Environment (LPC 2010) - Dan Magenheimer
![Page 39: Advances in Memory Management - Oracle › projects › tmem › dist › documentation › ... · 2010-11-01 · Motivation •Memory is increasingly becoming a bottleneck in virtualized](https://reader034.fdocuments.net/reader034/viewer/2022050719/5ed73f12d37f9f58ca6a8aaa/html5/thumbnails/39.jpg)
Memory Asceticism / Aggressive Self-ballooning
ISSUES
ISSUE #1: Pages evicted due to memory pressure are most likely to be clean pagecache pages. Eliminating these (without a crystal ball) results in refaults�additional disk reads
ISSUE #2: When no more clean pagecache pages can be evicted, dirty mapped pages get written … and rewritten… and rewritten to disk � additional disk writesand rewritten to disk � additional disk writes
ISSUE #3: Sudden large memory demands may occur unpredictably (e.g. from a new userland program launch) but the “ask for” mechanism can’t deliver enough memory fast enough� failed mallocs, swapping, and OOMs
Advances in Memory Management in a Virtualized Environment (LPC 2010) - Dan Magenheimer
![Page 40: Advances in Memory Management - Oracle › projects › tmem › dist › documentation › ... · 2010-11-01 · Motivation •Memory is increasingly becoming a bottleneck in virtualized](https://reader034.fdocuments.net/reader034/viewer/2022050719/5ed73f12d37f9f58ca6a8aaa/html5/thumbnails/40.jpg)
VMM Physical Memory Management
Solution Set C Summary
Solution Set C: Guests are dynamically “load balanced”
using some policy
• Guest-quantity-based policy
• Guest-pressure-driven host-control policy
• Guest-pressure-driven guest-control policy• Guest-pressure-driven guest-control policy
Advances in Memory Management in a Virtualized Environment (LPC 2010) - Dan Magenheimer
���� ALL POLICIES SUCK HAVE ISSUES BECAUSE:
1) MEMORY PRESSURE IS DIFFICULT TO MEASURE
2) HARD TO PREDICT THE FUTURE IS (Yoda)
![Page 41: Advances in Memory Management - Oracle › projects › tmem › dist › documentation › ... · 2010-11-01 · Motivation •Memory is increasingly becoming a bottleneck in virtualized](https://reader034.fdocuments.net/reader034/viewer/2022050719/5ed73f12d37f9f58ca6a8aaa/html5/thumbnails/41.jpg)
VMM Physical Memory Management
Solutions
Solution Set A: Each guest hogs all memory given to it,
but…
Solution Set B: Guest memory is dynamically
adjustable …somehow
Solution Set C: Total guest memory is dynamically
load-balanced across all guests …using some policy
Solution Set D: Host-provided “compensation”
… to correct for poor or non-omniscient policy
Advances in Memory Management in a Virtualized Environment (LPC 2010) - Dan Magenheimer
![Page 42: Advances in Memory Management - Oracle › projects › tmem › dist › documentation › ... · 2010-11-01 · Motivation •Memory is increasingly becoming a bottleneck in virtualized](https://reader034.fdocuments.net/reader034/viewer/2022050719/5ed73f12d37f9f58ca6a8aaa/html5/thumbnails/42.jpg)
Agenda
• Motivation and Challenge
• Memory Optimization Solutions in a Virtual Environment
• Transcendent Memory (“tmem”) Overview
• Self-ballooning + Tmem Performance Analysis
Advances in Memory Management in a Virtualized Environment (LPC 2010) - Dan Magenheimer
![Page 43: Advances in Memory Management - Oracle › projects › tmem › dist › documentation › ... · 2010-11-01 · Motivation •Memory is increasingly becoming a bottleneck in virtualized](https://reader034.fdocuments.net/reader034/viewer/2022050719/5ed73f12d37f9f58ca6a8aaa/html5/thumbnails/43.jpg)
Transcendent memory
creating the transcendent memory pool
• Step 1a: reclaim all fallow memory
• Step 1b: reclaim wasted guest
memory (e.g. via self-ballooning)
• Step 1c: collect it all into a pool
guest
guest
guestfallow
fallow
Advances in Memory Management in a Virtualized Environment (LPC 2010) - Dan Magenheimer
Transcendentmemorypool
guest
fallow
![Page 44: Advances in Memory Management - Oracle › projects › tmem › dist › documentation › ... · 2010-11-01 · Motivation •Memory is increasingly becoming a bottleneck in virtualized](https://reader034.fdocuments.net/reader034/viewer/2022050719/5ed73f12d37f9f58ca6a8aaa/html5/thumbnails/44.jpg)
Transcendent memory
creating the transcendent memory pool
• Step 2: provide indirect
access, strictly controlled by
the hypervisor and dom0
control
Transcendentmemory
guest
guest
guest
data
data
data
Advances in Memory Management in a Virtualized Environment (LPC 2010) - Dan Magenheimer
memorypool
guest
data
data
control
![Page 45: Advances in Memory Management - Oracle › projects › tmem › dist › documentation › ... · 2010-11-01 · Motivation •Memory is increasingly becoming a bottleneck in virtualized](https://reader034.fdocuments.net/reader034/viewer/2022050719/5ed73f12d37f9f58ca6a8aaa/html5/thumbnails/45.jpg)
Transcendent memory
API characteristics
Transcendent memory API
• paravirtualized (lightly)
• narrow
• well-specified
• operations are:
• synchronous
guest guest
Advances in Memory Management in a Virtualized Environment (LPC 2010) - Dan Magenheimer
• synchronous
• page-oriented (one page per op)
• copy-based
• multi-faceted
• extensibleTranscendentmemorypool
![Page 46: Advances in Memory Management - Oracle › projects › tmem › dist › documentation › ... · 2010-11-01 · Motivation •Memory is increasingly becoming a bottleneck in virtualized](https://reader034.fdocuments.net/reader034/viewer/2022050719/5ed73f12d37f9f58ca6a8aaa/html5/thumbnails/46.jpg)
Transcendent memory
four different subpool types
���� four different uses
flags ephemeral persistent
private “second-chance”
clean-page cache!!
� “cleancache”
Fast swap
“device”!!
� “frontswap”
Legend:Legend:
Implemented and Implemented and working todayworking today(Linux + (Linux + XenXen))
Working but Working but
Advances in Memory Management in a Virtualized Environment (LPC 2010) - Dan Magenheimer
� “cleancache” � “frontswap”
shared server-side cluster
filesystem cache
� “shared
cleancache”
inter-guest shared
memory?
Working but Working but limited testinglimited testing
Under Under investigationinvestigation
eph-em-er-al, adj.,… transitory, existing only briefly, short-lived (i.e. NOT persistent)
![Page 47: Advances in Memory Management - Oracle › projects › tmem › dist › documentation › ... · 2010-11-01 · Motivation •Memory is increasingly becoming a bottleneck in virtualized](https://reader034.fdocuments.net/reader034/viewer/2022050719/5ed73f12d37f9f58ca6a8aaa/html5/thumbnails/47.jpg)
Tmem guest kernel paravirtualization
cleancache
Filesystem restrictions to use cleancache
“Cleancache is a proposed new optional feature to be provided by the VFS layer that potentially dramatically increases page cache effectiveness for many
workloads in many environments at a negligible cost. Filesystems that are well-behaved and conform to certain restrictions can utilize cleancache simply by
making a call to cleancache_init_fs() at mount time. Unusual, misbehaving, or poorly layered filesystems must either add additional hooks and/or undergo
extensive additional testing… or should just not enable the optional cleancache.”
Filesystem restrictions to use cleancache• Little or no value for RAM-based filesystems
• Coherency: File removal/truncation must layer on VFS• or FS must add additional hooks to do same (issue in FScache net FS’s?)
• Inode numbers must be unique• no emulating 64-bit inode space on 32-bit inode numbers
• Superblock alloc/deactivate must layer on VFS• or FS must add additional hooks to do same
• Performance: Page fetching via VFS• or FS must add additional hooks to do same (e.g. btrfs)
• FS blocksize should match PAGE_SIZE• or existing backends will ignore
• Clustered FS should use “shared_init_fs” for best performance• on some backends, ignored on others
Advances in Memory Management in a Virtualized Environment (LPC 2010) - Dan Magenheimer
![Page 48: Advances in Memory Management - Oracle › projects › tmem › dist › documentation › ... · 2010-11-01 · Motivation •Memory is increasingly becoming a bottleneck in virtualized](https://reader034.fdocuments.net/reader034/viewer/2022050719/5ed73f12d37f9f58ca6a8aaa/html5/thumbnails/48.jpg)
cleancache
• a second-chance clean
page cache for a guest
• “put” clean pages only
• “get” only valuable pages
• pages eventually are evicted
• coherency managed by guest
• exclusive cache semantics
Transcendentmemory pool
(private+ephemeral)“put”
Advances in Memory Management in a Virtualized Environment (LPC 2010) - Dan Magenheimer
• exclusive cache semantics
ephemeral persistent
private “second-chance”
clean-page cache!!
� “cleancache”
Fast swap
“device”!!
� “frontswap”
shared server-side cluster
filesystem cache?
� “shared cleancache”
inter-domain
shared memory?
Transcendent Memory Pool types
guest
“put”
“get”
![Page 49: Advances in Memory Management - Oracle › projects › tmem › dist › documentation › ... · 2010-11-01 · Motivation •Memory is increasingly becoming a bottleneck in virtualized](https://reader034.fdocuments.net/reader034/viewer/2022050719/5ed73f12d37f9f58ca6a8aaa/html5/thumbnails/49.jpg)
Memory Asceticism / Aggressive Self-ballooning
ISSUES
ISSUE #1: Pages evicted due to memory pressure are most likely to be clean pagecache pages. Eliminating these (without a crystal ball) results in refaults�additional disk reads
ISSUE #2: When no more clean pagecache pages can be evicted, dirty mapped pages get written … and rewritten… and rewritten to disk � additional disk writesand rewritten to disk � additional disk writes
ISSUE #3: Sudden large memory demands may occur unpredictably (e.g. from a new userland program launch) but the “ask for” mechanism can’t deliver enough memory fast enough� failed mallocs, swapping, and OOMs
Advances in Memory Management in a Virtualized Environment (LPC 2010) - Dan Magenheimer
![Page 50: Advances in Memory Management - Oracle › projects › tmem › dist › documentation › ... · 2010-11-01 · Motivation •Memory is increasingly becoming a bottleneck in virtualized](https://reader034.fdocuments.net/reader034/viewer/2022050719/5ed73f12d37f9f58ca6a8aaa/html5/thumbnails/50.jpg)
“Frontswap is meant to deal with dirty pages that the kernel would like to get rid of… Like cleancache, frontswap can play tricks with stored pages to
stretch its memory resources. The real purpose behind this mechanism, though, appears to be to enable a hypervisor to respond quickly to memory
usage spikes in virtualized guests. Dan put it this way:
Frontswap serves nicely as an emergency safety valve when a guest has given up (too)
Tmem guest kernel paravirtualization
frontswap
Frontswap serves nicely as an emergency safety valve when a guest has given up (too) much of its memory via ballooning but unexpectedly has an urgent need that can’t be
serviced quickly enough by the balloon driver.
-- lwn.net, May 4, 2010,
http://lwn.net/Articles/386090/
Advances in Memory Management in a Virtualized Environment (LPC 2010) - Dan Magenheimer
![Page 51: Advances in Memory Management - Oracle › projects › tmem › dist › documentation › ... · 2010-11-01 · Motivation •Memory is increasingly becoming a bottleneck in virtualized](https://reader034.fdocuments.net/reader034/viewer/2022050719/5ed73f12d37f9f58ca6a8aaa/html5/thumbnails/51.jpg)
frontswap
• over-ballooned guests
experiencing unexpected
memory pressure have an
emergency swap disk
• much faster than swapping
• persistent (“dirty”) pages OK
Advances in Memory Management in a Virtualized Environment (LPC 2010) - Dan Magenheimer
• prioritized higher than hcache
• limited by domain’s maxmem
Transcendent Memory Pool types
ephemeral persistent
private “second-chance”
clean-page cache!!
� “cleancache”
Fast swap
“device”!!
� “frontswap”
shared server-side cluster
filesystem cache?
� “shared cleancache”
inter-domain
shared memory?
![Page 52: Advances in Memory Management - Oracle › projects › tmem › dist › documentation › ... · 2010-11-01 · Motivation •Memory is increasingly becoming a bottleneck in virtualized](https://reader034.fdocuments.net/reader034/viewer/2022050719/5ed73f12d37f9f58ca6a8aaa/html5/thumbnails/52.jpg)
Memory Asceticism / Aggressive Self-ballooning
ISSUES
ISSUE #1: Pages evicted due to memory pressure are most likely to be clean pagecache pages. Eliminating these (without a crystal ball) results in refaults�additional disk reads
ISSUE #2: When no more clean pagecache pages can be evicted, dirty mapped pages get written … and rewritten… and rewritten to disk � additional disk writesand rewritten to disk � additional disk writes
ISSUE #3: Sudden large memory demands may occur unpredictably (e.g. from a new userland program launch) but the “ask for” mechanism can’t deliver enough memory fast enough� failed mallocs, swapping, and OOMs
Advances in Memory Management in a Virtualized Environment (LPC 2010) - Dan Magenheimer
![Page 53: Advances in Memory Management - Oracle › projects › tmem › dist › documentation › ... · 2010-11-01 · Motivation •Memory is increasingly becoming a bottleneck in virtualized](https://reader034.fdocuments.net/reader034/viewer/2022050719/5ed73f12d37f9f58ca6a8aaa/html5/thumbnails/53.jpg)
Transcendent Memory Status
• Tmem support officially released in Xen 4.0.0
• Optional compression and page deduplication support
• Enterprise-quality concurrency
• Complete save/restore and live migration support
• Linux-side patches available, including• ocfs2, btrfs, ext3, ext4 filesystem support
Advances in Memory Management in a Virtualized Environment (LPC 2010) - Dan Magenheimer
• ocfs2, btrfs, ext3, ext4 filesystem support
• sysfs support for in-guest tmem statistics
• targeting upstream Linux 2.6.37 (cleancache), 2.6.38 (frontswap)
• Tmem “technology preview” releases:• Oracle VM 2.2
• OpenSuSE 11.2; SLE11 (?)
• Oracle Linux 5 update 5 rpm
![Page 54: Advances in Memory Management - Oracle › projects › tmem › dist › documentation › ... · 2010-11-01 · Motivation •Memory is increasingly becoming a bottleneck in virtualized](https://reader034.fdocuments.net/reader034/viewer/2022050719/5ed73f12d37f9f58ca6a8aaa/html5/thumbnails/54.jpg)
Agenda
• Motivation and Challenge
• Memory Optimization Solutions in a Virtual Environment
• Transcendent Memory (“tmem”) Overview
• Self-ballooning + Tmem Performance Analysis
Advances in Memory Management in a Virtualized Environment (LPC 2010) - Dan Magenheimer
![Page 55: Advances in Memory Management - Oracle › projects › tmem › dist › documentation › ... · 2010-11-01 · Motivation •Memory is increasingly becoming a bottleneck in virtualized](https://reader034.fdocuments.net/reader034/viewer/2022050719/5ed73f12d37f9f58ca6a8aaa/html5/thumbnails/55.jpg)
Test workload (overcommitted!)
• Dual core (Conroe) processor, 2GB RAM, IDE disk
• Four single vcpu PV VMs, in-kernel self-ballooning+tmem• Oracle Enterprise Linux 5 update 4; two 32-bit + two 64-bit
• mem=384MB (maxmem=512MB)… total = 1.5GB (2GB maxmem)
• virtual block device is tap:aio (file contains 3 LVM partitions: ext3+ext3+swap)
• Each VM waits for all VMs to be ready, then simultaneously
• two Linux kernel compiles (2.6.32 source), then force crash:
Advances in Memory Management in a Virtualized Environment (LPC 2010) - Dan Magenheimer
• two Linux kernel compiles (2.6.32 source), then force crash:• make clean; make –j8; make clean; make –j8
• echo c > /proc/sysrq-trigger
• Dom0: 256MB fixed, 2 vcpus• automatically launches all domains
• checks every 60s, waiting for all to be crashed
• saves away statistics, then reboots
![Page 56: Advances in Memory Management - Oracle › projects › tmem › dist › documentation › ... · 2010-11-01 · Motivation •Memory is increasingly becoming a bottleneck in virtualized](https://reader034.fdocuments.net/reader034/viewer/2022050719/5ed73f12d37f9f58ca6a8aaa/html5/thumbnails/56.jpg)
Measurement methodology• Four statistics measured for each run
• Temporal: (1) wallclock time to completion; (2) total vcpu including dom0
• Disk access: vbd sectors (3) read and (4) written
• Test workload run five times for each configuration
• high and low sample of each statistic discarded
• use average of middle three samples for “single-value” statistic
• Five different configurations:
Features Self- Tmem Page Compression
Advances in Memory Management in a Virtualized Environment (LPC 2010) - Dan Magenheimer
Features
enabled
Configuration
Self-
ballooningTmem Page
Dedup
Compression
Unchanged NO NO NO NO
Self-ballooning YES NO NO NO
Tmem YES YES NO NO
Tmem w/dedup YES YES YES NO
Tmem w/dedup+ comp YES YES YES YES
![Page 57: Advances in Memory Management - Oracle › projects › tmem › dist › documentation › ... · 2010-11-01 · Motivation •Memory is increasingly becoming a bottleneck in virtualized](https://reader034.fdocuments.net/reader034/viewer/2022050719/5ed73f12d37f9f58ca6a8aaa/html5/thumbnails/57.jpg)
Unchanged vs. Self-ballooning only
Temporal stats
Advances in Memory Management in a Virtualized Environment (LPC 2010) - Dan Magenheimer
![Page 58: Advances in Memory Management - Oracle › projects › tmem › dist › documentation › ... · 2010-11-01 · Motivation •Memory is increasingly becoming a bottleneck in virtualized](https://reader034.fdocuments.net/reader034/viewer/2022050719/5ed73f12d37f9f58ca6a8aaa/html5/thumbnails/58.jpg)
Unchanged vs. Self-ballooning only
Virtual block device stats
Advances in Memory Management in a Virtualized Environment (LPC 2010) - Dan Magenheimer
![Page 59: Advances in Memory Management - Oracle › projects › tmem › dist › documentation › ... · 2010-11-01 · Motivation •Memory is increasingly becoming a bottleneck in virtualized](https://reader034.fdocuments.net/reader034/viewer/2022050719/5ed73f12d37f9f58ca6a8aaa/html5/thumbnails/59.jpg)
AS EXPECTED: a performance hit!
Aggressive ballooning (by itself) doesn’t work very well!
• Self-ballooning indiscriminately shrinks the guest OS’s
page cache, causing refaults!
� PERFORMANCE WILL GET WORSEWHEN LARGE-
MEMORY GUESTS ARE AGGRESSIVELY BALLOONED
Advances in Memory Management in a Virtualized Environment (LPC 2010) - Dan Magenheimer
guest
![Page 60: Advances in Memory Management - Oracle › projects › tmem › dist › documentation › ... · 2010-11-01 · Motivation •Memory is increasingly becoming a bottleneck in virtualized](https://reader034.fdocuments.net/reader034/viewer/2022050719/5ed73f12d37f9f58ca6a8aaa/html5/thumbnails/60.jpg)
Self-ballooning AND Transcendent Memory…go together like a horse and carriage
• Self-ballooned memory is returned
to Xen and absorbed by tmem
• Most tmem memory can be
instantly reclaimed when needed
for a memory-needy or new guest
• Tmem also provides a safety valve
when ballooning is not fast enoughguest
guest
guestfallow
fallow
Advances in Memory Management in a Virtualized Environment (LPC 2010) - Dan Magenheimer
when ballooning is not fast enough
Transcendentmemorypool
guest
fallow
![Page 61: Advances in Memory Management - Oracle › projects › tmem › dist › documentation › ... · 2010-11-01 · Motivation •Memory is increasingly becoming a bottleneck in virtualized](https://reader034.fdocuments.net/reader034/viewer/2022050719/5ed73f12d37f9f58ca6a8aaa/html5/thumbnails/61.jpg)
79% utilization*
Self-ballooning AND Tmem
Temporal stats
Advances in Memory Management in a Virtualized Environment (LPC 2010) - Dan Magenheimer
72% utilization*
* 2 cores
5%-8% faster completion
![Page 62: Advances in Memory Management - Oracle › projects › tmem › dist › documentation › ... · 2010-11-01 · Motivation •Memory is increasingly becoming a bottleneck in virtualized](https://reader034.fdocuments.net/reader034/viewer/2022050719/5ed73f12d37f9f58ca6a8aaa/html5/thumbnails/62.jpg)
31-52% reductionin sectors read
Self-ballooning AND Tmem
virtual block device stats
Advances in Memory Management in a Virtualized Environment (LPC 2010) - Dan Magenheimer
(no significant change in sectors written)
![Page 63: Advances in Memory Management - Oracle › projects › tmem › dist › documentation › ... · 2010-11-01 · Motivation •Memory is increasingly becoming a bottleneck in virtualized](https://reader034.fdocuments.net/reader034/viewer/2022050719/5ed73f12d37f9f58ca6a8aaa/html5/thumbnails/63.jpg)
WOW! Why is tmem so good?• Tmem-enabled guests statistically multiplex one shared
virtual page cache to reduce disk refaults!• 252068 page (984MB) max (NOTE: actual tmem measurement)
• Deduplication and compression together transparently
QUADRUPLE apparent size of this virtual page cache!• 953166 page (3723MB) max (actually measured by tmem… on 2GB system!)
• Swapping-to-disk (e.g. due to insufficiently responsive ballooning)
Advances in Memory Management in a Virtualized Environment (LPC 2010) - Dan Magenheimer
• Swapping-to-disk (e.g. due to insufficiently responsive ballooning)
is converted to in-memory copies and statistically multiplexed• 82MB at workload completion, 319MB combined max (actual measurement)
• uses compression but not deduplication
• CPU “costs” entirely hidden by increased CPU utilization
� RESULTS MAY BE EVEN BETTERWHEN WORKLOAD
IS TEMPORALLY DISTRIBUTED/SPARSE
![Page 64: Advances in Memory Management - Oracle › projects › tmem › dist › documentation › ... · 2010-11-01 · Motivation •Memory is increasingly becoming a bottleneck in virtualized](https://reader034.fdocuments.net/reader034/viewer/2022050719/5ed73f12d37f9f58ca6a8aaa/html5/thumbnails/64.jpg)
Transcendent Memory Update
Summary
Tmem advantages:
• greatly increased memory utilization/flexibility
• dramatic reduction in I/O bandwidth requirements
• more effective CPU utilization
• faster completion of (some?) workloads
Advances in Memory Management in a Virtualized Environment (LPC 2010) - Dan Magenheimer
• faster completion of (some?) workloads
Tmem disadvantages:
• tmem-modified kernel required (cleancache and frontswap)
• higher power consumption due to higher CPU utilization
![Page 65: Advances in Memory Management - Oracle › projects › tmem › dist › documentation › ... · 2010-11-01 · Motivation •Memory is increasingly becoming a bottleneck in virtualized](https://reader034.fdocuments.net/reader034/viewer/2022050719/5ed73f12d37f9f58ca6a8aaa/html5/thumbnails/65.jpg)
Cleancache and Frontswap in ActionOracle Linux 5u5 (with tmem+selfballooning patch) on Xen 4.0
Advances in Memory Management in a Virtualized Environment (LPC 2010) - Dan Magenheimer
![Page 66: Advances in Memory Management - Oracle › projects › tmem › dist › documentation › ... · 2010-11-01 · Motivation •Memory is increasingly becoming a bottleneck in virtualized](https://reader034.fdocuments.net/reader034/viewer/2022050719/5ed73f12d37f9f58ca6a8aaa/html5/thumbnails/66.jpg)
For more information
http://oss.oracle.com/projects/tmemor xen-unstable.hg/docs/misc/tmem-internals.html
Advances in Memory Management in a Virtualized Environment (LPC 2010) - Dan Magenheimer
![Page 67: Advances in Memory Management - Oracle › projects › tmem › dist › documentation › ... · 2010-11-01 · Motivation •Memory is increasingly becoming a bottleneck in virtualized](https://reader034.fdocuments.net/reader034/viewer/2022050719/5ed73f12d37f9f58ca6a8aaa/html5/thumbnails/67.jpg)
BACKUP SLIDESBACKUP SLIDES
Advances in Memory Management in a Virtualized Environment (LPC 2010) - Dan Magenheimer
![Page 68: Advances in Memory Management - Oracle › projects › tmem › dist › documentation › ... · 2010-11-01 · Motivation •Memory is increasingly becoming a bottleneck in virtualized](https://reader034.fdocuments.net/reader034/viewer/2022050719/5ed73f12d37f9f58ca6a8aaa/html5/thumbnails/68.jpg)
Memory Technology Comparison
http://www.eetimes.com/design/memory-design/4204936/The-evolution-of-phase-change-memory
Table from: EE Times, July 26 2010, Greg Atwood
����
����
Advances in Memory Management in a Virtualized Environment (LPC 2010) - Dan Magenheimer
![Page 69: Advances in Memory Management - Oracle › projects › tmem › dist › documentation › ... · 2010-11-01 · Motivation •Memory is increasingly becoming a bottleneck in virtualized](https://reader034.fdocuments.net/reader034/viewer/2022050719/5ed73f12d37f9f58ca6a8aaa/html5/thumbnails/69.jpg)
☺☺☺☺
����
☺☺☺☺
Slide from: Linux kernel support to exploit phase change memory, Linux Symposium 2010, Youngwoo Park, EE KAIST. Data from Numonyx white paper, “PCM: A new memory technology to enable new memory usage models”
☺☺☺☺����
Advances in Memory Management in a Virtualized Environment (LPC 2010) - Dan Magenheimer
![Page 70: Advances in Memory Management - Oracle › projects › tmem › dist › documentation › ... · 2010-11-01 · Motivation •Memory is increasingly becoming a bottleneck in virtualized](https://reader034.fdocuments.net/reader034/viewer/2022050719/5ed73f12d37f9f58ca6a8aaa/html5/thumbnails/70.jpg)
• What does an OS do
with all that memory?
• Kernel code and data
• User code and data
• Page cache!
Kernel code
User data
OS Physical Memory Management
Advances in Memory Management in a Virtualized Environment (LPC 2010) - Dan Magenheimer
• Page cache!Kerneldata
User code
User dataPage cache
Everythingelse
![Page 71: Advances in Memory Management - Oracle › projects › tmem › dist › documentation › ... · 2010-11-01 · Motivation •Memory is increasingly becoming a bottleneck in virtualized](https://reader034.fdocuments.net/reader034/viewer/2022050719/5ed73f12d37f9f58ca6a8aaa/html5/thumbnails/71.jpg)
• What does an OS do
with all that memory?pagecache
OS Physical Memory Management
Page cache attempts to predict future needs of
Advances in Memory Management in a Virtualized Environment (LPC 2010) - Dan Magenheimer
predict future needs of pages from the disk…
sometimes it gets it right���� “good” pages
![Page 72: Advances in Memory Management - Oracle › projects › tmem › dist › documentation › ... · 2010-11-01 · Motivation •Memory is increasingly becoming a bottleneck in virtualized](https://reader034.fdocuments.net/reader034/viewer/2022050719/5ed73f12d37f9f58ca6a8aaa/html5/thumbnails/72.jpg)
• What does an OS do
with all that memory?pagecache
OS Physical Memory Management
Page cache attempts to predict future needs of
Advances in Memory Management in a Virtualized Environment (LPC 2010) - Dan Magenheimer
predict future needs of pages from the disk…
sometimes it gets it wrong���� “wasted” pages
![Page 73: Advances in Memory Management - Oracle › projects › tmem › dist › documentation › ... · 2010-11-01 · Motivation •Memory is increasingly becoming a bottleneck in virtualized](https://reader034.fdocuments.net/reader034/viewer/2022050719/5ed73f12d37f9f58ca6a8aaa/html5/thumbnails/73.jpg)
• What does an OS do
with all that memory?
…much of the time
mostly page cache
page cache
OS Physical Memory Management
Advances in Memory Management in a Virtualized Environment (LPC 2010) - Dan Magenheimer
… some of which will
be useful in the future
… and some (or
maybe most…)
of which is wastedEverything else
![Page 74: Advances in Memory Management - Oracle › projects › tmem › dist › documentation › ... · 2010-11-01 · Motivation •Memory is increasingly becoming a bottleneck in virtualized](https://reader034.fdocuments.net/reader034/viewer/2022050719/5ed73f12d37f9f58ca6a8aaa/html5/thumbnails/74.jpg)
Motivation: Memory-inefficient
workloads
Advances in Memory Management in a Virtualized Environment (LPC 2010) - Dan Magenheimer