VHPC'12: Pre-Copy and Post-Copy VM Live Migration for Memory Intensive Applications
-
Upload
aidanshribman -
Category
Technology
-
view
1.008 -
download
1
Transcript of VHPC'12: Pre-Copy and Post-Copy VM Live Migration for Memory Intensive Applications
Benoit Hudzia Sr. Researcher; SAP Research CEC Belfast (UK)
Aidan Shribman Sr. Researcher; SAP Research Israel
With the contribution of Roei Tell, Steve Walsh, Peter Izsak and Chaim Bendelac
Pre-Copy and Post-Copy VM Live Migration
for Memory Intensive Applications
© 2012 SAP AG. All rights reserved. 2
Agenda
Introduction
Live Migration
Memory Intensive Apps
Pre-Copy Live Migration
Post-Copy Live Migration
Summary
Introduction The challenge of live migrating memory intensive applications
© 2012 SAP AG. All rights reserved. 4
What is live migration?
Moving of a guest VM running on host A (the source host) to
host B (the destination host) without noticeable service
disruption.
Typically used for dynamic resource load balancing in cloud
environments. So the faster to migrate and the less the service
disruption the better.
.
© 2012 SAP AG. All rights reserved. 5
What is VM migration?
CPU state and device driver state migration
MBs of information can be transferred while VM is stopped
Memory state migration
GBs of information must (at least mostly) be transferred while VM is live
Requirement: is that memory can be migrated during guest execution
Storage state migration
TBs of information (requiring minutes up to hours to move)
Recommendation: use shared storage (accessed via NFS, etc)
NIC relocation
ARP announcement that IP address has been taken over
Requirement: both hosts reside on the same LAN
© 2012 SAP AG. All rights reserved. 6
Memory Intensive Applications
An application dominated by high rate of memory read/write accesses
Why must memory be transferred mid execution?
16 GBs takes 12 seconds to migrate over 10 GbE (1.25 GB/s) interconnects
12 seconds downtime would cause severe service disruption
The challenge (of live migration implementations) is to transfer memory a mid high rates of
memory shurn
Pre-Copy Live Migration Reducing the amount of page re-sends and the cost of page re-sends
© 2012 SAP AG. All rights reserved. 8
Pre-Copy Memory Transfer
Memory is transferred before VM relocation
The problem is how to copy memory while it is re-dirtied over and
over again by the guest VM
Solved by first copying all the memory followed by intervals of
copying newly dirtied pages until remaining state is small enough
Implemented by most (all) hypervisors VMWare, Xen, Qemu
Challenged by fast memory dirtying applications
© 2012 SAP AG. All rights reserved. 9
Pre-Copy Live Migration (pre-migration)
Host A Host B
Guest VM
Iterative
Pre-copy X
Rounds
Stop
and
Copy
Commit
Total Migration Time
Downtime Live on A Live on B
Reservation Pre-migrate
Degraded on A
© 2012 SAP AG. All rights reserved. 10
Pre-Copy Live Migration (reservation)
Host A Host B
Guest VM Guest VM
Iterative
Pre-copy X
Rounds
Stop
and
Copy
Commit
Total Migration Time
Downtime Live on A Live on B
Reservation Pre-migrate
Degraded on A
© 2012 SAP AG. All rights reserved. 11
Pre-Copy Live Migration (pre-copy)
Host A Host B
Guest VM Guest VM
Iterative
Pre-copy X
Rounds
Stop
and
Copy
Commit
Total Migration Time
Downtime Live on A Live on B
Reservation Pre-migrate
Degraded on A
© 2012 SAP AG. All rights reserved. 12
Pre-Copy Live Migration (stop and copy)
Host A Host B
Guest VM Guest VM
Iterative
Pre-copy X
Rounds
Stop
and
Copy
Commit
Total Migration Time
Downtime Live on A Live on B
Reservation Pre-migrate
Degraded on A
}
© 2012 SAP AG. All rights reserved. 13
Pre-Copy Live Migration (commit)
Host A Host B
Guest VM
Iterative
Pre-copy X
Rounds
Stop
and
Copy
Commit
Total Migration Time
Downtime Live on A Live on B
Reservation Pre-migrate
Degraded on A
© 2012 SAP AG. All rights reserved. 14
LRU Page Reordering (reducing number of pages transferred)
An LRU queue is used for tracking all historic page
dirty events (i.e. application wrote to memory)
QEMU (responsible for the live migration process)
queries the Linux Kernel KVM hypervisor for dirty
page events. It then inserts dirty page events into
the LRU queue (after a random shuffle)
Dirty pages (maintained in a separate LRU queue)
are inserted according to historic LRU queue and
transmitted head first
LRU reordering may have an adverse effect for
some workloads requiring a smarter cache (ARC?)
© 2012 SAP AG. All rights reserved. 15
XBZRLE Delta Encoder (reducing cost of page resent)
Cache page with highest dirtying rate during send operation
If old page is found in cache delta compress it; Else transfer new page uncompressed
Implemented using an LRU cache configured to fit working set size
Classic Pre-Copy
Delta Compression Pre-Copy
© 2012 SAP AG. All rights reserved. 16
XBZRLE Delta Encoder - Example
XBRLE (XOR + RLE) presented in VEE 2011: Evaluation of Delta Compression Techniques for
Efficient Live Migration of Large Virtual Machines by Benoit, Svard, Tordsson and Elmroth
XBZRLE (XOR + ZRLE) the current effort already released to QEMU community
0 1 2 3 4 5 6 7 Size
Old ‘a’ ‘b’ ‘c’ ‘d’ ‘e’ ‘f’ ‘g’ ‘h’ 8
New ‘a’ ‘b’ ‘c’ 0x00 0x01 ‘f’ ‘g’ ‘h’ 8
XOR 0x00 0x00 0x00 ‘d’ ‘f’ 0x00 0x00 0x00 8
RLE 0x00 0x03 ‘d’ 0x01 ‘f’ 0x01 0x00 0x03 8
ZRLE 0x00 0x03 ‘d’ ‘f’ 0x00 0x03 6
© 2012 SAP AG. All rights reserved. 17
XBZRLE Delta Encoder - Micro Benchmark
Sparse Medium Dense Very Dense
Jump 1111 701 203 121
Dirty 12 33 41 43
© 2012 SAP AG. All rights reserved. 18
Pre-copy Live Migration Evaluation
Host: 1 core; 2 GB
Benchmark: synthetic dirtying 512 MB; 1 T
Host: 2 core; 8 GB; Bandwidth 240 Mbps
Benchmark: appmembench dirtying 512 MB at 1 GB/s 8 T
Post-Copy Live Migration Fast live migration … but at a cost?
© 2012 SAP AG. All rights reserved. 20
Post-Copy Memory Transfer
Memory is transferred only after VM relocation
The problem is how to ensure that VM performance is not degraded (after
relocation) due to expensive network bound page faults
Solved by using fast interconnects and improved page fault mechanism
References: Yobusame PoC for KVM Linux (using char device user mode
transfer)
Challenged by fast memory reading applications
© 2012 SAP AG. All rights reserved. 21
Post-Copy Live Migration (pre-migration)
Host A Host B
Guest VM
Page Pushing
1
Round
Stop
and
Copy
Commit
Total Migration Time
Downtime Live on A Degraded on B Live on B
Reservation Pre-migrate
© 2012 SAP AG. All rights reserved. 22
Post-Copy Live Migration (reservation)
Host A Host B
Guest VM Guest VM
Page Pushing
1
Round
Stop
and
Copy
Commit
Total Migration Time
Downtime Live on A Degraded on B Live on B
Reservation Pre-migrate
© 2012 SAP AG. All rights reserved. 23
Post-Copy Live Migration (stop and copy)
Host A Host B
Guest VM Guest VM
Page Pushing
1
Round
Stop
and
Copy
Commit
Total Migration Time
Downtime Live on A Degraded on B Live on B
Reservation Pre-migrate
© 2012 SAP AG. All rights reserved. 24
Post-Copy Live Migration (post-copy)
Host A Host B
Guest VM Guest VM
Page Pushing
1
Round
Stop
and
Copy
Commit
Total Migration Time
Downtime Live on A Degraded on B Live on B
Reservation Pre-migrate
Page fault
Page push
© 2012 SAP AG. All rights reserved. 25
Post-Copy Live Migration (commit)
Host A Host B
Guest VM
Page Pushing
1
Round
Stop
and
Copy
Commit
Total Migration Time
Downtime Live on A Degraded on B Live on B
Reservation Pre-migrate
© 2012 SAP AG. All rights reserved. 26
Reducing Effects of Network Bound Page Faults
Low latency RDMA page transfer protocol (reducing latency/cost of page faults)
Implemented fully in kernel mode OFED VERBS
Can use the fastest RDMA hardware available (IB, IWARP, RoCE)
Demand pre-paging (pre-fetching) mechanism (reducing the number of page faults)
Currently only a simple fetching of pages surrounding page on which fault occured
Full Linux MMU integration (reducing the system-wide effects/cost of page fault)
Enabling to perform page fault transparency (only pausing the requesting thread)
Hybrid post-copy live migration (reducing the number of page faults)
Moving some of the pages to the destination in a bounded pre-copy phase prior to relocation
© 2012 SAP AG. All rights reserved. 27
Evaluation Pre-Copy vs. Post-Copy
Specification
Benchmark: Google Stress Application Test (SAT)
with a 1 GB working set
Communication: softIWARP over 1 GbE
Guest VM: 2 x vCPU; 1 GB – 14 GB memory
Results
Roughly the total migration time between pre-copy
to post-copy is 1:10
Summary
© 2012 SAP AG. All rights reserved. 30
Key takeaways
Pre-Copy and Post-Copy optimizations enhace the
support of live migrating memory intessive applications
To be released as open source under GPLv2 and LGPL
licenses to Qemu and Linux communities
Developed by SAP Research Technology Infrastructure
(TI) Practice
Thank you
Benoit Hudzia; Sr. Researcher;
SAP Research CEC Belfast
Aidan Shribman; Sr. Researcher;
SAP Research Israel
© 2012 SAP AG. All rights reserved. 32
Legal Disclaimer
The information in this presentation is confidential and proprietary to SAP and may not be disclosed without the permission of
SAP. This presentation is not subject to your license agreement or any other service or subscription agreement with SAP. SAP
has no obligation to pursue any course of business outlined in this document or any related presentation, or to develop or
release any functionality mentioned therein. This document, or any related presentation and SAP's strategy and possible future
developments, products and or platforms directions and functionality are all subject to change and may be changed by SAP at
any time for any reason without notice. The information on this document is not a commitment, promise or legal obligation to
deliver any material, code or functionality. This document is provided without a warranty of any kind, either express or implied,
including but not limited to, the implied warranties of merchantability, fitness for a particular purpose, or non-infringement. This
document is for informational purposes and may not be incorporated into a contract. SAP assumes no responsibility for errors or
omissions in this document, except if such damages were caused by SAP intentionally or grossly negligent.
All forward-looking statements are subject to various risks and uncertainties that could cause actual results to differ materially
from expectations. Readers are cautioned not to place undue reliance on these forward-looking statements, which speak only as
of their dates, and they should not be relied upon in making purchasing decisions.