Live Migration of Virtual Machines

Christopher Clark, Keir Fraser, Steven Hand, Jacob Gorm Hanseny,

Eric July, Christian Limpach, Ian Pratt, Andrew Wareld

Index

• Motivation• Migration• Pre-copy migration• Challenge in Pre-copy• Implementation• Managed Migration(Xen)• Self Migration• Experiment• Optimization

Motivation

• Why we need live migration? We want to move VM without interrupting

VM service.

Products:• Xen 2.0 : live migration• VMware:VMotion

Migration

• Two important consideration: 1. Downtime 2. Total migration time

• All migrations mechanism can be divided into three phases.

1. Push phase 2. Stop-and-copy phase 3. Pull phase

Migration

Example:• Stop-and-copy => Only stop-and-copy phase• Pure demand-migration => Stop-and-copy phase and Pull phase• Pre-copy => Push phase and Stop-and-copy phase

Pre-copy migration

Challenge in Pre-copy

• Every VM have some set of pages which updates very frequently.

• And the Sets size would influence downtime of the migration.

ÞWritable Working Sets

• The bandwidth of network also impact the migration time.

Implementation

• There are two kinds of implementation of migration.– Managed Migration• The physical machine has a management VM. (Xen)

– Self Migration

• There are three conditions to change into stop-and-copy phase.– The dirty rate is bigger than the upper bound.– The size if working set is small enough.– The limit of the rounds.

pagesdirty Current #pagesdirty Privious # rateDirty

Managed Migration(Xen)1. 1st round: – Copy all memory pages to destination machine. – Replace original page table with shadow page table whose all pages are

marked read-only . – Create a dirty bit map for the VM.

2. 2nd-(n-1)th round: – During the pages transferring, if VM want to modify a page, it will invoke

Xen to set the appropriate bit in the dirty bit map. – Dirty pages will be resend again.– Reset dirty bit map for the next round.

3. nth round: – When the Dirty rate is bigger than up bound, begin to do stop-and-copy.

Self Migration1. 1st round:

– Copy all memory pages to destination machine. – Mark all virtual address as write-protect. – The OS create a dirty bit map(used to tracking physical pages) to memory the modified

pages.2. 2nd-(n-1)th round:

– During the pages transferring, if write faults occur, OS will set the appropriate bit in the dirty map.

– Dirty pages will resend again. – Reset dirty bit map for the next round.

3. nth round: – When the dirty rate is bigger than the upper bound, OS would disable all activities.

4. Copy all dirty pages to shadow buffer.

5. Transfer the buffer and ignore new page updates.

Web Server migration

SPECweb99 benchmark

A worst case example

Optimization

• Dynamic Rate-Limiting• Rapid Page Dirtying• Para-virtualized Optimization 1. Ballooning mechanism 2. Stunning rogue Process