1

ARMVISOR

Linux symposium 2012Jiun-Hung Ding, Chang-Jung Lin, Ping-Hao ChangChieh-Hao Tsang, Wei-Chung Hsu, Yeh-Ching Chung

2

ARMVISORHypervisor on ARM architecture

3

KVM

Linux Kernel module Free, open Many developers maintain it!!! QEMU

4

System Architecture

QEMU

Linux

Device

Hardware

KVMDriver

Guest

5

Trap & emulate

User space

Kernel space

Guest OS

VMM

trap

6

Challenges

ARM is non-virtualizable. Sensitive instructions Privileged instructions

7

Non-virtualizable

31 sensitive instructions

6 privilegedinstructio

ns

25 sensitive but non-

privileged instructions

25 critical instructions

8

Challenges

ARM is non-virtualizable Critical instructions

Possible solutions Hardware support Dynamic binary translation (DBT) Para-virtualization

9

Implementation

CPU virtualization Memory virtualization IO virtualization Optimization

Kernel space

KVM

User space

QEMU

Guest Mode

Guest OS

2. Return to QEMU

1. VM initialization

3. Run VM 4. Enter

Guest5. Exit Guest

Enter Guest

Exit Guest Return to

QEMU Run VM

Enter Guest

Lightweight trap

Heavyweight trap

11

CPU Virtualization

12

CPU Virtualization

ARM: non-virtualizable ISA Para-virtualization

Manually insert software interrupts (SWI) into guest OS

13

Para-virtualization

…mov r0, r0add sp, spmovs pc, lr…

14

Para-virtualization (cont.)

…mov r0, r0add sp, spvirt_svc_movs “movs pc, lr”…

.macro virt_svc_movs, instSWI 0x190\inst.endm

15

KVM Trap Entry

KVM/Guest Context Switch

Unit

Host Trap Handler

Instruction

Emulation

Exception/Interrupt

Emulation

MMU Emulati

on

QEMU I/O

Emulation

KVM Trap Dispatcher

UND ABORT SWI IRQ/FIQ

16

KVM Vector

oxffff0000

oxffff1000

Kernel Vector

FIQIRQ

(Reserved*)Data Abort

Prefetch AbortSoftware Interrupt

Undefined Instruction

Reset

0x1C0x180x140x100x0C0x080x040x00

0xffff001c

17

KVM Vector

oxffff0000

oxffff1000

KVMVector

The KVM trap

Interface

0xffff001c

18

Memory Virtualization

19

Guest PTB

GVA

GPA

20

Shadow Page Tables

Map guest virtual address to host physical address

For each guest page table (GPT), ARMvisor will allocate a shadow page table (SPT) to map it.

21

Guest PTB GVA

GPA

HVA

HPA

HostPTB

New SPTE !!!

22

Emulation Flow

Guest permissio

n checker

Shadow page table

update

Shadow page table

mapping

MMIO access checker

PABT/DABT trap

Hidden protection fault

Hidden translation fault

MMIO emulation

True Translation fault

True permission fault

guest page table

walker

23

Shadow Page Tables

Map guest virtual address to host physical address

For each guest page table (GPT), ARMvisor will allocate a shadow page table (SPT) to map it.

How to keep coherence between SPT and GPT?

25

Synchronization

Write protect page table page Modification would cause a protection

fault. Reverse map (RMAP) : Record reverse

mapping form guest physical page to SPT entries

26

Guest OS in non-privileged mode Some instructions access kernel

space with user permission LDRBT, LDRT, STRBT, STRT

Double shadow page table

Permission Emulation (1)

28

Permission Mapping

Permission mechanism on ARM Permission bits Domain

No access Client Manager

29

Using Domain Mechanism

GUD GKDVirtual User Space

CLIENT NA

Virtual Kernel Space

CLIENT CLIENT

30

Optimizations

31

Virtualization Overhead -CPU CPU virtualization

Frequent lightweight traps result lots of context switch

Try to reduce… number of traps Overhead of emulation

33

CPU Optimization

Shadow file register (SFR) Map VCPU’s shadow state of the register

file into memory region that is both accessible for the VMM and guest with RW permission.

34oxffff0000

oxffff1000

oxffff2000

KVM vector

KVM/GuestContext Switch

Interface

Shadow Register File

mcr cpsr, r1

Read/Write

Instructions

VCPU Register File

Sync

36

CPU Optimization

Shadow file register (SFR) Map VCPU’s shadow state of the register

file into memory region that is both accessible for the VMM and guest with RW permission.

Para-virtualization: Fast instruction trap Sets of pre-defined macros which is

composed of encoded information of the replaced instructions.

37

Virtualization Overhead –MEM Memory virtualization

Synchronization model

Try to reduce… Protection faults

38

Memory Optimization

Para-virtualization: hyper calls for PT modifications When Guest OS sets PTEs When Guest OS is going to free a L2

page table

39

Implementation Status

40

Experiment Board

ARMvisor supports ARMv6 & ARMv7 architecture in host

ARM v6 11mpcore

ARM v7 cortex-a8

41

Environment

Host OS: Linux 2.6.35 Ubuntu Guest OS: Linux 2.6.31

LMBench

42

Experiment Data

0

20

40

60

80

100

base cpu opt mem opt all opt

43

Experiment Data

0

2

4

6

8

10

12

14

1615.20

11.24 12.46

4.39

9.37

4.63 5.78

1.89 1.43 1.23

8.45

10.81 12.17

7.18 7.97 8.04

5.66 4.41

9.94

all opt

44

Related Work

VMware MVP Xen-ARM B LABS CODEZEOR OK Labs OKL4 Virtual Open System

45

Conclusion

ARMvisor supports ARMv6 & ARMv7 architecture in host Support RealView EB and Beagle Board

CPU and memory optimization gain lots of performance improvement Reduce traps Decrease protection faults

46

Future Work

I/O optimization Support ARM hardware virtualization

extensions Multi-VM Multi-core …

47

Contact Us

SSLAB, NTHU, Taiwan

Website https://sites.google.com/a/sslab.cs.nthu.

edu.tw/armvisor/

Thanks For Your Attention

49

IO Virtualization Overhead

QEMU

Linux

Device

Hardware

KVMDriver

Guest

Driver