connect.linaro.org

SFO17-410: NEVE: Nested Virtualization Extensions for ARMJintack Lim, Christoffer Dall, Shih-Wei Li, Jason Nieh, and Marc Zyngier

ENGINEERS AND DEVICESWORKING TOGETHER

Nested Virtualization

Hardware

Hypervisor

VM

VM

Kernel

App App

Hypervisor

VM

Kernel

App App

VM

Kernel

App App

ENGINEERS AND DEVICESWORKING TOGETHER

Nested Virtualization

Hardware

Hypervisor

VM

VM

Kernel

App App

Hypervisor

VM

Kernel

App App

VM

Kernel

App App

• IaaS hosting private clouds

• Develop and test your hypervisor

ENGINEERS AND DEVICESWORKING TOGETHER

Terminology

Hardware

Host Hypervisor

VM

VM

Kernel

App App

Guest Hypervisor

Nested VM

Kernel

App App

Nested VM

Kernel

App App

ENGINEERS AND DEVICESWORKING TOGETHER

ARM Virtualization Extensions

Hypervisor

VM

OS Kernel

App AppApp

EL1

EL0

EL2

ENGINEERS AND DEVICESWORKING TOGETHER

ARM Nested Virtualization

Host Hypervisor

VM

EL1

EL0

EL2

Guest Hypervisor

Nested VM

OS Kernel

App AppApp

ENGINEERS AND DEVICESWORKING TOGETHER

ARM Nested Virtualization

Host Hypervisor

VM

EL1

EL0

EL2

Guest Hypervisor

Nested VM

OS Kernel

App AppApp

Trap and Emulate

ENGINEERS AND DEVICESWORKING TOGETHER

Host Hypervisor

VM

EL1

EL0

EL2

Guest Hypervisor

Nested VM

OS Kernel

App AppApp

??? and Emulate

ARMv8.0

ENGINEERS AND DEVICESWORKING TOGETHER

ARMv8.3

Host Hypervisor

VM

EL1

EL0

EL2

Guest Hypervisor

Nested VM

OS Kernel

App AppApp

Trap and Emulate

ENGINEERS AND DEVICESWORKING TOGETHER

Performance Evaluation

• Problem: No ARMv8.3 hardware available.

• Solution: Use ARMv8.0 hardware with the software modification

ENGINEERS AND DEVICESWORKING TOGETHER

Emulating v8.3 on v8.0

Host Hypervisor

VM

EL1

EL0

EL2

Guest Hypervisor

Nested VM

OS Kernel

App AppApp

ARMv8.0 Hardware

SW Modification

ENGINEERS AND DEVICESWORKING TOGETHER

Hypercall MicroBenchmark

Hypervisor

VM

OS Kernel

App AppApp

EL1

EL0

EL2Hypercall Return

Host Hypervisor

VM

EL1

EL0

EL2

Guest Hypervisor

Nested VM

OS Kernel

App AppApp

Hypercall Return

ENGINEERS AND DEVICESWORKING TOGETHER

Hypercall MicroBenchmark

ARMv8.3

VM Nested VM

Cycle counts 2,729 422,720

Ratio to VM 1 155x

ENGINEERS AND DEVICESWORKING TOGETHER

Nested VM Hypercall

Host Hypervisor

VM

EL1

EL0

EL2

Guest Hypervisor

Nested VM

OS Kernel

App AppApp

ReturnHypercall

…

ENGINEERS AND DEVICESWORKING TOGETHER

Hypercall MicroBenchmark

ARMv8.3

VM Nested VM

Cycle counts 2,729 422,720Ratio to VM 1 155xTrap counts 1 126

ENGINEERS AND DEVICESWORKING TOGETHER

All Traps are Necessary?

Host Hypervisor

VM

EL1

EL0

EL2

Guest Hypervisor

Nested VM

OS Kernel

App AppApp

ReturnHypercall

…

ENGINEERS AND DEVICESWORKING TOGETHER

NEVE: NEsted Virtualization Extensions for ARM

• Improves performance of nested virtualization

• Key Mechanisms

1. Redirects register accesses to memory

2. Redirects register accesses to other registers

ENGINEERS AND DEVICESWORKING TOGETHER

VNCR_EL2

• Virtual Nested Control Register

• Enable bit

• BADDR (Base Address)

ENGINEERS AND DEVICESWORKING TOGETHER

#1 Redirection to MemoryRegister

msr x0, TTBR0_EL1

VNCR_EL2.Enable == 0 Trap!

Memory

ENGINEERS AND DEVICESWORKING TOGETHER

#1 Redirection to MemoryRegister

msr x0, TTBR0_EL1

VNCR_EL2.Enable == 1

TTBR0_EL1

MemoryVNCR_EL2.BADDR

ENGINEERS AND DEVICESWORKING TOGETHER

Accessing EL2 State

Host Hypervisor

VM

EL1

EL0

EL2

Guest Hypervisor

Nested VM

OS Kernel

App AppApp

ReturnHypercall

…

ENGINEERS AND DEVICESWORKING TOGETHER

#2 Redirection to RegisterTTBR0_EL2

msr x0, TTBR0_EL2

TTBR0_EL1

VNCR_EL2.Enable == 0 Trap!

ENGINEERS AND DEVICESWORKING TOGETHER

#2 Redirection to Register

msr x0, TTBR0_EL2

TTBR0_EL1

VNCR_EL2.Enable == 1

TTBR0_EL2

ENGINEERS AND DEVICESWORKING TOGETHER

Hypercall MicroBenchmark

ARMv8.3 NEVE

VM Nested VM Nested VM

Cycle counts 2,729 422,720 92,385

Ratio to VM 155x 34x

Trap counts 1 126 15

ENGINEERS AND DEVICESWORKING TOGETHER

Application Workloads

Application Description Application Description

Kernbench Kernel compile Netperf TCP_RR Network performance

Hackbench Scheduler stress Netperf TCP STREAM Network performance

SPECjvm2008 Java Runtime Netperf TCP MAERTS Network performance

MySQL Database management Apache Web server stress

Memcached Key-Value store Nginx Web server stress

ENGINEERS AND DEVICESWORKING TOGETHER

Experimental Setup• Hardware

• APM X-Gene (ARMv8.0)

• 8-way SMP

• 64 GB RAM

• 10 Gb Ethernet

• Software

• KVM on KVM

• v4.10

• Native/VM/Nested VM Setup

• 4-way SMP

• 12 GB RAM

• Virt I/O(VM/nested VM)

ENGINEERS AND DEVICESWORKING TOGETHER

Application Benchmarks

01020304050 ARMv8.3 VM

ARMv8.3 NestedNEVE Nested

012345

Kernbench

Hackbench

SPECjvm2008TCP RR

TCP STREAM

TCP MAERTSApache

Nginx

MemcachedMySQL

Normalized overhead (lower is better)

ENGINEERS AND DEVICESWORKING TOGETHER

Application Benchmarks

01020304050 ARMv8.3 VM

ARMv8.3 NestedNEVE Nestedx86 Nested VM

012345

Kernbench

Hackbench

SPECjvm2008TCP RR

TCP STREAM

TCP MAERTSApache

Nginx

MemcachedMySQL

Normalized overhead (lower is better)

ENGINEERS AND DEVICESWORKING TOGETHER

Conclusion

• Nested virtualization performance on ARMv8.3 incurs high overhead

• Due to the exit multiplication problem

• NEVE enhances performance significantly by reducing number of traps

• NEVE is included in ARMv8.4