The Denali Project

28
The Denali Project Andrew Whitaker, Marianne Shaw, Rick Cox, Steven D. Gribble http://denali.cs.washington.edu {andrew,rick,mar,gribble}@cs.washington.

description

The Denali Project. Andrew Whitaker , Marianne Shaw, Rick Cox, Steven D. Gribble. http://denali.cs.washington.edu. {andrew,rick,mar,gribble}@cs.washington.edu. A Brief History of VMMs. Introduced by IBM in the late 1960’s: CP/67 (1966) VM/370 (1972) z/VM (2004) - PowerPoint PPT Presentation

Transcript of The Denali Project

Page 1: The Denali Project

The Denali Project

Andrew Whitaker, Marianne Shaw, Rick Cox, Steven D. Gribble

http://denali.cs.washington.edu{andrew,rick,mar,gribble}@cs.washington.edu

Page 2: The Denali Project

A Brief History of VMMs

• Introduced by IBM in the late 1960’s:– CP/67 (1966)

– VM/370 (1972)

– z/VM (2004)

• Found wide use as a time-sharing system system– Ideological competitor to Multics, UNIX

• Today: VMMs have enjoyed a popularity resurgence – Industry: VMWare, Virtual PC (Microsoft), IBM

– Academia: Disco, Denali, ReVirt, Xen

Page 3: The Denali Project

Virtualizable Architectures

• Previous systems focused on strict virtualizability [Goldberg, 74]– Goal: minimize OS

changes

– VMM strictly emulates the phys. hardware

virtual arch

phys arch

hardware

…VM

VMM=

VM VM VM

Page 4: The Denali Project

The Denali Philosophy

• Key insight: software is not hardware– We can modify the VMM to achieve other goals

• Two ways to exploit change:– Modifying the virtual architecture [OSDI 2002]

– Changing the VMM implementation [NSDI 2004]

Page 5: The Denali Project

Outline

• Virtual Machine Monitor Background

• Denali and Architectural Changes– Motivation and approach– Architecture changes– Results

• Denali and Implementation Changes

Page 6: The Denali Project

Motivation

• Growing importance of untrusted code:– Internet services: ASP model, service authors

upload code into hosting center– PlanetLab: “slice” distributed platforms amongst

competing experiments, apps– Your desktop: Java{script}, untrusted binaries,

email attachments

• Needed: strong isolation platform– must be secure, efficient, and extensible

Page 7: The Denali Project

Why we need a new system

• Previous platforms are are inadequate:– Java sandboxes, ActiveX controls, system call

monitoring, virtual private servers, enhanced reference monitors

• Why?– require complex security policy– complexity, wide OS API– performance and/or scaling bottlenecks

Page 8: The Denali Project

Basic approach

• Focus on Internet services– Little sharing required

• Straw man: one service per physical machine– not cost-effective for many

services

• Instead, use virtual machines to emulate separate hardware

hardware

VMM

Web

server

Game

server

Overlay

router

Page 9: The Denali Project

VMMs provide strong isolation

• namespace isolation– no sharing possible, except through network

– eliminates security policy problems

• type-I VMMs run on bare hardware– no layer-below vulnerabilities, unfettered resource

management ability

• no high-level abstractions– eliminates complexity, reduces width of API

Page 10: The Denali Project

But, VMMs do too much!

• VMMs support isolation and full HW emulation– though some VMMs are

moving away from this…

• Isolation kernels separate the goals of isolation and emulation– make a clean break from

the underlying phys arch.

Goal VMM Isolation kernel

strong isolation

full hardware emulation

simplicity, scale, performance

Page 11: The Denali Project

Outline

• Virtual Machine Monitor Background

• Denali and Architectural Changes– Motivation and approach– Architecture changes– Results

• Denali and Implementation Changes

Page 12: The Denali Project

The Denali Virtual Architecture

• Retain x86 instruction set for performance

• Change other aspects to gain simplicity, scale, performance– gives up legacy OS

support

virtual arch

phys arch

hardware

…VM

Denali

VM VM VM

Page 13: The Denali Project

Arch. Changes for Simplicity

• Real hardware is unnecessarily complex

• Denali “scrubs” the hardware architecture:– Omit rarely used features

– Revise other features (interrupts, MMU)

– Streamlined hardware devices• Hardware Ethernet drivers make 12 PIOs per send

• Denali’s Ethernet requires single PIO

Page 14: The Denali Project

Source Code Complexity

0

10000

20000

30000

40000

50000

60000

70000

Linux Denali

Source lines

core kernel

arch. dependent

drivers

0

10000

20000

30000

40000

50000

Linux Denali

Source lines

storage

networking

virtual memory

other

Denali’s core kernel is roughly an order-of-magnitude smaller than bare-bones Linux 2.4.16

a) total lines b) core kernel breakdown

Page 15: The Denali Project

Arch. Change for Scale• Denali exposes an idle-with-timeout instruction

• Avoids a 50% throughput reduction for many VMs

0

100

200

300

400

500

0 50 100 150 200 250 300# virtual machines

Served load (req/sec)

idle-with-timeout

idle-without-timeout

Page 16: The Denali Project

Arch. Change for Performance

• Denali uses asynchronous, batched interrupt delivery

– avoiding context switches results in a 30% performance improvement

VM 1 VM 2 VM 3 VM 4scheduled VM

virtual interrupts 4 4

physical interrupts 4 4 4 4

time

Page 17: The Denali Project

Outline

• Virtual Machine Monitor Background

• Denali and Architectural Changes– Motivation and approach– Architecture changes– Results

• Denali and Implementation Changes

Page 18: The Denali Project

Experimental Setup

• Hardware– Intel Pentium 4 CPU; 1 GB RAM– 3 IDE disks– Intel PRO/1000 gigabit Ethernet NIC

• Software: apps run on Ilwaco guest OS– includes threads package, TCP/IP stack, libc– doesn’t use Denali MMU

• apps linked against Ilwaco like a libOS

Page 19: The Denali Project

Ethernet Virtualization Overhead

• Virtualization overhead is less than 20%

user/kernel crossing 3% , 2%

VM’s device driver

1% , 1%

TCP Stack

37% , 42%

read( )

packet copy VNIC demux Ethernet driver

12% , 13% 5% , 4% 43% , 39%

Ethernetpacketarrival

Breakdown of UDP receive costs (100 byte , 1400 byte packets)

total cost (cycles): 43477, 49002

Page 20: The Denali Project

Web Server Performance

• BSD: one kernel crossing per syscall

• Denali: one kernel crossing per packet

0

100

200

300

400

500

600

1 10 100 1000

document size (KB)

throughput (Mb/s)

BSD Mb/sec

Denali Mb/sec

0

100

200

300

400

500

600

1 10 100 1000document size (KB)

throughput (Mb/s)

BSD Mb/sec

BSD-syscall Mb/sec

Denali Mb/sec

Page 21: The Denali Project

Swapping in Denali

• Mechanism: demand-loading of “virtual” physical memory from statically allocated swap

• Factors that influence performance:– # disks– memory size– popularity distribution– block size– data locality– striping policy

Page 22: The Denali Project

Scale in Denali: 10,000 Web Servers

0

1000

2000

3000

4000

5000

6000

0 200 400 600 800 1000 1200 1400

# VMs

req/sec

0

10

20

30

40

0 2000 4000 6000 8000 10000# VMs

req/sec

In memory performance Disk-bound performance

disk paging bandwidth

• Denali utilizes most of the available hardware resources

memoryexceeded

Page 23: The Denali Project

Outline

• Virtual Machine Monitor Background

• Denali and Architectural Changes– Motivation and approach– Architecture changes– Results

• Denali and Implementation Changes

Page 24: The Denali Project

The VMM as a Service Platform

• Adding services to a VMM:– Migration (Intel Research Pittsburgh, Stanford)

– Intrusion detection (Stanford)

– System replay (Michigan)

– Fault Tolerance (Cornell)

• Advantages:– Virtual devices are relatively simple

– VMM has a “whole-system” perspective

Page 25: The Denali Project

Denali: An Extensible VMM

• Problem: existing VMMs are hard to extend– New services require significant re-implementation or reverse-

engineering

Denali approach:– Factor out VMM design decisions (disk, swap, etc.)

• “Parent” virtual machines can implement functionality on behalf of their “children”

– Encapsulate these decisions with clean interfaces• Supply default implementations for commonly used components

Page 26: The Denali Project

A Denali Interface

// the virtual Disk device callback functionsstruct Disk {

int (*diskWrite)(char *buffer, int offset, int num_sectors);

int (*diskRead)(char *buffer, int offset, int num_sectors);

int (*getSizeInSectors)(void);

int (*getSectorSize)(void);

} Disk;

Page 27: The Denali Project

Migrating an Apache Web Server

0

50

100

150

200

250

300

0 5 10 15 20 25

Time (seconds)

Req/secMachine A

Machine B

Implemented in only 289 lines of C source code

Migration

Page 28: The Denali Project

Conclusions

• Denali exploits the degree-of-freedom offered by software to enhance VMMs– Architecture changes for scale,

performance, simplicity

– Implementation changes for providing virtual machine services