Efficient VM Introspection in KVM and

Performance Comparison with Xen

Kenichi KouraiKousuke Nakamura

Kyushu Institute of Technology

Intrusion Detection System (IDS)IDSes detect attacks against serversMonitor the systems and networks of serversAlert to administrators

Recently, attackers attempt to disable IDSesBefore they are detectedThis is easy because IDSes are running in

servers IDS

server

intrudedetect

IDS OffloadingOffloading IDSes using virtual machines

(VMs)Run a server in a VMExecute IDSes outside the VM

Prevent IDSes from being compromisedCan be provided as a cloud service

Cloud providers can protect users' VMs

VM

IDS offloading

IDS

In-VM monitoring

IDSmonitorVM

VM Introspection (VMI)A technique for monitoring VMs from the

outsideMemory introspection

Obtain raw memory contents and extract OS data

Disk introspectionObtain raw disk data and interpret a filesystem

Network introspectionObtain packets only from/to VMs

???IDS

VM

disk

??? memory

packets network

Performance of VMIPerformance has not been reported in

detailNo performance comparison

E.g., VMwatcher [Jiang+ CCS'07]

Implemented in Xen, QEMU, VMware, and UMLReported only for UML

E.g., EXTERIOR [Fu+ VEE'13]

Implemented in KVM and QEMUNo difference due to using memory dump

Performance data is importantFor user's selection of virtualization software

The Purpose of This WorkPerformance comparison among

virtualization software in terms of VMITarget: Xen and KVM

Widely used open source virtualization software

System architecture is different

hypervisor

VM VM

OS

VM

process

Xen KVM

Implementation for KVMNo efficient implementation of VMI for KVMSeveral studies have been done for KVM

The implementation details are unclearLibVMI [Payne+ '11] supports VMI for both Xen

and KVMThe performance of memory introspection is

too low in KVMOptimized for Xen

KVMonitorWe have developed an efficient VMI tool

for KVMExecute an IDS as a process of the host OSProvide functions for introspecting memory,

disks, and NICs in QEMU

host OS

diskQEMU

memory

NIC

KVM module

VM

IDS

KVMonitor

offload

monitor

Memory Introspection (1/2)Difficult to efficiently introspect QEMU's

memoryLibVMI obtains memory contents from QEMU

KVMonitor shares VM's physical memory with QEMU via a memory fileAccess As a memory-mapped fileEnable direct memory introspection

IDS

VM's physical memory

VM

VM's physical memory

memory file

VM's physical memory

KVMonitor QEMU

Memory Introspection (2/2)IDSes usually access OS data using virtual

addressesKVMonitor translates virtual addresses into

physical addressesLook up the page table for address

translationIntrospect the CR3 register using QMP

IDS VM

VM's physical memory

memory file

VM's physical memory

page table

KVMonitor QEMUCR3

Disk/Network IntrospectionKVMonitor introspects VM's disks via the

network block device (NBD)Interpret the qcow2 format in the NBD

serverInterpret the filesystem in the host OS

KVMonitor captures packets from a tap device

host OS

VM

QEMU

IDS

KVMonitor

tap

diskimage

fileNBD

server

NBD

network

Transcall with KVMonitorWe have ported Transcall [Iida+ '11] for Xen

to KVMEnable offloading legacy IDSes without any

modificationsConsist of a system call emulator and a

shadow filesystemIncluding the proc filesystemAnalyze OS data by memory introspection

QEMU

IDS

KVMonitor

VMTranscall

analyze

ExperimentsWe examined that KVMonitor achievedEfficient memory introspectionNo impact on memory performance of a VMEffective IDS offloading

CPU: Intel Xeon E5630 (12 MB L3 cache)Memory: 6 GB DDR3 PC3-8500HDD: 250 GB SATANIC: gigabit EthernetHypervisor: KVM 1.1.2Host OS: Linux 3.2.0

CPU: 1Memory: 512 MBDisk: 20 GB (ext3)Guest OS: Linux 2.6.27

PC VM

KVMonitor vs. LibVMIWe measured the performance of memory

introspectionCopy VM's physical memory

by 4KB

KVMonitor was32x faster than LibVMI

Series10

2

4

6

8

10

12

9.6

0.3

KVMonitor LibVMI

read

(G

B/s

) fast

memoryfile

Why is LibVMI so slow?LibVMI has to issue a QMP command for

each memory accessMemory contents are transferred from QEMU

to LibVMI

QEMU

LibVMI

QMPIDS

LibVMI

KVMonitor

IDS

VM's memory

VM's memory

VM's memory

QEMUKVMonit

or

VM VM

In-VM Memory PerformanceDoesn't using a memory file affect

memory performance of a VM?

Using a memory file wasas efficient as malloc

read write0123456789

108.6

6.6

8.5

6.3

memory file malloc

thro

ug

hp

ut

(GB

/s)

memoryfile VM's

memory

QEMU

VM

QEMU

VM

VM's memory

memory file malloc

KVMonitor vs. In-VM AccessKVMonitor was faster than

in-VM memory accessDue to virtualization

overhead

Series10

2

4

6

8

10 9.68.6

KVMonitor In-VM

read

(G

B/s

)memory

file

IDS

VM's memory

VM's memory

QEMUKVMonit

or

VM

fast

Offloading Legacy IDSes (1/3)TripwireCheck filesystem integrity in disks

We added, deleted, and modified filesOffloaded Tripwire detected changed files

Rule Name ... Added Removed ModifiedMonitor Filesystems 1 1 1Total Objects scanned: 67082Total violations found: 3

Tripwire

DB disk

VM

Offloading Legacy IDSes (2/3)SnortInspect network packets

We performed portscans from another hostOffloaded Snort detected portscans[**] [1:1421:11] SNMP AgentX/tcp request [**][Classification: Attempted Information Leak] ...01/28-10:47:13.406931 192.168.0.68:47962 -> 192.168.0.81:705

VMSnort

rule sets

packetsportscan

Offloading Legacy IDSes (3/3)ChkrootkitDetect rootkits using ps, netstat, and file

inspectionWe tampered with ps and netstat in a VMOffloaded chkrootkit detected tampered

commandsROOTDOR is ’/’Checking ’ps’...INFECTEDChecking ’netstat’...INFECTED :

chkrootkitdisk

VM

ps netstatps netstat

...

execute

Cross-view Diff (1/2)A technique for detecting hidden malwareCompare the results of VMI and in-VM

monitoringThe difference means the existence of

hidden malware

VM

A B D ...A B C D ...

cross-view diff

engine

C is hidden

monitorIDS IDS

Cross-view Diff (2/2)We tampered with ps in a VMA hidden process was detected as malicious

We tampered with netstat in a VMA hidden port was detected as a backdoor

PID TTY TIME CMD 1 ? 00:00:00 init 2 ? 00:00:00 kthreadd :

PID TTY TIME CMD 2 ? 00:00:00 kthreadd :

Proto ... Local Address ...tcp 0.0.0.0:22 :

Proto ... Local Address ...tcp 0.0.0.0:5900tcp 0.0.0.0:22 :

results from offloaded commands results from in-VM commands

ps

netstat

KVMonitor vs. XenWe compared the performance of VMI

between KVM and XenUsing a VMI tool for Xen

Memory: standard libraryDisk: loopback mountNetwork: tap device

hypervisor

Dom0 (VM)

Hypervisor: Xen 4.1.3Dom0 OS: Linux 3.2.0VM: fully virtualized

IDSlibxenct

rl

tap

VM

disk imagefile

Memory IntrospectionWe measured read throughputCopy VM's physical memory

by 4KB

KVMonitor was48x faster than Xen

VMI0

2

4

6

8

10

12

9.6

0.2

KVM Xen

read

(G

B/s

) fast

Why is Xen so slow?Xen has to map each memory pageIt cannot map all the pages in advanceIt takes time proportional to the number of

pagesKVMonitor can read a pre-mapped file

KVMonitor

IDS

Xen

VM's memory

IDS

map

VM

memoryfile

KVMonitor

libxenctrl

Kernel Integrity CheckingWe measured the execution time of the

kernel integrity checkerRead the code areaTranslate virtual to

physical addresses

KVMonitor was118x faster than Xen

0

50

100

150

200

250

1.9

224

KVM Xen

tim

e (

ms)

fast

Why is the speedup so larger?The speedup in the real IDS was much

larger48x (simple benchmark)118x (kernel checker)

Due to address translationIn Xen, the access cost of the page table is

highOnly 8 bytes are read after memory mapping

simple benchmark

IDS map &read

VM

real kernel checker

IDS map &read

VM

libxenctrl libxenctrl

Disk IntrospectionWe measured the execution time of

TripwireFor two formats of disks

raw and qcow2

KVMonitor wasComparable to Xen

The difference betweenformats was largerRaw was faster than qcow2

raw qcow20

2

4

6

8

10

7.5

9.4

7.5

9.2

KVM Xen

tim

e (

min

)fast

Network IntrospectionWe measured the packet loss rate in SnortSend many packets as

fast as possible

KVMonitor wasmore lightweight

than XenDom0 suffered from

virtualization overhead

Series10

2

4

6

8

10

12

6.2

10.4

KVM Xen

packet

loss r

ate

(%

)fast

ChkrootkitWe measured the execution time of

chkrootkit

KVMonitor was1.6x faster than Xen

Efficient memoryintrospection

No virtualizationoverhead

2x slower than in-VMDue to system call traps Offloading in-VM

0

10

20

30

40

50

60

35

18

55

21

KVM Xen

tim

e (

sec)

fast

Related WorkVMI toolsLivewire [Garfinkel+ NDSS'03] for VMwareXenAccess [Payne+ ACSAC'07] for Xen

Shm-snapshot for LibVMI [Xu+ PDL'13]

Take a VM's memory snapshot in shared memory

It takes 1.4 seconds for 3 GBVolatility [Walters '07]

A memory forensics frameworkVMI for KVM is enabled by a Python adapter,

PyVMI from LibVMI

ConclusionKVMonitorAchieve efficient VM introspection (VMI) in

KVM32x faster than existing LibVMI

Performance comparison with Xen118x faster at maximum

Chkrootkit was 1.6x faster

Future workComparison with other virtualization

softwareIntegration with LibVMI