Analysis of Virtualization Technologies for High Performance Computing Environments

25
Analysis of Virtualization Technologies for High Performance Computing Environments Andrew J. Younge, Robert Henschel, James T. Brown, Gregor von Laszewski, Judy Qiu, Geoffrey C. Fox Indiana University Bloomington https://portal.futuregrid.org 1

description

Analysis of Virtualization Technologies for High Performance Computing Environments. Andrew J. Younge , Robert Henschel , James T. Brown, Gregor von Laszewski , Judy Qiu , Geoffrey C. Fox Indiana University Bloomington. Outline. Introduction Related work Feature comparison - PowerPoint PPT Presentation

Transcript of Analysis of Virtualization Technologies for High Performance Computing Environments

Page 1: Analysis of Virtualization Technologies for High Performance Computing Environments

Analysis of Virtualization Technologies for High Performance

Computing Environments

Andrew J. Younge, Robert Henschel, James T. Brown, Gregor von Laszewski, Judy Qiu,

Geoffrey C. Fox

Indiana University Bloomington

https://portal.futuregrid.org 1

Page 2: Analysis of Virtualization Technologies for High Performance Computing Environments

Outline

• Introduction• Related work• Feature comparison• FutureGrid experimental setup• Performance comparison

– HPCC– SPEC

• Discussion2https://portal.futuregrid.org

Page 3: Analysis of Virtualization Technologies for High Performance Computing Environments

Introduction• What is Virtualization?

– A method of partitioning a physical computer into multiple “virtual” computers, each acting independently as if they were running directly on hardware.

• What is a Hypervisor?– A technique used to run multiple operating systems

simultaneously on a single resource.– Also called a Virtual Machine Monitor (VMM).

• What is a Virtual Machine?– A software implementation of a machine that executes as if it

was running on a physical resource directly.• Why does it matter?

– Cloud Computing!!!

3https://portal.futuregrid.org

Page 4: Analysis of Virtualization Technologies for High Performance Computing Environments

Motivation

• Most “Cloud” deployments rely on virtualization.– Amazon EC2, GoGrid, Azure, Rackspace Cloud …– Nimbus, Eucalyptus, OpenNebula, OpenStack …

• Number of Virtualization tools or Hypervisors available today.– Xen, KVM, VMWare, Virtualbox, Hyper-V …

• Need to compare these hypervisors for use within the scientific computing community.

4https://portal.futuregrid.org

Page 5: Analysis of Virtualization Technologies for High Performance Computing Environments

Current Hypervisors

5https://portal.futuregrid.org

Page 6: Analysis of Virtualization Technologies for High Performance Computing Environments

Hypervisors

• Evaluate Xen, KVM, and VirtualBox hypervisors against native hardware– Common, well documented– Open source, open architecture– Relatively mature & stable

• Cannot benchmark VMWare hypervisors due to proprietary licensing issues.

6https://portal.futuregrid.org

Page 7: Analysis of Virtualization Technologies for High Performance Computing Environments

Related Research• Some work has already been done to evaluate

performance…• Karger, P. & Safford, D. I/O for virtual machine monitors: Security and performance issues.

Security & Privacy, IEEE, IEEE, 2008, 6, 16-23 • Koh, Y.; Knauerhase, R.; Brett, P.; Bowman, M.; Wen, Z. & Pu, C. An analysis of performance

interference effects in virtual environments. Performance Analysis of Systems & Software, 2007. ISPASS 2007. IEEE International Symposium on, 2007, 200209.

• K. Jackson, L. Ramakrishnan, K. Muriki, S. Canon, S. Cholia, J. Shalf, H. Wasserman, and N. Wright, “Performance Analysis of High Performance Computing Applications on the Amazon Web Services Cloud,” in 2nd IEEE International Conference on Cloud Computing Technology and Science. IEEE, 2010, pp. 159–168.

• P. Barham, B. Dragovic, K. Fraser, S. Hand, T. L. Harris, A. Ho, R. Neugebauer, I. Pratt, and A. Warfield, “Xen and the art of virtualization,” in Proceedings of the 19th ACM Symposium on Operating Systems Principles, New York, U. S. A., Oct. 2003, pp. 164–177.

• Adams, K. & Agesen, O. A comparison of software and hardware techniques for x86 virtualization. Proceedings of the 12th international conference on Architectural support for programming languages and operating systems, 2006, 2-13.

7https://portal.futuregrid.org

Page 8: Analysis of Virtualization Technologies for High Performance Computing Environments

FeaturesXen KVM VirtualBox VMWare

Paravirtualization Yes No No No

Full Virtualization Yes Yes Yes Yes

Host CPU X86, X86_64, IA64 X86, X86_64, IA64, PPC

X86, X86_64 X86, X86_64

Guest CPU X86, X86_64, IA64 X86, X86_64, IA64, PPC

X86, X86_64 X86, X86_64

Host OS Linux, Unix Linux Windows, Linux, Unix Proprietary Unix

Guest OS Linux, Windows, Unix Linux, Windows, Unix Linux, Windows, Unix Linux, Windows, Unix

VT-x / AMD-v Opt Req Opt Opt

Supported Cores 128 16* 32 8

Supported Memory 4TB 4TB 16GB 64GB

3D Acceleration Xen-GL VMGL Open-GL Open-GL, DirectX

Licensing GPL GPL GPL/Proprietary Proprietary

8https://portal.futuregrid.org

Page 9: Analysis of Virtualization Technologies for High Performance Computing Environments

Usability

• KVM and VirtualBox trivial to install & deploy.– Xen requires special kernel, leading to more

complications.– VMWare ESX runs as a standalone hypervisor.

• All are supported under Libvirt API.– Used by many IaaS frameworks.

• Xen & Virtualbox have nice CLI, VMWare has an advanced web based GUI.

9https://portal.futuregrid.org

Page 10: Analysis of Virtualization Technologies for High Performance Computing Environments

Performance Analysis

• In order to assess various performance metrics, benchmarks are needed.– Provide a fair, apples-to-apples comparison

between each hypervisor.– Comparisons can be made across other

benchmark submissions on different machines.– Reproducible and verifiable results.– Open standards, no special optimizations or tricks

available (hopefully).

10https://portal.futuregrid.org

Page 11: Analysis of Virtualization Technologies for High Performance Computing Environments

• HW Resources at: Indiana University, San Diego Supercomputer Center, University of Chicago / Argonne National Lab, Texas Applied Computing Center, University of Florida, & Purdue• Software Partners: University of Southern California, University of Tennessee Knoxville, University of Virginia, Technische Universtität Dresden

FutureGridFutureGrid is an experimental grid and cloud test-bed with a wide variety of computing services available to users.

https://portal.futuregrid.org

Page 12: Analysis of Virtualization Technologies for High Performance Computing Environments

Testing Environment• All tests conducted on the

IU India cluster as part of FutureGrid.

• Identical nodes used for each hypervisor, as well as a bare-metal (native) machine for the control group.

• Each host OS runs RedHat Enterprise Linux 5.5.

• India: 256 1U compute nodes– 2 Intel Xeon 5570 Quad core

CPUs at 2.93Ghz– 24GB DDR2 Ram – 500GB 10k HDDs– InfiniBand DDR 20Gbs

12

Page 13: Analysis of Virtualization Technologies for High Performance Computing Environments

Testing Environment (cont)• Guest Operating System:

– RedHat Enterprise Linux 5.5 x86_64– Kernel 2.6.18-194.8.1.el5– Minimal server installation (Base, Core)– Default settings and services– Full virtualization using Intel VT-x– Each VM gets 16GB of memory– Virtualization Add- ons:

• Xen: version 3.1.0 with 2.6.18-194.8.1.el5xen kernel• KVM: version 83• VirtualBox: version 3.2.10_66523_rhel5• Bare-Metal OS = Host OS

13https://portal.futuregrid.org

Page 14: Analysis of Virtualization Technologies for High Performance Computing Environments

Benchmark Setup

HPCC• Industry standard HPC

benchmark suite from University of Tennessee.

• Sponsored by NSF, DOE, DARPA.

• Includes Linpack, FFT benchmarks, and more.

• Targeted for CPU and Memory analysis.

SPEC OMP• From the Standard

Performance Evaluation Corporation.

• Suite of applications aggrigated together

• Focuses on well-rounded OpenMP benchmarks.

• Full system performance analysis for OMP.

14https://portal.futuregrid.org

Page 15: Analysis of Virtualization Technologies for High Performance Computing Environments

15https://portal.futuregrid.org

Page 16: Analysis of Virtualization Technologies for High Performance Computing Environments

16https://portal.futuregrid.org

Page 17: Analysis of Virtualization Technologies for High Performance Computing Environments

17https://portal.futuregrid.org

Page 18: Analysis of Virtualization Technologies for High Performance Computing Environments

18https://portal.futuregrid.org

Page 19: Analysis of Virtualization Technologies for High Performance Computing Environments

19https://portal.futuregrid.org

Page 20: Analysis of Virtualization Technologies for High Performance Computing Environments

Performance Roundup• Hypervisor performance in Linpack reaches roughly 70% of native

performance during our tests, with Xen showing a high degree of variation.

• FFT benchmarks seem to be less influenced by hypervisors, with KVM and VirtualBox performing at native speeds yet Xen incurs a 50% performance hit with MPIFFT.

• With PingPong latency KVM and VirtualBox perform close to native speeds while Xen has large latencies under maximum conditions.

• Bandwidth tests vary largely, with the VirtualBox hypervisor performing the best, however having large performance variations.

• SPEC benchmarks show KVM at near-native speed, with Xen and VirtualBox close behind.

20https://portal.futuregrid.org

Page 21: Analysis of Virtualization Technologies for High Performance Computing Environments

Conclusion• Big Question: Is Cloud Computing viable for

scientific High Performance Computing?– Our answer is “Yes” (for the most part).

• Features: All hypervisors are similar.• Performance: KVM is fastest across most

benchmarks, VirtualBox close.• Overall, we have found KVM to be the best

hypervisor choice for HPC.– Currently moving to KVM for all of FutureGrid.

21https://portal.futuregrid.org

Page 22: Analysis of Virtualization Technologies for High Performance Computing Environments

Thank you!

22https://portal.futuregrid.org

Page 23: Analysis of Virtualization Technologies for High Performance Computing Environments

Virtual Machines

http://futuregrid.org 23

Page 24: Analysis of Virtualization Technologies for High Performance Computing Environments

Feature Roundup

• All hypervisors evaluated have acceptable level of features for x86 virtualization.

• Xen provides best expandability, supporting up to 128 CPUs and 4TB of RAM.– Can remove CPU limit for KVM.– VirtualBox needs to add support for >16GB RAM.

• All have API plugins to allow for simplified IaaS usage.

24https://portal.futuregrid.org

Need to learn more from a performance comparison….

Page 25: Analysis of Virtualization Technologies for High Performance Computing Environments

Benchmarks

• SPEC Benchmarks– specCPU 2006 – CPU bound benchmark– specMPI 2007 – MPI: cpu, memory, network– specOMP 2001 – Open MP: cpu, memory, IPC– specVIRT 2010 – CPU, memory, disk I/O, network

• HPCC Benchmarks– HPL (Linpack), DGEMM, STREAM, PTRANS,

RandomAccess, FFT, etc

http://futuregrid.org 25