EVGM08 1

Multi-Site Virtual Cluster: A User-Oriented, Distributed Deployment and Management

Mechanism for Grid Computing Environments

Takahiro Hirofuchi, Takeshi Yokoi, Tadashi Ebara,Yusuke Tanimura, Hirotaka Ogawa, Hidetomo Nakada,

Yoshio Tanaka, and Satoshi Sekiguchi

National Institute of Advanced IndustrialScience and Technology (AIST)

EVGM08 2

Background

Grid Computing Large-scale Distributed Heterogeneous

Grid Middleware Globus toolkit gLite

EVGM08 3

Software Deployment and Management

Problems Software complexity

Dependency Configuration

Resource diversity Hardware

• x86, x64, memory, storage, … Software

• CentOS 4, CentOS 5, Solaris, …• Library versions

Organizations• Administration policy

x Sites

Management CostExplosion

EVGM08 4

My Experience

VOMS Installation at 2007 spring Virtual organization management system Supports only SL3 (based on RHEL3 since 2003-12) It was nightmare

Debian Etch CentOS 5

VOMS depends on GT4.VOMS requires old GCC for C+

+.

GT4 requires new GCC for its patched Open

SSL

Me, too…?

EVGM08 5

Our Concept

Virtualization Isolate resources Capsulate environments Create new administrative domains

Site A

Site B

Site CCreate virtual machinesat each site

EVGM08 6

Our Concept

Virtualization Isolate resources Capsulate environments Create new administrative domains

Site A

Site B

Site C Group distributed VMsfor a Virtual Organization

EVGM08 7

Our Concept

Virtualization Isolate resources Capsulate environments Create new administrative domains

Site A

Site B

Site C Multi-Site Virtual Cluster

EVGM08 8

Multi-Site Virtual Cluster Integrate distributed VMs

Single cluster view. Allow single administrative domain

OS installation and full configuration Enable easy system deployment

Large-scale nodes

Application

Site A Site B

Application

Scientific applicationEmulation testbed

Deploymentand configuration

EVGM08 9

System Components (1)

Site A

Site B

Site C

Resource VirtualizationMechanism

EVGM08 10

System Components (2)

Site A

Site B

Site C

Web Service APIfor Virtualized Resource Control

EVGM08 11

System Components (3)

Site A

Site B

Site C

Easy management systemfor large-scale, distributed nodes

EVGM08 12

Resource Virtualization Mechanism

Design criteria Create completely-isolated VMs (Virtual Cluster)

Our virtual cluster system Physical nodes VMware Server and Xen VLAN LVM and iSCSI Free and Open Source

http://code.google.com/p/grivon

EVGM08 13

Allocating a Virtual Cluster (1)

ClusterManager VMM VMM VMM

Private Network

Public Network

EVGM08 14

Allocating a Virtual Cluster (2)

ClusterManager VMM VMM VMM

Create a new VLAN (eth0.1234)

Bridge to VLAN

EVGM08 15

Allocating a Virtual Cluster (3)

ClusterManager VMM VMM VMM

Create new storage volumes

LVM

Attach via iSCSI

EVGM08 16

Allocating a Virtual Cluster (4)

ClusterManager

VMM VMM VMM

Launch VMs on VMM

LVM

Attach via iSCSI

VM VM VM

EVGM08 17

Web Service API

REST API https://example.com/api/vc/ /api/vc

• List virtual clusters (GET), Create a new virtual cluster (POST) /api/vc/1234/

• Get the status of VC 1234 (GET) /api/vc/1234/vm

• List the VMs (GET), Add/Delete a new VM(POST) /api/vc/1234/vm/{0, 1, 2, 3, 4, 5}

• Get the status of a VM, Start/Stop a VM (POST) /api/vc/1234/vpn /api/vc/1234/vpn/{0, 1, 2}

EVGM08 18

Distributed Node Management

Exploit an existing cluster management system for the inside of a multi-site virtual cluster Designed for physical clusters Powerful node management

Node database, parallel command execution, automatic node installation

Integrate distributed VMs by Ethernet VPN Bridge internal networks of single-site virtual clusters Transparency for admins and users

Transparent package caching at each site Package-based software installer Quick installation and reconfiguration Flexible customizability

EVGM08 19

VM VM VM

VM VM VM

VM VM VMEthernet VPN among site-local VLANs

Package cacherepository

Managed by Rocks toolkit

Frontend Node Console Node DB PXE installation server

EVGM08 20

PrototypeImplementation

•Reservation portal Site resource monitoring Automatic reservation

• NPACI Rocks 4.2• VMware Server• VLAN, iSCSI• OpenVPN 2.0• Squid 3.0

EVGM08 21

Evaluation

Demo over the Pacific Software configuration Parallel command line tools Node status monitoring Condor job sumission

Evaluate ScalabilityWANLarge number of VMs

Reconfiguration time Network traffic

EVGM08 22

Experiment Setting

AMD Opteron 244, 3GB Mem, Gb Eth x2

16 nodes 134 nodes, reconfigured with 900MByte packages

AMD Opteron 246, 6GB Mem, Gb Eth x2

Node Config. DB

EVGM08 23

134 Node Reinstallation over WAN

Only 20 minutes for a134-nodes virtual cluster

900MB programs per node

EVGM08 24

30MB/s

800KB/s

10MB/s

Cache Enabled

Pre Cached

Cache DisabledWAN Traffic over VPNRTT 20ms

A cache server minimizesVPN traffic for reinstallation.

EVGM08 25

Conclusion

Multi-site virtual cluster Virtualization for Grid computing Isolate resources Give independent administrative domains Easy-to-use UI

Future work Amazon EC2 support Live Migration (demo sc08@Authtin)