INFSO-RI-223782

An On-Demand Dynamic Virtualization Manager

Øyvind Valen-Sendstad

CERN – IT/GD, ETICS

Oeyvind.valen-sendstad@cern.ch

Virtual Node bootstrapper (VNB)

INFSO-RI-223782

Contents

• Brief introduction to ETICS• ETICS build infrastructure• Improving the build/test process• The Virtual Node Bootstrapper• Achievements• Status, Lessons learned and Further work• Summary

2

INFSO-RI-223782

What ETICS is

• It’s a software engineering management system

• It’s a build and test infrastructure

• It provides tools and resources to configure, manage and analyse build and test runs

• It provides a common interface to diverse projects to facilitate knowledge sharing and operations management

• It has an open repository of configuration metadata, packages, reports. The goal is to share information, but also to reliably store and preserve information

• It has a plugin-based architecture and APIs to allow integrating ETICS into existing processes and extending it with custom actions

• It’s multi-platform and independent from any specific build or test tool

3

INFSO-RI-223782

4

ArchitectureETICS is not ‘just’ a build system

INFSO-RI-223782

ETICS build infrastructure - static platforms

X 5SL5 / 64

X 5Mac OSX

X 5SL5 / 32

X 5SL5 / 64

X 5SLC4

X 5 Debian

X 5 Ubuntu 7

X 5SL5

New SL5 Job

Physical node

Virtual static nodes

Problem 1:The composition of the pool is fixed. If we have 20 SL5 nodes all busy and a new request for SL5 arrives, it is queued even if many nodes are available with other platforms

INFSO-RI-223782

ETICS build infrastructure - static platforms

X 5SL5 / 64

X 5Mac OSX

X 5 SL5 / 32

X 5RH7

X 5SLC4

X 5Debian

X 5Ubuntu 7

X 5SL5

New RH7 Job

Physical node

Virtual static nodes

Problem 2:If a platform is very rarely used, but we want to make it available, one or more nodes will do nothing most of the time

INFSO-RI-223782The proposed solution: dynamic virtual platforms

New job

AvailableX 5 X 5

Physical machine

Virtual machine

AvailableX 5

AvailableX 5

Available

VM repository

X

AvailableX 5

AvailableX 5

AvailableX 5

AvailableX 5

INFSO-RI-223782

The bootstrap process

8

1. User submits a build/test for a specific platform2. Request arrives to VNB3. VNB determines platform and hypervisor support4. Downloads a suitable VM from repository (or local cache)5. Deploys the VM6. Executes job7. Returns results8. Cleans up

INFSO-RI-223782

Deploying the bootstrapper

INFSO-RI-223782

Deploying the bootstrapper

INFSO-RI-223782

INFSO-RI-223782

INFSO-RI-223782

Deploying the bootstrapper

INFSO-RI-223782

Deploying the virtual machine

INFSO-RI-223782

Deploying the virtual machine

INFSO-RI-223782

Deploying multiple virtual machines on the same node

INFSO-RI-223782

Deploying multiple virtual machines on the same node

INFSO-RI-223782

Deploying multiple virtual machines on the same node

INFSO-RI-223782

Achievements

• Perform platform deployment on demand• Increase availability• Increase capability• Provide reproducible environments• Reduce maintenance

• New releases• Upgrades• Security patches• Accommodate for specific software dependencies

• Offer privileged access (root access to the VM, not the host)• Enable post build analysis (VM snapshot can be saved with

results)• Virtual machine image customization (users can provide

their own)

19

INFSO-RI-223782

Status

Working• Bootstrap process, tested on VMware and KVM

Remaining• Testing on Xen• Support for NT-based operating systems• Caching VM images locally

20

INFSO-RI-223782

Lessons learned

Privileges• The bootstrapper runs in the user-space and need

privileges to use Xen and Vmware

Compatibility• The kernel in a virtual machine created in VMware is not

necessarily able to run in a Xen or KVM environment

21

INFSO-RI-223782

Further work

• Deploy virtual machine image into a folder and chroot in order to run natively on a worker node

• Update VNB on VM during startup

• Support for NT-based operating systems

INFSO-RI-223782

Open issues

Reproducibility• Different implementations of the virtual machine hardware

can cause different behaviour of a virtual machine

Firewall• Ephemeral port range can be blocked by strict firewalls

23

INFSO-RI-223782

Summary

• The VNB gives • Better service availability• Better resource utilization

• The VNB has• Generic design• Emulates when no hypervisors are present

• The VNB allows• Running multiple bootstrappers on a single host