4/7/2015 How to Set Up a Cluster of x86 Servers with Oracle VM 3

http://www.oracle.com/technetwork/articles/serversstorageadmin/o12033x86clusgeroraclevm1619861.html 1/14

Application DevelopmentFramework

Application Express

Big Data

Business Intelligence

Cloud Computing

Communications

Database Performance &Availability

Data Warehousing

Database

.NET

Dynamic Scripting Languages

Embedded

Digital Experience

Enterprise Architecture

Enterprise Management

Identity & Security

Java

Linux

Mobile

ServiceOriented Architecture

Solaris

SQL & PL/SQL

Systems All Articles

Virtualization

If you'd like to downloadsoftware, participate inforums, and get access toother technical howtogoodies in addition tocontent like this, becomean OTN member. No spam!

How to Set Up a Cluster of x86 Servers with Oracle VM 3by Kris Bakke and Jeffrey Kiely

Stepbystep instructions for using Oracle VM 3 to set up a cluster of highly available Sun x86 servers.

Published May 2012

Introduction About Oracle's Sun x86 SystemsConfiguration and Network TopologyDiscover Oracle VM Server for x86 Host Systems as a ResourceSet Up Network Resources for Oracle VM Configure the Management Network Bond the Management Network Ports Create an Additional Public Network Create the Virtual MAC AddressesCreate Oracle VM 3 Server Pool Assign a Storage RepositoryConclusionSee AlsoAbout the Authors

IntroductionThis article describes the process of setting up a cluster configuration of Oracle VM 3 using Oracle's Sun x86 systems. The primary design goal forthis cluster configuration is high availability (HA). Thus if one server or network port goes down, other components should automatically pick up itsload. Server redundancy is inherent in the virtualized architecture, and the design also utilizes multiple networks with bonded ports (see the "Bondthe Management Network Ports" section). This enables redundant network connections between the virtualized servers in the Oracle VM serverpool and the management server, thus giving Oracle VM Manager continuous access to the servers in the event of a single network failure.

The hardware and software infrastructure described in this article is considered a cloud infrastructure because itoffers the following capabilities:

Ondemand access—A shared pool of configurable computing resources is available as needed for runningapplication services.Elastic scalability—Add more servers, memory, disk, and so on and your applications automatically takeadvantage of it without having to know about it.Rapid provisioning—The management environment enables rapid provisioning and replication of systemresources and virtual machine (VM) guest environments.High availability—The infrastructure is configured to support continued operation in the event of a componentfailure, such as a server or network port.Portability—VMs are not tied to a hardware platform; they can be livemigrated and still access their storage.Our test configuration used blade servers running within Oracle's Sun Blade 6000 modular system to host the Oracle VM server pool. The setupprocess for Oracle VM and the network would be essentially the same if you were to use rackmount servers rather than blade servers. Theprocedure described in this article also works for any OS supported by Oracle VM.

Oracle Technology Network Articles Server and Storage Administration

Products Solutions Downloads Store Support Training Partners About OTN

Sign In/Register Help Country Communities I am a... I want to... Search

4/7/2015 How to Set Up a Cluster of x86 Servers with Oracle VM 3

http://www.oracle.com/technetwork/articles/serversstorageadmin/o12033x86clusgeroraclevm1619861.html 2/14

We start our setup process from the point where Oracle VM has already been installed on all of the Sun x86 systems in the configuration. The firststep is, therefore, to let Oracle VM Manager discover the baremetal systems available to participate in the Oracle VM server pool. We thenproceed to configure the network resources to create a cluster of Oracle VM virtual machines running in the blade server environment. The laststep of the process defined in this article is to create the Oracle VM server pool and assign a storage repository.

This article covers the assignment of a storage repository for the Oracle VM server pool, but it does not discuss the details of configuring thestorage environment or the storage networking. Also not covered in this article is the process of setting up the application environment that will runon the virtual machines. The virtual machine software stack, often called a golden image, is typically established on one VM and then replicated toother VMs for rapid deployment and easy maintenance. If all nodes on your cluster will run the same application environment, you can also installthe application environment as part of the golden image that you will replicate throughout the VMs. If your VMs will run different applications, thegolden image might consist of just a preconfigured version of the operating system of your choice.

Additional information about how to set up the storage environment, software infrastructure, and other aspects of an enterprise cloud deploymentcan be found in the "Oracle Optimized Solution for Enterprise Cloud Infrastructure—Implementation Guide (x86Linux)" white paper.

About Oracle's Sun x86 Systems

Oracle's Sun x86 systems are available in both rackmount and blade form factors and include an integrated Oracle software stack. Every modelcomes complete with virtualization, choice of OS, infrastructure provisioning, and Oracle's unique applicationtodisk system managementenvironment—all at no extra charge on servers with a support contract. In addition to simplifying cloud management, Sun x86 systems also providelarge memory capacities and scalable performance based on the Intel Xeon processor E52600 CPU. These features make Sun x86 systems anextremely good fit for an enterprise cloud infrastructure.

Configuration and Network TopologyThis article is based on the same server and storage configuration used in the Oracle Optimized Solution for Enterprise Cloud Infrastructure. Thistest configuration was deployed as part of the effort to identify best practices for the hardware and software architecture and implementation in theOracle Optimized Solution for Enterprise Cloud Infrastructure. Our test configuration consists of the following major components:

One Sun Blade 6000 modular system with three Sun x86 server blades for running Oracle VM guestsOne Sun x86 rackmount server running Oracle VM ManagerTwo of Oracle's Sun ZFS Storage Appliance controllers in a cluster configurationTwo topofrack network switches connected by an HA interconnectFigure 1 shows the network topology for connecting these components. Oracle VM Manager is deployed on a single Sun x86 rackmount serverwhile the Oracle VM guest virtual machines run on the three blade servers in the Sun Blade 6000 modular system.

The topofrack switches simplify the network cabling and enable redundant connections between all servers and storage systems in theconfiguration. The blade server modules in the Sun Blade 6000 modular system are surfaced to the network through two Sun Blade 6000 EthernetNetwork Express Modules (NEMs). These are referred to as NEM0 and NEM1 in the figure.

Quad SFP+ (QSFP+) network links are used for fiberbased physical network connections to the Sun Blade 6000 server modules. There are threeQSFP+ ports from each of NEM0 and NEM1, resulting in a total bandwidth of 120 Gb/sec for each NEM. This network configuration is designed todeliver both high performance and redundancy for high availability. The bandwidth available in this network configuration is also intended supportup to 10 server modules in the Sun Blade 6000 chassis. Since our test configuration had only three blade servers in the Sun Blade 6000 chassis,network throughput was not really challenged in our tests.

4/7/2015 How to Set Up a Cluster of x86 Servers with Oracle VM 3

http://www.oracle.com/technetwork/articles/serversstorageadmin/o12033x86clusgeroraclevm1619861.html 3/14

Figure 1. Network Topology for Connecting the Components

Discover Oracle VM Server for x86 Host Systems as a ResourceThe first step in setting up Oracle VM is to discover existing host systems that are on the same network as the server running Oracle VM Manager.Oracle VM Manager has a Discover Servers feature that makes it easy to bring the available servers into a default "unassigned" server pool whereyou can then manage them from within Oracle VM Manager. The servers must have already been assigned an IP address on the same network asOracle VM Manager. Otherwise, they will not be visible to the Discover Servers process of Oracle VM Manager.

After logging in to Oracle VM Manager, you should see a "home" view similar to Figure 2.

4/7/2015 How to Set Up a Cluster of x86 Servers with Oracle VM 3

http://www.oracle.com/technetwork/articles/serversstorageadmin/o12033x86clusgeroraclevm1619861.html 4/14

Figure 2. Home View of Oracle VM Manager

Click the Hardware icon on the lower left (highlighted in red in Figure 2). This will take you to the Hardware view shown in Figure 3, where you canadd networking and external storage as resources. Use Discover Servers to add them as resources by rightclicking the Resources folder in thenavigation tree and selecting Discover Servers from the menu.

Figure 3. Hardware View

Add the host systems, as shown in the Figure 4. You may use a range of IP addresses, as shown, or add them one at a time.

4/7/2015 How to Set Up a Cluster of x86 Servers with Oracle VM 3

http://www.oracle.com/technetwork/articles/serversstorageadmin/o12033x86clusgeroraclevm1619861.html 5/14

Figure 4. Adding IP Addresses

By default, the discovery process will add all the newly discovered servers to the Unassigned Servers resources pool. As shown in Figure 5, theUnassigned Servers pool now contains three servers: MyServer1, MyServer2, and MyServer3. These represent the three blade servers in the SunBlade 6000 modular system. The newly discovered x86 host systems will remain in the unassigned pool until you assign them to a server pool. Anew server pool will be created during a later step in this article and the servers will be assigned at that time.

4/7/2015 How to Set Up a Cluster of x86 Servers with Oracle VM 3

http://www.oracle.com/technetwork/articles/serversstorageadmin/o12033x86clusgeroraclevm1619861.html 6/14

Figure 5. Unassigned Servers Pool

Set Up Network Resources for Oracle VMOracle VM 3 allows finer control over networking compared to previous versions of Oracle VM. Oracle VM Manager can now be used to assigningnetwork roles and subnets to network devices on the x86 host systems, alleviating the need to log in to the host systems to configure networking byhand.

Most data centers will have multiple subnets with specific roles, such as a production front end where all users can connect to servers, databasesand applications (public network), dedicated storage (storage network), dedicated outofband server management (management network), and soon.

To keep things relatively simple, yet show some of the new powerful features of Oracle VM 3, this article describes the process of setting up twosubnets as follows:

A management subnet used for outofband management of the Oracle VM environmentA public subnet used for general access to storage and Oracle VM guests (as well as databases and applications)A single network was set up during the installation process for Oracle VM Manager and Oracle VM Server for x86. This existing network will beused as the management network and is the same network on which baremetal x86 servers were discovered in process described above. Thesecond step will be to create a new network that will become the public network for access to storage and Oracle VM guests.

To avoid confusion with network names, we will change the name of the existing management network to "Management network." This is not arequired change and is only meant to help reduce confusion about network roles in subsequent steps. Please adjust the names and roles of thesubnets in the following steps to conform to your particular network environment. If only a single subnet exists within the network, just modify theexisting network and assign all "uses" to that one subnet. Then skip the later "Create an Additional Public Network" section.

Configure the Management Network

Ensure that Resources is selected in the navigation pane, and then select Networks from management tab, as shown in Figure 6. Then click theEdit icon from the management toolbar just below the Networks tab.

4/7/2015 How to Set Up a Cluster of x86 Servers with Oracle VM 3

http://www.oracle.com/technetwork/articles/serversstorageadmin/o12033x86clusgeroraclevm1619861.html 7/14

Figure 6. Edit Icon

A dialog box from the Edit Network wizard will allow you to change the network name from the default subnet address to any string. In this case,change the Name field to Management network, as shown Figure 7, to allow easy identification in later steps.

The Network Use section on the right side of the GUI refers to the type of traffic for the subnet. In our test configuration, we had only one subnet.However, since most environments will have sufficient network ports to support two separate networks, we've written this article to show the stepsrequired to create two networks. Thus, we will detail the steps required to create a management network here in this section and a subsequentsection describes how to create an additional network for public use (VM guests and storage roles).

In our management network, we did not select the Storage or Virtual Machine checkboxes as Network Use roles, because this type of traffic will beconfigured in the public network in a subsequent section. As mentioned earlier, if you had only one subnet, then you would need to select theStorage and Virtual Machine options here in the management network and then skip the later "Create an Additional Public Network" section.

4/7/2015 How to Set Up a Cluster of x86 Servers with Oracle VM 3

http://www.oracle.com/technetwork/articles/serversstorageadmin/o12033x86clusgeroraclevm1619861.html 8/14

Figure 7. Changing the Network Name and Selecting Network Use Roles

The wizard will then step through the remaining four tasks. Simply accept the default on all subsequent dialog boxes until the Finish buttonbecomes active. The result should look like Figure 8.

4/7/2015 How to Set Up a Cluster of x86 Servers with Oracle VM 3

http://www.oracle.com/technetwork/articles/serversstorageadmin/o12033x86clusgeroraclevm1619861.html 9/14

Figure 8. Configured Management Network

Bond the Management Network Ports

The management network is critical to the operation of the server infrastructure, so it should be configured for high availability. This means that themanagement network ports should be bonded to allow the secondary network link to take over if the primary network links fails.

The bonding process described below establishes a logical connection between a primary and a secondary network port. The primary portremains the sole active port unless a problem occurs, and the secondary port is inactive unless the primary port fails. These bonded ports will allowthe blade server host systems to have uninterrupted management access even if the physical network layer for one of the network ports fails.

The steps that are described below assume that the basic network configuration is already in place. In other words, if network aggregation is to beused to consolidate multiple physical ports into a single logical network connection, that configuration has already been established.

We'll now set up network port bonding by repeating the steps below for each of the three servers.

First, select a server from the Unassigned Servers pool and click the Bonding tab.

4/7/2015 How to Set Up a Cluster of x86 Servers with Oracle VM 3

http://www.oracle.com/technetwork/articles/serversstorageadmin/o12033x86clusgeroraclevm1619861.html 10/14

Figure 9. Bonding Tab

There is an already existing bond (bond0) that was assigned to Bond Port(1) during the initial installation of Oracle VM Manager. Select this bondport and then select the Edit icon, as shown in Figure 10.

Figure 10. Editing Bond Port(1)

This brings up the Edit Port dialog box, as shown in Figure 11. From here, you can add the other management network port to the bond group.Note that the bonding type shown in the lower left corner of the screen is Active Backup. This means that the active port will handle the entirenetwork load unless it fails. If that happens, the secondary port will then take over the entire network load.

Move the secondary network port from the Available Ports list to the Selected Ports list, and click OK to continue.

Figure 11. Edit Port Dialog Box

As shown in Figure 12, multiple ports will now be associated with the bond.

4/7/2015 How to Set Up a Cluster of x86 Servers with Oracle VM 3

http://www.oracle.com/technetwork/articles/serversstorageadmin/o12033x86clusgeroraclevm1619861.html 11/14

Figure 12. Multiple Ports Are Now Associated with Bond Port(1)

Repeat the steps above to establish a bond/port group for each of the host systems in the environment.

Create an Additional Public Network

The process above focused on establishing high availability for the management network. Now we're ready to set up a new public network that willbe used for virtual machine I/O and storage I/O. This new network will allow Oracle VM Server for x86 host systems and Oracle VM Manager toaccess storage—as well as databases, applications, and so on—running on Oracle VM guests.

The create network task basically configures another network interface/bridge on the host servers. The network information is provided by theOracle VM administrator, as shown in the Oracle VM Manager Create Network wizard screenshots that follow.

Start by clicking the Create Network icon from the management toolbar just under the Network tab, as shown in the Figure 13.

Figure 13. Initiate the Create Network Function

Figure 14 shows the supported network types. Select the Create a network with bonds/ports only option because we want high availability.VLANs do not offer high availability, but they can be used with bond ports in a hybrid network. VLANs and hybrid networks are beyond the scope ofthis article. For more information about configuring VLANs, please refer to the Sun Ethernet Fabric Operating System VLAN Administration Guide.

Click Next.

Figure 14. Creating a New Network

Enter a name for the new network in the Name field and select the Virtual Machine and Storage checkboxes, as shown in Figure 15.

Figure 15. Naming the New Network

Next, select the host systems where the new network will be created and configured. All three of our blade servers will be part of this publicnetwork, so all of them are in the Selected Servers list in Figure 16.

Figure 16. Select the Host Systems to be Configured into the Network

The next step is to identify the port/interfaces. In Figure 17, the port/interfaces do not show the Linux name for the interface, such as eth1. Instead,they are shown as the logical names: such as Port(2),. Choose the appropriate interface to configure on each Oracle VM 3 server host, accordingto what is relevant to the environment. In our case, we're using Port(2) as the public network. Port(1) was already defined as the managementnetwork. Port(3) and Port(4) are unused.

Figure 17. Selected Ports

Finally, provide the IP address information of the storage subnet and Oracle VM guests on each host system.

Figure 18. Configuring IP Addresses

Figure 19 shows the public network after it has been configured on each host system.

Figure 19. Configured Public Network

Create the Virtual MAC Addresses

The last network task is to create a pool of virtual Ethernet (MAC) addresses for the Oracle VM guests. The virtual Ethernet addresses will berandomly assigned from the pool as each Oracle VM guest is created.

Note: The Oracle VM administrator can assign specific MAC addresses to specific Oracle VM guests by changing the randomly assigned MAC

4/7/2015 How to Set Up a Cluster of x86 Servers with Oracle VM 3

http://www.oracle.com/technetwork/articles/serversstorageadmin/o12033x86clusgeroraclevm1619861.html 12/14

addresses, as will be seen in a later step.

To begin, choose Vnic Manager from the Tools menu at the top of the GUI, as shown in Figure 20.

Figure 20. Opening the Vnic Manager

Enter any valid hex numbers for the last three octets in the Virtual Network Interface Card Manager dialog box (the first three octets cannot bechanged), and then click Generate.

Figure 21. Specifying MAC Addresses

Click Close once the pool of MAC addresses has been generated. The pool of MAC addresses will now be available for automatic assignment toany Oracle VM guests that are created. More Ethernet addresses can be generated later if needed.

Create the Oracle VM 3 Server PoolNow we want to create a server pool that will contain the server resources that can be used by Oracle VM Manager when allocating resources forOracle VM guests. The server pool requires a list of physical servers as well as networking and storage resources.

A server pool can be created only after all the other resources, such as host systems, networking, and storage, have been added to Oracle VMManager. The host systems and networking components were configured above and are, thus, already known by Oracle VM Manager. The storagesystems are assumed to have been registered with Oracle VM Manager at an earlier stage.

Select the Home shortcut in the navigation views pane to change the view to Home.

Figure 22. Returning to the Home View

Rightclick the Server Pool directory and select Create Server Pool.

Figure 23. Opening the Create Server Pool Dialog Box

In the Create Server Pool dialog box, enter the server pool name, the relocatable virtual IP address, and the NFS export used for the server poolfile system. The name MyServerPool has been entered as the server pool name in Figure 24. The virtual IP address is an address that will alwaysbe associated with the host system currently designated as the server pool master. Servers are not added to the server pool until later.

Select the Activate Cluster checkbox, as shown in Figure 24, to indicate that this server pool will be part of a cluster. Also select Network FileSystem rather than Physical Disk, because we'll be using shared storage on the Sun ZFS Storage Appliances.

Figure 24. Create Server Pool Dialog Box

Click the Location icon, shown in Figure 25, to select the server pool file system. This allows you to identify the NFS mount for use by the cluster ofvirtual machines.

Figure 25. Location Icon

This will bring up a dialog box that shows shared storage resources that have been previously registered with Oracle VM Manager. Select the NFSmount to be used for the server pool file system.

Figure 26. Selecting an NFS Mount

Click OK to create the server pool. The server pool does not yet have server host systems or a storage repository associated with it.

Figure 27. Creating the Server Pool

The next step is to add host systems to the newly created server pool. Simply rightclick the server pool name and select Add/Remove Servers.

4/7/2015 How to Set Up a Cluster of x86 Servers with Oracle VM 3

http://www.oracle.com/technetwork/articles/serversstorageadmin/o12033x86clusgeroraclevm1619861.html 13/14

Figure 28. Opening the Add/Remove Server from Server Pool Dialog Box

Select the host systems that will be included in the server pool, as shown in Figure 29.

Figure 29. Selected Servers

The server host systems will be moved from the unassigned folder and will now appear under the server pool name in the navigation tree, asshown in Figure 30.

Figure 30. Selected Servers Added to the Server Pool

The server pool is almost completed and just needs a storage repository.

Assign a Storage Repository

The final step in the process of creating a server pool is to assign an NFS mount that will act as the centralized storage repository where all of theOracle VM guest files, templates, and other resources will reside for the entire server pool.

Select the Repositories tab and then click the Create Repository icon on the management pane toolbar just under the tab.

Figure 31. Create Repositories Icon

Enter a name for the storage repository, choose Network File Server for the Repository Location, and then click the magnifying glass icon to bringup the Select Network File System dialog box.

Figure 32. Create Repository Dialog Box

Use the Select Network File System dialog box to choose the NFS export that was created earlier for use as a storage repository. Click OK whencompleted.

Figure 33. Choosing the NFS Export

Add an optional description and click OK to complete the creation of a storage repository.

Figure 34. Creating the Storage Repository

At this point, the storage repository has been created, but is not yet assigned to any server pool. So, the final step for creating a server pool is toassign the newly created storage repository to the specific host systems that will need access to the centralized storage.

To make this assignment, select the newly created storage repository, and then click the PresentUnpresent Selected Repository icon (up/downgreen arrow) from the toolbar just below the Repositories tab.

Figure 35. PresentUnpresent Selected Repository Icon

Select all of the server host systems that will have access to the repository and then click OK.

Figure 36. Selecting Servers for the Repository

The storage repository will now show the host systems whenever the storage repository is selected in the Repositories tab. Note that host systemscan have access to multiple storage repositories, so you can repeat the above steps for other storage repositories if additional storage repositoriesare desired.

Figure 37. Configured Cluster

The Oracle VM cluster is now ready for Oracle VM guest images to be created.

4/7/2015 How to Set Up a Cluster of x86 Servers with Oracle VM 3

http://www.oracle.com/technetwork/articles/serversstorageadmin/o12033x86clusgeroraclevm1619861.html 14/14

ConclusionBy performing the steps described in this article, you will have completed the core tasks to set up a highly available enterprise cloud infrastructure.The remaining steps include configuring the storage environment and installing other software components, such as the Oracle Database. Thesesteps and more are described in the "Oracle Optimized Solution for Enterprise Cloud Infrastructure—Implementation Guide (x86Linux)" whitepaper.

See AlsoAdditional information about the latest Sun x86 systems is available on the Sun x86 Systems Website.

About the AuthorsKris Bakke has spent a combined 16 years at Oracle and Sun Microsystems before Sun was acquired by Oracle. His current responsibilitiesinclude developing the Oracle Optimized Solution for Enterprise Cloud Infrastructure, which provides an integrated hardware and softwareinfrastructure that is cloudready. He has previous experience in product marketing and management as well as solution development fordatabase backup, virtualization, data protection, cloud computing, and video surveillance.

Jeff Kiely is a Senior Technical Product Manager in Oracle's Sun x86 systems group and has been involved with the past two generations of x86servers. He has held prior roles as a hardware engineer and a support engineer for x86 systems for over 7 years at Fujitsu America.

Revision 1.0, 05/03/2012

Follow us on Facebook, Twitter, or Oracle Blogs.

Email this page Printer View

ORACLE CLOUDLearn About Oracle CloudComputingGet a Free TrialLearn About DaaSLearn About SaaSLearn About PaaSLearn About IaaSLearn About Private CloudLearn About Managed Cloud

JAVALearn About JavaDownload Java forConsumersDownload Java forDevelopersJava Resources forDevelopersJava Cloud ServiceJava Magazine

CUSTOMERS AND EVENTSExplore and Read CustomerStoriesAll Oracle EventsOracle OpenWorldJavaOne

COMMUNITIESBlogsDiscussion ForumsWikisOracle ACEsUser GroupsSocial Media Channels

SERVICES AND STORELog In to My Oracle SupportTraining and CertificationBecome a PartnerFind a Partner SolutionPurchase from the OracleStore

CONTACT AND CHATPhone: +1.800.633.0738 Global ContactsOracle SupportPartner Support

Subscribe Careers Contact Us Site Maps Legal Notices Terms of Use Privacy Cookie Preferences Oracle Mobile