DELL EMC VxRAIL™ NETWORK GUIDE · DELL EMC VxRAIL™ NETWORK GUIDE Physical and Logical Network...

-

DELL EMC VxRAIL™ NETWORK GUIDE

Physical and Logical Network Considerations and

Planning

ABSTRACT

This is a planning and consideration guide for VxRail Appliances. It can be

used to understand better the networking required for VxRail implementation.

This whitepaper does not replace the requirement for implementation services

with VxRail Appliances and should not be used in an attempt to implement the

required networking for VxRail Appliances.

April 2018

WHITE PAPER

2 | Network Planning Guide © 2017 Dell Inc. or its subsidiaries

Table of contents

Intended Use and Audience ..................................................................................................................... 4

Introduction to VxRail ............................................................................................................................... 4

Decision Regarding vCenter Server ................................................................................................................. 4

Planning Your Network .................................................................................................................................... 5

Physical Network ............................................................................................................................................. 5

VxRail Clusters, Appliances and Nodes ........................................................................................................ 5

Network Switch ............................................................................................................................................ 7

Topology and Connections ......................................................................................................................... 10

Workstation/Laptop .................................................................................................................................... 11

Out-of-Band Management (optional)........................................................................................................... 12

Before Cabling VxRail Appliances .......................................................................................................... 12

Step 1: Plan Logical Network ......................................................................................................................... 12

Step 1A. Reserve VLANs (Best Practice) ....................................................................................................... 13

Step 1B. System ............................................................................................................................................ 14

Time Zone, NTP Server ............................................................................................................................. 14

DNS Server ................................................................................................................................................ 14

Step 1C. Management ................................................................................................................................... 15

ESXi Hostnames and IP Addresses............................................................................................................ 15

vCenter Server ........................................................................................................................................... 16

VxRail Manager and Networking ................................................................................................................ 17

Passwords ................................................................................................................................................. 18

Step 1D. vMotion and vSAN ........................................................................................................................... 18

Step 1E. Solutions ......................................................................................................................................... 19

Step 1F. Workstation/Laptop .......................................................................................................................... 19

Step 2: Set Up Switch .................................................................................................................................... 20

Step 2A. Understanding Switch Configuration ................................................................................................ 20

Network Traffic ........................................................................................................................................... 20

Inter-switch Communication ....................................................................................................................... 23

Disable Link Aggregation............................................................................................................................ 24

vSphere Security Recommendations .......................................................................................................... 24

Step 2B. Configure VLANs on Your Switch(es)............................................................................................... 24

Step 2C. Confirm Your Configuration ............................................................................................................. 25

After Planning and Switch Setup ............................................................................................................ 25

Unassigned Physical Ports .................................................................................................................... 26

Network Segregation .................................................................................................................................. 27

VxRail Network Configuration Table ....................................................................................................... 28


VxRail Setup Checklist ........................................................................................................................... 29

Appendix A: NSX Support on VxRail ...................................................................................................... 30

Appendix B: VxRail Open Ports Requirement ......................................................................................... 33

Appendix C: VxRail Firewall Rules & ACI Contracts Diagram ................................................................. 37

Appendix D: Physical Network Switch Examples .................................................................................... 38


Intended Use and Audience

This guide discusses the essential network details for VxRail deployment planning purposes only. It also introduces best

practices, recommendations, and requirements for both physical and virtual network environments. The guide has been

prepared for anyone involved in planning, installing, and maintaining VxRail, including Dell EMC field engineers and customer

system and network administrators. This guide should not be used to perform the actual installation and set-up of VxRail.

Please work with your Dell EMC service representative to perform the actual installation.

Introduction to VxRail

Dell EMC VxRail™ Appliances are a hyper-converged infrastructure (HCI) solution that consolidates compute and storage into

a single, highly available, network-ready unit. With careful planning, VxRail Appliances can be rapidly deployed into an existing

environment, and the infrastructure is immediately available to deploy applications and services.

VxRail is not a server. It is an appliance. The G Series consists of up to four nodes in a single appliance, all other models

based on Dell EMC PowerEdge Servers are a single node per appliance. A 10GbE switch (or a 1GbE switch for certain

models of VxRail) is required. A workstation/laptop for the VxRail user interface is also required.

VxRail has a simple, scale-out architecture, leveraging VMware vSphere® and VMware vSAN™ to provide server virtualization

and software-defined storage. Fundamental to the VxRail clustered architecture is network connectivity. It is through the logical

and physical networks that individual nodes act as a single system providing scalability, resiliency and workload balance.

The VxRail software bundle is preloaded onto hardware and consists of the following components (specific software versions

not shown):

VxRail Manager

VMware vCenter Server™

VMware vRealize Log Insight™

VMware vSAN

Dell EMC Secure Remote Support (ESRS)/VE

Also preloaded is VMware vSphere; however, licenses are required and can be purchased through Dell EMC, VMware or your

preferred VMware reseller partner.

The VxRail Appliances also includes licenses for software that can be downloaded, installed and configured:

Dell EMC RecoverPoint for Virtual Machines (RP4VM) - 15 Full Licenses per G-series appliance chassis or 5 Full

Licenses per all other single node per chassis VxRail series appliances

VxRail is fully compatible with other software in the VMware ecosystem, including VMware NSX. Refer to the VMware Product

Interoperability Matrixes for specific versions of NSX supported on vSphere specific versions.

Decision Regarding vCenter Server

A VxRail Cluster’s virtual infrastructure is managed by a single vCenter Server instance, either VxRail vCenter Server or

Customer Supplied vCenter Server. When a VxRail Appliance is deployed, the vCenter deployment type is selected and is

difficult to change. If a customer wants to make a change, for VxRail versions 3.5 and 4.0.1 it would require a factory reset and

all data would need to be wiped from the VxRail Appliance and reinstallation would be required. To migrate a VxRail running

4.0.200 vCenter Server to a Customer Supplied vCenter requires an RPQ. Starting with VxRail 4.0.301 your Dell EMC service

team can do this procedure.

The Customer Supplied vCenter Server option provides more configuration options and is the recommended choice.

Refer to the Dell EMC VxRail vCenter Server Planning Guide for details.

http://partnerweb.vmware.com/comp_guide2/sim/interop_matrix.php?#interop&93=&32=

http://partnerweb.vmware.com/comp_guide2/sim/interop_matrix.php?#interop&93=&32=

https://vxrail.is/vcenterplanning


Planning Your Network

The network considerations are no different from those of any enterprise IT infrastructure: availability, performance, and

extensibility. VxRail Appliances are delivered ready to deploy and attach to any 10GbE network infrastructure and use IPv4

and IPv6. Some models with single processors are available for 1GbE networks. Most production VxRail network topologies

use dual top-of-the-rack (ToR) switches to eliminate the switch as a single point of failure.

Follow all of the network prerequisites described in this document; otherwise, VxRail will not install properly, and it will not

function correctly in the future. If you have separate teams for network and servers in your data center, you will need to work

together to design the network and configure the switch(es). If your site is behind a firewall, make sure all necessary ports are

open, as demonstrated in Appendix B and C.

Physical Network

This section describes the physical components found in a VxRail cluster:

VxRail clusters, appliances and nodes

Network switch

Topology and connections

Workstation/laptop

Out-of-band management (optional)

VxRail Clusters, Appliances and Nodes

VxRail starts with a minimum of 3 nodes (either in a single G-series chassis or three individual appliance nodes for all other

models) connected to one or more network switches, deployed to form a VxRail cluster that contains the vSAN environment. A

maximum of 64 VxRail nodes can be in a VxRail cluster. The internal disks on each node combine to create a VxRail datastore

that is shared across all the nodes in the cluster. Within the cluster, multiple networks may serve different functions or types of

traffic.

The cluster is managed by a single instance of VxRail Manager and vCenter Server. A logical tag in each node and chassis is

used to display the identity of the appliance in VxRail Manager. These tags are 11 alphanumeric characters that uniquely

identify the appliance.

Please review the physical power, space and cooling requirements for your expected resiliency level.

The following illustrations show possible configurations of a VxRail Appliance.

Figure 1. VxRail P and V Series on 14th Generation Edge Servers, showing 4x10GbE ports

iDRAC port 10 GbE Ports


Figure 2. VxRail S Series node on 14th generation Dell EMC PowerEdge, showing 4x10GbE ports

Figure 3. VxRail E Series node on 14th generation Dell EMC PowerEdge, showing 4x10GbE ports

NOTE: The 2x10GbE ports will auto-negotiate to 1GbE when used with 1GbE networking

Figure 4. VxRail E, P, S and V Series node on 13th generation Dell EMC PowerEdge, showing the 10GbE and 1GbE ports

iDRAC

port

10 GbE 1 GbE

iDRAC

port

10 GbE

Ports

1 GbE

Ports




Figure 5. VxRail G Series appliance with four nodes, showing the 10GbE ports on each node

Figure 6. VxRail G Series appliance with four nodes, showing the 1GbE ports on each node

Network Switch

VxRail is broadly compatible with most customer networks and switches. VxRail nodes communicate over one or more

customer-provided network switch(es), typically a top-of-rack switch. One example is the Dell EMC Switch S4048 (more

details on planning configuration specific to that switch can be found in DELL EMC Switch Configuration Guide for VxRail.)

Switch requirements:

The switch(es) connected directly to VxRail Appliances must support multicast on 10GbE ports for all models of VxRail

except for the models that utilize 1GbE for their primary networking as specified:

o VxRail Releases prior to Release 4.5.0: both IPv4 and IPv6 multicast pass-through must be supported. Layer 3

multicast is not required.

o VxRail Releases starting with Release 4.5.0: IPv6 multicast pass-through and IPv4 unicast must be supported.1

Be sure to have access to the manufacturer’s documentation for your specific switch(es).

Keep in mind that while a Top-of-rack switch can work, it is a potential single point of failure. We recommend using dual

top-of-the-rack (ToR) switches.

1 vSAN no longer requires multicast, but is using unicast instead.

BMC port

port

1GbE ports

http://en.community.dell.com/cfs-file/__key/telligent-evolution-components-attachments/13-4629-00-00-20-44-34-24/Dell-EMC-Switch-Configuration-Guide-for-VxRail.pdf?forcedownload=true


Port availability:

Figure 7. VxRail 14th Generation Node Connectivity Summary

E, P, S and V Series (14th Generation Dell EMC PowerEdge Servers)

o Each VxRail node comes with a Network Daughter Card (NDC) consisting of either:

o 2x10GbE in either SFP+ or RJ-45 NIC ports

o 4x10GbE in either SFP+ or RJ-45 NIC ports

o 2x25GbE SFP28 (Starting with VxRail 4.5.200)

o For the E, P, and S series single processor models, only 1GbE connectivity is supported.

o Four corresponding ports are required for each VxRail node on one or more ToR switch(es).

o One additional port on the switch or one logical path on the VxRail management VLAN is required for a

workstation/laptop to access the VxRail user interface for the cluster.

o For the P, V and S series, additional PCI-e NICs can be added to the node, in either SFP+ or RJ45 interface.

Starting with VxRail 4.5.200, the additional PCI-e NICs can be 25GbE SFP28.

o All flash VxRail models must use either 10GbE or 25GbE NICs. 1GbE is not supported for all-flash.

o VxRail initialization process will not touch an additional PCI-e NIC. Customers can use the ports for their own

purposes such as VM networks, iSCSI, or NFS, etc.

o During the VxRail initialization process, the hardware configuration must have same NDC across all VxRail

nodes. There are no restrictions on the PCI-e NIC cards.


Figure 8. VxRail Pre-14th Generation Node Connectivity Summary

E, P, S and V Series (13th Generation Dell EMC PowerEdge Servers)

o Each VxRail node comes with a Network Daughter Card (NDC) consisting of 2x10GbE + 2x1GbE in either SFP+

or RJ-45 NIC ports.

o The 2x10GbE ports will auto-negotiate to 1GbE when used with 1GbE networking.

o Two (2) corresponding ports are required for each VxRail node on one or more 10GbE switch(es) when utilizing

10GbE as the primary networking speed.

o Four (4) corresponding ports are required for each VxRail node on one or more 1GbE switch(es) when utilizing

1GbE networking on the single processor modes. (Note: The P and V series do not offer any single processor

configurations.)

o All flash VxRail models must use either 10GbE or 25GbE NICs. 1GbE is not supported for all-flash.

o One (1) additional port on the switch or one logical path on the VxRail management VLAN is required for a


o Up to three additional PCI-e NIC can be added to the node, except single processor E460.

- The interface can be either SFP+ or RJ45.

- VxRail initialization process will not touch PCI-e NIC. Customers can use the ports for their own purposes

such as VM networks, iSCSI, or NFS, etc.

o During initial deployment, the hardware configuration must be identical including NDC and PCI-e NIC card. After

cluster has formed there is no requirement for PCI-e NIC card on additional nodes which means they may or may

not have the same type PCI-e NIC card as the existing nodes


This restriction only applies to VxRail 4.0.0. In an E, P, S, and V series VxRail 4.0.0 Appliances utilizing 10GbE, the

1GbE NIC ports must be disconnected during VxRail initialization, node addition and node replacement.

G Series

o VxRail nodes with 10GbE ports ships with either two SFP+ or RJ-45 NIC ports. Two (2) corresponding ports are

required for each VxRail node on one or more 10GbE switch(es). Six (6) ports are needed for a three-node initial

configuration.

o VxRail nodes with 1GbE ports ships with four RJ-45 NIC ports. Four (4) corresponding ports are required for

each VxRail node on one or more 1GbE switch(es). Twelve (12) ports are needed for a three-node initial

configuration.

o One (1) additional port on the switch or one logical path on the VxRail management VLAN is required for a


Cable requirements:

VxRail nodes with RJ-45 ports require CAT5 or CAT6 cables. CAT6 cables are included with every VxRail

VxRail nodes with SFP+ ports require optics modules (transceivers) and optical cables, or Twinax Direct-Attach-Copper

(DAC) cables. These cables and optics are not included; you must supply your own. The NIC and switch connectors and

cables must be on the same wavelength.

Please review the logical switch configuration requirements in the next section of this document.

Topology and Connections

Various network topologies for switch(es) and VLANs are possible with VxRail Appliances. Complex production environments

will have multiple core switches and VLANs. A site diagram showing the proposed network components and connectivity is

highly recommended before cabling and powering on VxRail Appliances.

Be sure to follow your switch vendor’s best practices for performance and availability. For example, packet buffer banks may

provide a way to optimize your network with your wiring layout.

Decide if you plan to use one or two switches for VxRail. One switch is acceptable and is often seen in test/development or

remote/branch office (ROBO) environments. However, two or more switches are used for high availability and failover in

production environments because VxRail is an entire software-defined data center in a box if one switch fails you are at risk of

losing availability of hundreds of virtual machines.


Figure 9 shows the recommended physical network setup using a management switch (for iDRAC) and two ToR switches.

Other network setups can be found in Appendix D.

Figure 9. Rear view of one deployment of a VxRail Appliance connected to two 10GbE switches and a separate switch for out-of-band management. These are 14G E series servers.

For 13th generation PowerEdge servers in the E, P, S and V series VxRail Appliances utilizing 1GbE with two

switches, the switches must be interconnected.

Workstation/Laptop

A workstation/laptop with a web browser for the VxRail user interface is required. It must be either plugged into the switch or

able to logically reach the VxRail management VLAN from elsewhere on your network; for example, a jump server

(https://en.wikipedia.org/wiki/Jump_server).

Don’t try to plug your workstation/laptop directly into a server node on a VxRail Appliance; plug it into your network or

switch and make sure that it is logically configured to reach VxRail.

You will use a browser for the VxRail user interface. The latest versions of Firefox, Chrome, and Internet Explorer 10+ are all

supported. If you are using Internet Explorer 10+ and an administrator has set your browser to “compatibility mode” for all

https://en.wikipedia.org/wiki/Jump_server


internal websites (local web addresses), you will get a warning message from VxRail. Contact your administrator to whitelist

URLs mapping to the VxRail user interface.

Out-of-Band Management (optional)

If the VxRail Appliances are located at a data center that you cannot access easily, we recommend setting up an out-of-band

management switch to facilitate direct communication with each node.

For E, P, S and V Series Based on PowerEdge Servers:

To use out-of-band management, connect the internal Dell Remote Access Controller (iDRAC) port to a separate switch to

provide physical network separation.

Default values, capabilities, and recommendations for out-of-band management are provided with server hardware

information. The default configuration is:

Username: root Password: calvin

You will need to reserve an IP address for each iDRAC in your VxRail cluster (one per node).

For G-Series (and VxRail Appliances Prior to VxRail 4.0):

To use out-of-band management, connect the BMC port on each node to a separate switch to provide physical network

separation.

Default values, capabilities, and recommendations for out-of-band management are provided with server hardware

information. The default configuration is via DHCP with:

Username: UserId Password: Passw0rd!

NOTE: Case sensitive and using a zero in place of a lowercase ‘o’ in the password

The <ApplianceID> can be found on a pullout tag located in front of the chassis. The default hostnames should be as follows:

BMC interface node 1: hostname = <ApplianceID>-01




Before Cabling VxRail Appliances

Step 1: Plan Logical Network

VxRail is not a simple server but is an entire data center in a box. Consequently, the network and virtualization teams need to

meet in advance to plan VxRail’s network architecture.

Use the VxRail Setup Checklist and the VxRail Network Configuration Table to help create your network plan. References

to rows in this document are to rows in the VxRail Network Configuration Table.

Once you set up VxRail Appliances, the configuration cannot be changed easily. Consequently, we strongly

recommend that you take care during this planning phase to decide on the configurations that will work most

effectively for your organization.

A VxRail cluster consists of three or more VxRail nodes starting in VxRail 4.0 and four or more VxRail nodes in earlier

releases. VxRail clusters can scale out to 64 ESXi hosts all on one vSAN datastore, backed by a single vCenter Server and

VxRail Manager. Deployment, configuration, and management are handled by VxRail, allowing the compute capacity and the

vSAN datastore to grow automatically. VxRail Manager automatically discovers the new node and configures the new node,

and automatically adds the new node to default vSphere Distributed Switch. vCenter propagates the port groups of default

VDS to the new node. However, if customers manually add a new VDS/VSS, or add unused physical network adapter(s) to


default/new VDS/VSS, then they need to configure the network on the new node manually. At this time all VxRail nodes in a

cluster must be the same hardware series.

You will be making decisions in the following areas:

Step 1A. Reserve VLANs (best practice)

Step 1B. System

Step 1C. Management

Step 1D. vMotion and vSAN

Step 1E. Solutions

Step 1F. Workstation/laptop

Step 1A. Reserve VLANs (Best Practice)

VxRail groups traffic in the following categories: management, vSphere vMotion, vSAN, and Virtual Machine. Traffic isolation

on separate VLANs is highly recommended (but not required) in VxRail. If you are using multiple switches, connect them via

VLAN trunked interfaces and ensure that all VLANs used for VxRail are carried across the trunk following the requirements in

this user guide.

Management traffic includes all VxRail, vCenter Server, and ESXi communication. The management VLAN also carries traffic

for vRealize Log Insight. All management traffic should be untagged and must be able to go over a Native VLAN on your

switch, or you will not be able to build VxRail and configure the ESXi hosts.

Special cases can allow you to tag management traffic in one of two ways:

1. Configure each VxRail port on your switch to tag the management traffic and route it to the desired VLAN.

2. Alternately, you can configure a custom management VLAN to allow tagged management traffic after you power on each node, but before

your run VxRail initial configuration. Your Dell EMC service representative will take care of this during installation.

In VxRail Appliances, vSphere vMotion and vSAN traffic cannot be routed. This traffic will be tagged for the VLANs you

specify in VxRail initial configuration.

Dedicated VLANs are preferred to divide virtual machine traffic. VxRail will create one or more VM Networks for you, based

on the name and VLAN ID pairs that you specify. Then when you create VMs in vSphere Web Client, you can easily assign

the virtual machine to the VM Network(s) of your choice. For example, you could have one VLAN for Development, one for

Production, and one for Staging.

Network Configuration

Table

Row 1

Enter the management VLAN ID for VxRail, ESXi, and vCenter Server. If you do

not plan to have a dedicated management VLAN and will accept this traffic as

untagged, enter “0” or “Native VLAN.”


Table

Row 36

Enter a VLAN ID for vSphere vMotion.

(Enter a 0 in the VLAN ID field for untagged traffic)


Table

Row 40

Enter a VLAN ID for vSAN.



Table

Rows 41-42

Enter a Name and VLAN ID pair for each VM network you want to create.

You must create at least one VM Network.



NOTE: If you have multiple independent VxRail clusters, we recommend using different VLAN IDs for vSAN traffic and

management traffic across multiple VxRail clusters to reduce network traffic congestion.

Step 1B. System

VxRail can configure connections to external servers in your network.

Time Zone, NTP Server

A time zone is required. It is configured on vCenter Server and each ESXi host.

An NTP server is not required, but it is recommended. If you provide an NTP server, vCenter Server will be configured to use

it. If you do not provide at least one NTP server, VxRail uses the time that is set on ESXi host #1 (regardless of whether the

time is correct or not).

A proxy server is optional and only applies to VxRail models prior to 3.5. If you have a proxy server on your network and

vCenter Server needs to access services outside of your network, supply the IP address, port, username, and password.


Table

Row 3

Enter your time zone.


Table

Row 4

Enter the hostname(s) or IP address(es) of your NTP server(s).


Table

Rows 6 and 7

Enter the proxy server IP address, port, username, and password.

DNS Server

One or more external DNS servers are required when using a Customer Supplied vCenter Server. When using a VxRail

vCenter Server, one or more external DNS servers are required starting in VxRail 4.5.0 even in isolated environments.

When using your corporate DNS server(s) for VxRail, be sure to add the hostnames and IP addresses for VxRail Manager,

vCenter Server, Log Insight, and each ESXi host (see the naming scheme in ESXi Hostnames and IP Addresses) prior to

VxRail initial configuration. vMotion and vSAN IP addresses are not configured for routing by VxRail, so there are no

hostnames to enter in the DNS server.

Example of VxRail hostnames and IP addresses configured on a DNS server:

esxi-host01.localdomain.local 192.168.10.1




vxrail.localdomain.local 192.168.10.100

vcserver.localdomain.local 192.168.10.101

loginsight.localdomain.local 192.168.10.102

Releases prior to VxRail 4.5.0 with a VxRail vCenter Server deployed did not require a DNS server in an isolated environment.

During initial configuration, VxRail sets up the VxRail vCenter Server to resolve hostnames to the DNS server. If you are in an

isolated environment, you will need to use the DNS server that is built into VxRail vCenter Server. To manage VxRail via your

workstation/laptop, configure your laptop’s network settings to use the vCenter Server IP address (Row 15) for DNS. VxRail’s

IP addresses and hostnames are configured for you.


Make sure that the DNS IP address is accessible from the network to which VxRail is connected and

functioning properly.

For Releases prior to VxRail 4.5.0, if the DNS server requires access via a gateway that is not reachable during

initial configuration, do not enter a DNS IP address. Instead, add a DNS server after you have configured VxRail using

VMware KB 2107249.


Table

Row 5

Enter the IP address(es) for your DNS server(s). This is required starting in

VxRail 4.5.0. It is also required when using a Customer Supplied vCenter

Server.

For releases prior to 4.5.0, leave blank in an isolated environment when using

the VxRail vCenter Server.

Step 1C. Management

VxRail does not have a single hostname. You must configure the hostnames for each ESXi host, VxRail Manager, and

vCenter Server.

You must configure the IP addresses for VxRail, vCenter Server, and your ESXi hosts. When selecting your IP addresses, you

must make sure that none of them conflict with existing IP addresses in your network. Also, make sure that these IP addresses

can reach other hosts in your network.

You cannot easily change the IP addresses after you have configured VxRail.

ESXi Hostnames and IP Addresses

All ESXi hostnames in a VxRail cluster are defined by a naming scheme that comprises: an ESXi hostname prefix (an

alphanumeric string), a separator (“None” or a dash ”-“), an iterator (Alpha, Num X, or Num 0X), an offset2 (empty or numeric),

a suffix3 (empty or alphanumeric string with no .) and a domain. The Preview field shown during VxRail initial configuration is

an example of the hostname of the first ESXi host. For example, if the prefix is “host,” the separator is “None,” the iterator is

“Num 0X”, the offset is empty, and the suffix is “lab”, and the domain is “local,” the first ESXi hostname would be

“host01lab.local”. The domain is also automatically applied to the vCenter Server and VxRail virtual machines. (Example: my-

vcenter.local).

Example 1 Example 2 Example 3

Prefix host myname esxi-host

Separator None - -

Iterator Num 0X Num X Alpha

Offset 4

Suffix lab

Domain local college.edu company.com

Resulting hostname host01.local myname-4lab.college.edu esxi-host-a.company.com

There are three or more ESXi hosts in your initial cluster and each requires an IP address. If you plan to scale out with

additional nodes in this VxRail cluster within the first few weeks after installation, we recommend you allocate extra IP

addresses for each of the ESXi, vMotion, and vSAN IP pools when you initially configure VxRail (three extra IP addresses per

node). Then when you add nodes to a cluster, you will only need to enter the ESXi and VxRail / vCenter Server passwords.

2 Offset is available starting in VxRail Release 4.0.200. It is only applicable when the iterator is numeric. 3 Suffix is available starting in VxRail Release 4.0.200.

http://kb.vmware.com/kb/2107249



Table

Rows 8-13

Enter an example of your desired ESXi host-naming scheme. Be sure to show

your desired prefix, separator, iterator, offset, suffix and domain.


Table

Rows 14 and 15

Enter the starting and ending IP addresses for the ESXi hosts - a continuous IP

range is required, with a minimum of 4 IPs.

vCenter Server

A new feature introduced in VxRail 3.5 is the ability to join a compatible Customer Supplied vCenter Server environment.

This allows a remote central vCenter Server to manage multiple VxRail clusters. Each VxRail environment appears within

vCenter Server as a cluster of hosts configured with a vSAN datastore. Prior to VxRail 4.5.200, the Customer Supplied

vCenter can NOT be hosted on the VxRail Cluster it is managing.4

Optionally, if you want VxRail to create a new VxRail vCenter Server, you will need to specify a hostname and IP address for

your VxRail vCenter Server and Platform Services Controller (PSC) virtual machines. (Rows 16-19)

If you want VxRail to join an existing Customer Supplied vCenter Server, you will need to:

The Customer Supplied vCenter Server version must match the VxRail vCenter Server version. In addition, if the

Customer Supplied vCenter Server is hosted outside the VxRail Cluster, then the VMware ESXi release must match the

ESXi release on the VxRail Cluster. Check the VxRail Release Notes for to determine the proper release numbers.

o VxRail 3.5 and vSphere 6.0, version details can be found in VxRail Appliance Software 3.5 Release Notes.

o VxRail 4.0.x and vSphere 6.0, version details can be found in VxRail Appliance Software 4.0.x Release Notes.

o VxRail 4.5.x and vSphere 6.5, version details can be found in VxRail Appliance Software 4.5.x Release Notes.

Know whether your Customer Supplied vCenter Server has an embedded or Customer Supplied Platform Services

Controller. If the PSC is Customer Supplied, enter the PSC FQDN (Row 20).

Know the Customer Supplied vCenter Server FQDN (Row 21), Single Sign-on domain (SSO) (Row 22).

(Optional) Create a VxRail admin user and password for initial deployment (Row 23). Two new roles will be created and

assigned to this user by your Dell EMC Representative.

Create or re-use a VxRail management user and password (Row 24) for this VxRail cluster on the Customer Supplied

vCenter Server. This user must be created with no permissions and no roles.

Create or select an existing datacenter (Row 25) on the Customer Supplied vCenter Server.

Specify the name of the cluster (Row 26) that will be created by VxRail in the selected datacenter when the cluster is built.

This name must be unique and not used anywhere in the datacenter on the Customer Supplied vCenter Server.

Starting with Release 3.5, the top-level domain of the Customer Supplied vCenter Server and PSC must be publicly

known, such as .com, .net, .edu, .local, and many country-specific suffixes. Most of those listed in this reference are

supported: https://en.wikipedia.org/wiki/List_of_Internet_top-level_domains

4 When using Stretched Clusters, the Customer Supplied vCenter Server can NOT be hosted on the VxRail Cluster.

https://support.emc.com/docu70688_VxRail-Appliance-3.5-Release-Notes.pdf?language=en_US

https://support.emc.com/docu80740_VxRail-Appliance-Software-4.0.x-Release-Notes.pdf?language=en_US

https://support.emc.com/docu86659_VxRail-Appliance-Software-4.5.x-Release-Notes.pdf?language=en_US

https://en.wikipedia.org/wiki/List_of_Internet_top-level_domains


VxRail vCenter Server (deployed when VxRail is built)


Table

Row 16

Enter an alphanumeric string for the new vCenter Server hostname. The domain specified in

Row 11 will be appended.


Table

Row 17

Enter the IP address of the new vCenter Server.


Table

Row 18

Enter an alphanumeric string for the new Platform Services Controller hostname. The

domain specified in Row 11 will be appended.


Table

Row 19

Enter the IP address of the new Platform Services Controller.

Customer Supplied vCenter Server


Table

Row 20

Enter the FQDN of the Customer Supplied Platform Services Controller (PSC) in the

hostname. In the user interface, there is a checkbox for Customer Supplied PSC.

Leave this row blank if the PSC is embedded in the Customer Supplied vCenter Server.


Table

Row 21

Enter the FQDN of the Customer Supplied vCenter Server in the hostname field.


Table

Row 22

Enter the Single Sign-on (SSO) domain for the Customer Supplied vCenter Server. (For

example vsphere.local)


Table

Row 23

Enter the administrative username/password for the Customer Supplied vCenter Server.

Optionally create a VxRail non-admin user on the Customer Supplied vCenter Server and

enter the VxRail non-admin username/password here.


Table

Row 24

Create or re-use the VxRail management user with no permissions and no roles for this

cluster. Enter the full VxRail management username/password.

(For example, [email protected])


Table

Row 25

Go to the Customer Supplied vCenter Server and select or create a datacenter.

Enter the name of a datacenter on the Customer Supplied vCenter Server.


Table

Row 26

Enter the name of the cluster that will be created by VxRail.

VxRail Manager and Networking

You must specify the hostname and IP address for the VxRail Manager virtual machine. In addition, you must specify the

subnet mask and gateway that VxRail Manager, vCenter Server, and the ESXi hosts all share.

We do not recommend using the default VxRail initial IP address (192.168.10.200/24) as your permanent VxRail IP

address (Row 26) because if you add more nodes to the VxRail cluster later or if you create more clusters, the

initial IP addresses will conflict with the existing cluster’s IP address.



Table

Row 27

Enter an alphanumeric string for the VxRail Manager hostname.


Table

Row 28

Enter the IP address for VxRail Manager after it is configured. We recommend that you do

not use the default 192.168.10.200/24


Table

Rows 29 and 30

Enter the subnet mask and gateway for all management IP addresses.

Passwords

Starting in VxRail 4.0.100, you can specify a different root password for each ESXi host in the cluster. Prior to that, the same

root password was used for all the ESXi hosts. You must also specify one password for the VxRail Manager virtual machine.

Unless you are using a Customer Supplied vCenter Server, the VxRail Manager and vCenter Server virtual machines will have

the same administrative password.

Passwords must contain between 8 and 20 characters with at least one lowercase letter, one uppercase letter, one numeric

character, and one special character. For more information about password requirements, see the vSphere password

documentation and vCenter Server password documentation.

For ESXi hosts, the username is root; the pre-configuration password is Passw0rd! and the post-configuration password is the

one you set in VxRail initial configuration (Row 29).

For VxRail Manager and the VxRail vCenter Server, the username for both user interfaces is [email protected] and

the console username is root. The pre-configuration password for VxRail is Passw0rd! and the post-configuration password is

the one you set in VxRail initial configuration (Row 30).


Table

Rows 31 and 32

Please check that you know your passwords in these rows, but for security reasons, we

suggest that you do not write them down.

Step 1D. vMotion and vSAN

vSphere vMotion and vSAN each require at least three IP addresses for the initial cluster.

Because VxRail supports up to 64 nodes in a cluster, you can allocate up to 64 vMotion IP addresses and 64 vSAN IP

addresses.

https://docs.vmware.com/en/VMware-vSphere/6.0/com.vmware.vsphere.security.doc/GUID-DC96FFDB-F5F2-43EC-8C73-05ACDAE6BE43.html

https://docs.vmware.com/en/VMware-vSphere/6.0/com.vmware.vsphere.security.doc/GUID-DC96FFDB-F5F2-43EC-8C73-05ACDAE6BE43.html

https://docs.vmware.com/en/VMware-vSphere/6.0/com.vmware.vsphere.security.doc/GUID-4BDBF79A-6C16-43B0-B0B1-637BF5516112.html



Table

Rows 33 and 34

Enter the starting and ending IP addresses for vSphere vMotion – a continuous IP range is

required, with a minimum of 4 IPs. Routing is not configured for vMotion.


Table

Row 35

Enter the subnet mask for vMotion.


Table

Rows 37 and 38

Enter the starting and ending IP addresses for vSAN – a continuous IP range is required,

with a minimum of 4 IPs. Routing is not configured for vSAN.


Table

Row 39

Enter the subnet mask for vSAN.

Step 1E. Solutions

VxRail is deployed with vRealize Log Insight. Alternately, you may choose to use your third-party syslog server(s). If you

choose to use vRealize Log Insight, it will always be available by pointing a browser to the configured IP address with the

username, admin. (If you ssh to Log Insight instead of pointing your browser to it, the username is root.) The password, in

either case, is the same password that you specified for VxRail Manager/VxRail vCenter Server (Row 30).

NOTE: The IP address for Log Insight must be on the same subnet as VxRail and vCenter Server.


Table

Rows 43 and 44 or

Row 45

Enter the hostname and IP address for vRealize Log Insight or the hostname(s) of your

existing third-party syslog server(s).

Step 1F. Workstation/Laptop

To access the VxRail for the first time, you must use the temporary VxRail initial IP address that was pre-configured, typically

192.168.10.200/24. You will change this IP address during VxRail initial configuration to your desired permanent address for

your new VxRail cluster.

Example

Configuration

VxRail Workstation/laptop

IP address/netmask IP address Subnet mask Gateway

Initial (temporary) 192.168.10.200/24 192.168.10.150 255.255.255.0 192.168.10.254

Post-configuration

(permanent) 10.10.10.100/24 10.10.10.150 255.255.255.0 10.10.10.254

Your workstation/laptop will need to be able to reach both the VxRail initial IP address (Row 2) and your selected permanent

VxRail IP address (Row 26). VxRail initial configuration will remind you that you may need to reconfigure your

workstation/laptop network settings to access the new IP address.

It may be possible to give your workstation/laptop or your jump server two IP addresses, which allows for a smoother

experience. Depending on your workstation/laptop, this can be implemented in several ways (such as dual-homing or multi-

homing). Otherwise, change the IP address on your workstation/laptop when instructed to and then return to VxRail Manager.

If you cannot reach the VxRail initial IP address, Dell EMC support team can configure a custom IP address, subnet mask, and

gateway.


Furthermore, if a custom management VLAN ID will be used for VxRail other than VLAN 1 (VLAN 1 is the default

management VLAN ID for most switches), make sure the workstation/laptop can also access this management

VLAN.


Table

Row 2

Please enter the VxRail initial IP address.

Enter 192.168.10.200/24 if you can reach this address on your network.

Otherwise, enter your custom IP address, subnet mask, and gateway.

Step 2: Set Up Switch

For VxRail to function properly, you must configure the ports that VxRail will use on your switch before you plug in VxRail

nodes and turn them on.

Set up your switch by following these steps:

Step 2A. Understanding switch configuration

Step 2B. Configure VLANs on your switch(es)

Step 2C. Confirm your configuration

Step 2A. Understanding Switch Configuration

Be sure to follow your switch vendor’s best practices for performance and availability. Ports on a switch operate in one of the

following modes:

Access mode – The port accepts only untagged packets and distributes the untagged packets to all VLANs on that port.

This is typically the default mode for all ports.

Trunk mode – When this port receives a tagged packet, it passes the packet to the VLAN specified in the tag. To

configure the acceptance of untagged packets on a trunk port, you must first configure a single VLAN as a “Native VLAN.”

A “Native VLAN” is when you configure one VLAN to use as the VLAN for all untagged traffic.

Tagged-access mode – The port accepts only tagged packets.

Network Traffic

For 13th generation PowerEdge Servers, each VxRail node will utilize either two 10GbE network ports or four 1GbE network

ports. Each port must be connected to a switch that supports IPv6 multicast, and for releases prior to VxRail 4.5.0, IPv4

multicast must also be supported.

For 14th generation PowerEdge Servers, each VxRail node will utilize the four 10GbE network ports. Each port must be

connected to a switch that supports IPv6 multicast.

Starting with Release 4.5.200, VxRail supports 25GbEnetwork ports on 14th generation PowerEdge Servers models E, P, V,

and S.

VxRail Appliances has four predefined network traffic types: management, vSphere vMotion, vSAN and Virtual Machine.

VxRail uses vSphere Network I/O Control (NIOC) to allocate and control network resources for these traffic types. The

respective NIOC settings for the predefined network traffic types are listed in the tables below for the various VxRail Models. 5

5 For a general overview on NIOC shares refer to http://frankdenneman.nl/2013/01/17/a-primer-on-network-io-control/.

http://frankdenneman.nl/2013/01/17/a-primer-on-network-io-control/


4x10GbE Traffic Configuration for VxRail 14th generation PowerEdge Servers

VxRail traffic on the E, P, S and V Series 10GbE NICs where all 4 NICs are utilized is separated as follows:

Traffic Type Requirements UPLINK1

(10Gb)

VMNIC0

UPLIINK2(10Gb)

VMNIC1

UPLINK3(10Gb)

VMNIC2

UPLINK4(10Gb)

VMNIC3

NIOC

Shares

Management IPv6 multicast Active Standby Unused Unused 40

vSphere vMotion Unused Unused Standby Active 50

vSAN IPv4 unicast Unused Unused Active Standby 100

Virtual Machines Standby Active Unused Unused 60

2x10GbE or 2x25GbE Traffic Configuration

VxRail traffic on the E, P, S and V Series 10GbE or 25 GbE NICs is separated as follows:

Traffic Type Requirements UPLINK1(10Gb or

25Gb)

VMNIC0

UPLINK2(10Gb or

25Gb)

VMNIC1

UPLINK3

No VMNIC

UPLINK4

No VMNIC

NIOC Shares


vSphere vMotion Active Standby Unused Unused 50

vSAN VxRail starting

with 4.5 (IPv4

unicast)

VxRail prior to

4.5 (IPv4

multicast)

Standby Active Unused Unused 100

Virtual Machines Active Standby Unused Unused 30

VxRail traffic on the G Series and models prior to VxRail 4.0 10GbE NICs is separated as follows:

Traffic Type Requirements 1st 10GbE NIC 2nd 10GbE NIC NIOC Shares

Management IPv6 multicast Active Standby 20

vSphere vMotion Active Standby 50

vSAN IPv4 multicast Standby Active 100

Virtual Machines Active Standby 30


1GbE Traffic Configuration

VxRail traffic on the G Series 1GbE NICs is separated as follows:

Traffic Type Requirements UPLINK1 (1Gb)

VMNIC0

UPLIINK2(1Gb)

VMNIC1

UPLINK3(1Gb)

VMNIC2

UPLINK4(1Gb)

VMNIC3

NIOC

Shares




with 4.5 (IPv4

unicast)

VxRail prior to

4.5 (IPv4

multicast)

Unused Unused Active Standby 100

Virtual Machines Standby Active Unused Unused 60

VxRail traffic on the E and S Series 1GbE NICs is separated as follows:

Traffic Type Requirements UPLINK1(1Gb)

VMNIC2

UPLINK2(1Gb)

VMNIC3

UPLINK3(1Gb)

VMNIC0

UPLINK4(1Gb)

VMNIC1

NIOC

Shares

Management IPv6 multicast Standby Active Unused Unused 40

vMotion Unused Unused Standby Active 50


with 4.5 (IPv4

unicast)

VxRail prior to 4.5

(IPv4 multicast)

Unused Unused Active Standby 100

Virtual

Machines

Active Standby Unused Unused 60

VxRail traffic on models prior to VxRail 4.0 1GbE NICs is separated as follows:

Traffic Type Requirements UPLINK1 (1Gb)

VMNIC0

UPLIINK2(1Gb)

VMNIC1

UPLINK3(1Gb)

VMNIC2

UPLINK4(1Gb)

VMNIC3

NIOC

Shares

Management IPv6 multicast Standby Active Unused Unused 40


vSAN IPv4 multicast Unused Unused Active Standby 100

Virtual Machines Active Standby Unused Unused 60


Multicast Traffic

VxRail Management VLAN

IPv6 multicast is required for the VxRail management VLAN. The network switch(es) that connect to VxRail must

allow for pass-through of multicast traffic on the VxRail Management VLAN. Multicast is not required on your

entire network, just on the ports connected to VxRail.

Why multicast? VxRail Appliances have no backplane, so communication between its nodes is facilitated via the network

switch. This communication between the nodes uses VMware Loudmouth auto-discovery capabilities, based on the RFC-

recognized "Zero Network Configuration" protocol. New VxRail nodes advertise themselves on a network using the VMware

Loudmouth service, which uses IPv6 multicast. This IPv6 multicast communication is strictly limited to the management VLAN

that the nodes use for communication.

VxRail creates very little traffic via IPv6 multicast for autodiscovery and management. It is recommended to limit traffic further

on your switch to enable MLD Snooping and MLD Querier.

If MLD Snooping is enabled, then MLD Querier must be enabled. If MLD Snooping is disabled, then MLD Querier

must be disabled.

vSAN VLAN (For Releases prior to VxRail 4.5.0)

Prior to VxRail 4.5.0, IPv4 multicast support is required for the vSAN VLAN. The network switch(es) that connect

to VxRail must allow for pass-through of multicast traffic on the vSAN VLAN. Multicast is not required on your

entire network, just on the ports connected to VxRail.

There are two options to handle vSAN IPv4 multicast traffic. Either limit multicast traffic by enabling both IGMP Snooping and

IGMP Querier or disable both of these features. We recommend enabling both IGMP Snooping and IGMP Querier if your

switch supports them.

IGMP Snooping software examines IGMP protocol messages within a VLAN to discover which interfaces are connected to

hosts or other devices interested in receiving this traffic. Using the interface information, IGMP Snooping can reduce

bandwidth consumption in a multi-access LAN environment to avoid flooding an entire VLAN. IGMP Snooping tracks ports that

are attached to multicast-capable routers to help manage IGMP membership report forwarding. It also responds to topology

change notifications. Disabling IGMP Snooping may lead to additional multicast traffic on your network.

IGMP Querier sends out IGMP group membership queries on a timed interval, retrieves IGMP membership reports from active

members, and allows updates to group membership tables. By default, most switches enable IGMP Snooping but disable

IGMP Querier. You will need to change the settings if this is the case.

If IGMP Snooping is enabled, then IGMP Querier must be enabled. If IGMP Snooping is disabled, then IGMP

Querier must be disabled.

If your switch does not support IGMP Snooping or MLD Snooping, VxRail multicast traffic will be broadcast in one

broadcast domain per VLAN. There is minimal impact on network overhead as management traffic is nominal.

For questions on your switch handles multicast traffic, contact your switch vendor.

Unicast Traffic (Starting in VxRail Release 4.5.0)

vSAN VLAN

Starting in VxRail Release 4.5.0, all vSAN traffic uses unicast. This change helps to reduce network configuration complexity

and simplifies switch configuration.

Inter-switch Communication

In a multi-switch environment, configure the ports used for inter-switch communication to carry IPv6 multicast traffic for the

VxRail management VLAN. Likewise, configure the ports to carry IPv4 traffic (unicast starting in VxRail Release 4.5.0 and


multicast in prior releases) between switches for the vSAN VLAN. Consult your switch manufacturer’s documentation for how

to do this.

Disable Link Aggregation

Do not use link aggregation, including protocols such as LACP and EtherChannel, on any ports directly connected to VxRail

Appliances. VxRail Appliances use active/standby configuration (NIC teaming) for network redundancy. However, LACP could

be enabled on non-system ports, such as additional NIC ports or 1G ports, for user traffic.

vSphere Security Recommendations

Security recommendations for vSphere should be followed can be found in General Network Security Recommendations

vSphere 6.5 or General Network Security Recommendations vSphere 6.0.

In particular, ensure that physical switch ports are configured with Portfast if spanning tree is enabled. Because VMware virtual

switches do not support STP, physical switch ports connected to an ESXi host must have Portfast configured if spanning tree

is enabled to avoid loops within the physical switch network. If Portfast is not set, potential performance and connectivity

issues might arise.

Step 2B. Configure VLANs on Your Switch(es)

Now that you understand the switch requirements, it is time to configure your switch(es). The VxRail network can be

configured with or without VLANs. For performance and scalability, it is highly recommended to configure VxRail with VLANs.

As listed in the VxRail Setup Checklist, you will be configuring the following VLANs:

Management VLAN (recommended is untagged/native): make sure that IPv6 multicast is configured/enabled on the

management VLAN.

vSAN VLAN: starting in VxRail 4.5.0, make sure that IPv4 unicast mode is used for vSAN traffic. For earlier releases,

make sure that IPv4 multicast is configured/enabled on the vSAN VLAN (enabling IGMP snooping and querier is highly

recommended).

vSphere vMotion VLAN

VM Networks VLANs

Figure 10. VxRail VLAN configuration, G Series.

Using the VxRail Network Configuration Table configure each switch port that will be connected to a VxRail node:

https://docs.vmware.com/en/VMware-vSphere/6.5/com.vmware.vsphere.security.doc/GUID-FA661AE0-C0B5-4522-951D-A3790DBE70B4.html




Configure the Management VLAN (Row 1) on the switch ports. If you entered “Native VLAN,” then set the ports on the switch

to accept untagged traffic and tag it to the custom management VLAN ID. Untagged management traffic is the recommended

management VLAN setting on VxRail.

You must set the management VLAN to allow IPv6 multicast traffic to pass through. Depending on the type of switch you have,

you may need to turn on IPv6 and multicast directly on the port or on the VLAN. Be sure to review the previous section, Step

2A. Understanding Switch Configuration, and consult the switch manufacturer for further instructions on how to configure

these settings.

Configure a vSphere vMotion VLAN (Row 34) on the switch ports.

Configure a vSAN VLAN (Row 38) on the switch ports for release prior to VxRail Release 4.5.0, set to allow IPv4 multicast

traffic to pass through. Starting in VxRail Release 4.5.0, set to allow IPv4 unicast traffic to pass through.

Configure the VLANs for your VM Networks (Rows 39-41) on the switch ports.

Step 2C. Confirm Your Configuration

Some network configuration errors cannot be recovered from, and you will need VxRail support to reset to factory defaults.

When VxRail is reset to factory defaults, all data is lost. Please confirm your switch setting in this step.

Read your vendor instructions for your switch:

a. Confirm that IPv4 multicast (VxRail release prior to 4.5.0) or unicast (VxRail Release 4.5.0 and beyond) and IPv6 multicast are enabled for the VLANs described in this document.

b. If you have two or more switches, confirm that IPv4 multicast/unicast and IPv6 multicast traffic is transported between them.

c. Remember that management traffic will be untagged on the native VLAN on your switch unless all ESXi hosts have been customized for a specific management VLAN.

Network design and accessibility:

a. Confirm that you can ping or point to the VxRail initial IP address (Row 2).

b. Confirm that your DNS server(s) are reachable unless you are in an isolated environment in VxRail releases prior to 4.5.0 (Row 5). The DNS server must be reachable from the VxRail, vCenter Server, and ESXi network addresses. Then update your DNS server with all VxRail hostnames and IP addresses.

c. Confirm that your management gateway IP address is accessible (Row 26).

d. If you have configured NTP servers, proxy servers, or a third-party syslog server, confirm that you can reach them from all of your configured VxRail IP addresses.

After Planning and Switch Setup

If you have successfully followed all of the previous steps, your network setup is complete, and you are ready to connect and

initialize your VxRail Appliance. These steps are done by Dell EMC service representatives. They are included here to help

you understand the complete process.

Step 1. Rack and cable the VxRail Appliance. After the nodes are cabled, power on all three or four initial nodes in your

VxRail cluster.

Do not turn on any other VxRail nodes until you have completed the full configuration of the first three or four

nodes.

Step 2. Connect a workstation/laptop to access the VxRail initial IP address on your selected management VLAN. It must

be either plugged into the switch or able to logically reach the VxRail management VLAN from elsewhere on your

network.

Step 3. Use the VxRail Pre-Engagement Questionnaire provided by the Dell EMC service representative to automatically

generate the JSON-formatted configuration file using the VxRail Network Configuration Table.

Step 4. Browse to the VxRail initial IP address (Row 2); for example, https://192.168.10.200.

Step 5. Click Get Started. Then if you agree, accept the VxRail End-User License Agreement (EULA).

Step 6. Click Configuration File to upload a JSON-formatted configuration file that you have created in Step 3.


Step 7. Click the Review First or Validate button. VxRail verifies the configuration data, checking for conflicts.

Step 8. After validation is successful, click the Build VxRail button.

Step 9. The new IP address for VxRail will be displayed.

Click Start Configuration. Ignore any browser messages about security (for example, by clicking “Advanced”

and “Proceed.”)

NOTE: You may need to manually change the IP settings on your workstation/laptop to be on the same subnet as

the new VxRail IP address (Row 26).

NOTE: If your workstation/laptop cannot connect to the new IP address that you configured, you will get a

message to fix your network and try again. If you are unable to connect to the new IP address after 20

minutes, VxRail will revert to its un-configured state and you will need to re-enter your configuration at the

initial IP address (Row 2).

NOTE: After the build process starts, if you close your browser, you will need to browse to the new IP address

(Row 26).

Step 10. Progress is shown as VxRail is built. VxRail implements services, creates the new ESXi hosts, and sets up vCenter

Server, vMotion, and vSAN.

When you see the Hooray! page, VxRail is built. Click the Manage VxRail button to continue to VxRail

management. You should also bookmark this IP address in your browser for future use.

Step 11. Configure your corporate DNS server for all VxRail hostnames and IP addresses unless you are in an isolated

environment in VxRail releases prior to 4.5.0.

Step 12. Connect to VxRail Manager using either the VxRail Manager IP address (Row 26) or the fully-qualified domain

name (FQDN) (Row 25) that you configured on your DNS server (e.g., https://vxrail.yourcompany.com).

Unassigned Physical Ports

For VxRail models based on Dell PowerEdge servers, VxRail Manager will not manage the optional ports on the PCI-e NIC.

Customers can configure the additional ports in vCenter for non-VxRail system traffics, such as VM networks, iSCSI, NFS, etc.

The supported operations include:

o Create a new vSphere Standard Switch(VSS), and connect unused ports to the VSS.

o Connect unused ports to the default vSphere Distributed Switch.

o Create a new vSphere Distributed Switch(VDS), add VxRail nodes to the new VDS, and connect their unused

network ports to the VDS.

o Create new VMKernel adapters and enable services of IP Storage and vSphere Replication.

o Create new VM Networks.

o Starting with VxRail 4.0.200, renaming the default VDS.

o Starting with VxRail 4.0.200, renaming the default port group.

Customers need to follow VMware’s official instructions/procedures for the above operations.

NOTE: Do NOT move VxRail system traffic to these ports. VxRail system traffic includes the management, vSAN,

vCenter Server and vMotion Networks.

Unsupported Operations:

o Migrating VxRail system traffic to other port groups.


Network Segregation

Some customers may want to separate VM networks and vSphere management network. They can leverage those unused

ports to enforce network segregation. Please be sure to work with your Dell EMC implementation and support teams to ensure

these additional ports are cabled and set-up in the appropriate order as prescribed by Dell EMC.


VxRail Network Configuration Table

The Dell EMC service representative will use a VxRail Pre-Site Installation tool with the following information:

Row Category Description

1 VxRail Management

VLAN ID

The recommended is untagged traffic on the Native VLAN. If you want the host to send only

tagged frames, manually configure the VLAN on each ESXi™ host using DCUI, and set

tagging for your management VLAN on your switch before you deploy VxRail.

2 VxRail initial IP If you cannot reach the default (192.168.10.200/24), set an alternate IP address

3 System Global settings Time zone

4 NTP server(s)

5 DNS server(s)

6 Proxy settings IP address and port

7 Username and password

8 Management ESXi

hostnames and

IP addresses

ESXi hostname prefix

9 Separator

10 Iterator

11 Offset

12 Suffix

13 Domain

14 ESXi starting address for IP pool

15 ESXi ending address for IP pool

16 vCenter Server

Leave blank if

Customer

Supplied VC

vCenter Server hostname

17 vCenter Server IP address

18 Platform Services Controller hostname

19 Platform Services Controller IP address

20 Customer

Supplied

vCenter Server

Leave blank if

VxRail VC

Customer Supplied Platform Services Controller (PSC) Hostname (FQDN)

Leave blank if PSC is VxRail PSC

21 Customer Supplied vCenter Server hostname (FQDN)

22 Customer Supplied vCenter Server SSO domain

23 admin username/password or the newly created VxRail non-admin username and password

24 New VxRail management username and password

25 Customer Supplied datacenter name

26 New cluster name

27 VxRail Manager VxRail hostname

28 VxRail IP address

29 Networking Subnet mask

30 Gateway

31 Passwords ESXI “root” passwords. Can be different for each host starting with Release 4.0.100.

32 VxRail Manager and VxRail vCenter Server “[email protected]”

33 vMotion Starting address for IP pool

34 Ending address for IP pool

35 Subnet mask

36 VLAN ID

37 vSAN Starting address for IP pool

38 Ending address for IP pool

39 Subnet mask

40 VLAN ID

41 VM

Networks

… (unlimited

number)

VM Network name and VLAN ID

42 VM Network name and VLAN ID

43 Solutions Logging vRealize Log Insight™ hostname

44 vRealize Log Insight IP address

45 Syslog server (instead of Log Insight)


VxRail Setup Checklist

Physical Network

VxRail cluster: Decide if you want to plan for additional nodes beyond the initial three (or four)-node cluster. You can have up to 64 nodes in a VxRail cluster

Network switch: Ensure your switch provides the connectivity option you chose in the Network Switch section. Check cable requirements.

Topology: Decide if you will have a single or multiple switch setup for redundancy.

Workstation/laptop: Any operating system with a browser to access the VxRail user interface. The latest versions of Firefox, Chrome, and Internet Explorer 10+ are all supported.

Out-of-band Management (optional): One available port that supports 100Mbps for each VxRail node.

Logical Network

Reserve

VLANs

One management VLAN with IPv6 multicast for traffic from VxRail, vCenter Server, ESXi (recommended is untagged/native).

One VLAN with IPv4 unicast (starting with VxRail 4.5.0) or IPv4 multicast (prior to Release 4.5.0) for vSAN traffic.

One VLAN for vSphere vMotion.

One or more VLANs for your VM Network(s).

System

Time zone.

Hostname or IP address of the NTP server(s) on your network (recommended).

IP address of the DNS server(s) on your network (required, except in isolated environments).

Optional: IP address, port, username, and password of your proxy server.

Management

Decide on your ESXi host naming scheme.

Reserve three or more contiguous IP addresses for ESXi hosts.

Decide if you will use a vCenter Server that is Customer Supplied or new to your VxRail cluster.

VxRail vCenter Server: Decide on hostnames for vCenter Server and PSC and reserve two IP addresses.

Customer Supplied vCenter Server: Determine PSC, hostname, administration user, and datacenter. Create a VxRail management user. Decide on a VxRail cluster name. (Optional) Create a VxRail non-admin user.

Decide on a hostname and reserve one IP address for VxRail Manager.

Determine IP address of the default gateway and subnet mask.

Select a root password for each ESXi hosts in the VxRail cluster. Prior to VxRail Release 4.0.100, all ESXi hosts use a single root password.

Select a single password for VxRail and VxRail vCenter Server.

vMotion and

vSAN

Reserve three or more contiguous IP addresses and a subnet mask for vSphere vMotion.

Reserve three or more contiguous IP addresses and a subnet mask for vSAN.

Solutions To use vRealize Log Insight: Reserve one IP address and decide on the hostname.

To use an existing syslog server: Get the hostname or IP address of your third-party syslog server.

Workstation Configure your workstation/laptop to reach the VxRail initial IP address.

Make sure you also know how to configure it to reach the VxRail Manger IP address after configuration.

Set up Switch

Configure your selected management VLAN (recommended is untagged/native). Confirm that IPv6 multicast is configured/enabled on the management VLAN.

Configure your selected VLANs for vSAN, vSphere vMotion, and VM Networks.

In multi-switch environments, configure the management and vSAN VLANs to carry the multicast and unicast traffic respectively between switches.

Confirm configuration and network access.


Appendix A: NSX Support on VxRail

VxRail supports VMware NSX software-defined networking (SDN) through vCenter Server. vCenter Server offers a fully

integrated option for SDN and network-layer abstraction with NSX. The NSX network-virtualization platform delivers for

networking what VMware delivers for compute and storage. In much the same way that server virtualization allows operators

to programmatically create, snapshot, delete, and restore software-based virtual machines (VMs) on demand, NSX enables

virtual networks to be created, saved, deleted, and restored on demand without requiring reconfiguration of the physical

network. The result fundamentally transforms the datacenter network-operational model, reduces network-provisioning time

from days or weeks to minutes, and dramatically simplifies network operations. NSX is a non-disruptive solution that is

deployed on any IP network, including existing datacenter network designs or next-generation fabric architectures from any

networking vendor.

With network virtualization, the functional equivalent of a “network hypervisor” reproduces the complete set of Layer 2 to Layer

7 networking services (e.g., switching, routing, access control, firewalling, QoS, and load balancing) in software. Just as VMs

are independent of the underlying x86 hardware platform and allow IT to treat physical hosts as a pool of compute capacity,

virtual networks are independent of the underlying IP network hardware and allow IT to treat the physical network as a pool of

transport capacity that can be consumed and repurposed on demand.

NSX coordinates ESXi’s vSwitches and the network services pushed to them for connected VMs to effectively deliver a

platform—or “network hypervisor”—for the creation of virtual networks. Similar to the way that a virtual machine is a software

container that presents logical compute services to an application, a virtual network is a software container that presents

logical network services—logical switches, logical routers, logical firewalls, logical load balancers, logical VPNs and more—to

connected workloads. These network and security services are delivered in software and require only IP packet forwarding

from the underlying physical network.

To connected workloads, a virtual network looks and operates like a traditional physical network. Workloads “see” the same

Layer 2, Layer 3, and Layers 4-7 network services that they would in a traditional physical configuration. It’s just that these

network services are now logical instances of distributed software modules running in the hypervisor on the local host and

applied at the vSwitch virtual interface.

The following NSX components are illustrated in Figure 11:

NSX vSwitch operates in ESXi server hypervisors to form a software abstraction layer between servers and the physical

network.

NSX Controller is an advanced, distributed state management system that controls virtual networks and overlays transport

tunnels. It is the central control point for all logical switches within a network and maintains information of all virtual

machines, hosts, logical switches, and VXLANs.

NSX Edge provides network-edge security and gateway services to isolate a virtualized network. You can install NSX

Edge either as a logical (distributed) router or as a services gateway.

NSX Manager is the centralized network management component of NSX, installed as a virtual appliance on an ESXi

host.


Figure 11. NSX component information flow: NSX Manager, NSX Controller, NSX Edge, NSX vSwitch

One NSX Manager maps to a single vCenter Server and multiple NSX Edge, vShield Endpoint, and NSX Data Security

instances. Before you install NSX in your vCenter Server environment, consider your network configuration and resources

using the chart below.


NSX Resource Requirements:

Memory Disk Space vCPU

NSX Manager 12GB 60GB 4

NSX Edge:

Compact

Large

Extra Large

Quad Large

512MB

1GB

8GB

1GB

512MB

512MB

4.5GB (with 4GB swap)

512MB

1

2

6

4

vShield Endpoint 1GB 4GB 2

NSX Data Security 512MB 6GB per ESXi host 1

In a VxRail cluster, the key benefits of NSX are consistent, simplified network management and operations, plus the ability to

leverage connected workload mobility and placement. With NSX, connected workloads can freely move across subnets and

availability zones. Their placement is not dependent on the physical topology and availability of physical network services in a

given location. Everything a VM needs from a networking perspective is provided by NSX, wherever it resides physically. It is

no longer necessary to over-provision server capacity within each application/network pod. Instead, organizations can take

advantage of available resources wherever they’re located, thereby allowing greater optimization and consolidation of

resources. VxRail easily inserts into existing NSX environments and provide NSX awareness so network administrators can

leverage simplified network administration. See the VMware NSX Design Guide for NSX best practices and design

considerations.

For additional information related to NSX, refer to the following materials:

VMware NSX Network Virtualization Platform Technical White Paper at http://www.vmware.com/files/pdf/products/nsx/VMware-NSX-Network-Virtualization-Platform-WP.pdf

Reference Design Guide: VMware NSX for vSphere at https://www.vmware.com/files/pdf/products/nsx/vmw-nsx-network-virtualization-design-guide.pdf

http://www.vmware.com/files/pdf/products/nsx/VMware-NSX-Network-Virtualization-Platform-WP.pdf

http://www.vmware.com/files/pdf/products/nsx/VMware-NSX-Network-Virtualization-Platform-WP.pdf

https://www.vmware.com/files/pdf/products/nsx/vmw-nsx-network-virtualization-design-guide.pdf

https://www.vmware.com/files/pdf/products/nsx/vmw-nsx-network-virtualization-design-guide.pdf


Appendix B: VxRail Open Ports Requirement

The following tables list the ports that set during VxRail initialization. Please consult with your Dell EMC or reseller rep for

more information.

Description Source Device(s) Destination Device(s) L4

Protocol Port

DNS VMware vCenter Servers,

Platforms Services Controllers,

ESRS Gateway VMs,

Host ESXi Management Interface,

NSX Managers,

Dell iDRAC port,

ToR Switch Management

DNS Server(s) UDP 53

ESX VMWare Agent Manager for

VIB installation (NSX)

Host ESXi Management Interface VMware vCenter Servers TCP 80

ESXi host modules to NSX

Controllers

Host ESXi Management Interface NSX Controllers TCP 1234

Host ESXi Management Administrators Host ESXi Management

Interface

TCP 427, 902

Host ESXi Management Administrators Host ESXi Management

Interface

UDP 902

Host ESXi Management Host ESXi Management Interface Administrators TCP &

UDP

902

GUI/Web Interfaces Administrators VMware vCenter Servers,

NSX Managers,

Host ESXi Management

Interface,

Dell iDRAC port,

vRealize LogInsight,

VxRail Manager

TCP 80, 443

VMware KVM Administrators Host ESXi Management

Interface

TCP 2068, 80

LDAP VMware Platform Service Controller

vApps,


VMware vCenter Servers,

Dell iDRAC port,

Host ESXi Management Interface

LDAP Server TCP 389, 636

Managed Hosts to vCenter Host ESXi Management Interface VMware vCenter Servers TCP 6501, 6502,

5989, 6500,

8000, 8001,

5988, 5989,

5353, 902,

443

Managed Hosts to vCenter

(Heartbeat)

Host ESXi Management Interface VMware vCenter Servers UDP 902



Protocol Port

NTP Client Host ESXi Management Interface,

NSX Managers,

NSX Controllers,

Dell iDRAC port,

vRealize LogInsight,

VMware vCenter Servers,

VxRail Manager,


NTP Server(s) UDP 123

Radius Radius Client IPs Radius Server TCP 1645, 1646,

1812, 1813

RDP Administrators VMware vCenter Servers,

VMware Platform Service

Controller vApps

TCP 3389

SMTP Dell iDRAC port, ESRS Gateway

VMs, vRealize LogInsight, ESRS

Policy Manager,

Cisco ToR Switch Management

SMTP Server (s) TCP 25

SNMP Polling SNMP Server (s) Host ESXi Management

Interface,

Dell iDRAC port,


UDP 161

SNMP Traps Host ESXi Management Interface,

Dell iDRAC port,


SNMP Servers UDP 162

NSX Manager to Controllers NSX Managers NSX Controllers TCP 443

NSX Manager to hosts,

management connection

NSX Managers Host ESXi Management

Interface

TCP 443, 902

NSX Manager to hosts,

management connection

NSX Managers NSX vCenter Servers TCP 443, 902

NSX User World Agent connection NSX Controllers Host ESXi Management

Interface

TCP 8672

NSX vib Install (Only if VDR

instances are associated with the

host)

NSX DLR Host ESXi Management

Interface

UDP 6999

NSX Manager Messaging &

Distributed Firewall Comms

Host ESXi Management Interface NSX Managers TCP 80, 5671

SSH & SCP Administrators Host ESXi Management

Interface,

ESRS Gateway VMs,

NSX Controllers,

NSX Managers,

Dell iDRAC port,

vRealize Log Insight,


TCP 22



Protocol Port

SYSLOG NSX Managers,

NSX vCenter Servers,

NSX Controllers,


vRealize Log Insight,


Syslog Server TCP 514

System port for SSHD

vCenter Server Appliance only

VMware vCenter Servers Administrators TCP/UDP 22

TACACS Clients Customer Preference TACACS Server TCP 49

vCenter to Managed Hosts (CIM

XMLm DPM, VC Agent,Data)

VMware vCenter Servers Host ESXi Management

Interface

TCP 80, 443,

5989, 902

vCenter to Managed Hosts (DPM

& Heartbeat)

VMware vCenter Servers Host ESXi Management

Interface

UDP 623, 902

Layer 3 vMotion Host ESXi Management Interface Host ESXi Management

Interface

TCP 8000

vSphere Clients to vCenter Server Administrators VMware vCenter Servers TCP 80, 443,

9443, 10080,

10443, 8443,

8080, 5480

vRealize Operations vRealize Operations Manager VMware vCenter Servers TCP 10443

VC6.0 to Active Directory Server VMware vCenter Servers Active Directory Server TCP 88

VxRail to ESRS Gateway VMs VxRail Manager ESRS Gateway VMs TCP 21, 25, 9443

ESRS GW to VxRail Manager ESRS Gateway VMs VxRail Manager TCP 21, 22, 5400-

5413, 25,

443, 8118,

9443

ESRS Policy Manager GUI Administrators ESRS Policy Manager TCP 8090, 8443

(Configurable)

ESRS V3 FTPS ESRS Gateway VMs ESRS V3 FTPS Servers:

Public Internet IP addresses

TCP 990

ESRS V3 ESRS Gateway VMs ESRS V3 Global Access

Servers: Public Internet IP

addresses

TCP 443, 8443

ESRS Management Administrators ESRS Gateway VMs TCP 22, 9443

ESRS Gateway VMs to VxRail

Manager

ESRS Gateway VMs VxRail Manager TCP 21, 22, 5400-

5413, 25,

443, 8118,

9443



Protocol Port

iDRAC Access Administrators Dell iDRAC TCP 22, 80, 443,

623, 5900,

5901

VxRail Internet Access VxRail Manager Proxy Server TCP 8080

VSAN Stretched Cluster VSAN Witness VxRail Node VSAN Interface UDP 12345,

23451, 12321

VSAN Stretched Cluster VSAN Witness VxRail Node VSAN Interface TCP 2233

VSAN Stretched Cluster VSAN Witness vCenter Server TCP 8080

VSAN Stretched Cluster VxRail Node VSAN Interface VSAN Witness UDP 12345,

23451, 12321

VSAN Stretched Cluster VxRail Node VSAN Interface VSAN Witness TCP 2233

VSAN Stretched Cluster Host ESXi Management Interface VMware vCenter Servers TCP 8080

VSAN Stretched Cluster VMware vCenter Servers VSAN Witness TCP 8080

VSAN Stretched Cluster VMware vCenter Servers Host ESXi Management

Interface

TCP 8080

References:

VMware vSphere Ports – TCP and UDP Ports required to access VMware vCenter Server, VMware ESXi and ESX hosts,

and other network components (VMware KB 1012382)

VMware vSAN – Configuring Virtual SAN Network

ESRS Port Requirements – EMC Secure Remote Service Release 3.24 Rev 01 – Port Requirements

ESRS Public IP Addresses – What IP addresses are used by the EMC Secure Remote Services (EMC KB 494729)

iDRAC - Integrated Dell Remote Access Controller 9 - iDRAC7 Port Information

https://kb.vmware.com/s/article/1012382

https://kb.vmware.com/s/article/1012382

https://docs.vmware.com/en/VMware-vSphere/6.0/com.vmware.vsphere.virtualsan.doc/GUID-D52F00FF-CA2C-4DDD-B76B-B8BF211BB0E8.html

https://www.emc.com/collateral/TechnicalDocument/docu55301.pdf

https://support.emc.com/kb/494729

http://topics-cdn.dell.com/pdf/iDRAC9-lifecycle-controller-v3.11.11.11_User's%20Guide_en-us.pdf


Appendix C: VxRail Firewall Rules & ACI Contracts Diagram

This diagram shows a typical VxRail deployment topology, in which the major ports are displayed for their respective usage

between VxRail components.


Appendix D: Physical Network Switch Examples

These diagrams show different physical network switch wiring examples. They are provided as illustrative examples.

Figure 12. Rear view of VxRail Appliance connected to 1x10GbE SFP+ ToR switch with no iDRAC


Figure 13. Rear view of VxRail Appliance connected to 1x(1GbeE + 10 GbE) SPF+ ToR switch with

iDRAC

Figure 14. Rear view of VxRail Appliance connected to 2x10GbE SPF+ ToR switches with no iDRAC


Figure 15. Rear view of VxRail Appliance connected to 2x(1Gbe + 10GbE) SPF+ ToR switches with iDRAC

© 2018 Dell Inc. or its subsidiaries. All Rights Reserved. Dell, EMC and other trademarks are

trademarks of Dell Inc. or its subsidiaries. Other trademarks may be trademarks of their respective

owners. Reference Number: H15300.6

Learn more about Dell

EMC VxRail

Appliances

Contact a Dell EMC Expert View more resources Join the conversation

@DellEMC_CI

with #VxRail

http://www.dellemc.com/vxrail

http://www.emc.com/contact/contact-us.htm

https://store.emc.com/Product-Family/VxRail-Products/Dell-EMC-VxRail-Appliance/p/VCE-VxRail?PID=EMC_SRS-VXRAIL-DC90_HERO

DELL EMC VxRAIL™ NETWORK GUIDE · DELL EMC VxRAIL™ NETWORK GUIDE Physical and Logical Network...

Documents

Transcript of DELL EMC VxRAIL™ NETWORK GUIDE · DELL EMC VxRAIL™ NETWORK GUIDE Physical and Logical Network...