Sr Iov Guide
Transcript of Sr Iov Guide
-
8/12/2019 Sr Iov Guide
1/8
SR-IOV
Version 3.0
Technology Preview Guide
Emulex Connects
P006796-01B Rev. A Servers, Storage and People
-
8/12/2019 Sr Iov Guide
2/8
SR-IOV Technology Preview Guide
Copyright 201-20121 Emulex. All rights reserved worldwide. No part of this document may be reproduced by any
means or translated to any electronic medium without the prior written consent of Emulex.
Information furnished by Emulex is believed to be accurate and reliable. However, no responsibility is assumed by
Emulex for its use; or for any infringements of patents or other rights of third parties which may result from its use.
No license is granted by implication or otherwise under any patent, copyright or related rights of Emulex.
Emulex, the Emulex logo, AutoPilot Installer, AutoPilot Manager, BlockGuard, Connectivity Continuum,
Convergenomics, Emulex Connect, Emulex Secure, EZPilot, FibreSpy, HBAnyware, InSpeed, LightPulse,
MultiPulse, OneCommand, OneConnect, One Network. One Company., SBOD, SLI, and VEngine are trademarks of
Emulex. All other brand or product names referenced herein are trademarks or registered trademarks of their
respective companies or organizations.
Emulex provides this manual "as is" without any warranty of any kind, either expressed or implied, including but not
limited to the implied warranties of merchantability or fitness for a particular purpose. Emulex may make
improvements and changes to the product described in this manual at any time and without any notice. Emulex
assumes no responsibility for its use, nor for any infringements of patents or other rights of third parties that may
result. Periodic changes are made to information contained herein; although these changes will be incorporated into
new editions of this manual, Emulex disclaims any undertaking to give notice of such changes.
Emulex, 3333 Susan Street
Costa Mesa, CA 92626
-
8/12/2019 Sr Iov Guide
3/8
SR-IOV Technology Preview User Guide Page iii
Introduction ........................... ........................... .......................... ............................ ................................ . 1
SR-IOV Advantages ....................... ........................... ............................ ......................... .................. 1
Requirements ....................... ............................ ......................... ............................ ........................... 1
Setting Up SR-IOV .......................... ............................ ........................... .......................... ................... 2
Configuration ......................... ........................... .......................... ........................... .............................. ... 4
MAC Address for Virtual Function Interfaces......................... ........................... ............................ ....... 4
Configuring the MAC Address for the VF.... .......................... ........................... ............................ ..... 4
Configuring VLAN on VF Interfaces Using vconfig..................... ......................... ......................... ..... 4
Transparent VLAN Tagging ........................ .......................... ........................... ........................... 4
Virtual Guest Tagging (VGT) ........................... ............................ ......................... ...................... 5
Configuring Rate Limit on VF Interfaces.................... ............................ ......................... ..................... 5
http://-/?-http://-/?- -
8/12/2019 Sr Iov Guide
4/8
SR-IOV Technology Preview Guide Page 1
Introduction
Emulexoffers Single Root I/O Virtualization (SR-IOV) functionality as a Technology Preview feature.
This guide contains requirements and steps to use SR-IOV in a RHEL 6 K Virtual Machine (KVM)environment. This environment supports capabilities of OneConnect to enable multiple virtual
functions (VFs) for a physical function (PF). Each of these VFs can be assigned to virtual machines
(VMs). A VF enables the guest operating system direct access to OneConnect, such that guest
performance is not limited by the overheads of the hypervisor.
It is important to remember that a Technology Preview feature is not to be used in production
environments and may not be functionally complete. Emulex cannot provide standard support levels
(including rapid response, fixes, troubleshooting, and workarounds). However, Emulex is very interested
in your feedback and early experiences with this feature.
SR-IOV Advantages
Direct I/O throughput Reduced CPU utilization
Increased scalabil ity
Improved device performance for virtual guests
Requirements
To set up SR-IOV on your system, the following is required:
A server/blade with an SR-IOV-capable motherboard BIOS.
The server/blade must be supported by one of the following operating systems, whichare supported as hypervisor and VMs:
RHEL 6.1 (64-bit)
RHEL 6.0 (64-bit)
RHEL 5.6
SLES 11 SP1
OneConnect has been tested with a limited set of servers based on Intel CPU.
A OneConnect OCx11102-xT or higher Universal Converged Network Adapter (UCNA).
Note: Currently, SR-IOV is supported only in non-multichannel setups.
Note: Configuration mechanisms for parameters such as MAC address, VLAN and QoS for
VF are supported only in RHEL6 and RHEL 6.1 distributions.
-
8/12/2019 Sr Iov Guide
5/8
SR-IOV Technology Preview Guide Page 2
Setting Up SR-IOV
1. Depending on your system, perform one or more of the following tasks to set up your BIOS. (For
more information, refer to the manufacturers instructions for your system.)
a. Enable SR-IOV in the system BIOS. This supports SR-IOV functionality.
b. Enable Intel Virtualization Technology support for Direct I/O VT-d. This enables support
of VMs.
2. To enable SR-IOV in the OneConnect firmware using the PXE Select menu, enter the PXE
Select utility menu by pressing +
when the PXE BIOS banner appears in during
system boot.
Figure 1: PXESelect Utility, SR-IOV enabled
3. Install Linux RHEL6.0 or 6.1.
4. Update the /boot/grub/menu.lst file to include the following command line load parameter for theLinux kernel:
intel_iommu=on iommu=pt
5. Install an Emulex driver for Linux that supports SR-IOV.
6. Reboot the server for the new firmware to become operational.
7. Use the "lspci -vvv" output command to check if SR-IOV is properly enabled:
lspci -vvv
This command returns an SR-IOV capability section for each OneConnect NIC PF, and the total/
initial VFs should be non-zero. Make a note of the lspci command output in the host. This output
is needed in step 12.
Example of the SR-IOV capabilities section output of the PF with SR-IOV enabled:
Capabilities: [180] Single Root I/O Virtualization (SR-IOV)
IOVCap: Migration-, Interrupt Message Number: 000
IOVCtl: Enable- Migration- Interrupt- MSE- ARIHierarchy-
IOVSta: Migration-
Initial VFs: 16, Total VFs: 16, Number of VFs: 0, Function Dependency
Link: 00
VF offset: 0, stride: 1, Device ID: 0710
Supported Page Size: 00000557, System Page Size: 00000001
-
8/12/2019 Sr Iov Guide
6/8
SR-IOV Technology Preview Guide Page 3
Region 0: Memory at 0000000000000000 (64-bit, non-prefetchable)
VF Migration: offset: 00000000, BIR: 0
Example of the SR-IOV capabilities section output of the PF with SR-IOV disabled:
Capabilities: [180] Single Root I/O Virtualization (SR-IOV)
IOVCap: Migration-, Interrupt Message Number: 000
IOVCtl: Enable- Migration- Interrupt- MSE- ARIHierarchy-
IOVSta: Migration-
Initial VFs: 0, Total VFs: 0, Number of VFs: 0, Function Dependency
Link: 00
VF offset: 0, stride: 1, Device ID: 0710
Supported Page Size: 00000557, System Page Size: 00000001
Region 0: Memory at 0000000000000000 (64-bit, non-prefetchable)
VF Migration: offset: 00000000, BIR: 0
8. Create a virtual machine using the Virtual Machine Manager utility and install the RHEL6
operating system on the VM.
9. Boot into the VM and install version 3.0 of the OneConnect software installation package or
installation CD.
10. Shut down the VM.11. Reload the newly installed NIC driver in the host (hypervisor) with the number of virtual PCI
functions (VFs) / physical functions (PFs) as module parameter:
rmmod be2net
modprobe be2net num_vfs=X
whereXis the number of VF per PF.
12. Rerun the lspci command and note that X (as specified in step 11) is the number of virtual NIC
functions that have been added per PF in the output of lspci from step 7.
View the PCI bus:device:function details for the VFs that you want to assign to the VM by using
the lspci command, then detach these VFs from the host. For example, if the lspci output has an
entry similar to:
04:04.0 Ethernet controller:
Emulex Corp.Emulex OneConnect 10Gb NIC (be3) (rev 01)
use the following command:
virsh nodedev-dettach pci_0000_04_04_0
13. Use the Virtual Machine Manager GUI to attach the VF (from step 11) to the guest operating
system by using the add physical device option.
14. Start the RHEL6 guest operating system. Once the guest operating system is booted, use the
lspci command to confirm the visibility of the OneConnect NIC function. The output will show an
OneConnect NIC function, for example:
03:00.0 Ethernet controller: Emulex Corp. Emulex OneConnect 10Gb NI (be3)
15. The OneConnect NIC driver automatically loads and creates the network interfaces. Use the
ifconfig command to verify that the interface is created.
16. After configuring the network interfaces with proper IP addresses, you can send and receive
network traffic from the VM.
Note: The maximum number of virtual functions per physical function is 16.
-
8/12/2019 Sr Iov Guide
7/8
SR-IOV Technology Preview Guide Page 4
Configuration
MAC Address for Virtual Function Interfaces
The Universal Converged Network Adapter (UCNA) is shipped with factory configured MAC addresses
for the network interfaces corresponding to the Physical Functions (PFs). The driver generates random
MAC addresses for the network interfaces corresponding to the Virtual Functions (VFs) using the factory
configured MAC address as seed. Other MAC addresses can be assigned for the interfaces
corresponding to the VF using the IP utility.
Configuring the MAC Address for the VF
To configure the MAC address for the VF, type:
# ip link set eth vf mac
where:
eth is the interface corresponding to the physical function.
is the VF number (0 based) corresponding to the interface for which you are configuring
the MAC address.
is the MAC address you are configuring.
Example:
# ip link set eth0 vf 0 mac 00:16:88:AA:BB:AA
Configuring VLAN on VF Interfaces Using vconfig
Transparent VLAN Tagging
When transparent tagging is configured for a VF, the NIC transparently tags all traffic from the VF with
the configured VLAN ID. The VM is not aware of the VLAN tag.
To assign a VLAN ID to the VF, run the following IP command in the host:
# ip link set eth vf vlan
where:
eth is the interface corresponding to the physical function.
is the VF number (0 based) corresponding to the interface for which you are configuring
the VLAN.
is the VLAN ID you are configuring.
Example:
# ip link set eth0 vf 0 vlan 5
-
8/12/2019 Sr Iov Guide
8/8
SR-IOV Technology Preview Guide Page 5
Virtual Guest Tagging (VGT)
When VGT is configured, the guest operating system can configure any VLAN ID on the VF. To configure
VGT, run the IP link command with the VLAN ID of 4095. For example:
# ip link set eth0 vf 0 vlan 4095
Configuring Rate Limit on VF Interfaces
Configure the rate limit on a VF interface from the hypervisor using the ip command:
# ip link set eth vf rate
where:
eth is the interface corresponding to the physical function.
is the VF number (0 based) corresponding to the interface for which you are configuring the TX-
rate.
is the transmission rate in Mb/s that you are configuring. The TX-rate value is interpreted as
a percent of the maximum link speed (currently 10 Gb/s) irrespective of current PF QoS (Quality of
Service). The TX-rate value is not interpreted as an absolute QoS value.
For example, to set a transmit bandwidth rate limit of 5000 Mb/s for the VF 0, run the command:
# ip link set eth0 vf 0 rate 5000
The default rate for a VF is 100Mbps, which is the lowest rate you can program per function in
OneConnect.
Note: Only one VLAN ID per VF is currently supported.
Note: In the above example, the Tx rate value indicates that 50 percent of the PF QoS is
assigned to a VF.