Mellanox BlueField BMC Software User Manual · BMC software enables control and management of the...

Mellanox Technologies Confidentialwww.mellanox.com

Mellanox BlueField BMC Software User Manual

Rev 1.0Software Version 6.0.0200-64 (beta)

Rev 1.0Doc #: MLNX-15-52010 2Mellanox Technologies Confidential

Table of Contents

Table of Contents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3List of Tables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5List of Figures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6Document Revision History . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7About this Manual . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8Chapter 1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9Chapter 2 Connecting to BMC Interfaces. . . . . . . . . . . . . . . . . . . . . . . . . . . . 10

2.1 BMC Console Interface. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 102.2 BMC Management Interface Port . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 102.3 BMC USB Port . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12

2.3.1 Provides Removable Storage via USB Stick . . . . . . . . . . . . . . . . . . . . . . . . . 122.3.2 Provides Network Connection from BMC to BlueField SoC. . . . . . . . . . . . 122.3.3 Pushing Boot-stream from BMC to BlueField Arm . . . . . . . . . . . . . . . . . . . 122.3.4 Access of BlueField RShim Console from BMC . . . . . . . . . . . . . . . . . . . . . . 13

Chapter 3 System Management. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 143.1 System EEPROMs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 143.2 Management Interface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 143.3 Serial Over LAN (SOL) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 143.4 Serial Console Capture (SCC) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15

3.4.1 Setting Up Serial Capture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 153.4.2 Reading FRU Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 173.4.3 History Buffer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17

3.5 NC-SI Interface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 183.6 Chassis Management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19

3.6.1 System Fan Management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 193.6.2 FRU Reading. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 193.6.3 Sensor Database Rate (SDR) Support . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 203.6.4 IPMI Stored Information. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 213.6.5 Rebooting BlueField with BMC. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 223.6.6 Shutting Down BlueField with BMC. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 223.6.7 Turning on BlueField with BMC . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22

Chapter 4 BMC Management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 234.1 System Components. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23

4.1.1 Software Versioning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 234.1.2 Boot Sequence Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23

3Mellanox Technologies Confidential

4.2 User Management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 234.3 BMC Image Upgrade. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 244.4 CPLD Upgrade . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24

Chapter 5 Mellanox Custom Commands . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25Chapter 6 List of Supported IPMItool Commands . . . . . . . . . . . . . . . . . . . . 26



List of Tables

Table 1: Related Documentation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .8Table 2: Glossary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .8Table 3: BMC to BlueField . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .12Table 4: Available SDR Information . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .20Table 5: Event Log Retrieval Commands . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .21Table 6: Sensor Data Retrieval Commands . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .22Table 7: Custom IPMI Tool Commands. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .25


List of Figures

Figure 1: BMC Console Interface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .10Figure 2: BMC Management Interface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .11

Document Revision History

Rev 1.0 – August 08, 2018First release


About this ManualThis manual provides general information concerning the BMC on the BlueField Reference Plat-form.

Intended AudienceThis manual is intended for those who want to familiarize themselves with the functionality pro-vided by the BMC on the BlueField Reference Platform.

Related DocumentationAll of these documents can be found on the Mellanox website. They are available either through the product pages or through the support page with a login and password.

Glossary

Table 1 - Related Documentation

Document Description

BlueField 2U Reference Plat-form Hardware User Manual

Provides details as to the interfaces of the Reference Platform, specifi-cations and hardware installation instructions.

Mellanox BlueField Reference Platforms Bring Up Guide

This document describes a step-by-step procedure of how to bring up the BlueField Reference Platform.

Table 2 - Glossary

Term Definition

BMC Baseboard Management Controller

EEPROM Electrically Erasable Programmable Read Only Memory

FRU Field Replaceable Unit

IPMI Intelligent Platform Management Interface

NC-SI Network Controller Sideband Interface

SoC System-on-chip

SOL Serial Over LAN

SEL System Event Log

SDR Sensor Database Record

UART Universal Asynchronous Receiver Transmitter


Introduction

1 IntroductionBMC software enables control and management of the baseboard management controller’s (BMC) hardware components. The BMC software supports the Intelligent Platform Management Interface (IPMI).The BMC node enables remote power cycling, board environment monitoring, BlueField chip temperature monitoring, board power and consumption monitoring, and individual interface resets. The BMC also supports the ability to push a bootstream to the BlueField. Most of the monitoring is performed through IPMI commands, although sysfs entry infrastructure to monitor is available as well.The procedures described in this manual assume that you have already installed and powered on your device according to the instructions in its bring up guide.• Support for IPMI 2.0 (v1.1) Standards

• Thermal control – access to all relevant temperature sensors, fan control

• System management – power state control, power on/off, reboot/reset

• Environmental monitoring – voltage/current/power

• Serial over LAN (SOL)

• RMCP/RMCP+

• Event log management

• Event alerting

• VLAN support

• Support for DMTF Standards

• Network Controller Sideband Interface (NC-SI) Specification (DMTF DSP0222)

• Support for BMC Image Update

• Support for CPLD image upgrade


Connecting to BMC Interfaces

2 Connecting to BMC Interfaces

2.1 BMC Console InterfaceThe BMC UART1 console is available on the IO panel. The BMC is connected to an RJ45 con-nector which allows the Linux console to be monitored.

Figure 1: BMC Console Interface

2.2 BMC Management Interface PortThe BMC has a separate Ethernet interface, distinct from the NC-SI interface, which provides network connection for management traffic to the BMC. The rear panel of the 1U and 2U Blue-Field Reference Platform has an RJ45 port labeled “BMC ENET” which is the management interface port. The management port is configured with auto-negotiation capabilities by default (100MbE to 1GbE).The BMC interface “eth1” is the management interface, so any information displayed by “ifcon-fig eth1” pertains to the management interface. The MAC address to be used for “eth1” is pre-programmed in the BMC Mezz EEPROM. By default, the IP address used for “eth1” is acquired via DHCP, but can be configured differently.

Make sure you plug the console cable into the BMC console port and not the BMC Ethernet port. Failure to make proper connections could damage the system.



Figure 2: BMC Management Interface

There are standard ways to configure Linux network interfaces for the BMC. There is a daemon (system-networkd) that manages network configurations. The default network configuration for this daemon is stored on the BMC at /usr/lib/systemd/network/default.network.By default, the initial network configuration settings in that file are:

[Match]Name=eth*[Network]DHCP=ipv4

This means that both “eth0” (NC-SI port) and “eth1” (management port) use DHCP to acquire an IPv4 address. To change that default behavior, the “default.network” file can be modified directly, or settings for a specific interface can be configured in a separate file. The naming con-vention for that separate file is /etc/systemd/network/<interfaceName>.network.To customize settings for a specific interface “ethX”:1. Modify the “default.network” file to no longer configure “ethX”2. Create a file called “ethX.network” and place it in the proper location3. Restart the systemd network daemon to enable these new settings

systemctl stop systemd-networkdsystemctl start systemd-networkd

For further information about this topic, please refer to the following URL: https://wiki.archlinux.org/index.php/systemd-networkd.Note: This is simply one way to configure IP addresses on the BMC interfaces, there are other methods used in Linux for doing this. Any network configuration changes performed with “ifconfig” from the command line, and not placed into a file, are lost following a reboot.


https://wiki.archlinux.org/index.php/systemd-networkd


2.3 BMC USB PortThis section describes the use cases for the BMC USB port, which is accessible from the back of the 1U and 2U chassis. It is worth noting that only BMC Linux has access to the USB port and its feature set. There is no access to BMC USB port while running U-Boot.

2.3.1 Provides Removable Storage via USB StickOnce a USB stick is plugged in to the BMC’s USB port, issue the command “lsusb” and/or check the dmesg log to see if the USB stick has been detected. The successful insertion of a USB stick will create a device under /dev called “sda” (or “sdb”), and a mountable partition “/dev/sda1”. To mount the USB stick as a filesystem, just issue the command “mount /dev/sda1 /mnt” to mount it at “/mnt”. The command “umount /mnt” unmounts the device.

2.3.2 Provides Network Connection from BMC to BlueField SoCThe cabling of “BMC USB” to “SoC USB” ports on the rear panel provides network connectivity between the BMC and the BlueField. By default these interfaces are configured as follows:

Table 3 - BMC to BlueField

BMC BlueField

Interface Name “eth2” “eth0”

MAC address 00:1A:CA:FF:FF:02 00:1A:CA:FF:FF:01

IP address <not configured> <not configured>

These USB interfaces do not come pre-configured with IP addresses, so consult the “Configuring Network Interfaces” section for configuration techniques.“eth2” on the BMC pertains to the BMC ↔ BlueField connection via USB, this has two require-ments: external USB cable, rshim_net kernel module must be loaded.Please issue “modprobe rshim_net” in order to load this kernel module.

2.3.3 Pushing Boot-stream from BMC to BlueField ArmThis section describes how to utilize the BMC to push a boot stream from an external x86 server to the BlueField Arm. The initial boot stream is contained in the BFB file and is pushed from BMC to BlueField via the RShim interface. The details of the steps are shown below and involve commands issued from one of three consoles:• “X86” = Console of external x86-based server

• “BMC” = BMC console

• “BlueField Arm” = BlueField console

Prerequisites:• BMC management port “eth1” cabled to network, and active

• BMC USB port cabled to BlueField USB port



• Three RShim kernel modules loaded on BMC: rshim, rshim_net, rshim_usb

2.3.4 Access of BlueField RShim Console from BMCThis section describes how to utilize the BMC to access the BlueField RShim console. If the required software is running on the BMC, and there is a USB cable between the BMC and the BlueField, a device node called “/dev/rshim0/console” is created on the BMC. This device node can be used as an argument to the “screen” program and thus providing access to BlueField Arm console via the RShim.For example:

BMC: screen /dev/rshim0/console 115200

Note that if nested screen programs are being used (i.e. one program to the BMC, and then another to the BlueField RShim), there is a special sequence needed to exit the “inner” (i.e. Blue-Field RShim) screen. The sequence “CTRL-a a k” should be used to kill just the inner screen pro-gram, leaving the outer screen program in use.Prerequisites:• BMC USB port cabled to BlueField USB port

• Two RShim kernel modules loaded on BMC: rshim, rshim_usb

Power on the 1U or 2U systemX86: cd /auto/mswg/release/sw_mc_soc/BlueField-1.0.xxx.yyyX86: Push the install-bluewhale BFB file to the BlueField, via SSH pipe to the BMC e.g. cat BlueField-1.0.alpha6.10485_install-bluewhale.bfb | ssh root@<BMC Mgmt IP> "cat > /dev/rshim0/boot"BlueField ARM: <booting, wait for prompt to return>BlueField ARM: /opt/mlnx/scripts/bfinstBMC: mkdir /ramdisk // if not created alreadyBMC: mount -t tmpfs -o size=300m tmpfs /ramdisk // size should be large enough to hold core image full tarballX86: scp the core-image-full.tar.xz file to the BMC e.g. X86: scp core-image-full-BlueField-1.0.alpha6.10485.2.4.1.tar.xz root@<BMC Mgmt IP>:/ramdisk/core-image-full.tar.xzBMC: modprobe rshim_netBMC: ifconfig eth2 192.168.100.1 netmask 255.255.255.0BMC: scp /ramdisk/core-image-full.tar.xz [email protected]:/tmpBlueField ARM: mount /dev/mmcblk0p2 /mntBlueField ARM: tar -C /mnt -xf /tmp/core-image-full.tar.xzBlueField ARM: rebootBlueField ARM: cat /etc/bluefield_version // verify the expected version


3 System Management

3.1 System EEPROMsThe BMC has access via I2C buses to board-level system EEPROM devices which contain pro-duction data (e.g. product name, part number, serial number, etc). The EEPROM devices avail-able are the following:• Reference Platform EEPROM

• Mezz EEPROM

All the EEPROM devices are programmed during manufacturing, so there should be no need for BMC software to write to the EEPROMs. There are two methods to read the EEPROM content:1. By issuing commands from the BMC Linux console (e.g. hexdump, i2cdump, dd) directed to

the sysfs path for the appropriate EEPROM device. Note that the I2C bus numbering is differ-ent in Linux than the hardware I2C numbering. Linux numbering starts at 0 and is one less than the hardware numbering, so access to the BlueField Reference Platform and BMC Mezz EEPROMs is via I2C bus 6 (/sys/bus/i2c/devices/{6-0050,6-0055}).

2. By IPMI commands directed to BMC. The “ipmitool” command “fru print XXX” displays the contents of FRU XXX. The numbering of FRUs within the system is “0” for the system FRU, and “1” for the BMC Mezz FRU.

The BMC u-boot console cannot access system EEPROMs.From a content perspective, the four EEPROMs are formatted to follow IPMI FRU format, as documented in the IPMI Platform Management FRU Information Storage Definition, which is downloadable on the Internet. The Reference Platform, Riser, and Backplane EEPROMs only have the “board info” and “product info” areas populated. The BMC Mezz EEPROM utilizes the “multi-record info” area to specify Mellanox-specific information, in this case it is the MAC addresses for the two BMC Ethernet ports (eth0 and eth1).

3.2 Management InterfaceThe BlueField Reference Platform provides management interfaces to the BMC and the Blue-Field device.The BlueField Reference Platform BMC, based on the IPMI standard, supports both out-of-band (OOB) dedicated interfaces, and a serial port to access the CLI of the BMC.The IPMI interface can be accessed from an external server using the standard tools like the IPMI tool. For more information on how to use IPMI, please refer to the IPMI 2.0 standard.

3.3 Serial Over LAN (SOL)If the external BlueField serial connection is not available to the switch (i.e. not connected), BMC software enables access to the BlueField through an internal serial connection redirected over an IP address.In order to connect to serial-over-LAN use the following IPMI command from an external server:

ipmitool -I lanplus -H <IP address of BMC > -U ADMIN -P ADMIN sol activate

Mellanox Technologies

System Management

14Confidential

https://www.intel.com/content/dam/www/public/us/en/documents/product-briefs/platform-management-fru-document-rev-1-2-feb-2013.pdf

For example:ipmitool -I lanplus -H 10.10.10.10 -U ADMIN -P ADMIN sol activate[SOL Session operational. Use ~? for help]

Poky (Yocto Project Reference Distro)

2.3.1 bluefield /dev/ttyAMA0

bluefield login:

3.4 Serial Console Capture (SCC)The BMC has 2 UART ports:• UART0 (aka UART1 in HW) which connects to BlueField’s UART0 and allows the

BMC to access the BlueField’s console

• UART4 (aka UART5 in HW) which connects to RJ45 and is used for the BMC console

This section gives the steps on how to achieve a serial capture of the BlueField console via BMC UART0.The serial capture enables us to capture the most recent messages on the BlueField’s console and store them in a buffer. It could be the BlueField’s booting messages, panic messages, or com-mands typed by the user.The size of the captured buffer is limited to 65535 bytes, and can be controlled by an IPMI tool command or the /etc/ipmi/mlxbw.lan.conf file from the BMC. The buffer can be dumped using the IPMI SOL session 2 which is explained in more details below.

3.4.1 Setting Up Serial CaptureThe first thing to do is to make sure that the BMC’s UART is connected to the BlueField’s UART.On the BlueField Reference Platform board, for example, the JP1 header allows the UART con-nection between the BMC and the BlueField. Jumpers should be used to connect the following pins on JP1, if SOL is not working:• Pin 2 (SE_BMC0_RS232_RXD) ↔ pin 1 (SE_SOC0_RS232_TXD)

• Pin 4 (GND) ↔ pin 3 (GND)

• Pin 6 (SE_BMC0_RS232_TXD) ↔ pin 5 (SE_SOC0_RS232_RXD)

The next step is to set up serial console redirection using the BMC which allows administrators to remotely view the content of the BlueField’s console.

SOL feature can be used even if BlueField is configured to use UART1/ttyAMA1.


System Management

15Confidential

System Management

To set up the serial console redirection, configure an SOL session in the /etc/ipmi/mlxbw.lan.conf file located on the BMC as follows:

sol <device> <baud_rate> [nortscts] [history=buffer_size[,backupfile=filename]] [historyfru=N]

Where:• device – the serial console is configured as “/dev/ttyS0”

• baud rate – 115200

• nortscts – disables RTS and CTS on the port. It changes the RTS output line, but does not listen to CTS.

• history – enables serial console redirection and sets the history buffer size. The maxi-mum buffer size is always 65535 bytes. This means that DCD and CTS are always enabled to the device, and constantly collect history for the device. Connecting to SOL interface 2 causes the full history buffer to be dumped.

• backupfile – enables storing the history if the BMC shuts down so the data persists while rebooting

• historyfru – assigns the BMC FRU device which fetches the contents of the history buf-fer

The following is an example for the above command on a BlueField Reference Platform board:sol "/dev/ttyS0" 115200 history=4000,backupfile=/home/root/serial_history historyfru=10

Normally the data is left in the history buffer when read. If you want the history data to be cleared from the buffer on read, add “readclear”:

sol "/dev/ttyS0" 115200 nortscts readclear

Whenever modifications are made to the /etc/ipmi/mlxbw.lan.conf file, the BMC needs to be rebooted to apply the changes. Once the BMC is up, make sure there is activity on the Blue-Field’s console, otherwise the captured buffer comes up empty.The captured data from the BlueField’s console can be dumped using sol session 2:

ipmitool -I lanplus -H <bmc_ip> -U ADMIN -P ADMIN sol activate instance=2

If you want to end the session just enter “~.”.Sometimes, a session remains active even after it is ended, you may deactivate it by running:

ipmitool -I lanplus -H <bmc_ip> -U ADMIN -P ADMIN sol deactivate instance=2

Make sure to have IPMI tools installed on your server.


3.4.2 Reading FRU DataAdding [historyfru=N] to the serial console redirection set up file enables FRU N to fetch the contents of the history buffer.The following command dumps the binary data pertaining to the specified FRU to an FRU file:

ipmitool -I lanplus -H <bmc_ip> -U ADMIN -P ADMIN fru read <fru-id> <fru-file>

Output example:Fru Size : 2068 bytesDone

You may convert the binary FRU file to a human readable text format file by using the following command:

ipmitool -I lanplus -H <bmc_ip> -U ADMIN -P ADMIN ekanalyzer frushow a1=<fru file>Note that the aforementioned command outputs general information about the FRU such as the manufacturer’s ID and device ID. That information is human readable, but then the BlueField’s console text will be in a hexadecimal format. So this is not as helpful as using SOL session 2 to dump the captured buffer.

3.4.3 History BufferIf the mlxbw.lan.conf file has been previously set up to have backupfile=<history_file>, and you activated an SOL session before rebooting the BMC, then, the backup file will contain the con-tent of the captured buffer from the last active SOL session.

3.4.3.1 Changing the Buffer Size

Once the SOL 2 connection is made, the buffer size can be overridden by running:ipmitool -I lanplus -H <bmc_ip> -U ADMIN -P ADMIN raw 0x2e 0x01 0x74 0x9f 0x00 274 9f 00

Where:• 0x2e – OEM command NETNF

• 0x1 – flag for SET

• 0x74 0x9f 0x00 – OpenIPMI IANA code

• 2 is the buffer size value (will be multiplied by 1024)

• 74 9f 00 – what is returned is the OpenIPMI IANA code

Note that you can only change the buffer size to a smaller size than the default size you set in the mlxbw.lan.conf file. For instance, if the buffer size was set as history=4000 in the mlxbw.lan.conf file, the highest buffer size value that you can set using the raw command is 3 because 3x1024 = 3072. If you try to set the size of the buffer to 4 (4x1024=4096) that would be higher than 4000, which causes your request to be discarded.


System Management

17Confidential

System Management

3.4.3.2 Reading the Buffer Size

To get the buffer size, run:ipmitool -I lanplus -H <bmc_ip> -U ADMIN -P ADMIN raw 0x2e 0x02 0x74 0x9f 0x0074 9f 00 02

Where:• 0x2e – OEM command NETNF

• 0x2 – flag for GET

• 0x74 0x9f 0x00 – OpenIPMI IANA code

• 74 9f 00 02 – what is returned is the OpenIPMI IANA code, and “02” is the size of the buffer divided by 1024

3.5 NC-SI InterfaceThe Mellanox Reference Platform design supports the Network Controller Sideband Interface (NC-SI) feature set. NC-SI is the DMTF-defined sideband protocol, and is fully documented in the DSP0222 – Network Controller Sideband Interface (NC-SI) Specification. The NC-SI is a board-level connection between the BlueField RBT port and the BMC MAC1 port. The NC-SI is based on RMII signaling, and runs at 100Mbps full duplex.From a functional perspective, the NC-SI allows the BMC to communicate with the external net-work via the QSFP ports of the BlueField SoC. It is worth noting that while QSFP ports run at 100Gb/s, the NC-SI only runs at 100Mbps. From a packet format point of view, NC-SI traffic is simply Ethernet packets with one of two EtherTypes:1. Control packets (EtherType=0x88F8) which contain information shared only between BMC

and BlueField. This traffic does not get forwarded to the QSFP ports. One example of this traffic type is the “get link status” command sent by BMC to BlueField to query for port sta-tus.

2. Pass-through packets (all other EtherTypes) which contain information shared between the BMC and the external network. All pass-through packets contain L2 information pertaining to the BMC’s eth0 MAC address.

From a BMC point of view, the interface eth0 is the NC-SI. So, any information displayed by “ifconfig eth0” pertains to the NC-SI. The MAC address used for BMC’s eth0 is contained in the BMC Mezz EEPROM. The IP address used for BMC’s eth0 is configurable. The DHCP request and reply only resolve if the QSFP ports on the BlueField’s ConnectX®-5 side are up and active. If the QSFP ports are not up, then the BMC’s eth0 interface does not get an IP through DHCP.

NC-SI is only active when the BMC is running Linux. While the BMC is running u-boot, there is no support for packet communication over NC-SI.


https://www.dmtf.org/sites/default/files/standards/documents/DSP0222_1.0.0.pdf

3.6 Chassis Management

3.6.1 System Fan ManagementBMC fan speed auto-control is enabled by default allowing fan speeds to change automatically depending on system temperature.To disable fan auto-control, run:

systemctl stop obmc-mellanox-fand

To enable fan auto-control, run:systemctl start obmc-mellanox-fand

To retrieve fan speed, run:ipmitool sdr list | grep fanipmitool sdr get fan1_rpmipmitool sdr get fan2_rpmipmitool sdr get fan3_rpm

It is also possible to see fan parameters from BMC console under /bsp/fan/fan1_rpm, /bsp/fan/fan2_rpm, and /bsp/fan/fan3_rpm.

BMC continuously monitors fan speeds. If any speed deviation occurs, the corresponding log is added to the SEL:

25 | OEM record e0 | 00017600000000000000000000 | Abnormal FAN speed, too low FAN = 0x7626 | OEM record e0 | 00017700000000000000000000 | Abnormal FAN speed, too low FAN = 0x77

3.6.2 FRU ReadingTo retrieve FRU info, run:

ipmitool fru print <fru-id>

The list of available FRUs is as follows:• 0 – system

• 1 – BMC

It is possible to dump the binary FRU data pertaining to the specified FRU entity into a file. Run:ipmitool fru read <fru-id> <filename>

From a networking point-of-view, the NC-SI port is on the same IP subnet as the first active ConnectX®-5 QSFP port. Therefore, the BMC’s management port (eth1) should be on a different subnet than eth0. Also, to access the BMC via the management inter-face, the remote server should not be in the same subnet as the NC-SI (eth0) interface.


System Management

19Confidential

System Management

3.6.3 Sensor Database Rate (SDR) SupportSDR contains information about the type and number of sensors. The following is a list of the available SDR information:

The parameter <filename> is the absolute path to the file.

Table 4 - Available SDR Information

SDR Entry Description

Board Temperature Monitoring

bf_inlet_temp Temperature on Reference Platform board near BlueField device

qsfp_inlet_temp Temperature on Reference Platform board near QSFP cages

pci_inlet_temp Temperature on Reference Platform board near PCI connectorsFans

fan1_rpm RPM of Fan1

fan2_rpm RPM of Fan2

fan3_rpm RPM of Fan3Voltage Regulator

vr_vin Input voltage input as measured by voltage regulator (normally 12V)

vr_vcore_iout Vcore output current as measured by voltage regulator

vr_bfddr_1p2v_iout 1.2 BlueField DDR output current as measured by voltage regulatorVoltage Monitors for Reference Platform and BlueField (Power1014a)

mb_3p3v_aux 3.3V AUX rail as measured by Reference Platform board voltage monitor

mb_3p3v_stby 3.3V standby rail as measured by Reference Platform board voltage monitor

mb_3p3v 3.3V rail as measured by Reference Platform board voltage monitor

mb_5v_stby 5V standby rail as measured by Reference Platform board voltage monitor

mb_5v_aux 5V AUX rail as measured by Reference Platform board voltage monitor

mb_pci0_12v PCI Slot 0 12V rail as measured by Reference Platform board voltage monitor

mb_pci0_3p3v PCI Slot 0 3.3V rail as measured by Reference Platform board voltage monitor

mb_pci1_12v PCI Slot 1 12V rail as measured by Reference Platform board voltage monitor

mb_pci1_3p3v PCI Slot 1 3.3V rail as measured by Reference Platform board voltage monitor

mb_vcca_3p3v 3.3 sequencer voltage at VCCA pin as measured by Reference Platform board voltage monitor

mb_vccinp_3p3v 3.3 sequencer voltage at VCCINP pin as measured by Reference Platform board voltage monitor

bf_vcore_0p95v Vcore voltage as measured by BlueField voltage monitor


System Management

3.6.4 IPMI Stored Information

3.6.4.1 Event Management

The system event log (SEL) is non-volatile repository for system events and certain system con-figuration information. SEL entries have a unique “record ID” field. This field is used for retriev-ing log entries from the SEL. Record IDs are not required to be sequential or consecutive. Applications should not assume that the SEL record ID follows any particular numeric ordering.

3.6.4.2 System Event Log

Event logs are chassis events, recorded in the BMC software which can be read using IPMI com-mands.If the SEL is full and a new event is raised, the oldest record is removed and the new one is placed at the end of the SEL.SEL may be accessed, even after BlueField failure, on the server through IPMI LAN access.Table 5 lists the command to use in order to view event logs.

3.6.4.3 SDR Repository

BMC software supports reading chassis sensor information using the IPMI tool.Table 6 lists commands which allow reading sensor data.

bf_io_3p3v 3.3 BlueField IO voltage as measured by BlueField voltage monitor

bf_ddr0_2p5v 2.5 BlueField DDR0 voltage as measured by BlueField voltage monitor

bf_ddr1_2p5v 2.5 BlueField DDR1 voltage as measured by BlueField voltage monitor

bf_vddo_1p8v 1.8 VDDO voltage as measured by BlueField voltage monitor

bf_ddr_1p2v 1.2 DDR voltage as measured by BlueField voltage monitor

bf_serdes_1p2v 1.2 SerDes voltage as measured by BlueField voltage monitor

bf_vcca_3p3v 3.3 sequencer voltage at VCCA pin as measured by BlueField voltage monitor

bf_vccinp_3p3v 3.3 sequencer voltage at VCCINP pin as measured by BlueField voltage monitor

Table 5 - Event Log Retrieval Commands

Command Description

ipmitool sel Displays information about SEL

ipmitool sel list Displays list of events

ipmitool sel elist Displays extended info list of events

ipmitool sel time set “mm/dd/yyyy hh:mm:ss” Modifies time on the BMC

ipmitool sel clear Clears SEL

Table 4 - Available SDR Information

SDR Entry Description


System Management

3.6.5 Rebooting BlueField with BMCBMC software enables resetting the BlueField.To reset the main CPU, Run:

ipmitool chassis power reset

3.6.6 Shutting Down BlueField with BMCTo shut down the BlueField, Run:

ipmitool chassis power off

3.6.7 Turning on BlueField with BMCTo turn on the BlueField, Run:

ipmitool chassis power on

Table 6 - Sensor Data Retrieval Commands

Command Description

ipmitool sdr list Displays sensor data repository entry readings and their status

ipmitool sdr elist Displays extended sensor information

ipmitool sensor list Displays sensors and thresholds in a wide table format

ipmitool sdr get <name> Displays information for sensor data records specified by sensor ID

ipmitool sdr type <type> Displays all records from the SDR repository of a specific type

ipmitool sensor get <sensor_name> Displays information for sensors specified by name

ipmitool sensor reading <name>…<name> Displays readings for sensors specified by name


4 BMC Management

4.1 System ComponentsOpenBMC is an open software framework to build a complete Linux image for a Board Manage-ment Controller (BMC). It uses the Yocto Project as the underlying building and distro genera-tion framework.The OpenBMC project can be found at: https://github.com/openbmc/openbmc.The primary software components are listed below:• U-Boot bootloader

• Linux kernel

• OpenBMC Distro

• OpenIPMI package

4.1.1 Software VersioningThere is a software version for each of the BMC software components. You may retrieve this information by running the following for each component:• U-boot version: “version” command from the u-boot prompt

• Linux version: “uname -a” command from the Linux prompt

• OpenBMC version: “cat /etc/os-release” from the Linux prompt

4.1.2 Boot Sequence Overview1. BMC starts booting through u-boot bootloader once the power supply is powered on.2. By default, the BMC automatically boots into Linux. If the user wants to stop at the u-boot

prompt, the user must type the password “0penBmc” within 5 seconds (note use of the digit zero in “0pen”). If you want to boot Linux from the u-boot prompt, type “boot”.

3. Redundant boot flash – helps with recovery: By default, BMC boots off the CS0 flash device. If the watchdog expires (after 20 seconds) the BMC boots off CS1 flash device.

4. The default password for the root user, to be typed in once Linux is booted, is “admin”.

4.2 User ManagementTo see the list of BMC users, run the following command:

ipmitool user list [<channel number>]

To create a new user, run:ipmitool user set name <user id> <user name>ipmitool user set password <user id> <password>ipmitool user enable <user id>ipmitool user priv <user id> <privilege level(1-4)> [<channel number>]


BMC Management

23Confidential

https://github.com/openbmc/openbmc

To disable a user, run:ipmitool user disable <user id>

4.3 BMC Image UpgradeBMC software may be upgraded from the BMC shell or a remote server. To upgrade BMC software from shell: Step 1. Log into BMC Step 2. Upload BMC image using standard Linux SCP protocol. Run from the remote server:

scp <bmc-image> root@<bmc_ip>:/tmp

Step 3. Run the update script from the BMC using the following command:/run/initramfs/update_all <absolute image path>

The script automatically reboots the BMC to update to the new version. Allow the system to boot without interruption for a few minutes until the upgrade procedure is complete.

To upgrade BMC software from remote server: Step 1. Upload BMC image using standard Linux SCP protocol. Run:

scp <bmc-image> root@<bmc_ip>:/tmp

Step 2. Run the update script using the following command:sshpass -p "<root-password>" ssh root@<bmc_ip> '/run/initramfs/update_all <absolute image path>

The script automatically reboots the BMC to update to the new version. Allow the system to boot without interruption for a few minutes until the upgrade procedure is complete.

4.4 CPLD UpgradeCPLD may be upgraded from the BMC shell. To upgrade CPLD: Step 1. Log into the BMC. Step 2. Upload the image using standard Linux SCP protocol. Step 3. Run the update script from the BMC:

update_cpld <absolute image path>


BMC Management

24Confidential

Mellanox Custom Commands


5 Mellanox Custom CommandsNot all functionalities are covered with a standard set of IPMI tool commands. Therefore, a set of custom Mellanox IPMItool “raw” commands have been added. The first two parameters of the “raw” command are NetFN and CMD. IPMI tool “raw” commands follow the template “ipmitool raw <netfunc> <cmd> <data>” where:• netfunc – network function which identifies the functional message class, and clusters

IPMI commands into sets

• cmd – one byte command within a network function

• data – optional element which provides additional parameters for a request or response message

Table 7 lists the supported IPMI tool “raw” commands. Just use the template “ipmitool raw <net-func> <cmd> <data>” in order to run these commands.

Table 7 - Custom IPMI Tool Commands

Netfunc Cmd Data Description

0x2e 0x01 0x74 0x9f 0x00 2 Changes the serial console capture buffer size

0x2e 0x02 0x74 0x9f 0x00 Retrieves the serial console capture buffer size

0x6 0x5b <size_LSB> <size_MSB> Configures system event log buffer size

0x6 0x63 Retrieves BMC uptime

0x6 0x64 [sdr_type] [direction] [event_type] Adds record to the SEL

List of Supported IPMItool Commands


6 List of Supported IPMItool Commands

The IPMItool program allows you to remotely manage the IPMI functions of the BlueField BMC. The commands below may be directed to the BMC’s Ethernet interface by invoking:

ipmitool -I lanplus -H <BMC IP addr> -U ADMIN -P ADMIN <ipmitool arguments>

The following table provides a full list of the IPMItool arguments supported by BlueField BMC.chassis power offchassis power onchassis power resetchassis status (to be implemented in future release)frufru print 0fru print 1fru read 0 /tmp/frufru read 1 /tmp/frulan printmc infomc reset coldsdr elistsdr get <sensor name>sdr listsdr type <type>selsel clearsel elistsel listsel time set "MM/DD/YYYY HH:MM:SS"sensor get <sensor name>sensor listsol activatesol activate instance=2user disable <user id>user enable <user id>user list [<channel number>]user priv <user id> <privilege level(1-4)> [<channel number>]user set name <user id> <user name>user set password <user id> <password>

Mellanox Technologies does not support IPMI functionalities that are not supported by OpenIPMI by default.

Mellanox BlueField BMC Software User Manual · BMC software enables control and management of the...

Documents

Transcript of Mellanox BlueField BMC Software User Manual · BMC software enables control and management of the...