Inspur NF5288M5 Supercomputer White Paper · 2019-05-23 · 1 1 Product Overview Inspur NF5288M5...

Inspur NF5288M5 Supercomputer

White Paper

V1.5

Inspur Electronic Information Industry Co., Ltd.

Contents

1 Product Overview ............................................................................................................... 1

2 Product Features ................................................................................................................ 2

3 Product Overview ............................................................................................................... 4

3.1 Appearance............................................................................................................................... 4

3.2 Physical Structure ................................................................................................................... 5

3.3 Motherboard ............................................................................................................................. 7

3.4 GPU board ................................................................................................................................ 8

3.5 Fan module ............................................................................................................................. 10

3.6 Rear IO board ......................................................................................................................... 11

3.7 Front IO riser board ............................................................................................................... 11

3.8 Logical Structure ................................................................................................................... 12

3.9 Technical Parameters ............................................................................................................ 15

4 Module and Compatibility ................................................................................................ 18

4.1 GPU .......................................................................................................................................... 18

4.2 CPU .......................................................................................................................................... 18

4.3 Memory ................................................................................................................................... 19

4.4 Hard disk ................................................................................................................................. 21

4.4.1 Front hot swap hard disk ........................................................................................................... 21

4.4.2 Built-in hard disk ......................................................................................................................... 24

4.5 PCIe expansion card ............................................................................................................. 24

4.6 PSU .......................................................................................................................................... 26

4.7 Operating system ................................................................................................................... 27

5 BMC management system ............................................................................................... 28

5.1 Function module .................................................................................................................... 29

6 Certification and License ................................................................................................. 30

1

1 Product Overview

Inspur NF5288M5 Supercomputer (also referred to as AGX-2) is an "AI Supercomputer"

for intelligent computing, high-performance computing and accelerated video applications.

It is also a new generation 8GPU server with 2U height for Internet, intelligent security,

intelligent transportation, intelligent finance, intelligent healthcare, intelligent

manufacturing and other applications. Inspur Supercomputer NF5288M5 is the world’s

leading 2U server with 8 high-performance GPU accelerators interconnected through

300GB/s high-speed NVIDIA® NVLink ™. It is also an AI server with the world’s highest

density and best performance.

Figure 1-1 NF5288M5 External View

2

2 Product Features

Inspur Supercomputer NF5288M5 has following features:

Highest density, excellent performance:

2U standard space, 2-socket server, 8 × GPU card on NVIDIA® NVLink ™ or FHFL

dual-width PCIe GPU card.

Support 2× Intel ® Xeon ® Scalable Processor, up to 28-core processor to provide

excellent system performance, maximum 3.6 GHz base frequency, 38.5 MB L3 cache and

up to 2 × UPI at 10.4 GT/s.

Support 16 × DDR4 ECC memory of 2666 MT/s, 12 × memory channel, RDIMM and

LRDIMM, providing excellent speed, high availability, memory capacity up to 1,024GB,

and maximum memory bandwidth of 249.9375GB/s.

Support 8 × hot swap NVMe SSD and 2 × SATA M.2 SSD to maximize disk caching

rate of stored data.

Support 4 × 100G RDMA(Remote Direct Memory Access), greatly optimizing horizontal

expansion capacity.

For AI training, provide peak computing capability up to 960 Tensor TFLOPs at a single

node, over 200 times higher than traditional 2-socket server.

For HPC application, a single node delivers a peak dual-precision floating point

capability higher than 60 TFLOPs. And high-density cluster of NF5288M5 in a single

cabinet (42U) can easily reach the peak performance of more than 1 PFLOPs, (thousand

trillion of floating-point computing per second).

Support NVIDIA®NVLink2.0, allow 300GB/s bandwidth among GPUs. Hybrid Cube

Mesh reduces latency of multi-GPU data sharing, and keep cache consistency between

multiple GPUs, providing a better acceleration ratio for computing.

Flexible Configuration, Expandable as Needed

Support Tesla®SXM2 V100/P100 based on NVIDIA®NVLink ™, or V100 / P100 / P40 /

P4 GPU based on PCIe3.0 port, etc. Support 2 type interface and multiple GPU models,

allowing users to select GPU card as needed.

Connect CPU and GPU with PCIe cable, flexibly adjust CPU bandwidth and quantity,

and allocate PCIe resources more effectively as needed in different AI applications.

Flexible computing architecture allows one or two CPUs to manage up to 8 GPUs.

3

PCIe I / O and 8 × U.2 slots in the server flexibly adjust topology based on computing.

Flexible heterogeneous platforms are sufficient to support diverse AI application

scenarios.

Front panel supports 2 × PCIe3.0 X16 HHHL slots.

For SXM2 port GPU, rear panel supports 4 × PCIe3.0 X16 HHHL slots.

Support LSI3108 2G/4G cache RAID Mezz card on-board without occupying external

PCIe slot, provide customers with higher IO extensibility.

Ingenious Design and Green Innovation:

(1 + 1) redundant 3,000W 80Plus Platinum grade PSU with ultra-low power loss.

Reserve liquid cooling connectors, support gas-liquid heat radiation mode, and transfer

80% heat via GPU liquid cooling module, significantly reducing data center PUE.

Independent design and intellectual property rights of Inspur.

4

3 Product Overview

3.1 Appearance

Front panel of NF5288M5 is shown in Figure 3-1.

Figure 3-1 Front Panel

Rear panel of NF5288M5 is shown in Figure 3-2.

Figure 3-2 Rear Panel

Table 3-1 Technical Specifications for Front and Rear Panels

No. Name Quantity Description

1 PCIe I/O 2 2 × PCIe X16 HHHL front slot

2 Power module 2 80Plus Platinum PSU, 1+1

3 VGA port 1

4 USB 3.0 port 2

5 System reset button 1

5

6 System power button & LED 1

7 UID button & LED 1 Press and hold for 7 seconds to reset

BMC

8 2.5 inch hard disk holder 8

9 Rear PCIe slot 4 2 × PCIe X16 HHHL slots

(only for NVLink GPU)

10 Liquid cooling connector 1 GPU liquid-cooling connector reserved

11 C20 power connector 2 Only for C19 power cord

12 Serial port 1

13 UID button & LED 1

14 Rear VGA port 1

15 BMC management port 1

16 USB 3.0 port 2

17 10GE optical port 4 10GE optical port does not support

10/100M network. Support NC-SI

function to help you access BMC

management system via service port.

This port do not support Finisar Gen2

AOC or Rosenberger AOC cable

3.2 Physical Structure

Physical structure of NF5288M5 consists of chassis, power supply, motherboard, hard

disk, front IO riser board, fan module, GPU board and rear IO board. There are two GPU

board replacement options for NVLink GPU and PCIe GPU, to allow NF5288M5 to

support SXM2 port GPU on NVLink and standard PCIe3.0 port GPU.

NVLink GPU of NF5288M5 is shown in Figure 3-3:

6

Figure 3-3 NF5288M5 NVLink Physical Structure

PCIe GPU of NF5288M5 is shown in Figure 3-4:

Figure 3-4 NF5288M5 PCIe Physical Structure

Table 3-1 NF5288M5 Physical Structure

No. Name No. Name

1 PSU 5 System fan module

2 Hard disk 6 NVLink GPU board

3 Front IO Riser card 7 Rear IO board

7

4 Motherboard 8 PCIe GPU board

3.3 Motherboard

Motherboard of NF5288M5 is shown in Figure 3-5.

Figure 3-5 NF5288M5 Motherboard

Table 3-2 Technical Specification of NF5288M5 Motherboard

No. Name and description

1 DIMM (for CPU1)

2 Hard disk backplane power supply port and SAS, SATA signal port

3 PCIe3.0 X16 HHHL front slot

4 M.2 Riser card slot (supporting 2 × M.2 SSD hard disk)

5 ID button and ID LED & BMC reset button (Press and hold for 7 seconds to

reset BMC)

6 Power button & LED

7 System reset button

8 2*USB 3.0 port

9 VGA port

10

LSI3108 2G / 4G Cache RAID Mezz Card (supporting RAID

0/1/1E/10/5/50/6/60 with hot swap SAS/SATA hard disk, providing RAID

Cache, optional supercapacitor for data protection during power loss)

11 PCH chipset

12 Signal port (Internet, USB and VGA to rear IO board)

13 DIMM (for CPU0)

8

14 Signal port (BMC, UID and serial port to rear IO board)

15 System fan, management and signal port

16 System fan power supply port

17/26 PSU0/PSU1 power supply, management and signal ports

18/25 PSU0/PSU1 power supply port

19/24 CPU0/CPU1

20/21/22/23 2 * Slimline X8 port (each No. indicates to 2 × Slimline X8 port, one on

upper motherboard and one on lower motherboard)

3.4 GPU board

NF5288M5 support following two types of GPU boards: NVLink GPU board and PCIe

GPU board. Support two types GPU port (SXM2 port and PCIe port).

NF5288M5 NVLink GPU board is shown in Figure 3-6.

Figure 3-6 NF5288M5 NVLink GPU Board

NF5288M5 PCIe GPU board is shown in Figure 3-7.

9

Figure 3-7 NF5288M5 PCIe GPU Board

Table 3-3 Technical Specifications of NVLink GPU Board & PCIe GPU Board


1/2/3/4 Slimline X8 port

5 PCIe switch (PEX9797)

6 NVLink GPU slot 0

7 NVLink GPU slot 1

8 NVLink GPU slot 2

11 NVLink GPU slot 3





9/10/13/14 PCIe X16 HHHL slot


19/28/31/38 GPU power supply port (power supply per port to 2 GPUs)

20/21/26/27/32/33/36/37 PCIeX16 connector (to GPU Riser card)


34/35 PCIe switch (PEX9797)

Installation sequence of PCIe GPU card is shown in Figure 3-8.

10

Figure-3-8 PCIe GPU Card Sequence

3.5 Fan module

NF5288M5 uses 5 pairs of dual rotor pairs of rotating redundant system fan, N+N

redundant mode. When 1 rottors are broken in each group of fans, the fan group exhaust

volume can be kept constant

NF5288M5 fan module is shown in Figure 3-9.

Figure-3-9 System Fan Module

Table 3-4 Technical Specification of System Fan Module


1 5 × dual-rotor redundant system fan

GPU0 GPU2 GPU1 GPU3 GPU4 GPU5 GPU6 GPU7

11

2 System fan power supply port

3 System fan, management and signal port

3.6 Rear IO board

NF5288M5 rear IO board is shown in Figure 3-10.

Figure 3-10 Rear IO Board

Table 3-5 Technical Specification of Rear IO Riser Board


1 Serial port

2 ID button & LED

3 VGA port

4 Management port

5 2*USB port

6 4×10G SFP+ Ethernet (supporting NC-SI from left to right eth0 - eth3,

helping you access BMC management system via service network)

7 Signal port (Internet, USB and VGA to motherboard)

8 Signal port (BMC, UID and serial port to motherboard)

3.7 Front IO riser board

Front IO Riser board of NF5288M5 is shown in Figure 3-11.

12

Figure 3-11 Front IO Riser Board

Table 3--7 Technical Specification of Front IO Riser Board


1 PCIeX16 slot (2 × Slimline X8 is converted into PCIE signal)

2 2 Slimline X8 interface (to motherboard or PCIe GPU board)

3 PCIe X16 slot (Motherboard PCIe switch)

3.8 Logical Structure

Motherboard logical block diagram of NF5288M5 is shown in Figure 3-12.

13

Figure 3-12 NF5288M5 Motherboard Logical Block Diagram

NVLink GPU board logical block diagram of NF5288M5 is shown in Figure 3-13:

14

Figure 3-13 NF5288M5 NVLink GPU Board Logical Block Diagram

PCIe GPU board logical block diagram of NF5288M5 is shown in Figure 3-14:

15

Figure 3-14 NF5288M5 PCIe GPU Board Logical Block Diagram

3.9 Technical Parameters

Complete technical parameters of NF5288M5 are listed in Table 3-8.

Table 3-8 Technical Parameters

Module Name Specification description

Form 2U rack server

Processor 1/2 Intel® Xeon® 3100, 4100, 5100, 6100, 8100 series

Scalable Processor:

Support up to 28 cores (2.5GHz frequency)

Maximum frequency 3.6GHz (4 cores)

Support L3 cache up to 38.5MB (28 cores)

2 × UPIs with maximum rate 10.4GT/s per link

Maximum thermal design power 165W.

Chipset Intel® C624 chipset

Memory Every CPU support 8 x DIMM, total 16 x DIMM

Every CPU support 6 x memory channel, total 12 x memory

channel

IO port 2 front set USB 3.0 port, 1 VGA 1 ID button & ID LED & BMC

reset button, 1 system power button & LED, 1 system reset

button

16

2 rear set USB 3.0 port, 1 VGA, 1 serial port, 1 ID button &

LED, 1 BMC management port

RAID card Support 1pcs 8-channel high-performance SAS RAID Mezz

card without occupying standard PCIe slot, improving system

expansibility.

Optional super-capacitor protection module, support Raid

0/1/10/5/50/6/60.

Storage Front 8 × 2.5 inch hot swap SAS / SATA / U.2 port NVMe SSD

hard disks; 2× M.2(SATA) SSDs built-in

On-board Network Integrated Intel X722 provide 4*10GE optical port, support

NCSI, Wake-on-LAN, network redundancy, load balancing

and other advanced features

PSU 2pcs 3,000W 80Plus Platinum PSU (Redundant under

restrictive condition)

Support 100V-240V AC and 240V DC input

C20 power port (with C19 power cord)

Heat radiation 5 × redundant hot swap system fan,

Reserved NVLink GPU liquid cooling connector

GPU For NVLink GPU board:

8 × SXM2 port, support NVIDIA Tesla V100 / P100 (SXM2) on

NVLink, 4 × PCIeX16 HHHL rear slot

For PCIe GPU board:

8 built-in GPU PCIe3.0 X16 FHFL and dual-width slot, support

NVIDIA Tesla V100 (PCIe) /P100 (PCIe) /P40/P4, etc.

PCIe I/O For PCIe GPU:

2 × PCIe 3.0 X16 HHHL front slot

For NVLink GPU:

2 × PCIe 3.0 X16 HHHL front slot and 4 × PCIe 3.0 X16 HHHL

rear slot

Monitor and

Management

Provide specific Ethernet port for management. Supports

remote monitoring, SMTP KVM, SNMP management, Virtual

Media and redundant management system

Support OS Windows server 2012 R2/2016

Redhat Linux Enterprise Server 7.2/7.3/7.4

Suse Linux Enterprise Server 12 and other main operating

systems

Chassis Size 448mm width × 87.5mm height × 899.5mm depth (17.64in

width × 3.44in height × 35.41in depth)

Net Weight ≤48kg

Operating Environment

Temperature 10℃ to 35℃

17

Transportation &Storage

temperature -40℃ to 60℃

Operating humidity 20%-80% RH

Storage humidity 20%-93% RH

Storage &Operating

altitude ≤3000M

18

4 Module and Compatibility

4.1 GPU

Support 0/4/8 ×standard PCIe GPU card. Or

Support 0/4/6/8 × SXM2 NVLink GPU card.

Support up to 8 × NVLink NVIDIA® TESLA® V100 / P100 GPU card

Support up to 8 PCIe GPU cards such as NVIDIA® TESLA® V100 / P100 / P40 / P4,

etc.

supports both NVIDIA® NVLink ™ 1.0 and NVIDIA® NVLink ™ 2.0

Maximum P2P bandwidth 300GB/s

Support up to 300W GPU TDP

For SXM2 interface NVLink-connection GPU card, NVLink GPU board should be

configured, supported GPUs are shown in Table 4-1.

Table 4-1 NVLink GPU Compatibility List

GPU No. GPU model

X01700D000000000 GPU_NV_16G_Tesla-P100_4096b_S_CAC

X01700K000000000 GPU_NV_16GB_Tesla-V100_4096b_S_SXM2

For standard PCIe interface GPU card, PCIE GPU board should be configured, supported

GPUs are shown in Table 4-2.

Table 4-2 PCIe GPU Compatibility List

GPU No. GPU model

X01700B000000000 GPU_NV_12G_TESLA-P100_4096b_P_CAC

X01700C000000000 GPU_NV_16G_TESLA-P100_4096b_P_CAC

X01700F000000000 GPU_NV_24G_TESLA-P40_384b_P_CAC

X01700G000000000 GPU_NV_8G_P4_256b_P_CAC

X01700L000000000 GPU_NV_16GB_Tesla-V100_4096b_P_CAC

4.2 CPU

Support 1/2 Intel® Xeon® 3100, 4100, 5100, 6100, 8100 series Scalable Processor:

Support up to 28 cores (2.5GHz frequency)

19

Maximum frequency 3.6GHz (4 cores)

2 × UPIs with maximum rate 10.4GT/s per link

Support L3 cache up to 38.5MB (28 cores)

Maximum thermal design power 165W

For 2 × CPUs, CPUs are installed on CPU0 and CPU1 slots of the motherboard. For only

1× CPU, CPU is installed on CPU0 slot.

CPU supported by NF5288M5 is shown in Figure 4-3.

Table 4-3 CPU Compatibility List

CPU No. CPU model

V00103P000000000 CPU_I_4114-Xeon2.2_10_9.6_13.75_85

V00103V000000000 CPU_I_4116-Xeon2.1_12_9.6_16.5_85

V00102J000000000 CPU_I_5117-Xeon2.0_14C_19.25M_105W_BD

V00103N000000000 CPU_I_5118-Xeon2.3_12_10.4_16.5_105

V001037000000000 CPU_I_6126-Xeon2.6_12_10.4_19.25_125

V001039000000000 CPU_I_6130-Xeon2.1_16_10.4_22_125

V00103W000000000 CPU_I_6132-Xeon2.6_14_10.4_19.25_140

V00102U000000000 CPU_I_6134-Xeon3.0_8_10.4_24.75_130

V00102T000000000 CPU_I_6138-Xeon2.0_20_10.4_27.5_125

V001045000000000 CPU_I_6138-Xeon2.0_20C_27.5M_125W

V001030000000000 CPU_I_6140-Xeon2.3_18_10.4_24.75_140

V00102R000000000 CPU_I_6142-Xeon2.6_16_10.4_22_150

V001032000000000 CPU_I_6148-Xeon2.4_20_10.4_27.5_150

V001031000000000 CPU_I_6152-Xeon2.1_22_10.4_0.25_140

V00102V000000000 CPU_I_8160-Xeon2.1_24_10.4_33_150

V00102K000000000 CPU_I_8163-Xeon2.5_24C_33M_165W_ALI

V00102Z000000000 CPU_I_8164-Xeon2.0_26_10.4_35.75_150

V001038000000000 CPU_I_8170-Xeon2.1_26_10.4_35.75_165

V001034000000000 CPU_I_8176-Xeon2.1_28_10.4_38.5_165

4.3 Memory

NF5288M5 has 16 × DDR4 DIMM slots. Support DDR4 2133/2400/2666MT/s

RDIMM/LRDIMM. Each processor provides 6 memory channels, in which CPU0_C0,

CPU0_C3, CPU1_C0, CPU1_C3 support 2 × DIMMs.

NF5288M5 supports a single DIMM of memory capacity 16GB, 32GB and 64GB,

maximum memory support is up to 1024GB.

20

Memory installation rules are shown in table 4-4 and table 4-5.

Table 4-4 NF5288M5 Dual-CPU Memory Configuration

DRAM supported by dual

CPUs

Memory quantity and memory installation location

2 4 6 8 10 12 14 16

DRAM

supported

by CPU0

CPU0_C0D0 ● ● ● ● ● ● ● ●

CPU0_C0D1 ● ●

CPU0_C1D0 ● ● ● ● ● ● ●

CPU0_C2D0 ● ● ● ● ● ●

CPU0_C3D0 ● ● ● ● ●

CPU0_C3D1 ●

CPU0_C4D0 ● ● ● ●

CPU0_C5D0 ● ● ●

DRAM

supported

by CPU1

CPU1_C0D0 ● ● ● ● ● ● ● ●

CPU1_C0D1 ● ●

CPU1_C1D0 ● ● ● ● ● ● ●

CPU1_C2D0 ● ● ● ● ● ●

CPU1_C3D0 ● ● ● ● ●

CPU1_C3D1 ●

CPU1_C4D0 ● ● ● ●

CPU1_C5D0 ● ● ●

Table 4-5 NF5288M5 Single-CPU Memory Configuration

DRAM supported by a

single CPU

Memory quantity and memory installation location

1 2 3 4 5 6 7 8

DRAM

supported

by CPU0

CPU0_C0D0 ● ● ● ● ● ● ● ●

CPU0_C0D1 ● ●

CPU0_C1D0 ● ● ● ● ● ● ●

CPU0_C2D0 ● ● ● ● ● ●

CPU0_C3D0 ● ● ● ● ●

CPU0_C3D1 ●

CPU0_C4D0 ● ● ● ●

CPU0_C5D0 ● ● ●

21

● Memory installation rules:

One server supports only one memory type. Memory installation rules are described as

follows:

A. With priority to white slot, CPU1 and CPU0 memory needs to be symmetrically

installed.

B. In case of single CPU, memory is installed in the following order: CPU0_C0D0,

CPU0_C1D0, CPU0_C2D0, CPU0_C3D0, CPU0_C4D0, CPU0_C5D0; CPU0_C0D1,

CPU0_C3D1

C. In case of double CPUs, memory at CPU0 is installed in the following order:

CPU0_C0D0, CPU0_C1D0, CPU0_C2D0...; memory at CPU1 needs to be

symmetrical with memory at CPU0: CPU1_C0D0, CPU1_C1D0 and CPU1_C2D0...

DIMM position of NF5288M5 is shown in Figure 4-1.

Figure 4-1 NF5288M5 DIMM Slot Position Diagram

4.4 Hard disk

4.4.1 Front hot swap hard disk

NF5288M5 supports front 8 × 2.5 inch SAS / SATA / NVMe hard disk, and mix use with

multi-capacity and multi-type hard disk.

For SAS / SATA hard disk, hard disk sequence at delivery is shown in Figure 4-2.

22

Figure 4-2 SAS / SATA Hard Disk Configuration

There are two types of hard disk backplane, 1 backplane support 4 hard disk and support

up to 2 backplanes. as shown in Table 4-6.

Table 4-6 Hard Disk Backplane List

No. Backplane name

YZBB-00906-101 Backplane_Inspur_Vancouver_HDDBP_SATA_NVMe

YZBB-00835-101 Backplane_Inspur_Vancouver_HDDBP_SAS_NVMe

YZBB-00906-101 supports 2.5 inch SATA hard disk. SATA signal port is connected

from PCH chip on motherboard. With NVMe cable, supports NVMe hard disk, SATA and

NVMe hard disks in any capacity and number, and software RAID0/1/5/10. This

configuration does not need RAID Mezz card.

YZBB-00835-101 is used with Inspur 3108 RAID Mezz card, and supports 2.5 inch

SATA / SAS hard disk. SAS signal port is connected from RAID Mezz card, supports Raid

0/1/10/5/50/6/60 and RAID card direct pass-through mode; with NVMe cable, support

NVMe hard disk, and support mix use of SATA/SAS and NVMe hard disk in any capacity

and quantity.

SAS/SATA hard disk compatibility is shown in Table 4-7.

Table 4-7 SAS/SATA Hard Disk Compatibility List

Type Name

SAS_HDD_2.5 300G_15k_12G rate_2.5 inch SAS hard disk

300G_10k_12G rate_2.5 inch SAS hard disk







1T_7.2k_12G rate_2.5 inch SAS hard disk

1.2T_10k_12G rate_2.5 inch SAS hard disk

23

1.8T_10k_12G rate_2.5 inch SAS hard disk

2T_7.2k_12G rate_2.5 inch SAS hard disk

SATA_HDD_2.5 1TB_7.2k_6G rate_2.5 inch SATA hard disk

2TB_7.2k_6G rate_2.5 inch SATA hard disk

SATA_SSD_2.5 150GB_SATA SSD hard disk

240GB_SATA SSD hard disk



(support Micron 5100 series)



1.2TB_SATA SSD hard disk



(support Intel S4600 series)


For full NVMe hard disk configuration, hard disk sequence at delivery is shown in Figure

4-2.

Figure 4-3 NVMe Hard Disk Configuration

NVMe hard disk is used with built-in NVMe hard disk cable.

While doing RAID0/1/10 for NVME hard disk，RAID Key code is V08902F000000000

Raidkey_I_Stand_RAID0-1-10_Black

While doing RAID0/1/5/10 for NVME hard disk，RAID Key code is V08902E000000000

Raidkey_I_Prem_RAID0-1-5-10_Red

For NVLink GPU card and add-in card in upper front PCIe slot, only support 4 NVMe

hard disks.

For 1pcs CPU and NVLink GPU card, front NVMe hard disk is not supported.

For a single processor a PCIe GPU and add-in card in upper front PCIe slot, only

support 4 NVMe hard disks.

24

NVMe hard disk compatibility is shown in Figure 4-8.

Table 4-8 NVMe Hard Disk Compatibility List

Type Name

NVMe SSD

960GB_U.2 port NVMe hard disk

1TB_U.2 port NVMe hard disk



1.9TB_U.2 port NVMe hard disk

3.8TB_U.2 port NVMe hard disk

4.4.2 Built-in hard disk

NF5288M5 supports optional built-in 1 or 2 × M.2 SATA SSD with M.2 Riser card. building

soft RAID under BIOS, and RAID 0/1 mode, but does not support mixing two different M.2

SSDs.

M.2 SSD compatibility is shown in Figure 4-9.

Table 4-9 M.2 SSD Compatibility List

Type Name

M.2 port SATA bus SSD

80GB_M.2 port SATA bus SSD








4.5 PCIe expansion card

In addition to GPU card, NF5288M5 also supports PCIe extension cards with different

types and rates.

PCIe card collocation is as follows:

RAID Mezz card is installed on motherboard without occupying external PCIe slot.

For NVLink GPU, support up to 6 × external PCIe slot.

25

For PCIe GPU, support up to 2 × external PCIe slot.

For NVLink GPU configuration, support 4 × PCIe X16 HHHL rear, and 100G InfiniBand

card RDMA.

For 1 processor and NVLink GPU, support 4 x PCIE, do not support front PCIE and

NVME hard disk.

For 2 processors, NVLink GPU and 8 × NVMe hard disk, do not support front upper

PCIe slot.

For 1 processor and PCIE GPU, support front upside PCIE and 4 x NVME without front

downside PCIE support, or support 8 x NVME without front PCIE support.

Support mixing of Ethernet card, IB card, and OPA card support in any type and

quantity.

Current PCIe expansion card compatibility is shown in Table 4-10.

Table 4-10 PCIe Expansion Card Compatibility List

Type Name

RAID card

Inspur self-developed 2GB cache 3108 RAID Mezz card (support RAID or

direct pass-through)

Inspur self-developed 4GB cache 3108 RAID Mezz card (support RAID or

direct pass-through)

Ethernet

card

Mellanox_100G one-port network card (with optional 100G multi-mode optical

module)

Mellanox_100G two-port network card (with optional 100G multi-mode optical

module)

Mellanox_40G one-port network card（with optional 40G multi-mode optical

module）

Intel_XL710_40G one-port network card

Mellanox _25G one-port network card

(with optional 25G multi-mode optical module or special fixed-length cable with

optical module)

Mellanox _25G dual-port network card (with optional 25G multi-mode optical

module)

(with optional 25G multi-mode optical module or special fixed-length cable with

optical module)

Inspur self-developed 82599_10G dual - optical port network card (with

optional 10G SFP+ single-mode/multi-mode optical module)

Mellanox_10G dual - optical port network card (with optional 10G SFP+

single-mode/multi-mode optical module)

26

Intel_82599_10G dual - optical port network card

(including optical module)

I350 1G dual port RJ45

Intel X540 10G dual port RJ45

IB card

Mellanox EDR IB (100Gb/s) single-port card (with optional special fixed-length

cable including optical module)

Mellanox EDR IB (100Gb/s) dual-port card (with optional special fixed-length

cable including optical module)

Mellanox IB 56Gb/s single-port card(with optional special fixed-length cable

including optical module)

OPA card Intel OPA 100Gb/s single-port card (with optional special fixed-length cable)

4.6 PSU

NF5288M5 configure 2 power modules as default. While 8pcs 250/300W GPU configured

and running NVqual under heavy power test, the power can not meet 1+1 redundancy.

But under most training model of deep learning the power have 1+1 redundancy. Optional

power module available are shown in Table 4-11.

Table 4- 11 Power Supply Specification

Power mode Rated power Rated input voltage

3000W 80Plus Platinum PSU 3000W 100V AC - 240V AC

Ready for 240 v DC

Note: Standard G16-C19 interface or C19-C20 interface 16A AC power cord is needed.

And every power cord must need 1pcs power cord filter.

Power cord filter for NF5288M5 is shown in table 4-12.

Table 4-12 power cord filter

Model of power cord filter note

power filter_GW_16A_GW-CDM3000AA-EMI Match standard G16-C19 or C19-C20

interface 16A AC power cord，And every

power cord must need 1pcs power cord

filter.

Power cord filter is a low-pass filter that allows DC or 50Hz working current to

pass through, and does not allow high frequency EMI signals to pass through.

Power cord filter is bidirectional, not only prevent interference from the power

grid go into the equipment to produce adverse effects on the equipment,

making equipment meet requirements of conduction sensitivity ; but also

prevent EMI from the equipment go into the power grid through the power cord,

making equipment meet requirements of conduction emission.

27

4.7 Operating system

Operating system options supported by NF5288M5 are shown in Figure 4-12.

Table 4-12 Operating System Compatibility

part/mode

OS

RAID

card_S

AS3108

_2GB

RAID

card_S

AS3108

_4GB

sSATA

AHCI

mode

sSATA

RAID

mode

SATA

AHCI

mode

SATA

RAID

mode

NVMe

single

disk

mode

NVMe

RAID

mode

Microsoft Windows

Server 2012 R2 UEFI √ √ √ √ √ √ √ √

Microsoft Windows

Server 2012 R2

Legacy

√ √ √ √ √ √ N/A N/A

Microsoft Windows

Server 2016 UEFI √ √ √ √ √ √ √ √

Microsoft Windows

Server 2016 Legacy √ √ √ √ √ √ N/A N/A

Red Hat Enterprise

Linux 7 U3 64Bit UEFI √ √ √ √ √ √ √ √

Red Hat Enterprise

Linux 7 U3 64Bit

Legacy

√ √ √ √ √ √ N/A N/A

Red Hat Enterprise

Linux 7 U4 64Bit UEFI √ √ √ √ √ √ √ √

Red Hat Enterprise

Linux 7 U4 64Bit

Legacy

√ √ √ √ √ √ N/A N/A

CentOS 7.3 UEFI √ √ √ √ √ √ √ √

CentOS 7.3 Legacy √ √ √ √ √ √ N/A N/A

Ubuntu 14.04 UEFI √ √ √ √ √ √ √ √

Ubuntu 14.04 Legacy √ √ √ √ √ √ N/A N/A

Ubuntu 16.04 UEFI √ √ √ √ √ √ √ √

Ubuntu 16.04 Legacy √ √ √ √ √ √ N/A N/A

28

5 BMC management system

NF5288M5 integrates a new generation Inspur server BMC management system that is a

self-developed by Inspur for remote management of server. It conforms to server industry

management standard IPMI 2.0, and possesses highly reliable hardware monitoring and

management functions.

Main functions and features of Inspur server BMC management system are listed as

follows:

● Remote control

Server control is realized via functions such as KVM (Keyboard Video and Mouse), SOL

(Serial Over Lan) and virtual media.

Note: SOL function has to be realized via third-party tools such as IPMI Tool.

● Alarm management

Report alarm information in real time, and deal with it accordingly.

● Condition monitor

Monitor various operating conditions of each monitoring unit in real-time.

● Equipment information management

Provide equipment version information, model and asset information function.

● Heat radiation control

Adjust fan speed dynamically according to ambient temperature and workload.

● Support IPMI Tool management

Support commands from IPMI Tool. Download IPMI Tool at will.

Note: IPMI Tool download website: http://ipmitool.sourceforge.net/manpage.html

● Support WEB interface

Provide friendly visual interface management, allowing quick setting and query by clicking.

● Support centralized account management

Support centralized account storage in Active Directory server, complete authentication

process in server, and allow domain account to log in management system.

29

5.1 Function module

Inspur Server BMC management system consists of IPMI module, command line module,

WEB module, KVM Over IP, virtual media, etc.

● Command line module allows calling IPMI module. Users can operate IPMI module via

command line.

● WEB module completes routine management of server in the form of a visual interface

by calling IPMI command, integrates KVM and virtual media.

IPMI module:

IPMI module completes the server system management function according to IPMI2.0

standard.

IPMI module support following functions:

● Real-time system monitor

In case of failure detected, report and indicate alarms.

● Remote control of system

Remotely power on/off and reset system, etc. via command line and Web.

Command line module:

Command line module consists of query and setup commands of network, sensor, fan,

user management, system, server, etc.

Remote control module:

●KVM Over IP: Users monitor and control remote devices by using local video, keyboard

and mouse on client, so as to operate remote devices in real time.

● Virtual media: Provide remote access to local media (CD-ROM, floppy disk drive, or

compact / floppy disk mirror image file) by means of virtual CD drive and floppy drive on

server.

Remote control function needs corresponding browser version and Java operating

environment.

Note:

If Java operating environment fails to meet requirements, download from

http://www.oracle.com/technetwork/java/javase/downloads/index.html

http://www.oracle.com/technetwork/java/javase/downloads/index.html

30

6 Certification and License

Inspur NF5288M5 Supercomputer White Paper · 2019-05-23 · 1 1 Product Overview Inspur NF5288M5...

Documents

Transcript of Inspur NF5288M5 Supercomputer White Paper · 2019-05-23 · 1 1 Product Overview Inspur NF5288M5...