Post on 31-Mar-2015
Company ProprietaryKarl Kortkamp, SE, SGI
New Mexico Computing Applications Center(NMCAC)
System Diagrams&
Physical Layouts
Company ProprietaryKarl Kortkamp, SE, SGI
New Mexico Computing Applications Center HPC for New Mexico
Company ProprietaryKarl Kortkamp, SE, SGI
Overall Top-level Diagram
10 Gig-E
EncantoAltix ICE System
14,336 Compute Cores1,792 Nodes
28 Racks
UNMAltix ICE Exemplar176 Compute Cores
22 Nodes1 Rack
NMSUAltix ICE Exemplar176 Compute Cores
22 Nodes1 Rack
New Mexico TechAltix ICE Exemplar176 Compute Cores
22 Nodes1 Rack
Company ProprietaryKarl Kortkamp, SE, SGI
Main System “Encanto” Overview
Major Components
• 172 TFLOPS Altix ICE 8200 Cluster
• 1792 Nodes, 3,584 Sockets, 14,336 Cores
• 28.7 TB memory (2GB/core)
• 172 TB Parallel File Subsystem (Lustre)
• 20 OSS Service Nodes
• 20 TB NAS & Archive Storage Subsystem
• 1 System Admin Controller
• 28 Rack Leader Controllers
• 1 Batch Service Node
• 5 Login Service Nodes
• Lambda Rail Connectivity
• 36 Racks – 28 Compute – 5 Storage – 3 Support
Company ProprietaryKarl Kortkamp, SE, SGI
Encanto Top-level Diagram
33 IB Lines
1Lamba Rail
Fiber
120
Lambda RailGateway
5
Login ServiceNodes
IB DDRGig-E 10 Gig-E
Login ServiceNodesLogin Service
NodesLogin ServiceNodesLogin Service
Nodes
Qty 5
11System Admin
ControllerBatchNode
20
Lustre-Based ParallelStorage Sub-system
172 TB Usable
5
2
4
NAS CUBE / ArchiveStorage Sub-systemNAS: 20 TB Usable
Arch: 40TB Uncompressed
10
Altix ICE 8200 Cluster14,336 Compute Cores
1,792 Nodes28 Racks
Company ProprietaryKarl Kortkamp, SE, SGI
NMCAC Qwest Fiber Connections to New Mexico “GigaPop”
505 MarquetteAlbuquerque, NM
“Lambda Rail”Point of Presence
4100 Sara RoadRio Rancho, NM
“Encanto”Supercomputer
Existing National Lamba RailFiber Ring
Company ProprietaryKarl Kortkamp, SE, SGI
1 3
Encanto Lambda-Rail Router DiagramLamba Rail
Fiber
IB DDRGig-E 10 Gig-E
10
Future Expansion
21
21
21
21
21
Gateway/Router Cisco 6509 10-GigE Router
5
Login Service Node
Login Service Node
Login Service Node
Login Service Node
Login Service Node
Altix ICE 8200 Cluster14,336 Compute Cores
1,792 Nodes28 Racks
5
1
1
1
1
1
Company ProprietaryKarl Kortkamp, SE, SGI
Login Node
Login Node
Login Node
Login Node
Login Node
NMCAC Switch• Owned by NMCAC• Router between Fibre & Encanto• Redundant AC Power • Routing module to 5 Login Nodes• Load-leveling Module
NMCAC Switch• Owned by NMCAC• Redundant DC Power
Cisco
6509 Sw
itch
Qw
es
t Ge
om
ax
Qw
es
t G
eo
ma
x
Cis
co 6
509
Sw
itch
Cis
co 6
513
Sw
itch
UNM Switch• Owned by UNM• Connected to NLR• Connects to NMSU• Connects to NMTECH• Redundant DC Power
UNM
NMTECH
NMSU
NationalLamba Rail
Encanto External Network Diagram
505 Marquette Site 4100 Sara Road Site
Company ProprietaryKarl Kortkamp, SE, SGI
…
Encanto Lustre-Based Storage System
10 Lustre Storage Units
Lustre Storage Unit
1
1
20
SASIB DDRGig-EHeartbeat FC4
Altix ICE 8200 Cluster14,336 Compute Cores
1,792 Nodes28 Racks
Meta-data Servers
IS220
6015B
6015B
IS4500OST
XE310 Node
OSS1
OSS1
XE310 Node
IS4500OST
XE310 Node
OSS1
OSS1
XE310 Node
IS4500OST
XE310 Node
OSS1
OSS1
XE310 Node
IS4500OST
XE310 Node
OSS1
OSS1
XE310 Node
Place OSS XE310 servers on topof the following ICE racks
(5, 6, 7, 8, 9, 10, 11, 12, 24, 25)
Company ProprietaryKarl Kortkamp, SE, SGI
Object StorageServers
Encanto Lustre Storage Unit
IS4500Dual-controller 2GB Cache
FC FC
1 1
Object StorageTargets
12x ( 4 + 1 ) LUNs + 4 Hot SparesAcross 4 FC Loops
FC RAIDDrive Tray
Loop
FC RAIDDrive Tray
Loop
FC RAIDDrive Tray
Loop
FC RAIDDrive Tray
Loop
Altix XE 310 Node2x3 GHz quad-core
16GB memory
FC
1
IB
Altix XE 310 Node2x3 GHz quad-core
16GB memory
FC
1
IB
1 1
64x 400GB 15K rpmFibre Channel drives
Company ProprietaryKarl Kortkamp, SE, SGI
OSSs
Lustre Storage Cabling Details
OSTs
IS4500Dual-controller
FC FC
FC RAIDDrive Tray
FC4
FC RAIDDrive Tray
FC4
FC RAIDDrive Tray
FC4
FC RAIDDrive Tray
FC4
Altix XE 310Node
FC
1
IB
Altix XE 310Node
FC
IS4500Dual-controller
FC FC
FC RAIDDrive Tray
FC4
FC RAIDDrive Tray
FC4
FC RAIDDrive Tray
FC4
FC RAIDDrive Tray
FC4
Altix XE 310Node
FC
Altix XE 310Node
FC
1
IB
1
IB
1
IB
IB DDRHeartbeat FC4Gig-E
Company ProprietaryKarl Kortkamp, SE, SGI
Altix 450
Combined NAS & Archive Storage Subsystem
Altix 450 Blade EnclosureAtempo Archive Server
SL500Dual-LT03 Tape Drives
FC FC
FC FC
Control Line Data Lines
TapeCartridgeTape
CartridgeTapeCartridgeTape
CartridgeTapeCartridgeTape
Cartridge
TapeCartridgeTape
CartridgeTapeCartridgeTape
CartridgeTapeCartridgeTape
Cartridge
TapeCartridgeTape
CartridgeTapeCartridgeTape
CartridgeTapeCartridgeTape
Cartridge
TapeCartridgeTape
CartridgeTapeCartridgeTape
CartridgeTapeCartridgeTape
Cartridge
Altix 450 Blade EnclosureNAS Head Node
IS4000RAID-controller
FC FC
FC FC
4
IB
NUMAlink
1 2 11
SATA RAIDDrive Tray
FC
SATA RAIDDrive Tray
FC
SATA RAIDDrive Tray
FC
Company ProprietaryKarl Kortkamp, SE, SGI
Altix ICERack014
(Front)
Altix ICERack013
(Front)
Altix ICERack015
(Front)
Altix ICERack016
(Front)
Altix ICERack006
(Front)
Altix ICERack005
(Front)
Altix ICERack007
Altix ICERack008
(Front)
Altix ICERack010
(Front)
Altix ICERack009
Altix ICERack011
(Front)
Altix ICERack012
(Front)
(Front)
ArchiveRack
{Front)028
Altix ICERack
{Front)027
Altix ICERack
{Front)026
Altix ICERack
{Front)025
Altix ICERack
{Front)020
Altix ICERack
{Front)019
Altix ICERack
{Front)018
Altix ICERack
{Front)017
Altix ICERack
{Front)024
Altix ICERack
{Front)023
Altix ICERack
{Front)022
Altix ICERack
{Front)021
Altix ICERack
(Front)Storage
Rack001
(Front)Storage
Rack002
(Front)Storage
Rack003
(Front)Storage
Rack004
(Front)Storage
Rack005
Encanto Floor Plan
{Front)SWITCH
Rack
Altix ICERack002
(Front)
Altix ICERack001
(Front)
Altix ICERack003
(Front)
Altix ICERack004
(Front)
ADMINRack
(Front)
Lustre IB connections Chilled Water & CondensateIndicates overhead cable trellis
XE310
XE310
XE310 XE310 XE310XE310XE310
XE310
XE310XE310
Company ProprietaryKarl Kortkamp, SE, SGI
Encanto Compute Rack (28)
Rack Leader Ctrl – 6015B
Lustre OSS I/O Service Nodes(XE310 servers) are placed on top of the following ICE racks
(5, 6, 7, 8, 9, 10, 11, 12, 24, 25)
Altix ICE 8200 Blade Enclosure (16 blades)
Altix ICE 8200 Blade Enclosure (16 blades)
Altix ICE 8200 Blade Enclosure (16 blades)
Altix ICE 8200 Blade Enclosure (16 blades)
Altix ICERack
(Front)
Company ProprietaryKarl Kortkamp, SE, SGI
Encanto Admin Rack
Sys Admin Ctrl – 6015BLogin Node – 6015B
Batch Node – 6015B
Login Node – 6015BLogin Node – 6015BLogin Node – 6015BLogin Node – 6015B
Administrator Console
Altix 450 Blade Enclosure
IS4000 Drive Tray (16 drives)
IS4000 Drive Tray (16 drives)
NAS CUBE & Archive
Main Cluster Service Nodes
IS4000 Controller
Altix 450 Blade Enclosure
ADMINRack
(Front)
Company ProprietaryKarl Kortkamp, SE, SGI
MDS Node – 6015BMDS Node – 6015B
IS220 (12 drives)
Encanto Switch Rack
{Front)Support
ICERack
Sys Admin Ctrl (Test) – 6015BLogin Node (Test) – 6015B
Rack Leader Ctrl (Test) – 6015B
Administrator Console (Test)
Lustre Meta Data Server
Test ICE SystemAltix ICE 8200 Blade Enclosure
(Test) (16 blades)
Cisco 6509 10-GigE Switch
Company ProprietaryKarl Kortkamp, SE, SGI
Encanto Luster Storage Rack (5)
IS4500 Controller
IS4500 Drive Tray (16 drives)
IS4500 Drive Tray (16 drives)
IS4500 Drive Tray (16 drives)
IS4500 Drive Tray (16 drives)
IS4500 Drive Tray (16 drives)
IS4500 Drive Tray (16 drives)
IS4500 Drive Tray (16 drives)
IS4500 Drive Tray (16 drives)
IS4500 Controller Lustre Storage Unit
StorageRack
(Front)
Lustre Storage Unit
Company ProprietaryKarl Kortkamp, SE, SGI
Encanto Archive Library Cartridge Rack
(Front)
ArchiveRack
STK SL500
Cartridge Expansion Unit
Archive & BackupCartridge Storage Units
Company ProprietaryKarl Kortkamp, SE, SGI
Major Components
• 2.1 TFLOPS Altix ICE 8200 Cluster
• 22 Nodes, 44 Sockets, 176 Cores
• 352 TB memory (2GB/core)
• 16 TB NAS Storage Subsystem
• 1 System Admin Controller
• 1 Rack Leader Controller
• 1 Login Service Node
• Lambda Rail Connectivity
• 1 Consolidated Rack
Exemplar System Overview
Company ProprietaryKarl Kortkamp, SE, SGI
Exemplar Top-level Diagram
33 IB Lines
1Lamba Rail
Fiber
Lambda RailGateway
33
IB DDRGig-E 10 Gig-E
Login ServiceNode
System AdminController
4
Altix ICE 8200 Cluster176 Compute Cores
22 Nodes1 Rack
2
NASStorage Subsystem
16 TB Raw
Company ProprietaryKarl Kortkamp, SE, SGI
Altix 450NAS Head Node
Exemplar NAS Storage Subsystem
4
IB
IS4000RAID-controller
FC FC
FC FC
11
SATA RAIDDrive Tray
FC
SATA RAIDDrive Tray
FC
Company ProprietaryKarl Kortkamp, SE, SGI
Exemplar System Rack
Rack Leader Controller – 6015B
Altix ICE 8200 Blade Enclosure (11 blades)
Administrator Console
Altix 450
NAS Storage CUBEIS4000 Controller and 16 Drives
Altix ICE 8200 Blade Enclosure (11 blades)
Sys Admin Controller – 6015BLogin Node – 6015B
Drive Tray (16 Drives)
Altix ICE Cluster