PAN Cluster NeSI @ The University of Auckland Centre for ... · PDF fileAdded 10G Core to 1G...
Transcript of PAN Cluster NeSI @ The University of Auckland Centre for ... · PDF fileAdded 10G Core to 1G...
IS5200
4036E-1 Test Rack 1 4036E-2 Test Rack 2
IS5300
SX6036-4
SX6036-3SX6036-2SX6036-1
A iDataplexB iDataplexA2 A3C2 D iDataplex C3 C iDataplex Test Rack 2
10G G8124-3 10G G8124-4
1G G
8000
1G G
8000
1G G
8052
1G G
8000
10G
G81
24-2
10G
G81
24-1
40Gb/s Infiniband
1Gb/s Management,Provisioning &
GPFS Cntl
UOACampus
TSM
KAREN
Test Rack 1
9.6T Metadata24 x SSD 400G
144T48 x 7.2K 3T21.6T NeVE
24 x 10K 900G120T
60 x 7.2K 2T36T
60 x 15K 600G
Quad connected8G FC
T1DS1 T2DS1
120T60 x 7.2K 2T
T2DS2 NeVE + T2DS4T1DS2 SSDDS1T2DS3
96T24 x 7.2K 1T
GPFS NSD 1 GPFS NSD 2 GPFS NSD 3 GPFS NSD 4 GPFS Meta NSD 5 GPFS Meta NSD 6
1G Firewall x 2
4T Metadata10 x SSD 400G
3.5T24 x 15K 146G
7.2T24 x 15K 300G
Stage 1A Rack iDataplex 80 M3 Nodes, with 96G RAM 960 Westmere Cores 4096 Fermi GPU CoresIS5200 infiniband switchsimple 1G management network280T Storage in 4 SANSFour GPFS NSD ServersFour NeVE Servers3 Service nodes
Jan 2012
Stage 2AB Rack iDataplex 8 M3 Nodes with 96G RAM 68 M4 Nodes with 128G RAM 96 Westmere Cores 1088 Sandybridge Cores 8192 Fermi GPU CoresAdded leaf cards to IS5200Expanded 1G management networkAdded 7.2T to Metadata SANAdded 22T NeVE SANDual connected Fibre Channel
Stage 2BC Rack iDataplex 60 M4 nodes, each with 128G Ram 960 Sandybridge Cores 4096 Fermi GPU CoresAdded 4x36 Port SX6036 IB switchesExpanded 1G management networkAdded 96T SANRestructured GPFS filesystem
Stage 3D Rack iDataplex 77 M4 nodes, with 128G Ram 1232 Sandybridge Cores 24,960 Kepler GPU Cores 240 Intel Phi Cores3 x 512G 20 Core Westmere nodesAdditional xcat nodeAdded IS5300 IB switchRedesigned infiniband networkAdded 10G Core to 1G Mng NetworkAdded 144T to NeVE SANAdded 9.6T SSD SAN for MetadataAdded 4T SSD to existing Metadata SANQuad connected Fibre ChannelAdded dedicated Metadata GPFS NSDs
Aug 2012
Jan 2013
May 2013
Moved DatacentreMoved From Ak City to Tamaki
Nov 2012
PAN Cluster NeSI @ The University of Auckland Centre for eResearch
Infiniband100ns switch delay300ns max delayMPI latency measured at 6µs
Core Count1116 Westmere3280 Sandybridge16384 Fermi24960 Kepler240 Phi
Dual 10Gb/sManagement CoreInfiniband