Cray Corporate Update
Transcript of Cray Corporate Update
![Page 1: Cray Corporate Update](https://reader034.fdocuments.net/reader034/viewer/2022051505/586a90c01a28ab063d8bbfb1/html5/thumbnails/1.jpg)
![Page 2: Cray Corporate Update](https://reader034.fdocuments.net/reader034/viewer/2022051505/586a90c01a28ab063d8bbfb1/html5/thumbnails/2.jpg)
(Jaguarpf/Kraken/Hopper)
3
![Page 3: Cray Corporate Update](https://reader034.fdocuments.net/reader034/viewer/2022051505/586a90c01a28ab063d8bbfb1/html5/thumbnails/3.jpg)
Microkernel on Compute PEs, full featured Linux on Service PEs.
Service PEs specialize by function
Software Architecture eliminates OS “Jitter”
Software Architecture enables reproducible run times
Large machines boot in under 30 minutes, including filesystem
Service PartitionSpecialized Linux nodes
Compute PE Login PE
Network PE
System PE
I/O PE
4
![Page 4: Cray Corporate Update](https://reader034.fdocuments.net/reader034/viewer/2022051505/586a90c01a28ab063d8bbfb1/html5/thumbnails/4.jpg)
GigE
10 GigE
GigE
RAIDSubsystem
Fibre Channels
SMW
Compute node
Login nodeNetwork node
Boot/Syslog/Database nodes
I/O and Metadata nodes
X
ZY
5
![Page 5: Cray Corporate Update](https://reader034.fdocuments.net/reader034/viewer/2022051505/586a90c01a28ab063d8bbfb1/html5/thumbnails/5.jpg)
6.4 GB/sec direct connect HyperTransport
CraySeaStar2+
Interconnect
25.6 GB/sec direct connect memory
Characteristics
Number of Cores 8 or 12
Peak PerformanceShanghai (2.4)
76 Gflops/sec
Peak Performance Istanbul (2.6)
124 Gflops/sec
Memory Size 16 or 32 GB per node
Memory Bandwidth
25.6 GB/sec
6
![Page 6: Cray Corporate Update](https://reader034.fdocuments.net/reader034/viewer/2022051505/586a90c01a28ab063d8bbfb1/html5/thumbnails/6.jpg)
Cray XT5 systems ship with the SeaStar2+ interconnect
Custom ASIC Integrated NIC / RouterMPI offload engineConnectionless ProtocolLink Level ReliabilityProven scalability to 225,000 cores
HyperTransport Interface
Memory
PowerPC440 Processor
DMAEngine6Port
Router
BladeControl
ProcessorInterface
Now Scaledto 225,000
cores
7
![Page 7: Cray Corporate Update](https://reader034.fdocuments.net/reader034/viewer/2022051505/586a90c01a28ab063d8bbfb1/html5/thumbnails/7.jpg)
8
![Page 8: Cray Corporate Update](https://reader034.fdocuments.net/reader034/viewer/2022051505/586a90c01a28ab063d8bbfb1/html5/thumbnails/8.jpg)
9
![Page 9: Cray Corporate Update](https://reader034.fdocuments.net/reader034/viewer/2022051505/586a90c01a28ab063d8bbfb1/html5/thumbnails/9.jpg)
High
Vel
ocity
Airf
low
High
Vel
ocity
Airf
low
Low
Velo
city A
irflo
w
Low
Velo
city A
irflo
w
Low
Velo
city
Airf
low
10
![Page 10: Cray Corporate Update](https://reader034.fdocuments.net/reader034/viewer/2022051505/586a90c01a28ab063d8bbfb1/html5/thumbnails/10.jpg)
11
![Page 11: Cray Corporate Update](https://reader034.fdocuments.net/reader034/viewer/2022051505/586a90c01a28ab063d8bbfb1/html5/thumbnails/11.jpg)
Hot air stream passes through evaporator, rejects heat to R134a via liquid‐vapor phase change
(evaporation).
R134a absorbs energy only in the presence of heated air.
Phase change is 10x more efficient than pure water cooling.
Liquid/Vapor Mixture out
Liquid in
Cool air is released into the computer room
12
![Page 12: Cray Corporate Update](https://reader034.fdocuments.net/reader034/viewer/2022051505/586a90c01a28ab063d8bbfb1/html5/thumbnails/12.jpg)
R134a piping Exit Evaporators
Inlet Evaporator
13
![Page 13: Cray Corporate Update](https://reader034.fdocuments.net/reader034/viewer/2022051505/586a90c01a28ab063d8bbfb1/html5/thumbnails/13.jpg)
14
![Page 14: Cray Corporate Update](https://reader034.fdocuments.net/reader034/viewer/2022051505/586a90c01a28ab063d8bbfb1/html5/thumbnails/14.jpg)
New enhanced blower to handle the 130 Watt Magny‐Cours Processor
Enhanced sound kit to reduce noise
More efficient designNew VFD (Variable Frequency Diode) for blower
An upgrade kit product code will be available for existing XT5 customers which will contain the required components
15
![Page 15: Cray Corporate Update](https://reader034.fdocuments.net/reader034/viewer/2022051505/586a90c01a28ab063d8bbfb1/html5/thumbnails/15.jpg)
16
Air taken from top, no line of sight for sound
Foam lined duct for sound absorption
Extra foam added to front. Door now seals to front IO extension
![Page 16: Cray Corporate Update](https://reader034.fdocuments.net/reader034/viewer/2022051505/586a90c01a28ab063d8bbfb1/html5/thumbnails/16.jpg)
(Jaguar/Athena/Franklin)
17
![Page 17: Cray Corporate Update](https://reader034.fdocuments.net/reader034/viewer/2022051505/586a90c01a28ab063d8bbfb1/html5/thumbnails/17.jpg)
2 – 8 GB
12.8 GB/sec direct connect memory(DDR 800)
6.4 GB/sec direct connect HyperTransport
CraySeaStar2+Interconnect
4-way SMP >35 Gflops per
node Up to 8 GB per
node OpenMP Support
within socket
18
![Page 18: Cray Corporate Update](https://reader034.fdocuments.net/reader034/viewer/2022051505/586a90c01a28ab063d8bbfb1/html5/thumbnails/18.jpg)
4 DIMM Slotswith Chipkill
CRAYSeaStar™
CRAYSeaStar™
CRAYSeaStar™
CRAYSeaStar™
Redundant VRMs
Embedded HyperTransport Link
Blade Control Processor
Blade Backplane Connector (>100 GB/sec)
19
![Page 19: Cray Corporate Update](https://reader034.fdocuments.net/reader034/viewer/2022051505/586a90c01a28ab063d8bbfb1/html5/thumbnails/19.jpg)
20
![Page 20: Cray Corporate Update](https://reader034.fdocuments.net/reader034/viewer/2022051505/586a90c01a28ab063d8bbfb1/html5/thumbnails/20.jpg)
Site specificPublic DomainISV Applications
21
Cray Software Ecosystem
Applications
Compilers
Debuggers
Schedulers
Tools
OS
CrayPatCray Apprentice
LibrariesPublic Domain Tools
Cray Linux Enviroment
![Page 21: Cray Corporate Update](https://reader034.fdocuments.net/reader034/viewer/2022051505/586a90c01a28ab063d8bbfb1/html5/thumbnails/21.jpg)
Service nodes run a full‐featured SLES10 Linux installation We add our tools, libraries, and services
Compute nodes run a slim‐line Linux kernel with only necessary services Only run what’s needed so the application can rule the roost
Libraries MPT –Message Passing Toolkit LibSci – Cray Scientific Libraries (BLAS, LAPACK, SCALAPACK, FFTW, etc) I/O Libraries – HDF5 & NetCDF
Tools Compilers – PGI, Cray, GNU, Pathscale, Intel CrayPAT – Performance Analysis Tools
ALPS Application placement, job launching, application clean‐up Users interface with ALPS primarily via aprun
PBS/TORQUE & MOAB All jobs on the local XTs are batch jobs MOAB is an advanced job scheduler that is used on Jaguar and Kraken
22
![Page 22: Cray Corporate Update](https://reader034.fdocuments.net/reader034/viewer/2022051505/586a90c01a28ab063d8bbfb1/html5/thumbnails/22.jpg)
Parallel Data Virtualization Service supportScalable Dynamic LibrariesVirtual Cluster EnvironmentCore Specialization for codes with high synchronization requirements
NodeKARE (Node Knowledge and Reconfiguration) resiliency features
Checkpoint / Restart
23
![Page 23: Cray Corporate Update](https://reader034.fdocuments.net/reader034/viewer/2022051505/586a90c01a28ab063d8bbfb1/html5/thumbnails/23.jpg)
Compute NodeDVS Client
Compute NodeDVS Client
Compute NodeDVS Client
Compute NodeDVS Client
Compute NodeDVS Client
DVS ServerSIO Node 4
SeaStar Interconnect
DVS ServerSIO Node 4
DVS ServerSIO Node 4
DVS ServerSIO Node 4
StornextServers &Storage
Stornext Client
GPFSServers &Storage
GPFS Client
NFSServers &Storage
IB or 10GigE
NFS Client
Servers & Storage
Panfs Client
Cray XTSystem
24
![Page 24: Cray Corporate Update](https://reader034.fdocuments.net/reader034/viewer/2022051505/586a90c01a28ab063d8bbfb1/html5/thumbnails/24.jpg)
Benefit: root file system environment available to applications
Shared root from SIO nodes will be available on compute nodes
Standard libraries / tools will be in the standard places
Able to deliver customer‐provided root file system to compute nodes
Programming environment will support static and dynamic linking
Performance impact negligible, due to scalable implementation
25
![Page 25: Cray Corporate Update](https://reader034.fdocuments.net/reader034/viewer/2022051505/586a90c01a28ab063d8bbfb1/html5/thumbnails/25.jpg)
DisklessCompute Node 0/dvs
DisklessCompute Node 1/dvs
DisklessCompute Node 5000
/dvs
DisklessCompute Node 2/dvs
DisklessCompute Node 3/dvs
DVS Server Node 0
Requests for shared libraries (.so files) are routed through DVS Servers
Provides similar functionality as NFS,but scales to 1000s of compute nodes
Central point of administration for shared libraries
DVS Servers can be “re‐purposed” compute nodes
SeaStar Interconnect
NFSSharedLibraries
26
![Page 26: Cray Corporate Update](https://reader034.fdocuments.net/reader034/viewer/2022051505/586a90c01a28ab063d8bbfb1/html5/thumbnails/26.jpg)
Capacity/Production (Mid-weight Linux Image)
LessCompatibility
FullCompatibility
27550
27750
27950
28150
28350
0 1 2 3
Capability (Ultra-light Linux Image)
Shrink-wrap3rd Party Application
(Full Linux Image and all services)
High Scale
Low Scale
![Page 27: Cray Corporate Update](https://reader034.fdocuments.net/reader034/viewer/2022051505/586a90c01a28ab063d8bbfb1/html5/thumbnails/27.jpg)
Benefit: Eliminate noise with overhead (interrupts, daemon execution) directed to a single core
Rearranges existing work Without core specialization: overhead affects every core With core specialization: overhead is confined, giving app exclusive access to remaining cores
Helps some applications, hurts others POP 2.0.1 on 8K cores on XT5: 23% improvement Larger jobs should see larger benefit Future nodes with larger core counts will see even more benefit
This feature is adaptable and available on a job‐by‐job basis
28
![Page 28: Cray Corporate Update](https://reader034.fdocuments.net/reader034/viewer/2022051505/586a90c01a28ab063d8bbfb1/html5/thumbnails/28.jpg)
Feature Also Known As “Node Health Checker” Benefit: verify that nodes are healthy so that jobs are not started on unhealthy nodes, that is, improved application completion rates
Checks more possible sources of error: file system checks, memory usage, application termination, site‐specific check
Configurable: when to run, what to do on errors, callout to site‐specific script
Suspect Mode minimizes burden on administrator Future release will dump and restart downed nodes
Checkpoint / restart Released in CLE 2.2 (Jul 09) Supported by PBS Pro (10.1 or later) and Moab/Torque MPI and SHMEM
29
![Page 29: Cray Corporate Update](https://reader034.fdocuments.net/reader034/viewer/2022051505/586a90c01a28ab063d8bbfb1/html5/thumbnails/29.jpg)
Cray continues the partnership with PGI to provide compilers on XT
Cray Compilation Environment UPC implementation Co‐Array implementation Smooth transition to Cascade Laying support for integrating accelerators
Intel compiler also available for XT systemsCray acquired Pathscale TechnologySupport for dynamic libraries and ISV codes
30
![Page 30: Cray Corporate Update](https://reader034.fdocuments.net/reader034/viewer/2022051505/586a90c01a28ab063d8bbfb1/html5/thumbnails/30.jpg)