Power Measurements on MiraArgonne Leadership Computing FacilitySusan CoghlanMira Project ManagerArgonne National Laboratory

June 18, 2012

The Mira System

Argonne National Laboratory - ISC 2012

2

Mira Statistics

10 PF Peak 8.1 PF HPL

48 Racks 48K compute nodes

768K cores 3.1 B threads

768 TB memory 16 GB memory per node

28 PB disk 240 GB/s

4.8MW peak 48 racks at 100kw each

Mira 480V Compute Power Distribution System

Argonne National Laboratory - ISC 2012

3

13.2KVSubstations Transformers

480V

DistributionPanels

EatonDigitripOptim 1050

BG/Q Compute Racks

Measuring points

Bulk PowerModules

(BPM)

NodesDCAs7 Domains

Bulk PowerModules

(BPM)

NodesDCAs7 Domains

Bulk PowerModules

(BPM)

NodesDCAs7 Domains

36

36

36

64

64

64

1024

1024

1024

ALCF Power Measurement Methodology Challenges Challenge #1: Length of time required for HPL run

– Mira integration not complete, system still in hands of vendor– Difficult to justify so many hours just to measure power– Recommend using a benchmark that runs in an hour or two

Challenge #2: Access to external AC power data required for L2 and L3– Building and all infrastructure is owned by a 3rd party Trust– Distribution panel power measurement data only visible to the building management

company – All contact with building management company must go through the Argonne Data

Center Manager (ADCM)– Still no data after months of working through ADCM– Attempted to find boxes to measure between panels and racks

• Impossible to find existing boxes – 480V 3 Phase, receptacle HBL560R7W • Custom ones cost just under $10K per rack to rent

Argonne National Laboratory - ISC 2012

4

Mira 480V Compute Power Distribution System

Argonne National Laboratory - ISC 2012

5

13.2KVSubstations Transformers

480V

DistributionPanels

EatonDigitripOptim 1050

BG/Q Computes

Measuring points

Bulk PowerModules

(BPM)

NodesDCAs7 Domains

Bulk PowerModules

(BPM)

NodesDCAs7 Domains

Bulk PowerModules

(BPM)

NodesDCAs7 Domains

36

36

36

64

64

64

1024

1024

1024Accessible to ALCF

Data from 48 racks running HPL

Argonne National Laboratory - ISC 2012

6