guillimin@ March 16, 2017 Guillimin HPC ... · PDF fileGuillimin HPC Users Meeting ... •...

Guillimin HPC Users Meeting - March 2017

Guillimin HPC Users MeetingMarch 16, 2017

[email protected]

McGill University / Calcul Québec / Compute CanadaMontréal, QC Canada


• Please be kind to your fellow user meeting attendees • Limit to two slices of pizza per person to start please• And please recycle your pop cans.• Thank you!

2


• Compute Canada News• System Status• Software Updates• Training News• Special Topic

• Best Practices for Job Submission

Outline

3


• 2017 Resource Allocation Competitions• Scientific reviews completed• Announcement of Awards: Soon!• Implementation of Awards: Mid April 2017• Important notes:

• There will be some small number of migrations of allocations, either partial or full, from Guillimin to other Compute Canada systems (such as the new Cedar and Graham)

• Once announced, we can help answer any questions regarding migration if your allocation is on a different system

Compute Canada News

4


• 2017 High Performance Computing Symposium• HPCS 2017 - June 5th – 9th• Queen's University, Kingston (Ontario)• http://2017.hpcs.ca/• Call for papers and posters: submissions due April 17

Compute Canada News

5

http://2017.hpcs.ca/

http://2017.hpcs.ca/


• March 6 - GPFS unresponsive on login nodes• Caused by long waiters in GPFS

• GPFS communications unable to complete their actions• Source: Infiniband communications issues between some

worker nodes and the rest of the cluster, which can have adverse effects on general GPFS functions

• Problematic nodes were identified and removed from the cluster network

• GPFS waiters were cleaned and regular access restored in the afternoon of March 6

System Status

6


• Upcoming scheduled power maintenance: April 23-29• Precise dates within the week still to be confirmed• Major power maintenance by ETS to upgrade 25kV

feeds to campus and therefore to the HPC Centre• Impact

• Significantly reduced or no access to worker nodes• To be Confirmed: Guillimin storage and login nodes may

be placed on generator so as to enable access to data during that week

• Recommendations• Attempt to complete any important project beforehand• If you need to work on any code or data during that week,

make sure to keep a copy at another site, when feasible• More details to be announced soon

System Status

7


Please use “module spider modulename” for load instructions.• Java/1.6.0_24 - Programming language (old version for compatibility)• Python/{2.7.12, 3.5.2} - Programming language• NAMD/2.9-PACE - Molecular dynamics code with PACE force field support• Stacks/1.45 - Pipeline for building loci from short-read sequences• SAMtools/1.3.1 - Manipulates alignments in the SAM format• HDF5/1.8.15-serial - Library for storing and managing data (no-MPI version)• Bazel/0.4.4 - Build tool for Tensorflow• Tensorflow/{0.10.0-Python-2.7.10, 0.10.0-Python-2.7.12, 1.0.0-Python-2.7.12,

1.0.0-Python-3.5.2} - Package for machine learning• Vim/8.0 - The ubiquitous text editor (Vi Improved)• tmux/2.3 - Terminal multiplexer• RandomLib/1.10 - Library for random numbers

New Software Installations

8


• All upcoming events: calculquebec.eventbrite.ca• March 23 - Introduction to OpenMP (McGill)• Apr. 4 - Analyse de données massives avec Spark (U. Laval)• Apr. 6 - Programmation en R intermédiaire (UdeM)• May 15-19 - https://ecole2017.calculquebec.ca/

• Recently completed:• Feb. 22 - Data Analysis in Ecology, R/Python (UQAM)• Mar. 2, 9 - Software Carpentry, Python (McGill)• Mar. 7, 8 - Software Carpentry, Python (U. Laval)• Mar. 15 - Introduction a OpenMP (U. Sherbrooke)

• All materials from previous workshops are available online: wiki.calculquebec.ca/w/Formations/en

• All user meeting presentations online at www.hpc.mcgill.ca

Training News

9

http://calculquebec.eventbrite.ca

https://ecole2017.calculquebec.ca/

https://wiki.calculquebec.ca/w/Formations/en

http://www.hpc.mcgill.ca


• Questions? Comments?• We value your feedback. Contact us at:

[email protected]

• Guillimin Operational News for Users– Status Pages

• http://www.hpc.mcgill.ca/index.php/guillimin-status• http://serveurscq.computecanada.ca (all CQ systems)

– Follow us on Twitter• http://twitter.com/McGillHPC

User Feedback and Discussion

10

mailto:[email protected]

mailto:[email protected]

http://www.hpc.mcgill.ca/index.php/guillimin-status

http://www.hpc.mcgill.ca/index.php/guillimin-status

http://serveurscq.computecanada.ca

http://serveurscq.computecanada.ca

http://twitter.com/McGillHPC





McGill University / Calcul Québec / Compute CanadaMontréal, QC Canada

Best Practices for Job SubmissionMarch 16, 2017

[email protected]


The Scheduler is Playing Tetris

12

Time Low priority

High priority (reservation)

Unused cores

Lower priority

Nodes


The Scheduler is Playing Tetris

13

Time

Unused cores

Nodes

Backfill (small, low priority job can run when higher priority jobs can't)


Hardware Resources Available on Guillimin

14

Partition Count(W, SB)

Memory per core onWestmere nodes

(ppn=12)

Memory per core onSandy Bridge nodes

(ppn=16)

Debug (SW2) - for short test jobs 0, 3 4 GB

Serial Workload (SW, SW2) - for serial jobs and "light" parallel jobs 576, 216 3 GB 4 GB

High Bandwidth (HB) - for massively parallel jobs 384, 0 2 GB

Large Memory (LM, LM2) - for jobs, requiring large memory footprint 192, 144 6 GB 8 GB

Extra Large Memory (XLM2) - limited selection of extra large memory nodes 0, 12 12, 16 or 32 GB

Accelerated Workload (AW) - nodes with GPUs and Xeon Phis 0, 58+50 4 or 8 GB


Let the Scheduler Choose the Right Queue

15

Description(queue name) nodes=1:ppn<12 ppn=12 ppn=16 procs=n, n≥12

Default(metaq)

SW, SW2, AW SW, HB, LM SW2, LM2, XLM2

SW, HB, LM, SW2, LM2, XLM2

High Bandwidth(hb)

NOT ALLOWED HB SW2 HB

Serial Workload(sw)

SW, SW2 SW SW2 SW, SW2

Large Memory(lm)

NOT ALLOWED LM LM2 LM, LM2

Accelerated Workload (k20, phi)

AW AW AW AW

Debug(debug)

SW2 SW2 SW2 SW2


Let the Default Queue Route Your Serial Job

16

PBS -l value where n<12 walltime ≤ 36h(serial-short)

walltime > 36h(sw-serial)

#PBS -l nodes=1:ppn=n SW, SW2, AW SW

#PBS -l nodes=1:ppn=n:westmere SW SW

#PBS -l nodes=1:ppn=n:sandybridge SW2, AW SW2

• Serial:• Default memory: pmem=2700m (2.7G per core)• Recommended: n≤6, or n=12 otherwise (full node)

• Serial (Sandy Bridge):• Optional memory: pmem=3700m (3.7G per core)• Recommended: n≤8, or n=16 otherwise (full node)


• The Linux operating system can run your process “in the background” so that your script continues without waiting for it to finish

• Use the ampersand symbol, &• The ‘wait’ command says to wait for all background processes

to finish

How to Pack Serial Jobs

17

#!/bin/bash#PBS -l walltime=30:00:00#PBS -l nodes=1:ppn=12 SRC=$HOME/program_dircd $SCRATCH/dir1 ; $SRC/prog > output &cd $SCRATCH/dir2 ; $SRC/prog > output &cd $SCRATCH/dir3 ; $SRC/prog > output &...cd $SCRATCH/dir12 ; $SRC/prog > output& wait

#!/bin/bash#PBS -l walltime=30:00:00#PBS -l nodes=1:ppn=12 SRC=$HOME/program_dirfor i in $(seq 12)do cd $SCRATCH/dir$i $SRC/prog > output &done wait


• GNU Parallel is an easy-to-use tool for launching processes in parallel

• Example: testing all combinations of two parameters: {1, 2, 3} x {94, 95, 96}

$ parallel echo {1} x {2} ::: $(seq 1 3) ::: $(seq 94 96)1 x 94

1 x 95

1 x 96

2 x 94

2 x 95

2 x 96

3 x 94

3 x 95

3 x 96

How to Pack Thousands of Serial Tasks

18


• Run different commands in parallel• parallel ::: hostname date 'echo hello world'

• Input sources from a file• parallel -a input-file echo

• Input sources from the command line• parallel echo ::: A B C

• Input sources from STDIN• cat input-file | parallel echo

• Input from multiple sources• parallel -a abc-file -a def-file echo• cat abc-file | parallel -a - -a def-file echo• # Will operate on each pair of inputs

GNU Parallel

19


Let the Default Queue Route Your Parallel Job

20

pmem value walltime <= 72h walltime > 72h

ppn=12 ppn=16 procs ppn=12 ppn=16 procs

1700m(*) HB, SW, LM SW2, LM2 HB, SW, LM,SW2, LM2

HB SW2 HB

2700m(†) SW, LM SW2, LM2 SW, LM,SW2, LM2

SW SW2 SW

3700m(‡) LM SW2, LM2 LM,SW2, LM2

LM SW2 SW2

5700m LM LM2 LM, LM2 LM LM2 LM, LM2

7700m N. A. LM2 LM2 N. A. LM2 LM2

>7800m XLM2 XLM2 XLM2 XLM2 XLM2 XLM2

(*) pmem=1700m is default if procs>12 or nodes>1(†) pmem=2700m is default if procs=12 or nodes=1:ppn=12(‡) pmem=3700m is default if ppn=16


Let the Default Queue Route Your Parallel Job

21

• Parallel (ppn=12, Westmere):#PBS -l nodes=n:ppn=12

• Default pmem: 2700m if n=1, 1700m otherwise• Parallel (ppn=16, Sandy Bridge):

#PBS -l nodes=n:ppn=16• Default pmem: 3700m

• Parallel (procs=k, k>11, multiples of 48 are best):#PBS -l procs=k

• Default pmem: 2700m if k=12, 1700m otherwise


• GPUs:#PBS -l nodes=2:ppn=16:gpus=2#PBS -l pmem=123200m

• Reserves two full nodes with 2 GPUs each• pmem is per node for GPUs!

• Xeon Phi:#PBS -l nodes=1:ppn=8:mics=1,pmem=29600m

• Public Queues:• Default queue: metaq, generally no need to specify

queue name• Exception: debug queue: #PBS -q debug, for test

jobs (default walltime 30 mins, max 2 hours)

Submission styles (accelerators, debug)

22


Private Queues

23

pmem value walltime <= 72h walltime > 72h

ppn=12 ppn=16

1700m hbplus hb sw2-parallel

2700m swplus sw-parallel sw2-parallel

3700m sw2plus sw2-parallel sw2-parallel

5700m lm lm lm

7700m lm2 N. A. lm2

>7800m xlm2 xlm2 xlm2

Other queues:● k20● phi● debug


How to Monitor Your Job in Queue

24

• Idle queues with partitions for accurate priority:• showq -i -p gm-1r16-n04• showq -i -p k20• showq -i -p phi

• Idle queue for your account:• showq -i -w acct=abc-123-ax -v

• Idle queue for serial jobs:• showq -i -p gm-1r16-n04 -w qos=serial

• Idle queue for any queue. For example, debug:• showq -i -w class=debug

• Note: priority ranking goes by “QOS” = Quality of Service: serial,normal,avx(SW2),lm,xlm2,aw This way, lm jobs get priority over normal jobs on LM nodes.


• Any question?• For other questions: [email protected]

Conclusion

25

guillimin@ March 16, 2017 Guillimin HPC ... · PDF fileGuillimin HPC Users Meeting ... •...

Documents

Transcript of guillimin@ March 16, 2017 Guillimin HPC ... · PDF fileGuillimin HPC Users Meeting ... •...