Steinbuch Centre for Computing (SCC)

Funding: www.bwhpc-c5.de

File Systems and Software Modules

Simon Raffeiner, Scientific Computing and Simulation Dept., SCC, KIT

File Systems and Software modules / S.Raffeiner05/04/2017 2

Reference: bwHPC-C5 Best Practices Repository

Most information given by this talk can be found at http://bwhpc-c5.de/wiki:

Category:Hardware_and_Architecture

Environment_Modules

Batch_Jobs

File Systems and Software modules / S.Raffeiner05/04/2017 3

Material: Slides & Scripts

https://indico.scc.kit.edu/indico/event/278/

@bwUniCluster/ForHLR I/ForHLR II:/pfs/data1/software_uc1/bwhpc/kit/workshop/2017-04-05

Abbreviation/Colour code Full meaning

$ command -option value $ = prompt of the interactive shellThe full prompt may look like: user@machine:path$The command has been entered in the interactive shell session

<integer><string>

<> = Placeholder for integer, string etc

foo, bar Metasyntactic variables

How to read the following slides

File Systems and Software modules / S.Raffeiner05/04/2017 4

File Systems

File Systems and Software modules / S.Raffeiner05/04/2017 5

File Systems

bwUniCluster, ForHLR I / ForHLR II, bwForCluster, …

Too many file systems to list them all, please see documentation!

Common „rules of thumb“

There is a small, global, permanent $HOME directory→ Usually backed up

There are one or more large, global temporary $WORK directories→ Old files might be deleted automatically→ Usually NOT backed up

There might be local, non-global, temporary storage on the computing nodes

→ Usually named $TMP, $SCRATCH or something similar→ Will be wiped when the job ends

File Systems and Software modules / S.Raffeiner05/04/2017 6

$HOME = Home directory

$HOME:

HowTo goto:

@ bwUniCluster/ForHLR: Current quota:

Diskusage:

@ KIT: $HOME directories of bwUniCluster, ForHLR I / II are the sameBut: different hardware, libraries, queueing etc.→ generalise your scripts to work on all systems using $CLUSTER

$ lfs quota -u $(whoami) $HOME

if [ ${CLUSTER} == “uc1“ ]; then <operations>fi

if [ ${CLUSTER} == “uc1“ ]; then <operations>fi

$ grep -E "$(whoami)|Account" ~/../../diskusage

$ cd $HOME

$ cd

File Systems and Software modules / S.Raffeiner05/04/2017 7

$HOME @ bwForClusters

$HOME:

@bwForCluster: Please check the documentation!

File Systems and Software modules / S.Raffeiner05/04/2017 8

$PROJECT = Project directory of ForHLR I/II

ONLY ForHLR I/II:All features of $HOME

Access granted based on approved projects

→ assigned „name/acronym“, e.g. /project/fh1-project-scs/mr2515

→ $PROJECT_GROUP, e.g. „fh1-project-scs“

Access project home directory:

Do not use: $HOME → it has very low quota for the project group!

Quota of Project:

$ cd $PROJECT

$ lfs quota -g ${PROJECT_GROUP} ${PROJECT}

File Systems and Software modules / S.Raffeiner05/04/2017 9

$WORK = Working directory

bwUniCluster/ForHLR I/II → addional parallel file system with limited lifetime, no redundancy, quotas

especially designed for parallel access and for a high throughput to large files

2 concepts of access via:(A) → $WORK(B) → workspaces

(A) $WORK:Change to it via:

Quota:

But: files no longer needed should be removed

→ any file inside your $WORK older than 28 days will be deleted

$ cd $WORK

$ lfs quota -u $(whoami) $WORK

File Systems and Software modules / S.Raffeiner05/04/2017 10

Workspaces: lifetime on allocated folder HowTo:

→ http://www.bwhpc-c5.de/wiki/index.php/BwUniCluster_File_System#Workspaces

$ ws_allocate foo 10 Allocate workspace foo for 10 days 

$ ws_list -a List all your workspaces 

$ ws_find foo Get absolute path of workspace foo

$ ws_extend foo 5 Extend lifetime of your workspace foo by 5 days from now. You can extend 3 times 

 → max. lifetime of foo =                                     240 days (U+F)                                      90 days (J)

$ ws_release foo Manually erase your workspace foo

$ ws_allocate scratch 300WS: duration is longer than allowed for this filesystem!WS: setting duration to allowed maximum of 60 days!Workspace created. Duration is 1440 hours. Further extensions available: 3/work/workspace/scratch/mr2515-scratch-0

Example:

Workspaces = Working directory (1)

File Systems and Software modules / S.Raffeiner05/04/2017 12

Exercise 1

Allocate two workspaces

$ ws_allocate test 30Workspace created. Duration is 720 hours. Further extensions available: 3/work/workspace/scratch/mr2515-test-0

$ ws_allocate scratch 50Workspace created. Duration is 1200 hours. Further extensions available: 3/work/workspace/scratch/mr2515-scratch-0

File Systems and Software modules / S.Raffeiner05/04/2017 13

Exercise 2

List workspaces

$ ws_listFilesystem: defaultWorkspace ID Workspace location Creation date Remaining time------------------------------------------------------------------------------test Apr 5 09:35:42 29 days 23 hours /work/workspace/scratch/mr2515-test-0 acctcode:900000 available extensions:3scratch Apr 5 09:35:01 49 days 23 hours /work/workspace/scratch/mr2515-scratch-0 acctcode:900000 available extensions:3

File Systems and Software modules / S.Raffeiner05/04/2017 14

Exercise 3

Find workspace path and switch to it

$ ws_find scratch/work/workspace/scratch/mr2515-scratch-0

$ ws_find test/work/workspace/scratch/mr2515-test-0

$ cd $(ws_find test)$ pwd/work/workspace/scratch/mr2515-test-0

File Systems and Software modules / S.Raffeiner05/04/2017 15

Exercise 4

Extend the lifetime of a workspace

$ ws_extend test 60Duration of workspace is successfully changed!New duration is 1440 hours. Further extensions available: 2

$ ws_listFilesystem: defaultWorkspace ID Workspace location Creation date Remaining time--------------------------------------------------------------------------------test Apr 5 09:37:12 59 days 23 hours /work/workspace/scratch/mr2515-test-0 acctcode:900000 available extensions:2scratch Apr 5 09:35:01 49 days 23 hours /work/workspace/scratch/mr2515-scratch-0 acctcode:900000 available extensions:2

File Systems and Software modules / S.Raffeiner05/04/2017 16

Exercise 5

Release workspaces

$ ws_release test/work/workspace/scratch/mr2515-test-0Info: Workspace was deleted.

$ ws_release scratch/work/workspace/scratch/mr2515-scratch-0Info: Workspace was deleted.

File Systems and Software modules / S.Raffeiner05/04/2017 18

Software System

File Systems and Software modules / S.Raffeiner05/04/2017 19

Software (=Environment) modules

By default manual setup of $PATH, $LD_LIBRARY_PATH … forcompilers, libraries and software packages etc.

→ Getting complicated if multiple versions of same software installed

Solution:dynamic modification of the session environment by

→ instruction sets stored in modulefiles

HowTo?load and unload instruction sets (= modulefiles)

How to use modulefiles in general?

More information:http://www.bwhpc-c5.de/wiki/index.php/Environment_Modules

$ module help $ module help

File Systems and Software modules / S.Raffeiner05/04/2017 20

modulefiles: available / search

Display all modulefiles

Display all modulefiles with prefix „compiler“

$ module avail

$ module avail compiler

File Systems and Software modules / S.Raffeiner05/04/2017 21

modulefiles: help / whatis

Show help of modulefiles, e.g.

Show short info modulefile

$ module help compiler/intel

$ module whatis compiler/intel

compiler/intel : Intel(R) compiler suite (icc, icpc, ifort), debugger (gdb-ia), IPP and TBB

File Systems and Software modules / S.Raffeiner05/04/2017 22

modulefiles: show

Show all instructions of modulefile $ module show compiler/gnu/4.7

File Systems and Software modules / S.Raffeiner05/04/2017 23

modulefiles: load (1)

Modulefiles are sorted in categories, software name and versions:

Load a default software:

e.g. Intel compiler

→ loads currently Intel compiler suite 16

→ loads currently Intel-MPI 5.1.3 for Intel compiler 16.0

Display all loaded modules

$ module load <category>/<software_name>/<version>

$ module load <category>/<software_name>

$ module load compiler/intel mpi/impi

$ module list

Currently Loaded Modulefiles: 1) compiler/intel/16.0(default) 2) mpi/impi/5.1.3-intel-16.0(default)

File Systems and Software modules / S.Raffeiner05/04/2017 24

modulefiles: search

@ ForHLR

$ module spider openmpi

------------------------------------------------------------------------------ Core/mpi/openmpi:------------------------------------------------------------------------------ Versions: Core/mpi/openmpi/1.8_intel Core/mpi/openmpi/1.8 Core/mpi/openmpi/1.10_intel Core/mpi/openmpi/1.10 Core/mpi/openmpi/2.0

(..)

File Systems and Software modules / S.Raffeiner05/04/2017 25

modulefiles: categories & dependencies

Module names already implicate dependencies:

→ Category/softwarename/version_attributes-dependenciese.g numlib/fftw/3.3.5-impi-5.1.3-intel-16.0

→ fftw package version 3.3.5, compiled with Intel 16.0 and Intel-MPI 5.1.3

Categories:

compiler/ for compiler, e.g. intel, gnu, pgi, open64

devel/ for debugger, e.g. ddt, and development tools, e.g. cmake, itrac

mpi/ for MPI libraries, e.g. impi, openmpi, mvapich(2)

numlib/ for numerical libraries, e.g. Intel MKL, ACML, nag, gsl, fftw

lib/ for other libraries, e.g. netcdf, global array

bio/ for biology software, e.g. bowtie, abyss, mrbayes

cae/ for CAE software, e.g. ansys, abaqus, fluent

chem/ for chemistry software, e.g. gromacs, dacapo, turbomole

math/ for mathematics software, e.g. matlab, R

phys/ for physics software, e.g. geant4

vis/ for visualisation software, e.g. vmd, tigervnc

File Systems and Software modules / S.Raffeiner05/04/2017 26

modulefiles: load (2)

@ bw{Uni,For}Cluster - Conflicts:a) load different software version in the same session, e.g. Intel:

b) load module with dependencies on other modules

→ NOT an issue if the cluster uses Lmod (ForHLR I/II)

$ module load compiler/intel/14.0$ module load compiler/intel/15.0

compiler/intel/13.1(394):ERROR:150: Module 'compiler/intel/15.0'conflicts with the currently loaded module(s) 'compiler/intel/14.0'

$ module load mpi/openmpi/1.10-intel-16

Loading module dependency 'compiler/intel/16.0'.compiler/intel/16.0(394):ERROR:150: Module 'compiler/intel/16.0'Conflicts with the currently loaded module(s) 'compiler/intel/15.0'

File Systems and Software modules / S.Raffeiner05/04/2017 28

modulefiles: list

Depending on the cluster, default modules might be loaded or not

$ module list

Currently Loaded Modules: 1) dot 2) numlib/mkl/11.3 3) mpi/openmpi/1.8 4) compiler/gnu/5

File Systems and Software modules / S.Raffeiner05/04/2017 29

modulefiles: unload/swap (1)

To remove module foo:

or

@ bw{Uni,For}Cluster

be aware that you might create inconsistencies

e.g. you can remove

compiler/intel/16.0 while mpi/openmpi/1.10-intel-16.0 is still loaded

Swap = remove + load

e.g.:

$ module unload foo $ module remove foo

$ module swap compiler/intel/15.0 compiler/intel/16.0

File Systems and Software modules / S.Raffeiner05/04/2017 30

modulefiles: unload/swap (2)

@ ForHLR: automatic swap

$ module load compiler/gnu/5

Lmod is automatically replacing "compiler/intel/16.0" with "compiler/gnu/5".

File Systems and Software modules / S.Raffeiner05/04/2017 31

Private modulefiles

Each user can create own modulefiles:e.g. modulefiles that adds path of own programs, $HOME/special, to $PATH

→ content of this modulefile „mybin“

→ place „mybin“ under $HOME/privatemodules

→ to make all own modules visible to “module avail” command, enter:

or → former: own modules have lower priority than system ones if equally named

→ latter: own module have higher priority

Remove own modules:

or

#%Module1.0

Append-path PATH “$env(HOME)/special”

$ module load use.own $ module use $HOME/privatemodules

$ module unload use.own $ module unuse $HOME/privatemodules

File Systems and Software modules / S.Raffeiner05/04/2017 33

Exercise 1

Check loaded modules after login

Check default GNU Compiler version

$ module list

No Modulefiles Currently Loaded.

$ gcc --versiongcc (GCC) 4.8.5 20150623 (Red Hat 4.8.5-11)Copyright (C) 2015 Free Software Foundation, Inc.This is free software; see the source for copying conditions. There is NOwarranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

File Systems and Software modules / S.Raffeiner05/04/2017 34

Exercise 2

Load a different GNU compiler version

$ module load compiler/gnu/5.2

$ echo $PATH/opt/bwhpc/common/compiler/gnu/5.2.0/bin:/software/all/bin:/usr/lib64/qt-3.3/bin:/opt/moab/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/home/kit/scc/mr2515/.local/bin:/home/kit/scc/mr2515/bin

$ gcc --versiongcc (GCC) 5.2.0Copyright (C) 2015 Free Software Foundation, Inc.This is free software; see the source for copying conditions. There is NOwarranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

File Systems and Software modules / S.Raffeiner05/04/2017 35

Exercise 3

Switch GNU compiler with Intel Compiler

$ echo $CCgcc

$ module load compiler/intel

$ echo $CCicc

File Systems and Software modules / S.Raffeiner05/04/2017 36

Exercise 4

Try to compile an MPI application

Try to load OpenMPI

$ mpiccbash: mpicc: command not found…

$ module load mpi/openmpi

module load mpi/openmpiLoading module dependency 'compiler/intel/14.0'.compiler/intel/14.0(401):ERROR:150: Module 'compiler/intel/14.0' conflictswith the currently loaded module(s) 'compiler/intel/16.0'compiler/intel/14.0(401):ERROR:102: Tcl command execution failed: conflictcompiler/intel

ERROR: Failed to load module dependency 'compiler/intel/14.0'. Please checkconflicts with already loaded modules.

File Systems and Software modules / S.Raffeiner05/04/2017 37

Exercise 4

What is happening?

The default OpenMPI module was compiled against Intel Compiler version 14, but we loaded version 16 before

$ module show mpi/openmpi-------------------------------------------------------------------/opt/bwhpc/common/modulefiles/mpi/openmpi/1.8-intel-14.0:

(..)

File Systems and Software modules / S.Raffeiner05/04/2017 38

Exercise 4

Load the correct OpenMPI version

Now mpicc works

$ module load mpi/openmpi/1.10-intel-16.0

$ mpicc --versionicc (ICC) 16.0.4 20160811Copyright (C) 1985-2016 Intel Corporation. All rights reserved.