Viglen Rocks Cluster Training University of Liverpool David Power HPC Systems Engineer.

Viglen Rocks Cluster Training

University of Liverpool

David PowerHPC Systems Engineer

Viglen Rocks Cluster Training

• Rocks Overview

• Managing Rocks

• Under the hood with Torque/Maui

• IPMI Management

• Implementing Disk Quotas

• Using Rocks

Rocks Overview

• Rocks Overview

• Managing Rocks


• IPMI Management


• Using Rocks

Rocks Overview

• Rocks is an open-source Linux cluster distribution that enables end users to easily build computational clusters, grid endpoints and visualization tiled-display walls

• It was started by National Partnership for Advanced Computational Infrastructure and the SDSC (San Diego Supercomputer Centre) in 2000

• It is based on CentOS with a modified anaconda installer that simplifies mass installation onto many computers.

• Installations are customised with additional software packages called rolls

Adding Software to Compute Nodes

• Rocks Overview

• Managing Rocks


• IPMI Management


• Using Rocks

Building RPMs in Rocks

• Use rpmbuild if a .spec file is provided

• Use the Rocks built in mechanism to create RPMs from directories.

• Build your software from source and install on the frontend:– configure– make– make install

• Or, just untar a binary bundle

Building RPMs in Rocks

• Creating RPM packages in Rocks:

rocks create package <path> <package-name>

rocks create package /opt/myapp myapp

Contribute the RPM

• Your distribution looks for packages from Rolls and in the contrib area.

• Copy your RPMs in to the contrib directory.

cp myapp-1.0-1.x86_64.rpm /export/rocks/install/contrib/5.X/x86_64/RPMS

Extend XML

• Compute Node installation profile can be extended with additional applications by creating an extend-compute.xml file and editing this:

cd /export/rocks/install/site-profiles/5.1/nodes/

cp skeleton.xml extend-compute.xml

vi extend-compute.xml

Add a package to the Compute Profile

Rebuild Distribution

• RPM package is present in the contrib directory

• XML node file is extended (and updated)

• Now we need to rebuild the distribution (apply changes above)

• This must be done in /export/rocks/install

$ cd /export/rocks/install/

$ rocks create distro

Reinstall the nodes

• PXE Boot– Network boot is first in BIOS boot order– Set Rocks Boot action to install– Reboot the host

$ rocks set host boot <host> action=<boot-action>

$ rocks set host boot compute-0-0 action=install

(or action=os to cancel reinstall)

Reinstall the nodes

• Files updated in /tftpboot

/tftpboot/pxelinux/pxelinux.cfg/XXYYZZAABBCC

default rocks

prompt 0

label rocks

localboot 0

Reinstall the nodes

• Initiate a reboot on the nodes to re-install.

$ rocks run host

<host>

<command>

$ rocks run host

compute-0-0

/sbin/init 6

Check the PXE boot action

• List the current PXE boot action on all the nodes.

$ rocks list host boot

HOST ACTION

frontend: os

compute-0-0: os

compute-0-1: os

compute-1-0: os

compute-1-1: install

compute-2-0: os

compute-2-1: os

<post> section

• The <post> section of the extend-compute.xml can be used to do any post installation configuration

• The file is XML based and certain characters are not permitted:

Special Character XML syntax > > < < “ " & &

<post>.../path/to/daemon > /path/to/outputfile</post>

<post> section

• You can put in standard shell commands in to the <post> section

<post>...chkconfig --level 2345 rocks-grub off</post>

• You can also put in python commands

<post><eval shell=“python”>

python shell code </eval>

</post>

Append files in the <post> section

<post>

...

<file name=“/etc/motd” mode=“append”>

Updated MOTD message

</file>

</post>

Create files in the <post> section

<post>

...

<file name=“/etc/rc.d/rocksconfig.d/post-99-dofinalconf” mode=“create” perms=“0755”>

#!/bin/bash

cd /share/apps/myapp

cp file /opt/myapp

rm -f /etc/rc.d/rocksconfig.d/post-99-dofinalconf

</file>

</post>

Problems with adding new packages

• Verify the syntax of the xml extend-compute file

$ xmllint –noout /path/to/extend-compute.xml

• Are kickstart files being generated correctly?

$ /export/rocks/sbin/kickstart.cfg

--client=compute-0-0

Sync Files across Cluster (411)

• 411 is used to keep files consistent across the cluster

• Used for passwd/shadow/group etc• Can be used for other files also

/var/411/Files.mk

FILES_NOCOMMENT = /etc/passwd \

/etc/group \

/etc/shadow

$ rocks sync users

Torque/Maui

• Rocks Overview

• Managing Rocks


• IPMI Management


• Using Rocks

Software Included

• Torque http://www.clusterresources.com/products/torque

• Maui http://www.clusterresources.com/products/maui

• mpiexec http://www.osc.edu/~pw/mpiexec

• pbstools http://www.osc.edu/~troy/pbs

• pbspython ftp://ftp.sara.nl/pub/outgoing/

Installed daemons

• Frontend– maui– pbs_server– pbs_mom (not running)– mpiexec (mostly for the man-page)

• Compute– pbs_mom– mpiexec

Scheduling Features

• Maui provides a rich set of scheduling features

• Maui can schedule on– CPUs– Walltime– Memory– Disk size– Network Topology

Needed Job Info

• For scheduling to be useful one needs info about the jobs

– At least number of cpus and walltime– Memory requirement also useful

#PBS –lwalltime=HH:MM:SS

#PBS –lnodes=10:ppn=8

#PBS –lpmem=1gb

Memory handling on Linux

• Torque/Maui supports two memory specification types, (p)mem and (p)vmem on linux.

• pmem is not enforced– Used only as information to the sceduler

• pvmem is enforced– Terminating procs that cross the limit– Limits vmem size setting ulimit –v on the processes

Torque Hacking

• Torque is installed in /opt/torque

• qmgr is the torque mgt. command

• Note: backup your working config

$ qmgr –c “print server” > /tmp/pbsconfig.txt

Torque Hacking

• Roll back to escape from a messed up system:

$ qterm

$ pbs_server –t create

$ qmgr < /tmp/pbsconfig.txt

• This will bring you back to where you started

Note: this will wipe the whole queue setup and all current queued and running jobs will be lost!

Maui Hacking

• Most things can be achieved by modifying:

/etc/maui/maui.cfg

• Maui needs a restart after changing the config file:

$ service maui restart

Should I edit Torque or Maui?

• If you can achieve the same thing by changing either torque or maui, use maui.

• Restarting maui is rather lightweight operation, and seldom causes problems for live systems.

• Restarting pbs_server can make the system oscillatory for a few minutes– pbs_server needs to contact all pbs_moms to get

back in state.

Prioritising Short Jobs

• Often it is useful to give shorter jobs higher priority (maui configuration).

• Use the XFACTOR feature in maui rather than torque queues with different priorities

XFACTORWEIGHT 1000

Prioritising Short Jobs

• XFACTOR is defined as XFACTOR=(walltime+queuetime)/walltime

• XFACTOR will increase faster for shorter walltimes thus giving higher priorities for short jobs.

• Depends on users giving reasonable walltime limits.

Prioritising large jobs (maui)

• In a cluster with a diverse mix of jobs it is useful to prioritize the large jobs and make the smaller ones fill the gaps.

CPUWEIGHT 1000

MEMWEIGHT 100

• This should be combined with fairshare to avoid starving users falling outside this prioritisation.

Fairshare (maui)

• Also known as“keeping all users equally unhappy”

• Can be done on several levels– users, groups …

• Set a threshold

USERCFG[DEFAULT] FSTARGET=10FSWEIGHT 100

• Users having more than 10% will get reduced priority and vice versa.

Adjusting your policy

• You can play with the weights to fine tune your scheduling policies

XFACTORWEIGHT 100FSWEIGHT 1000RESWEIGHT 10CPUWEIGHT 1000MEMWEIGHT 100

• Analyse the prioritisation with

$ diagnose -p

Job Node Distribution

• Default is MINRESOURCE– Run on the nodes which gives at least unused resources.

• Spread or pack?NODEALLOCATIONPOLICY PRIORITY

– Select the most busy nodes

NODECFG[DEFAULT] PRIORITYF=JOBCOUNT

– Select the least busy nodesNODECFG[DEFAULT] PRIORITYF=-1.0*JOBCOUNT

Node Access Policy

• Default access policy is SHARED

• Can choose to limit this to SINGLEJOB or SINGLEUSER, i.e. NODEACCESSPOLICY SINGLEUSER

• Single user access prevents users from stepping on each others toes while allowing good utilisation for serial jobs.

Throttling Policies

• Sometimes one needs to limit the user from taking over the system…– MAXPROC– MAXNODE– MAXPS– MAXJOB– MAXIJOB (I indicating idle, scheduled but not running)

• All can be set for all individuals users and groups – USERCFG[DEFAULT]– USERCFG[USERA] MAXPROC=16

http://www.clusterresources.com/products/maui/docs/6.2throttlingpolicies.shtml

Debugging and Analysing

• Lots of tools:– pbsnodes -- node status– qstat –f -- all details of a job– diagnose –n -- node status from maui– diagnose –p -- job priority calculation– showres –n -- job reservation per node– showstart -- estimated job start time– checkjob -- check job status– checknode -- check node status

Example: Express Queue

• Goal: Supporting development and job script testing, but prevents misuse

• Basic philosophy:

– Create a separate queue– Give it the highest priority– Throttle is so it is barely usable


• Create the queue with qmgr

create queue express

set queue express queue_type = Execution

set queue express resources_max.walltime = 08:00:00

set queue express resources_default.nodes = 1:ppn=8

set queue express resources_default.walltime = 08:00:00

set queue express enabled = True

set queue express started = True


• Increase the priority and limit the usage (maui.cfg)

CLASSWEIGHT 1000

CLASSCFG[express] PRIORITY=1000 MAXIJOB=1 \

MAXJOBPERUSER=1 QLIST=express QDEF=express \

QOSCFG[express] FLAGS=IGNUSER

This will allow users to test job scripts and run interactive jobs with a good turnaround.

Check jobs running on the nodes

• Check for jobs running on the cluster (showq):[root@fotcluster2 nodes]# showq

ACTIVE JOBS--------------------

JOBNAME USERNAME STATE PROC REMAINING STARTTIME

1472 asmith Running 64 99:13:35:10

1473 klangfeld Running 120 99:22:01:12

2 Active Jobs 184 of 948 Processors Active (19.41%)

16 of 79 Nodes Active (20.25%)

Check host jobs are running on

• Whats nodes are the jobs running on: (checkjob JOBID) :[root@fotcluster2 nodes]# checkjob 1473checking job 1473...Req[0] TaskCount: 120 Partition: DEFAULTNetwork: [NONE] Memory >= 0 Disk >= 0 Swap >= 0Opsys: [NONE] Arch: [NONE] Features: [NONE]Allocated Nodes:[compute-2-31:8][compute-2-30:8][compute-2-29:8]

[compute-2-28:8][compute-2-27:8][compute-2-26:8][compute-2-25:8]

[compute-2-24:8][compute-2-23:8][compute-2-35:8][compute-2-34:8]

[compute-2-33:8]

[compute-2-32:8][compute-2-21:8][compute-2-20:8]

Take nodes OFFLINE

• Take nodes offline (no jobs will be allocated) :[root@fotcluster2 nodes]# pbsnodes -o compute-2-0[root@fotcluster2 nodes]# pbsnodes compute-2-0compute-2-0 state = offline np = 12 ntype = cluster status = opsys=linux,uname=Linux compute-2-0.local

2.6.18-128.el5 #1 SMP Wed Jan 21 10:41:14 EST 2009 x86_64,sessions=? 0,nsessions=? 0,nusers=0,idletime=248891,totmem=25696960kb,availmem=25431280kb,physmem=24676844kb,ncpus=12,loadave=0.00,netload=1104506203,state=free,jobs=,varattr=,rectime=1288610352

Bring nodes ONLINE

• Take nodes offline (no jobs will be allocated) :[root@fotcluster2 nodes]# pbsnodes -c compute-2-0[root@fotcluster2 nodes]# pbsnodes compute-2-0compute-2-0 state = free np = 12 ntype = cluster status = opsys=linux,uname=Linux compute-2-0.local

2.6.18-128.el5 #1 SMP Wed Jan 21 10:41:14 EST 2009 x86_64,sessions=? 0,nsessions=? 0,nusers=0,idletime=248981,totmem=25696960kb,availmem=25431356kb,physmem=24676844kb,ncpus=12,loadave=0.00,netload=1104881434,state=free,jobs=,varattr=,rectime=1288610442

Create Queues• Create additional queues using qmgr

qmgr -c "create queue infiniband“

qmgr -c "set queue infiniband queue_type = Execution“

qmgr -c "set queue infiniband enabled = True“

qmgr -c "set queue infiniband started = True“

qmgr -c "set queue infiniband resources_default.neednodes = ib“

qmgr -c "create queue primare“

qmgr -c "set queue primare queue_type = Execution“

qmgr -c "set queue primare enabled = True“

qmgr -c "set queue primare started = True“

qmgr –c “set queue default resources_default.neednodes = ethernet”

Create Queues• Assign resources to nodes:

In file: /opt/torque/server_priv/nodes

...

compute-1-7 np=8 ethernet



compute-2-0 np=12 ib




...

Create Queues• Verify nodes report resources correctly:

[root@fotcluster2 ~]# pbsnodes compute-1-0

compute-1-0

state = free

np = 8

properties = ethernet

[root@fotcluster2 ~]# pbsnodes compute-2-0

compute-2-0

state = free

np = 12

properties = ib

Queue Priority• Set the priority levels for the queues:

In File: /opt/maui/maui.cfg

...

CLASSWEIGHT 1

CLASSCFG[default] PRIORITY=1

CLASSCFG[infiniband] PRIORITY=1

CLASSCFG[primare] PRIORITY=10000

...

• Save and restart mauiservice maui restart

Check Job Priorities• (As root) use the diagnose command:

[root@fotcluster2 ~]# diagnose -p

diagnosing job priority information (partition: ALL)

Job PRIORITY* Cred(Class) Serv(QTime)

Weights -------- 1( 1) 1( 1)

1548 10000 100.0(10000) 0.0( 0.0)

1545 1 84.5( 1.0) 15.5( 0.2)

1547 1 92.3( 1.0) 7.7( 0.1)

Percent Contribution -------- 100.0(100.0) 0.0( 0.0)

*indicates system priority set on job

Red text indicates a job on high priority queue (classcgf)

Submitting to the queues• Submit to Infiniband Queue

#PBS -q infiniband

• Submit to Primare Queue#PBS -q primare

• Submit to Primare Queue (on IB nodes)#PBS -q primare

#PBS -l nodes=NN:ppn=PP:ib

• Submit to Primare Queue (on Ethernet nodes)#PBS -q primare

#PBS -l nodes=NN:ppn=PP:ethernet

Restricting User Access to Queue

• Allow only users from the primare group use primare Q

qmgr -c "set queue primare acl_group_enable=true"

qmgr -c "set queue primare acl_groups=primare"

Summary

• Limit the number of queues.

• You need good information about walltime.

IPMI Management

• Rocks Overview

• Managing Rocks


• IPMI Management


• Using Rocks

IPMI Overview

• IPMI is a standard which defines a set of common interfaces to monitor system health and manage the system.

• IPMI operates independent of the OS

• IPMI operates even if the monitored system is powered off, but connected to a power source

• IPMI BMCs can be communicated to using IP addressing or queried locally.

Configuring the IPMI Module

• RPMs installed:– OpenIPMI– OpenIPMI-tools (inc ipmitool)– OpenIPMI-libs

• Kernel Modules required:– ipmi_msghandler– ipmi_devinf– ipmi_si

Configuring the IPMI Module

• Set the IP address (static):

ipmitool lan set 1 ipsrc static

ipmitool lan set 1 ipaddr 10.1.1.100

ipmitool lan set 1 netmask 255.255.0.0

ipmitool lan set 1 defgw ipaddr 10.1.1.1

• Set the IP address (dhcp):

ipmitool lan set 1 ipsrc dhcp

Configuring the IPMI Network

• IPMI modules part of the Rocks Network• Edit the /etc/hosts.local file

10.1.1.100 compute-ipmi-0-0

10.1.1.101 compute-ipmi-0-1

• Run dbreport hosts or rocks report hosts• Sync the /etc/hosts file using 411

(/var/411/Files.mk)

$ rocks sync users


• IPMI Modules on a separate subnet• Step 1: Add a network

$ rocks add network <network-name>

<network>

<subnet>

$ rocks add network ipmi

192.168.1.0

255.255.255.0


• List the networks on your Rocks cluster

$ rocks list network

NETWORK SUBNET NETMASK

private: 10.12.0.0 255.255.0.0

public: 169.228.3.0 255.255.255.240

ipmi: 192.168.1.0 255.255.255.0


• Step 2: Add the interfaces through Rocks

– Host first must be installed– Then secondary NICs can be added– After all hosts are configured just re-install


• Add host interface:

$ rocks add host interface <host> <iface>

ip=<address> subnet=<name> gateway=<address>

name=<hostname>

$ rocks add host interface compute-0-0 ipmi

ip=192.168.1.1 subnet=ipmi gateway=ipmi

gateway=1 name=compute-ipmi-0-0


• List host interface:

$ rocks list host interface compute-0-0

SUBNET IFACE MAC IP NETMASK GATEWAY MODULE NAME

private eth0 00:15:17:79:d3:c0 10.12.0.12 255.255.0.0 ------- e1000e compute-0-0

Ipmi ipmi ----------------- 192.168.1.2 255.255.255.0 2 ------ ipmi-0-0

(Alternative) Configuring the IPMI Network

• Add hosts to /etc/hosts.local:10.1.10.10 compute-1-0-ipmi10.1.10.11 compute-1-1-ipmi10.1.10.12 compute-1-2-ipmi10.1.10.13 compute-1-3-ipmi

• Regenerate the /etc/hosts file (which will append all entries from /etc/hosts.local$ dbreport hosts > /etc/hosts...# import from /etc/hosts.local10.1.10.10 compute-1-0-ipmi10.1.10.11 compute-1-1-ipmi10.1.10.12 compute-1-2-ipmi10.1.10.13 compute-1-3-ipmi

Check the IPMI serivice is running

• Use the tentakel command:tentakel -g rack2 chkconfig --list '| grep ipmi'

### compute-2-35(stat: 0, dur(s): 0.16):

ipmi 0:off 1:off 2:on 3:on 4:on 5:on 6:off

### compute-2-23(stat: 0, dur(s): 0.17):

ipmi 0:off 1:off 2:on 3:on 4:on 5:on 6:off

• If any nodes have the service set to off, turn on using:tentakel -g rack2 chkconfig --level 2345 ipmi on

Using IPMItool

• Chassis Status

ipmitool chassis statusSystem Power : onPower Overload : falsePower Interlock : inactiveMain Power Fault : falsePower Control Fault : falsePower Restore Policy : always-offLast Power Event : ac-failedChassis Intrusion : activeFront-Panel Lockout : inactiveDrive Fault : falseCooling/Fan Fault : false

Using IPMItool

• Chassis Status over the network

ipmitool –U ADMIN –P ADMIN –h compute-ipmi-0-0 chassis statusSystem Power : onPower Overload : falsePower Interlock : inactiveMain Power Fault : falsePower Control Fault : falsePower Restore Policy : always-offLast Power Event : ac-failedChassis Intrusion : activeFront-Panel Lockout : inactiveDrive Fault : falseCooling/Fan Fault : false

Using IPMItool

• Reset the IPMI Module (sometimes causes network issues)

ipmitool mc reset cold

ipmitool –U ADMIN –P ADMIN –h compute-ipmi-0-0 mc reset cold

• Then reload network modules on nodes:rmmod e1000e ; modprobe e1000e

Using IPMItool

• Controlling the Power

Initiate a clean shutdownipmitool –U ADMIN –P ADMIN –h compute-ipmi-0-0 power soft

Power Offipmitool –U ADMIN –P ADMIN –h compute-ipmi-0-0 power off

Power Onipmitool –U ADMIN –P ADMIN –h compute-ipmi-0-0 power on

Power Cycleipmitool –U ADMIN –P ADMIN –h compute-ipmi-0-0 power cycle

Implementing Disk/User Quotas

• Rocks Overview

• Managing Rocks


• IPMI Management

• Implementing Disk/User Quotas

• Using Rocks

Disk Quotas (general)

• In order to use disk quotas, they must first be enabled:

– Modify /etc/fstab– Remount the filesystem(s)– Run quotacheck– Assign Quotas

Disk Quotas (general)

• Modify /etc/fstab

• Edit the line with your home directoryLABEL=/state/partition

• On this line change the mount options in include grpquota and usrquota

defaults -> grpquota,usrquota,defaults

• Reboot the machine

Disk Quotas (ext3)

• When the system has come back up, run quotacheck• This examines quota-enabled file systems, building a table of

the current disk usage for each one.

quotacheck –guvma

-g -- group quotas-u -- user quotas-v -- report the operation as it progresses-m -- do not remount filesystems as RO-a -- check all mounted non-NFS filesystems

Disk Quotas (ext3)

• Turn File system quotas on

quotaon –guva

• Setup a user quota

edquota –u <username>

• Modify the number of days for the hard and soft limits

edquota –t

• Propagate these settings to all other users

edquota –p <user_with_updated_quota> -u users..

Disk Quotas (ext3)

• Editing the quota limits (edquota)

Disk quotas for user viglen (uid 501): Filesystem blocks soft hard inodes soft hard /dev/sda5 24 5000 0 7 0 0

• Blocks: The amount of space in 1K blocks the user is currently using.

• Inodes: The number of files the user is currently using.

• Soft Limit: The maximum blocks/inodes a quota user may have on a partition. The role of a soft limit changes if grace periods are used. When this occurs, the user is only warned that their soft limit has been exceeded. When the grace period expires, the user is barred from using additional disk space or files. When set to zero, limits are disabled.

• Hard Limit: The maximum blocks/inodes a quota user may have on a partition when a grace period is set. Users may exceed a soft limit, but they can never exceed their hard limit.

Disk Quotas (ext3)

• Reporting on quota usage:

repquota /state/partition1

• Useful TIP: Display user quotas at their login (/etc/profile)

quota –u $USER

Disk Quotas (XFS)

*[Steps below assume /etc/fstab has been updated and the system has been rebooted]*

• Confirm quotas are enabled

xfs_quota –x –c state

• Check the current space usage

xfs_quota –x –c quot

Disk Quotas (XFS)

• Set user limits (where username is the user account name)Limits below implement a 6GB hard limit (cant write more than 6GB) and a 5GB soft limit (warnings issued after 5GB)

xfs_quota –x /state/partition1

xfs_quota> limit bsoft=5g bhard=6g username

xfs_quota> quota -h username

Disk quotas for User username(id)

Filesystem Blocks Quota Limit Warn/Time Mounted on /dev/sdb1 2.4G 5G 6G 00 [------] /state/partition1

Disk Quotas (XFS)

• Disable user quota, set the limits back to 0

xfs_quota –x /state/partition1

xfs_quota> limit bsoft=0 bhard=0 username

xfs_quota> quota -h username

Disk quotas for User username(id)

Filesystem Blocks Quota Limit Warn/Time Mounted on /dev/sdb1 2.4G 0 0 00 [------] /state/partition1

Disk Quotas (XFS)

• Report on XFS Quota Usage[root@fotcluster2 scripts]# xfs_quota -xxfs_quota> reportUser quota on /state/partition1 (/dev/sdb1) BlocksUser ID Used Soft Hard Warn/Grace---------- --------------------------------------------------root 18050992 0 0 00 [0 days]maui 2447696 0 0 00 [--------]icr 105732 0 0 00 [--------]biouser 0 0 0 00 [--------]viglen 8343092 0 0 00 [--------]klangfeld 1590916 99614720 104857600 00 [--------]pmills 15474924 99614720 104857600 00 [--------]troc 436572 99614720 104857600 00 [--------]ngardiner 4826800 99614720 104857600 00 [--------]

Implementing Disk/User Quotas

• Rocks Overview

• Managing Rocks


• IPMI Management

• Implementing Disk/User Quotas

• Using Rocks

System Access

• Access the Rocks Cluster via ssh (port 22)• From Linux systems this is already built in (ssh from the CLI)• For graphical forwarding use ssh with X-forwarding

ssh –X headnode

• To copy file use scpscp file username@headnode:path/to/file

scp –r directory username@headnode:/path/to/dir

• From Windows systems use puttyhttp://www.chiark.greenend.org.uk/~sgtatham/putty/download.html

• For graphical forwarding in Windows use Xminghttp://www.straightrunning.com/XmingNotes/

System Access

• From Windows systems use puttyhttp://www.chiark.greenend.org.uk/~sgtatham/putty/download.html

• For graphical forwarding in Windows use Xminghttp://www.straightrunning.com/XmingNotes/

• To copy files use WinSCPhttp://winscp.net/

System Environment

• To setup you environment to compile/run applications, u se the modules environment[viglen@ulgqcd ~]$ which pgcc

/usr/bin/which: no pgcc in $PATH

[viglen@ulgqcd ~]$ pgcc -V

-bash: pgcc: command not found

[viglen@ulgqcd ~]$ module available

--------------- /usr/share/Modules/modulefiles --------------------------------------

pgi/32/10.9 pgi/64/10.9 rocks-mpich2/gnu/1.1.1-p1

pgi/32/10.9-mpich pgi/64/10.9-mpich

[viglen@ulgqcd ~]$ module load pgi/64/10.9

[viglen@ulgqcd ~]$ which pgcc

/share/apps/pgi/10.9/linux86-64/10.9/bin/pgcc

[viglen@ulgqcd ~]$ pgcc -V

pgcc 10.9-0 64-bit target on x86-64 Linux -tp istanbul-64

Job Control

• Check Jobs running on the queue[viglen@ulgqcd ~]$ showq

ACTIVE JOBS--------------------


0 Active Jobs 0 of 496 Processors Active (0.00%)


IDLE JOBS----------------------

JOBNAME USERNAME STATE PROC WCLIMIT QUEUETIME

0 Idle Jobs

BLOCKED JOBS----------------

JOBNAME USERNAME STATE PROC WCLIMIT QUEUETIME

Sample Submission Script

#!/bin/bash #PBS -l nodes=4

#PBS –l walltime=2:00 #PBS -o mytest.out #PBS -e mytest.err

cd $PBS_O_WORKDIR

NPROCS=`wc -l $PBS_NODEFILE | awk '{ print $1 }'`

/opt/mpich/bin/mpirun -np $NPROCS -machinefile $PBS_NODEFILE hello

Job Control

• Submit a job to the queue[viglen@ulgqcd qsub]$ qsub q_sleep.sh

110.ulgqcd.ph.liv.ac.uk

[viglen@ulgqcd qsub]$ showq

ACTIVE JOBS--------------------


110 viglen Running 1 00:00:00 Wed Nov 3 23:20:45

1 Active Job 1 of 496 Processors Active (0.20%)


Job Control

• Delete a job from the queue[viglen@ulgqcd qsub]$ qsub q_sleep.sh

110.ulgqcd.ph.liv.ac.uk

[viglen@ulgqcd qsub]$ qdel 110

Job Control

• Take a look at the output before jobs finishes[viglen@ulgqcd qsub]$ qpeek 190memtester version 4.0.8 (64-bit)Copyright (C) 2007 Charles Cazabon.Licensed under the GNU General Public License version 2 (only).

pagesize is 4096pagesizemask is 0xfffffffffffff000want 90000MB (94371840000 bytes)got 90000MB (94371840000 bytes), trying mlock ...locked.Loop 1/1: Stuck Address : ok Random Value : ok Compare XOR : Compare SUB : Compare MUL :

Compare DIV : Compare OR : Compare AND : Sequential Increment: ok

Solid Bits : testing 49 Block Sequential : setting 11

Torque/PBS Submit Options

#PBS –I # Submit an interactive job

#PBS –q queue_name # Submit to a certain queue

#PBS –l nodes=compute-0-0 # Submit to a specific host

#PBS –l nodes=4:ppn=4 # 4 Host with 4 Processes per node

#PBS –j oe # Combine stdout and stderr

#PBS –o stdout_file # Specify the Output file

#PBS –e stderr_file # Specify the Error file

#PBS –V # Import you environment vars

Torque/PBS Variables

$PBS_O_HOST # Hostname where job launched

$PBS_O_QUEUE # Originating Queue

$PBS_O_WORKDIR # Execution Queue

$PBS_JOBID # Job Identifier

$PBS_JOBNAME # Job name

$PBS_NODEFILE # Nodefile (for parallel jobs)

$PBS_O_HOME # Current Home Directory

$PBS_PATH # PATH

Thank You!

Web: http://www.viglen.co.uk/hpcEmail: [email protected]

http://www.viglen.co.uk/hpc

mailto:[email protected]

Viglen Rocks Cluster Training University of Liverpool David Power HPC Systems Engineer.

Documents

Transcript of Viglen Rocks Cluster Training University of Liverpool David Power HPC Systems Engineer.