Changes between Version 19 and Version 20 of Public/User_Guide/Batch_system

Timestamp:

Nov 21, 2019, 4:41:51 PM (4 years ago)

Author:

Jacopo de Amicis

Comment:

Added information about heterogeneous jobs, binding/pinning and SMT.

Public/User_Guide/Batch_system

@@ -136 +136 @@
 }}}
 
+== Heterogeneous jobs ==
+
+As of version 17.11 of Slurm, heterogeneous jobs are supported. For example, the user can run:
+
+{{{
+srun --partition=dp-cn -N 1 -n 1 hostname : --partition=dp-dam -N 1 -n 1 hostname
+dp-cn01
+dp-dam01
+}}}
+
+In order to submit a heterogeneous job, the user needs to set the batch script similarly to the following:
+
+{{{#!sh
+#!/bin/bash
+
+#SBATCH --job-name=imb_execute_1
+#SBATCH --account=deep
+#SBATCH --mail-user=
+#SBATCH --mail-type=ALL
+#SBATCH --output=job.out
+#SBATCH --error=job.err
+#SBATCH --time=00:02:00
+
+#SBATCH --partition=dp-cn
+#SBATCH --nodes=1
+#SBATCH --ntasks=12
+#SBATCH --ntasks-per-node=12
+#SBATCH --cpus-per-task=1
+
+#SBATCH packjob
+
+#SBATCH --partition=dp-dam
+#SBATCH --constraint=
+#SBATCH --nodes=1
+#SBATCH --ntasks=12
+#SBATCH --ntasks-per-node=12
+#SBATCH --cpus-per-task=1
+
+srun ./app_cn : ./app_dam
+}}}
+
+Here the `packjob` keyword allows to define Slurm parameter for each sub-job of the heterogeneous job.
+
+When submitting a heterogeneous job with this colon notation using ParaStationMPI, a unique `MPI_COMM_WORLD` is created, spanning across the two partitions. If this is not desired, one can use the `--pack-group` key to submit independent job steps to the different node-groups of a heterogeneous allocation:
+
+{{{#!sh
+srun --pack-group=0 ./app_cn ; srun --pack-group=1 ./app_dam
+}}}
+
+Using this configuration implies that inter-communication must be established manually by the applications during run time, if needed.
+
+More information about heterogeneous and cross-module jobs (including how to used gateway nodes) can be found on [https://apps.fz-juelich.de/jsc/hps/jureca/modular-jobs.html this page] of the JURECA documentation. All information available there applies for the DEEP system as well. Please be aware that the DEEP system currently includes 2 gateway nodes between the Infiniband and EXTOLL fabrics.
+
+Also, for more information about heterogeneous jobs please refer to the [https://slurm.schedmd.com/heterogeneous_jobs.html relevant page] of the Slurm documentation.
+
+{{{#!comment
+If you need to load modules before launching the application, it's suggested to create wrapper scripts around the applications, and submit such scripts with srun, like this:
+
+{{{#!sh
+...
+srun ./script_sdv.sh : ./script_knl.sh
+}}}
+
+where a script should contain:
+
+{{{#!sh
+#!/bin/bash
+
+module load ...
+./app_sdv
+}}}
+
+This way it will also be possible to load different modules on the different partitions used in the heterogeneous job.
+}}}
+
 == Available Partitions ==
 
@@ -160 +235 @@
 }}}
 
-{{{#!comment information included with the examples
-== Interactive Jobs ==
-
-== Batch Jobs ==
-}}}
+== Information on past jobs and accounting ==
+
+The `sacct` command can be used to enquire the Slurm database about a past job.
+
 
 == FAQ ==
 
+=== Is there a cheat sheet for all main Slurm commands? ===
+
+Yes, it is available [https://slurm.schedmd.com/pdfs/summary.pdf here].
+
 === Why's my job not running? ===
 
@@ -198 +276 @@
 entry.
 
-=== How can get a list of broken nodes? ===
-
-The command to use is
-
-{{{
-sinfo -Rl -h -o "%n %12U %19H %6t %E" | sort -u
-}}}
-
-See also the translation table below.
-
-=== Can I join stderr and stdout like it was done with {{{-joe}}} in Torque? ===
+=== How can check the status of partitions and nodes? ===
+
+The main command to use is `sinfo`. By default, when called alone, `sinfo` will list the available partitions and the number of nodes in each partition in a given status. For example:
+
+{{{
+[deamicis1@deepv hybridhello]$ sinfo
+PARTITION    AVAIL  TIMELIMIT  NODES  STATE NODELIST
+sdv             up   20:00:00     16   idle deeper-sdv[01-16]
+knl             up   20:00:00      1  drain knl01
+knl             up   20:00:00      3   idle knl[04-06]
+knl256          up   20:00:00      1  drain knl01
+knl256          up   20:00:00      1   idle knl05
+knl272          up   20:00:00      2   idle knl[04,06]
+snc4            up   20:00:00      1   idle knl05
+dam             up   20:00:00      1  down* protodam01
+dam             up   20:00:00      3   idle protodam[02-04]
+extoll          up   20:00:00     16   idle deeper-sdv[01-16]
+ml-gpu          up   20:00:00      1   idle ml-gpu01
+dp-cn           up   20:00:00      1  drain dp-cn49
+dp-cn           up   20:00:00      2  alloc dp-cn[01,50]
+dp-cn           up   20:00:00     47   idle dp-cn[02-48]
+dp-dam          up   20:00:00      1 drain* dp-dam01
+dp-dam          up   20:00:00      1  drain dp-dam02
+dp-dam          up   20:00:00     14   down dp-dam[03-16]
+dp-sdv-esb      up   20:00:00      2   idle dp-sdv-esb[01-02]
+psgw-cluster    up   20:00:00      1  down* nfgw01
+psgw-booster    up   20:00:00      1  down* nfgw02
+debug           up   20:00:00      1 drain* dp-dam01
+debug           up   20:00:00      1  down* protodam01
+debug           up   20:00:00      3  drain dp-cn49,dp-dam02,knl01
+debug           up   20:00:00     14   down dp-dam[03-16]
+debug           up   20:00:00      2  alloc dp-cn[01,50]
+debug           up   20:00:00     69   idle deeper-sdv[01-16],dp-cn[02-48],knl[04-06],protodam[02-04]
+}}}
+
+Please refer to the man page for `sinfo` for more information.
+
+=== Can I join stderr and stdout like it was done with {{{-joe}}} in Torque? === 
 
 Not directly. In your batch script, redirect stdout and stderr to the same file:
@@ -221 +326 @@
 (The {{{%j}}} will place the job id in the output file). N.B. It might be more efficient to redirect the output of your script's commands to a dedicated file.
 
-=== What's the equivalent of {{{qsub -l nodes=x:ppn=y:cluster+n_b:ppn=p_b:booster}}}? ===
-
-As of version 17.11 of Slurm, heterogeneous jobs are supported. For example, the user can run:
-
-{{{
-srun --partition=sdv -N 1 -n 1 hostname : --partition=knl -N 1 -n 1 hostname
-deeper-sdv01
-knl05
-}}}
-
-In order to submit a heterogeneous job, the user needs to set the batch script similarly to the following:
-
-{{{#!sh
-#!/bin/bash
-
-#SBATCH --job-name=imb_execute_1
-#SBATCH --account=deep
-#SBATCH --mail-user=
-#SBATCH --mail-type=ALL
-#SBATCH --output=job.out
-#SBATCH --error=job.err
-#SBATCH --time=00:02:00
-
-#SBATCH --partition=sdv
-#SBATCH --constraint=
-#SBATCH --nodes=1
-#SBATCH --ntasks=12
-#SBATCH --ntasks-per-node=12
-#SBATCH --cpus-per-task=1
-
-#SBATCH packjob
-
-#SBATCH --partition=knl
-#SBATCH --constraint=
-#SBATCH --nodes=1
-#SBATCH --ntasks=12
-#SBATCH --ntasks-per-node=12
-#SBATCH --cpus-per-task=1
-
-srun ./app_sdv : ./app_knl
-}}}
-
-Here the `packjob` keyword allows to define Slurm parameter for each sub-job of the heterogeneous job.
-
-If you need to load modules before launching the application, it's suggested to create wrapper scripts around the applications, and submit such scripts with srun, like this:
-
-{{{#!sh
-...
-srun ./script_sdv.sh : ./script_knl.sh
-}}}
-
-where a script should contain:
-
-{{{#!sh
-#!/bin/bash
-
-module load ...
-./app_sdv
-}}}
-
-This way it will also be possible to load different modules on the different partitions used in the heterogeneous job.
-
-
+
+=== What is the default binding/pinning behaviour on DEEP? ===
+
+DEEP uses a !ParTec-modified version of Slurm called psslurm. In psslurm, the options concerning binding and pinning are different from the ones provided in Vanilla Slurm. By default, psslurm will use a ''by rank'' pinning strategy, assigning each Slurm task to a different physical thread on the node starting from OS processor 0. For example:
+
+{{{#!sh
+[deamicis1@deepv hybridhello]$ OMP_NUM_THREADS=1 srun -N 1 -n 4 -p dp-cn ./HybridHello | sort -k9n -k11n
+Hello from node dp-cn50, core 0; AKA rank 0, thread 0
+Hello from node dp-cn50, core 1; AKA rank 1, thread 0
+Hello from node dp-cn50, core 2; AKA rank 2, thread 0
+Hello from node dp-cn50, core 3; AKA rank 3, thread 0
+}}}
+
+**Attention:** please be aware that the psslurm affinity settings only affect the tasks spawned by Slurm. When using threaded  applications, the thread affinity will be inherited from the task affinity of the process originally spawned by Slurm. For example, for a hybrid MPI-OpenMP application:
+{{{#!sh
+[deamicis1@deepv hybridhello]$ OMP_NUM_THREADS=4 srun -N 1 -n 4 -c 4 -p dp-dam ./HybridHello | sort -k9n -k11n
+Hello from node dp-dam01, core 0-3; AKA rank 0, thread 0
+Hello from node dp-dam01, core 0-3; AKA rank 0, thread 1
+Hello from node dp-dam01, core 0-3; AKA rank 0, thread 2
+Hello from node dp-dam01, core 0-3; AKA rank 0, thread 3
+Hello from node dp-dam01, core 4-7; AKA rank 1, thread 0
+Hello from node dp-dam01, core 4-7; AKA rank 1, thread 1
+Hello from node dp-dam01, core 4-7; AKA rank 1, thread 2
+Hello from node dp-dam01, core 4-7; AKA rank 1, thread 3
+Hello from node dp-dam01, core 8-11; AKA rank 2, thread 0
+Hello from node dp-dam01, core 8-11; AKA rank 2, thread 1
+Hello from node dp-dam01, core 8-11; AKA rank 2, thread 2
+Hello from node dp-dam01, core 8-11; AKA rank 2, thread 3
+Hello from node dp-dam01, core 12-15; AKA rank 3, thread 0
+Hello from node dp-dam01, core 12-15; AKA rank 3, thread 1
+Hello from node dp-dam01, core 12-15; AKA rank 3, thread 2
+Hello from node dp-dam01, core 12-15; AKA rank 3, thread 3
+}}}
+
+Be sure to explicitly set the thread affinity settings in your script (e.g. exporting environment variables) or directly in your code. Taking the previous example:
+{{{#!sh
+[deamicis1@deepv hybridhello]$ OMP_NUM_THREADS=4 OMP_PROC_BIND=close srun -N 1 -n 4 -c 4 -p dp-dam ./HybridHello | sort -k9n -k11n
+Hello from node dp-dam01, core 0; AKA rank 0, thread 0
+Hello from node dp-dam01, core 1; AKA rank 0, thread 1
+Hello from node dp-dam01, core 2; AKA rank 0, thread 2
+Hello from node dp-dam01, core 3; AKA rank 0, thread 3
+Hello from node dp-dam01, core 4; AKA rank 1, thread 0
+Hello from node dp-dam01, core 5; AKA rank 1, thread 1
+Hello from node dp-dam01, core 6; AKA rank 1, thread 2
+Hello from node dp-dam01, core 7; AKA rank 1, thread 3
+Hello from node dp-dam01, core 8; AKA rank 2, thread 0
+Hello from node dp-dam01, core 9; AKA rank 2, thread 1
+Hello from node dp-dam01, core 10; AKA rank 2, thread 2
+Hello from node dp-dam01, core 11; AKA rank 2, thread 3
+Hello from node dp-dam01, core 12; AKA rank 3, thread 0
+Hello from node dp-dam01, core 13; AKA rank 3, thread 1
+Hello from node dp-dam01, core 14; AKA rank 3, thread 2
+Hello from node dp-dam01, core 15; AKA rank 3, thread 3
+}}}
+
+Please refer to the [https://apps.fz-juelich.de/jsc/hps/jureca/affinity.html following page] on the JURECA documentation for more information about how to affect affinity on the DEEP system using psslurm options. Please be aware that different partitions on DEEP have different number of sockets per node and cores/threads per socket with respect to JURECA. Please refer to the [wiki:System_overview] or run the `lstopo-no-graphics` on the compute nodes to get more information about the hardware configuration on the different modules. 
+ 
+
+
+=== How do I use SMT on the DEEP CPUs? ===
+
+On DEEP, SMT is enabled by default on all nodes. Please be aware that on all JSC systems (including DEEP), each hardware thread is exposed by the OS as a physical core. For a ''n''-core node, with ''m'' hardware threads per core, the OS cores from ''0'' to ''n-1'' will correspond to the first hardware thread of all hardware cores (from all sockets), the OS cores from ''n'' to ''2n-1'' to the second hardware thread of the hardware cores, and so on.
+
+For instance, on a Cluster node (with two sockets with 12 cores each, with 2 hardware threads per core):
+{{{
+[deamicis1@deepv hybridhello]$ srun -N 1 -n 1 -p dp-cn lstopo-no-graphics --no-caches --no-io --no-bridges --of ascii
+┌────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
+│ Machine (191GB total)                                                                                                                                  │
+│                                                                                                                                                        │
+│ ┌────────────────────────────────────────────────────────────────────────┐  ┌────────────────────────────────────────────────────────────────────────┐ │
+│ │ ┌────────────────────────────────────────────────────────────────────┐ │  │ ┌────────────────────────────────────────────────────────────────────┐ │ │
+│ │ │ NUMANode P#0 (95GB)                                                │ │  │ │ NUMANode P#1 (96GB)                                                │ │ │
+│ │ └────────────────────────────────────────────────────────────────────┘ │  │ └────────────────────────────────────────────────────────────────────┘ │ │
+│ │                                                                        │  │                                                                        │ │
+│ │ ┌────────────────────────────────────────────────────────────────────┐ │  │ ┌────────────────────────────────────────────────────────────────────┐ │ │
+│ │ │ Package P#0                                                        │ │  │ │ Package P#1                                                        │ │ │
+│ │ │                                                                    │ │  │ │                                                                    │ │ │
+│ │ │ ┌─────────────┐  ┌─────────────┐  ┌─────────────┐  ┌─────────────┐ │ │  │ │ ┌─────────────┐  ┌─────────────┐  ┌─────────────┐  ┌─────────────┐ │ │ │
+│ │ │ │ Core P#0    │  │ Core P#1    │  │ Core P#2    │  │ Core P#3    │ │ │  │ │ │ Core P#0    │  │ Core P#3    │  │ Core P#4    │  │ Core P#8    │ │ │ │
+│ │ │ │             │  │             │  │             │  │             │ │ │  │ │ │             │  │             │  │             │  │             │ │ │ │
+│ │ │ │ ┌─────────┐ │  │ ┌─────────┐ │  │ ┌─────────┐ │  │ ┌─────────┐ │ │ │  │ │ │ ┌─────────┐ │  │ ┌─────────┐ │  │ ┌─────────┐ │  │ ┌─────────┐ │ │ │ │
+│ │ │ │ │ PU P#0  │ │  │ │ PU P#1  │ │  │ │ PU P#2  │ │  │ │ PU P#3  │ │ │ │  │ │ │ │ PU P#12 │ │  │ │ PU P#13 │ │  │ │ PU P#14 │ │  │ │ PU P#15 │ │ │ │ │
+│ │ │ │ └─────────┘ │  │ └─────────┘ │  │ └─────────┘ │  │ └─────────┘ │ │ │  │ │ │ └─────────┘ │  │ └─────────┘ │  │ └─────────┘ │  │ └─────────┘ │ │ │ │
+│ │ │ │ ┌─────────┐ │  │ ┌─────────┐ │  │ ┌─────────┐ │  │ ┌─────────┐ │ │ │  │ │ │ ┌─────────┐ │  │ ┌─────────┐ │  │ ┌─────────┐ │  │ ┌─────────┐ │ │ │ │
+│ │ │ │ │ PU P#24 │ │  │ │ PU P#25 │ │  │ │ PU P#26 │ │  │ │ PU P#27 │ │ │ │  │ │ │ │ PU P#36 │ │  │ │ PU P#37 │ │  │ │ PU P#38 │ │  │ │ PU P#39 │ │ │ │ │
+│ │ │ │ └─────────┘ │  │ └─────────┘ │  │ └─────────┘ │  │ └─────────┘ │ │ │  │ │ │ └─────────┘ │  │ └─────────┘ │  │ └─────────┘ │  │ └─────────┘ │ │ │ │
+│ │ │ └─────────────┘  └─────────────┘  └─────────────┘  └─────────────┘ │ │  │ │ └─────────────┘  └─────────────┘  └─────────────┘  └─────────────┘ │ │ │
+│ │ │                                                                    │ │  │ │                                                                    │ │ │
+│ │ │ ┌─────────────┐  ┌─────────────┐  ┌─────────────┐  ┌─────────────┐ │ │  │ │ ┌─────────────┐  ┌─────────────┐  ┌─────────────┐  ┌─────────────┐ │ │ │
+│ │ │ │ Core P#4    │  │ Core P#9    │  │ Core P#10   │  │ Core P#16   │ │ │  │ │ │ Core P#9    │  │ Core P#10   │  │ Core P#11   │  │ Core P#16   │ │ │ │
+│ │ │ │             │  │             │  │             │  │             │ │ │  │ │ │             │  │             │  │             │  │             │ │ │ │
+│ │ │ │ ┌─────────┐ │  │ ┌─────────┐ │  │ ┌─────────┐ │  │ ┌─────────┐ │ │ │  │ │ │ ┌─────────┐ │  │ ┌─────────┐ │  │ ┌─────────┐ │  │ ┌─────────┐ │ │ │ │
+│ │ │ │ │ PU P#4  │ │  │ │ PU P#5  │ │  │ │ PU P#6  │ │  │ │ PU P#7  │ │ │ │  │ │ │ │ PU P#16 │ │  │ │ PU P#17 │ │  │ │ PU P#18 │ │  │ │ PU P#19 │ │ │ │ │
+│ │ │ │ └─────────┘ │  │ └─────────┘ │  │ └─────────┘ │  │ └─────────┘ │ │ │  │ │ │ └─────────┘ │  │ └─────────┘ │  │ └─────────┘ │  │ └─────────┘ │ │ │ │
+│ │ │ │ ┌─────────┐ │  │ ┌─────────┐ │  │ ┌─────────┐ │  │ ┌─────────┐ │ │ │  │ │ │ ┌─────────┐ │  │ ┌─────────┐ │  │ ┌─────────┐ │  │ ┌─────────┐ │ │ │ │
+│ │ │ │ │ PU P#28 │ │  │ │ PU P#29 │ │  │ │ PU P#30 │ │  │ │ PU P#31 │ │ │ │  │ │ │ │ PU P#40 │ │  │ │ PU P#41 │ │  │ │ PU P#42 │ │  │ │ PU P#43 │ │ │ │ │
+│ │ │ │ └─────────┘ │  │ └─────────┘ │  │ └─────────┘ │  │ └─────────┘ │ │ │  │ │ │ └─────────┘ │  │ └─────────┘ │  │ └─────────┘ │  │ └─────────┘ │ │ │ │
+│ │ │ └─────────────┘  └─────────────┘  └─────────────┘  └─────────────┘ │ │  │ │ └─────────────┘  └─────────────┘  └─────────────┘  └─────────────┘ │ │ │
+│ │ │                                                                    │ │  │ │                                                                    │ │ │
+│ │ │ ┌─────────────┐  ┌─────────────┐  ┌─────────────┐  ┌─────────────┐ │ │  │ │ ┌─────────────┐  ┌─────────────┐  ┌─────────────┐  ┌─────────────┐ │ │ │
+│ │ │ │ Core P#18   │  │ Core P#19   │  │ Core P#25   │  │ Core P#26   │ │ │  │ │ │ Core P#17   │  │ Core P#18   │  │ Core P#24   │  │ Core P#26   │ │ │ │
+│ │ │ │             │  │             │  │             │  │             │ │ │  │ │ │             │  │             │  │             │  │             │ │ │ │
+│ │ │ │ ┌─────────┐ │  │ ┌─────────┐ │  │ ┌─────────┐ │  │ ┌─────────┐ │ │ │  │ │ │ ┌─────────┐ │  │ ┌─────────┐ │  │ ┌─────────┐ │  │ ┌─────────┐ │ │ │ │
+│ │ │ │ │ PU P#8  │ │  │ │ PU P#9  │ │  │ │ PU P#10 │ │  │ │ PU P#11 │ │ │ │  │ │ │ │ PU P#20 │ │  │ │ PU P#21 │ │  │ │ PU P#22 │ │  │ │ PU P#23 │ │ │ │ │
+│ │ │ │ └─────────┘ │  │ └─────────┘ │  │ └─────────┘ │  │ └─────────┘ │ │ │  │ │ │ └─────────┘ │  │ └─────────┘ │  │ └─────────┘ │  │ └─────────┘ │ │ │ │
+│ │ │ │ ┌─────────┐ │  │ ┌─────────┐ │  │ ┌─────────┐ │  │ ┌─────────┐ │ │ │  │ │ │ ┌─────────┐ │  │ ┌─────────┐ │  │ ┌─────────┐ │  │ ┌─────────┐ │ │ │ │
+│ │ │ │ │ PU P#32 │ │  │ │ PU P#33 │ │  │ │ PU P#34 │ │  │ │ PU P#35 │ │ │ │  │ │ │ │ PU P#44 │ │  │ │ PU P#45 │ │  │ │ PU P#46 │ │  │ │ PU P#47 │ │ │ │ │
+│ │ │ │ └─────────┘ │  │ └─────────┘ │  │ └─────────┘ │  │ └─────────┘ │ │ │  │ │ │ └─────────┘ │  │ └─────────┘ │  │ └─────────┘ │  │ └─────────┘ │ │ │ │
+│ │ │ └─────────────┘  └─────────────┘  └─────────────┘  └─────────────┘ │ │  │ │ └─────────────┘  └─────────────┘  └─────────────┘  └─────────────┘ │ │ │
+│ │ └────────────────────────────────────────────────────────────────────┘ │  │ └────────────────────────────────────────────────────────────────────┘ │ │
+│ └────────────────────────────────────────────────────────────────────────┘  └────────────────────────────────────────────────────────────────────────┘ │
+└────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘
+┌────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
+│ Host: dp-cn50                                                                                                                                          │
+│                                                                                                                                                        │
+│ Indexes: physical                                                                                                                                      │
+│                                                                                                                                                        │
+│ Date: Thu 21 Nov 2019 15:22:31 CET                                                                                                                     │
+└────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘
+}}}
+The `PU P#X` are the Processing Units numbers exposed by the OS.
+
+To exploit SMT, simply run a job using a number of tasks*threads_per_task higher than the number of physical cores available on a node. Please refer to the [https://apps.fz-juelich.de/jsc/hps/jureca/smt.html relevant page] of the JURECA documentation for more information on how to use SMT on the DEEP nodes.
+
+**Attention**: currently the only way of assign Slurm tasks to hardware threads belonging to the same hardware core is to use the `--cpu-bind` option of psslurm using `mask_cpu` to provide affinity masks for each task. For example:
+{{{#!sh
+[deamicis1@deepv hybridhello]$ OMP_NUM_THREADS=2 OMP_PROC_BIND=close OMP_PLACES=threads srun -N 1 -n 2 -p dp-dam --cpu-bind=mask_cpu:$(printf '%x' "$((2#1000000000000000000000000000000000000000000000001))"),$(printf '%x' "$((2#10000000000000000000000000000000000000000000000010))") ./HybridHello | sort -k9n -k11n
+Hello from node dp-dam01, core 0; AKA rank 0, thread 0
+Hello from node dp-dam01, core 48; AKA rank 0, thread 1
+Hello from node dp-dam01, core 1; AKA rank 1, thread 0
+Hello from node dp-dam01, core 49; AKA rank 1, thread 1
+}}}
+
+This can be cumbersome for jobs using a large number of tasks per node. In such cases, a tool like [https://www.open-mpi.org/projects/hwloc/ hwloc] (currently available on the compute nodes, but not on the login node!) can be used to calculate the affinity masks to be passed to psslurm.
+
+
+{{{#!comment
 == pbs/slurm dictionary ==
 || '''PBS command''' || '''closest slurm equivalent''' ||
@@ -301 +480 @@
 || setres -u <user> ALL || scontrol create reservation ReservationName=\<some name> user=\<user> Nodes=ALL startTime=now duration=unlimited FLAGS=maint,ignore_jobs ||
 || releaseres || scontrol delete ReservationName= <reservation> ||
+}}}