Changes between Version 15 and Version 16 of Public/User_Guide/Batch_system

Timestamp:

Feb 6, 2019, 6:29:40 PM (5 years ago)

Author:

Jacopo de Amicis

Comment:

Updated info on modules, available partitions and heterogeneous jobs.

Public/User_Guide/Batch_system

@@ -5 +5 @@
 Please confer /etc/slurm/README.
 
+The documentation of Slurm can be found [https://slurm.schedmd.com/ here]. 
+
 == Overview ==
 
-Slurm offers interactive and batch jobs (scripts submitted into the system). The relevant commands are {{{srun}}} and {{{sbatch}}}. The {{{srun}}} command can be used to spawn processes ('''please do not use mpiexec'''), both from the frontend and from within a batch script. You can also get a shell on a node to work locally there (e.g. to compile your application natively for a special platform.
-
-== !!!OUTDATED!!! Remark about modules ==
-
-Slurm passes the environment from your job submission session directly to the execution environment. The setup as used with torque therefore doesn't work anymore. Please use 
-
-{{{
-# workaround for missing module file
-. /etc/profile.d/modules.sh
-
-module purge
-module load  intel/16.3 parastation/intel1603-e10-5.1.9-1_11_gc11866c_e10 extoll
-}}}
-
-instead.
+Slurm offers interactive and batch jobs (scripts submitted into the system). The relevant commands are `srun` and `sbatch`. The `srun` command can be used to spawn processes ('''please do not use mpiexec'''), both from the frontend and from within a batch script. You can also get a shell on a node to work locally there (e.g. to compile your application natively for a special platform.
+
+== Remark about modules ==
+
+By default, Slurm passes the environment from your job submission session directly to the execution environment. Please be aware of this when running jobs with `srun` or when submitting scripts with `sbatch`. This behavior can be controlled via the `--export` option. Please refer to the [https://slurm.schedmd.com/ Slurm documentation] to get more information about this.
+
+In particular, when submitting job scripts, it is recommended to load the necessary modules within the script and submit the script from a clean environment.
+
 
 == An introductory example ==
 
 Suppose you have an mpi executable named {{{hello_mpi}}}. There are three ways to start the binary.
+
 
 === From a shell on a node ===
@@ -140 +136 @@
 Please note that there is no default partition configured. In order to run a job, you have to specify one of the following partitions, using the {{{--partition=...}}} switch:
 
- * cluster: decommisioned ~~The old DEEP cluster nodes {{{deep[001-128]}}}~~
  * sdv: The DEEP-ER sdv nodes
  * knl: The DEEP-ER knl nodes (all of them, regardless of cpu and configuration)
@@ -147 +142 @@
  * snc4: the knls configured in SNC-4 mode
  * knm: The DEEP-ER knm nodes
+ * ml-gpu: the machine learning nodes equipped with 4 Nvidia Tesla V100 GPUs each
  * extoll: the sdv and knl nodes in the extoll fabric
 
@@ -200 +196 @@
 
 See also the translation table below.
-
-=== Can I still use the old DEEP Booster nodes? ===
-
-Yes, please use
-
-{{{
-qsub -q booster ...
-}}}
-
-You cannot run a common job on both the old DEEP cluster and DEEP booster.
 
 === Can I join stderr and stdout like it was done with {{{-joe}}} in Torque? ===
@@ -226 +212 @@
 === What's the equivalent of {{{qsub -l nodes=x:ppn=y:cluster+n_b:ppn=p_b:booster}}}? ===
 
-Mixing nodes from different partitions will appear in version 17.11 of slurm. As a workaround, you can explicitly request nodes:
-
-{{{
-srun/sbatch --partition=extoll -w cluster1,...,clusterx,booster1,...,boostern_b -n ...
-}}}
-
-With this the same number of processes will be launched on all allocated nodes. With the following example the number of processes per node can be different for each partition. one node of the sdv partition and one of the knl partition is allocated here. The -m plane=X option sets the number of processes on the first part of nodes (in this case 4 and then 1 process is left for the knl node, because -n is set to 5):
-
-{{{
--bash-4.1$ srun --partition=extoll -N2 -n 5 -C '[sdv*1&knl*1]' -m plane=4 hostname
-deeper-sdv16
-deeper-sdv16
-deeper-sdv16
-deeper-sdv16
-knl01
-}}}
-
-To change the node where to start your job (e.g. start on one partition and then spawn the rest of the processes later within your code) please use the -r option for srun. 
-
-{{{
--bash-4.1$ salloc --partition=extoll -N2 -n 5 -C '[sdv*1&knl*1]' -m plane=4
-salloc: Granted job allocation 5581
--bash-4.1$ srun -n 1 -r 1 hostname
-knl02
-}}}
+As of version 17.11 of Slurm, heterogeneous jobs are supported. For example, the user can run:
+
+{{{
+srun --partition=sdv -N 1 -n 1 hostname : --partition=knl -N 1 -n 1 hostname
+deeper-sdv01
+knl05
+}}}
+
+In order to submit a heterogeneous job, the user needs to set the batch script similarly to the following:
+
+{{{#!sh
+#!/bin/bash
+
+#SBATCH --job-name=imb_execute_1
+#SBATCH --account=deep
+#SBATCH --mail-user=
+#SBATCH --mail-type=ALL
+#SBATCH --output=job.out
+#SBATCH --error=job.err
+#SBATCH --time=00:02:00
+
+#SBATCH --partition=sdv
+#SBATCH --constraint=
+#SBATCH --nodes=1
+#SBATCH --ntasks=12
+#SBATCH --ntasks-per-node=12
+#SBATCH --cpus-per-task=1
+
+#SBATCH packjob
+
+#SBATCH --partition=knl
+#SBATCH --constraint=
+#SBATCH --nodes=1
+#SBATCH --ntasks=12
+#SBATCH --ntasks-per-node=12
+#SBATCH --cpus-per-task=1
+
+srun ./app_sdv : ./app_knl
+}}}
+
+Here the `packjob` keyword allows to define Slurm parameter for each sub-job of the heterogeneous job.
+
+If you need to load modules before launching the application, it's suggested to create wrapper scripts around the applications, and submit such scripts with srun, like this:
+
+{{{#!sh
+...
+srun ./script_sdv.sh : ./script_knl.sh
+}}}
+
+where a script should contain:
+
+{{{#!sh
+#!/bin/bash
+
+module load ...
+./app_sdv
+}}}
+
+This way it will also be possible to load different modules on the different partitions used in the heterogeneous job.
+
 
 == pbs/slurm dictionary ==