Changes between Version 31 and Version 32 of Public/User_Guide/Batch_system

Timestamp:

Apr 14, 2020, 10:02:11 PM (4 years ago)

Author:

Zia Ul Huda

Comment:

—

Public/User_Guide/Batch_system

@@ -319 +319 @@
 
 
+== Workflows ==
+
+The new version of the installed slurm now supports workflows. The idea is to have an overlap between the dependent jobs so that they can communicate the data over the network instead of writing and reading it on storage. To enable the workflows, we have introduced a new switch {{{delay}}} to {{{sbatch}}} command. Here is a simple example script.
+
+{{{
+[huda1@deepv scripts]$ cat test.sh 
+#!/bin/sh
+
+NAME=$(hostname)
+echo "$NAME: Going to sleep for $1 seconds"
+sleep $1
+echo "$NAME: Awake"
+
+[huda1@deepv scripts]$ cat batch_workflow.sh 
+#!/bin/bash
+#SBATCH -p sdv -N2 -t3
+
+#SBATCH packjob
+
+#SBATCH -p sdv -N1 -t3 --delay 2
+
+srun test.sh 175
+
+[huda1@deepv scripts]$ 
+}}}
+
+In the above {{{sbatch}}} script, the usage of {{{--delay}}} can be seen. It takes thee values in minutes. The idea is to delay the corresponding job of a heterogeneous job by the provided number of minutes from the start of the first job in the job pack. After submission of this job pack, slurm divides it into separate jobs at the time of the resource reservation. So you can see multiple jobs in the output of {{{squeue}}} command. Here is the example execution of this script.
+
+{{{
+[huda1@deepv scripts]$ sbatch batch_workflow.sh 
+Submitted batch job 81458
+[huda1@deepv scripts]$ squeue -u huda1
+             JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
+             81458       sdv batch_wo    huda1 CF       0:01      2 deeper-sdv[02-03]
+             81459       sdv batch_wo    huda1 PD       0:00      1 (Reservation)
+
+[huda1@deepv scripts]$ 
+}}}
+
+Here the second job(81458) will start 2 minutes after the start of the first job(81459). Similarly, the output files will be different for each separated job in the job pack. the final outputs are:
+{{{
+[huda1@deepv scripts]$ cat slurm-81458.out 
+deeper-sdv02: Going to sleep for 175 seconds
+deeper-sdv03: Going to sleep for 175 seconds
+deeper-sdv02: Awake
+deeper-sdv03: Awake
+
+[huda1@deepv scripts]$ cat slurm-81459.out 
+deeper-sdv01: Going to sleep for 175 seconds
+deeper-sdv01: Awake
+
+[huda1@deepv scripts]$ 
+}}}
+
+Another feature to note is that if there are multiple jobs in a job pack and any number of consecutive jobs have the same {{{delay}}} values, they are combined into a new heterogeneous job. Here is an example of such a script:
+{{{
+[huda1@deepv scripts]$ cat batch_workflow_complex.sh 
+#!/bin/bash
+
+#SBATCH -p sdv -N 2 -t 3
+#SBATCH -J first 
+
+#SBATCH packjob
+
+#SBATCH -p sdv -N 1 -t 3 --delay 2
+#SBATCH -J second
+
+#SBATCH packjob
+
+#SBATCH -p sdv -N 1 -t 2 --delay 2
+#SBATCH -J second
+
+#SBATCH packjob
+
+#SBATCH -p sdv -N 2 -t 3 --delay 4
+#SBATCH -J third
+
+if [ "$SLURM_JOB_NAME" == "first" ]
+then
+        srun ./test.sh 150
+
+elif [ "$SLURM_JOB_NAME" == "second" ]
+then
+        srun ./test.sh 150 : ./test.sh 115
+
+elif [ "$SLURM_JOB_NAME" == "third" ]
+then
+        srun ./test.sh 155
+
+fi
+
+[huda1@deepv scripts]$ 
+}}}
+
+Note the {{{delay}}} values for the second and third job in the script are equal. Also, note the usage of the environment variable {{{SLURM_JOB_NAME}}} in the script to decide which command to run in which job. The example execution leads to the following:
+{{{
+[huda1@deepv scripts]$ sbatch batch_workflow_complex.sh 
+Submitted batch job 81460
+
+[huda1@deepv scripts]$ squeue -u huda1
+             JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
+           81461+0       sdv   second    huda1 PD       0:00      1 (Resources)
+           81461+1       sdv   second    huda1 PD       0:00      1 (Resources)
+             81463       sdv    third    huda1 PD       0:00      2 (Resources)
+             81460       sdv    first    huda1 PD       0:00      2 (Resources)
+
+[huda1@deepv scripts]$ 
+}}}
+
+Note that the submitted heterogeneous job has been divided into a single job (81460), a job pack (81461) and again a single job (81643). Similarly, three different output files will be generated, one for each new job.
+{{{
+[huda1@deepv scripts]$ cat slurm-81460.out 
+deeper-sdv03: Going to sleep for 150 seconds
+deeper-sdv04: Going to sleep for 150 seconds
+deeper-sdv03: Awake
+deeper-sdv04: Awake
+
+[huda1@deepv scripts]$ cat slurm-81461.out 
+deeper-sdv01: Going to sleep for 150 seconds
+deeper-sdv02: Going to sleep for 115 seconds
+deeper-sdv02: Awake
+deeper-sdv01: Awake
+
+[huda1@deepv scripts]$ cat slurm-81463.out 
+deeper-sdv01: Going to sleep for 155 seconds
+deeper-sdv02: Going to sleep for 155 seconds
+deeper-sdv01: Awake
+deeper-sdv02: Awake
+
+[huda1@deepv scripts]$ 
+}}}
+If a job exits earlier than the allocated time asked by the user, the corresponding reservation for this job is deleted automatically and the resources become available for the other jobs. However, users should be careful with the requested time when submitting workflows as the larger time values can delay the scheduling of the workflows depending on the situation of the resources.
 
 == Information on past jobs and accounting ==