Changes between Version 36 and Version 37 of Public/User_Guide/Batch_system

Timestamp:

Jul 28, 2020, 11:40:21 AM (4 years ago)

Author:

Zia Ul Huda

Comment:

—

Public/User_Guide/Batch_system

@@ -321 +321 @@
 == Workflows ==
 
-The new version of the installed slurm now supports workflows. The idea is to have an overlap between the dependent jobs so that they can communicate the data over the network instead of writing and reading it on storage. To enable the workflows, we have introduced a new switch {{{delay}}} to {{{sbatch}}} command. Here is a simple example script.
+The new version of the installed slurm now supports workflows. The idea is to have an overlap between the dependent jobs so that they can communicate the data over the network instead of writing and reading it on storage. We have provided two ways to achieve a workflow. One way is to use the new {{{delay}}} switch provided in {{{sbatch}}} command. While the other method is to submit jobs with dependencies of type {{{afterok}}} and later the independent job changes the dependency type of the dependent job to {{{after}}} using our provided shared library (explained below). Jacopo has developed an example project [https://gitlab.version.fz-juelich.de/deamicis1/mpi_connect_test/-/tree/test_zia_workflows] that uses all the features discussed here.
+
+The following simple example script helps understanding the mechanism of new {{{delay}}} switch for workflows.
 
 {{{
@@ -450 +452 @@
 [huda1@deepv scripts]$ 
 }}}
-If a job exits earlier than the allocated time asked by the user, the corresponding reservation for this job is deleted automatically and the resources become available for the other jobs. However, users should be careful with the requested time when submitting workflows as the larger time values can delay the scheduling of the workflows depending on the situation of the resources.
+If a job exits earlier than the allocated time asked by the user, the corresponding reservation for this job is deleted 5 minutes after the end of the job, automatically and the resources become available for the other jobs. However, users should be careful with the requested time when submitting workflows as the larger time values can delay the scheduling of the workflows depending on the situation of the resources.
+
+The workflows created using {{{delay}}} switch ensure overlap between the applications. The second method that includes dependencies among jobs, does not ensure an overlap but avoids users to guess the time a job will take and how much should be the delay between jobs. The process is simple. A user submits a job and later a dependent job with a dependency of type {{{afterok}}}. Inside the first (independent) job, the application running calls the function provided in {{{slurm_workflow}}} library, that changes the dependency type of the dependent job to {{{after}}}. This enables the dependent job to be eligible for allocation by slurm immediately. However, the allocation of resources depends upon the situation of resources available in the system. The following script helps to submit jobs in the form of a chain with a provided dependency type.
+{{{
+[huda1@deepv scripts]$ cat chain_jobs.sh 
+#!/usr/bin/env bash
+
+if [ $# -lt 3 ]
+then
+    echo "$0: ERROR (MISSING ARGUMENTS)"
+    exit 1
+fi
+
+LOCKFILE=$1
+DEPENDENCY_TYPE=$2
+shift 2
+SUBMITSCRIPT=$*
+
+
+if  [ -f $LOCKFILE ]
+then
+    if [[ "$DEPENDENCY_TYPE" =~ ^(after|afterany|afterok|afternotok)$ ]]; then  
+        DEPEND_JOBID=`head -1 $LOCKFILE`
+        echo "sbatch --dependency=${DEPENDENCY_TYPE}:${DEPEND_JOBID} $SUBMITSCRIPT"
+        JOBID=`sbatch --dependency=${DEPENDENCY_TYPE}:${DEPEND_JOBID} $SUBMITSCRIPT`
+    else
+        echo "$0: ERROR (WRONG DEPENDENCY TYPE: choose among 'after', 'afterany', 'afterok' or 'afternotok')"
+    fi
+else
+    echo "sbatch $SUBMITSCRIPT"
+    JOBID=`sbatch $SUBMITSCRIPT`
+fi
+
+echo "RETURN: $JOBID"
+# the JOBID is the last field of the output line
+echo ${JOBID##* } > $LOCKFILE
+
+exit 0
+}}}
+
+Here is the example of submission.
+{{{
+[huda1@deepv scripts]$ ./chain_jobs.sh lockfile afterok simple_job.sh 
+sbatch simple_job.sh
+RETURN: Submitted batch job 98626
+[huda1@deepv scripts]$ ./chain_jobs.sh lockfile afterok simple_job.sh 
+sbatch --dependency=afterok:98626 simple_job.sh
+RETURN: Submitted batch job 98627
+[huda1@deepv scripts]$ ./chain_jobs.sh lockfile afterok simple_job.sh 
+sbatch --dependency=afterok:98627 simple_job.sh
+RETURN: Submitted batch job 98628
+[huda1@deepv scripts]$ squeue -u huda1
+             JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
+             98627       sdv simple_j    huda1 PD       0:00      2 (Dependency)
+             98628       sdv simple_j    huda1 PD       0:00      2 (Dependency)
+             98626       sdv simple_j    huda1  R       0:21      2 deeper-sdv[01-02]
+[huda1@deepv scripts]$ scontrol show job 98628 | grep Dependency
+   JobState=PENDING Reason=Dependency Dependency=afterok:98627
+[huda1@deepv scripts]$ cat lockfile 
+98628
+}}}
+Note that the {{{lockfile}}} contains the id of last submitted job.
 
 === {{{slurm_workflow}}} Library ===