Changes between Version 28 and Version 29 of Public/User_Guide/Batch_system

Timestamp:

Apr 6, 2020, 8:37:10 AM (4 years ago)

Author:

Simon Pickartz

Comment:

Update example script for "Heterogeneous jobs with MPI communication across modules"

Public/User_Guide/Batch_system

@@ -228 +228 @@
 
 In order to establish MPI communication across modules using different interconnect technologies, some special Gateway nodes must be used. On the DEEP-EST system, MPI communication across gateways is needed only between Infiniband and Extoll interconnects.
+
 **Attention:** Only !ParaStation MPI supports MPI communication across gateway nodes.
 
@@ -239 +240 @@
 # Use the packjob feature to launch separately CM and DAM executable
 
-#SBATCH --job-name=imb
+
+# General configuration of the job
+#SBATCH --job-name=modular-imb
 #SBATCH --account=deep
-#SBATCH --output=IMB-%j.out
-#SBATCH --error=IMB-%j.err
-#SBATCH --time=00:05:00
+#SBATCH --time=00:10:00
+#SBATCH --output=modular-imb-%j.out
+#SBATCH --error=modular-imb-%j.err
+
+# Configure the gateway daemon
 #SBATCH --gw_num=1
-#SBATCH --gw_binary=/opt/parastation/bin/psgwd.extoll
 #SBATCH --gw_psgwd_per_node=1
 
+# Configure node and process count on the CM
 #SBATCH --partition=dp-cn
 #SBATCH --nodes=1
-#SBATCH --ntasks=1
+#SBATCH --ntasks-per-node=1
 
 #SBATCH packjob
 
+# Configure node and process count on the DAM
 #SBATCH --partition=dp-dam-ext
 #SBATCH --nodes=1
-#SBATCH --ntasks=1
-
+#SBATCH --ntasks-per-node=1
+
+# Echo job configuration
 echo "DEBUG: SLURM_JOB_NODELIST=$SLURM_JOB_NODELIST"
 echo "DEBUG: SLURM_NNODES=$SLURM_NNODES"
 echo "DEBUG: SLURM_TASKS_PER_NODE=$SLURM_TASKS_PER_NODE"
 
-# Execute
-srun hostname : hostname
-srun module_dp-cn.sh : module_dp-dam-ext.sh
-}}}
-
-It uses two execution scripts for loading the correct environment and starting the IMB on the CM and the DAM node (this approach can also be used to start different programs, e.g. one could think of a master and worker use case). The execution scripts could look like:
-
-{{{
-#!/bin/bash 
-# Script for the CM using InfiniBand 
-
+
+# Set the environment to use PS-MPI
 module --force purge
 module use $OTHERSTAGES
@@ -279 +277 @@
 module load ParaStationMPI
 
-# Execution 
-EXEC=$PWD/mpi-benchmarks/IMB-MPI1 
-${EXEC} PingPong 
-}}}
-
-{{{
-#!/bin/bash 
-# Script for the DAM using Extoll 
-
-module --force purge
-module use $OTHERSTAGES
-module load Stages/Devel-2019a
-module load Intel
-module load ParaStationMPI
-
-# Execution 
-EXEC=$PWD/mpi-benchmarks/IMB-MPI1 
-${EXEC} PingPong
-}}}
+# Show the hosts we are running on
+srun hostname : hostname
+
+# Execute
+APP="/p/project/cfa_partec/pickartz/mpi-benchmarks/src_c/IMB-MPI1 Uniband"
+srun ${APP}  : ${APP}
+}}}
+
+
 
 **Attention:** During the first part of 2020, only the DAM nodes will have Extoll interconnect, while the CM and the ESB nodes will be connected via Infiniband. This will change later during the course of the project (expected Summer 2020), when the ESB will be equipped with Extoll connectivity (Infiniband will be removed from the ESB and left only for the CM).