Changes between Version 17 and Version 18 of Public/User_Guide/DEEP-EST_DAM

Timestamp:

Dec 8, 2020, 11:52:07 AM (3 years ago)

Author:

Jacopo de Amicis

Comment:

Restructured information about multi-node MPI jobs on the DAM nodes.

Public/User_Guide/DEEP-EST_DAM

@@ -115 +115 @@
 Multi-node jobs can be launched on the `dp-dam` partition with ParaStationMPI by loading the `pscom` module (currently `pscom/5.3.1-1`) and the `extoll` module. Please beware that the `extoll` module can be loaded only on nodes with an EXTOLL device, therefore it cannot be loaded on the login node: please load it in a batch script for `sbatch` or directly on the compute nodes within an interactive session (see [wiki:Batch_system#Fromashellonanode here] for more information on the interactive sessions).
 }}}
-Multi-node jobs can be launched on the `dp-dam` partition with !ParaStationMPI by loading Intel (or GCC) and ParaStationMPI modules. There is no need to manually load the `extoll` or `pscom` modules anymore unless you would like to test new features only available in a certain development version of the pscom.
+Multi-node MPI jobs can be launched on the DAM nodes with !ParaStation MPI by loading the `Intel` (or `GCC`) and `ParaStationMPI` modules.
+{{{#!comment JDA: 2020-12-08 I believe this would just confuse the users
+There is no need to manually load the `extoll` or `pscom` modules anymore unless you would like to test new features only available in a certain development version of the pscom.
+}}}
+
+**Extoll:** As of 12.12.2019, the first half of the DAM nodes (`dp-dam[01-08]`) has only GbE connectivity, while the second half has also the faster Extoll interconnect active (nodes `dp-dam[09-16]`). To run multi-node MPI jobs on the DAM nodes, it is strongly recommended to use the `dp-dam-ext` partition, which includes only the nodes providing EXTOLL connectivity. If necessary, users can also run MPI jobs on the other DAM nodes (using the `dp-dam` partition) by setting the `PSP_TCP=1` environment variable in their scripts. This will cause any MPI communication to go through the slower 40 Gb Ethernet fabric.
 
 A release-candidate version of ParaStationMPI with CUDA awareness and GPU direct support for Extoll is currently being tested. Once released it will become available on the DAM nodes with the modules environment.
@@ -123 +128 @@
 For using Cluster nodes in heterogeneous jobs together with CM and/or ESB nodes, please see info about [https://deeptrac.zam.kfa-juelich.de:8443/trac/wiki/Public/User_Guide/Batch_system#Heterogeneousjobs heterogeneous jobs].
 
-**Extoll:** As of 12.12.2019, the first half of the DAM nodes has GbE network (partition=dp-dam,nodeslist=dp-dam[01-16]), the second half has Extoll interconnect (partition=dp-dam-ext,nodeslist=dp-dam[09-16]).