Changes between Version 24 and Version 25 of Public/ParaStationMPI

Timestamp:

Aug 11, 2020, 12:00:23 PM (5 years ago)

Author:

Carsten Clauß

Comment:

—

Public/ParaStationMPI

@@ -16 +16 @@
 }}}
 The optimal chunk size is highly dependent on the communication pattern and therefore has to be chosen for each application individually.
+
+== Reporting of Statistical Information ==
+
+The recently installed !ParaStation MPI version 5.4.7-1 offers the possibility to collect statistical information and to print a respective report on the number of messages and the distribution over their length at the end of an MPI run.
+This new feature is currently enabled on DEEP-EST for the psmpi installation in the `Devel-2019a` stage:
+{{{
+> module use $OTHERSTAGES
+> module load Stages/Devel-2019a
+> module load GCC/8.3.0
+> module load ParaStationMPI/5.4.7-1
+}}}
+
+For activating this feature for an MPI run, the `PSP_HISTOGRAM=1` environment variable has to be set:
+{{{
+> PSP_HISTOGRAM=1 srun --gw_num=1 -A deep --partition=dp-cn -N2 -n2 ./IMB-MPI1 Bcast -npmin 4 : --partition=dp-dam-ext -N2 -n2 ./IMB-MPI1 Bcast -npmin 4
+
+srun: psgw: requesting 1 gateway nodes
+srun: job 101384 queued and waiting for resources
+srun: job 101384 has been allocated resources
+#------------------------------------------------------------
+#    Intel(R) MPI Benchmarks 2019 Update 5, MPI-1 part
+#------------------------------------------------------------
+
+...
+
+#----------------------------------------------------------------
+# Benchmarking Bcast
+# #processes = 4
+#----------------------------------------------------------------
+       #bytes #repetitions  t_min[usec]  t_max[usec]  t_avg[usec]
+            0         1000         0.03         0.04         0.04
+            1         1000         0.81         6.70         2.81
+            2         1000         0.86         6.79         2.69
+            4         1000         0.84         6.79         2.69
+            8         1000         0.86         6.80         2.72
+           16         1000         0.85         6.76         2.68
+           32         1000         0.87         6.88         2.67
+           64         1000         0.95         7.43         3.38
+          128         1000         0.98         7.02         3.18
+          256         1000         0.91         8.11         3.68
+          512         1000         0.91        10.46         4.80
+         1024         1000         1.01        11.13         5.59
+         2048         1000         1.07        11.91         6.12
+         4096         1000         1.35        12.77         6.78
+         8192         1000         1.77        14.81         8.23
+        16384         1000         3.24        18.66        11.19
+        32768         1000         4.93        25.96        16.14
+        65536          640        30.06        38.71        34.03
+       131072          320        44.85        60.80        52.53
+       262144          160        66.28       100.63        83.20
+       524288           80       109.16       180.59       144.57
+      1048576           40       199.61       343.00       271.12
+      2097152           20       377.66       666.27       521.72
+      4194304           10       736.83      1314.28      1025.35
+
+
+# All processes entering MPI_Finalize
+
+     bin  freq
+      64  353913
+     128  6303
+     256  6303
+     512  6303
+    1024  6311
+    2048  6303
+    4096  6303
+    8192  6303
+   16384  6303
+   32768  6303
+   65536  4035
+  131072  2019
+  262144  1011
+  524288  507
+ 1048576  255
+ 2097152  129
+ 4194304  66
+ 8388608  0
+16777216  0
+33554432  0
+67108864  0
+}}}
+
+As one can see, the messages being exchanged between all processes of the run are sorted into ''bins'' according to their message lengths.
+The number of bins as well as their limits can be adjusted by the following variables:
+
+ * `PSP_HISTOGRAM_MIN` (default: 64 bytes) Set the lower limit regarding the message size for controlling the number of bins of the histogram.
+ * `PSP_HISTOGRAM_MAX` (default: 64 MByte) Set the upper limit regarding the message size for controlling the number of bins of the histogram.
+ * `PSP_HISTOGRAM_SHIFT` (default: 1 bit position) Set the bit shift regarding the step width for controlling the number of bins of the histogram.
+
+Example:
+{{{
+> PSP_HISTOGRAM=1 PSP_HISTOGRAM_SHIFT=2 PSP_HISTOGRAM_MAX=4096 srun --gw_num=1 -A deep --partition=dp-cn -N2 -n2 ./IMB-MPI1 Barrier -npmin 4 : --partition=dp-dam-ext -N2 -n2 ./IMB-MPI1 Barrier -npmin 4
+
+...
+
+#---------------------------------------------------
+# Benchmarking Barrier
+# #processes = 4
+#---------------------------------------------------
+ #repetitions  t_min[usec]  t_max[usec]  t_avg[usec]
+         1000         5.02         5.02         5.02
+
+
+# All processes entering MPI_Finalize
+
+ bin  freq
+  64  16942
+ 256  0
+1024  8
+4096  0
+}}}
+
+In this example, 16942 messages were smaller than or equal to 64 Byte of MPI payload, while 8 messages were greater than 256 Byte but smaller than or equal to 1024 Byte.
+
+Please note at this point that all messages larger than `PSP_HISTOGRAM_MAX` are as well counted and always fall into the last bin.
+Therefore, in this example, no message of the whole run was larger than 1024 Byte, because the last bin, labeled with 4096 but collecting all messages larger than 1024, is empty.