Changes between Version 7 and Version 8 of Public/User_Guide/TAMPI_NAM

Timestamp:

Mar 3, 2021, 7:38:08 PM (3 years ago)

Author:

Kevin Sala

Comment:

—

Public/User_Guide/TAMPI_NAM

@@ -31 +31 @@
 When there is a timestep that requires a snapshot, the application instantiates multiple tasks that save the matrix data into the corresponding NAM subregion. Each MPI rank creates a task for saving the data of each matrix block into the NAM subregion. These communication tasks do not have any data dependency between them, so they can run in parallel writing data to the NAM region using regular `MPI_Put`. Ranks only write to the subregions that belong to themselves, never in other ranks' subregions. Even so, all `MPI_Put` calls must be done inside an RMA access epoch, so there must be one fence call before all the `MPI_Put` calls and another one after them to close the epoch for each of the timesteps with snapshot. Thus, here is where we use the new function `MPI_Win_ifence` together with the TAMPI non-blocking support. In this way, we taskify both synchronization and writing of NAM regions, keeping the data-flow model, and without having to stop the parallelism (e.g., with a `taskwait`) to perform the snapshots. Thanks to the task data dependencies and TAMPI, we cleanly include the snapshots in the application's data-flow execution as any other regular task.
 
+The following pseudo-code shows how the saving of snapshots work in `02.heat_itampi_ompss2_tasks.bin`:
+
+{{{#!c
+void solve() {
+    int namSnapshotFreq = ...;
+    int namSnapshotId = 0;
+
+    for (t = 1; t <= timesteps; ++t) {
+        // Computation and communication tasks declaring
+        // dependencies on the blocks they process
+        gaussSeidelSolver(...);
+
+        if (t % namSnapshotFreq == 0) {
+            namSaveMatrix(namSnapshotId, namWindow, ...);
+            ++namSnapshotId;
+        }
+    }
+    #pragma oss taskwait
+}
+}}}
+
+{{{#!c
+void namSaveMatrix(int namSnapshotId, MPI_Win namWindow, ...) {
+    // Compute destination offset in NAM region
+    int snapshotOffset = namSnapshotId*sizeof(..all blocks..);
+
+    // Open RMA access epoch to write the NAM window for this timestep
+    #pragma oss task in(..all blocks..) inout(namWindow)
+    {
+        MPI_Request request;
+        MPI_Win_ifence(namWindow, 0, &request);
+        TAMPI_Iwait(&request, MPI_STATUS_IGNORE);
+    }
+
+    // Write all blocks from the current rank to NAM subregions concurrently
+    for (B : all blocks) {
+        #pragma oss task in(..block B..) in(namWindow)
+        {
+            MPI_Put(/* origin */ ..block B..,
+                /* target rank */ currentRank,
+                /* target offset */ snapshotOffset + B,
+                /* target window */ namWindow);
+        }
+    }
+
+    // Close RMA access epoch to write the NAM window for this timestep
+    #pragma oss task in(..all blocks..) inout(namWindow)
+    {
+        MPI_Request request;
+        MPI_Win_ifence(namWindow, 0, &request);
+        TAMPI_Iwait(&request, MPI_STATUS_IGNORE);
+    }
+}
+}}}
+
+
 === Requirements ===
 The requirements of this application are shown in the following lists. The main requirements are: