Changes between Version 33 and Version 34 of Public/ParaStationMPI

Timestamp:

May 31, 2021, 1:02:22 PM (3 years ago)

Author:

Carsten Clauß

Comment:

—

Public/ParaStationMPI

@@ -538 +538 @@
 }
 }}}
+
+
+=== Releasing PSNAM Memory ===
+
+According to the standard, an MPI RMA window must be freed by the collective call of `MPI_Win_free()`.
+In case of a PSNAM window, the selection of the `psnam_consistency` MPI info key decided whether the corresponding NAM memory regions are to be freed, too.
+Since `MPI_Win_free()` has no info parameter, the corresponding selection has either already to be made when calling `MPI_Win_allocate()` and/or can also be made/changed later by using `MPI_Win_info_set()`.
+
+A sound MPI application must free all MPI window objects before calling `MPI_Finalize()` -- regardless whether the corresponding NAM region should be persistent or not.
+According to this, there are different degrees with respect to the lifetime of an MPI window:
+Common MPI windows just live as long as `MPI_Win_free()  has not been called and the related session is still alive.
+In contrast to this, persistent NAM windows exist as long as the assigned NAM space is granted by the NAM manager.
+Upon an `MPI_Win_free()` call, such windows are merely freed from the perspective of the MPI current application, not from the view of the NAM manager. 
+
+
+=== Attaching to Persistent Memory Regions ===
+
+Obviously, there needs to be a way for subsequent MPI sessions to attach to the persistent NAM regions previous MPI sessions have created.
+The PSNAM wrapper layer enables this to be done via a call to `MPI_Comm_connect()`, which is normally used for establishing communication between distinct MPI sessions:
+
+{{{
+MPI_Comm_connect(window_name, info, root, comm, newcomm)
+IN  window_name // globally unique window name (string, used only on root)
+IN  info    // implementation-dependent information (handle, used only on root)
+IN  root    // rank in comm of root node (integer)
+IN  comm    // intra-communicator over which call is collective (handle)
+OUT newcomm //inter-communicator with server as remote group (handle)
+}}}
+
+When passing a valid name of a persistent NAM window plus an info argument with the key `psnam_window_connect` and the value true, this function will return an inter-communicator that then serves for accessing the remote NAM memory regions.
+However, this returned inter-communicator is just a pseudo communicator that cannot be used for any point-to-point or collective communication, but that rather acts like a handle for RMA operations on a virtual window object embodied by the remote NAM memory.
+In doing so, the original structure of the NAM window is being retained.
+That means that the window is still divided (and thus addressable) in terms of the MPI ranks of that process group that created the window before.
+Therefore, a call to `MPI_Comm_remote_size()` on the returned inter-communicator reveals the former number of processes in that group.
+For actually creating the local representative for the window in terms of an `MPI_Win` datatype, the `MPI_Win_create_dynamic()` function can be used with the inter-communicator as the input and the window handle as the output parameter.
+
+==== Querying Information about a Remote Window ====
+
+After determining the size of the former progress group via `MPI_Comm_remote_size()`, there might also be a demand for getting the information about the remote region sizes as well as the related unit sizes for displacements.
+For this purpose, the PSNAM wrapper hooks into the `MPI_Win_shared_query()` function that returns these values according to the passed rank:
+
+{{{
+MPI_Win_shared_query(win, rank, size, disp_unit, baseptr)
+IN  win       // window object used for communication (handle) 
+IN  rank      // remote region rank 
+OUT size      // size of the region at the given rank 
+OUT disp_unit // local unit size for displacements
+OUT baseptr   // always NULL in case of PSNAM windows
+}}}
+
+
+==== Example ====
+
+{{{
+MPI_Info_create(&win_info);
+MPI_Info_set(win_info, "psnam_window_connect", "true");
+MPI_Comm_connect(window_name, info, 0, MPI_COMM_WORLD, &inter_comm);
+MPI_Info_free(&info);
+
+printf("Connection to persistent memory region established!\n");
+MPI_Comm_remote_size(inter_comm, &remote_group_size);
+printf("Number of former process group that created the NAM window: %d\n", remote_group_size);
+MPI_Win_create_dynamic(MPI_INFO_NULL, inter_comm, &win);
+…
+For (int region_rank=0; region_rank < remote_group_size; region_rank++) {
+MPI_Win_shared_query(win, region_rank, &region_size[i], &disp_unit[i], NULL);
+}
+…
+}}}