Changes between Version 31 and Version 32 of Public/ParaStationMPI

Timestamp:

May 31, 2021, 11:33:40 AM (3 years ago)

Author:

Carsten Clauß

Comment:

—

Public/ParaStationMPI

@@ -367 +367 @@
 * [https://deeptrac.zam.kfa-juelich.de:8443/trac/attachment/wiki/Public/ParaStationMPI/DEEP-EST_Task_6.1_MPI-NAM-Manual.pdf This Manual for using the NAM as a PDF]
 
-More to come...
+
+=== Introduction ===
+
+One distinct feature of the DEEP-EST prototype is the Network Attached Memory (NAM):
+Special memory regions that can directly be accessed via !Put/Get-operations from any node within the Extoll network.
+For executing such RMA operations on the NAM, a new version of the libNAM is available to the users that features a corresponding low-level API for this purpose.
+However, to make this programming more convenient—and in particular to also support parallel access to shared NAM data by multiple processes—an MPI extension with corresponding wrapper functionalities to the libNAM API has also been developed in the DEEP-EST project.
+
+This extension, which is called PSNAM, is a complementary part of the !ParaStation MPI—which is the MPI library of choice in the DEEP-EST project—and is as such also available to users on the DEEP-EST prototype system.
+In this way, application programmers shall be enabled to use known MPI functions (especially those of the MPI RMA interface) for accessing NAM regions in a standardized (or at least harmonized) way under the familiar roof of an MPI world.
+In doing so, the PSNAM extensions try to stick to the current MPI standard as close as possible and to avoid the introduction of new API functions wherever possible.
+
+----
+**Attention:** ''Currently (as of May 2021), changes to the DEEP-EST system are foreseeable, which will also affect the availability of libNAM and the SW-NAM mockup.
+The following text still reflects the current state and will soon be adapted accordingly.''
+----
+
+=== Acquiring NAM Memory ===
+
+==== General Semantics ====
+
+The main issue when mapping the MPI RMA interface onto the libNAM API is the fact that MPI assumes that all target and memory regions for RMA operations are always associated with an MPI process being the owner of that memory. That means that in an MPI world, remote memory regions are always addressed by means of a process rank (plus handle, which is the respective window object, plus offset), whereas the libNAM API merely requires an opaque handle for addressing the respective NAM region (plus offset). Therefore, a mapping between remote MPI ranks and the remote NAM memory needs somehow to be realized. In PSNAM, this is achieved by sticking to the notion of an ownership in a sense that definite regions of the NAM memory space are logically assigned to particular MPI ranks. However, it has to be emphasised that this is a purely software-based mapping being conducted by the PSNAM wrapper layer. That means that the related MPI window regions (though globally accessible and located within the NAM) have then to be addressed by means of the rank of that process to which the NAM region is assigned.
+
+==== Semantic Terms ====
+
+At this point, the semantic terms of memory ''allocation'', memory ''region'' and memory ''segment'' are to be determined for their use within this proposal.
+The reason for this is that, for example, the term "allocation" is commonly used for both: a resource, as granted by the job scheduler, and a memory region, as returned e.g. by malloc.
+Therefore, we need a stricter nomenclature here:
+
+"''NAM Memory Allocation''":
+A certain amount of contiguous NAM memory space that has been requested from the NAM Manager (and possibly granted through the job scheduler) for an MPI session.
+
+"''NAM Memory Segment''":
+A certain amount of contiguous NAM memory space that is part of a NAM allocation.
+According to this, a NAM allocation can logically be subdivided by the PSNAM wrapper layer into multiple memory segments, which can then again be assigned to MPI RMA windows.
+
+"''NAM Memory Region''":
+A certain amount of contiguous NAM memory space that is associated to a certain MPI rank in the context of an MPI RMA window.
+
+For performance and also for management reasons, allocation requests towards the NAM and/or the resource manager should preferably occur rarely--so, for instance, only once at the beginning of an MPI session.
+In order to provide MPI applications with the ability to handle multiple MPI RMA windows within such an allocation, PSNAM implements a further layer of memory management that allows for a logical acquiring and releasing of NAM segments within the limits of the granted allocation.
+
+
+==== Interface Specification ====
+
+For assigning memory regions on the NAM with MPI RMA windows, a semantic extension to the well-known `MPI_Win_allocate()` function via its MPI info parameter can be used:
+
+{{{
+MPI_Win_allocate(size, disp_unit, info, comm, baseptr, win)
+IN   size      size of memory region in bytes (non-negative integer, may differ between processes)
+IN   disp_unit local unit size for displacements, in bytes (positive integer)
+IN   info      info argument (handle) with psnam info keys and values
+IN   comm      intra-communicator (handle)
+OUT  baseptr   always NULL in case of PSNAM windows
+OUT  win       window object returned by the call (handle)
+}}}
+
+`MPI_Win_allocate()`is a collective call to be executed by all processes in the group of `comm`.
+This in turn enables the PSNAM wrapper layer to treat the set of allocated memory regions as an entity and logically link the regions to a shared RMA window.
+
+The semantic extension compared to the MPI standard is the evaluation of the following keys within the given MPI info object:
+ * `psnam_manifestation`
+ * `psnam_consistency`
+ * `psnam_structure`
+
+The `psnam_manifestation` key specifies which memory type shall be used for a region.
+The value for using the NAM is `psnam_manifestation_libnam` -- but it should be mentioned that also node-local persistent shared-memory ( `psnam_manifestation_pershm`) can here be chosen as another supported manifestation.
+In fact, each process in `comm` can even select a different manifestation of these two for the composition of the window.
+
+The `psnam_consistency` key specifies whether the memory regions of an RMA window shall be persistent (`psnam_consistency_persistent`) or whether they shall be released during the respective `MPI_Win_free()` call (`psnam_consistency_volatile`).
+This key must be selected equally among all processes in `comm`.
+
+The `psnam_structure` key specifies the memory layout as formed by the multiple regions of an MPI window.
+Currently, the following three different memory layouts are supported:
+ * `psnam_structure_raw_and_flat`}
+ * `psnam_structure_managed_contiguous`}
+ * `psnam_structure_managed_distributed`}
+
+The chosen memory layout also decides whether and how the PSNAM layer stores further meta data in the NAM regions to allow a later recreation of the structure while reconnecting to a persistent RMA window by another MPI session.
+The chosen structure must be the same for all processes in `comm`.
+
+"''Raw and Flat''":
+The `psnam_structure_raw_and_flat` layout is intended to store raw data (i.e. untyped data) in the NAM without adding meta information.
+According to this layout, only rank 0 of comm is allowed to pass a size parameter greater than zero during the `MPI_Win_allocate()` call.
+Hence, only rank 0 allocates one (contiguous) NAM region forming the window and all RMA operations on such a flat window have therefore to be addressed to target rank = 0.
+
+"''Managed Contiguous''":
+In the `psnam_structure_managed_contiguous` case, also only rank 0 allocates (contiguous) NAM space, but this space is then subdivided according to the size parameters as passed by all processes in `comm`.
+That means that here also processes with rank > 0 can pass a size > 0 and hence acquire a rank-addressable (sub-)region within this window.
+Furthermore, the information about the number of processes and the respective region sizes forming that window is being stored as meta data within the NAM.
+That way, a subsequent MPI session re-connecting to this window can retrieve this information and hence recreate the former structure of the window. 
+
+"''Managed Distributed''":
+In a `psnam_structure_managed_distributed` window, each process that passes a size > 0 also allocates NAM memory explicitly and on its own.
+It then contributes this memory as a NAM region to the RMA window so that the corresponding NAM allocation becomes directly addressable by the respective process rank.
+The following Figure to illustrates the differences between these three structure layouts.
+
+