Latest news on the DEEP-EST prototype system

This is a summary of the latest news concerning the system. For a list of known problems related to the system, please refer to this page.

Last update: 2022-09-21

System hardware

CM nodes

• the cluster nodes have direct EBR IB access to the SSSM storage nodes now (without using the IB ↔ 40 GbE gateway)

ESB nodes

• all ESB nodes (dp-esb[01-75]) are using EDR Infiniband interconnect (no Extoll anymore)
• SSSM and AFSM file servers can be directly accessed through IB

DAM nodes

• DAM nodes are using EDR Infiniband (instead of using 40 GbE and Extoll) now
• SSSM and AFSM file servers can be directly accessed through IB

Network Federation Gateways

• with the all IB solution NFGWs are not needed anymore ! this also affects heterogenous jobs

• dp-nfgw[01,04] (IB EDR ↔ 40GbE) still present, but not in use anymore

• remaining NFGWs are being used for BXI testing now: dp-nfgw[02,03,05,06]

SDV

• 4 node BXI test setup has been installed using the former GW nodes

• FPGA test nodes available for using FPGAs with oneAPI, OpenCL:
  • Arria10: deeper-sdv[09,10]
  • Stratix10: dp-sdv-esb[01,02]

File Systems

please also refer to the Filesystems overview

• quota has been added to /tmp on deepv to avoid congestion
• the All Flash Storage Module (AFSM) provides a fast work file system mounted to /afsm (symbolic link to /work) on all compute nodes (CM, DAM, ESB) and the login node (deepv)
• the older System Services and Storage Module (SSSM) work file system is obsolete, but still available at (/work_old) for data migration
• SSSM still serves the /usr/local/software file system, but
  • starting from Rocky 8 image /usr/local will be a local file system
  • /usr/local/software will be shared and provided by the SSSM storage
  • in addition to the !Easybuild software stack the shared /usr/local/software filesystem will contain manually installed software in a legacy subfolder

System software

• ParaStation? update (psmgmt) to 5.1.50-5 has been performed

OS

• preparing update to Rocky 8.6

• the decision was taken to use one Rocky linux image for all compute nodes

• transition from CentOS to Rocky Linux 8.5 has been finished on compute nodes

• login node deepv has moved to Rocky linux 8 as well
• file servers and master nodes to follow

EasyBuild

• 2022 EasyBuild stage is the default now