System overview

This page is supposed to give a short overview on the available systems from a hardware point of view. All hardware can be reached through a login node via SSH: deep@fz-juelich.de. The login node is implemented as virtual machine hosted by the master nodes (in a failover mode). Please, see also information about getting an account and using the batch system.

DEEP-EST Modular Supercomputer (prototype system)

The DEEP-EST system is a prototype of Modular Supercomputing Architecture (MSA) consisting of the following components:

• Cluster Module (CM)
• Data Analytics Module (DAM)
• Scalable Storage and Servcies Module (SSSM)
• Software Development Vehicle (SDV)

In addition to the two compute modules (CM and DAM) an All Flash Storage Module (AFSM) leverages a fast local work filesystem (/afsm) on the compute nodes. The AFSM is integrated in the Scalable Storage and Servcies Module (SSSM). An additional Software Development Vehicle comprises extra Hardware for Software Development. All modules are connected via a 100 Gbp/s EDR IB network in a non-blocking tree topology accompanied by a Gigabit Ethernet service network. In addition the system is connected to the Jülich storage system (JUST) to share home and project file systems with other HPC systems hosted at Jülich Supercompting Centre (JSC). JUST also hosts the entire software stack für the DEEP-EST prototype.

Cluster Module

The CM is composed of 50 nodes with the following hardware specifications:

Cluster [50 nodes]: dp-cn[01-50]: 2 Intel Xeon 'Skylake' Gold 6146 (12 cores (24 threads), 3.2GHz) 192 GB RAM 1 x 400GB NVMe SSD network: InfiniBand EDR (100 Gb/s)

Data Analytics Module

The DAM is composed of 16 nodes with the following hardware specifications:

Data Analytics Module [16 nodes]: dp-dam[01-16] 2 x Intel Xeon 'Cascade Lake' Platinum 8260M CPU @ 2.40GHz dp-dam[01-08]: 1 x Nvidia V100 Tesla GPU (32 GB HBM2) dp-dam[09-12]: 2 x Nvidia V100 Tesla GPU (32 GB HBM2) dp-dam[13-16]: 2 x Intel STRATIX10 FPGA (32 GB DDR4) 384 GB RAM + 3 TB non-volatile memory 2 x 1.5 TB Intel Optane SSD (1 for local scratch, 1 for BeeOND) 1 x 240 GB SSD (for boot and OS) network: IB EDR (100 Gb/s)

Software Development Vehicle (SDV)

Along with the actual prototype system several test nodes and so called software development vehicles (SDVs) have been installed in the scope of the DEEP(-ER,EST) projects. These are located in the SDV rack. intelmax and ml-gpu nodes can be accessed by the users via SLURM. Access to the remaining SDV nodes can be given on demand:

• Intel MAX [2 nodes]: dp-intelmax[01,01]
  • 2 x Intel® Xeon® CPU Max 9462 (2.7 GHz, 32 cores per socket)
  • 1 TB RAM + 128 GB HBM
  • 1 NVMe with 1 TB per node
  • network: EDR Infiniband 100 Gb/s

• GPU nodes for Machine Learning [3 nodes]: ml-gpu[01-03]
  • 2 x Intel Xeon 'Skylake' Silver 4112 (2.6 GHz)
  • 192 GB RAM
  • 4 x Nvidia Tesla V100 GPU (PCIe Gen3), 16 GB HBM2
  • network: 40GbE connection inbetween, 1 GbE external

Scalable Storage And Services Module

This module provides the required frontends in terms of two master nodes (running several service VMs in a high availability setup) and additional service nodes

Here are the specifications of the main hardware components more in detail:

SSSM [8 servers]: dp-master[01,02], dp-fs[01-06]: 2 Intel Xeon Silver 4114 (20 cores, 2.2 GHz) 96 GB RAM 2 x 240 GB SSD (additional 2 x 480 GB SSD in dp-fs[01-02]) network: IB EDR (100 Gb/s)

The SSSM also includes the All Flash Storage Module which is based on PCIe3 NVMe SSD storage devices. It is composed of 6 volume data server systems and 2 metadata servers interconnected with a 100 Gbps EDR-InfiniBand fabric. The BeeGFS global parallel file system is used to make 1.3 PB of data storage capacity available.

Here are the specifications of the main hardware components more in detail:

AFSM [2 metadata servers]: dp-afsm-m[01-02]: 2 Intel Xeon Scalable Gold 6246 (12 cores (24 threads), 3.30 GHz) 192 GB RAM 25.6 TB SSD PCIe3 NVMe (using 8 x 3.2 TB Intel SSD DC P4610) network: 1 x 100 Gbps EDR-InfiniBand HCA (PCIe3 x16) AFSM [6 volume data servers]: dp-afsm-o[01-06]: 2 Intel Xeon Scalable Gold 6226R (16 cores (32 threads), 2.90 GHz) 384 GB RAM 308 TB SSD PCIe3 NVMe (using 24 x 15.36 TB Intel SSD DC P4326) network: 1x 100 Gbps EDR-InfiniBand HCA (PCIe3 x16)

Interconnect

As shown in the system overview an EDR IB non-blocking fat tree is used as fast interconnect inside and between all modules along with a Gigabit Ethernet service network (used for administration). The IB fat tree is composed of 6 spine and 10 leaf switches:

Rack plan

This is a sketch of the available hardware reflecting the current rack layout.

Further information

• Information about the batchsystem
• Filesystems
• Information on available software and tools

• Use the DEEP-EST Cluster Module
• Use the DEEP-EST Data Analytics Module
• Use the SDV Cluster
• Use the SDV KNLs