Table of Contents
System overview
This page is supposed to give a short overview on the available systems from a hardware point of view. All hardware can be reached through a login node via SSH: deep@fz-juelich.de. The login node is implemented as virtual machine hosted by the master nodes (in a failover mode). Please, see also information about getting an account and using the batch system.
DEEP-EST Modular Supercomputer (prototype system)
The DEEP-EST system is a prototype of Modular Supercomputing Architecture (MSA) consisting of the following modules:
- Cluster Module (CM)
- Extreme Scale Booster (ESB)
- Data Analytics Module (DAM)
In addition to the three compute modules, a Scalable Storage Service Module (SSSM) provides shared storage infrastructure for the DEEP-EST prototype (/usr/local) and is accompanied by the All Flash Storage Module (AFSM) leveraging a fast local work filesystem (/afsm) on the compute nodes.
All modules are connected via a 100 Gbp/s EDR IB network in a non-blocking tree topology accompanied by a Gigabit Ethernet service network. In addition the system is connected to the Jülich storage system (JUST) to share home and project file systems with other HPC systems hosted at Jülich Supercompting Centre (JSC).
Cluster Module
It is composed of 50 nodes with the following hardware specifications:
|
|
Extreme Scale Booster
It is composed of 75 nodes with the following hardware specifications:
|
|
Data Analytics Module
It is composed of 16 nodes with the following hardware specifications:
|
|
Scalable Storage Service Module
It is based on spinning disks and composed of 4 volume data server systems, 2 metadata servers and 2 RAID enclosures. The RAID enclosures each host 24 spinning disks with a capacity of 8 TB each. Both RAIDs expose two 16 Gb/s fibre channel connections, each connecting to one of the four file servers. There are 2 volumes per RAID setup. The volumes are driven with a RAID-6 configuration. The BeeGFS global parallel file system is used to make 292 TB of data storage capacity available.
Here are the specifications of the main hardware components more in detail:
|
|
All Flash Storage Module
It is based on PCIe3 NVMe SSD storage devices. It is composed of 6 volume data server systems and 2 metadata servers interconnected with a 100 Gbps EDR-InfiniBand fabric. The BeeGFS global parallel file system is used to make 1.3 PB of data storage capacity available.
Here are the specifications of the main hardware components more in detail:
|
|
Interconnect
As shown in the system overview an EDR IB non-blocking fat tree is used as fast interconnect inside and between all modules along with a Gigabit Ethernet service network (used for administration). The IB fat tree is composed of 6 spine and 10 leaf switches:
Rack plan
This is a sketch of the available hardware reflecting the current rack layout.
SSSM rack
This rack hosts the master nodes (frontends), SSSM file servers and the storage as well as network components for the Gigabit Ethernet administration and service networks. Users can access the login node via deep@fz-juelich.de (implemented as virtual machine running on the master nodes). The rack is air-cooled.
CM rack
Contains the hardware of the DEEP-EST Cluster Module including compute nodes, a management node for this module, network components and a liquid cooling unit.
DAM rack
This rack hosts the compute nodes of the Data Analytics Module of the DEEP-EST prototype, a management node for this module, network components and 4x BXI test nodes plus switch. The rack is air-cooled.
SDV rack
Along with the actual prototype system several test nodes and so called software development vehicles (SDVs) have been installed in the scope of the DEEP(-ER,EST) projects. These are located in the SDV rack (07). KNL and ml-GPU nodes can be accessed by the users via SLURM. Access to the remaining SDV nodes can be given on demand:
- Prototype DAM [4 nodes]:
protodam[01-04]- 2 x Intel Xeon 'Skylake' (26 cores per socket)
- 192 GB RAM
- network: 1 Gigabit Ethernet
- Old DEEP-ER Cluster Module SDV [10 nodes]:
deeper-sdv[01-10]- 2 Intel Xeon 'Haswell' E5-v2680 v3 (2.5 GHz)
- 128 GB RAM
- 1 NVMe with 400 GB per node( accessible through BeeGFS on demand)
- network: 1 Gigabit Ethernet
- deeper-sdv[09,10]: 1 x Arria 10 FPGA PAC
- KNLs [4 nodes]:
knl[01,04-06]- 1 Intel Xeon Phi (64-68 cores)
- 1 NVMe with 400 GB per node (accessible through BeeGFS on demand)
- 16 GB MCDRAM plus 96 GB RAM per KNL
- network: 1 Gigabit Ethernet
- GPU nodes for Machine Learning [3 nodes]:
ml-gpu[01-03]- 2 x Intel Xeon 'Skylake' Silver 4112 (2.6 GHz)
- 192 GB RAM
- 4 x Nvidia Tesla V100 GPU (PCIe Gen3), 16 GB HBM2
- network: 40GbE connection inbetween, 1 GbE external
Further information
Attachments (9)
- CM_node_hardware.png (151.4 KB) - added by 5 years ago.
- DAM_node_hardware.png (226.5 KB) - added by 5 years ago.
- ESB_node_hardware.png (130.5 KB) - added by 5 years ago.
- DEEP-EST_Modules.png (377.0 KB) - added by 3 years ago.
- SSSM.png (88.3 KB) - added by 3 years ago.
- AFSM.png (341.7 KB) - added by 3 years ago.
-
SystemOverview.png (20.9 KB) - added by 3 years ago.
MSA Overview
- IB_non-blocking_fat_tree.png (411.2 KB) - added by 3 years ago.
- DEEP-EST_Prototype_Rackplan.png (214.0 KB) - added by 11 months ago.
Download all attachments as: .zip
