Changes between Version 6 and Version 7 of Public/User_Guide/DEEP-EST_DAM
- Timestamp:
- Oct 15, 2019, 1:16:32 PM (5 years ago)
Legend:
- Unmodified
- Added
- Removed
- Modified
-
Public/User_Guide/DEEP-EST_DAM
v6 v7 1 1 = System usage = 2 3 2 The DEEP-EST Data Analytics Module (DAM) can be used through the SLURM based batch system that is also used for (most of) the Software Development Vehicles (SDV). You can request DAM nodes (`dp-dam[01-16]`) with an interactive session like this: 4 3 … … 15 14 dp-dam04 16 15 }}} 17 18 19 16 When using a batch script, you have to adapt the partition option within your script: `--partition=dp-dam` 20 17 21 18 == Using Cuda == 22 23 19 To compile and run Cuda applications on the Nvidia V100 cards included in the DAM nodes, it is necessary to load the CUDA module: 24 20 … … 35 31 }}} 36 32 33 == Using FPGAs == 34 35 Each node is equipped with a Stratix 10 FPGA. For getting started using OpenCL with the FPGAs you can find some hints as well as the slides and exercises from the Intel FPGA workshop held at JSC 36 37 {{{ 38 /usr/local/fpga 39 }}} 40 41 It is recommended to do the first steps in an interactive session on a DAM node. To set up and check the FPGA environment, do the following: 42 43 {{{ 44 source /usr/local/fpga/FPGA_init.sh 45 lspci | grep -i 'accel' 46 aocl list-devices 47 aoc -list-boards 48 # -- optional for doing the exercises: 49 # export CL_CONTEXT_EMULATOR_DEVICE_INTELFPGA=1 50 }}} 51 52 53 You can copy and untar the lab into your home directory to do the exercises step by step. The exercises use the emulator device instead of the actual FPGA device due to the long 54 compilation time for the FPGAs. For using the FPGA device you will have to compile your OpenCL kernels using `-board=pac_s10_dc` option: 55 56 {{{ 57 # compile for the emulator 58 aoc -march=emulator -fast-emulator kernel-file.cl 59 60 # compile for the FPGA device 61 aoc -board=pac_s10_dc kernel-file.cl 62 }}} 63 64 In addition, you will have to adapt the OpenCL host file to select the correct platform ("Intel(R) FPGA SDK for OpenCL(TM)" or "Intel(R) FPGA Emulation Platform for OpenCL(TM) (preview)"). 65 **Attention:** Compiling kernels for the FPGA device (instead of the emulator) might take several hours. 66 67 Although eclipse is available on the DAM nodes, compiling and running the example applications might not work out, so you have to fall back to the command line as described in the exercise manual and by using 68 the provided `simple_compile.sh` scripts. 37 69 38 70 == Filesystems and local storage == 71 The home filesystem on the DEEP-EST Cluster Module is provided via GPFS/NFS and hence the same as on (most of) the remaining compute nodes. The local storage system of the DAM running BeeGFS is available at 39 72 40 The home filesystem on the DEEP-EST Cluster Module is provided via GPFS/NFS and hence the same as on (most of) the remaining compute nodes.41 The local storage system of the DAM running BeeGFS is available at42 73 {{{ 43 74 /work … … 45 76 The file servers are reachable through the 40 GbE interface of the DAM nodes. 46 77 47 This is NOT the same storage being used on the DEEP-ER SDV system. Both, the DEEP-EST prototype system and the DEEP-ER SDV have their own local storage. 78 This is NOT the same storage being used on the DEEP-ER SDV system. Both, the DEEP-EST prototype system and the DEEP-ER SDV have their own local storage. 48 79 49 80 It's possible to access the local storage of the DEEP-ER SDV (`/sdv-work`), but you have to keep in mind that the file servers of that storage can just be accessed through 1 GbE ! Hence, it should not be used for performance relevant applications since it is much slower than the DEEP-EST local storages mounted to `/work`. … … 51 82 There is node local storage available for the DEEP-EST DAM node (2 x 1.5 TB NVMe SSD), but configuration is to be done for those devices. 52 83 53 54 84 == Multi-node Jobs == 55 56 85 Currently, multi-node MPI jobs are possible on the DAM only by modifying the environment in the following way: 57 86 … … 62 91 Hello World from processor dp-dam01, rank 0 out of 2 63 92 }}} 64 65 93 **Attention:** This is a temporary workaround. 66 94 67 {{{#!comment 68 **Attention:** Since the Extoll network is not in place yet multi-node MPI Jobs are currently disabled. 95 {{{#!comment 96 **Attention:** Since the Extoll network is not in place yet multi-node MPI Jobs are currently disabled. 69 97 }}} 70 98 71 99 {{{#!comment 72 100 73 Loading the most recent ParaStation module will be enough to run multi-node MPI jobs over Extoll 101 Loading the most recent ParaStation module will be enough to run multi-node MPI jobs over Extoll 74 102 75 103 {{{ 76 104 module load ParaStationMPI 77 105 }}} 78 79 106 }}}
