80 | | |
81 | | == File Systems == |
82 | | On the DEEP-EST system, three different groups of filesystems are available: |
83 | | |
84 | | * the [http://www.fz-juelich.de/ias/jsc/EN/Expertise/Datamanagement/OnlineStorage/JUST/Filesystems/JUST_filesystems_node.html JSC GPFS filesystems], provided via [http://www.fz-juelich.de/ias/jsc/EN/Expertise/Datamanagement/OnlineStorage/JUST/JUST_node.html JUST] and mounted on all JSC systems; |
85 | | |
86 | | * the DEEP-EST (and SDV) parallel BeeGFS filesystems, available on all the nodes of the DEEP-EST system; |
87 | | |
88 | | * the filesystems local to each node. |
89 | | |
90 | | The users home folders are placed on the shared GPFS filesystems. With the advent of the new user model at JSC ([http://www.fz-juelich.de/ias/jsc/EN/Expertise/Supercomputers/NewUsageModel/NewUsageModel_node.html JUMO]), the shared filesystems are structured as follows: |
91 | | |
92 | | * $HOME: each JSC user has a folder under `/p/home/jusers/`, in which different home folders are available, one per system he/she has access to. These home folders have a low space quota and are reserved for configuration files, ssh keys, etc. |
93 | | |
94 | | * $PROJECT: In JUMO, data and computational resources are assigned to projects: users can request access to a project and use the resources associated to it. As a consequence, each user has a folder within each of the projects he/she is part of. For the DEEP project, such folder is located under `/p/project/cdeep/`. Here is where the user should place data, and where the old files generated in the home folder before the JUMO transition can be found. |
95 | | |
96 | | The DEEP-EST system doesn't mount the $SCRATCH and $ARCHIVE filesystems, as it is expected to provide similar functionalities with its own parallel filesystems. |
97 | | |
98 | | The following table summarizes the characteristics of the file systems available in the DEEP-EST and DEEP-ER (SDV) systems: |
99 | | |
100 | | |
101 | | == Stripe Pattern Details == |
102 | | It is possible to query this information from the deep login node, for instance: |
103 | | |
104 | | {{{ |
105 | | manzano@deep $ fhgfs-ctl --getentryinfo /work/manzano |
106 | | Path: /manzano |
107 | | Mount: /work |
108 | | EntryID: 1D-53BA4FF8-3BD3 |
109 | | Metadata node: deep-fs02 [ID: 15315] |
110 | | Stripe pattern details: |
111 | | + Type: RAID0 |
112 | | + Chunksize: 512K |
113 | | + Number of storage targets: desired: 4 |
114 | | |
115 | | manzano@deep $ beegfs-ctl --getentryinfo /sdv-work/manzano |
116 | | Path: /manzano |
117 | | Mount: /sdv-work |
118 | | EntryID: 0-565C499C-1 |
119 | | Metadata node: deeper-fs01 [ID: 1] |
120 | | Stripe pattern details: |
121 | | + Type: RAID0 |
122 | | + Chunksize: 512K |
123 | | + Number of storage targets: desired: 4 |
124 | | }}} |
125 | | Or like this: |
126 | | |
127 | | {{{ |
128 | | manzano@deep $ stat -f /work/manzano |
129 | | File: "/work/manzano" |
130 | | ID: 0 Namelen: 255 Type: fhgfs |
131 | | Block size: 524288 Fundamental block size: 524288 |
132 | | Blocks: Total: 120178676 Free: 65045470 Available: 65045470 |
133 | | Inodes: Total: 0 Free: 0 |
134 | | |
135 | | manzano@deep $ stat -f /sdv-work/manzano |
136 | | File: "/sdv-work/manzano" |
137 | | ID: 0 Namelen: 255 Type: fhgfs |
138 | | Block size: 524288 Fundamental block size: 524288 |
139 | | Blocks: Total: 120154793 Free: 110378947 Available: 110378947 |
140 | | Inodes: Total: 0 Free: 0 |
141 | | }}} |
142 | | See http://www.beegfs.com/wiki/Striping for more information. |
143 | | |
144 | | == Additional infos == |
145 | | Detailed information on the '''BeeGFS Configuration''' can be found [https://trac.version.fz-juelich.de/deep-er/wiki/BeeGFS here]. |
146 | | |
147 | | Detailed information on the '''BeeOND Configuration''' can be found [https://trac.version.fz-juelich.de/deep-er/wiki/BeeOND here]. |
148 | | |
149 | | Detailed information on the '''Storage Configuration''' can be found [https://trac.version.fz-juelich.de/deep-er/wiki/local_storage here]. |
150 | | |
151 | | Detailed information on the '''Storage Performance''' can be found [https://trac.version.fz-juelich.de/deep-er/wiki/SDV_AdminGuide/3_Benchmarks here]. |
152 | | |
153 | | == Notes == |
154 | | * The /work file system which is available in the DEEP-EST prototype, is as well reachable from the nodes in the SDV (including KNLs and KNMs) but through a slower connection of 1 Gig. The file system is therefore not suitable for benchmarking or I/O task intensive jobs from those nodes |
155 | | |
156 | | * Performance tests (IOR and mdtest) reports are available in the BSCW under DEEP-ER -> Work Packages (WPs) -> WP4 -> T4.5 - Performance measurement and evaluation of I/O software -> Jülich DEEP Cluster -> Benchmarking reports: https://bscw.zam.kfa-juelich.de/bscw/bscw.cgi/1382059 |
157 | | |
158 | | * Test results and parameters used stored in JUBE: |
159 | | |
160 | | {{{ |
161 | | user@deep $ cd /usr/local/deep-er/sdv-benchmarks/synthetic/ior |
162 | | user@deep $ jube2 result benchmarks |
163 | | |
164 | | user@deep $ cd /usr/local/deep-er/sdv-benchmarks/synthetic/mdtest |
165 | | user@deep $ jube2 result benchmarks |
166 | | }}} |