[[TOC]] = Integrate applications in JUBE = == Introduction == * JUBE Benchmarking Environment: provides a script based framework to easily create benchmark sets, run those sets on different computer systems and evaluate the results. * Documentation: see [http://apps.fz-juelich.de/jsc/jube/jube2/docu/index.html JUBE Online Documentation] * Current version available in the DEEP, DEEP-ER and DEEP-EST systems: please check [https://apps.fz-juelich.de/jsc/llview/deep_modules/ here] * Example commands: {{{ ssh user@deep $ jube info benchmark_directory $ jube status benchmark_directory [-i run_id] $ jube run benchmark_xml_file [--tag tag1 tag2 ...] $ jube analyse benchmark_directory [-i run_id] $ jube result benchmark_directory [-i run_id] }}} == Example application == {{{ ssh manzano@deep cd /usr/local/deep-er/sdv-benchmarks/applications/MAXW-DGTD }}} === JUBE compile input file === See for instance the JUBE xml file of the example application: {{{ vim MAXW-DGTD-jube-master-SDV.compile.xml }}} Important parts in the file are: 1) Platform {{{ /usr/local/jube2/platform/deep }}} Under /usr/local/jube2/platform/deep there are a series of files with default values for the DEEP, DEEP-ER and DEEP-EST systems: {{{ deep-chainJobs.sh submit.job.in platform.xml }}} These files won't be modified but the default values can be overwritten in the JUBE xml file. 2) Benchmark name and outpath {{{ }}} The outpath describes the benchmark run directory (relative to the position of the input file). This directory will be managed by JUBE and will be automatically created if it doesn’t exist. The directory name and position are very important, because they are the main interface to communicate with your benchmark, after it was submitted. 3) Source files {{{ MAXW-DGTD.tar.gz tar -xzf MAXW-DGTD.tar.gz }}} The source files will be copied to the user sandbox directory and untar before the compilation. 4) Parameterset {{{ intel/15.2.164 parastation/intel-5.1.4-1_1_g064e3f7 $jube_benchmark_home/executable/k$k mpif90 -align dcommons -openmp -no-opt-prefetch -O3 -r8 -axCORE-AVX2 -fpp }}} The values in the set of parameters with name "systemParamter" will be initialized with the default values found in the file "MAXW-DGTD_specs.xml". This last file takes in turn some of the default values of platform.xml and overwrites them. See the file for more information. We can reference other parameters with $ like in the example $k or $jube_benchmark_home. $jube_benchmark_home is a JUBE variable with the value of the original input file location and $k is 2: {{{ 2 }}} So the value of the targetdir parameter will be /usr/local/deep-er/sdv-benchmarks/applications/MAXW-DGTD/executable/k2. 5) Step compile {{{ sources MAXW-DGTDMakefileFile compileset systemParameter MAXW-DGTDParameter-head MAXW-DGTDMakefileSub module purge; module load $modules; export LD_LIBRARY_PATH=/opt/parastation/mpi2/lib:/usr/local/deep-er/sdv-benchmarks/libraries/scr/lib:/direct/Software/Extoll/SDV/lib:/usr/local/parastation/pscom/lib64:$LD_LIBRARY_PATH; cd MAXW-DGTD; rm -f Makefile; cp -p ../Makefile.out Makefile; make $k }}} In the step compile we tell which parameters and source files to use with the tag . The contains a single shell command. This command will run inside of a sandbox directory environment (inside the outpath directory tree). 6) Step copy_exec {{{ systemParameter MAXW-DGTDParameter-head mkdir -p $targetdir cp -p compile/MAXW-DGTD/exe/gdL$k $targetdir/gdL$k-jubeID_$jube_benchmark_id rm -f $targetdir/gdL$k; ln -s $targetdir/gdL$k-jubeID_$jube_benchmark_id $targetdir/gdL$k }}} The copy_exec step depends on the compile step and will wait until it is finished for copying the executables resulting from the compilation to the $targetdir directory. === Compile application === {{{ jube run MAXW-DGTD-jube-master-SDV.compile.xml }}} A series of output files will be generated and stored under /usr/local/deep-er/sdv-benchmark/applications/MAXW-DGTD/compile/benchmark_id/. The structure is as follows: {{{ compile # the given outpath | +- 000000 # the benchmark id | +- configuration.xml # the stored benchmark configuration +- workpackages.xml # workpackage information +- run.log # log information +- 000000_compile # the workpackage | | | +- done # workpackage finished marker | +- work # user sanbox folder | | | +- stderr # standard error messages of used shell commands | +- stdout # standard output of used shell commands +- 000000_copy_exec # the workpackage | +- done # workpackage finished marker +- work # user sanbox folder | +- stderr # standard error messages of used shell commands +- stdout # standard output of used shell commands }}} The executable resulting of the compilation will be stored under /usr/local/deep-er/sdv-benchmark/applications/MAXW-DGTD/executable/. === JUBE run job input file === See for instance the JUBE xml file of the example application: {{{ vim MAXW-DGTD-jube-master-SDV.modules.test.xml }}} Important parts in the file are: 1) Use of tags Notice the 2 different parameter sets: {{{ [...] [...] }}} You can tell JUBE to use one parameter set or another depending on the tag. For doing so you can specify it with the jube2 run command: {{{ jube run MAXW-DGTD-jube-master-SDV.modules.test.xml --tag woman }}} will use MAXW-DGTDParameter-woman, {{{ jube run MAXW-DGTD-jube-master-SDV.modules.test.xml }}} will use MAXW-DGTDParameter-head. 2) Several values for the parameters {{{ 1,3 [...] "/nvme/tmp/manzano/MAXW-DGTD-test/","/sdv-work/manzano/MAXW-DGTD-test/" }}} You can specify comma separated values for the parameters. In this case JUBE will execute the benchmark 4 times with the following values: {{{ k=1, iopath=/nvme/tmp/manzano/MAXW-DGTD-test/ k=1, iopath=/sdv-work/manzano/MAXW-DGTD-test/ k=3, iopath=/nvme/tmp/manzano/MAXW-DGTD-test/ k=3, iopath=/sdv-work/manzano/MAXW-DGTD-test/ }}} 3) Substitute set See for instance the following section in platform.xml {{{ }}} And in MAXW-DGTD-jube-master-SDV.modules.test.xml: {{{ }}} This will tell JUBE to take the file ${submit_script}.in and create an output file named ${submit_script} with the occurrences #NOTIFY_EMAIL#, #QUEUE#, etc. substituted by the corresponding values. 4) Step execute - chainjob_script NOTE: For integrating an application the step prepare is not necessary and won't be covered in this guide. A slightly modificated version of the xml file is explained here. {{{ MAXW-DGTD-exec MAXW-DGTDInputFile executeset executesub chainsub systemParameter MAXW-DGTDParameter-head MAXW-DGTDParameter-woman MAXW-DGTDInputSub MAXW-DGTDSCRSub jobfiles chainfiles mkdir exe; chgrp deeper exe; chmod g+s exe; cp -pr exe_$testcase/* exe/ cp -p gdL$k exe/ rm -f exe/Data_Num*; cp -p Data_Num.out exe/Data_Num rm -f exe/Data_IO*; cp -p Data_IO.out exe/Data_IO rm -f exe/scr.sh*; cp -p scr.sh.out exe/scr.sh rm -f exe/test.part*; cp -p test.part.gz exe/; gzip -d exe/test.part.gz $chainjob_script $shared_job_info $submit_script chgrp deeper job.log job.err }}} The interesting parts of this section are at the end. With {{{ $chainjob_script $shared_job_info $submit_script }}} you can submit a series of jobs (in this example 4, see section 2). The first job will run and the next one will wait until the first one is finished. The ${chainjob_script} looks as follows: {{{ cat /usr/local/jube2/platform/deep/deep-chainJobs.sh #!/usr/bin/env bash if [ $# -lt 2 ] then echo "$0: ERROR (MISSING ARGUMENTS)" exit 1 fi LOCKFILE=$1 shift SUBMITSCRIPT=$* if [ -f $LOCKFILE ] then DEPEND_JOBID=`head -1 $LOCKFILE` JOBID=`qsub -W depend=afterany:${DEPEND_JOBID} $SUBMITSCRIPT` else JOBID=`qsub $SUBMITSCRIPT` fi echo ${JOBID} > $LOCKFILE exit 0 }}} In ${shared_jobinfo} the jobid of the preceding job is stored. Notice that {{{ ${shared_jobinfo} = ${shared_folder}/jobid = ./shared/jobid }}} and all jobs have in their sandbox a soft link called "shared" pointing to ./benchmarks_modules_test/benchmark_id/execute_shared. 5) Analyse {{{ pattern exe/maxdg3DP$k.out job.log }}} After the job has run, you can analyse the results with: {{{ jube analyse benchmarks_modules_test --id jube_id }}} The analyse step will look for the corresponding patterns in the output files, in this case exe/maxdg3DP$k.out and job.log, and the data will be stored inside the benchmark directory. The patterns to look for are specified in the MAXW-DGTD_specs.xml file: {{{ \s*Flux of the Poynting vector :\s+$jube_pat_wrd\s* \s*AT TAU =\s+$jube_pat_wrd\s* \s*T =\s+$jube_pat_wrd\s* \s*USING\s+$jube_pat_wrd\s* \s*MAX CPU TIME :\s+$jube_pat_wrd\s* \s*REAL TIME :\s+$jube_pat_wrd\s* \s*TIME TO SOLUTION:\s+$jube_pat_wrd\s* \s*I/O TIME :\s+$jube_pat_wrd\s* \s*ELECTR TIME :\s+$jube_pat_wrd\s* \s*OMP TIME :\s+$jube_pat_wrd\s* \s*T FLUX ELECTR :\s+$jube_pat_wrd\s* \s*T FLUX MAGNET :\s+$jube_pat_wrd\s* \s*T UP ELECTR :\s+$jube_pat_wrd\s* \s*T UP MAGNET :\s+$jube_pat_wrd\s* \s*T FOURIER :\s+$jube_pat_wrd\s* \(start\)$jube_pat_bl($jube_pat_wrd$jube_pat_bl$jube_pat_wrd) ^\s*\(finished\)\s*(.*)\s*\( }}} If there are multiple matches inside a single file you can add a reduce option. Normally only the first match will be extracted. See https://apps.fz-juelich.de/jsc/jube/jube2/docu/glossar.html#term-pattern-tag for more information. 6) Result {{{ analyse modulesiopathiotimestarttimestoptime
}}} The last part is the result table creation. Here you have to use an existing analyser. The contains a pattern or a parameter name. sort is the optional sorting order (separated by ,). The style attribute can be csv or pretty to get different ASCII representations. To create the result table you have to write: {{{ jube result benchmarks_modules_test --id jube_id }}} The result table will be written to STDOUT and into a result.dat file inside ./benchmarks_modules_test/benchmark_id/result.