Integrate applications in JUBE

Introduction

• JUBE Benchmarking Environment: provides a script based framework to easily create benchmark sets, run those sets on different computer systems and evaluate the results.
• Documentation: ​http://apps.fz-juelich.de/jsc/jube/jube2/docu/index.html
• Current version available in the DEEP and DEEP-ER systems: 2.1.0
• Example commands:

ssh user@deep

$ jube2 info benchmark_directory
$ jube2 status benchmark_directory [-i run_id]
$ jube2 run benchmark_xml_file [--tag tag1 tag2 ...]
$ jube2 analyse benchmark_directory [-i run_id]
$ jube2 result benchmark_directory [-i run_id]

Example application

ssh manzano@deep

cd /usr/local/deep-er/sdv-benchmarks/applications/MAXW-DGTD

JUBE compile input file

See for instance the JUBE xml file of the example application:

vim MAXW-DGTD-jube-master-SDV.compile.xml

Important parts in the file are:

1) Platform

  <include-path>
    <path>/usr/local/jube2/platform/deep</path>
  </include-path>

Under /usr/local/jube2/platform/deep there are a series of files with default values for the DEEP and DEEP-ER systems:

deep-chainJobs.sh
submit.job.in
platform.xml

These files won't be modified but the default values can be overwritten in the JUBE xml file.

2) Benchmark name and outpath

<benchmark name="MAXW-DGTD-compile" outpath="./compile">

The outpath describes the benchmark run directory (relative to the position of the input file). This directory will be managed by JUBE and will be automatically created if it doesn’t exist. The directory name and position are very important, because they are the main interface to communicate with your benchmark, after it was submitted.

3) Source files

    <fileset name="sources">
      <copy>MAXW-DGTD.tar.gz</copy>
      <prepare>tar -xzf MAXW-DGTD.tar.gz</prepare>
    </fileset>

The source files will be copied to the user sandbox directory and untar before the compilation.

4) Parameterset

    <parameterset init_with="MAXW-DGTD_specs.xml" name="systemParameter">
      <parameter name="modules">intel/15.2.164 parastation/intel-5.1.4-1_1_g064e3f7</parameter>
      <parameter name="targetdir">$jube_benchmark_home/executable/k$k</parameter>
      <parameter name="FC" type="string" >mpif90</parameter>
      <parameter name="FFLAGS" type="string" >-align dcommons -openmp -no-opt-prefetch -O3 -r8 -axCORE-AVX2 -fpp</parameter>
    </parameterset>

The values in the set of parameters with name "systemParamter" will be initialized with the default values found in the file "MAXW-DGTD_specs.xml". This last file takes in turn some of the default values of platform.xml and overwrites them. See the file for more information.

We can reference other parameters with $ like in the example $k or $jube_benchmark_home. $jube_benchmark_home is a JUBE variable with the value of the original input file location and $k is 2:

    <parameterset init_with="MAXW-DGTD_specs.xml" name="MAXW-DGTDParameter-head">
      <parameter name="k" type="int" >2</parameter>
    </parameterset>

So the value of the targetdir parameter will be /usr/local/deep-er/sdv-benchmarks/applications/MAXW-DGTD/executable/k2.

5) Step compile

    <step name="compile">
      <use>sources</use>
      <use from="MAXW-DGTD_specs.xml">MAXW-DGTDMakefileFile</use>
      <use from="platform.xml">compileset</use>
      <use>systemParameter</use>
      <use>MAXW-DGTDParameter-head</use>
      <use from="MAXW-DGTD_specs.xml">MAXW-DGTDMakefileSub</use>
      <do>module purge; module load $modules; export LD_LIBRARY_PATH=/opt/parastation/mpi2/lib:/usr/local/deep-er/sdv-benchmarks/libraries/scr/lib:/direct/Software/Extoll/SDV/lib:/usr/local/parastation/pscom/lib64:$LD_LIBRARY_PATH; cd MAXW-DGTD; rm -f Makefile; cp -p ../Makefile.out Makefile; make $k</do>
    </step>

In the step compile we tell which parameters and source files to use with the tag <use></use>. The <do></do> contains a single shell command. This command will run inside of a sandbox directory environment (inside the outpath directory tree).

6) Step copy_exec

    <step depend="compile" name="copy_exec">
      <use>systemParameter</use>
      <use>MAXW-DGTDParameter-head</use>
      <do>mkdir -p $targetdir</do>
      <do>cp -p compile/MAXW-DGTD/exe/gdL$k $targetdir/gdL$k-jubeID_$jube_benchmark_id</do>
      <do>rm -f $targetdir/gdL$k; ln -s $targetdir/gdL$k-jubeID_$jube_benchmark_id $targetdir/gdL$k</do>
    </step>

The copy_exec step depends on the compile step and will wait until it is finished for copying the executables resulting from the compilation to the $targetdir directory.

Compile application

jube2 run MAXW-DGTD-jube-master-SDV.compile.xml

A series of output files will be generated and stored under /usr/local/deep-er/sdv-benchmark/applications/MAXW-DGTD/compile/benchmark_id/. The structure is as follows:

compile                 # the given outpath
|
+- 000000               # the benchmark id
   |
   +- configuration.xml # the stored benchmark configuration
   +- workpackages.xml  # workpackage information
   +- run.log           # log information
   +- 000000_compile    # the workpackage
   |  |
   |  +- done           # workpackage finished marker
   |  +- work           # user sanbox folder
   |     |
   |     +- stderr      # standard error messages of used shell commands
   |     +- stdout      # standard output of used shell commands
   +- 000000_copy_exec  # the workpackage
      |
      +- done           # workpackage finished marker
      +- work           # user sanbox folder
         |
         +- stderr      # standard error messages of used shell commands
         +- stdout      # standard output of used shell commands

The executable resulting of the compilation will be stored under /usr/local/deep-er/sdv-benchmark/applications/MAXW-DGTD/executable/.

JUBE run job input file

See for instance the JUBE xml file of the example application:

vim MAXW-DGTD-jube-master-SDV.modules.test.xml

Important parts in the file are:

1) Use of tags

Notice the 2 different parameter sets:

    <parameterset name="MAXW-DGTDParameter-head" init_with="MAXW-DGTD_specs.xml" tag="!woman">
    [...]
    </parameterset>


    <parameterset name="MAXW-DGTDParameter-woman" init_with="MAXW-DGTD_specs.xml" tag="woman">
    [...]
    </parameterset>

You can tell JUBE to use one parameter set or another depending on the tag. For doing so you can specify it with the jube2 run command:

jube2 run MAXW-DGTD-jube-master-SDV.modules.test.xml --tag woman

will use MAXW-DGTDParameter-woman,

jube2 run MAXW-DGTD-jube-master-SDV.modules.test.xml

will use MAXW-DGTDParameter-head.

2) Several values for the parameters

<parameter name="k" type="int" >1,3</parameter>
[...]
<parameter name="iopath" type="string" >"/nvme/tmp/manzano/MAXW-DGTD-test/","/sdv-work/manzano/MAXW-DGTD-test/"</parameter>

You can specify comma separated values for the parameters. In this case JUBE will execute the benchmark 4 times with the following values:

k=1, iopath=/nvme/tmp/manzano/MAXW-DGTD-test/
k=1, iopath=/sdv-work/manzano/MAXW-DGTD-test/
k=3, iopath=/nvme/tmp/manzano/MAXW-DGTD-test/
k=3, iopath=/sdv-work/manzano/MAXW-DGTD-test/