Job Arrays
Jobarrays are very useful for jobs where only parameters differ.
Example:
First assume we have a parameterlist, in “paramlist” text-file, like:
#Generated parameterlist to demonstrate job arrays.
#1st 2nd 3rd 4th
-s A 100 100
-s B 100 200
-s C 100 300
-s D 100 400
-s E 200 100
-s F 200 200
-s G 200 300
-s H 200 400
-s I 300 400
-s J 400 100
-s K 400 200
-s L 400 300
-s M 400 400
#EOF
And program (e.g. ./example.sh) that takes these arguments:
#!/bin/bash
steptype="$1"
stepsize="$2"
arraysize=$( echo $3 * $4 | bc -l)
echo $arraysize
If we want to start our program with several of these parameter sets by a loop inside a single job we could run into several problems, e.g.:
- exceeding maximum walltime and job gets killed
- all of the output get stored in one file by default
- no save state until job ended
To avoid these problems and iterate over these list with our example.sh we could use a job array. By example here for the first 10 parameters, this could be done by using an arrayjob with following script:
#!/bin/bash
#SBATCH --array=1-10 # Describe your array first
#SBATCH --job-name=job_array
#SBATCH --nodes=1
#SBATCH --ntasks=1
#SBATCH --output=ex_%A_%a.out # err/out file with <JOBID>_<JOBARRAY_INDEX>
#SBATCH --error=ex_%A_%a.err
#SBATCH --time=00:05:00
#Starting the example.sh script with <n>th parameter-set from list.
#Using "grep -v "^#" to ignore commentlines
line=${SLURM_ARRAY_TASK_ID}
./example.sh $( cat paramlist | grep -v "^#" | sed -ne ${line}p )
#EOF
this will start the “./example.sh” with
$ ./example.sh -s 8 100 100
...
$ ./example.sh -s 8 300 200