Actions

Defining and submitting your job

From ALICE Documentation

Defining and submitting your job

Usually, you will want to have your program running in batch mode, as opposed to interactively as you may be accustomed to. The point is that the program must be able to start and run without user intervention, i.e., without you having to enter any information or to press any buttons during program execution. All the necessary input or required options have to be specified on the command line or needs to be put in input or configuration files. As an example, we will run a Perl script, which you will find in the examples subdirectory on the HPC. When you received an account to the HPC a subdirectory with examples was automatically generated for you. Remember that you have copied the contents of the HPC examples directory to your home directory, so that you have your own personal copy (editable and over-writable) and that you can start using the examples. If you haven’t done so already, run these commands now:

 $ cd
 $ cp -r /apps/alice/tutorials/Intro-HPC/examples ∼/

First go to the directory with the first examples by entering the command:

 $ cd ∼/examples/Running-batch-jobs

Each time you want to execute a program on the HPC you’ll need 2 things: The executable The program to execute from the end-user, together with its peripheral input files, databases and/or command options. A batch job script, which will define the computer resource requirements of the program, the required additional software packages and which will start the actual executable. The HPC needs to know: 1. the type of compute nodes; 2. the number of CPUs; 3. the amount of memory; 4. the expected duration of the execution time (wall time: Time as measured by a clock on the wall); 5. the name of the files which will contain the output (i.e., stdout) and error (i.e., stderr) messages; 6. what executable to start, and its arguments. Later on, the HPC user shall have to define (or to adapt) his/her own job scripts. For now, all required job scripts for the exercises are provided for you in the examples subdirectories. List and check the contents with:

 $ ls -l
 total 512
 -rw-r--r-- 1 vsc40000 193 Sep 11 10:34 fibo.pbs
 -rw-r--r-- 1 vsc40000 609 Sep 11 10:25 fibo.pl

In this directory you find a Perl script (named “fibo.pl”) and a job script (named “fibo.pbs”). 1. The Perl script calculates the first 30 Fibonacci numbers. 2. The job script is actually a standard Unix/Linux shell script that contains a few extra comments at the beginning that specify directives to PBS. These comments all begin with

  1. PBS.

We will first execute the program locally (i.e., on your current login-node), so that you can see what the program does. On the command line, you would run this using:

 $ ./fibo.pl
 [0] -> 0
 [1] -> 1
 [2] -> 1
 [3] -> 2
 [4] -> 3
 [5] -> 5
 [6] -> 8
 [7] -> 13
 [8] -> 21
 [9] -> 34
 [10] -> 55
 [11] -> 89
 [12] -> 144
 [13] -> 233
 [14] -> 377
 [15] -> 610
 [16] -> 987
 [17] -> 1597
 [18] -> 2584
 [19] -> 4181
 [20] -> 6765
 [21] -> 10946
 [22] -> 17711
 [23] -> 28657
 [24] -> 46368
 [25] -> 75025
 [26] -> 121393
 [27] -> 196418
 [28] -> 317811
 [29] -> 514229

Remark: Recall that you have now executed the Perl script locally on one of the login-nodes of the HPC cluster. Of course, this is not our final intention; we want to run the script on any of the compute nodes. Also, it is not considered as good practice, if you “abuse” the login-nodes for testing your scripts and executables. It will be explained later on how you can reserve your own compute-node (by opening an interactive session) to test your software. But for the sake of acquiring a good understanding of what is happening, you are pardoned for this example since these jobs require very little computing power. The job script contains a description of the job by specifying the command that need to be executed on the compute node: — fibo.pbs —

 #!/bin/bash -l
 cd $PBS_O_S_O_WORKDIR
 ./fibo.pl

So, jobs are submitted as scripts (bash, Perl, Python, etc.), which specify the parameters related to the jobs such as expected runtime (walltime), e-mail notification, etc. These parameters can also be specified on the command line. This job script that can now be submitted to the cluster’s job system for execution, using the qsub (Queue SUBmit) command:

 $ qsub fibo.pbs
 123456

The qsub command returns a job identifier on the HPC cluster. The important part is the number (e.g., “123456 ”); this is a unique identifier for the job and can be used to monitor and manage your job. Your job is now waiting in the queue for a free workernode to start on. Go and drink some coffee . . . but not too long. If you get impatient you can start reading the next section for more information on how to monitor jobs in the queue. After your job was started and ended, check the contents of the directory:

 $ ls -l
 total 768
 -rw-r--r-- 1 vsc40000 vsc40000 44 Feb 28 13:33 fibo.pbs
 -rw------- 1 vsc40000 vsc40000 0 Feb 28 13:33 fibo.pbs.e123456
 -rw------- 1 vsc40000 vsc40000 1010 Feb 28 13:33 fibo.pbs.o123456
 -rwxrwxr-x 1 vsc40000 vsc40000 302 Feb 28 13:32 fibo.pl

Explore the contents of the 2 new files:

 $ more fibo.pbs.o123456
 $ more fibo.pbs.e123456

These files are used to store the standard output and error that would otherwise be shown in the terminal window. By default, they have the same name as that of the PBS script, i.e., “fibo.pbs” as base name, followed by the extension “.o” (output) and “.e” (error), respectively, and the job number (’123456 ’ for this example). The error file will be empty, at least if all went well. If not, it may contain valuable information to determine and remedy the problem that prevented a successful run. The standard output file will contain the results of your calculation (here, the output of the Perl script)

When will my job start?

ALICE uses a fair-share scheduling policy(see Policies). There is no guarantee on when a job will start, since it depends on a number of factors. One of these factors is the priority of the job, which is determined by

  • All our resources use a fair-share scheduling policy.
  • No guarantees on when job will start, so plan ahead!
  • Job priority is determined by:
    • historical use: the aim is to balance usage over users, so infrequent (in terms of total compute time used) users get a higher priority
    • aim is to balance usage over users
    • requested resources (# nodes, Jobs and walltime, )
      • larger resource request => lower priority
    • time waiting in queue
      • queued jobs get higher priority over time
    • user limits
      • avoid that a single user fills up an entire cluster
Partitions/queues
Partition MaxNodesPerUser MaxJobsinQueue (running and waiting) MaxTime
cpu-short no-limit 100 2 hours
cpu-medium 18 30 1 day
cpu-long 15 30 7 days
gpu-short no-limit 100 2 hours
gpu-medium 8 16 1 day
gpu-long 7 14 7 days
mem 15 30 7 days
test no-limit 4 1 hour
playground-cpu no-limit 1 infinite
playground-gpu no-limit 1 infinite
notebook-cpu no-limit 1 infinite
notebook-gpu no-limit 1 infinite

It might be beneficial to request less resources (e.g., not requesting all cores in a workernode), since the scheduler often finds a “gap” to fit the job into more easily.

Specifying the cluster on which to run

To use other clusters, you can swap the cluster module. This is a special module that change what modules are available for you, and what cluster your jobs will be queued in. By default you are working on victini. To switch to, e.g., skitty you need to redefine the environment so you get access to all modules installed on the skitty cluster, and to be able to submit jobs to the skitty scheduler so your jobs will start on skitty instead of the default victini cluster.

 $ module swap cluster/alice

Note: the skitty modules may not work directly on the login nodes, because the login nodes do not have the same architecture as the skitty cluster, they have the same architecture as the victini cluster however, so this is why by default software works on the login nodes. See

Running software that is incompatible with host for why this is and how to fix this.

To list the available cluster modules, you can use the module avail cluster/ command:

 $ module avail cluster/ 
 -----------------------------------------------------------------------------------
   /etc/modulefiles/alice 
 -----------------------------------------------------------------------------------
 cluster/golett (S) cluster/phanpy (S) cluster/skitty (S) cluster/swalot 
    (S) cluster/victini (S,L)
 Where: 
  S: Module is Sticky, requires --force to unload or purge 
  L: Module is loaded
 If you need software that is not listed, request it helpdesk@alice.leidenuniv.nl

As indicated in the output above, each cluster module is a so-called sticky module, i.e., it will not be unloaded when module purge (see Purging all modules) is used.

The output of the various commands interacting with jobs (qsub, stat, ...) all depend on which cluster module is loaded.