Scheduling and priorisation

The following chaptes shows the flow and priority calculation of Slurm scheduling with the multifactor/fairtree priority plugin.

Slurm Scheduling Workflow

Each job goes through three to four stages: submission → priority calculation → main scheduler (by priority) → usually also the backfill scheduler as a "gap filler".

The overall priority of a job in the Multifactor plugin (which forms the basis for the main scheduler) is calculated as a weighted sum:

Priority = PriorityWeightAge       × F_age
         + PriorityWeightFairshare × F_fairshare
         + PriorityWeightJobSize   × F_jobsize
         + PriorityWeightPartition × F_partition
         + PriorityWeightQOS       × F_qos
         + Σ (TRESBillingWeight_i  × F_tres_i)

All factors F lie between 0 and 1. F_fairshare has the largest weight at UBT – it is derived from the Fair Tree algorithm.

Fair Tree: Calculating Fairshare

Fair Tree compares sibling nodes (accounts/users with the same parent node) at each level of the tree. For each node i at a given level:

S_i  = shares_i / Σ shares_siblings        (normalized share)
U_i  = usage_i  / Σ usage_siblings        (normalized usage)
LF_i = S_i / U_i                          (Level-Fairshare)

A high LF value means that, relative to its assigned share, the node has consumed comparatively little so far → resulting in a higher priority. The ranks are propagated from the root down to the leaf; the final F_fairshare is derived from the global rank position (F = (N − rank) / N), normalized to [0,1].

Example calculation: physik has 40% of the cluster shares, but has consumed only 25% of the normalized usage → LF = 0.40 / 0.25 = 1.6. chemie has 60% of the shares and 75% of the usage → LF = 0.60 / 0.75 = 0.8. Therefore, physik ranks higher – Slurm effectively "owes" it additional compute time. Within physik, the same comparison is then performed between user_a and user_b, independently of the ranking at the account level. This is the key principle of Fair Tree: comparisons are always made only between siblings, never across the entire tree. This prevents a large account from simply "pushing out" a small account.

Backfill Scheduling

Backfill allows lower-priority jobs to run ahead of higher-priority jobs, provided that doing so does not delay a higher-priority job that is already waiting.

The condition for job_low is:

start_low + walltime_low ≤ t_reserved(job_high)
AND enough free nodes are available immediately

t_reserved is the time at which the blocked top-priority job A is allowed to start according to the backfill reservation. This is calculated based on the estimated remaining runtimes of currently running jobs.

Example (10-node cluster)

• Currently free: 6 nodes (Job X occupies 4). Job A (highest priority) requires 8 nodes → it cannot start immediately.
• Job X finishes at t = 2h → all 10 nodes will then be free → backfill reservation for Job A: t_reserved(A) = 2h.
• Job B (4 nodes, 1h walltime): 0h + 1h = 1h ≤ 2h ✓ and 4 ≤ 6 currently free nodes ✓ → Job B is started immediately via backfill without delaying Job A.
• Job C (4 nodes, 3h walltime): 0h + 3h = 3h > 2h ✗ → it would delay Job A → Job C remains in the queue, even though there are currently enough free nodes to run it.

Settings @ UBT

The following priority weights are configured for the Fair Tree algorithm:

Priority = 1000  × F_age
         + 20000 × F_fairshare
         + 1     × F_jobsize
         + 4     × F_partition
         + 8     × F_qos
         + 0     (PriorityWeightTRES = null → term is omitted completely)

At UBT, the share assigned to a cluster clearly dominates the priority calculation. The share for each chair/department (account) is based on its contribution to the system, expressed in euros. Accounts that do not contribute financially are assigned a fictitious euro contribution. Users belonging to an account, or using resources under that account, inherit the account's share (share=parent).

Other important settings on the KeyLabHPC systems

• PriorityDecayHalfLife = 3-00:00:00: Previously accumulated usage loses half of its weight every 3 days (exponential decay).
• PriorityUsageResetPeriod = WEEKLY: Usage is completely reset once per week.
• PriorityMaxAge = 10-00:00:00: F_age stops increasing after 10 days.
• PriorityFlags = SMALL_RELATIVE_TO_TIME: F_jobsize is calculated relative to walltime × CPU rather than the number of nodes.
• bf_window = 1440: The backfill scheduler only plans reservations for the next 24 hours.