Skip to content

MPI with Apptainer and Slurm

This example demonstrates how to run an MPI application across multiple compute nodes using Apptainer (w Intel oneAPI MPI,-image) and Slurm.

The MPI implementation is provided entirely by the container. Slurm is responsible for launching and managing the MPI processes across the allocated nodes.

1. Load the Apptainer module

Load the Apptainer module:

module load apptainer/1.3.3

2. Create a minimal MPI application

Create a file named mpi-hello.c:

#include <stdio.h>
#include <unistd.h>
#include <mpi.h>

int main(int argc, char **argv)
{
    int rank, size, len;
    char hostname[MPI_MAX_PROCESSOR_NAME];

    MPI_Init(&argc, &argv);

    MPI_Comm_rank(MPI_COMM_WORLD, &rank);
    MPI_Comm_size(MPI_COMM_WORLD, &size);

    MPI_Get_processor_name(hostname, &len);

    printf("Hello from rank %d/%d on node %s\n",
           rank, size, hostname);

    MPI_Finalize();
    return 0;
}

The application initializes MPI, determines the rank and size of MPI_COMM_WORLD, and prints the hostname of the node executing each MPI process.

3. Configure the Apptainer cache

Set the Apptainer cache directory to a location in your working directory:

export APPTAINER_CACHEDIR="$WORKDIR/.apptainer/cache"
mkdir -p "${APPTAINER_CACHEDIR}"

This keeps downloaded container layers and temporary cache data out of the default home-directory cache.

4. Pull the oneAPI container image

Download the Intel oneAPI development image from the registry of your choice and convert it into an Apptainer SIF image:

apptainer pull oneapi.sif \
    docker://containers.rz.uni-bayreuth.de/public-images/oneapi-toolkit:2026.0.0-devel-ubuntu24.04

The resulting oneapi.sif image contains the Intel oneAPI environment, including the MPI compiler and runtime.

5. Compile the MPI application

Compile the application using the MPI compiler provided by the container:

apptainer --quiet exec oneapi.sif mpicc mpi-hello.c -o mpi-hello.x

The resulting executable is created in the current working directory.

6. Run the MPI application across multiple nodes

Use Slurm to allocate eight compute nodes and launch one MPI process per node:

srun \
    --nodes=2 \
    --ntasks-per-node=2 \
    --mpi=pmi2 \
    --cpu-bind=cores \
    apptainer --quiet exec \
    --env FI_VERBS_DEVICE_NAME=mlx5_0 \
    oneapi.sif \
    ./mpi-hello.x

Important:

  • --nodes=2: Allocate eight compute nodes.
  • --ntasks-per-node=2: Start exactly two MPI processes per node.
  • --mpi=pmi2: Use Slurm's PMI-2 interface for MPI process initialization (INTEL MPI), with openmpi --mpi=pmix_v[2-6] may also work.
  • --cpu-bind=cores: Bind each process to a CPU core.
  • --env FI_VERBS_DEVICE_NAME=mlx5_0: Make use of the host's InfiniBand devices.

Slurm launches the MPI processes directly. Therefore, do not use mpirun or mpiexec inside the container. This avoids nested process launchers and allows Slurm to manage the MPI job.

7. Expected output

The application should produce output similar to:

Hello from rank 0/8 on node s71a01
Hello from rank 1/8 on node s71a02
Hello from rank 2/8 on node s71a03
Hello from rank 3/8 on node s71a04
Hello from rank 4/8 on node s71a05
Hello from rank 5/8 on node s71a06
Hello from rank 6/8 on node s71a07
Hello from rank 7/8 on node s71a08

The actual node names depend on the Slurm allocation.

Summary

This setup demonstrates the following architecture:

  • Apptainer: Provides the isolated user-space environment.
  • Intel oneAPI MPI: Provides the MPI implementation and runtime inside the container.
  • Slurm: Allocates compute resources and launches MPI processes across nodes.
  • InfiniBand: Host devices are exposed to the container for RDMA-capable communication.

No host MPI outside the container is required. The container provides its own MPI implementation, while the host provides the Slurm infrastructure and InfiniBand kernel drivers.