Open MPI Deployment Guide: Compiling, SSH, Slurm, and Performance Validation

EdwardMoon

Deploying Open MPI is more than copying the same executable to every node. Align compiler and Open MPI versions, runtime libraries, network transports, and scheduler integration, then validate progressively from a small program to communication performance.

This operational guide covers distribution packages versus official tarballs, wrapper compilers, unmanaged SSH launches, mpirun within Slurm allocations, PMIx/PRRTE, CPU binding, and UCX validation.

Open MPI workflow: wrapper compiler, Slurm allocation, mpirun, PMIx/PRRTE, and MPI ranks on compute nodes — Build with a wrapper compiler, then run inside a Slurm allocation with mpirun and PMIx/PRRTE

Choose the Execution Model

Environment	Recommended launch method	Prerequisites
Single-host development	mpirun -n	Local libraries and slots
Small unmanaged cluster	mpirun + hostfile	SSH, identical paths, and host-key verification
Production Slurm cluster	mpirun within an allocation	Compatible Slurm/PMIx builds and resource policies

Do not bypass a production Slurm scheduler by using arbitrary hostfiles and SSH to reach compute nodes. Running outside an allocation breaks CPU, memory, and GPU isolation and job accounting.

Distribution Packages or a Source Build

Distribution packages simplify updates and dependencies, but their version or UCX/PMIx features may not match requirements. Inspect package information first. Build an official Open MPI release tarball into a separate prefix only when needed. GitHub's automatically generated source archives are not substitutes for official release tarballs.

dnf info openmpi openmpi-devel
rpm -q openmpi openmpi-devel
module avail 2>&1 | grep -i mpi

Verify the Official Tarball and Build to a Separate Prefix

sha256sum -c openmpi-X.Y.Z.tar.bz2.sha256
tar -xjf openmpi-X.Y.Z.tar.bz2
cd openmpi-X.Y.Z

./configure   --prefix=/opt/openmpi/X.Y.Z   --with-slurm   --with-pmix   --with-ucx
make -j"$(nproc)"
make check
sudo make install

Check the configure summary to ensure requested features were actually found. Relying on automatic discovery when UCX or PMIx paths differ can link nodes against different libraries. Packaging and deploying the same RPM improves reproducibility in production.

/opt/openmpi/X.Y.Z/bin/ompi_info --version
/opt/openmpi/X.Y.Z/bin/ompi_info --all   | grep -Ei 'slurm|pmix|prrte|ucx'
ldd /opt/openmpi/X.Y.Z/bin/mpirun

Environment Modules and Node Consistency

Matching PATH alone is insufficient. The headers and libraries used by wrapper compilers, runtime libmpi, and plugin paths must be consistent across nodes.

export PATH=/opt/openmpi/X.Y.Z/bin:$PATH
export LD_LIBRARY_PATH=/opt/openmpi/X.Y.Z/lib:$LD_LIBRARY_PATH

which mpicc
which mpirun
mpicc --showme:command
mpicc --showme:compile
mpicc --showme:link
ompi_info --version

for host in node01 node02 node03 node04; do
  ssh "$host" '/opt/openmpi/X.Y.Z/bin/ompi_info --version'
done

Environment modules improve version switching and reproducibility. Point PATH and library paths to a single version prefix, and explicitly load that version in job scripts.

Build a Minimal MPI Program with the Wrapper Compiler

mpicc and mpicxx are wrappers that add MPI options to the selected C/C++ compiler, not standalone compilers. Use them rather than assembling header and library paths manually for gcc.

#include <mpi.h>
#include <stdio.h>

int main(int argc, char **argv) {
    int rank, size, name_len;
    char name[MPI_MAX_PROCESSOR_NAME];

    MPI_Init(&argc, &argv);
    MPI_Comm_rank(MPI_COMM_WORLD, &rank);
    MPI_Comm_size(MPI_COMM_WORLD, &size);
    MPI_Get_processor_name(name, &name_len);
    printf("rank=%d size=%d host=%s\n", rank, size, name);
    MPI_Finalize();
    return 0;
}

mpicc -O2 -Wall -Wextra hello_mpi.c -o hello_mpi
ldd ./hello_mpi | grep -E 'libmpi|open-rte|pmix'
mpirun -n 4 ./hello_mpi

Do not run Open MPI as root. Even inside containers, use a dedicated unprivileged user and scheduler resource limits. Do not make root-execution override flags part of the production standard.

Launch over SSH on an Unmanaged Cluster

Use a hostfile only for an experimental cluster without a scheduler. Executables and dynamic libraries must exist at identical absolute paths on every node, with SSH key authentication and host-key verification already configured.

node01 slots=8
node02 slots=8

mpirun --hostfile hosts.txt   --map-by ppr:4:node   --bind-to core   ./hello_mpi

Do not unintentionally launch more ranks than available slots. Oversubscription can be useful for deliberate development tests, but it distorts production performance results and creates memory pressure.

Use mpirun Within a Slurm Allocation

Open MPI 5 recommends mpirun in Slurm environments as well. Within an allocation, mpirun reads Slurm's node and task information, so hostfile, --host, and -n do not need to be repeated.

#!/usr/bin/env bash
#SBATCH --job-name=mpi-smoke
#SBATCH --partition=compute
#SBATCH --nodes=2
#SBATCH --ntasks-per-node=4
#SBATCH --cpus-per-task=1
#SBATCH --time=00:05:00
#SBATCH --output=mpi-%j.out

set -euo pipefail
module purge
module load openmpi/X.Y.Z

echo "nodes=${SLURM_JOB_NODELIST} tasks=${SLURM_NTASKS}"
mpirun --bind-to core --map-by slot ./hello_mpi

JOB_ID=$(sbatch --parsable mpi-smoke.sbatch)
squeue --job "$JOB_ID"
sacct --jobs "$JOB_ID"   --format=JobID,State,Elapsed,AllocCPUS,ExitCode
cat "mpi-${JOB_ID}.out"

Use Direct srun Launch Only After Checking PMIx Support

Direct srun launch is possible, but Slurm and Open MPI must be built with compatible PMIx support. Inspect the available plugins first and document whether the site's standard launcher is mpirun or srun.

srun --mpi=list
ompi_info --all | grep -Ei 'pmix|slurm'
srun --mpi=pmix ./hello_mpi

Validate TCP, UCX, and CPU Binding

TCP may be selected over Ethernet, and UCX over InfiniBand or RoCE. Before forcing a transport, inspect Open MPI's detected components, NIC state, MTU, and RDMA device permissions.

ompi_info --param pml all
ompi_info --param btl all
ompi_info --param osc all
ucx_info -d
ibv_devinfo
ip -br link

mpirun --report-bindings   --display-map   --bind-to core   --map-by slot   ./hello_mpi

On nodes with hyperthreading, NUMA, or GPUs, benchmark map-by and bind-to policies against the actual application. Do not exceed physical core counts based on the assumption that more ranks always run faster.

Test Functionality and Performance Separately

Verify initialization and shutdown with two to four ranks on one node.
Run a two-node hello program to check name resolution and process launch.
Record latency and bandwidth with a small ping-pong benchmark.
Measure CPU, memory, networking, and scalability with real application inputs.
Check for orphaned processes after node/NIC failures and job cancellation.

mpirun -n 2 osu_latency
mpirun -n 2 osu_bw
sstat --jobs "${SLURM_JOB_ID}.batch"   --format=JobID,AveCPU,MaxRSS,AveRSS

Troubleshooting

mpirun --version
mpicc --showme
ompi_info --version
ldd ./hello_mpi
env | grep -E '^(PATH|LD_LIBRARY_PATH|OPAL|OMPI|PMIX|PRTE|SLURM)'
scontrol show job "$SLURM_JOB_ID"
hostnamectl --static
getent hosts node01

If libmpi versions differ, correct modules and LD_LIBRARY_PATH, then redeploy the same build.
For remote launch failures, inspect host keys, non-interactive SSH, identical paths, and firewall rules.
If a Slurm job targets nodes outside its allocation, remove hostfile and --host options.
For UCX errors, check devices, drivers, MTU, memory locking, and container device permissions.
For poor performance, measure CPU binding, NUMA placement, oversubscription, and application communication patterns together.

Official Documentation and Related Articles

A complete Open MPI deployment reproduces the build, scheduler resource compliance, binding, transport, job accounting, and real performance, rather than just running hello once. Record versions, modules, compiler options, and benchmark results so behavior can be compared before and after upgrades.