DRBD and Pacemaker on Rocky Linux 9: STONITH and Two-Node HA

EdwardMoon

DRBD replicates block devices between two servers. Pacemaker and Corosync decide which node runs the Primary role, filesystem, virtual IP, and service. Promoting both nodes to Primary or mounting XFS on both at once can corrupt data. Follow an active/passive design with an enforced single writer.

Do not apply the previous CentOS 7.9 example directly to a new production environment: it relied on an end-of-life OS, older Master/Slave terminology, and omitted fencing. This guide explains design and validation using Rocky Linux 9, DRBD 9, and Pacemaker's current Promoted/Unpromoted model. Recheck package versions and supported combinations against distribution and LINBIT support documentation.

Architecture and Failure Boundaries

Prerequisites

• Pin compatible OS, DRBD kernel module and utilities, Pacemaker, Corosync, pcs, and resource agent versions on both nodes.
• Verify forward and reverse hostname resolution and NTP synchronization.
• Separate failure domains for replication, Corosync, service traffic, and BMC/fencing management wherever possible.
• Check backing-device sizes and sector information on both nodes, and confirm the partitions contain no filesystem or LVM signatures.
• Prepare a suitable fence agent, BMC credentials, and a qdevice or other third decision point.
• Document RPO/RTO and manual recovery procedures for corruption, split brain, and site failures.

Preflight Checks for Nodes and Storage

hostnamectl --static
timedatectl status
chronyc tracking
getent hosts ha1.example.com ha2.example.com

lsblk -o NAME,SIZE,TYPE,FSTYPE,MOUNTPOINTS,MODEL,SERIAL
sudo blkid /dev/vdb1
sudo wipefs --no-act /dev/vdb1

The example uses /dev/vdb1; identify the actual WWID, multipath, and LVM layers in your environment. Confirm device identity survives reboot, and stop if existing signatures are found. wipefs --no-act only inspects signatures; this guide does not automate their removal.

Check Packages and Resource Agents

sudo dnf repolist
sudo dnf --showduplicates list   drbd-utils kmod-drbd pacemaker corosync pcs resource-agents fence-agents-all

rpm -q drbd-utils pacemaker corosync pcs resource-agents
modinfo drbd | head
drbdadm --version
pcs --version

pcs resource standards
pcs resource providers ocf
pcs resource agents ocf:linbit

DRBD package names and kmod delivery vary by repository. Verify RHEL compatibility, Secure Boot signatures, kernel update policy, and the support provider, and use only validated repositories. Do not create resources before ocf:linbit:drbd is actually installed.

Configure the DRBD Replication Resource

DRBD Protocol C acknowledges completion after the remote disk confirms the write, making it suitable for synchronous replication. It does not automatically guarantee application fsync behavior or storage cache policy. Test both nodes' data paths and write durability under power loss separately.

Generate a DRBD Shared Secret

umask 077
openssl rand -base64 32

# Store the output in an approved secret store
# and deploy the same secret to the DRBD configuration on both nodes.

Example /etc/drbd.d/r0.res

resource r0 {
  protocol C;
  device /dev/drbd0;
  disk /dev/vdb1;
  meta-disk internal;

  net {
    cram-hmac-alg sha256;
    shared-secret "REPLACE_WITH_GENERATED_SECRET";
  }

  on ha1.example.com {
    node-id 0;
    address ipv4 10.10.10.11:7789;
  }

  on ha2.example.com {
    node-id 1;
    address ipv4 10.10.10.12:7789;
  }
}

Restrict the configuration file to mode 0600 so only root can read it, and prevent configuration management from logging the plaintext secret. Replace the REPLACE marker with a real secret before proceeding. Allow TCP 7789 only from the replication peer, and keep SELinux and firewalld enabled.

Create Metadata and Bring Up the Device

sudo chown root:root /etc/drbd.d/r0.res
sudo chmod 0600 /etc/drbd.d/r0.res
sudo drbdadm dump r0
sudo drbdadm create-md r0
sudo drbdadm up r0

sudo drbdadm status r0 --verbose
sudo cat /proc/drbd

Initial Synchronization and Filesystem Creation

# For new empty volumes only: run on ha1 alone during an authorized change window
sudo drbdadm primary --force r0
watch -n 2 sudo drbdadm status r0

# Create a new XFS filesystem only after both peers report UpToDate
sudo mkfs.xfs -L ha_data /dev/drbd0
sudo mkdir -p /srv/ha-data
sudo mount /dev/drbd0 /srv/ha-data
sudo touch /srv/ha-data/cluster-marker
sudo umount /srv/ha-data
sudo drbdadm secondary r0

mkfs.xfs destroys existing content. Do not run it on a reused volume or when either side contains data to retain. Choosing the wrong initial synchronization source and target can overwrite valid data with empty blocks. Confirm both devices are new and empty, then wait for UpToDate/UpToDate.

Create the Cluster and Configure STONITH

Authenticate pcs and Create the Corosync Cluster

# Set the hacluster password interactively on both nodes
sudo passwd hacluster

# Run on one node and enter the password at the prompt
sudo pcs host auth ha1.example.com ha2.example.com -u hacluster
sudo pcs cluster setup ha-drbd ha1.example.com ha2.example.com --start
sudo pcs cluster enable --all

sudo pcs cluster status
sudo pcs quorum status
sudo corosync-cfgtool -s

During a communication failure, two nodes cannot independently determine which peer is still alive. Consider a Corosync qdevice as a third vote, but remember that it does not replace STONITH. If using DRBD's internal quorum and a diskless tiebreaker, choose a consistent supported design rather than indiscriminately layering Pacemaker fencing and DRBD resource-level fencing.

Find the Fence Agent and Test It

sudo pcs stonith list
sudo pcs stonith describe fence_ipmilan
sudo pcs stonith config
sudo pcs property config --all

# Create and verify fencing with the actual BMC, PDU, or cloud agent and parameters
sudo pcs stonith config --full
sudo pcs stonith status

fence_ipmilan is only an example. Use the fence agent, pcmk_host_map, independent management address, and least-privilege account appropriate to the hardware or platform. Keep passwords out of command history and use the agent's supported secure secret-delivery method.

# Run only in a maintenance window after double-checking service impact and the target node
sudo pcs stonith fence ha2.example.com

sudo pcs status --full
sudo journalctl -u pacemaker -u corosync --since '-10 min'

Use maintenance mode to prevent automatic startup while creating the following resources and constraints in a new, empty cluster. Do not copy this lab command into a running production cluster: it changes the management state of all resources.

sudo pcs property set maintenance-mode=true
sudo pcs property config

Create the Promotable DRBD Resource

DRBD Agent and the Promoted Role

sudo pcs resource create drbd_r0 ocf:linbit:drbd   drbd_resource=r0   op monitor interval=15s role=Promoted   op monitor interval=30s role=Unpromoted

sudo pcs resource promotable drbd_r0   promoted-max=1 promoted-node-max=1   clone-max=2 clone-node-max=1   notify=true

sudo pcs resource config drbd_r0
sudo pcs status --full

Current Pacemaker uses Promoted/Unpromoted instead of Master/Slave. promoted-max=1 limits the cluster to one Primary at a time. Check the resource and clone names generated by your pcs version, and use those exact names in subsequent constraints.

Group the Filesystem, VIP, and Service

sudo pcs resource create fs_data ocf:heartbeat:Filesystem   device=/dev/drbd0 directory=/srv/ha-data fstype=xfs   op monitor interval=20s timeout=40s

sudo pcs resource create vip_app ocf:heartbeat:IPaddr2   ip=192.0.2.50 cidr_netmask=24   op monitor interval=20s

sudo pcs resource create app_service systemd:myapp   op monitor interval=20s timeout=40s

sudo pcs resource group add app_group fs_data vip_app app_service

Adjust systemd enablement on both nodes so myapp cannot start automatically outside the cluster. Adapt the IP address, interface, Filesystem agent options, and service timeouts to your environment. XFS is a single-writer filesystem mounted on one node only; do not turn this into a dual-primary example.

Ordering and Colocation Constraints

# Use the actual promotable clone name shown by pcs status/config
sudo pcs constraint order   promote drbd_r0-clone then start app_group

sudo pcs constraint colocation add   app_group with promoted drbd_r0-clone INFINITY

sudo pcs constraint config --full
sudo pcs status --full

The order constraint starts the group after DRBD is Promoted; colocation places it on that Promoted node. Within the group, resources start as Filesystem, VIP, then service, and stop in reverse order. Check constraint names and role syntax against your installed pcs help and generated CIB.

Validate Cluster State and Data

Review every resource, ordering constraint, colocation constraint, and fencing setting before disabling maintenance mode on the new lab cluster and confirming management starts.

sudo pcs constraint config --full
sudo pcs stonith status
sudo pcs property set maintenance-mode=false
sudo pcs status --full

Check Healthy State

sudo pcs status --full
sudo crm_mon -1Arf
sudo pcs resource config
sudo pcs constraint config --full
sudo drbdadm status r0 --verbose

findmnt /srv/ha-data
ip -brief address | grep '192.0.2.50'
curl --fail --silent --show-error http://192.0.2.50/healthz

Test Failover

Start with a normal standby transition to validate resource constraints and the application before pulling cables. Then follow an approved test plan to test application process failure, node shutdown, Corosync link loss, replication network loss, and BMC failure separately.

sudo pcs node standby ha1.example.com
watch -n 2 sudo pcs status --full

curl --fail --silent --show-error http://192.0.2.50/healthz
ssh ha2.example.com 'findmnt /srv/ha-data; sudo drbdadm status r0'

sudo pcs node unstandby ha1.example.com
sudo pcs status --full

Once Pacemaker manages the resources, do not change their state manually with drbdadm primary, mount, or systemctl start. Manual intervention separates actual state from the CIB's expected state. Use standby or maintenance-mode procedures, and verify the management state of all resources afterward.

Respond to Split Brain

1. Stop the service and automatic recovery, and physically isolate nodes so they cannot both write data.
2. Preserve DRBD roles and disk states, Pacemaker/Corosync/fencing logs, and the last known-good backup time.
3. Work with the application owner to identify the authoritative data using transaction-level evidence.
4. Have two people review the chosen victim node before discarding its data through LINBIT's official split-brain procedure.
5. After resynchronization, check the filesystem, data consistency, service functionality, and backups.
6. Correct the root cause in replication, quorum, STONITH, or constraints, then repeat the same failure test.

Commands for Collecting Evidence

sudo drbdadm status r0 --verbose
sudo drbdsetup status --statistics
sudo pcs status --full
sudo crm_mon -1Arf
sudo corosync-cfgtool -s
sudo pcs quorum status

sudo journalctl -u drbd -u pacemaker -u corosync   --since '-30 min' --no-pager

Operations Checklist

• Replace the CentOS 7 example with a pinned, supported OS/DRBD/Pacemaker combination.
• Identify backing devices by serial or WWID and have destructive initialization double-checked.
• Separate the failure domains of replication, Corosync, service traffic, and BMC fencing networks.
• Test STONITH in practice and keep stonith-enabled on.
• Set DRBD promoted-max=1 and mount XFS only on the single Promoted node.
• Validate ordering and colocation for the Filesystem/VIP/service group.
• Record expected results and RTO for standby, node shutdown, link loss, and fencing failure.
• Create versioned backups independent of DRBD and restore them in an isolated environment.

Official Documentation and Related Articles

• LINBIT DRBD 9 User's Guide
• Official ocf:linbit:drbd resource agent documentation
• RHEL 9: Getting started with Pacemaker and fencing
• RHEL 9 cluster quorum documentation
• ClusterLabs Pacemaker Explained
• Choosing Between GlusterFS and DRBD
• GlusterFS Replica 3 Deployment and Healing

Conclusion

Two-node HA is about ensuring the previous owner is truly isolated before one node writes the correct data, not merely restarting a service quickly. Validate Protocol C, one Promoted role, working STONITH, quorum decisions, and Filesystem/VIP/service constraints together. Replication is not backup; failure tests and independent restore tests are part of making the cluster ready for operation.