GlusterFS Deployment Guide: Replica 3, Mounting, Healing, and Operations

EdwardMoon

GlusterFS presents disks across multiple servers as a single filesystem. Installing packages and joining two nodes does not by itself provide high availability. Design replication counts, quorum, failure domains, backups, and package support lifecycles together to reduce data loss and split-brain risk.

This guide follows a Replica 3 lab that stores a full data copy on each of three servers. Commands and configuration are separated into code blocks, with verification steps and stop conditions throughout. Before applying these examples in production, verify device names, network ranges, and the package supplier's support scope.

Check Current Versions and Support Before Deployment

The latest community release found in the official download directory is 11.1. However, the official release policy describes a major-version maintenance period of roughly 12 months, so 11.1 cannot be recommended unconditionally for new production deployments in 2026. Check whether your distribution or commercial provider supplies security updates. Without a supported path, also evaluate alternatives such as Ceph, NFS HA, or cloud file services. The commands here are lab examples assuming a validated EL9 repository is already configured.

Example Topology

• Place the three servers in separate failure domains and verify time synchronization and forward/reverse name resolution.
• Separate the OS and brick data onto different disks or logical volumes.
• Replica 3 stores three full copies, requiring roughly three times the source data capacity.
• Replication propagates deletion, encryption, and application errors too, so independent backups are essential.

Check Name Resolution and Time on Every Node

getent hosts gluster01 gluster02 gluster03
chronyc tracking
timedatectl status

Example Host Mappings

10.20.30.11  gluster01
10.20.30.12  gluster02
10.20.30.13  gluster03

Step 1: Prepare Dedicated Brick Storage

The following steps can erase existing device data. First verify the actual device and mount state on every server, and secure backups and change approval. The example device is /dev/sdb; replace it for your environment.

lsblk -f
findmnt --real
sudo wipefs --no-act /dev/sdb

sudo mkfs.xfs -f -i size=512 /dev/sdb
sudo mkdir -p /bricks/brick1
sudo blkid /dev/sdb

Use the UUID reported by blkid to add an entry like the following to each server's /etc/fstab.

UUID=<각-서버의-실제-UUID> /bricks/brick1 xfs defaults,noatime 0 2

sudo mount -a
findmnt /bricks/brick1
df -hT /bricks/brick1
sudo mkdir -p /bricks/brick1/gv0

Step 2: Verify Packages and the glusterd Service

On EL9, first query whether the configured repository actually provides glusterfs-server. If the package is missing or its source/signature is unclear, stop rather than downloading arbitrary old RPMs.

sudo dnf repolist
sudo dnf repoquery --info glusterfs-server
sudo dnf install -y glusterfs-server glusterfs-fuse
rpm -q glusterfs-server glusterfs-fuse
glusterfs --version

sudo systemctl enable --now glusterd
sudo systemctl --no-pager --full status glusterd
sudo journalctl -u glusterd -b --no-pager | tail -n 80

Step 3: Restrict Firewall Access to the Storage Network

Since Gluster 10, brick ports are selected randomly within the configured base-port/max-port range. Allow management ports 24007–24008 and the actual brick port range only from the storage subnet. The range below is an example; align it with the brick count and glusterd configuration before applying it in a change window.

sudo grep -E 'base-port|max-port' /etc/glusterfs/glusterd.vol
sudo firewall-cmd --get-active-zones
sudo firewall-cmd --permanent --zone=internal \
  --add-rich-rule='rule family=ipv4 source address=10.20.30.0/24 port port=24007-24008 protocol=tcp accept'
sudo firewall-cmd --permanent --zone=internal \
  --add-rich-rule='rule family=ipv4 source address=10.20.30.0/24 port port=49152-49200 protocol=tcp accept'
sudo firewall-cmd --reload
sudo firewall-cmd --zone=internal --list-all

For a new lab server using these firewall rules, set option base-port 49152 and option max-port 49200 inside the existing volume management block of /etc/glusterfs/glusterd.vol on every node, then restart glusterd. Do not duplicate existing options. For production volumes, inspect the ports and counts of existing bricks and allow the current range first; do not shrink it without a maintenance procedure.

sudoedit /etc/glusterfs/glusterd.vol
sudo systemctl restart glusterd
sudo systemctl status glusterd --no-pager
sudo grep -E "base-port|max-port" /etc/glusterfs/glusterd.vol

Step 4: Create the Trusted Storage Pool

Probe the other two servers from gluster01, then probe gluster01 once from gluster02 so hostnames are recorded consistently. Do not create the volume until every peer is Connected.

# Run on gluster01
sudo gluster peer probe gluster02
sudo gluster peer probe gluster03
sudo gluster peer status
sudo gluster pool list

# Run on gluster02
sudo gluster peer probe gluster01
sudo gluster peer status

Step 5: Create and Validate a Replica 3 Volume

Replica 3 copies files to three bricks. A plain distributed volume increases capacity without replicas, so losing one brick can lose the files it contains. If the goal is high availability, carefully inspect the Type and brick layout in the volume create output.

sudo gluster volume create gv0 replica 3 transport tcp \
  gluster01:/bricks/brick1/gv0 \
  gluster02:/bricks/brick1/gv0 \
  gluster03:/bricks/brick1/gv0

sudo gluster volume info gv0
sudo gluster volume start gv0
sudo gluster volume status gv0 detail

After starting the volume, verify that all three bricks are Online. The force option can bypass warnings about incorrect paths or topology; do not use it without understanding the cause.

6. Mount Clients and Verify Reboot Behavior

Install glusterfs-fuse on the client. Set the other two servers as backup-volfile-servers in case the first volfile server is unavailable. This option does not replace data replication.

sudo dnf install -y glusterfs-fuse
sudo mkdir -p /mnt/gv0
sudo mount -t glusterfs gluster01:/gv0 /mnt/gv0 \
  -o backup-volfile-servers=gluster02:gluster03
findmnt /mnt/gv0
df -hT /mnt/gv0

For automatic mounting after reboot, add the following entry to the client's /etc/fstab.

gluster01:/gv0 /mnt/gv0 glusterfs defaults,_netdev,backup-volfile-servers=gluster02:gluster03 0 0

sudo umount /mnt/gv0
sudo mount -a
findmnt /mnt/gv0
sudo touch /mnt/gv0/.mount-test
stat /mnt/gv0/.mount-test

7. Test Writes, Replication, and Recovery

Before storing production data, test basic writes and replication with separate test files. Direct application access to brick paths bypasses Gluster metadata, so clients must always use the Gluster mount point.

date -Is | sudo tee /mnt/gv0/healthcheck.txt
sha256sum /mnt/gv0/healthcheck.txt
sudo gluster volume status gv0
sudo gluster volume heal gv0 info summary

Run planned failure tests in a maintenance window after checking backups and recovery plans. Add test files while one node is isolated, then verify pending heal entries converge to zero after it returns. Do not force writes after losing the majority or write to multiple partitions simultaneously.

sudo gluster peer status
sudo gluster volume status gv0 detail
sudo gluster volume heal gv0 info summary
sudo gluster volume heal gv0 info
sudo gluster volume heal gv0 info split-brain

8. Handle Healing and Split Brain Safely

Pending heal entries can be repaired automatically. Split brain means Gluster cannot determine which copy is authoritative. Automatically choosing the newest or largest file can overwrite valid data. Stop application writes, make a backup, and compare file contents and checksums first.

sudo gluster volume heal gv0 info split-brain
sudo getfattr -d -m . -e hex /bricks/brick1/gv0/<문제-파일>
sudo stat /bricks/brick1/gv0/<문제-파일>
sudo sha256sum /bricks/brick1/gv0/<문제-파일>

Use source-brick recovery for the affected file only after an operator and the application owner have confirmed the authoritative brick. Replace the host, brick, and file paths below with verified values.

sudo gluster volume heal gv0 split-brain \
  source-brick gluster01:/bricks/brick1/gv0 /<문제-파일>

sudo gluster volume heal gv0 info split-brain
sudo gluster volume heal gv0 info summary

9. Diagnose Failures with Logs and State Together

Before restarting servers in response to a mount error, collect peer state, volume state, brick ports, healing, capacity, inode usage, and logs at the same point in time. A full /var/lib/glusterd can disrupt the management daemon in particular.

sudo gluster pool list
sudo gluster volume info gv0
sudo gluster volume status gv0 detail
sudo gluster volume heal gv0 info summary
df -hT /var/lib/glusterd /bricks/brick1
df -i /var/lib/glusterd /bricks/brick1
sudo ss -lntp | grep -E ':(2400[78]|4915[2-9]|491[6-9][0-9]|49200)\b'
sudo journalctl -u glusterd -b --no-pager | tail -n 200

If the heal command itself fails, inspect glfsheal and the relevant brick logs. Log paths and process ports vary by version and packaging; compare them with volume status output.

sudo ls -1 /var/log/glusterfs/
sudo tail -n 200 /var/log/glusterfs/glfsheal-gv0.log
sudo find /var/log/glusterfs/bricks -maxdepth 1 -type f -name '*.log' -print

10. Operations Checklist

1. Document the package supplier's security support period and upgrade path.
2. Place the three servers in independent power, rack, and network failure domains.
3. Allow management and brick ports only from required source networks, and keep SELinux enabled.
4. Continuously monitor peers, bricks, healing, split brain, capacity, inode usage, and latency.
5. Maintain independent backups against deletion and ransomware, and test restoration regularly.
6. Rehearse node replacement, package upgrades, and authoritative-copy selection before making changes.

Minimum Daily or Automated Monitoring Checks

sudo gluster peer status
sudo gluster volume status gv0
sudo gluster volume heal gv0 info summary
df -P /bricks/brick1
df -Pi /bricks/brick1

Related Articles

• GlusterFS vs. DRBD: Replication, Consistency, and HA Selection
• Linux No space left on device Troubleshooting

Conclusion

Safe GlusterFS deployment starts by defining failure behavior, not counting installation commands. Build a Replica 3 topology with majority decisions, then operate dedicated bricks, restricted firewall rules, consistent name resolution, state validation, and healing observation as one workflow. Independent backups, restore tests, and a supported package source complete an operational file service.