The Operations Loop 03 — Validating Failover and Recovery

EdwardMoon

The criterion for verification is whether data and paths are actually connected, rather than whether a screen opens. This article records the tasks executed and observation results from the 2026-09-20 VirtualBox lab. It distinguishes between design expectations, actual tests, and the scope not yet performed.

• 1. Validation Environment and Acceptance Criteria
• 2. Actual Successful Onboarding Records
• 3. Issues Discovered and Fixed Along the Normal Path
• 4. Backup and Isolated Restoration
• 5. Central Node Failure Test
• 6. Proxy Disconnection and Delayed Data Retransmission
• 7. Asset Data Protection and Path Validation
• 8. Target Installation Without Internet and Package Validation
• 9. Architecture Screen Validation
• 10. Validation Checklist and Boundaries
• 11. Recommended Reading and Configuration Materials

1. Validation Environment and Acceptance Criteria

Rocky Linux 10.2, Zabbix 7.0.30 LTS, NetBox 4.7.1, and AWX 24.6.1 were used. Two central VMs, a quorum, a gateway, a provisioner, and proxies/bastions for three environments were configured. All VMs share the memory, CPU, and storage of a single physical PC.

Successful onboarding passes only when PXE installation → approved SSH path → actual asset collection → NetBox native IPAM → Zabbix registration → NetBox-based AWX inventory sync → verification of actual latest values all succeed. API registration response and latest data reception are separate check items.

2. Actual Successful Onboarding Records

The PROD target was installed on an empty 32GiB disk with one internal-network NIC. SSH through the bastion confirmed Rocky Linux 10.2, management IP 10.77.20.101/24, SELinux Enforcing and the PXE completion marker. Memory was reduced from 4GiB to 1GiB after installation.

The verification job confirmed receipt of actual current server uptime data. Passing required both successful registration and working monitoring data collection.

NetBox stored vCPU 1, guest available memory 954MiB, disk 32768MiB, Rocky Linux 10.2, virtualization type virtualbox, management NIC enp0s3, and the PROD environment and actual partition structure. The guest memory observed value of 954MiB and the VirtualBox allocated value of 1024MiB are different values.

AWX’s NetBox inventory received ansible_host=10.77.20.101, ansible_user=labadmin, fml_environment=prod, and fml_subnet=20. It did not just synchronize asset names, but also verified the management IP and environment variables required for calculating the Bastion path.

DEV was installed by PXE on an empty disk and completed Workflow 30: registration job 31, inventory update 32 and verification job 34. Monitoring host 10684 used DEV Proxy 2, confirming collection through a path separate from PROD.

STG automatically executed upon detecting installation completion

After approving STG’s first SSH identity with the VirtualBox serial console’s public key and fingerprint, the installation RAM was reduced from 4GiB to 1GiB. The installation completion controller verified the approval key, installation mark, hostname, and machine ID, and synchronized the SCM and bootstrap inventory. Afterward, Job 38 was started without manual Workflow execution.

STG also supplied actual current monitoring data. AWX listed all three environment hosts with their management IPs and environment variables. Initial SSH identity approval remained an operator check; unapproved keys were not bypassed.

3. Issues found up to the normal path and fixed

Failed jobs were not deleted either. They were kept so that fix commits can be compared with subsequent Job results. Do not confuse cases that failed prior to NetBox/Zabbix registration with a situation where actual assets were duplicated.

4. Backup and isolated restoration

The NetBox database was backed up with pg_dump and restored with pg_restore into a separate verification database. The restored fml-prod-app-01 asset and its primary management IP relationship were checked.

The backup SHA-256 is 06914eb419d88edb97245ed435177b396309a625b65eb14b72ae9fb627667390. A separate verification DB was used without overwriting the existing DB. This result is a NetBox logical DB restoration test, not a disaster recovery test that recovers all VMs, media, AWX, and Git at once.

5. Central node outage test

Just before the first test, the PostgreSQL leader was ops02, ops01 was a sync standby, and the replication lag was 0. For Zabbix, ops01 was active and ops02 was standby. Power to unit 1 was forcibly cut to simulate VM power loss.

VIP 10.77.10.10 moved to unit 2. However, during the transition phase, the NetBox backend returned 500 and the VIP returned 503. Zabbix subsequently started active tasks on unit 2 and received connections from all three proxies. This test was not judged as a “zero-downtime HA success”.

The monitoring tool’s first log file was not generated due to a path incompatible with the SELinux policy. Therefore, precise RTO is not calculated from that test. The logging path was moved under /var/log, normal recording was verified, and the test was re-run.

Initial failure tests exposed delayed NetBox recovery caused by cache connection retries. After checking the configuration code, local Sentinel lookup was prioritized and connection and retry limits were set. The failure test was then repeated.

Power loss of active unit 2 for DB, Redis, and Zabbix after modifications

Central node 2 was forcibly powered off while it held the active PostgreSQL, Redis and Zabbix roles. Role transfer and service responses on node 1 were verified. Transient response errors occurred during the transition, so this was not classified as interruption-free failover.

With node 2 off, a test marker was written to a NetBox asset, read back and restored to its original value. The asset and management IP were preserved, verifying writes through the surviving database. Recovery of the AWX status endpoint did not prove continuity of running jobs.

After node 2 restarted, PostgreSQL returned as a synchronous standby and NetBox became healthy. Service survival and restoration of the failed node as spare capacity were checked separately.

Power Loss on Active Node 1 with VIP, DB, and Zabbix After Modification

The reverse direction was tested by powering off node 1. Node 2 retained the shared service IP and database write role; NetBox asset queries and new Zabbix monitoring values resumed. Asset identity and management IP were preserved.

AWX was installed only on node 1 and was unavailable while that node was off. Restarting node 1 restored AWX, database replication and NetBox. This was recovery of the original server, not successful AWX HA.

Both tests include timeouts and HTTP 500/503 errors during the transition phase. We do not use the term ‘uninterrupted’. Performance improvement effects must not be concluded simply from ratios when comparing pre- and post-improvement metrics of similar tests, as the active DB location and load differed at the time.

6. Proxy Disconnection and Delayed Data Retransmission

Only monitoring transport from PROD Proxy 10.77.20.10 to the central network on TCP 10051 was blocked at the gateway. SSH, local agent collection and the DEV path remained available. Central PROD updates stopped while DEV collection continued.

A read-only check of the PROD Proxy database confirmed storage of data not yet delivered centrally. Total row counts were not treated as unsent counts because already transmitted records may remain before cleanup.

The temporary blocking rule remained despite its expiry setting. It was removed explicitly and connectivity was checked. Subsequent tests require checking actual rule removal instead of relying on the timer alone.

After connectivity returned, central history showed the buffered data in its original collection order and cadence, and current values resumed. This test covered a temporary transport interruption. Long-term retention, disk exhaustion and failure of the Proxy VM itself remain separate tests.

7. Asset Data Protection and Path Verification

Based on actual PROD collection results, the same NetBox registration was executed again. VM ID 1 and IPAddress ID 1 were retained, and only one instance of the same name and address existed. We distinguished between collection timestamp updates and duplicate asset generation.

Partial collection failure is an input created for testing and is not presented as an actual disk failure case. Rejection conditions are checked first before making changes, and situations where only some APIs succeed are handled so that successfully created objects are not unconditionally deleted, allowing retries.

8. Air-gapped Target Installation and Package Verification

The PXE target has only a single Internal Network NIC and lacks NAT or bridge NICs. The OS was received from Provision’s Rocky 10.2 Minimal installation tree. After clearing the Agent RPM cache in PROD and enabling only two internal repositories, zabbix-agent2 7.0.30 was reinstalled. We verified the actual 6.4MB download, GPG verification, and service active status.

NetBox HTTPS using internal repo metadata and CA verification returned 200 on the same target where external connections were denied. Reinstalling the entire central system in a new air-gapped network is out of scope. What was verified this time is OS installation, Agent deployment, and internal API integration on air-gapped targets.

9. Architecture Screen Verification

The architecture’s failure state and flow playback are for explanation purposes. It is not an operational dashboard that sends commands to servers or displays real-time status.

10. Verification Checklist and Boundaries

Actual physical server UEFI PXE, BMC/RAID automation, failures between two physical hosts, total storage failure, and version detection for all commercial DB/WAS products are not verified by this VM practice. Separate the implementation of each feature from the actual test scope.

Also distinguish between online initial installation and installation/Agent deployment for targets without internet access. Tests that reinstall the entire central system from scratch in a new isolated network must include validation of import bundle dependencies. Disconnecting external NICs from an existing service does not substitute for this.

Items passed in this practice and the next verification tasks are listed in the same table. Do not rephrase the results from a single personal PC as guaranteeing operational SLA or zero data loss across all failures.

Further Reading and Configuration Materials

Portfolio Original · Online and Isolated Network Setup Guide · Failure Scenarios and Verification Records

Practice Setup & Automation Source ZIP · ZIP SHA-256

The public ZIP contains configuration templates and automation sources. It does not include OS, RPMs, container images, or credentials. Configure with your own addresses, public CAs, authorized SSH keys, and secret storage before applying.