The Operations Loop Verification Report — From Registration to Failure Recovery

AI_Manager

The criterion for verification is whether data and paths are actually connected, rather than whether a screen opens. This article records the tasks executed and observation results from the 2026-09-20 VirtualBox lab. It distinguishes between design expectations, actual tests, and the scope not yet performed.

• 1. Validation Environment and Acceptance Criteria
• 2. Actual Successful Onboarding Records
• 3. Issues Discovered and Fixed Along the Normal Path
• 4. Backup and Isolated Restoration
• 5. Central Node Failure Test
• 6. Proxy Disconnection and Delayed Data Retransmission
• 7. Asset Data Protection and Path Validation
• 8. Target Installation Without Internet and Package Validation
• 9. Architecture Screen Validation
• 10. Validation Checklist and Boundaries
• 11. Recommended Reading and Configuration Materials

1. Validation Environment and Acceptance Criteria

Rocky Linux 10.2, Zabbix 7.0.30 LTS, NetBox 4.7.1, and AWX 24.6.1 were used. Two central VMs, a quorum, a gateway, a provisioner, and proxies/bastions for three environments were configured. All VMs share the memory, CPU, and storage of a single physical PC.

Successful onboarding passes only when PXE installation → approved SSH path → actual asset collection → NetBox native IPAM → Zabbix registration → NetBox-based AWX inventory sync → verification of actual latest values all succeed. API registration response and latest data reception are separate check items.

2. Actual Successful Onboarding Records

The PROD target was installed with an empty 32GiB disk and a single internal network NIC. On 2026-09-20 04:13 KST, via SSH through the Bastion, the completion marker for Rocky 10.2, 10.77.20.101/24, SELinux Enforcing, and PXE installation was confirmed. The installation RAM of 4GiB was adjusted to 1GiB after a normal shutdown.

The Verify Job checked lastclock=1789846239, lastvalue=942 and the verification time was 1789846262. This is the actual uptime value collected 23 seconds before the verification point. This value is read from the Job results, not artificially generated.

NetBox stored vCPU 1, guest available memory 954MiB, disk 32768MiB, Rocky Linux 10.2, virtualization type virtualbox, management NIC enp0s3, and the PROD environment and actual partition structure. The guest memory observed value of 954MiB and the VirtualBox allocated value of 1024MiB are different values.

AWX’s NetBox inventory received ansible_host=10.77.20.101, ansible_user=labadmin, fml_environment=prod, and fml_subnet=20. It did not just synchronize asset names, but also verified the management IP and environment variables required for calculating the Bastion path.

DEV also performed a PXE installation on an empty disk and ran Workflow 30. From 05:00:40 to 05:03:34 KST, Onboard 31, inventory update 32, and Verify 34 all completed as successful. It was connected to a different environment than PROD with Zabbix host 10684, proxy 2, and uptime item 50884.

STG automatically executed upon detecting installation completion

After approving STG’s first SSH identity with the VirtualBox serial console’s public key and fingerprint, the installation RAM was reduced from 4GiB to 1GiB. The installation completion controller verified the approval key, installation mark, hostname, and machine ID, and synchronized the SCM and bootstrap inventory. Afterward, Job 38 was started without manual Workflow execution.

We even checked the actual uptime value 17 seconds prior to the verification timestamp. The three hosts (PROD, DEV, and STG) along with their respective management IPs and environment variables were also queried in AWX’s NetBox inventory. The initial SSH identity approval is an administrative step left intentionally, and automatic connection was not established by bypassing unapproved keys.

3. Issues found up to the normal path and fixed

Failed jobs were not deleted either. They were kept so that fix commits can be compared with subsequent Job results. Do not confuse cases that failed prior to NetBox/Zabbix registration with a situation where actual assets were duplicated.

4. Backup and isolated restoration

On 2026-09-20 at 04:40 KST, the NetBox DB was backed up in pg_dump custom format and pg_restored to netbox_restore_20260919194012 separate from the production NetBox DB. VM ID 1, fml-prod-app-01, and primary_ip4_id 1 were queried from the restored DB.

The backup SHA-256 is 06914eb419d88edb97245ed435177b396309a625b65eb14b72ae9fb627667390. A separate verification DB was used without overwriting the existing DB. This result is a NetBox logical DB restoration test, not a disaster recovery test that recovers all VMs, media, AWX, and Git at once.

5. Central node outage test

Just before the first test, the PostgreSQL leader was ops02, ops01 was a sync standby, and the replication lag was 0. For Zabbix, ops01 was active and ops02 was standby. Power to unit 1 was forcibly cut to simulate VM power loss.

VIP 10.77.10.10 moved to unit 2. However, during the transition phase, the NetBox backend returned 500 and the VIP returned 503. Zabbix subsequently started active tasks on unit 2 and received connections from all three proxies. This test was not judged as a “zero-downtime HA success”.

The monitoring tool’s first log file was not generated due to a path incompatible with the SELinux policy. Therefore, precise RTO is not calculated from that test. The logging path was moved under /var/log, normal recording was verified, and the test was re-run.

Checking the actual configuration code of NetBox 4.7.1, SENTINEL_TIMEOUT for caching was not mapped in the same way as the task queue. In the second test with only the timeout added, a single cache lookup still took 55.6 seconds. At that time, the first successful NetBox response after powering off unit 1 was about 104 seconds later. Afterwards, local Sentinel priority lookup and explicit retry limits were applied together.

Power loss of active unit 2 for DB, Redis, and Zabbix after modifications

At 05:08:20.777 KST, ops02 was forcibly terminated. Immediately prior, ops02 was PostgreSQL primary, Redis master, and Zabbix active, while ops01 was DB sync standby with lag 0. Independent APIs were queried from node 1 at 5-second intervals with a 4-second request timeout. The values below are from the failure command time to the time the state was first observed, and do not represent precise internal transition times or a guaranteed SLA.

At 05:10:11, with node 2 powered off, a test mark was PATCHed to the comments of the NetBox asset, read via GET, and then restored to its original content. Asset ID 1 and the primary IP were retained. We confirmed not only that the query screen opened, but also writes to the surviving DB path. The recovery of AWX ping does not imply uninterrupted execution of running jobs. Do not overinterpret this as a result indicating successful execution of a new job during this test.

Node 2 was restarted at 05:11:19. PostgreSQL recovered following timeline 3 and was confirmed to be sync standby with lag 0 during the 05:18 inspection. NetBox was ready later due to initialization and Python worker startup, and was confirmed healthy with HTTP 200 during the 05:23 inspection. Service survival time differs from the time it takes for a failed node to return to spare capacity.

Power Loss on Active Node 1 with VIP, DB, and Zabbix After Modification

The opposite direction was also tested. At 05:24:28.045 KST, ops01 was forcibly terminated. On ops02, VIP 10.77.10.10 was verified, and PostgreSQL primary was observed at approx. 39 seconds, Zabbix API normal response after error at approx. 44 seconds, NetBox asset query at approx. 69 seconds, and monitoring values with post-failure timestamps at approx. 111 seconds. The asset ID and management IP were retained.

Since AWX resides only on ops01, it was inaccessible while the node was powered off. After powering ops01 back on at 05:26:44, the AWX ping recovered at 05:29:24. This value indicates service resumption due to the recovery of the original node, not successful AWX native HA. After restart, PostgreSQL returned to timeline 4 sync standby with lag 0, and the NetBox App was also healthy.

Both tests include timeouts and HTTP 500/503 errors during the transition phase. We do not use the term ‘uninterrupted’. Performance improvement effects must not be concluded simply from ratios when comparing pre- and post-improvement metrics of similar tests, as the active DB location and load differed at the time.

6. Proxy Disconnection and Delayed Data Retransmission

At 05:12:43 KST, only TCP 10051 from PROD Edge 10.77.20.10 to the central network was blocked at the Gateway. SSH, Agent-to-Proxy, and DEV paths were maintained. Central PROD uptime stopped at clock 1789848759, while DEV continued to update at 30-second intervals.

Querying the PROD Proxy SQLite database in read-only mode revealed that proxy_history contained values such as clock 1789848879 that were not yet present in the central server. Since the total row count in SQLite may include rows whose transmission completed prior to cleanup, the row count itself was not interpreted as the ‘unsent count’.

Although timeout=240 was specified for the temporary policy, the rule was still present during the 05:18:52 check. The test rule was explicitly removed, and the actual blocking duration was recorded as approx. 6 minutes and 9 seconds. We did not write that it recovered after 4 minutes based solely on the timer option. Repeated tests are conducted with external recovery scheduling and verification of rule absence.

At 05:19:30, the central history.get results showed that values collected during the disconnection period—from 1789848789, 8819, 8849, 8879 up to 9119—reappeared, maintaining 30-second intervals. The latest value was also updated to 1789849149. We verified through actual data that local collection during transmission interruption → disk storage → history reflection for the original timestamps after recovery works properly. This short test did not validate 24-hour buffer retention, disk full conditions, or Proxy VM internal failures.

7. Asset Data Protection and Path Verification

Based on actual PROD collection results, the same NetBox registration was executed again. VM ID 1 and IPAddress ID 1 were retained, and only one instance of the same name and address existed. We distinguished between collection timestamp updates and duplicate asset generation.

Partial collection failure is an input created for testing and is not presented as an actual disk failure case. Rejection conditions are checked first before making changes, and situations where only some APIs succeed are handled so that successfully created objects are not unconditionally deleted, allowing retries.

8. Air-gapped Target Installation and Package Verification

The PXE target has only a single Internal Network NIC and lacks NAT or bridge NICs. The OS was received from Provision’s Rocky 10.2 Minimal installation tree. After clearing the Agent RPM cache in PROD and enabling only two internal repositories, zabbix-agent2 7.0.30 was reinstalled. We verified the actual 6.4MB download, GPG verification, and service active status.

NetBox HTTPS using internal repo metadata and CA verification returned 200 on the same target where external connections were denied. Reinstalling the entire central system in a new air-gapped network is out of scope. What was verified this time is OS installation, Agent deployment, and internal API integration on air-gapped targets.

9. Architecture Screen Verification

The architecture’s failure state and flow playback are for explanation purposes. It is not an operational dashboard that sends commands to servers or displays real-time status.

10. Verification Checklist and Boundaries

Actual physical server UEFI PXE, BMC/RAID automation, failures between two physical hosts, total storage failure, and version detection for all commercial DB/WAS products are not verified by this VM practice. Separate the implementation of each feature from the actual test scope.

Also distinguish between online initial installation and installation/Agent deployment for targets without internet access. Tests that reinstall the entire central system from scratch in a new isolated network must include validation of import bundle dependencies. Disconnecting external NICs from an existing service does not substitute for this.

Items passed in this practice and the next verification tasks are listed in the same table. Do not rephrase the results from a single personal PC as guaranteeing operational SLA or zero data loss across all failures.

Further Reading and Configuration Materials

Portfolio Original · Online and Isolated Network Setup Guide · Failure Scenarios and Verification Records · VirtualBox Configuration Troubleshooting Log

Practice Setup & Automation Source ZIP · ZIP SHA-256

The public ZIP contains configuration templates and automation sources. It does not include OS, RPMs, container images, or credentials. Configure with your own addresses, public CAs, authorized SSH keys, and secret storage before applying.