The Operations Loop — Turning Server Installation into the Start of Operations

AI_Manager

Installing a single server is only part of the preparation for operations. A server only becomes operable when it is tracked—who deployed it and to what environment, what software is running, how to connect to it, and how failures are detected and recovered.

This project is an infrastructure lab that connects everything from Rocky Linux server installation and basic configuration, detailed asset registration, and automated inventory updates, to receiving actual monitoring metrics. We divided the central operations network and PROD, DEV, and STG on Oracle VirtualBox, configuring Zabbix, NetBox, AWX, Git, and PXE into a single operational workflow.

The practice environment consists of VMs on a single personal PC. It is an environment for testing services and VM failures, not an operational system with redundancy for physical hosts or storage. Verification results are categorized based on job IDs, timestamps, and observed values from separate verification projects.

• 1. Project Overview
• 2. Operational Issues to Solve
• 3. Server Configuration and Networking
• 4. The Process of a Server Joining Operations
• 5. How Attached Asset Collection YAMLs Were Applied
• 6. Why It Was Designed This Way
• 7. Failure Scenarios and the Boundaries of Availability
• 8. Operations Considering Both Online and Air-Gapped Networks
• 9. Competencies Demonstrated in This Project
• 10. Further Reading and Technical Basis
• Related Reading and Configuration Materials

1. Project Overview

Rather than simply completing the installation of each tool, the completion criterion was set as whether a server’s identifier and management IP remain consistent across installation, asset management, automation, and monitoring. Even if the API returns 200, onboarding is not treated as successful if the target is missing from the inventory or if the latest Zabbix values are not received.

Actual onboarding has been completed for all three targets: PROD, DEV, and STG. AWX Workflows 22, 30, and 38 successfully performed asset registration, inventory updates, and latest monitoring value verification. We also tested bidirectional power loss of central nodes, proxy transmission disconnects and delayed data recovery, asset conflict rejection, and separate DB recovery. The conditions for each success and unverified scope are documented in the validation article.

2. Operational Issues to Solve

Performing OS installation, Excel asset ledger updates, Ansible host additions, and monitoring registration separately leads to omissions and discrepancies. Issues may arise such as a reinstalled server overwriting previous assets, a NAT address being registered as the management IP, or monitoring showing only the host without any data.

To mitigate this, approved hostnames, environments, and management IPs are set as the input criteria and compared against the hostname, machine ID, and interfaces actually read from the guest. In NetBox, not only string IPs but also VM/Device → Interface → IPAddress → primary_ip4 relationships are established, and AWX pulls inventory from those relationships. When rerunning registration, existing identifiers and management entities are verified.

The second challenge is defining the blast radius of failures. The mere presence of two Zabbix servers does not mean the database itself is HA. Similarly, two NetBox web instances are meaningful only when PostgreSQL, Redis, shared storage, and connection addresses work together. The failover conditions and remaining single points of failure for each tier are explicitly noted.

Interactive Architecture — Overall Structure and Operational Flow

3. Server Configuration and Network

The physical PC runs Windows 11 Pro with an Intel Core Ultra 9 185H (16 cores / 22 logical processors), approximately 31.4GiB of usable memory, and a 2TB SSD. VMs are allocated memory strictly as needed, and PXE installation targets are executed sequentially. Targets requiring 4GiB during installation are reduced to 1GiB post-installation.

In hostnames, ops stands for production service, quorum for failure determination, edge for environment entry point, and provision for installation base. The FQDN recognized by the operating system is 호스트명.fullmoon.test. Public site domains and practice DNS have been separated.

Do not attach NAT or bridge NICs to PXE targets per environment. SSH from the center to the target server goes through the Bastion of the corresponding environment. Inter-environment communication is blocked by default, and necessary sources, destinations, and ports are specified. The NAT and internal service paths of the management VM used for initial package importing are distinguished in the deployment article.

4. The process of a server joining production

Do not place the NetBox administrator password on the remote server. Remote collection is executed with administrator privileges within the necessary scope, and API requests are executed by injecting credentials from the AWX Execution Environment. SSH private keys are not copied to the Bastion.

Automatic inventory addition in AWX is a feasible feature. We connected the official NetBox Ansible inventory plugin as an SCM inventory source, and configured the Workflow to continuously execute inventory sync and verification tasks after a successful registration job. NetBox tokens were separated for asset write and inventory read purposes.

Connections after PXE are handled by the central installation completion controller. After verifying the approved SSH host key, installation completion mark, hostname, and machine ID, the workflow for that server is started. Initial server identity approval remains as an administrator review step, and after approval, asset, inventory, and monitoring registrations are executed sequentially. If the machine ID changes due to reinstallation, it is left as a review target instead of automatic overwriting.

5. How the attached asset collection YAML was reflected

The attached YAML is not a simple hostname registrar. It is an asset register that collects OS, CPU, disk, network, product, virtualization, and backup agent traces using running Linux processes and /proc information as a starting point. Maintaining this intent, four items—collection status, timestamp, error, and machine ID—were added to the 34 original Custom Fields.

The original onsite address, Site, and environment determination rules were replaced with PROD/DEV/STG and explicit inventory variables for this practice. NetBox native IPAM relationships were added to what previously stored only primary_ipv4 strings. Existing approved Site, role, Platform, and user tags are not unconditionally replaced by collection results.

Version detection for all commercial DBs/WASs has not been installation-verified. Verify the ownership, whitelist, and namespace of executable files, and do not run uncertain startup scripts as root. Unverified versions are left as unknown. Product collection inside containers is also categorized as a separate task from host asset collection.

6. Why it was designed this way

Reasons for placing multiple services on the two central VMs

In a production environment, it is generally advantageous to separate servers considering resource, security, and failure isolation by role. Since this lab needs to verify service interworking and failover within a 32GB PC, the roles were consolidated onto two central VMs. NetBox, Zabbix, and data services are separated using Compose projects, volumes, and accounts, while AWX was deployed on K3s to follow the official operator’s installation and management structure.

However, separating containers does not isolate VM failures. Many services use the host network and share the impact of CPU, memory, disk I/O, kernel, and VM restarts. Memory limits and task concurrency limits are the means of resource management under this condition.

Reasons for separating Zabbix Server HA and Proxy

The two central servers divide Active/Standby roles using Native HA and share a common DB. Active proxies for each environment gather local agent data and forward it to the center. When the central connection is lost, the proxy’s disk buffer can continue collection, but if the proxy VM itself stops, that function also disappears. Agent-to-Proxy and Proxy-to-Server are each authenticated using PSKs.

Reasons for separating NetBox HA down to the data tier

NetBox App and Worker are placed on the two central nodes, PostgreSQL is configured with Patroni, Redis with Sentinel, and the access address with Keepalived/HAProxy. The two NetBox nodes use the same SECRET_KEY and token pepper, and media references a common NFS. The third etcd/Sentinel ensures a majority when one of the two central nodes is lost.

NFS itself is a single node. Therefore, the NetBox HA scope of this configuration is defined around the failure of a single central node. It is not described as complete storage HA that survives media storage failures. Although PostgreSQL uses synchronous mode, it is not strict mode, so RPO 0 is not guaranteed in all failures.

Reasons for putting Proxy and Bastion together

By consolidating each environment’s monitoring collection point and automation entry point onto a single VM, resource usage in a small lab was reduced. Bastion forwards only to approved destination SSH ports and restricts interactive shells and agent forwarding. This deployment comes with the trade-off that a failure of a single Proxy/Bastion affects both monitoring and new automation. In actual operations, they can be separated depending on scale and security boundaries.

Reasons for having Git and AWX

Having playbook files is different from being able to reproduce the exact same configuration. By linking Git commits, AWX Project sync, Job ID, target inventory, and Execution Environment versions, you can track what was changed with which code. While NetBox is used as the standard for actual assets, the initial installation approval list is maintained as a separate bootstrap inventory.

7. Failure scenarios and boundaries of availability

This table shows the expected scope by design. Actual failures encountered, collection gaps, recovery times, and data retention are determined separately in the verification posts. The failure animations on the screen do not query actual server states or execute commands.

8. Operations considering both online and air-gapped networks

In the online network, download verified versions from official repositories and record the image digest, RPM signature, and Git commit. In the air-gapped network, bring in the same version of RPMs and dependencies, container images, K3s images, AWX EE, Ansible collections, Git bundles, and OS installation trees. CA, DNS, and NTP must also be provided internally.

For target servers without internet access, provide signed RPMs via an internal HTTP repository, and use HTTPS verified by the lab CA for APIs. Distinguish between HTTP transport for the internal RPM repository and HTTPS authentication for the API. Internal plaintext connections in the data layer are a limitation of the current lab, and TLS, secret management, and access auditing must be reinforced when applied to production.

9. Capabilities demonstrated in this project

The core of this lab is not the number of tools, but the ability to connect operational outcomes. Identifiers from the installation phase are carried through to NetBox and Zabbix, re-execution and partial failures are accounted for, and what is maintained versus what stops after a failure—as well as normal states—is verified.

Repository paths, API schemas, token methods, AWX execution environments, and HAProxy health check issues discovered during setup are recorded along with reproduction conditions and the basis for fixes. VirtualBox firmware, virtual CPU, and installation memory issues will be organized in a separate blog post, distinct from service design.

When expanding to a future operational environment, prioritize physical host and storage redundancy, AWX/Git availability, end-to-end encryption, external secret storage, backup policies, and regular restoration tests. UEFI PXE, NIC driver, RAID, and BMC integration on actual physical servers must be verified separately from this VM test.

10. Further reading and technical basis

Installation procedures and verification results are linked as a series of projects in the same portfolio. The status and scope of each post are updated based on actual test records.

Zabbix Native HA, Zabbix release lifecycle, NetBox required settings and Redis Sentinel, AWX Operator installation, NetBox Ansible inventory plugin, Rocky Linux support lifecycle.

Recommended reading and configuration materials

Portfolio Original Text · Online and Air-Gapped Network Setup Guide · Failure Scenarios and Verification Records · VirtualBox Configuration Troubleshooting Log

Lab Configuration & Automation Source ZIP · ZIP SHA-256

The public ZIP contains configuration templates and automation sources. It does not include the OS, RPMs, container images, or credentials. Configure with your own addresses, public CAs, approved SSH keys, and secret stores before applying.