Building The Operations Loop — From PXE to NetBox, AWX, and Zabbix

AI_Manager

This article is an actual implementation procedure connecting server installation, assets, automation, and monitoring across four internal networks in VirtualBox. The central services are co-located on fml-ops-01/02, but the management units are separated using Compose and K3s. Online and air-gapped network procedures are written together for each step.

Commands are based on the addresses and fixed versions of this lab. Passwords, tokens, and SSH private keys are not included in examples or Git. Placeholders in configuration files are injected from your own secret storage. The scope of execution and actual results should be verified based on the verification report.

• 1. Versions and Installation Sequence
• 2. VM and Network Preparation
• 3. Routing, DNS, NTP, and Access Paths
• 4. Container Runtime and Import Bundles
• 5. PostgreSQL, etcd, and Redis Configuration
• 6. VIP, HTTPS, and NetBox High Availability
• 7. Zabbix LTS HA and Environment-specific Proxy/Bastion
• 8. Git, K3s, AWX, and Execution Environment
• 9. Provisioning Empty Servers with PXE and Kickstart
• 10. Connecting Asset APIs, Inventories, and Monitoring
• 11. Access Information and Brief Operations Guide
• 12. Conclusion with Backup, Updates, and Verification
• Related Reading and Configuration Materials

1. Versions and Installation Sequence

The installation order is OS and network → DNS/NTP and CA → etcd/PostgreSQL/Redis → access VIP → NetBox/Zabbix → environment-specific Proxy/Bastion → K3s/AWX and Git → PXE targets → automation integration → failure and recovery verification. Even if the UI for a later stage comes up, first verify that the DB roles, name resolution, and time synchronization of the preceding stages are operating normally.

“Latest” refers to the fixed version as of the 2026-09-20 setup. For subsequent rebuilds, recheck the support policies and image digests, and do not mix the versions from this article with new versions. Distinguish between Zabbix’s latest regular release and latest official LTS.

2. Preparing VMs and Network

CORE is 10.77.10.0/24, PROD is 10.77.20.0/24, DEV is 10.77.30.0/24, and STG is 10.77.40.0/24. Create them as fml-core/prod/dev/stg Internal Networks respectively. Attach all four internal network NICs to the Gateway and Provision, and attach Edge and targets to their respective environment networks.

When configuring in an online network

Verify the official ISO checksum and create a Rocky minimal installation template. For the management VM, separate the NAT NIC and internal network NIC during the installation period. Bind host SSH port forwarding to 127.0.0.1 only. Do not attach NAT to target PXE VMs.

$VBox = 'C:\Program Files\Oracle\VirtualBox\VBoxManage.exe'
& $VBox list vms
& $VBox showvminfo fml-ops-01 --machinereadable
# Create the target only on an approved empty disk.
.\lab-v2\Create-PxeTarget.ps1 -Environment prod

When cloning templates, generate a new hostname, machine-id, SSH host key, and MAC address for each. Ensure that the original template’s static addresses, keys, and machine-ids do not remain across multiple VMs. Keep SELinux in Enforcing mode and maintain firewalld, while restricting root remote login and password login.

When configuring in an isolated (air-gapped) network

After bringing in the ISO, verified installation packages, internal CA, public keys, and configuration templates, build the same internal network structure. You must be able to access OS installation and internal DNS/NTP even without connecting a NAT NIC. Record “an existing installation that disabled NAT later” and “installation from scratch by bringing in required dependencies” as separate tests.

On this PC, booting a new BIOS PXE target stalled at 2 vCPU, so 1 vCPU was used. Central Unit 1 ran with 4 vCPU under the Intel Core i7-6700K CPU profile. This profile name does not reflect the actual PC CPU. Hyper-V, VBS, and memory integrity were left unchanged. Detailed VM issues are covered in a separate blog.

3. Routing, DNS, NTP, and Access Paths

Inter-environment communication is permitted by the Gateway’s firewalld policy. Provision is directly connected to each environment to provide DHCP, PXE, and storage, and is not used as a router.

The table above summarizes communication purposes. For actual allow rules, check both host firewall and router policies. Do not expose DB, etcd, or Redis to the internet or all environment networks.

When configuring in an online network:

Install the required packages on the management VM and create the internal interface and static route. Do not mix the default route for the internet with the static route for the lab network.

nmcli connection modify fml-internal \
  +ipv4.routes '10.77.20.0/24 10.77.10.1'
nmcli connection modify fml-internal \
  +ipv4.routes '10.77.30.0/24 10.77.10.1'
nmcli connection modify fml-internal \
  +ipv4.routes '10.77.40.0/24 10.77.10.1'
nmcli device reapply enp0s8
ip route
chronyc tracking

When configuring in an air-gapped network:

The internal DNS resolves netbox.fullmoon.test, zabbix.fullmoon.test, and awx.fullmoon.test to VIP 10.77.10.10, and git.fullmoon.test to 10.77.10.12. Do not rely on external DNS fallback. Use Provision’s chrony as the internal reference, but prepare a separately verified time source and synchronization policy for actual operations.

getent hosts netbox.fullmoon.test git.fullmoon.test
chronyc sources -v
ip route get 10.77.20.10

Browser access is performed using HTTPS names from a management terminal that trusts the lab CA. A loopback SSH tunnel can be used when the internal network is not directly routed from Windows. First, check that no local port 443 is already in use.

ssh -i <관리용_개인키> -p 22031 -N -L 127.0.0.1:443:10.77.10.10:443 labadmin@127.0.0.1

In the management terminal’s hosts file, map the three service names to 127.0.0.1, and register the verified public CA certificate in the trust store. Do not import the server CA private key. To access the central node 1 during a failure test, use the management SSH port 22032 of central node 2. Ignoring certificate warnings must not be adopted as a regular access procedure.

4. Container Runtime and Import Bundle

When configuring in an online network:

Install Docker and Compose from the official Docker RHEL repository. Separate the Compose project paths into /opt/fullmoon-lab/core, /opt/fullmoon-lab/apps, and /opt/fullmoon-lab/edge. Set sensitive environment files to root-owned 0600 and parent directories to 0700.

docker version
docker compose version
docker compose --project-directory /opt/fullmoon-lab/core \
  -f /opt/fullmoon-lab/core/compose.yml config --quiet
docker image inspect --format '{{json .RepoDigests}}' \
  zabbix/zabbix-server-pgsql:alpine-7.0.30

Since the rendered compose full output may contain passwords, do not store config content in public logs. Also distinguish between whether the host SELinux is Enforcing and whether the Docker daemon’s SELinux integration is enabled. This lab maintains host Enforcing but does not claim an environment where Docker SELinux integration is even applied.

When configuring in an air-gapped network:

Download RPMs and dependencies from a connected preparation VM with the same CPU architecture and Rocky major release. Docker image archives and K3s/containerd archives are separate. Loading an image into Docker does not mean it can be used in K3s.

# Preparation VM: save the required RPM versions and all dependencies
dnf download --resolve --alldeps --destdir ./rpms \
  docker-ce docker-ce-cli containerd.io docker-compose-plugin
createrepo_c ./rpms
docker image save -o images.tar <반입할_고정_이미지_목록>
sha256sum images.tar > images.tar.sha256

# After transferring the bundle to the isolated network
sha256sum -c images.tar.sha256
docker image load -i images.tar
k3s ctr images import fullmoon-ee-r2.tar

The import list consists of OS ISO and RPM repositories, Docker/Compose, Patroni build results, NetBox/Zabbix/Redis/etcd images, K3s binaries and airgap images, Operator/RBAC proxy/AWX/EE images, Git bundles, configuration templates and public CAs, signing keys, and checksums. Verify on a new target without an external NIC to avoid accidentally passing due to the preparation VM’s cache.

5. PostgreSQL, etcd, and Redis Configuration

etcd and Sentinel are deployed on the two central nodes and the quorum node. PostgreSQL/Patroni and Redis are deployed on the two central nodes. The database separates accounts and databases for NetBox, Zabbix, and AWX.

When configuring in an online network:

Pull/build fixed images, and deploy node-specific NODE_NAME/NODE_IP, Patroni settings, Redis/Sentinel settings, and restricted secret files. Start the central DB after verifying that the three etcd nodes can communicate with each other.

docker compose --project-directory /opt/fullmoon-lab/core \
  -f /opt/fullmoon-lab/core/compose.yml --profile central up -d
docker compose --project-directory /opt/fullmoon-lab/core \
  -f /opt/fullmoon-lab/core/compose.yml exec -T postgres \
  patronictl -c /etc/patroni/patroni.yml list
curl --fail http://10.77.10.11:8008/patroni
curl --fail http://10.77.10.12:8008/patroni

The quorum node runs only etcd and Sentinel without the central profile. Patroni’s synchronous_mode is enabled and synchronous_mode_strict is set to false. Replication status and potential commit loss during a failure must be evaluated according to operational conditions. Do not use the method of making each of the two databases a manual primary.

When configuring in an air-gapped network:

Since the Patroni Dockerfile requires external repositories like pip and apt, complete the images in the online preparation stage and import them. Avoid building on the spot in the field and encountering internet dependencies. Load the same archive onto both central nodes, compare the image IDs, and start with the same procedure.

HAProxy’s database access port is 6432. Only nodes where Patroni /primary returns 200 are used as DB write backends. Since idle DB connections may be dropped at the initial 60-second timeout, the DB listener’s client/server timeout was separated to 1 hour. Adjust the values according to actual query and connection pool policies.

6. VIP, HTTPS, and NetBox Redundancy

Keepalived’s VIP is 10.77.10.10, and the normal preferred node is ops01. HAProxy on both nodes connects to NetBox 8082, Zabbix Web 8080, and AWX 30080. The active roles of Node, DB, Redis, and VIP do not always need to be the same node.

When configuring in an online network:

Issue certificates with the three web names in SAN using the lab CA, and place the certificates/keys on the two central nodes with restricted permissions. Use the same SECRET_KEY, API token pepper, and Redis Sentinel settings for NetBox App and Worker.

# Key relationships in netbox-configuration.py
DATABASES = {'default': {
    'ENGINE': 'django.db.backends.postgresql',
    'NAME': 'netbox', 'USER': 'netbox',
    'PASSWORD': '<비밀_파일에서_주입>',
    'HOST': '127.0.0.1', 'PORT': 6432,
}}
# Use matching SECRET_KEY / API_TOKEN_PEPPERS values on both App nodes.
# For REDIS tasks/caching, specify the Sentinel service and the respective DB numbers.

Use the actual version-specific formats of default configuration files based on the provided NetBox settings and official documentation. If you only match environment variable names and do not read them in Python settings, they will not be applied. Complete the initial DB migration on one node first, and then start the remaining nodes and Worker.

Media is shared via NFSv4 at quorum’s /srv/fullmoon/netbox-media. Allow only the two central nodes for export and maintain root_squash. Check the actual UID of the NetBox image to match directory permissions. The NFS SELinux boolean verified in this Rocky environment is virt_use_nfs.

findmnt /srv/fullmoon/netbox-media
haproxy -c -f /etc/haproxy/haproxy.cfg
keepalived -t -f /etc/keepalived/keepalived.conf
systemctl is-active haproxy keepalived
curl --fail https://netbox.fullmoon.test/login/ -o /dev/null

When configuring in an air-gapped network:

The NetBox image includes Python dependencies required for execution, so import images with the same digest to both nodes. Put the internal CA not only on the host but also in the AWX EE. Prepare a secret recovery procedure to ensure the SECRET_KEY/pepper of the two nodes do not diverge. NFS outages affect media independently of web API success, so test them separately.

An issue actually encountered was the NetBox health check. When inspecting /login/ without an HTTP/1.1 Host, NetBox returned 400, HAProxy judged all backends as down, and the VIP returned 503. Host was specified as follows.

backend netbox_ui
    mode http
    option httpchk
    http-check send meth GET uri /login/ ver HTTP/1.1 hdr Host netbox.fullmoon.test
    http-check expect status 200
    server ops01 10.77.10.11:8082 check
    server ops02 10.77.10.12:8082 check

During the failure test, a single NetBox cache lookup took about 55.6 seconds due to the default retry of the Redis connection. We checked the installed NetBox 4.7.1 and django-redis code, and changed it to query the local Sentinel first. The cache specified connection and response timeouts and retry policies in the supported KWARGS. Loopback is a different access address of the same Sentinel, and no majority member was added.

from redis.backoff import NoBackoff
from redis.retry import Retry

# Query the local Sentinel on each NetBox node first.
SENTINELS = [('127.0.0.1', 26379), ('10.77.10.13', 26379),
             ('10.77.10.11', 26379), ('10.77.10.12', 26379)]
REDIS['caching']['KWARGS'] = {
    'socket_connect_timeout': 3,
    'socket_timeout': 3,
    'retry': Retry(NoBackoff(), 0),
}

Because this value is restricted to return errors quickly, retries for individual requests and service recovery times differ. The time until API recovery after an actual DB/Redis primary failure can be verified in the verification article. Do not generalize that the same results will occur under operational load or other versions.

7. Zabbix LTS HA and Environment-Specific Proxy/Bastion

When configuring in an online network:

Server/Web/Proxy use the alpine-7.0.30 tag, and Agent 2 targeting Rocky uses the 7.0.30 RPM for el10. Both servers point to a common Zabbix DB but are configured with different HANodeName and NodeAddress. Web is configured to find active nodes from HA information without fixing a specific server address.

# ops01
ZBX_HANODENAME=fml-ops-01
ZBX_NODEADDRESS=10.77.10.11:10051
# ops02
ZBX_HANODENAME=fml-ops-02
ZBX_NODEADDRESS=10.77.10.12:10051
# Join the Active Proxy addresses belonging to the same HA cluster with a semicolon
ZBX_SERVER_HOST=10.77.10.11:10051;10.77.10.12:10051

The Proxy name must exactly match the name registered in the Zabbix API. Distinguish between environment-specific Proxy PSKs and Agent PSKs. Keep Proxy SQLite data on a persistent volume and apply a disk buffer policy. Buffer operation can only be considered verified after confirming that delayed data was retransmitted after disconnection.

When configuring in an isolated network

Bring in the proxy image using docker load. Provide the agent RPM through the internal repository while maintaining GPG signature verification. This el10 RPM was verified with the B5333005 key. Since verification failed in the configuration using only the existing A14FE591 key, the official signing key’s fingerprint was checked and added.

rpm --import /etc/pki/rpm-gpg/RPM-GPG-KEY-ZABBIX-B5333005
rpm -K zabbix-agent2-7.0.30-release1.el10.x86_64.rpm
# Verified key fingerprint
# 4C3D6F2CC75F5146754FC374D913219AB5333005

The Bastion forwards only to the environment target .101:22. Interactive shells and agent forwarding are not allowed. Connect to the target using the central key and do not leave the private key on the Edge. Check with sshd -t and reload while preserving the existing management account’s SSH access.

Match User bastion
    AuthenticationMethods publickey
    AllowTcpForwarding local
    PermitOpen 10.77.20.101:22
    PermitTTY no
    AllowAgentForwarding no
    ForceCommand /bin/false
Match all

8. Git, K3s, AWX, and Execution Environment

When configuring in an online network

The Git bare repository is /srv/git/fullmoon-automation.git on ops02. AWX fetches the main branch using fmlgit’s read-only deployment key. The forced command is enclosed in single quotes up to the path so that git-shell can interpret it.

restrict,command="git-upload-pack '/srv/git/fullmoon-automation.git'" ssh-ed25519 <공개키>

K3s was installed on ops01, and SELinux and Secret encryption were enabled. Traefik, ServiceLB, and metrics-server were excluded as they are unnecessary for this lab. AWX is fixed to Operator 2.19.1 and AWX 24.6.1, and the PostgreSQL Secret points to the 6432 path of the existing HA DB. The administrator password, DB password, and CA bundle are injected as Kubernetes Secrets.

k3s kubectl -n awx get pods,jobs
k3s kubectl -n awx apply -f awx.yml
curl --fail https://awx.fullmoon.test/api/v2/ping/

Because the Operator’s initial RBAC proxy image gcr.io path returned a 404, it was corrected to the official quay.io/brancz image of the same v0.15.0. In the AWX CRD, cookie settings must be strings and host_aliases must be arrays. Putting SYSTEM_TASK_ABS_MEM as a number causes the corresponding AWX code to call string methods and the dispatcher to fail, so unnecessary manual settings were removed.

When configuring in an isolated network

Import the K3s airgap images and all images generated by the Operator. In particular, if the init/migration/EE/RBAC proxy images are missing, installation will not proceed even if the Web image is present. After importing into K3s containerd, verify that the actual image names match the CR names.

The lab EE has netbox.netbox:3.23.0, public CA, and verified Git/Edge SSH host keys added to awx-ee:24.6.1. The image name is localhost/fullmoon-ee:24.6.1-netbox3.23.0-r2 and the pull policy of the AWX execution environment is Never. Having localhost in the name does not mean a registry is running; it uses the local image pre-imported into K3s.

k3s ctr images import fullmoon-ee-r2.tar
k3s ctr images list
git bundle verify fullmoon-automation.bundle

In AWX, create a Project, approved bootstrap inventory, NetBox inventory source, SSH credential, NetBox/Zabbix API credential, Onboard/Verify Job Template, and Workflow. The Workflow success path is Onboard → NetBox inventory sync → Verify. AWX itself is a single-node configuration on central unit 1. Do not judge that it is a physically distributed HA based solely on the ha field of the API ping.

9. Provisioning Empty Servers with PXE and Kickstart

When configuring in an online network

Verify the official Rocky Minimal ISO and provide the installation tree at /var/www/html/rocky on Provision. The actual ISO contains Minimal/repodata. Using the DVD’s BaseOS/AppStream paths as-is causes a 404. DHCP maps only approved MACs per environment to .101.

fml-prod-app-01 : 08:00:27:a0:20:65 → 10.77.20.101
fml-dev-app-01  : 08:00:27:a0:30:65 → 10.77.30.101
fml-stg-app-01  : 08:00:27:a0:40:65 → 10.77.40.101

In this PC’s VirtualBox, BIOS + official ipxe-legacy.iso worked from DHCP → HTTP kernel/initrd → Kickstart. The iPXE file is the boot medium that initiates network installation, and OS packages are received from Provision. Create the new VM with an empty 32GiB disk, 4GiB installation RAM, and 1 vCPU, reducing it to 1GiB after installation.

Kickstart’s %pre allows partitioning only after verifying the MAC, DMI VirtualBox identification, presence of /dev/sda, and absence of a disk signature. Lock the root password and configure the labadmin public key, sshd, sudo, chrony, and firewalld. Register the SSH host key left by %post by comparing it with trusted VM console records. Do not accept unknown host keys blindly.

When configuring in an isolated network

Bring the same ISO, boot files, and Kickstart into internal HTTP. The target has no external NIC and receives DNS, NTP, and packages from Provision in the same environment. By configuring an internal repository for Agent 2, basic setup can be performed even without an external RPM repository.

When applying to physical servers, do not remove the VirtualBox DMI guard as-is and run it. After separately mapping approved actual serial/BMC/MACs, RAID logical disks and installation target WWNs, UEFI/Secure Boot and NIC drivers, you must review the scope of disk destruction. The actual installation evidence in this article is a VM, and it should not be expressed as physical server bare-metal verification results.

10. Connecting Asset API, Inventory, and Monitoring

When configuring in an online network

Grant NetBox’s asset write account the necessary VM/Device/Interface/IP/catalog permissions, and allow the inventory account view-only access. Initial Custom Field and catalog creation are performed via a separate bootstrap. NetBox 4.7 v2 tokens are in the format Bearer nbt_. The Zabbix automation account restricts managed groups and API methods and sets an expiration date for tokens.

plugin: netbox.netbox.nb_inventory
api_endpoint: https://netbox.fullmoon.test
token:
  type: Bearer
  value: "{{ lookup('env', 'NETBOX_TOKEN') }}"
validate_certs: true
query_filters:
  - tag: auto
  - status: active

For the collection phase, deploy the Python collector to a temporary random directory and always remove it after execution. Compare the target’s actual hostname, machine ID, and management NIC/IP with the approved inventory. If values differ or the existing IP is assigned to a different asset, stop before making changes in NetBox. Do not include API tokens or the entire /proc environment in collection results.

API operations are executed in the AWX EE with delegate_to: localhost. If target ansible_become variables are propagated to local tasks, it may fail looking for sudo inside the EE, so specify it as follows.

delegate_to: localhost
become: false
vars:
  ansible_become: false
  ansible_python_interpreter: "{{ ansible_playbook_python }}"

The registration order is actual asset/Interface/IPAM → Zabbix host/Proxy/template → NetBox inventory source sync → verification of target baseline and latest Zabbix data. Since NetBox and Zabbix APIs are not a single DB transaction, they support re-execution after partial success, and compensation tasks that unconditionally delete successfully registered assets are not performed.

When configuring in an isolated network

The method is the same because API code uses only internal HTTPS. The EE’s CA, collection, Python libraries, pinned SSH host keys, and Git commits must all be brought in. Do not download packages from Galaxy or pip during Job execution. Specify only the fullmoon-minimal and fullmoon-zabbix repositories on the target to install the Agent.

ServerActive=10.77.20.10:10051
Hostname=fml-prod-app-01
TLSConnect=psk
TLSAccept=psk
TLSPSKIdentity=fml-prod-app-01
TLSPSKFile=/etc/zabbix/agent.psk

Keep the PSK file owned by zabbix with 0400 permissions and do not leave its value in AWX logs. Check for duplicate management hosts and verify the latest item’s lastclock and lastvalue. Do not treat creation as complete based solely on the host creation response.

Installation Completion and Automatic Workflow Connection

Central unit 1’s fullmoon-postinstall.timer verifies the completion of the approved target’s installation. SSH goes through the corresponding Bastion and verifies the pinned host key. If /var/lib/fullmoon-lab/pxe-installed, hostname, and machine ID match, execute the Workflow specifying Project sync → bootstrap inventory sync → one target as a limit. Job IDs and statuses are left in state.json, which is readable only by root.

Compare the initially generated SSH host key with the trusted VirtualBox serial console and add it to the approval list. The controller waits until this identity is approved. Do not auto-accept unknown keys. Subsequent installation completion detection, API registration, and monitoring verification proceed automatically. On physical servers, this identity registration must be linked to the BMC console or the organization’s SSH CA issuance procedure.

systemctl status fullmoon-postinstall.timer
journalctl -u fullmoon-postinstall.service --since '-30min'
# Success: launched → AWX workflow ID → completed
# Failure: manual_review_required. Do not register again automatically before checking the cause.

This controller and AWX reside on ops01, so new onboarding halts during a failure of that node. If ops02, which holds Git, goes down, new Project syncs fail. If state files are lost or processes terminate immediately after execution responses, the same Workflow may be retried, so asset registration itself was structured in a re-executable manner that checks for identifier and IP duplication.

11. Connection Information and Brief Operations Guide

Routine inspections are performed in the order of time synchronization and disk/memory → DB/Redis role → App/Worker → HA/VIP → Proxy → actual Agent data → recent AWX failures. Do not simply check whether containers are Up.

df -h
free -m
chronyc tracking
docker ps --format '{{.Names}} {{.Status}}'
k3s kubectl -n awx get pods
systemctl is-active haproxy keepalived

When adding a new server, first register the approved inventory, MAC/IP, Bastion PermitOpen, and SSH host key, and commit them to Git. After a project sync, target only that specific host using the limit option. Verify NetBox’s primary IP and environment group before transferring to operations. Do not blindly change all running servers to an unrestricted all target.

Password changes and token renewals must manage the secret store, AWX Credentials, and service restart scope together. Expired tokens, changed SSH host keys, and different machine IDs are not bypassed automatically. After a node failure, restore the replica based on the surviving primary, and do not arbitrarily enable writing on a separated node.

12. Concluding with backup, update, and verification

Manage PostgreSQL logical backups, NetBox media, Git repositories, AWX SECRET_KEY and DB, and CA/config/secrets separately. VM snapshots alone do not replace application-consistent backups. Having a backup file and restoring it to an isolated DB to read the same assets are two different verification processes.

Record image tags/digests along with EE/collections for updates and proceed after verification. NetBox migrations or major DB upgrades may not be recoverable simply by reverting to the previous image, so prepare data backup and restoration procedures. In an air-gapped environment, create a new update import bundle and checksum while preserving the previous verification bundle.

Build completion criteria include asset registration, inventory sync, latest monitoring values, re-execution, address conflict rejection, service failover, Proxy buffer recovery, backup restoration, and target installation without external communication. Clearly separate actually performed tests and unexecuted items in the verification project.

Further reading and configuration materials

Portfolio original text · Online and air-gapped network build guide · Failure scenarios and verification records · VirtualBox configuration troubleshooting log

Lab configuration and automation source ZIP · ZIP SHA-256

The public ZIP contains configuration templates and automation sources. OS, RPM, container images, and credentials are not included. Apply after configuring with your own address, public CA, approved SSH keys, and secret store.