Fullmoon System

Deploy Elastic Stack on Rocky Linux 9: TLS, ILM, and Operations

EdwardMoon

Deploying Elastic Stack involves more than installing four packages. Design ingestion authentication, Elasticsearch TLS and permissions, Kibana exposure, retention policies, and snapshot recovery as one operational system. This guide provides a reproducible installation and validation procedure for Rocky Linux 9 and Elastic Stack 9.4.2.

The lab starts on a single host; production should separate roles and failure domains. Each command is accompanied by its purpose and verification steps. Do not hardcode passwords or API keys in published configuration or shell history, and keep Elastic's default security settings enabled.

Elastic Stack architecture: TLS ingestion, three-node Elasticsearch, Kibana, ILM, and external snapshots
TLS-protected ingestion, search, and visualization alongside a three-node cluster, external snapshot storage, and data lifecycle management

Scope and Components

Component Role Operational priorities
Elasticsearch Document indexing, search, aggregation, and shard management Three-node quorum, TLS, capacity, and shard counts
Kibana Search, dashboards, and administration UI Loopback binding, HTTPS proxy, and RBAC
Logstash Parsing, transformation, and routing Backpressure, persistent queues, and secrets
Filebeat/Elastic Agent Log and metric collection TLS trust, field conventions, and retries
ILM Automated rollover and retention Shard sizes, retention requirements, and execution state
Snapshot/SLM Off-cluster backup and recovery Repository validation, recovery drills, and separate access controls
Here, ELK refers to the historical Elasticsearch/Logstash/Kibana combination. Elastic Stack includes Beats and Agent together with security, lifecycle management, and backups. For simple collection, Agent or Filebeat can send directly to Elasticsearch without Logstash.

Design and Requirements

Rocky Linux 9 is RHEL-compatible, but if you need commercial support, verify the exact OS combination in Elastic's support matrix. Align all Elastic Stack components to the same version. This guide uses 9.4.2 as its example baseline; it does not claim that version is the latest. Check package availability in the official repository and the support matrix when installing.

Separate Lab and Production Topologies

Item Single-node lab Production direction
Elasticsearch One node, discovery.type=single-node At least three master-eligible nodes across failure domains
Kibana 127.0.0.1 on the same host Separate hosts or multiple instances behind an HTTPS proxy
Logstash Local pipeline Dedicated nodes sized for ingestion volume, with persistent queues
Backups A separate path for functional testing Shared storage or object storage outside the cluster
Exposed ports Loopback only Source-restricted firewall rules, TLS, and a separate management network

Check Host Resources and Time Synchronization

sudo dnf install -y curl jq
cat /etc/rocky-release
uname -r
timedatectl status
free -h
df -hT /var/lib /var/log
sysctl vm.max_map_count
ulimit -n

Elasticsearch makes extensive use of the filesystem cache as well as the JVM heap. Heavy Logstash processing on the same host can therefore create memory contention. Start with automatic heap sizing, and set Xms and Xmx to the same explicit value only when load testing supports the change.

Configure the 9.4.2 Repository and Install Packages

Use Elastic's official GPG key and 9.x RPM repository. Keeping the repository disabled by default reduces unexpected version changes during routine dnf updates. Review change history and compatibility before installing.

Register the Official GPG Key and Repository

sudo rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch

sudo tee /etc/yum.repos.d/elastic-9.x.repo >/dev/null <<'EOF'
[elastic-9.x]
name=Elastic repository for 9.x packages
baseurl=https://artifacts.elastic.co/packages/9.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=0
type=rpm-md
EOF

sudo dnf clean metadata
sudo dnf --disablerepo='*' --enablerepo=elastic-9.x list available   elasticsearch kibana logstash filebeat

Install Matching Versions and Verify the Pins

STACK_VERSION='9.4.2'
sudo dnf install --enablerepo=elastic-9.x   "elasticsearch-${STACK_VERSION}"   "kibana-${STACK_VERSION}"   "logstash-${STACK_VERSION}"   "filebeat-${STACK_VERSION}"

rpm -q elasticsearch kibana logstash filebeat
sudo dnf install python3-dnf-plugin-versionlock
sudo dnf versionlock add elasticsearch kibana logstash filebeat
If the dnf versionlock plugin is missing, install python3-dnf-plugin-versionlock first. Pinning versions does not mean postponing patches forever; it lets you explicitly unlock and upgrade during a maintenance window after validating snapshots, compatibility, and rollback.

Secure a Single Elasticsearch Node

Bind Elasticsearch only to loopback for the single-node lab. On its first startup, Elastic 9.x automatically configures authentication and HTTP/transport TLS. Configure each client to trust the generated CA rather than turning security off.

Minimal Lab Settings in elasticsearch.yml

Back up and edit the file with the commands below. Set each subsequent YAML key only once in the existing file, preserving automatically generated authentication and TLS settings. Do not replace the entire file with this short YAML fragment.

sudo cp -a /etc/elasticsearch/elasticsearch.yml /etc/elasticsearch/elasticsearch.yml.before-lab
sudoedit /etc/elasticsearch/elasticsearch.yml
cluster.name: logs-lab
node.name: es01
network.host: 127.0.0.1
http.host: 127.0.0.1
transport.host: 127.0.0.1
discovery.type: single-node
sudo systemctl daemon-reload
sudo systemctl enable --now elasticsearch
sudo systemctl status elasticsearch --no-pager

If a key already exists, edit its value instead of adding a duplicate. If startup fails, inspect the journal and Elasticsearch logs for bootstrap checks, permissions, or YAML errors before considering configuration changes; do not disable security.

sudo journalctl -u elasticsearch --since '-10 min' --no-pager
sudo tail -n 100 /var/log/elasticsearch/logs-lab.log
sudo ss -lntp | grep -E ':(9200|9300)\b'

Reset the Password and Verify HTTPS with the CA

Copy only the public CA certificate to a separate location readable by curl from an ordinary shell. Do not copy the CA private key or the transport/HTTP keystores.

sudo install -d -m 0755 /etc/elastic-client
sudo install -o root -g root -m 0644 /etc/elasticsearch/certs/http_ca.crt /etc/elastic-client/http_ca.crt
openssl x509 -in /etc/elastic-client/http_ca.crt -noout -fingerprint -sha256
sudo /usr/share/elasticsearch/bin/elasticsearch-reset-password   -u elastic -i

read -rsp 'elastic password: ' ELASTIC_PASSWORD
echo
export ELASTIC_PASSWORD

curl --fail --silent --show-error   --cacert /etc/elastic-client/http_ca.crt   -u "elastic:${ELASTIC_PASSWORD}"   https://127.0.0.1:9200/ | jq .

unset ELASTIC_PASSWORD

Skipping certificate verification may establish a connection, but it prevents detection of an impersonating server. Distribute the generated CA fingerprint or certificate, and use service-specific least-privilege accounts or API keys instead of personal passwords.

Enroll Kibana and Publish It over HTTPS

Kibana can enroll with a token that provides Elasticsearch CA and authentication information. A straightforward setup binds Kibana to 127.0.0.1 and terminates HTTPS at a reverse proxy using your organization's certificate, rather than exposing port 5601 directly to browsers.

Create a Kibana Enrollment Token and Enroll

sudo /usr/share/elasticsearch/bin/elasticsearch-create-enrollment-token   -s kibana

# Enter the short-lived token from the previous command in the next command.
sudo /usr/share/kibana/bin/kibana-setup   --enrollment-token '<ONE_TIME_ENROLLMENT_TOKEN>'

sudo tee -a /etc/kibana/kibana.yml >/dev/null <<'EOF'
server.host: "127.0.0.1"
server.port: 5601
server.publicBaseUrl: "https://kibana.example.com"
EOF

sudo systemctl enable --now kibana
sudo journalctl -u kibana --since '-10 min' --no-pager
Replace example domains, certificate paths, and allowed networks with actual values. Enrollment tokens are short-lived; do not preserve them in documents or tickets. Generate a new token if one is exposed or expires.

Example Nginx HTTPS Reverse Proxy

First install the domain's certificate and private key at the indicated paths, then save the server block below to /etc/nginx/conf.d/kibana.conf. Confirm that the http block in nginx.conf includes that directory.

sudo dnf install -y nginx policycoreutils-utils
sudoedit /etc/nginx/conf.d/kibana.conf
# Allow Nginx to connect to Kibana on loopback with SELinux enforcing.
sudo setsebool -P httpd_can_network_connect on
server {
    listen 443 ssl http2;
    server_name kibana.example.com;

    ssl_certificate     /etc/pki/tls/certs/kibana-fullchain.pem;
    ssl_certificate_key /etc/pki/tls/private/kibana.key;

    location / {
        proxy_pass http://127.0.0.1:5601;
        proxy_http_version 1.1;
        proxy_set_header Host $host;
        proxy_set_header X-Forwarded-Proto https;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
    }
}
sudo nginx -t
sudo systemctl enable --now nginx
sudo systemctl reload nginx

# Replace this with your management network range.
sudo firewall-cmd --permanent --new-zone=kibana-admin || true
sudo firewall-cmd --permanent --zone=kibana-admin   --add-source=10.20.0.0/16
sudo firewall-cmd --permanent --zone=kibana-admin --add-service=https
sudo firewall-cmd --reload
sudo firewall-cmd --zone=kibana-admin --list-all

Prepare Data Streams and ILM Before Ingestion

Create the policy and template before sending the first logs. Template changes do not apply retroactively to existing backing indices. For an existing stream, inspect its current ILM policy and handle policy changes and rollover through a separate change procedure. Run the ILM explain command below after ingestion creates a backing index.

Continuously writing time-based logs to one index produces oversized shards and longer recovery times. Data streams and ILM can roll over to a new backing index when conditions are met and automatically delete data after its retention period. This example rolls over at 50 GB or one day.

Create a 30-Day ILM Policy

read -rsp 'elastic password: ' ELASTIC_PASSWORD
echo

curl --fail --silent --show-error   --cacert /etc/elastic-client/http_ca.crt   -u "elastic:${ELASTIC_PASSWORD}"   -X PUT https://127.0.0.1:9200/_ilm/policy/logs-30d   -H 'Content-Type: application/json' -d @- <<'JSON'
{
  "policy": {
    "phases": {
      "hot": {
        "actions": {
          "rollover": {"max_primary_shard_size": "50gb", "max_age": "1d"}
        }
      },
      "delete": {
        "min_age": "30d",
        "actions": {"delete": {}}
      }
    }
  }
}
JSON

Create an Index Template for the Data Stream

curl --fail --silent --show-error   --cacert /etc/elastic-client/http_ca.crt   -u "elastic:${ELASTIC_PASSWORD}"   -X PUT https://127.0.0.1:9200/_index_template/logs-app   -H 'Content-Type: application/json' -d @- <<'JSON'
{
  "index_patterns": ["logs-app-*"],
  "priority": 500,
  "data_stream": {},
  "template": {
    "settings": {
      "index.lifecycle.name": "logs-30d",
      "number_of_shards": 1,
      "number_of_replicas": 1
    }
  }
}
JSON
On a single-node lab, replicas=1 leaves replica shards unassigned and cluster health yellow. Use replicas=0 only for testing, or deploy at least two data nodes in production. Calculate retention and rollover thresholds from actual daily ingestion and recovery objectives.

Inspect ILM State

curl --fail --silent --show-error   --cacert /etc/elastic-client/http_ca.crt   -u "elastic:${ELASTIC_PASSWORD}"   'https://127.0.0.1:9200/_ilm/status?pretty'

curl --fail --silent --show-error   --cacert /etc/elastic-client/http_ca.crt   -u "elastic:${ELASTIC_PASSWORD}"   'https://127.0.0.1:9200/.ds-logs-app-*/_ilm/explain?pretty'

Build the Logstash and Filebeat Ingestion Pipeline

In this local example, Filebeat sends to Logstash on loopback, and Logstash verifies Elasticsearch's certificate against its CA. If remote Beats connect on port 5044, deploy server and client certificates separately and restrict firewall sources.

Create a Least-Privilege Ingestion Account

read -rsp 'elastic password: ' ELASTIC_PASSWORD
echo
read -rsp 'logstash_writer password: ' LOGSTASH_WRITER_PASSWORD
echo

curl --fail --silent --show-error   --cacert /etc/elastic-client/http_ca.crt   -u "elastic:${ELASTIC_PASSWORD}"   -X PUT https://127.0.0.1:9200/_security/role/logstash_writer_role   -H 'Content-Type: application/json' -d @- <<'JSON'
{
  "cluster": ["monitor"],
  "indices": [
    {
      "names": ["logs-app-*"],
      "privileges": ["auto_configure", "create_doc"]
    }
  ]
}
JSON

jq -n --arg password "$LOGSTASH_WRITER_PASSWORD"   '{password:$password,roles:["logstash_writer_role"]}' | curl --fail --silent --show-error   --cacert /etc/elastic-client/http_ca.crt   -u "elastic:${ELASTIC_PASSWORD}"   -X PUT https://127.0.0.1:9200/_security/user/logstash_writer   -H 'Content-Type: application/json' --data-binary @-

unset LOGSTASH_WRITER_PASSWORD

Validate Logstash Syntax and Connectivity

sudo install -d -o logstash -g logstash -m 0750 /etc/logstash/certs
sudo install -o logstash -g logstash -m 0640   /etc/elasticsearch/certs/http_ca.crt   /etc/logstash/certs/http_ca.crt

sudo /usr/share/logstash/bin/logstash-keystore   --path.settings /etc/logstash create
sudo /usr/share/logstash/bin/logstash-keystore   --path.settings /etc/logstash add ES_PASSWORD

ES_PASSWORD should contain the password of the dedicated least-privilege ingestion account, not the elastic superuser used for initial learning. The configuration below references keystore variables instead of literal secrets, keeping credentials out of the configuration file.

sudo tee /etc/logstash/conf.d/beats-to-elasticsearch.conf >/dev/null <<'EOF'
input {
  beats { host => "127.0.0.1" port => 5044 }
}

filter {
  if ![@metadata][pipeline] { mutate { add_tag => ["pipeline-default"] } }
}

output {
  elasticsearch {
    hosts => ["https://127.0.0.1:9200"]
    user => "logstash_writer"
    password => "${ES_PASSWORD}"
    ssl_enabled => true
    ssl_certificate_authorities => ["/etc/logstash/certs/http_ca.crt"]
    ecs_compatibility => "v8"
    manage_template => false
    ilm_enabled => false
    data_stream => "true"
    data_stream_type => "logs"
    data_stream_dataset => "app"
    data_stream_namespace => "default"
  }
}
EOF

sudo chown logstash:logstash /etc/logstash/logstash.keystore
sudo chmod 0600 /etc/logstash/logstash.keystore

sudo -u logstash /usr/share/logstash/bin/logstash   --path.settings /etc/logstash --config.test_and_exit
sudo systemctl enable --now logstash

Configure Filebeat filestream Input and Output

sudo tee /etc/filebeat/filebeat.yml >/dev/null <<'EOF'
filebeat.inputs:
  - type: filestream
    id: system-messages
    enabled: true
    paths:
      - /var/log/messages

output.elasticsearch:
  enabled: false

output.logstash:
  hosts: ["127.0.0.1:5044"]

logging.level: info
EOF

sudo filebeat test config -e
sudo filebeat test output -e
sudo systemctl enable --now filebeat
sudo journalctl -u filebeat --since '-10 min' --no-pager

Confirm that Filebeat can read /var/log/messages. Review the package's service account and permission model instead of automatically running it as root. Drop or redact sensitive fields, personal information, and tokens before collection. Track parsing failures through a separate tag and dashboard.

Expand to a Production Three-Node Cluster

Place three master-eligible nodes in separate failure domains for production. Two nodes cannot retain a majority after either node fails. Join new nodes with enrollment tokens, then clean up discovery settings once the cluster has formed.

Enroll Nodes and Check the Cluster

You cannot add nodes while retaining the earlier loopback-only, single-node configuration. On the existing node, remove discovery.type: single-node and set transport.host to a dedicated network address reachable by the other nodes. Enrollment also needs the existing node's HTTPS API, so configure http.host to include that private address and loopback, and verify certificate SANs. Allow port 9200 only from enrollment management hosts and port 9300 only between cluster nodes. Preserve TLS settings.

cluster.name: logs-lab
node.name: es01
network.host: 10.20.30.11
http.host: ["127.0.0.1", "10.20.30.11"]
transport.host: 10.20.30.11
discovery.seed_hosts: ["10.20.30.11:9300", "10.20.30.12:9300", "10.20.30.13:9300"]

Replace these addresses with the actual three-node addresses and edit the existing YAML keys. Keep cluster.initial_master_nodes removed from the existing cluster. After bootstrap checks and the service restart succeed, generate a separate token for each new node. Before their first startup, verify each new node's cluster.name, node.name, addresses, and seed hosts.

sudo systemctl restart elasticsearch
sudo systemctl status elasticsearch --no-pager
sudo ss -lntp | grep -E ":(9200|9300)\b"
# Run on an existing cluster node
sudo /usr/share/elasticsearch/bin/elasticsearch-create-enrollment-token   -s node

# Run on the new node immediately after installing the package
sudo /usr/share/elasticsearch/bin/elasticsearch-reconfigure-node   --enrollment-token '<ONE_TIME_NODE_TOKEN>'
sudo systemctl enable --now elasticsearch

# Check nodes and roles from the existing cluster
curl --cacert /etc/elastic-client/http_ca.crt   -u "elastic:${ELASTIC_PASSWORD}"   'https://127.0.0.1:9200/_cat/nodes?v&h=name,ip,node.role,master,heap.percent,disk.avail'

cluster.initial_master_nodes is only for the first election of a new cluster. Remove it from every node once the cluster has formed, and do not set it again. Leaving it in place during restarts risks accidentally bootstrapping a separate cluster. Populate discovery.seed_hosts with master-eligible nodes discoverable after restart.

Snapshots and Recovery Drills

A replica protects against node failure; it is not a backup. An accidental document deletion is immediately reflected in replicas. A complete Elastic Stack deployment needs an off-cluster snapshot repository, automated SLM policies, and actual recovery tests.

Register a Shared Filesystem Repository

Mount the same real shared filesystem at the same path on every master/data node, and verify read/write access for the Elasticsearch service account. Separate local directories with identical names do not form shared storage.

# On every master/data node, set the existing path.repo key in elasticsearch.yml
# to path.repo: ["/mnt/elastic-backup"].

# Register the repository after a rolling restart.
curl --fail --silent --show-error   --cacert /etc/elastic-client/http_ca.crt   -u "elastic:${ELASTIC_PASSWORD}"   -X PUT https://127.0.0.1:9200/_snapshot/prod_backup   -H 'Content-Type: application/json' -d @- <<'JSON'
{
  "type": "fs",
  "settings": {
    "location": "/mnt/elastic-backup",
    "compress": true
  }
}
JSON

Verify the Repository and Create a Test Snapshot

curl --fail --silent --show-error   --cacert /etc/elastic-client/http_ca.crt   -u "elastic:${ELASTIC_PASSWORD}"   -X POST 'https://127.0.0.1:9200/_snapshot/prod_backup/_verify?pretty'

curl --fail --silent --show-error   --cacert /etc/elastic-client/http_ca.crt   -u "elastic:${ELASTIC_PASSWORD}"   -X PUT 'https://127.0.0.1:9200/_snapshot/prod_backup/smoke-test?wait_for_completion=true&pretty'

curl --fail --silent --show-error   --cacert /etc/elastic-client/http_ca.crt   -u "elastic:${ELASTIC_PASSWORD}"   'https://127.0.0.1:9200/_snapshot/prod_backup/smoke-test?pretty'

Copying /var/lib/elasticsearch with rsync or filesystem snapshots is not a supported backup method. External processes modifying repository contents can also corrupt them. Regularly restore to a separate test cluster and verify indices, feature states, and dashboards.

Validate and Troubleshoot Elastic Stack

Check Services, TLS, and Cluster Health Together

systemctl --no-pager --full status   elasticsearch kibana logstash filebeat

sudo ss -lntp | grep -E ':(9200|9300|5601|5044)\b'

curl --fail --silent --show-error   --cacert /etc/elastic-client/http_ca.crt   -u "elastic:${ELASTIC_PASSWORD}"   'https://127.0.0.1:9200/_cluster/health?pretty'

curl --fail --silent --show-error   --cacert /etc/elastic-client/http_ca.crt   -u "elastic:${ELASTIC_PASSWORD}"   'https://127.0.0.1:9200/_cat/shards?v&s=state,index'

unset ELASTIC_PASSWORD

Diagnostic Sequence

  1. Find the first error timestamp in systemd exit codes and recent journal entries.
  2. Confirm that ports listen only on their intended addresses.
  3. Check CA trust, hostnames, certificate validity periods, and client trust configuration.
  4. Inspect unassigned shard reasons and disk watermarks alongside cluster health.
  5. Check Logstash configuration validation, persistent queues, rejected events, and Filebeat output tests.
  6. Verify retention and backup automation using ILM explain and snapshot repository verification.
  7. Reproduce the issue with the same input and configuration in an isolated environment, then record any changes.

Common Mistakes and Alternatives

Mistake Problem Safer alternative
Setting security features to false Removes authentication, authorization, and TLS protection Keep automatic security configuration and distribute the CA
Skipping curl certificate verification Cannot detect an impersonated server certificate Verify with --cacert or a fingerprint
Ingesting as elastic Excessive superuser privileges Use per-service roles/users or API keys
Exposing 9200 and 5601 broadly Directly exposes management APIs and UI Use loopback, a management network, and a reverse proxy
One index without bounds Oversized shards and long recovery Use data streams, rollover, and ILM
Copying the data directory Inconsistent backups that may not be recoverable Use a snapshot repository and restore tests
Two master-eligible nodes Loses majority agreement after a failure Use three master-eligible nodes

Upgrade and Operations Checklist

  • Align target component versions and review release notes and breaking changes.
  • Before upgrading, check cluster health, unassigned shards, disk watermarks, and successful snapshots.
  • Upgrade one node at a time, waiting at each step for it to rejoin and for shard recovery.
  • Validate compatibility of Kibana saved objects, Logstash plugins, Beats modules, and custom templates.
  • Regularly review certificate expiration, keystore entries, API key rotation, and least privilege.
  • Alert on ingestion delay, JVM pressure, rejected requests, shard sizes, and ILM/SLM failures.

Related Resources

Conclusion

A successful Elastic Stack deployment is an operational system that collects data securely, retains it at predictable sizes, and recovers after failures. Start with matching official package versions and retain automatic TLS and authentication. Then verify least privilege, loopback and firewall boundaries, data streams and ILM, and off-cluster snapshots in order. This provides a path from a single-node lab to an operational three-node cluster.