Deployment guide

How to put a keelinfra Keycloak cluster into production: what runs where, what the network must allow, how backups leave the cluster, where alerts go, and how to prove each piece works before users depend on it. It covers the 3-node HA layout; a single node runs the same components without the redundancy.

Everything below is what the roles in this repository do. Where something is not tested, the guide says so.

What you are deploying

Three identical nodes. Each one runs:

The first node also runs the shared services:

The optional off-site copy (pgBackRest repository 2) lives in an S3-compatible bucket you provide.

What survives what, on a 3-node cluster:

• One node down: logins keep working through the other two nodes' HAProxy. If it was the PostgreSQL leader, Patroni promotes a replica. ./verify --drill failover and ./verify --drill session exercise both, and CI runs them on three nodes.
• Two nodes down: etcd loses quorum and PostgreSQL stops accepting writes. Keycloak cannot serve logins. Three nodes tolerate one failure, not two.
• The first node down: the cluster keeps serving, but backups, Prometheus, Alertmanager and Grafana are down with it until it returns. If its disk is lost, so is repository 1. This is why production clusters should set backup_s3.

Before you start

Nodes. Three machines running Ubuntu 24.04 (the only OS tested in CI; the roles use apt). For evaluation, 4 GB RAM and 2 vCPU per node is enough. The defaults size Keycloak's heap at 512 MB to 1 GB and PostgreSQL's shared buffers at 512 MB; for production, size to your login volume and watch the dashboard.

Control machine. Your laptop or a jump host with ansible-core, SSH access to every node as a user with passwordless sudo, and outbound internet (the nodes download Keycloak, PostgreSQL, Grafana and the rest from upstream during install).

DNS and certificate. A name for the service (say sso.example.com). Either let the install generate a private CA (tls_mode: selfsigned), or bring a certificate for that name (tls_mode: custom, see the configuration reference).

Network. Open these, and nothing more:

Prometheus (9090) and Grafana (3000) listen on all interfaces. Prometheus has no login, so keep both reachable only from your operators' network.

Traffic between the nodes is not encrypted yet. etcd, PostgreSQL replication, the Patroni API and Keycloak's cluster traffic travel in plaintext between the nodes. Put the three nodes on a private network you trust. Encrypting this traffic is on the roadmap.

1. Describe the cluster

Copy examples/ha-3node.yml and edit it. It is the only file you edit. The production-relevant blocks:

cluster_name: keycloak-prod
nodes:
  - {host: 10.0.0.11, name: kc-node1}
  - {host: 10.0.0.12, name: kc-node2}
  - {host: 10.0.0.13, name: kc-node3}
ssh_user: ubuntu
domain: sso.example.com
vip: ""                      # or a free address on the nodes' L2 segment
keycloak_version: "26.8.1"
tls_mode: custom
tls_cert_file: tls/sso.example.com.crt
tls_key_file: tls/sso.example.com.key

backup_s3:
  endpoint: s3.eu-central-1.amazonaws.com
  region: eu-central-1
  bucket: acme-keycloak-backups
  uri_style: host            # "path" for most self-hosted stores

alerting:
  webhook_url: https://hooks.example.com/keycloak
  email:
    to: oncall@example.com
    from: keycloak-alerts@example.com
    smarthost: smtp.example.com:587
    username: keycloak-alerts@example.com

Commit this file to your own infrastructure repository if you like: it holds no secrets.

2. Configure

export KEELINFRA_S3_ACCESS_KEY=...   # only on the first run with backup_s3
export KEELINFRA_S3_SECRET_KEY=...
export KEELINFRA_SMTP_PASSWORD=...   # only if alerting.email.username is set
./configure -c cluster.yml

./configure checks the definition, the certificate and SSH access to every node, then writes inventory/: the generated inventory, and an ansible-vault holding every password, the backup encryption passphrase, and the bucket and SMTP credentials. The vault key is inventory/.vault_pass.

Back up inventory/ somewhere safe and separate from the cluster. Backups in both repositories are encrypted with the passphrase in that vault. Without it, a backup cannot be restored.

3. Install

./install

The playbooks run in order: OS preparation, etcd, PostgreSQL and Patroni, HAProxy, pgBackRest (including a first full backup to every repository), TLS material, Keycloak, realm configuration, the VIP if set, and monitoring.

Every install also creates a demo realm (user demo, password demo12345) that the session drill uses. Change the password or delete the realm before go-live; the session drill needs it to run.

4. Prove it

./verify                     # health of every component, backups, monitoring
./verify --drill failover    # switch the PostgreSQL leader to a replica, write through it
./verify --drill restore     # restore the latest backup of every repository to a scratch dir
./verify --drill session     # rolling restart of every Keycloak node; sessions must survive
./verify --drill alert       # send a test alert; every configured receiver must accept it
./verify --drill all         # all of the above

./verify (health) fails if a component is down, if any backup repository has no backup, if any Prometheus target is not being scraped, if any metric an alert rule or the dashboard depends on has no data, or if Grafana is not serving the dashboard.

CI runs this whole sequence, install plus ./verify --drill all, on a three-node cluster with an S3 bucket and a webhook receiver whenever the roles change.

Backups

pgBackRest archives every WAL segment continuously and takes scheduled backups:

Both are encrypted (AES-256) with the passphrase in the vault. Times are the first node's local time. ./upgrade also takes a differential backup to repository 1 before it changes anything.

WAL goes to both repositories as it is written, so either one can restore to any point since its oldest kept backup. If the bucket is unreachable, archiving fails and PostgreSQL keeps the WAL on disk and retries; watch the root-filesystem alert during a long bucket outage.

What is tested: that each repository holds a backup and that the latest one restores into a scratch directory (./verify --drill restore), on every commit for one node and on three nodes when the roles change. What is not scripted yet: restoring over the live cluster after a total loss. That procedure (stop Patroni everywhere, restore on one node with pgbackrest --stanza=main --repo=N restore, re-bootstrap the replicas) is documented upstream by pgBackRest and Patroni; keelinfra does not automate or drill it today.

Monitoring and alerts

Prometheus scrapes every node every 15 seconds. The rules that page you:

Alertmanager sends each alert, and its resolution, to the webhook and/or email in alerting, grouped by alert and node, repeating every 4 hours while it fires. Without alerting, alerts are only visible in Grafana and Alertmanager. Run ./verify --drill alert after any change to the receivers.

Grafana is on http://<first node>:3000, user admin, password vault_grafana_admin_password in the vault. The keelinfra: Keycloak cluster dashboard shows nodes up, PostgreSQL leader and replica lag, hours since the last backup per repository, logins and failed logins per realm, HTTP traffic and errors, JVM heap, database pool use, and host CPU, memory and disk.

To read a vault value:

ANSIBLE_VAULT_PASSWORD_FILE=inventory/.vault_pass ansible-vault view inventory/group_vars/all/vault.yml

Day two

• Upgrades: ./upgrade --to <version>, only along a path listed in UPGRADES.md. Every listed path is drilled nightly on one and three nodes.
• Changing backup or alert settings after install: edit the definition, re-run ./configure (with the credential variables if they changed) and ./install. Re-running ./install on a live cluster to change these settings is not covered by CI yet; do it in a maintenance window and run ./verify --drill all afterwards.
• Certificates: with tls_mode: custom, replace the files, re-run ./configure and ./install.
• Security fixes upstream: see the CVE policy.

Known gaps

Stated plainly, so you can plan around them:

• Traffic between the nodes is not encrypted (see Network above).
• Ubuntu 24.04 only. The roles use apt; RHEL and Rocky are not supported.
• Monitoring and repository 1 live on the first node. Losing it stops backups and alerting until it returns; backup_s3 keeps the backups safe, nothing yet keeps alerting up.
• Prometheus and Grafana listen on all interfaces; restrict them with your firewall.
• No scripted restore over a live cluster (see Backups).
• Growing a single node into a 3-node cluster means a fresh install plus a realm import.
• Install needs outbound internet. An offline bundle is planned, not available.

Synced from keelinfra/keycloak/DEPLOYMENT.md @ 88206d8 — to change it, edit it there.