<?xml version="1.0" encoding="UTF-8"?>
<feed xmlns="http://www.w3.org/2005/Atom" xml:lang="en">
    <title>keelinfra - Blog</title>
    <subtitle>Production-grade self-hosted Keycloak distribution: HA cluster, Postgres auto-failover, backups with PITR, monitoring, and upgrade paths re-proven nightly in public CI. Apache-2.0.</subtitle>
    <link rel="self" type="application/atom+xml" href="https://keelinfra.io/blog/atom.xml"/>
    <link rel="alternate" type="text/html" href="https://keelinfra.io/blog/"/>
    <generator uri="https://www.getzola.org/">Zola</generator>
    <updated>2026-09-01T00:00:00+00:00</updated>
    <id>https://keelinfra.io/blog/atom.xml</id>
    <entry xml:lang="en">
        <title>Keycloak 26.7.3 fixes 20 CVEs. Here&#39;s which ones can actually reach you.</title>
        <published>2026-09-01T00:00:00+00:00</published>
        <updated>2026-09-01T00:00:00+00:00</updated>
        
        <author>
          <name>Unknown</name>
        </author>
        
        <link rel="alternate" type="text/html" href="https://keelinfra.io/blog/keycloak-26-7-3-cve-triage/"/>
        <id>https://keelinfra.io/blog/keycloak-26-7-3-cve-triage/</id>
        
        <content type="html" xml:base="https://keelinfra.io/blog/keycloak-26-7-3-cve-triage/">&lt;p&gt;&quot;20 CVEs fixed&quot; is the kind of release note that gets forwarded to a CISO with no
further context. Most of the time, three of those twenty can reach your cluster
and the rest are gated behind features you never turned on. Nobody tells you
which three. We did the sorting for
&lt;a rel=&quot;external&quot; href=&quot;https://github.com/keycloak/keycloak/releases/tag/26.7.3&quot;&gt;26.7.3&lt;/a&gt; — released
2026-08-31 — and we&#39;re publishing the method, not just the answer, so you can
redo it on the next release yourself.&lt;/p&gt;
&lt;p&gt;There&#39;s a second problem 26.7.3 fixes that the CVE count doesn&#39;t cover at all:
six functional regressions introduced inside the 26.7 stream itself, including
one that pins CPU high on every node after upgrade. If you&#39;re running
26.7.0–26.7.2, that alone is worth reading past the CVE table for.&lt;/p&gt;
&lt;h2 id=&quot;the-three-questions-cheapest-first&quot;&gt;The three questions, cheapest first&lt;/h2&gt;
&lt;p&gt;Before treating any CVE in a batch as urgent, ask in this order:&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;&lt;strong&gt;Is your version before the fix?&lt;/strong&gt; On the fixed release or later — stop here.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Is the feature it touches turned on?&lt;/strong&gt; Most Keycloak CVEs live inside a
subsystem — fine-grained admin permissions, organizations, token exchange,
SAML brokering, LDAP federation, dynamic client registration. A cluster that
never enabled it cannot be reached through it.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Who can reach the surface?&lt;/strong&gt; An admin-API flaw exploitable by a delegated
admin is a different risk with three admins than with three hundred realm
managers. An unauthenticated endpoint doesn&#39;t care how many admins you have.&lt;/li&gt;
&lt;/ol&gt;
&lt;p&gt;Full method and the reasoning behind it: &lt;a rel=&quot;external&quot; href=&quot;https://github.com/keelinfra/keycloak/blob/main/CVE-POLICY.md&quot;&gt;CVE-POLICY.md&lt;/a&gt;.&lt;/p&gt;
&lt;h2 id=&quot;26-7-3-sorted&quot;&gt;26.7.3, sorted&lt;/h2&gt;
&lt;p&gt;Twenty CVEs, grouped by what has to be true for each one to matter
(&lt;a rel=&quot;external&quot; href=&quot;https://www.keycloak.org/2026/08/keycloak-2673-released&quot;&gt;upstream release notes&lt;/a&gt;):&lt;/p&gt;
&lt;table&gt;&lt;thead&gt;&lt;tr&gt;&lt;th&gt;Only relevant if you run&lt;/th&gt;&lt;th&gt;CVEs&lt;/th&gt;&lt;/tr&gt;&lt;/thead&gt;&lt;tbody&gt;
&lt;tr&gt;&lt;td&gt;Fine-grained admin permissions (FGAP v2)&lt;/td&gt;&lt;td&gt;2026-16108, 16104, 16106, 17059, 18571&lt;/td&gt;&lt;/tr&gt;
&lt;tr&gt;&lt;td&gt;Delegated admin over roles (any admin API role management)&lt;/td&gt;&lt;td&gt;2026-16105&lt;/td&gt;&lt;/tr&gt;
&lt;tr&gt;&lt;td&gt;Organizations&lt;/td&gt;&lt;td&gt;2026-16072, 18201&lt;/td&gt;&lt;/tr&gt;
&lt;tr&gt;&lt;td&gt;Token exchange with Microsoft or Google IdPs&lt;/td&gt;&lt;td&gt;2026-18215, 18214&lt;/td&gt;&lt;/tr&gt;
&lt;tr&gt;&lt;td&gt;UMA / authorization services&lt;/td&gt;&lt;td&gt;2026-18572&lt;/td&gt;&lt;/tr&gt;
&lt;tr&gt;&lt;td&gt;LDAP user federation&lt;/td&gt;&lt;td&gt;2026-35563&lt;/td&gt;&lt;/tr&gt;
&lt;tr&gt;&lt;td&gt;&lt;strong&gt;Any OIDC deployment&lt;/strong&gt;&lt;/td&gt;&lt;td&gt;2026-16093, 16089, 18218, 18209, 18573, 18570, 79652&lt;/td&gt;&lt;/tr&gt;
&lt;tr&gt;&lt;td&gt;&lt;strong&gt;Any deployment&lt;/strong&gt;&lt;/td&gt;&lt;td&gt;2026-19729&lt;/td&gt;&lt;/tr&gt;
&lt;/tbody&gt;&lt;/table&gt;
&lt;p&gt;Two rows are worth reading twice. &lt;strong&gt;2026-16105&lt;/strong&gt; sits under upstream&#39;s
&lt;code&gt;admin/rbac&lt;/code&gt; label, not fine-grained permissions — a quick skim of &quot;these are
all FGAP issues&quot; skips it, and it reaches any deployment that delegates role
management through the admin API, FGAP on or off. And the &quot;any OIDC
deployment&quot; row is seven flaws, not one — including an authorization code
retargetable to another client session and an incomplete fix for
&lt;code&gt;redirect_uri&lt;/code&gt; parameter injection. Nobody who runs OIDC (which is nearly
everyone) walks away from that row clean.&lt;/p&gt;
&lt;p&gt;Add the one &quot;any deployment&quot; path-traversal flaw and you have eight CVEs that
apply regardless of what you&#39;ve configured. If you&#39;re on 26.7.0–26.7.2, those
eight alone are the upgrade argument, before you even look at the gated rows.&lt;/p&gt;
&lt;h2 id=&quot;the-other-reason-not-to-sit-on-26-7-0-26-7-2&quot;&gt;The other reason not to sit on 26.7.0-26.7.2&lt;/h2&gt;
&lt;p&gt;26.7.3 also repairs six regressions that shipped inside the 26.7 stream — none
of them CVEs, all of them the kind of thing that pages someone at 3am:&lt;/p&gt;
&lt;table&gt;&lt;thead&gt;&lt;tr&gt;&lt;th&gt;Upstream&lt;/th&gt;&lt;th&gt;What breaks on 26.7.0–26.7.2&lt;/th&gt;&lt;/tr&gt;&lt;/thead&gt;&lt;tbody&gt;
&lt;tr&gt;&lt;td&gt;&lt;a rel=&quot;external&quot; href=&quot;https://github.com/keycloak/keycloak/issues/51523&quot;&gt;#51523&lt;/a&gt;&lt;/td&gt;&lt;td&gt;Sustained high CPU on every node after upgrading&lt;/td&gt;&lt;/tr&gt;
&lt;tr&gt;&lt;td&gt;&lt;a rel=&quot;external&quot; href=&quot;https://github.com/keycloak/keycloak/issues/51554&quot;&gt;#51554&lt;/a&gt;&lt;/td&gt;&lt;td&gt;Admin API per-request cost grows super-linearly with realm count (since 26.7.1)&lt;/td&gt;&lt;/tr&gt;
&lt;tr&gt;&lt;td&gt;&lt;a rel=&quot;external&quot; href=&quot;https://github.com/keycloak/keycloak/issues/51707&quot;&gt;#51707&lt;/a&gt;&lt;/td&gt;&lt;td&gt;Lightweight access tokens resolve every role in every realm on each admin API request&lt;/td&gt;&lt;/tr&gt;
&lt;tr&gt;&lt;td&gt;&lt;a rel=&quot;external&quot; href=&quot;https://github.com/keycloak/keycloak/issues/51920&quot;&gt;#51920&lt;/a&gt;&lt;/td&gt;&lt;td&gt;&lt;code&gt;OFFLINE_CLIENT_SESSION&lt;/code&gt; write conflicts — &quot;Record has changed since last read&quot;&lt;/td&gt;&lt;/tr&gt;
&lt;tr&gt;&lt;td&gt;&lt;a rel=&quot;external&quot; href=&quot;https://github.com/keycloak/keycloak/issues/52038&quot;&gt;#52038&lt;/a&gt;&lt;/td&gt;&lt;td&gt;Client session note removals not persisted with persistent user sessions&lt;/td&gt;&lt;/tr&gt;
&lt;tr&gt;&lt;td&gt;&lt;a rel=&quot;external&quot; href=&quot;https://github.com/keycloak/keycloak/issues/51792&quot;&gt;#51792&lt;/a&gt;&lt;/td&gt;&lt;td&gt;Aurora detection logs an ERROR on every startup&lt;/td&gt;&lt;/tr&gt;
&lt;/tbody&gt;&lt;/table&gt;
&lt;p&gt;A CVE scanner won&#39;t flag any of these. We only found the full list because we
run every supported path in CI and read what actually happens, not just what
the release notes summarize.&lt;/p&gt;
&lt;h2 id=&quot;the-tested-way-off-it&quot;&gt;The tested way off it&lt;/h2&gt;
&lt;p&gt;We don&#39;t list an upgrade path we haven&#39;t run end-to-end — install, log in,
upgrade, assert the pre-upgrade session survives. Two paths land you on
26.7.3:&lt;/p&gt;
&lt;pre&gt;&lt;code data-lang=&quot;bash&quot;&gt;./upgrade --to 26.7.3   # from 26.6.2 (stop-start) or 26.7.0 (rolling)
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;Both are single-node CI-verified nightly as of 2026-08-31; the 3-node HA drill
hasn&#39;t run against this specific target yet, so if you&#39;re on HA, drill it in
staging first. Full matrix, updated as paths get verified:
&lt;a rel=&quot;external&quot; href=&quot;https://github.com/keelinfra/keycloak/blob/main/UPGRADES.md&quot;&gt;UPGRADES.md&lt;/a&gt;.&lt;/p&gt;
&lt;h2 id=&quot;what-we-will-and-won-t-do-about-a-cve&quot;&gt;What we will and won&#39;t do about a CVE&lt;/h2&gt;
&lt;p&gt;We track upstream, do this triage for every release, and point you at a
drilled upgrade path. We do not backport patches to old branches — a real
backport program means rewriting a 26.7 fix for a 26.2 codebase and owning the
regression risk of that rewrite, release after release, which is a different
business than the one we&#39;re running. When upstream tags a maintenance release
with no community artifact, we build the unmodified tag and publish it — see
the &lt;a rel=&quot;external&quot; href=&quot;https://github.com/keelinfra/keycloak/blob/main/UPGRADES.md#keelinfra-lts-builds&quot;&gt;LTS builds&lt;/a&gt;
— but that&#39;s compiling, not patching. Full boundary, including what a
subscription adds on top: &lt;a rel=&quot;external&quot; href=&quot;https://github.com/keelinfra/keycloak/blob/main/CVE-POLICY.md&quot;&gt;CVE-POLICY.md&lt;/a&gt;.&lt;/p&gt;
&lt;h2 id=&quot;run-the-triage-yourself&quot;&gt;Run the triage yourself&lt;/h2&gt;
&lt;pre&gt;&lt;code data-lang=&quot;bash&quot;&gt;git clone https://github.com/keelinfra/keycloak &amp;amp;&amp;amp; cd keycloak
./configure -c examples/single-node.yml
./install
./upgrade --to 26.7.3
./verify --drill session   # pre-upgrade session must still refresh
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;Then read the &lt;a rel=&quot;external&quot; href=&quot;https://www.keycloak.org/2026/08/keycloak-2673-released&quot;&gt;26.7.3 release notes&lt;/a&gt;
against the three questions above for your own configuration — the table in
this post is a starting point, not a substitute for reading the advisory when
a specific CVE is load-bearing for a decision you&#39;re making.&lt;/p&gt;
&lt;hr /&gt;
&lt;p&gt;&lt;em&gt;keelinfra is an open-source (Apache-2.0), production-ready self-hosted
Keycloak distribution. A subscription adds CVE notification matched to the
version you actually run, and upgrade help along a path we&#39;ve already drilled:
&lt;a rel=&quot;external&quot; href=&quot;https://keelinfra.io/pricing&quot;&gt;keelinfra.io/pricing&lt;/a&gt; · hello@keelinfra.io.&lt;/em&gt;&lt;/p&gt;
&lt;p&gt;&lt;em&gt;Keycloak is a trademark of The Linux Foundation. keelinfra is an independent
project, not affiliated with or endorsed by The Linux Foundation or the
Keycloak project.&lt;/em&gt;&lt;/p&gt;
</content>
        
    </entry>
    <entry xml:lang="en">
        <title>Zero-downtime Keycloak patch upgrades, measured: 156/156 probes</title>
        <published>2026-08-26T00:00:00+00:00</published>
        <updated>2026-08-26T00:00:00+00:00</updated>
        
        <author>
          <name>Unknown</name>
        </author>
        
        <link rel="alternate" type="text/html" href="https://keelinfra.io/blog/zero-downtime-keycloak-upgrades/"/>
        <id>https://keelinfra.io/blog/zero-downtime-keycloak-upgrades/</id>
        
        <content type="html" xml:base="https://keelinfra.io/blog/zero-downtime-keycloak-upgrades/">&lt;p&gt;Everyone claims zero-downtime upgrades. Almost nobody publishes the probe log.&lt;/p&gt;
&lt;p&gt;We upgraded a 3-node HA Keycloak cluster from 26.6.0 to 26.6.2 while hitting it with
a request every second. &lt;strong&gt;156 probes, 156 × HTTP 200, zero failures&lt;/strong&gt; — and a session
opened &lt;em&gt;before&lt;/em&gt; the upgrade was still valid &lt;em&gt;after&lt;/em&gt; every node had been replaced.&lt;/p&gt;
&lt;p&gt;Then we did a minor upgrade (26.6.2 → 26.7.0), measured a 65-second outage window,
got annoyed, re-ordered the orchestration, and cut it to &lt;strong&gt;16 seconds&lt;/strong&gt;. Sessions
survived that too.&lt;/p&gt;
&lt;p&gt;This post is the numbers, the orchestration, and the three mistakes we made on the
way. Everything here is reproducible from the open-source repo —
&lt;a rel=&quot;external&quot; href=&quot;https://github.com/keelinfra/keycloak&quot;&gt;keelinfra/keycloak&lt;/a&gt; — with one command.&lt;/p&gt;
&lt;h2 id=&quot;why-keycloak-upgrades-scare-people&quot;&gt;Why Keycloak upgrades scare people&lt;/h2&gt;
&lt;p&gt;Two reasons, historically:&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;&lt;strong&gt;Sessions lived in Infinispan.&lt;/strong&gt; Restart the cluster wrong and every user is
logged out. Nothing pages a CISO faster than &quot;SSO logged out the whole company.&quot;&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;No rolling updates.&lt;/strong&gt; Until recently, mixed-version clusters were unsupported,
full stop.&lt;/li&gt;
&lt;/ol&gt;
&lt;p&gt;Keycloak 26 changed the physics of both. &lt;a rel=&quot;external&quot; href=&quot;https://www.keycloak.org/docs/latest/release_notes/&quot;&gt;Persistent user
sessions&lt;/a&gt; put sessions in
PostgreSQL by default — a node (or the whole cluster) can restart without logging
anyone out. And since 26.6, &lt;a rel=&quot;external&quot; href=&quot;https://www.keycloak.org/server/update-compatibility&quot;&gt;rolling updates for patch
releases&lt;/a&gt; are GA: 26.6.0 →
26.6.2 can be done node by node with zero downtime.&lt;/p&gt;
&lt;p&gt;Upstream gives you the &lt;em&gt;capability&lt;/em&gt;. What it doesn&#39;t give you is the
&lt;em&gt;orchestration&lt;/em&gt; — the drain/replace/verify choreography — or any proof that your
particular path works. That&#39;s the part we automated and measured.&lt;/p&gt;
&lt;h2 id=&quot;the-setup&quot;&gt;The setup&lt;/h2&gt;
&lt;p&gt;Three clean VMs (this run: Multipass on Apple Silicon; same playbooks run on any
Ubuntu 24.04 hosts). On each node:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;Keycloak 26.6.0 — official tarball under systemd, &lt;code&gt;jdbc-ping&lt;/code&gt; discovery,
DB-persisted sessions (26.x defaults)&lt;/li&gt;
&lt;li&gt;PostgreSQL 17 with Patroni (etcd DCS), automatic failover&lt;/li&gt;
&lt;li&gt;HAProxy — port 443 load-balances all Keycloak nodes; 5433 always routes to the
Patroni leader&lt;/li&gt;
&lt;li&gt;pgBackRest — encrypted backups + WAL archiving&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;The whole stack comes up with &lt;code&gt;./configure &amp;amp;&amp;amp; ./install&lt;/code&gt; in about 12 minutes on this
laptop-hosted Multipass rig (CI installs on clean VMs in ~10 — timings are public in the
&lt;a rel=&quot;external&quot; href=&quot;https://github.com/keelinfra/keycloak/actions/workflows/smoke.yml&quot;&gt;smoke workflow&lt;/a&gt;).&lt;/p&gt;
&lt;h2 id=&quot;the-patch-upgrade-26-6-0-26-6-2-rolling&quot;&gt;The patch upgrade: 26.6.0 → 26.6.2, rolling&lt;/h2&gt;
&lt;p&gt;The orchestration per node (Ansible, &lt;code&gt;serial: 1&lt;/code&gt;):&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;1. STAGE      download + unpack 26.6.2, render config, kc.sh build
              → the running service is untouched during the slow part
2. DRAIN      disable the node on every HAProxy
              (runtime API: &amp;quot;disable server keycloak_https/&amp;lt;node&amp;gt;&amp;quot;)
3. WAIT 5s    let in-flight requests finish
4. CUT OVER   /opt/keycloak/current -&amp;gt; keycloak-26.6.2   (symlink flip)
5. RESTART    systemd restart, wait for /health/ready = 200
6. RETURN     re-enable the node on every HAProxy
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;The measurement: a probe loop on the operator machine, one request per second
against the public entrypoint, straight through the upgrade:&lt;/p&gt;
&lt;pre&gt;&lt;code data-lang=&quot;bash&quot;&gt;while :; do
  curl -sk -o /dev/null -w &amp;quot;$(date +%T) %{http_code}\n&amp;quot; \
    --max-time 3 https://sso.example/realms/demo/.well-known/openid-configuration
  sleep 1
done
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;Result:&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;$ awk &amp;#39;{print $2}&amp;#39; uptime_probe.log | sort | uniq -c
    156 200
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;Every probe during the entire rolling upgrade returned 200. No timeouts, no 5xx,
no connection resets.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Session survival&lt;/strong&gt;, verified the honest way: log in &lt;em&gt;before&lt;/em&gt; the upgrade, keep
the &lt;code&gt;refresh_token&lt;/code&gt;, upgrade all three nodes, then redeem the token:&lt;/p&gt;
&lt;pre&gt;&lt;code data-lang=&quot;bash&quot;&gt;# before: password grant against the demo realm → save refresh_token
# after:
curl -sk https://sso.example/realms/demo/protocol/openid-connect/token \
  -d &amp;quot;grant_type=refresh_token&amp;amp;client_id=admin-cli&amp;amp;refresh_token=$OLD_TOKEN&amp;quot;
# → 200, fresh access_token. The session outlived every node it was born on.
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;That&#39;s the DB-persisted sessions doing their job: the session state lives in
PostgreSQL, so it doesn&#39;t care which Keycloak process serves the request.&lt;/p&gt;
&lt;h2 id=&quot;the-minor-upgrade-26-6-2-26-7-0-and-a-4x-improvement-we-owe-you-honesty-about&quot;&gt;The minor upgrade: 26.6.2 → 26.7.0, and a 4× improvement we owe you honesty about&lt;/h2&gt;
&lt;p&gt;Minor upgrades may run schema migrations, so upstream&#39;s supported procedure is
stop → migrate → start, not rolling. Sessions still survive — they&#39;re in the
database — but there is a real service window. The only question is how long.&lt;/p&gt;
&lt;p&gt;Our first implementation was naive:&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;stop all nodes  →  download 26.7.0  →  unpack  →  build  →  start
                   ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
                   all of this inside the outage window
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;Measured: &lt;strong&gt;65 seconds of downtime.&lt;/strong&gt; Most of it was downloading a 250 MB tarball
three times — with the service already stopped. Embarrassing, easy to fix:&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;STAGE on all nodes (download + unpack + build — service still running)
  → stop all
  → flip symlinks
  → start node 1 (runs DB migrations)
  → start nodes 2..n
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;Measured after re-ordering: &lt;strong&gt;16 seconds.&lt;/strong&gt;&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;     57 200      ← before
     16 000     ← the window (connection refused)
     28 200      ← after; pre-upgrade session still refreshes fine
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;The lesson generalizes: &lt;em&gt;never put artifact acquisition inside a service window.&lt;/em&gt;
Obvious in retrospect. We publish the number anyway, because &quot;measured 16s
stop-start window&quot; is a claim you can hold us to, and &quot;zero downtime&quot; for minor
upgrades would be a lie — if a vendor tells you otherwise, ask for their probe log.&lt;/p&gt;
&lt;h2 id=&quot;three-mistakes-worth-stealing-from&quot;&gt;Three mistakes worth stealing from&lt;/h2&gt;
&lt;p&gt;&lt;strong&gt;1. The official tarball ships with a default build — and it breaks naive
idempotence.&lt;/strong&gt; Keycloak&#39;s tarball comes pre-built for dev-mode h2. If your
automation checks &quot;does the optimized build output exist?&quot; the answer is &lt;em&gt;always
yes&lt;/em&gt;, your &lt;code&gt;kc.sh build&lt;/code&gt; silently never runs, and the server dies on startup with
&lt;code&gt;--optimized&lt;/code&gt;. We now stamp builds with a checksum of the effective config and
rebuild when it drifts.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;2. systemd quietly ate our heap flag.&lt;/strong&gt; &lt;code&gt;Environment=JAVA_OPTS_KC_HEAP=-Xms512m -Xmx1024m&lt;/code&gt; — without quotes, systemd parses &lt;code&gt;-Xmx1024m&lt;/code&gt; as a second (invalid)
assignment and drops it. The service runs fine with an uncapped heap… until the
day it doesn&#39;t. &lt;code&gt;journalctl&lt;/code&gt; had been warning about it the whole time:
&lt;code&gt;Invalid environment assignment, ignoring: -Xmx1024m&lt;/code&gt;. Read your warnings.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;3. Drain on &lt;em&gt;every&lt;/em&gt; load balancer, not just the local one.&lt;/strong&gt; Each node runs its
own HAProxy, and each HAProxy backends &lt;em&gt;all&lt;/em&gt; Keycloak nodes. Draining a node only
on its local proxy still leaves it taking traffic from the other two. The drain
loop talks to every proxy&#39;s runtime socket before touching the node.&lt;/p&gt;
&lt;h2 id=&quot;reproduce-it-yourself&quot;&gt;Reproduce it yourself&lt;/h2&gt;
&lt;pre&gt;&lt;code data-lang=&quot;bash&quot;&gt;git clone https://github.com/keelinfra/keycloak &amp;amp;&amp;amp; cd keycloak
./configure -c examples/ha-3node.yml   # 3 clean Ubuntu 24.04 hosts + SSH
./install                              # ≈12 min: HA KC + PG HA + backups + monitoring
./verify --drill session               # login → rolling restart → session must survive
./upgrade --to 26.6.2                  # the rolling upgrade from this post
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;Every upgrade path we support is listed in
&lt;a rel=&quot;external&quot; href=&quot;https://github.com/keelinfra/keycloak/blob/main/UPGRADES.md&quot;&gt;UPGRADES.md&lt;/a&gt; — and a
path only gets listed after it has been executed end-to-end, sessions verified.
That CI matrix now runs nightly:
&lt;a rel=&quot;external&quot; href=&quot;https://github.com/keelinfra/keycloak/actions/workflows/upgrade-matrix.yml&quot;&gt;the upgrade matrix&lt;/a&gt;
re-proves every supported path in public — install the old version, log in, upgrade,
and the pre-upgrade session must still refresh.&lt;/p&gt;
&lt;hr /&gt;
&lt;p&gt;&lt;em&gt;keelinfra is an open-source (Apache-2.0), production-ready self-hosted Keycloak
distribution. If you&#39;d rather have the people who wrote this run your deployment,
migration, or a very bad Keycloak day: &lt;a rel=&quot;external&quot; href=&quot;https://keelinfra.io&quot;&gt;keelinfra.io&lt;/a&gt; ·
hello@keelinfra.io.&lt;/em&gt;&lt;/p&gt;
&lt;p&gt;&lt;em&gt;Keycloak is a trademark of The Linux Foundation. keelinfra is an independent
project, not affiliated with or endorsed by The Linux Foundation or the
Keycloak project.&lt;/em&gt;&lt;/p&gt;
</content>
        
    </entry>
</feed>
