IGNITE-28867 Implement TLS certificate hot redeployment via control.sh --ssl reload - #13335
IGNITE-28867 Implement TLS certificate hot redeployment via control.sh --ssl reload#13335anton-vinogradov wants to merge 5 commits into
Conversation
2cc7619 to
edaee64
Compare
|
/runall 🚀 RunAll queued — build 9200721. The verdict lands here when the run finishes. |
|
Ignite PR Checker verdict · RunAll build 9200721 · 147 suites ran, 0 reused
✅ No test blockers otherwise; 21 pre-existing/flaky filtered out. ♻️ Settled after 3 auto re-run wave(s): #1 — 2 blocker suite(s); #2 — 3 broken suite(s); #3 — 1 broken suite(s). |
edaee64 to
d39d081
Compare
|
Ignite PR Checker verdict · RunAll build 9205392 · 147 suites ran, 0 reused ✅ No blockers — nothing in this run looks caused by this PR. 31 pre-existing/flaky tests filtered out. |
|
Ignite PR Checker verdict · RunAll build 9206015 · 17 suites ran, 130 reused ✅ No blockers — nothing in this run looks caused by this PR. 10 pre-existing/flaky tests filtered out. |
d39d081 to
f4bb8e7
Compare
|
Ignite PR Checker verdict · RunAll build 9215889 · 147 suites ran, 0 reused 🔍 1 suite(s) ran fewer tests than on master (tests that never ran can't fail):
✅ No test blockers otherwise; 26 pre-existing/flaky filtered out. ♻️ Settled after 1 auto re-run wave(s): #1 — 1 broken suite(s). |
1b83933 to
16b6dfa
Compare
|
/runall 🚀 RunAll queued — build 9229101. The verdict lands here when the run finishes. 🚀 RunAll queued — build 9232130 · live progress & verdict: Ignite PR Checker. The verdict lands here when the run finishes. |
|
Ignite PR Checker verdict · RunAll build 9232130 · 16 suites ran, 131 reused ✅ No blockers — nothing in this run looks caused by this PR. 14 pre-existing/flaky tests filtered out. |
16b6dfa to
ac2a1b5
Compare
|
/run-all 🚀 RunAll queued — build 9235255 · live progress & verdict: Ignite PR Checker. The verdict lands here when the run finishes. |
|
Ignite PR Checker verdict · RunAll build 9235255 · 147 suites ran, 0 reused ✅ No blockers — nothing in this run looks caused by this PR. 24 pre-existing/flaky tests filtered out. |
…deployment
Reload TLS certificates on running nodes without a restart:
control.sh --ssl reload re-reads the configured key and trust stores and
puts them in use
control.sh --ssl ensure reports whether they can be put in use, without
changing anything
Covers the SSL-enabled transports of a server node: communication, discovery,
client connector and binary (TCP) REST. Connections opened after a reload use
the updated certificates, established sessions are not interrupted.
Discovery now binds a plain server socket and secures every accepted connection
with the context current at that moment, so reloaded certificates apply to
incoming connections as well.
Components register themselves in GridInternalSubscriptionProcessor once they
have set SSL up, so the commands do not reach into the transports.
Before a rebuilt context replaces the one in use on communication and discovery,
it has to complete a mutually authenticated TLS handshake in memory. A
certificate that the node's own trust store rejects therefore never reaches the
inter-node transports.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
ac2a1b5 to
c59d08a
Compare
…hearse it The reload now covers what connects to the cluster as well as the cluster itself: control.sh --ssl reload [--dry-run] --ssl ensure is gone: a dry run is a flag on the reload, so both forms take the same path and cannot drift apart. The command reaches client nodes too, and a node that leaves while it runs is reported as having left rather than failing the run. Every node answers with what it reloaded and the certificate it serves now, so a partial rotation is visible instead of silent. HTTP REST is reloaded through its Jetty connector when the configuration points at key store files. Jetty rebuilds in place and has no rollback of its own, so the stores are verified on a throw-away factory first, and the context in use is pinned back if the rebuild fails regardless. A thin client rebuilds its context from disk after a refused TLS handshake, so an application locked out by a rotation recovers once the right files are placed next to it. A connection that merely dropped does not count: that happens on every node restart, and rebuilding there would also pick up a key store that was staged but is not meant to be presented yet. For that to be possible at all, the reason a handshake failed now reaches whoever waits on it. It used to be replaced by a generic "connection closed", which a plain network drop produces just as well, so the real cause was lost to callers and to the log alike. The field it was taken from was shared by every session of a filter and never reset, which could attribute one session's failure to another. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Rotating a node certificate does not need a new key pair: the key the node already runs on stays, and only the certificate on it changes. The steps showed keytool -genkeypair and a signing request instead, which is both more work than the rotation needs and the least likely way a certificate is issued in a deployment that has a certificate authority to begin with. The import is also one command rather than two whenever the authority returns the whole chain, which is the usual case; a reply carrying the node certificate alone is the one that needs the authority imported first. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…ry refused
The names components register under are also what the reload command reports,
so they are what an operator reads and scripts against. They were string
literals in five places and in the tests; SslContextReloadable now declares
them.
Discovery does not go through GridNioSslFilter, so it never got the hint the
other transports got when a handshake is refused. Its own message guessed one
cause and, during a rotation, the wrong one: it blamed missing SSL on the
remote node, while the node in fact presents a certificate this one no longer
trusts. It now carries the reason the TLS engine reported, and the rule a
rotation has to follow. Discovery is the transport where this matters most: a
refusal there is what drops a node out of the ring.
Tests: a second rotation is now covered, because nothing checked that a
component compares against the context in use rather than the one it started
with. Two assertions were satisfied by the report line that says the opposite
of what the test wanted ("not reloaded" contains "reloaded"), and one accepted
any exception at all; all three are anchored now. The rest is removal: two
nested factory classes that a lambda expresses, a temporary file that was an
immutable copy of an immutable resource, a hardcoded Jetty port that the
protocol is free not to bind, and assertions that could not fail.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Possible compatibility issues. Please, check rolling upgrade casesThis PR modifies protected classes (with Order annotation). Affected files:
|
|
Ignite PR Checker verdict · RunAll build 9241877 · 147 suites ran, 0 reused ✅ No blockers — nothing in this run looks caused by this PR. 24 pre-existing/flaky tests filtered out. |
1 participant
Overview
Implements IGNITE-28867: hot redeployment of TLS certificates on running nodes, without a full restart.
TLS certificates were previously loaded once at node startup, so rotating them required restarting every node. This PR adds a new
control.sh --ssl reloadcommand that re-reads the configured key and trust stores from disk and replaces the activeSSLContext. New connections use the updated certificates while established sessions are not interrupted.Mechanism
AbstractSslContextFactory.reload()rebuilds the cachedSSLContextfrom the current factory configuration (re-reading the key/trust stores from disk). On failure the previously cached context is kept intact.GridNioSslFilter.sslCtxis nowvolatilewithupdateSslContext(...); new sessions build theirSSLEnginefrom the new context, existing sessions keep their engines.SslContextReloadableinterface (reloadSslContext()) is implemented by each SSL-enabled transport.Coverage per transport:
sslSockFactory/sslSrvSockFactoryrebuiltCommand
A new root
--sslcommand group with areloadsub-command. It broadcasts aVisorMultiNodeTaskto all server nodes; each node reloads its own certificates locally and reports which transports were reloaded. Modelled afterencryption change_master_key.Known limitation (documented)
For the discovery transport, new outgoing connections use the reloaded certificates immediately, but the already-bound listening server socket keeps serving the previously loaded certificate to incoming connections until it is recreated (e.g. when the node re-enters the ring), because
SSLServerSocketcaptures the SSL context at creation time. Recreating the discovery server socket at runtime is intentionally avoided as too disruptive for the ring.The Jetty HTTP REST protocol manages its own SSL and is out of scope (the default
GridRestProtocol.reloadSslContext()is a no-op).Tests
SslContextFactoryReloadTest— a real loopback TLS handshake confirms that after the key store file is replaced on disk,reload()serves a different certificate, whilecreate()keeps its caching semantics.SslContextReloadNodeTest— a two-node SSL cluster runs the actual--ssl reloadtask; asserts communication/discovery/client-connector are reloaded and the cluster keeps operating (established sessions are not interrupted); a non-SSL node reports nothing to reload.Both
control.sh --helpgolden files were updated. Strict-Pcheckstylepasses.Docs / Release Notes
docs/_docs/tools/control-script.adoc— new "TLS Certificate Hot Reload" section (Docs Required flag).🤖 Generated with Claude Code