fix(clustering): propagate system table set() cluster wide - #37286
Merged
danielsolis-dotcms merged 2 commits intoAug 29, 2026
Conversation
SystemTableImpl.set() notified only the local node while delete() also published a CLUSTER_WIDE_EVENT, so configuration written through POST/PUT /api/v1/system-table applied only to the node that handled the request. Subscribers latch the resolved value in memory, so every other node kept the stale value for the lifetime of its JVM. Both operations now route through a single commit listener that notifies locally and publishes cluster wide, logging a push failure at ERROR with the affected key. SystemTableUpdatedKeyEvent also gains a @JsonCreator constructor. The cluster wide payload is rebuilt on the receiving node by PayloadDeserializer; without a Jackson creator that throws, and since ConversionUtils.convert has no per-item catch the whole polled batch is discarded. Publishing from set() without this would have been a net regression. Verified on a two-node cluster: set, update and delete each publish an event that node 2 receives and re-resolves. Refs: #36828 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
6 tasks
danielsolis-dotcms
approved these changes
Aug 28, 2026
danielsolis-dotcms
deleted the
issue-36828-system-table-set-cluster-wide
branch
August 29, 2026 16:45
3 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Proposed Changes
SystemTableImpl.set()now publishes a cluster-wideSystemTableUpdatedKeyEventthe same waydelete()already did. Both operations route through one commit listener (notifyKeyUpdatedOnCommit), so they cannot drift apart again — that drift is what caused this bug.delete()logged onlye.getMessage(), no key, no stack trace). The silentTry.run(...)wrapper around listener registration is replaced withHibernateUtil.addCommitListenerNoThrow(...).SystemTableUpdatedKeyEventgains a@JsonCreatorconstructor. The cluster-wide payload is rebuilt on the receiving node byPayloadDeserializer; without a Jackson creator that throws, and becauseConversionUtils.converthas no per-item catch, the entire polled batch is discarded. Publishing fromset()without this would have been a net regression, not a partial fix.@Operationdocs onPOST/PUT /api/v1/system-tablestate the change is cluster-wide (regeneratedopenapi.yamlcommitted alongside). The response string is deliberately unchanged — it is asserted verbatim by 6 Postman tests.Checklist
CMS_ADMINISTRATOR_ROLEgate and key blacklist are untouched)Additional Info
Integration tests — two added to
SystemAPITest(already registered inMainSuite2b):test_set_publishes_cluster_wide_event— the fixtest_delete_publishes_cluster_wide_event— guards the pre-existing delete behaviourBoth assert the event is published and that its payload reads back as a typed
SystemTableUpdatedKeyEvent, which is what proves the receiving node can rebuild it.Verified on a real two-node cluster (
docker/docker-compose-examples/cluster-mode), A/B againstdotcms/dotcms:26.08.03-01:set()->CLUSTER_WIDE_EVENTrowset()-> node 2 subscriber notifiedPUT-> row / node 2delete()-> row / node 2Cross-node reads confirmed (write node 1, read node 2), and full CRUD still works on both nodes.
Known gaps (not introduced here)
This does not on its own guarantee convergence, as the issue itself notes:
SystemEventsJob's high-water mark drops events in transit. Being fixed in parallel.SWITCH_SITEevents carryvisibilityType: com.dotcms.api.system.event.UserSessionBean, which has no Jackson creator.PayloadDeserializerthrows on it and — becauseSystemEventsFactory.convertSystemEventDTOhas no per-item catch — the whole poll batch is discarded, silently (SystemEventsJobDelegatelogs it at DEBUG). Observed live on both the fixed and unfixed images. SinceSWITCH_SITEfires on admin login, it lands in the same 5s window as a system-table write very easily (measured 51ms apart in one run). Worth its own issue; the systemic fix is a per-item catch inconvertSystemEventDTO, plus raising that DEBUG to WARN.One acceptance criterion is only half-met by design: "does not report success to the caller as though it had propagated". The push happens in a post-commit listener, after the caller's transaction has committed, so there is no safe way to fail the response without publishing pre-commit. This PR delivers the ERROR log; the caller-facing half needs a product decision.
Refs: #36828
This PR fixes: #36828