Skip to content

fix(clustering): propagate system table set() cluster wide - #37286

Merged
danielsolis-dotcms merged 2 commits into
mainfrom
issue-36828-system-table-set-cluster-wide
Aug 29, 2026
Merged

fix(clustering): propagate system table set() cluster wide#37286
danielsolis-dotcms merged 2 commits into
mainfrom
issue-36828-system-table-set-cluster-wide

Conversation

@danielsilva-dotcms

@danielsilva-dotcms danielsilva-dotcms commented Aug 28, 2026

Copy link
Copy Markdown
Contributor

Proposed Changes

  • SystemTableImpl.set() now publishes a cluster-wide SystemTableUpdatedKeyEvent the same way delete() already did. Both operations route through one commit listener (notifyKeyUpdatedOnCommit), so they cannot drift apart again — that drift is what caused this bug.
  • A failed cluster-wide push is logged at ERROR with the affected key (previously delete() logged only e.getMessage(), no key, no stack trace). The silent Try.run(...) wrapper around listener registration is replaced with HibernateUtil.addCommitListenerNoThrow(...).
  • SystemTableUpdatedKeyEvent gains a @JsonCreator constructor. The cluster-wide payload is rebuilt on the receiving node by PayloadDeserializer; without a Jackson creator that throws, and because ConversionUtils.convert has no per-item catch, the entire polled batch is discarded. Publishing from set() without this would have been a net regression, not a partial fix.
  • @Operation docs on POST/PUT /api/v1/system-table state the change is cluster-wide (regenerated openapi.yaml committed alongside). The response string is deliberately unchanged — it is asserted verbatim by 6 Postman tests.

Checklist

  • Tests
  • Translations
  • Security Implications Contemplated (none — no auth, input-validation or data-exposure surface changes; the endpoint's existing CMS_ADMINISTRATOR_ROLE gate and key blacklist are untouched)

Additional Info

Integration tests — two added to SystemAPITest (already registered in MainSuite2b):

  • test_set_publishes_cluster_wide_event — the fix
  • test_delete_publishes_cluster_wide_event — guards the pre-existing delete behaviour

Both assert the event is published and that its payload reads back as a typed SystemTableUpdatedKeyEvent, which is what proves the receiving node can rebuild it.

Tests run: 6, Failures: 0, Errors: 0, Skipped: 0 -- com.dotcms.business.SystemAPITest

Verified on a real two-node cluster (docker/docker-compose-examples/cluster-mode), A/B against dotcms/dotcms:26.08.03-01:

Operation Before (unfixed) After
set() -> CLUSTER_WIDE_EVENT row 0 1
set() -> node 2 subscriber notified 0 1
PUT -> row / node 2 - 1 / 1
delete() -> row / node 2 1 / 0 1 / 1

Cross-node reads confirmed (write node 1, read node 2), and full CRUD still works on both nodes.

Known gaps (not introduced here)

This does not on its own guarantee convergence, as the issue itself notes:

  1. SystemEventsJob silently drops the majority of system events in a cluster #36827SystemEventsJob's high-water mark drops events in transit. Being fixed in parallel.
  2. A third defect found while QA'ing this, not yet filed: SWITCH_SITE events carry visibilityType: com.dotcms.api.system.event.UserSessionBean, which has no Jackson creator. PayloadDeserializer throws on it and — because SystemEventsFactory.convertSystemEventDTO has no per-item catch — the whole poll batch is discarded, silently (SystemEventsJobDelegate logs it at DEBUG). Observed live on both the fixed and unfixed images. Since SWITCH_SITE fires on admin login, it lands in the same 5s window as a system-table write very easily (measured 51ms apart in one run). Worth its own issue; the systemic fix is a per-item catch in convertSystemEventDTO, plus raising that DEBUG to WARN.

One acceptance criterion is only half-met by design: "does not report success to the caller as though it had propagated". The push happens in a post-commit listener, after the caller's transaction has committed, so there is no safe way to fail the response without publishing pre-commit. This PR delivers the ERROR log; the caller-facing half needs a product decision.

Refs: #36828

This PR fixes: #36828

SystemTableImpl.set() notified only the local node while delete() also
published a CLUSTER_WIDE_EVENT, so configuration written through
POST/PUT /api/v1/system-table applied only to the node that handled the
request. Subscribers latch the resolved value in memory, so every other
node kept the stale value for the lifetime of its JVM.

Both operations now route through a single commit listener that notifies
locally and publishes cluster wide, logging a push failure at ERROR with
the affected key.

SystemTableUpdatedKeyEvent also gains a @JsonCreator constructor. The
cluster wide payload is rebuilt on the receiving node by
PayloadDeserializer; without a Jackson creator that throws, and since
ConversionUtils.convert has no per-item catch the whole polled batch is
discarded. Publishing from set() without this would have been a net
regression.

Verified on a two-node cluster: set, update and delete each publish an
event that node 2 receives and re-resolves.

Refs: #36828

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@danielsolis-dotcms
danielsolis-dotcms added this pull request to the merge queue Aug 29, 2026
Merged via the queue into main with commit 3192816 Aug 29, 2026
82 of 92 checks passed
@danielsolis-dotcms
danielsolis-dotcms deleted the issue-36828-system-table-set-cluster-wide branch August 29, 2026 16:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Area : Backend PR changes Java/Maven backend code

Projects

Status: No status

Development

Successfully merging this pull request may close these issues.

System table changes made with set() only apply to the node handling the request

2 participants