Skip to content

Project notifications to RabbitMQ through an outbox (Spring Modulith) - #20

Merged
andreasarf merged 7 commits into
claude/practical-dirac-t5uv58from
claude/events-outbox
Nov 5, 2026
Merged

andreasarf merged 7 commits into
claude/practical-dirac-t5uv58from
claude/events-outbox

Conversation

@andreasarf

@andreasarf andreasarf commented Nov 1, 2026 •

Copy link
Copy Markdown
Owner

Summary

Second of three event/messaging PRs, stacked on #19 (Spring events). Other systems now learn about project changes from RabbitMQ, and no notification is lost if the broker is down or the app stops right after a change.

Event Exchange (topic) Routing key
ProjectCreated resourceapi.projects (resourceapi.notifications.exchange) project.created
ProjectUpdated (with the changed fields) same project.updated

Messages are JSON, carry occurredAt (from #19) and the request's trace (traceparent header).

How the outbox works

  1. Store. Spring Modulith's event publication registry (JDBC, table event_publication in schema.sql) stores each event in the transaction that changed the project. A row exists exactly if the change was committed.
  2. Send. After commit, ProjectNotifier (@Async @TransactionalEventListener) sends the event and waits for the broker's publisher confirm. Only then is the row deleted (completion-mode=delete).
  3. Resend. A failed, unconfirmed or refused send leaves the row. UnsentNotifications (@Scheduled, every 1 min, for events older than 1 min) sends it again. With several instances, a Redis lock makes one instance do it at a time.
  4. Duplicates and order. Delivery is at least once and not in order, so consumers should handle duplicates and use occurredAt. This is documented in the README.

Decisions worth reviewing

  • Not using Modulith's RabbitMQ externalization (spring-modulith-events-amqp). I started with it. The test that makes the broker refuse a message (exchange deleted) showed the event being deleted from the outbox although RabbitMQ never took it. In its default (module-listener) mode it uses fire-and-forget convertAndSend without confirms; its confirm-waiting variant only exists for third-party outbox libraries (JobRunr, Namastack). ProjectNotifier is about 20 lines and does convertAndSend + waitForConfirmsOrDie (publisher-confirm-type=simple).
  • Own executor for sends (notificationExecutor): virtual threads, trace propagated, at most 16 sends at once.
    • When all 16 are busy, a further send is refused rather than queued (changed in the review round). Before the change, with each send waiting for an unreachable broker, further sends made the committing request wait for a free slot, which coupled API latency to the broker. Now the event just stays in the outbox. Spring logs the TaskRejectedException, and the request is unaffected, because after-commit listeners run in afterCompletion.
    • Declaring an executor bean makes Boot drop its applicationTaskExecutor. spring.task.execution.mode=force keeps it, and the AsyncConfigurer points plain @Async at it.
  • One-line logging for failed fire-and-forget tasks. A RabbitMQ outage would otherwise log a stack trace per event.
  • The registry tracks every @TransactionalEventListener, including SyncOnProjectChange from Project events; sync a project's name after create or re-key #19. An unfinished one is re-run by the resubmission too. A listener whose condition doesn't match leaves nothing behind; a test checks this.
  • JDBC, not JPA, for the registry: no extra entities in the persistence unit, and less for the native image to handle. The table is in schema.sql like the others (H2 version of Modulith's v2 schema).

Local and Docker

  • compose.yaml adds rabbitmq:4-management-alpine (5672, management UI on 15672).
  • docs/docker.md mentions SPRING_RABBITMQ_HOST.
  • Startup without RabbitMQ works, because the connection is lazy. Notifications then wait in the table.
  • AOT image. The training run in docker build needs no broker.

Tests

  • Test setup. Real RabbitMQ 4 via Testcontainers (RabbitTestContainerInitializer, shared container, an exchange per test context, like the Redis key prefix).
  • NotificationsIntegrationTest:
    • create → project.created with the project's JSON and a traceparent header, outbox emptied;
    • rename → project.updated with "changed":["NAME"];
    • broker refuses (exchange deleted) → the event stays in the outbox and nothing is delivered; once the exchange is back, resubmission delivers it and empties the outbox;
    • all 16 senders blocked → a create still answers within 1 s, its event stays in the outbox and is sent once a sender is free (new in the review round).
  • Mutation checks, each failing a test:
    • no confirm wait (the original externalizer's behavior);
    • no resubmission;
    • ProjectUpdated not listened to;
    • waiting instead of refusing when all senders are busy (the request then takes 3 s).

Full suite: 124 tests pass.

Docker check of the stack's top (#21), AOT and native images, against Redis 8, RabbitMQ 4 and a WireMock stand-in for Jira:

  • create → project.created, then project.updated (NAME) after the automatic sync;
  • with RabbitMQ stopped, a create still succeeds; its notification waits in the outbox and arrives after RabbitMQ is back;
  • no reflection errors in the native image.

🤖 Generated with Claude Code

https://claude.ai/code/session_012MLyPXGHeCc1fvTeB8JnT2

andreasarf and others added 3 commits November 1, 2026 05:51
Spring Modulith's JDBC event publication registry stores ProjectCreated and
ProjectUpdated in event_publication within the changing transaction.
ProjectNotifier sends them after commit to the resourceapi.projects topic
exchange (project.created / project.updated, JSON, traceparent) on its own
virtual-thread executor, and only counts a send once the broker confirmed
it; the row is then deleted. UnsentNotifications resends what is left,
one instance at a time (Redis lock). Delivery is at least once.

Modulith's RabbitMQ externalization isn't used: in module-listener mode it
doesn't wait for publisher confirms, so a refused message was counted as
sent (caught by the test that deletes the exchange).

Also: spring.task.execution.mode=force keeps Boot's application task
executor next to the notification executor; failed fire-and-forget tasks
are logged in one line. RabbitMQ in compose.yaml and in tests
(Testcontainers, an exchange per test context).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012MLyPXGHeCc1fvTeB8JnT2
POST /sync now puts a ProjectSyncRequest on the durable queue
resourceapi.project-sync (202 once the broker confirmed it, 503 if RabbitMQ
is unreachable) instead of running it on the in-memory @async executor.
Requests survive restarts and wait for a free consumer instead of being
refused.

ProjectSyncConsumer (4 consumers per instance, virtual threads, prefetch 1,
trace continued from the request) runs the sync with Spring's RetryTemplate:
SDLC failures that may pass (unreachable, 5xx, 429, our own rate limit) are
retried after 1s and 2s; a sync that still fails, or fails otherwise, is
dead-lettered to <queue>.dead and logged in one line, without the upstream
body. The per-project lock is taken by the request and released by the
consumer via the owner id carried in the message.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012MLyPXGHeCc1fvTeB8JnT2
With 16 sends each waiting for an unreachable broker, further sends made
the committing request wait for a free slot, coupling API latency to the
broker. A refused send leaves its event in the outbox, sent again later.
README: consumers should order notifications by occurredAt.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012MLyPXGHeCc1fvTeB8JnT2
@andreasarf
andreasarf merged commit 2d80a3a into claude/practical-dirac-t5uv58 Nov 5, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants