[andreasarf]

Summary

Third of three event/messaging PRs, stacked on #20 (outbox notifications), which is stacked on #19 (Spring events).

Before: POST /api/v2/projects/{id}/sync ran the sync on the in-memory @Async executor (at most 8 at once). A crash lost queued work, and a busy executor answered 503.

Now the request goes on a durable RabbitMQ queue:

• Request side (ProjectSyncService):
  • checks run as before: 404 → lock (409) → rate limit (429);
  • then it sends a ProjectSyncRequest to resourceapi.project-sync and waits for the broker's publisher confirm;
  • 202 only once the broker has the request;
  • RabbitMQ unreachable → 503 + Retry-After: 5, and the lock is released.
• Consumer (ProjectSyncConsumer, @RabbitListener):
  • 4 consumers per instance, virtual threads, prefetch=1 so waiting requests go to whichever instance is free;
  • the trace continues from the request (listener observation, traceparent header).
• Retries use Spring Framework 7's RetryTemplate: up to 2 retries after 1 s and 2 s. They apply only to failures that may pass:
  • SDLC system unreachable;
  • SDLC 5xx or 429;
  • our own SDLC rate limit.
• Dead-lettering: a sync that still fails, or fails for another reason (e.g. 403), goes to resourceapi.project-sync.dead, where it can be inspected and shovelled back. default-requeue-rejected=false, so a failing message never loops. A project deleted in the meantime is skipped.
• Lock (from the Redis PRs): taken by the request, released by whichever instance runs the sync. The message carries the lock's owner id, and ProjectSyncLock.lease(projectId, ownerId) rebuilds the lease. A request waiting longer than the lease (60 s) loses its lock (documented).
• ProjectSyncWorker is now a plain synchronous sync(id) that throws; the consumer decides what to retry and logs.

Things that came up

• Upstream body leaked into logs. Spring AMQP's default error handler logs the listener exception with its cause chain, which contained the SDLC system's response body. The existing ProjectSyncIntegrationTest assertion caught it. Fixed: the consumer logs the status only and rejects without the cause, and a ContainerCustomizer makes deliberate dead-lettering log one line instead of a stack trace. Other listener failures still log in full.
• Test isolation. Test contexts share one RabbitMQ, so each context gets its own sync queue (like the exchange and Redis key prefix).
• Removed ProjectSyncBackpressureTest (executor full → 503): that behavior no longer exists. A burst now waits in the queue, covered by the new test below.
• Docker AOT build. The training run (spring.context.exit=onRefresh) exits before listener containers start, so docker build still needs no broker. No training profile was needed.

Tests

• ProjectSyncQueueTest (one consumer):
  • 5 requests are all accepted and synced;
  • SDLC 503 then 200: synced after one retry (2 calls);
  • SDLC keeps failing (500): 3 calls, then dead-lettered, and the lock is free again;
  • SDLC 403: no retry (1 call), dead-lettered.
• RabbitUnavailableTest: POST …/sync → 503 + Retry-After within the connection timeout, the lock is released, and reads still work.
• Unchanged and passing: ProjectSyncIntegrationTest (trace through the queue, status-only logging, re-key guard) and the lock tests (409, release when the sync ends, lease expiry).
• Mutation checks, each failing a test:
  • no lock release;
  • never retry;
  • always retry.

Full suite: 128 tests pass.

AOT and native images of this head, run against Redis 8, RabbitMQ 4 and a WireMock stand-in for Jira:

• Create: a project created without a name gets it from Jira through the queue.
• Sync: 202, then 409 while it runs (native checked with a slow SDLC stand-in, because the native app finishes a fast sync before the second request arrives), then 202 again once it ended.
• RabbitMQ stopped:
  • sync → 503;
  • create → 201, and its notification is delivered after RabbitMQ is back;
  • only Spring AMQP's reconnect warnings in the log.
• Native image: no reflection errors.
• Build note: the native build is unchanged, peak about 12 GB RSS.

🤖 Generated with Claude Code

https://claude.ai/code/session_012MLyPXGHeCc1fvTeB8JnT2