Skip to content

Distributed lock: one sync per project across instances (Redisson) - #15

Merged
andreasarf merged 14 commits into
claude/practical-dirac-t5uv58from
claude/redis-lock
Nov 1, 2026
Merged

andreasarf merged 14 commits into
claude/practical-dirac-t5uv58from
claude/redis-lock

Conversation

@andreasarf

Copy link
Copy Markdown
Owner

Summary

Second of three Redis PRs, stacked on #14 (cache). Merge #14 first; GitHub then retargets this PR to main.

  • Redisson client. It uses the plain org.redisson:redisson 4.8.0 library. Its Spring Boot starter is built against Boot 3.5 and Boot 4 moved the Redis auto-configuration, so the starter isn't used.
    • Same Redis as the cache. It connects with Boot's DataRedisConnectionDetails and the same timeouts, so Docker Compose and Testcontainers connections work too.
    • Lazy connection. It connects on first use, so the app, including the AOT training run during docker build, starts without Redis.
    • Prefixed names. Object names get resourceapi.redisson.key-prefix.
  • ProjectSyncLock. POST /api/v2/projects/{id}/sync takes a per-project RLock (project-sync:{id}). A second sync of the same project, from any instance, gets 409 until the first ends.
    • Release. The lock is released when the background sync's future completes, or right away if the task executor rejects the sync.
    • Owner. The lock is held by a random owner ID, not the request thread, because it is released on the background thread.
    • Lease. It expires after resourceapi.sync.lock-lease-time (60 s) if never released, e.g. when an instance dies.
    • Redis down. The sync runs without the lock and a warning is logged: a duplicate sync only costs an extra SDLC call, while refusing would stop syncs entirely.
  • spring.cache.type=redis. Redisson brings a JCache provider (javax.cache:cache-api), and Boot prefers JCache over Redis when choosing the cache. Without this setting, every cache call failed with "Cannot find cache named 'sdlc-systems'".

Tests (real Redis)

  • ProjectSyncLockIntegrationTest:
    • lock held by "another instance" gives 409 and no SDLC call; once released, sync works again;
    • slow sync: the second POST gets 409, and the lock is free within 500 ms of the first sync ending, long before its lease;
    • an abandoned lock expires after its lease;
    • lock keys carry the test context's prefix.
  • CacheUnavailableTest: with Redis down, a sync still runs and completes.
  • ProjectSyncBackpressureTest: now uses two projects, because the lock refuses the same project before the executor limit applies. It also checks that a rejected sync released its lock.
  • Mutation checks. Each was checked to fail with its fix removed: the release when a sync ends (the first version of the test didn't catch that, because the short lease expired in time; fixed with a 30 s app lease in the test) and the fail-open handling.

Full suite: 95 tests pass.


Generated by Claude Code

- Redisson client (plain library; its Boot starter targets Boot 3) on the
  same Redis connection details as the cache, connecting lazily so the app
  starts without Redis; object names prefixed (resourceapi.redisson.key-prefix)
- ProjectSyncLock: per-project RLock taken by POST /sync and released when
  the background sync ends (held by a random owner id, as it is released on
  another thread); expires after resourceapi.sync.lock-lease-time (60s); a
  concurrent sync gets 409; if Redis is unreachable the sync runs unlocked
- spring.cache.type=redis: Redisson brings a JCache provider that Boot would
  otherwise pick over Redis
- Tests (real Redis): lock held elsewhere -> 409 and no SDLC call; slow sync
  -> second refused, then allowed once the first released (verified to fail
  without the release); abandoned lock expires; keys carry the context's
  prefix; Redis down -> sync still runs (verified to fail without fail-open)
- Backpressure test uses two projects (the lock refuses the same one first)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012MLyPXGHeCc1fvTeB8JnT2
- ProjectSyncService.requestSync takes and releases the lock; the async
  work moves to ProjectSyncWorker, so no caller can start a sync unlocked
- A lock request that fails (e.g. times out after Redis applied it) keeps
  its lease, so the sync's end still tries to release it
- Redisson is only created for a standalone Redis; with Sentinel/Cluster
  the app starts and syncs run unlocked (was: startup failure)
- The task executor depends on Redisson, so on shutdown running syncs
  release their locks before Redisson stops
- No Redisson retries (a failed request took 2-3s), small connection pool
  (defaults were 24 idle / 64 max)
- Tests: Sentinel config starts and syncs unlocked; executor shutdown
  order; SDLC call count checked before the next sync can call; slow-sync
  test waits for its last sync; backpressure test polls instead of sleeping

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012MLyPXGHeCc1fvTeB8JnT2
- RateLimiters: Redisson RRateLimiter token buckets shared by all
  instances; created with the configured rate, updated when the
  configuration changed, expiring after an hour unused; requests are
  allowed when Redis is unreachable
- Calls to one SDLC system (resourceapi.rate-limit.sdlc.*, 10/s, waiting up
  to 1s): only real calls count, not cache hits; exceeded -> 503 +
  Retry-After (our own limit), logged by a background sync
- POST /sync per project (resourceapi.rate-limit.sync.*, 6/min): checked
  after the 404 and lock checks, so refused requests don't use a permit;
  exceeded -> 429 + Retry-After
- Tests (real Redis): third uncached call refused and not sent, permits
  shared with another instance and refilled, cache hits don't count,
  changed rate applied (verified to fail without the update), sync 429 with
  Retry-After, 409s don't use permits; Redis down still allows (verified
  to fail without fail-open)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012MLyPXGHeCc1fvTeB8JnT2
…ted limits

- Redisson's RRateLimiter is a sliding window (a used permit returns one
  period later): Retry-After is now the full period (was period/permits,
  so clients retried into another 429); README says sliding window
- A limiter is set up once per instance (trySetRate no longer on every
  call), and again only when Redis dropped it after expiring; it is marked
  set up only after that succeeded, so a failed update is retried
- The set of set-up limiters is bounded (LRU, 10,000)
- Rate limit properties are validated at startup (@positive permits,
  required period and max-wait)
- README: a sync refused for a full executor (503) does use a permit
- Tests: removed/expired limiter is set up again (verified to fail
  without it); invalid limits rejected; Retry-After is one full window

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012MLyPXGHeCc1fvTeB8JnT2
# Conflicts:
#	src/main/resources/application.properties
They are deprecated on the server config (logged as warnings at startup).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012MLyPXGHeCc1fvTeB8JnT2
…lient

- external-projects-missing cache (30s): a project the SDLC system doesn't
  know is answered from the cache instead of asking again each time, and
  found soon after it is created there; a sync replaces it with the
  project. Used directly through the CacheManager, so failures are
  caught and logged like the cache error handler does
- ClientRateLimitFilter on /api/* (FilterRegistrationBean, after the trace
  filter): requests per client IP across all instances (100/s, 429 +
  Retry-After); server.forward-headers-strategy=native takes the client IP
  from X-Forwarded-For only from internal proxy addresses
- Tests: unknown answer remembered then found after expiry, sync clears it;
  per-client 429 with Retry-After and X-Trace-Id, separate per client,
  non-API paths unlimited; the repeated-lookup test counts calls after the
  first lookup, as the HTTP client may retry a GET on a stale connection

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012MLyPXGHeCc1fvTeB8JnT2
Startup validation reads the record fields reflectively; the native image
failed to start with MissingReflectionRegistrationError on Limit.permits.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012MLyPXGHeCc1fvTeB8JnT2
…ases

# Conflicts:
#	src/main/java/com/company/resourceapi/config/RateLimitProperties.java
…ed 429

- "Unknown project" is cached as Spring's null value in the external-projects
  cache with a per-entry TTL (30s unknown, 5m found), replacing the separate
  cache and the hand-written lookups; a sync simply @CachePut-replaces it.
- RateLimiters stops asking Redis for 10s after a failure, so an outage
  doesn't add a timeout to every request.
- Limiters expire 10 periods after last use (at least a minute) instead of
  an hour, so per-IP limiters don't pile up.
- The per-client filter answers through the shared exception handler;
  refused callers are logged at DEBUG.
- README: server.tomcat.remoteip.internal-proxies for the per-IP limit.
- Tests: longer negative-cache TTL, no timing-sensitive 404 assertion,
  delta-based call counts, unit test for the outage pause.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012MLyPXGHeCc1fvTeB8JnT2
@andreasarf
andreasarf added this pull request to stack #18 October 31, 2026 14:21
@andreasarf
andreasarf merged commit 39f3d56 into main Nov 1, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants