Skip to content

Distributed rate limits for SDLC calls and sync requests (Redisson) - #16

Merged
andreasarf merged 9 commits into
claude/redis-lockfrom
claude/redis-rate-limiter
Nov 1, 2026
Merged

andreasarf merged 9 commits into
claude/redis-lockfrom
claude/redis-rate-limiter

Conversation

@andreasarf

Copy link
Copy Markdown
Owner

Summary

Third of three Redis PRs, stacked on #15 (lock), which is stacked on #14 (cache). Merge in order #14 → #15 → #16.

RateLimiters uses Redisson RRateLimiter token buckets in Redis, shared by all instances.

  • Created and updated. A limiter is created with the configured rate on first use. If the configuration changed, an existing limiter is updated the first time an instance uses it.
  • Expiry. Limiters unused for an hour expire from Redis.
  • Redis down. Requests are allowed: limiting is a protection, and losing it shouldn't stop the app.
Limit Default When exceeded
Calls to one SDLC system (resourceapi.rate-limit.sdlc.*) 10 per 1 s, waiting up to 1 s for a permit 503 + Retry-After, because it's our own limit, not the SDLC system failing. A background sync logs it
POST /api/v2/projects/{id}/sync per project (resourceapi.rate-limit.sync.*) 6 per minute 429 + Retry-After
  • SDLC limit. It sits in ExternalProjectService around the real call, so cache hits don't use permits.
  • Sync order. The checks run 404 → lock (409) → rate limit (429) → start (503 if no slot). Requests refused anyway don't use a permit, and a rate-limited request releases the lock it took.

Tests (real Redis)

  • RateLimitIntegrationTest:
    • the third uncached call to an SDLC system gets 503 + Retry-After and is not sent;
    • permits used by "another instance" limit this one too, and refill after the period;
    • cache hits don't count;
    • a changed rate is applied to an existing limiter;
    • the third sync of a project in a minute gets 429 + Retry-After: 30;
    • 409s don't use permits.
  • ProjectSyncBackpressureTest: sync limit raised in that test, because it retries refused syncs and is about the executor.
  • Mutation checks. Each was checked to fail with its fix removed: the config update, and the fail-open handling (via CacheUnavailableTest).

Full suite: 103 tests pass, twice in a row.


Generated by Claude Code

- RateLimiters: Redisson RRateLimiter token buckets shared by all
  instances; created with the configured rate, updated when the
  configuration changed, expiring after an hour unused; requests are
  allowed when Redis is unreachable
- Calls to one SDLC system (resourceapi.rate-limit.sdlc.*, 10/s, waiting up
  to 1s): only real calls count, not cache hits; exceeded -> 503 +
  Retry-After (our own limit), logged by a background sync
- POST /sync per project (resourceapi.rate-limit.sync.*, 6/min): checked
  after the 404 and lock checks, so refused requests don't use a permit;
  exceeded -> 429 + Retry-After
- Tests (real Redis): third uncached call refused and not sent, permits
  shared with another instance and refilled, cache hits don't count,
  changed rate applied (verified to fail without the update), sync 429 with
  Retry-After, 409s don't use permits; Redis down still allows (verified
  to fail without fail-open)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012MLyPXGHeCc1fvTeB8JnT2
…ted limits

- Redisson's RRateLimiter is a sliding window (a used permit returns one
  period later): Retry-After is now the full period (was period/permits,
  so clients retried into another 429); README says sliding window
- A limiter is set up once per instance (trySetRate no longer on every
  call), and again only when Redis dropped it after expiring; it is marked
  set up only after that succeeded, so a failed update is retried
- The set of set-up limiters is bounded (LRU, 10,000)
- Rate limit properties are validated at startup (@positive permits,
  required period and max-wait)
- README: a sync refused for a full executor (503) does use a permit
- Tests: removed/expired limiter is set up again (verified to fail
  without it); invalid limits rejected; Retry-After is one full window

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012MLyPXGHeCc1fvTeB8JnT2
…lient

- external-projects-missing cache (30s): a project the SDLC system doesn't
  know is answered from the cache instead of asking again each time, and
  found soon after it is created there; a sync replaces it with the
  project. Used directly through the CacheManager, so failures are
  caught and logged like the cache error handler does
- ClientRateLimitFilter on /api/* (FilterRegistrationBean, after the trace
  filter): requests per client IP across all instances (100/s, 429 +
  Retry-After); server.forward-headers-strategy=native takes the client IP
  from X-Forwarded-For only from internal proxy addresses
- Tests: unknown answer remembered then found after expiry, sync clears it;
  per-client 429 with Retry-After and X-Trace-Id, separate per client,
  non-API paths unlimited; the repeated-lookup test counts calls after the
  first lookup, as the HTTP client may retry a GET on a stale connection

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012MLyPXGHeCc1fvTeB8JnT2
Startup validation reads the record fields reflectively; the native image
failed to start with MissingReflectionRegistrationError on Limit.permits.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012MLyPXGHeCc1fvTeB8JnT2
…ases

# Conflicts:
#	src/main/java/com/company/resourceapi/config/RateLimitProperties.java
…ed 429

- "Unknown project" is cached as Spring's null value in the external-projects
  cache with a per-entry TTL (30s unknown, 5m found), replacing the separate
  cache and the hand-written lookups; a sync simply @CachePut-replaces it.
- RateLimiters stops asking Redis for 10s after a failure, so an outage
  doesn't add a timeout to every request.
- Limiters expire 10 periods after last use (at least a minute) instead of
  an hour, so per-IP limiters don't pile up.
- The per-client filter answers through the shared exception handler;
  refused callers are logged at DEBUG.
- README: server.tomcat.remoteip.internal-proxies for the per-IP limit.
- Tests: longer negative-cache TTL, no timing-sensitive 404 assertion,
  delta-based call counts, unit test for the outage pause.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012MLyPXGHeCc1fvTeB8JnT2
@andreasarf
andreasarf added this pull request to stack #18 October 31, 2026 14:21
@andreasarf
andreasarf merged commit 39f3d56 into main Nov 1, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants