Distributed lock: one sync per project across instances (Redisson)

README.md

@@ -79,14 +79,45 @@ The application uses Redis (`spring.data.redis.host`/`port`, default `localhost:
 | Cache | What | TTL |
 |---|---|---|
 | `sdlc-systems` | SDLC systems by id, read on every create, update and SDLC call | `resourceapi.cache.sdlc-systems-ttl` (10m) |
-| `external-projects` | `GET /api/v2/projects/{id}/external` answers per SDLC system and external id; "unknown" answers are not cached | `resourceapi.cache.external-projects-ttl` (5m) |
+| `external-projects` | `GET /api/v2/projects/{id}/external` answers per SDLC system and external id | `resourceapi.cache.external-projects-ttl` (5m); "the SDLC system doesn't know this project" (stored as a cached null) only `resourceapi.cache.missing-external-projects-ttl` (30s), so a project created there is found soon |
 
-A background sync always asks the SDLC system and refreshes the cached answer (or removes it when the SDLC
-system no longer knows the project). SDLC systems are not changed
+A background sync always asks the SDLC system and replaces the cached answer with its own. SDLC systems are not changed
 through this API; one changed directly in the database is picked up when its entry expires. The cache is an
 optimization: if Redis is unreachable, cache reads and writes are logged and skipped (each waits at most the
 500 ms Redis timeouts) and requests are served without it.
 
+**Lock** ([Redisson](https://redisson.org/) `RLock`): at most one `POST /api/v2/projects/{id}/sync` per project
+runs at a time, across all instances; a second one gets `409 Conflict` until the first ends. The lock is
+released when the background sync ends and expires after `resourceapi.sync.lock-lease-time` (60s) in case
+an instance dies mid-sync. If Redis can't be reached, the sync runs without the lock (a duplicate sync only
+costs an extra SDLC call; the failed lock request takes about a second). Redisson connects lazily and uses the
+same Redis connection settings; its keys are prefixed with `resourceapi.redisson.key-prefix`. Only a
+standalone Redis is supported for the lock: with Sentinel or Cluster the application still starts and syncs
+run unlocked; TLS uses the JVM's trust store (Boot SSL bundles aren't passed to Redisson). (Redisson's Spring Boot starter isn't used: it targets
+Spring Boot 3. Redisson brings a JCache provider, so `spring.cache.type=redis` is set explicitly.)
+
+**Rate limits** (Redisson `RRateLimiter`, sliding windows shared by all instances: a used permit becomes
+available again one period later, which is also what `Retry-After` says):
+
+| Limit | Default | When exceeded |
+|---|---|---|
+| Calls to one SDLC system (`resourceapi.rate-limit.sdlc.*`), protecting its API quota; cache hits don't count | 10 per 1s, waiting up to 1s for a permit | `503` + `Retry-After` (our own limit, not the SDLC system's failure); a background sync logs it |
+| `POST /api/v2/projects/{id}/sync` per project (`resourceapi.rate-limit.sync.*`); requests refused anyway (unknown project, sync already running) don't count, one refused because no background slot is free (`503`) does | 6 per minute | `429` + `Retry-After` |
+| All `/api/**` requests per client IP (`resourceapi.rate-limit.client.*`); behind a reverse proxy the IP comes from `X-Forwarded-For`, trusted only from internal proxy addresses (`server.forward-headers-strategy=native`, see below) | 100 per second | `429` + `Retry-After` |
+
+A changed rate is applied to the existing limiter when an instance first uses it; unused limiters expire from
+Redis after 10 periods (at least a minute), so per-IP limiters don't pile up. Invalid limits (e.g.
+`permits=0`) stop the application at startup. If Redis can't be reached, requests are allowed (limiting
+protects, losing it shouldn't stop the application), and Redis is not asked again for 10 seconds, so an
+outage doesn't add a timeout to every request.
+
+The per-IP limit only means something if the application sees the client's address. Tomcat takes it from
+`X-Forwarded-For` when the request comes from an address matching `server.tomcat.remoteip.internal-proxies`
+(by default 10/8, 192.168/16, 172.16/12, 169.254/16, 127/8 and IPv6 local addresses). If the reverse proxy or
+load balancer has another address, set that property to match it; otherwise every request appears to come from
+the proxy and all clients share one limit. Never match addresses clients can connect from directly, or they
+can choose their own key by sending `X-Forwarded-For`.
+
 **Local development**: `./mvnw spring-boot:run` starts Redis from `compose.yaml` automatically
 (`spring-boot-docker-compose`); with `java -jar`, start it yourself with `docker compose up -d` (it listens on
 `localhost:6379`). Tests start their own Redis with Testcontainers, so they need Docker.

pom.xml

@@ -17,6 +17,7 @@
 	<properties>
 		<java.version>25</java.version>
 		<springdoc-openapi.version>3.2.0</springdoc-openapi.version>
+		<redisson.version>4.8.0</redisson.version>
 		<wiremock.version>3.13.2</wiremock.version>
 		<jacoco.version>0.8.16</jacoco.version>
 		<!-- Overridden by agents such as JaCoCo; keeps @{argLine} resolvable when none is set -->
@@ -58,6 +59,12 @@
 			<groupId>org.springframework.boot</groupId>
 			<artifactId>spring-boot-starter-cache</artifactId>
 		</dependency>
+		<dependency>
+			<!-- Distributed lock and rate limiter (plain client; its Spring Boot starter targets Boot 3) -->
+			<groupId>org.redisson</groupId>
+			<artifactId>redisson</artifactId>
+			<version>${redisson.version}</version>
+		</dependency>
 		<dependency>
 			<!-- Pooled HTTP client for RestClient (Boot picks it up when present) -->
 			<groupId>org.apache.httpcomponents.client5</groupId>

src/main/java/com/company/resourceapi/config/CacheConfig.java

@@ -10,6 +10,7 @@
 import org.springframework.cache.annotation.EnableCaching;
 import org.springframework.cache.interceptor.CacheErrorHandler;
 import org.springframework.cache.interceptor.LoggingCacheErrorHandler;
+import org.springframework.cache.support.NullValue;
 import org.springframework.context.annotation.Bean;
 import org.springframework.context.annotation.Configuration;
 import org.springframework.context.annotation.Role;
@@ -26,31 +27,39 @@
 @EnableCaching
 @EnableConfigurationProperties(CacheTtlProperties.class)
 // cached values are rebuilt from JSON by type name: reflection a native image must know about
-@RegisterReflectionForBinding({SdlcSystem.class, ExternalProject.class})
+@RegisterReflectionForBinding({SdlcSystem.class, ExternalProject.class, NullValue.class})
 public class CacheConfig {
 
     public static final String SDLC_SYSTEMS = "sdlc-systems";
     public static final String EXTERNAL_PROJECTS = "external-projects";
 
     /**
      * Starts from Boot's defaults (spring.cache.redis.*: key prefix, null values, ...) and only
-     * replaces the value serializer and sets each cache's time to live.
+     * replaces the value serializer and sets each cache's time to live. External projects rely on
+     * null values being cached (spring.cache.redis.cache-null-values, true by default).
      */
     @Bean
     RedisCacheManagerBuilderCustomizer redisCacheCustomizer(CacheTtlProperties ttl) {
         var typeValidator = BasicPolymorphicTypeValidator.builder()
                 .allowIfSubType("com.company.resourceapi.")
                 .allowIfSubType("java.lang.")
                 .allowIfSubType("java.time.")
+                // Spring's marker for a cached null ("the SDLC system doesn't know this project")
+                .allowIfSubType(NullValue.class)
                 .build();
         var serializer = GenericJacksonJsonRedisSerializer.builder()
                 .enableDefaultTyping(typeValidator)
+                .enableSpringCacheNullValueSupport()
                 .build();
         return builder -> {
             var defaults = builder.cacheDefaults().serializeValuesWith(SerializationPair.fromSerializer(serializer));
             builder.cacheDefaults(defaults)
                     .withCacheConfiguration(SDLC_SYSTEMS, defaults.entryTtl(ttl.sdlcSystemsTtl()))
-                    .withCacheConfiguration(EXTERNAL_PROJECTS, defaults.entryTtl(ttl.externalProjectsTtl()));
+                    // a found project is kept longer than "unknown", so a project created there is found soon
+                    .withCacheConfiguration(EXTERNAL_PROJECTS, defaults.entryTtl((key, value) ->
+                            value == null || value instanceof NullValue
+                                    ? ttl.missingExternalProjectsTtl()
+                                    : ttl.externalProjectsTtl()));
         };
     }
 

src/main/java/com/company/resourceapi/config/CacheTtlProperties.java

@@ -9,9 +9,12 @@
  *
  * @param sdlcSystemsTtl      SDLC systems by id; they change rarely and only outside this API
  * @param externalProjectsTtl projects as described by their SDLC system ({@code GET .../external})
+ * @param missingExternalProjectsTtl "the SDLC system doesn't know this project" answers; short, so a
+ *                                   project created there is found soon
  */
 @ConfigurationProperties("resourceapi.cache")
 public record CacheTtlProperties(
         @DefaultValue("10m") Duration sdlcSystemsTtl,
-        @DefaultValue("5m") Duration externalProjectsTtl) {
+        @DefaultValue("5m") Duration externalProjectsTtl,
+        @DefaultValue("30s") Duration missingExternalProjectsTtl) {
 }

src/main/java/com/company/resourceapi/config/RateLimitProperties.java

@@ -0,0 +1,42 @@
+package com.company.resourceapi.config;
+
+import jakarta.validation.Valid;
+import jakarta.validation.constraints.NotNull;
+import jakarta.validation.constraints.Positive;
+import java.time.Duration;
+import org.springframework.aot.hint.MemberCategory;
+import org.springframework.aot.hint.annotation.RegisterReflection;
+import org.springframework.boot.context.properties.ConfigurationProperties;
+import org.springframework.validation.annotation.Validated;
+
+/**
+ * Distributed rate limits ({@code resourceapi.rate-limit.*}), shared by all instances through Redis.
+ * Validated at startup: a broken limit would otherwise fail, or silently stop limiting, at runtime.
+ *
+ * @param sdlc   calls to one SDLC system (protects its API quota)
+ * @param sync   sync requests for one project
+ * @param client API requests from one client (IP address)
+ */
+@Validated
+@ConfigurationProperties("resourceapi.rate-limit")
+// validation reads the record fields reflectively, which a native image must know about
+@RegisterReflection(classes = {RateLimitProperties.class, RateLimitProperties.Limit.class},
+        memberCategories = {MemberCategory.ACCESS_DECLARED_FIELDS, MemberCategory.INVOKE_DECLARED_METHODS})
+public record RateLimitProperties(@NotNull @Valid Limit sdlc, @NotNull @Valid Limit sync, @NotNull @Valid Limit client) {
+
+    /**
+     * At most {@code permits} per sliding window of {@code period}: each permit becomes available again
+     * one period after it was used.
+     *
+     * @param permits how many per period
+     * @param period  the sliding window
+     * @param maxWait how long a caller may wait for a permit before being refused
+     */
+    public record Limit(@Positive long permits, @NotNull Duration period, @NotNull Duration maxWait) {
+
+        /** When a permit is surely available again: a used permit only comes back after a full period. */
+        public long retryAfterSeconds() {
+            return Math.max(1, (long) Math.ceil(period.toMillis() / 1000.0));
+        }
+    }
+}

src/main/java/com/company/resourceapi/config/RedissonConfig.java

@@ -0,0 +1,94 @@
+package com.company.resourceapi.config;
+
+import java.time.Duration;
+import java.util.Arrays;
+import java.util.stream.Stream;
+import org.redisson.Redisson;
+import org.redisson.api.RedissonClient;
+import org.redisson.config.Config;
+import org.redisson.config.NameMapper;
+import org.springframework.beans.factory.config.BeanFactoryPostProcessor;
+import org.springframework.beans.factory.annotation.Value;
+import org.springframework.boot.autoconfigure.condition.ConditionalOnExpression;
+import org.springframework.boot.context.properties.EnableConfigurationProperties;
+import org.springframework.boot.data.redis.autoconfigure.DataRedisConnectionDetails;
+import org.springframework.boot.data.redis.autoconfigure.DataRedisProperties;
+import org.springframework.context.annotation.Bean;
+import org.springframework.context.annotation.Configuration;
+
+/**
+ * Redisson client for distributed locks and rate limiters, connected to the same Redis as the cache
+ * (Boot's connection details, so Docker Compose and Testcontainers connections work too).
+ *
+ * <p>It connects lazily, on first use: the application starts without Redis (for example the AOT
+ * training run while building an image), and Redis being down only affects the calls that need it.
+ * Only a standalone Redis is supported; with Sentinel or Cluster there is no client and its users fall
+ * back to running without it. TLS uses the JVM's trust store (Boot SSL bundles aren't passed on).
+ */
+@Configuration
+@EnableConfigurationProperties(RateLimitProperties.class)
+public class RedissonConfig {
+
+    static final String REDISSON_CLIENT = "redissonClient";
+    static final String TASK_EXECUTOR = "applicationTaskExecutor";
+
+    @Bean(name = REDISSON_CLIENT, destroyMethod = "shutdown")
+    @ConditionalOnExpression("'${spring.data.redis.sentinel.master:}' == '' and '${spring.data.redis.cluster.nodes:}' == ''")
+    RedissonClient redissonClient(DataRedisConnectionDetails connection, DataRedisProperties redis,
+            @Value("${resourceapi.redisson.key-prefix}") String keyPrefix) {
+        var standalone = connection.getStandalone();
+        var config = new Config()
+                .setLazyInitialization(true)
+                .setUsername(connection.getUsername())
+                .setPassword(connection.getPassword())
+                .setNameMapper(new KeyPrefixNameMapper(keyPrefix));
+        config.useSingleServer()
+                .setAddress((connection.getSslBundle() != null ? "rediss://" : "redis://")
+                        + standalone.getHost() + ":" + standalone.getPort())
+                .setDatabase(standalone.getDatabase())
+                .setConnectTimeout(millis(redis.getConnectTimeout(), Duration.ofSeconds(10)))
+                .setTimeout(millis(redis.getTimeout(), Duration.ofSeconds(3)))
+                // fail fast like the cache: callers decide what to do without Redis
+                .setRetryAttempts(0)
+                // a lock or rate limit check now and then needs few connections (defaults: 24 idle, 64 max)
+                .setConnectionMinimumIdleSize(1)
+                .setConnectionPoolSize(8)
+                .setSubscriptionConnectionMinimumIdleSize(1)
+                .setSubscriptionConnectionPoolSize(4);
+        return Redisson.create(config);
+    }
+
+    /**
+     * On shutdown, the task executor waits for running syncs, which release their locks when they end;
+     * Redisson must still be up then, so the executor depends on it and is destroyed first.
+     */
+    @Bean
+    static BeanFactoryPostProcessor taskExecutorShutsDownBeforeRedisson() {
+        return beanFactory -> {
+            if (beanFactory.containsBeanDefinition(TASK_EXECUTOR) && beanFactory.containsBeanDefinition(REDISSON_CLIENT)) {
+                var executor = beanFactory.getBeanDefinition(TASK_EXECUTOR);
+                var dependsOn = executor.getDependsOn() == null ? new String[0] : executor.getDependsOn();
+                executor.setDependsOn(Stream.concat(Arrays.stream(dependsOn), Stream.of(REDISSON_CLIENT))
+                        .distinct().toArray(String[]::new));
+            }
+        };
+    }
+
+    private static int millis(Duration value, Duration fallback) {
+        return (int) (value != null ? value : fallback).toMillis();
+    }
+
+    /** Prefixes every Redisson object name (locks, rate limiters) in Redis. */
+    record KeyPrefixNameMapper(String prefix) implements NameMapper {
+
+        @Override
+        public String map(String name) {
+            return prefix + name;
+        }
+
+        @Override
+        public String unmap(String name) {
+            return name.startsWith(prefix) ? name.substring(prefix.length()) : name;
+        }
+    }
+}

src/main/java/com/company/resourceapi/controllers/ProjectSyncController.java

@@ -5,7 +5,6 @@
 import static com.company.resourceapi.controllers.ProjectRestController.PATH_VARIABLE_ID;
 
 import com.company.resourceapi.exceptions.NotFoundException;
-import com.company.resourceapi.services.ProjectService;
 import com.company.resourceapi.services.ProjectSyncService;
 import io.swagger.v3.oas.annotations.Operation;
 import io.swagger.v3.oas.annotations.Parameter;
@@ -23,18 +22,15 @@
 @RequiredArgsConstructor
 public class ProjectSyncController {
 
-    private final ProjectService projectService;
     private final ProjectSyncService projectSyncService;
 
     @Operation(summary = "Refresh a Project's name from its SDLC system, in the background")
     @PostMapping(ENDPOINT_ID + "/sync")
     public ResponseEntity<Void> syncProject(
             @Parameter(name = PATH_VARIABLE_ID, description = "ID", required = true)
             @PathVariable(PATH_VARIABLE_ID) final long projectId) throws NotFoundException {
-        // an unknown project is reported now, not only in the background (404 via ApiExceptionHandler);
-        // a full task executor rejects the sync (503 via ApiExceptionHandler)
-        projectService.getProject(projectId);
-        projectSyncService.syncName(projectId);
+        // unknown project: 404, sync already running: 409, no background slot: 503 (ApiExceptionHandler)
+        projectSyncService.requestSync(projectId);
         return ResponseEntity.accepted().build();
     }
 }

src/main/java/com/company/resourceapi/exceptions/RateLimitExceededException.java

@@ -0,0 +1,20 @@
+package com.company.resourceapi.exceptions;
+
+import lombok.Getter;
+
+/** One of our own rate limits refused the request; it can be retried after {@link #getRetryAfterSeconds()}. */
+@Getter
+public class RateLimitExceededException extends RuntimeException {
+
+    /** Whose limit: the caller's own requests (429) or the application's calls to a system (503). */
+    public enum Scope { CALLER, UPSTREAM }
+
+    private final Scope scope;
+    private final long retryAfterSeconds;
+
+    public RateLimitExceededException(Scope scope, String message, long retryAfterSeconds) {
+        super(message);
+        this.scope = scope;
+        this.retryAfterSeconds = retryAfterSeconds;
+    }
+}

src/main/java/com/company/resourceapi/exceptions/SyncInProgressException.java

@@ -0,0 +1,8 @@
+package com.company.resourceapi.exceptions;
+
+public class SyncInProgressException extends RuntimeException {
+
+    public SyncInProgressException(long projectId) {
+        super("A sync of project %d is already running.".formatted(projectId));
+    }
+}