Redis cache for SDLC systems and SDLC project lookups

README.md

@@ -71,6 +71,57 @@ The OpenAPI document is served at `/v3/api-docs` and Swagger UI at `/swagger-ui.
 annotations. Only `/api/**` is documented.
 To turn both off (e.g. in production), set `springdoc.api-docs.enabled=false` and `springdoc.swagger-ui.enabled=false`.
 
+## Redis
+The application uses Redis (`spring.data.redis.host`/`port`, default `localhost:6379`).
+
+**Cache** (Spring Cache on Spring Data Redis, values stored as JSON):
+
+| Cache | What | TTL |
+|---|---|---|
+| `sdlc-systems` | SDLC systems by id, read on every create, update and SDLC call | `resourceapi.cache.sdlc-systems-ttl` (10m) |
+| `external-projects` | `GET /api/v2/projects/{id}/external` answers per SDLC system and external id | `resourceapi.cache.external-projects-ttl` (5m); "the SDLC system doesn't know this project" (stored as a cached null) only `resourceapi.cache.missing-external-projects-ttl` (30s), so a project created there is found soon |
+
+A background sync always asks the SDLC system and replaces the cached answer with its own. SDLC systems are not changed
+through this API; one changed directly in the database is picked up when its entry expires. The cache is an
+optimization: if Redis is unreachable, cache reads and writes are logged and skipped (each waits at most the
+500 ms Redis timeouts) and requests are served without it.
+
+**Lock** ([Redisson](https://redisson.org/) `RLock`): at most one `POST /api/v2/projects/{id}/sync` per project
+runs at a time, across all instances; a second one gets `409 Conflict` until the first ends. The lock is
+released when the background sync ends and expires after `resourceapi.sync.lock-lease-time` (60s) in case
+an instance dies mid-sync. If Redis can't be reached, the sync runs without the lock (a duplicate sync only
+costs an extra SDLC call; the failed lock request takes about a second). Redisson connects lazily and uses the
+same Redis connection settings; its keys are prefixed with `resourceapi.redisson.key-prefix`. Only a
+standalone Redis is supported for the lock: with Sentinel or Cluster the application still starts and syncs
+run unlocked; TLS uses the JVM's trust store (Boot SSL bundles aren't passed to Redisson). (Redisson's Spring Boot starter isn't used: it targets
+Spring Boot 3. Redisson brings a JCache provider, so `spring.cache.type=redis` is set explicitly.)
+
+**Rate limits** (Redisson `RRateLimiter`, sliding windows shared by all instances: a used permit becomes
+available again one period later, which is also what `Retry-After` says):
+
+| Limit | Default | When exceeded |
+|---|---|---|
+| Calls to one SDLC system (`resourceapi.rate-limit.sdlc.*`), protecting its API quota; cache hits don't count | 10 per 1s, waiting up to 1s for a permit | `503` + `Retry-After` (our own limit, not the SDLC system's failure); a background sync logs it |
+| `POST /api/v2/projects/{id}/sync` per project (`resourceapi.rate-limit.sync.*`); requests refused anyway (unknown project, sync already running) don't count, one refused because no background slot is free (`503`) does | 6 per minute | `429` + `Retry-After` |
+| All `/api/**` requests per client IP (`resourceapi.rate-limit.client.*`); behind a reverse proxy the IP comes from `X-Forwarded-For`, trusted only from internal proxy addresses (`server.forward-headers-strategy=native`, see below) | 100 per second | `429` + `Retry-After` |
+
+A changed rate is applied to the existing limiter when an instance first uses it; unused limiters expire from
+Redis after 10 periods (at least a minute), so per-IP limiters don't pile up. Invalid limits (e.g.
+`permits=0`) stop the application at startup. If Redis can't be reached, requests are allowed (limiting
+protects, losing it shouldn't stop the application), and Redis is not asked again for 10 seconds, so an
+outage doesn't add a timeout to every request.
+
+The per-IP limit only means something if the application sees the client's address. Tomcat takes it from
+`X-Forwarded-For` when the request comes from an address matching `server.tomcat.remoteip.internal-proxies`
+(by default 10/8, 192.168/16, 172.16/12, 169.254/16, 127/8 and IPv6 local addresses). If the reverse proxy or
+load balancer has another address, set that property to match it; otherwise every request appears to come from
+the proxy and all clients share one limit. Never match addresses clients can connect from directly, or they
+can choose their own key by sending `X-Forwarded-For`.
+
+**Local development**: `./mvnw spring-boot:run` starts Redis from `compose.yaml` automatically
+(`spring-boot-docker-compose`); with `java -jar`, start it yourself with `docker compose up -d` (it listens on
+`localhost:6379`). Tests start their own Redis with Testcontainers, so they need Docker.
+
 ## Tracing
 Every request gets a trace id ([Micrometer Tracing](https://micrometer.io/docs/tracing) with Brave, W3C trace context):
 

compose.yaml

@@ -0,0 +1,7 @@
+# Local development services. `./mvnw spring-boot:run` starts them automatically
+# (spring-boot-docker-compose); or run `docker compose up -d` yourself.
+services:
+  redis:
+    image: redis:8-alpine
+    ports:
+      - "6379:6379"

docs/docker.md

@@ -125,9 +125,12 @@ the build. The executable is built for the architecture of the machine (or `--pl
 
 ## Configuration
 
-All three images are configured like the application itself, e.g. with environment variables:
+All three images are configured like the application itself, e.g. with environment variables. The
+application uses Redis (see the README); point it at one with `SPRING_DATA_REDIS_HOST`/`SPRING_DATA_REDIS_PORT`
+(without Redis it still works, uncached):
 
 ```sh
+docker run --rm -p 8080:8080 -e SPRING_DATA_REDIS_HOST=redis --network my-network resourceapi
 docker run --rm -p 9090:9090 -e SERVER_PORT=9090 resourceapi
 docker run --rm -p 8080:8080 -e SPRING_HTTP_CLIENTS_READ_TIMEOUT=10s resourceapi:aot
 ```

pom.xml

@@ -17,6 +17,7 @@
 	<properties>
 		<java.version>25</java.version>
 		<springdoc-openapi.version>3.2.0</springdoc-openapi.version>
+		<redisson.version>4.8.0</redisson.version>
 		<wiremock.version>3.13.2</wiremock.version>
 		<jacoco.version>0.8.16</jacoco.version>
 		<!-- Overridden by agents such as JaCoCo; keeps @{argLine} resolvable when none is set -->
@@ -49,6 +50,21 @@
 			<groupId>io.micrometer</groupId>
 			<artifactId>micrometer-tracing-bridge-brave</artifactId>
 		</dependency>
+		<dependency>
+			<!-- Redis cache (Spring Cache abstraction backed by Spring Data Redis / Lettuce) -->
+			<groupId>org.springframework.boot</groupId>
+			<artifactId>spring-boot-starter-data-redis</artifactId>
+		</dependency>
+		<dependency>
+			<groupId>org.springframework.boot</groupId>
+			<artifactId>spring-boot-starter-cache</artifactId>
+		</dependency>
+		<dependency>
+			<!-- Distributed lock and rate limiter (plain client; its Spring Boot starter targets Boot 3) -->
+			<groupId>org.redisson</groupId>
+			<artifactId>redisson</artifactId>
+			<version>${redisson.version}</version>
+		</dependency>
 		<dependency>
 			<!-- Pooled HTTP client for RestClient (Boot picks it up when present) -->
 			<groupId>org.apache.httpcomponents.client5</groupId>
@@ -59,6 +75,13 @@
 			<artifactId>spring-boot-h2console</artifactId>
 		</dependency>
 
+		<dependency>
+			<!-- Starts compose.yaml (Redis) for spring-boot:run; not packaged, not used by tests -->
+			<groupId>org.springframework.boot</groupId>
+			<artifactId>spring-boot-docker-compose</artifactId>
+			<scope>runtime</scope>
+			<optional>true</optional>
+		</dependency>
 		<dependency>
 			<groupId>org.springframework.boot</groupId>
 			<artifactId>spring-boot-devtools</artifactId>
@@ -87,6 +110,12 @@
 			<artifactId>spring-boot-starter-webmvc-test</artifactId>
 			<scope>test</scope>
 		</dependency>
+		<dependency>
+			<!-- Real Redis for tests, started once in Docker (see RedisTestContainerInitializer) -->
+			<groupId>org.springframework.boot</groupId>
+			<artifactId>spring-boot-testcontainers</artifactId>
+			<scope>test</scope>
+		</dependency>
 		<dependency>
 			<!-- Standalone jar shades WireMock's own Jetty and Jackson, so they can't clash with Boot's -->
 			<groupId>org.wiremock</groupId>

src/main/java/com/company/resourceapi/config/CacheConfig.java

@@ -0,0 +1,79 @@
+package com.company.resourceapi.config;
+
+import com.company.resourceapi.clients.ExternalProject;
+import com.company.resourceapi.entities.SdlcSystem;
+import org.springframework.aot.hint.annotation.RegisterReflectionForBinding;
+import org.springframework.beans.factory.config.BeanDefinition;
+import org.springframework.boot.cache.autoconfigure.RedisCacheManagerBuilderCustomizer;
+import org.springframework.boot.context.properties.EnableConfigurationProperties;
+import org.springframework.cache.annotation.CachingConfigurer;
+import org.springframework.cache.annotation.EnableCaching;
+import org.springframework.cache.interceptor.CacheErrorHandler;
+import org.springframework.cache.interceptor.LoggingCacheErrorHandler;
+import org.springframework.cache.support.NullValue;
+import org.springframework.context.annotation.Bean;
+import org.springframework.context.annotation.Configuration;
+import org.springframework.context.annotation.Role;
+import org.springframework.data.redis.serializer.GenericJacksonJsonRedisSerializer;
+import org.springframework.data.redis.serializer.RedisSerializationContext.SerializationPair;
+import tools.jackson.databind.jsontype.BasicPolymorphicTypeValidator;
+
+/**
+ * Redis-backed Spring Cache. Values are stored as JSON with their type, limited to the classes the
+ * cached values consist of. A cache that can't be reached is logged and skipped, so Redis being down
+ * makes requests slower, not fail.
+ */
+@Configuration
+@EnableCaching
+@EnableConfigurationProperties(CacheTtlProperties.class)
+// cached values are rebuilt from JSON by type name: reflection a native image must know about
+@RegisterReflectionForBinding({SdlcSystem.class, ExternalProject.class, NullValue.class})
+public class CacheConfig {
+
+    public static final String SDLC_SYSTEMS = "sdlc-systems";
+    public static final String EXTERNAL_PROJECTS = "external-projects";
+
+    /**
+     * Starts from Boot's defaults (spring.cache.redis.*: key prefix, null values, ...) and only
+     * replaces the value serializer and sets each cache's time to live. External projects rely on
+     * null values being cached (spring.cache.redis.cache-null-values, true by default).
+     */
+    @Bean
+    RedisCacheManagerBuilderCustomizer redisCacheCustomizer(CacheTtlProperties ttl) {
+        var typeValidator = BasicPolymorphicTypeValidator.builder()
+                .allowIfSubType("com.company.resourceapi.")
+                .allowIfSubType("java.lang.")
+                .allowIfSubType("java.time.")
+                // Spring's marker for a cached null ("the SDLC system doesn't know this project")
+                .allowIfSubType(NullValue.class)
+                .build();
+        var serializer = GenericJacksonJsonRedisSerializer.builder()
+                .enableDefaultTyping(typeValidator)
+                .enableSpringCacheNullValueSupport()
+                .build();
+        return builder -> {
+            var defaults = builder.cacheDefaults().serializeValuesWith(SerializationPair.fromSerializer(serializer));
+            builder.cacheDefaults(defaults)
+                    .withCacheConfiguration(SDLC_SYSTEMS, defaults.entryTtl(ttl.sdlcSystemsTtl()))
+                    // a found project is kept longer than "unknown", so a project created there is found soon
+                    .withCacheConfiguration(EXTERNAL_PROJECTS, defaults.entryTtl((key, value) ->
+                            value == null || value instanceof NullValue
+                                    ? ttl.missingExternalProjectsTtl()
+                                    : ttl.externalProjectsTtl()));
+        };
+    }
+
+    /**
+     * Caching infrastructure reads this before regular beans exist; as an infrastructure bean of its
+     * own, without dependencies, that early creation is expected and not reported as a problem.
+     */
+    @Configuration
+    @Role(BeanDefinition.ROLE_INFRASTRUCTURE)
+    static class CacheErrorHandling implements CachingConfigurer {
+
+        @Override
+        public CacheErrorHandler errorHandler() {
+            return new LoggingCacheErrorHandler();
+        }
+    }
+}

src/main/java/com/company/resourceapi/config/CacheTtlProperties.java

@@ -0,0 +1,20 @@
+package com.company.resourceapi.config;
+
+import java.time.Duration;
+import org.springframework.boot.context.properties.ConfigurationProperties;
+import org.springframework.boot.context.properties.bind.DefaultValue;
+
+/**
+ * Time to live of the Redis caches ({@code resourceapi.cache.*}).
+ *
+ * @param sdlcSystemsTtl      SDLC systems by id; they change rarely and only outside this API
+ * @param externalProjectsTtl projects as described by their SDLC system ({@code GET .../external})
+ * @param missingExternalProjectsTtl "the SDLC system doesn't know this project" answers; short, so a
+ *                                   project created there is found soon
+ */
+@ConfigurationProperties("resourceapi.cache")
+public record CacheTtlProperties(
+        @DefaultValue("10m") Duration sdlcSystemsTtl,
+        @DefaultValue("5m") Duration externalProjectsTtl,
+        @DefaultValue("30s") Duration missingExternalProjectsTtl) {
+}

src/main/java/com/company/resourceapi/config/RateLimitProperties.java

@@ -0,0 +1,42 @@
+package com.company.resourceapi.config;
+
+import jakarta.validation.Valid;
+import jakarta.validation.constraints.NotNull;
+import jakarta.validation.constraints.Positive;
+import java.time.Duration;
+import org.springframework.aot.hint.MemberCategory;
+import org.springframework.aot.hint.annotation.RegisterReflection;
+import org.springframework.boot.context.properties.ConfigurationProperties;
+import org.springframework.validation.annotation.Validated;
+
+/**
+ * Distributed rate limits ({@code resourceapi.rate-limit.*}), shared by all instances through Redis.
+ * Validated at startup: a broken limit would otherwise fail, or silently stop limiting, at runtime.
+ *
+ * @param sdlc   calls to one SDLC system (protects its API quota)
+ * @param sync   sync requests for one project
+ * @param client API requests from one client (IP address)
+ */
+@Validated
+@ConfigurationProperties("resourceapi.rate-limit")
+// validation reads the record fields reflectively, which a native image must know about
+@RegisterReflection(classes = {RateLimitProperties.class, RateLimitProperties.Limit.class},
+        memberCategories = {MemberCategory.ACCESS_DECLARED_FIELDS, MemberCategory.INVOKE_DECLARED_METHODS})
+public record RateLimitProperties(@NotNull @Valid Limit sdlc, @NotNull @Valid Limit sync, @NotNull @Valid Limit client) {
+
+    /**
+     * At most {@code permits} per sliding window of {@code period}: each permit becomes available again
+     * one period after it was used.
+     *
+     * @param permits how many per period
+     * @param period  the sliding window
+     * @param maxWait how long a caller may wait for a permit before being refused
+     */
+    public record Limit(@Positive long permits, @NotNull Duration period, @NotNull Duration maxWait) {
+
+        /** When a permit is surely available again: a used permit only comes back after a full period. */
+        public long retryAfterSeconds() {
+            return Math.max(1, (long) Math.ceil(period.toMillis() / 1000.0));
+        }
+    }
+}

src/main/java/com/company/resourceapi/config/RedissonConfig.java

@@ -0,0 +1,94 @@
+package com.company.resourceapi.config;
+
+import java.time.Duration;
+import java.util.Arrays;
+import java.util.stream.Stream;
+import org.redisson.Redisson;
+import org.redisson.api.RedissonClient;
+import org.redisson.config.Config;
+import org.redisson.config.NameMapper;
+import org.springframework.beans.factory.config.BeanFactoryPostProcessor;
+import org.springframework.beans.factory.annotation.Value;
+import org.springframework.boot.autoconfigure.condition.ConditionalOnExpression;
+import org.springframework.boot.context.properties.EnableConfigurationProperties;
+import org.springframework.boot.data.redis.autoconfigure.DataRedisConnectionDetails;
+import org.springframework.boot.data.redis.autoconfigure.DataRedisProperties;
+import org.springframework.context.annotation.Bean;
+import org.springframework.context.annotation.Configuration;
+
+/**
+ * Redisson client for distributed locks and rate limiters, connected to the same Redis as the cache
+ * (Boot's connection details, so Docker Compose and Testcontainers connections work too).
+ *
+ * <p>It connects lazily, on first use: the application starts without Redis (for example the AOT
+ * training run while building an image), and Redis being down only affects the calls that need it.
+ * Only a standalone Redis is supported; with Sentinel or Cluster there is no client and its users fall
+ * back to running without it. TLS uses the JVM's trust store (Boot SSL bundles aren't passed on).
+ */
+@Configuration
+@EnableConfigurationProperties(RateLimitProperties.class)
+public class RedissonConfig {
+
+    static final String REDISSON_CLIENT = "redissonClient";
+    static final String TASK_EXECUTOR = "applicationTaskExecutor";
+
+    @Bean(name = REDISSON_CLIENT, destroyMethod = "shutdown")
+    @ConditionalOnExpression("'${spring.data.redis.sentinel.master:}' == '' and '${spring.data.redis.cluster.nodes:}' == ''")
+    RedissonClient redissonClient(DataRedisConnectionDetails connection, DataRedisProperties redis,
+            @Value("${resourceapi.redisson.key-prefix}") String keyPrefix) {
+        var standalone = connection.getStandalone();
+        var config = new Config()
+                .setLazyInitialization(true)
+                .setUsername(connection.getUsername())
+                .setPassword(connection.getPassword())
+                .setNameMapper(new KeyPrefixNameMapper(keyPrefix));
+        config.useSingleServer()
+                .setAddress((connection.getSslBundle() != null ? "rediss://" : "redis://")
+                        + standalone.getHost() + ":" + standalone.getPort())
+                .setDatabase(standalone.getDatabase())
+                .setConnectTimeout(millis(redis.getConnectTimeout(), Duration.ofSeconds(10)))
+                .setTimeout(millis(redis.getTimeout(), Duration.ofSeconds(3)))
+                // fail fast like the cache: callers decide what to do without Redis
+                .setRetryAttempts(0)
+                // a lock or rate limit check now and then needs few connections (defaults: 24 idle, 64 max)
+                .setConnectionMinimumIdleSize(1)
+                .setConnectionPoolSize(8)
+                .setSubscriptionConnectionMinimumIdleSize(1)
+                .setSubscriptionConnectionPoolSize(4);
+        return Redisson.create(config);
+    }
+
+    /**
+     * On shutdown, the task executor waits for running syncs, which release their locks when they end;
+     * Redisson must still be up then, so the executor depends on it and is destroyed first.
+     */
+    @Bean
+    static BeanFactoryPostProcessor taskExecutorShutsDownBeforeRedisson() {
+        return beanFactory -> {
+            if (beanFactory.containsBeanDefinition(TASK_EXECUTOR) && beanFactory.containsBeanDefinition(REDISSON_CLIENT)) {
+                var executor = beanFactory.getBeanDefinition(TASK_EXECUTOR);
+                var dependsOn = executor.getDependsOn() == null ? new String[0] : executor.getDependsOn();
+                executor.setDependsOn(Stream.concat(Arrays.stream(dependsOn), Stream.of(REDISSON_CLIENT))
+                        .distinct().toArray(String[]::new));
+            }
+        };
+    }
+
+    private static int millis(Duration value, Duration fallback) {
+        return (int) (value != null ? value : fallback).toMillis();
+    }
+
+    /** Prefixes every Redisson object name (locks, rate limiters) in Redis. */
+    record KeyPrefixNameMapper(String prefix) implements NameMapper {
+
+        @Override
+        public String map(String name) {
+            return prefix + name;
+        }
+
+        @Override
+        public String unmap(String name) {
+            return name.startsWith(prefix) ? name.substring(prefix.length()) : name;
+        }
+    }
+}