fix(jwks): read the public keys document under one name, and document rotation

.changeset/eager-pandas-verify.md

@@ -0,0 +1,37 @@
+---
+'seamless-auth-api': minor
+---
+
+Read the JWKS public keys document under one name, and document rotation properly.
+
+**Fixes a deployment trap.** `validateEnvs.sh` required `JWKS_PUBLIC_KEYS`, and the
+configuration reference and `.env.example` named only that, but token verification read
+`SEAMLESS_JWKS_PUBLIC_KEYS`. A deployment that set exactly what this API asked for
+started cleanly and then threw on every JWT verification, because
+`getPublicKeyByKid` could not find the secret. The failure arrives at the first
+authenticated request rather than at boot, which is the worst place for it.
+
+Everything now uses `SEAMLESS_JWKS_PUBLIC_KEYS`: the entrypoint check, the
+`/.well-known/jwks.json` handler, the configuration reference, and `.env.example`.
+
+**Breaking for anyone setting only the unprefixed name**, who is already broken and
+does not know it. After this they fail to start, with the variable named, instead of
+serving a deployment that cannot verify a token it just issued.
+
+**Rotation is documented rather than implemented.** The acceptance criteria in the
+rotation issue are met by the read path that already exists: the document is a list,
+every key in it is published and can verify, and only the active kid signs. What was
+missing was a written procedure, which `docs/production-operations.md` now carries as
+the three-step overlap (add, flip, retire), including why the steps cannot be
+collapsed and why key ids must be environment-variable safe.
+
+The server deliberately does not rotate its own keys. It has no secret-store write
+path, and environment variables are fixed for a process's lifetime, so a server that
+rotated could not observe the result without a restart it cannot trigger. The document
+belongs to whatever manages the secrets.
+
+Accordingly the empty `ensureKeys()` production branch is removed, along with
+`initKeys`, their specs and the dev-stack invocation. It advertised a runtime rotation
+capability that is not going to exist, and its development branch wrote a keypair to
+`./keys` that nothing has ever read: `signingKeyStore` keeps dev keys under
+`./keys/dev` and generates them lazily.

.env.example

@@ -165,7 +165,7 @@ MESSAGING_TWILIO_AUTH_TOKEN=
 # Required when NODE_ENV=production.
 # SEAMLESS_JWKS_ACTIVE_KID=main_2026_04
 # SEAMLESS_JWKS_KEY_main_2026_04_PRIVATE="-----BEGIN PRIVATE KEY-----..."
-# JWKS_PUBLIC_KEYS={"keys":[{"kid":"main_2026_04","pem":"-----BEGIN PUBLIC KEY-----...","createdAt":"2026-04-22T00:00:00.000Z"}]}
+# SEAMLESS_JWKS_PUBLIC_KEYS={"keys":[{"kid":"main_2026_04","pem":"-----BEGIN PUBLIC KEY-----...","createdAt":"2026-04-22T00:00:00.000Z"}]}
 
 # FIDO2 CONFORMANCE
 # Mounts the FIDO2 conformance test interface at /conformance. That surface takes no

docker-compose.dev.yml

@@ -25,13 +25,13 @@ services:
       - node-modules-dev:/app/node_modules
     # Dockerfile.dev has no build step, so the compiled entrypoint cannot run from a
     # fresh clone. Install any dependencies added since this container last booted,
-    # generate dev signing keys, migrate (creating the database on first boot), then
-    # run the watcher against the TypeScript sources.
+    # migrate (creating the database on first boot), then run the watcher against the
+    # TypeScript sources. Dev signing keys are generated lazily by signingKeyStore on
+    # first use, so nothing has to create them up front.
     entrypoint: ['/bin/sh', '-c']
     command:
       - >
         ./syncDevDeps.sh &&
-        npx tsx src/scripts/initKeys.ts &&
         (npm run migrate:up || (npm run db:create && npm run migrate:up)) &&
         npm run dev:container
     depends_on:

docs/configuration.md

@@ -245,11 +245,11 @@ combinations (for example, Twilio requires SID, auth token, and a from-number).
 
 Required when `NODE_ENV=production`. In development, signing keys are generated locally.
 
-| Variable                          | Required  | Default | Notes                                                                              |
-| --------------------------------- | --------- | ------- | ---------------------------------------------------------------------------------- |
-| `SEAMLESS_JWKS_ACTIVE_KID`        | Prod only | -       | Key ID of the active signing key.                                                  |
-| `SEAMLESS_JWKS_KEY_<KID>_PRIVATE` | Prod only | -       | PKCS8 private key PEM for the active KID.                                          |
-| `JWKS_PUBLIC_KEYS`                | Prod only | -       | JSON `{ "keys": [{ "kid", "pem", "createdAt" }] }` published at the JWKS endpoint. |
+| Variable                          | Required  | Default | Notes                                                                                                                                                                                                                                 |
+| --------------------------------- | --------- | ------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| `SEAMLESS_JWKS_ACTIVE_KID`        | Prod only | -       | Key ID of the active signing key.                                                                                                                                                                                                     |
+| `SEAMLESS_JWKS_KEY_<KID>_PRIVATE` | Prod only | -       | PKCS8 private key PEM for the active KID.                                                                                                                                                                                             |
+| `SEAMLESS_JWKS_PUBLIC_KEYS`       | Prod only | -       | JSON `{ "keys": [{ "kid", "pem", "createdAt" }] }`. Every published key, not just the active one: the endpoint serves them all and token verification resolves by `kid`, which is what makes a rotation overlap rather than cut over. |
 
 See [docs/production-operations.md](./production-operations.md) for key rotation.
 

docs/production-operations.md

@@ -24,25 +24,60 @@ Production deployments should define:
 - `OAUTH_STATE_SECRET`
 - `SEAMLESS_JWKS_ACTIVE_KID`
 - `SEAMLESS_JWKS_KEY_<kid>_PRIVATE`
-- `JWKS_PUBLIC_KEYS`
+- `SEAMLESS_JWKS_PUBLIC_KEYS`
 - OAuth client-secret environment variables referenced by provider `clientSecretEnv`
 - Messaging provider credentials when direct delivery is enabled
 
 Do not store raw secrets in `system_config`.
 
 ## Signing Keys
 
-Access tokens are signed with configured JWKS signing keys. A typical rotation is:
+### What the server does, and does not do
 
-1. Generate a new key pair.
-2. Publish the new public key in `JWKS_PUBLIC_KEYS`.
-3. Deploy with both old and new public keys available.
-4. Switch `SEAMLESS_JWKS_ACTIVE_KID` to the new key id.
-5. Keep retired public keys until all tokens signed with them expire.
-6. Remove retired public keys after the token TTL window.
+`SEAMLESS_JWKS_PUBLIC_KEYS` holds **every** published key, not just the active one. The server
+treats it as a list:
 
-Use key ids with letters, numbers, and underscores because the active key id is used to derive the
-private-key environment variable name: `SEAMLESS_JWKS_KEY_<kid>_PRIVATE`.
+- `/.well-known/jwks.json` publishes all of them.
+- Token verification resolves a key by the `kid` in the token's own header, so any published key
+  can verify a token it signed.
+- Only `SEAMLESS_JWKS_ACTIVE_KID`, and that key's private half, is used to sign.
+
+That separation is what makes rotation an overlap rather than a cutover. Nothing has to be
+invalidated to change signing keys.
+
+**The server never writes any of this.** It has no secret-store write path and no cloud SDK, and
+environment variables are fixed for a process's lifetime, so a server that rotated its own keys
+could not observe the result without a restart it cannot trigger. Whatever manages your secrets
+owns the document; the server reads it.
+
+### Rotating without downtime
+
+Three steps, each one deployed before the next. Do not collapse them: the overlap is the whole
+mechanism.
+
+1. **Add.** Generate a key pair, add its public half to `SEAMLESS_JWKS_PUBLIC_KEYS`, and deploy.
+   The outgoing key still signs. Verifiers now accept both.
+2. **Flip.** Point `SEAMLESS_JWKS_ACTIVE_KID` at the new key id, inject its
+   `SEAMLESS_JWKS_KEY_<kid>_PRIVATE`, and deploy. The new key signs. Tokens already issued keep
+   verifying, because the outgoing public key is still published.
+3. **Retire.** Once `refresh_token_ttl` has passed, so no token signed by the outgoing key can
+   still be live, remove it from the document and drop its private key. Deploy.
+
+Each step needs a restart to take effect, because the values arrive as environment variables. The
+server re-reads the document every five minutes within a process, which covers a secret updated in
+place, but a changed environment variable needs a new process.
+
+Key ids must be letters, digits and underscores, and must not start with a digit, because the
+private-key variable name is derived from the id: `SEAMLESS_JWKS_KEY_<kid>_PRIVATE`. A key id with
+a hyphen produces a variable name that cannot be set.
+
+### Managed deployments
+
+For instances built with `seamless-iac`, this is already automated and should not be done by hand.
+Terraform owns the document, signing keys are a set of labels with `jwks_active_key_label`
+selecting the signer, and the three steps above are three applies. See ADR 0011,
+`Terraform owns the JWKS document, and rotation overlaps`, and the procedure in that repo's
+`portal-auth/README.md`.
 
 ## Refresh Tokens
 

src/controllers/jwks.ts

@@ -29,7 +29,7 @@ export function __resetJwksCache() {
 async function loadJwksFromSecrets(): Promise<JWK[]> {
   logger.info('Loading JWKS from Secrets Manager');
 
-  const raw = await getSecret('JWKS_PUBLIC_KEYS');
+  const raw = await getSecret('SEAMLESS_JWKS_PUBLIC_KEYS');
   const parsed = JSON.parse(raw);
 
   const jwks: JWK[] = [];

tests/integration/jwks/jwks.spec.ts

@@ -77,7 +77,7 @@ describe('JWKS - Production Mode', () => {
     expect(res.status).toBe(200);
     expect(res.body.keys[0].kid).toBe('key-1');
 
-    expect(getSecret).toHaveBeenCalledWith('JWKS_PUBLIC_KEYS');
+    expect(getSecret).toHaveBeenCalledWith('SEAMLESS_JWKS_PUBLIC_KEYS');
     expect(res.headers['cache-control']).toContain('max-age=300');
   });
 });