Skip to content

fix(jwks): read the public keys document under one name, and document rotation - #246

Merged
Bccorb merged 1 commit into
mainfrom
feat/jwks-key-rotation
Aug 31, 2026
Merged

fix(jwks): read the public keys document under one name, and document rotation#246
Bccorb merged 1 commit into
mainfrom
feat/jwks-key-rotation

Conversation

@Bccorb

@Bccorb Bccorb commented Aug 31, 2026

Copy link
Copy Markdown
Contributor

Closes #175 and #243.

A deployment trap, reproduced

validateEnvs.sh required JWKS_PUBLIC_KEYS. The configuration reference, .env.example and the /.well-known/jwks.json handler all named that too. But token verification read SEAMLESS_JWKS_PUBLIC_KEYS, and getSecret throws on a missing secret rather than returning empty.

So a deployment that set exactly what this API documented started cleanly and then threw on every JWT verification. Reproduced against the built image with only the documented variable set:

THREW: Secret "SEAMLESS_JWKS_PUBLIC_KEYS" is not defined.

The failure lands on the first authenticated request, not at boot, which is the worst place for it.

The managed instance was never affected: its Terraform injects the same secret under both names, with a comment explaining why, and portal-auth/README.md documents it as "Injecting only the former passes startup but fails JWT verification at runtime." That is an API defect worked around in the infrastructure repo, so it never got fixed here.

Everything now uses SEAMLESS_JWKS_PUBLIC_KEYS.

Breaking, deliberately. Anyone setting only the unprefixed name is already broken and does not know it. After this they fail to start, with the variable named, instead of serving an instance that cannot verify a token it just issued. Trading a silent runtime failure for a loud boot failure is the point.

#175, rescoped against ADR 0011

The issue asks the API to implement rotation. It should not, and I stopped to check rather than build it.

seamless-iac ADR 0011 (Accepted, 2026-08-28) decides "Terraform owns the JWKS document. The runtime never writes it." It reaches that by reading this repo's secretsStore.ts and observing that runtime ownership would require a real secret store with write access, a task role granted write on a bundle it currently only reads, and a live fetch instead of environment variables, because a process cannot see its own new key without a restart it has no way to trigger. That repo's AGENTS.md adds: "Do not add a runtime write path without revisiting ADR 0011."

The same ADR notes rotation needs no image change at all, because the runtime already handles a multi-key document. That matches what is here: the document is a list, loadAllPublicKeys caches every entry by kid, verification resolves by the token's own kid, and only the active kid signs.

So the acceptance criteria were already met by the read path plus Terraform. What was genuinely missing was the written procedure, and a test.

docs/production-operations.md now carries the three-step overlap (add, flip, retire), why the steps cannot be collapsed, why a restart is needed at each one, why key ids must be environment-variable safe, and a pointer to the automated procedure for managed instances.

#243 falls out

The empty ensureKeys() production branch was the placeholder reserved for the rotation that is not going to live here, so keyManager.ts, initKeys.ts, their two specs, and the dev-stack invocation are removed. Its development branch wrote a keypair to ./keys that nothing has ever read: signingKeyStore keeps dev keys under ./keys/dev and generates them lazily, and that is what the JWKS route serves.

Tests

A new keyRotationOverlap.spec.ts exercises the contract with real RSA keys and real jose, rather than asserting about mocks:

  • the active kid signs, and only it
  • the outgoing key still resolves after the active kid moves on
  • the acceptance criterion: a token signed with the outgoing key verifies after the incoming key has taken over
  • a retired key stops resolving once it leaves the document, which is why step three is tied to refresh_token_ttl
  • the document is read under the documented name, so the split cannot come back

Verification

1216 tests passing, 1 skipped. Typecheck, lint, format clean. sh -n and shellcheck clean on validateEnvs.sh. Coverage 98.93% statements, 96.45% branches.

Follow-up for seamless-iac

JWKS_PUBLIC_KEYS is now unused by the API, so the duplicate injection in grand-cross/main.tf and portal-auth can be dropped and the README note retired. Harmless until then. Worth doing in the same pass as the #205 migration task.

… rotation

Closes #175 and #243.

validateEnvs.sh required JWKS_PUBLIC_KEYS, and the configuration reference and
.env.example named only that, but token verification read
SEAMLESS_JWKS_PUBLIC_KEYS. A deployment that set exactly what this API asked for
started cleanly and then threw on every JWT verification, because
getPublicKeyByKid could not find the secret. Reproduced against the built image
with only the documented variable set. The managed instance is unaffected because
its Terraform injects the same secret under both names, a workaround recorded in
that repo rather than fixed here.

Everything now uses SEAMLESS_JWKS_PUBLIC_KEYS. Anyone setting only the unprefixed
name is already broken and does not know it; after this they fail to start with
the variable named.

Rotation is documented rather than built. The read path already satisfies the
acceptance criteria: the document is a list, every key in it is published and can
verify by kid, and only the active kid signs. What was missing was a written
procedure, now in docs/production-operations.md as the three-step overlap, with a
test that signs a token with the outgoing key and verifies it after the active
kid has moved on.

The server does not rotate its own keys, and will not. It has no secret-store
write path, and environment variables are fixed for a process's lifetime, so a
server that rotated could not observe the result without a restart it cannot
trigger. This matches seamless-iac ADR 0011, which settles ownership by reading
this repo's secretsStore.

So the empty ensureKeys() production branch is removed, with initKeys, their
specs and the dev-stack invocation. It advertised a capability that is not going
to exist, and its development branch wrote a keypair to ./keys that nothing
reads: signingKeyStore keeps dev keys under ./keys/dev and generates them lazily.
@Bccorb
Bccorb merged commit 365bc95 into main Aug 31, 2026
5 checks passed
@Bccorb
Bccorb deleted the feat/jwks-key-rotation branch August 31, 2026 19:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Production JWKS key rotation is not implemented

1 participant