[ThomasK33]

Summary

Goal: every open coder/coder-k8s issue is fixed and lands on main. This PR fixes two CoderProvisioner key-ownership issues.

1. A CoderProvisioner no longer deletes or replaces a Coder provisioner key that its status does not record (🤖 CoderProvisioner: a rotation at the target deletes another provisioner's same-named key #414). Before, a provisioner P that met a key with its key name at its target deleted that key by name and created its own. When the key belonged to provisioner Q, Q's Secret silently held a dead key.
2. Each key record and the applied key fields now bind their operator-token source to the CoderControlPlane UID (🤖 CoderProvisioner: bind each key record's operator token to the control plane UID #417). After a control plane is deleted and created again under the same name, the operator no longer sends the new control plane's token for a key of the old one, and creates a new key for the provisioner.
3. A key that status records under an organization ID stays tracked after the organization is renamed and its name reused. Before, the lookup by organization name missed such a key, the provisioner replaced it without a record, and deletion left it valid in Coder. An independent verifier found this on 0149387.

Fixes #414
Fixes #417
Refs #416 (not changed here, see the decision below)

Background

• 🤖 fix: revoke a CoderProvisioner key created at an intermediate control plane endpoint #413 added status.pendingKeys (record before create) and made the cleanup of a record without key IDs delete nothing. The rotation paths at the target (drift, Secret loss or unpublished record, metadata backfill) still deleted a same-named key by name when Coder returned it without plaintext. That is 🤖 CoderProvisioner: a rotation at the target deletes another provisioner's same-named key #414.
• 🤖 fix: revoke a CoderProvisioner key created at an intermediate control plane endpoint #413 records name their token source by control plane name only. revokeKey read the control plane by name and sent its token to the record's URL. That is 🤖 CoderProvisioner: bind each key record's operator token to the control plane UID #417.

Implementation

1. 🤖 CoderProvisioner: a rotation at the target deletes another provisioner's same-named key #414, before any create or replacement at the target:
   • The operator looks the target key name up in Coder (ProvisionerKeyByName).
   • A key whose ID is neither status.provisionerKeyID nor the keyID of a status.pendingKeys entry is not deleted or replaced. The operator keeps the applied key, the Secret and the Deployment, and writes no target record. It sets ProvisionerKeyReady=False, reason KeyNameInUse, with a message that names the URL, organization, key name and key ID. It requeues after 2 minutes, because nothing signals when that key goes.
   • During the conflict, the operator still deletes the replaced keys that status.pendingKeys records outside the target slot, each with the token of its own control plane (revokePendingKeys, the cleanup loop of 🤖 fix: revoke a CoderProvisioner key created at an intermediate control plane endpoint #413 moved into a function). Unpublished entries stay, because the Secret can hold their key. The Codex security review asked for this.
   • The refusal comes before the record-before-create status write, so a repeated conflict writes no status. For() has no predicate, so a write per reconcile would loop. The envtest checks that the resourceVersion stays the same.
   • A recorded key is deleted by its ID (DeleteProvisionerKeyByID) after the record-before-create write, and then one create runs. A create answer without plaintext means another key took the name after the lookup. The operator then also refuses with KeyNameInUse.
   • The three rotation branches (drift, rotate, metadata backfill) were three copies of "ensure, and on no plaintext delete by name and ensure again". They are now one block.
   • coderbootstrap.Client.DeleteProvisionerKey (delete by name) is removed with its SDK implementation and tests. No operator path can delete a key by name now.
   • ProvisionerKeyReady=True is now set on every reconcile that passes the key step, so a revert to the applied key name after a conflict clears KeyNameInUse.
2. 🤖 CoderProvisioner: bind each key record's operator token to the control plane UID #417:
   • API: new string fields status.controlPlaneUID and status.pendingKeys[].controlPlaneUID. A new target record and a new applied key record the UID of the control plane.
   • revokeKey (pending cleanup and deletion) checks tokenBoundTo before it reads the token. A record with a UID gets its control plane's token only while the control plane has that UID, also at the same URL. With TLS, a control plane created again under the same name keeps its in-cluster Coder API URL and can serve another Coder backend there. That backend's "no such key" answer does not show that the old key is gone.
   • Otherwise the outcome is keyControlPlaneChanged: nothing is sent, the record stays, and the new condition ControlPlaneChangedKeyKept=True names each such entry. Each reconcile rebuilds it, like UnverifiedKeyKept. Deletion logs such a record and removes the finalizer.
   • A changed control plane UID (status.controlPlaneUID set and different) is drift. The provisioner creates a new key at the new control plane, instead of keeping a key that a new backend can reject.
   • Records with different control plane UIDs name different key slots (sameKeySlot), so the replaced key of the old control plane stays recorded and reported, also at the same URL. A record without a UID matches any UID.
   • When the target lookup finds a key that status records under the old control plane (the new one serves the same backend), the target replacement deletes it by ID with the current token at the current URL, and drops every record of that key ID.
   • Migration: a record without a UID (status from v0.1.5 or a :main build) is bound only at the current Coder API URL of its control plane. This is the safe side. An old URL never gets the token of a control plane that the operator cannot match. The cost: after an upgrade, a URL change of the same control plane before the next reconcile leaves the replaced key in Coder, and ControlPlaneChangedKeyKept names it.
   • For status without status.controlPlaneUID, the next reconcile looks the applied key up at the current control plane by its recorded IDs (see item 3). When Coder lists that key ID there, the operator records the control plane's UID, so a later recreate is drift. When Coder answers that the key is gone there, it creates a new key. This needs one lookup per upgraded provisioner.
3. Organization renames. Coder lets an admin rename an organization and give its name to another one. A lookup by organization name then misses a key that status records under the old organization ID.
   • Applied key check for status without a UID: the operator lists the organization status.organizationID and matches status.provisionerKeyID (ProvisionerKeyByID, new in coderbootstrap, which shares its list-and-match code with DeleteProvisionerKeyByID). Coder's own 404 for that organization, or a list without that key ID, means the key is gone at this control plane: the operator creates a new key and drops the old record. Any other error fails the reconcile and changes nothing.
   • Replacement at the target (Secret loss, metadata backfill, a tags change, or an applied key that the check above did not find): before the create, the operator deletes by their recorded IDs the applied key and the target entries in status.pendingKeys that have key IDs, beside the key that the name lookup found. A failed delete fails the reconcile before the create. main has the same loss on these paths.
   • Drift that moves the replaced key of status without a UID to status.pendingKeys now records in that entry the UID of the control plane it names, while that control plane has the recorded Coder API URL (bindToControlPlane). A control plane created again under that name before Coder confirms the delete then gets no token for the entry, and ControlPlaneChangedKeyKept names it.
4. Docs: docs/tutorials/deploy-coderprovisioner.md describes the key name rule, KeyNameInUse with its recovery, the UID binding, ControlPlaneChangedKeyKept, and the migration rule. CRD, dist/install.yaml and the API reference are regenerated.

Decision: #416 stays open (Refs)

#416 asks to delete off-target recorded keys while the current target fails its token or entitlement checks. It does not conflict with the fail-static credential checks. I did not fold it in for these reasons:

• It needs a second cleanup site before the token check, beside the cleanup that this PR changes. Each site needs its own rules for unpublished entries (the Secret can hold their key) and for the target slot.
• The KeyNameInUse gate added here already runs the cleanup for published entries outside the target slot. The token and entitlement gates do not. There the cleanup would retry dead endpoints every 15 seconds (the operator-access retry interval) instead of every 2 minutes, and the target slot would have to be computed before the token check. revokePendingKeys is the function a 🤖 CoderProvisioner: keys recorded at an older endpoint are not deleted while the current target fails its checks #416 fix can call there.
• Deletion of the provisioner still deletes every recorded key, so the gap stays bounded.

Validation

• TestCoderProvisionerKey_OtherProvisionersKeyKept (envtest, 5 cases, stateful fake Coder with one key per name). Provisioner Q holds key provisioner. Provisioner P reaches that key name in three ways: its first key, a key name change, and status without key records. P must keep Q's key, keep its own key, Secret and Deployment, report KeyNameInUse with Q's key ID, write no status on a repeated conflict, and record no create. P recovers when Q's key goes, after a rename to a distinct name, or after a revert to its applied name (Q's key and P's old key ID stay). In one case P also has a replaced key recorded at another key name; the conflict reconcile deletes it and drops its record (this case fails on 9b59a91).
• TestCoderProvisionerKey_ControlPlaneRecreated (envtest, 10 cases). Control plane coder at http://coder-old.example.com is deleted and created again under the same name with another token, at http://coder-new.example.com or at the same URL. Then P rotates or is deleted, with and without a recorded UID. No request may carry the new token to the old URL, and with a recorded UID the new token is never used for the old key. On rotation, P creates a new key, the record of the old key stays, and ControlPlaneChangedKeyKept names it. When the recreated control plane serves the old backend, the rotation deletes the old key there and keeps no record of it. For status without a UID: a recreate at the same URL against a fresh backend creates a new key, and a reconcile before such a recreate binds the key that the backend holds, so the recreate is then drift. One case keeps the control plane, moves P to another one with status without a UID, and expects the old key deleted at its URL with its own token.
• Commit 4094802 adds both tests on the code of main, where they fail. The legacy "control plane kept" case passes on main too, as a migration guard. The same-URL cases came with the fixes for the Codex review (cd33036, 94bdf89):

On main (4094802)

--- FAIL: TestCoderProvisionerKey_ControlPlaneRecreated (0.00s)
    --- FAIL: .../deletion_after_the_recreate (2.26s)
            Messages:   the new control plane's token must not reach http://coder-old.example.com, sent [...]
    --- FAIL: .../status_without_control_plane_UID,_deletion_after_the_recreate (2.26s)
            Messages:   the new control plane's token must not reach http://coder-old.example.com, sent [...]
    --- FAIL: .../status_without_control_plane_UID,_rotation_after_the_recreate (2.28s)
            Messages:   the new control plane's token must not reach http://coder-old.example.com, sent [...]
    --- FAIL: .../rotation_after_the_recreate (2.29s)
            Messages:   the new control plane's token must not reach http://coder-old.example.com, sent [...]
--- FAIL: TestCoderProvisionerKey_OtherProvisionersKeyKept (0.00s)
    --- FAIL: .../first_key_of_P (2.28s)
            Messages:   the key of Q must still exist
    --- FAIL: .../status_without_key_records (2.31s)
            Messages:   the key of Q must still exist
    --- FAIL: .../key_name_change_of_P (2.33s)
            Messages:   the key of Q must still exist
FAIL

• TestCoderProvisionerKey_OrganizationRenamed (envtest, 4 cases). It runs the real SDK client against an HTTP Coder fixture. Organization A, named team, holds the applied key. A is renamed and organization B takes the name team. For status without a UID, the key stays applied, and deleting P deletes it in A by its IDs. This is the verifier's regression test. For a Secret loss (with and without a UID) and a tags change, the rotation deletes the old key in A by its IDs and creates the new key in B. In every case, deleting P leaves no key in Coder.
• TestCoderProvisionerKey_StatusWithoutUIDReplacedKeyBound (envtest). Status without a UID, P moves to another control plane, and Coder fails the delete of the replaced key. The old control plane is then created again at its URL with another token. The new token must not be used for the old key, its entry stays, and ControlPlaneChangedKeyKept names it.
• TestProvisionerKeyByID (SDK): a listed key, another ID, Coder's own 404 (nil), and a 404 not from Coder (error).
• Each new test landed before its fix and failed there:

Failing before each fix

# c86c1f1c (verifier test, on the code of 0149387c)
--- FAIL: TestCoderProvisionerKey_StatusWithoutUIDOrganizationRenamed (2.21s)
        Messages:   deleting the provisioner must revoke its recorded key in the renamed organization by ID
# c410f572 (on 1126867b, before bindToControlPlane)
--- FAIL: TestCoderProvisionerKey_StatusWithoutUIDReplacedKeyBound (2.18s)
        Messages:   the new control plane's token must not be used for the old key, sent [... token:operator-token-new ...]
# cf7d2a53 (on 7087cb85, before the replacement deletes by recorded IDs)
    --- FAIL: TestCoderProvisionerKey_OrganizationRenamed/Secret_loss (2.11s)
    --- FAIL: TestCoderProvisionerKey_OrganizationRenamed/status_without_control_plane_UID,_Secret_loss (2.11s)
    --- PASS: TestCoderProvisionerKey_OrganizationRenamed/status_without_control_plane_UID (2.11s)
    --- FAIL: TestCoderProvisionerKey_OrganizationRenamed/tags_change (2.11s)
        Messages:   the rotation must delete the replaced key in the renamed organization by ID

• Negative controls on the fix, each reverted afterwards:
  • Treating every existing key as recorded fails all three conflict cases ("the key of Q must still exist").
  • tokenBoundTo returning true fails the four different-URL recreate cases.
  • Counting a record without a UID as bound fails both legacy recreate cases.
  • Dropping the UID allowance (current URL only) fails TestCoderProvisionerKey_RevertAfterFailedRotation and TestCoderProvisionerDeletion_TriesCurrentControlPlaneURL.
  • Accepting the current URL despite a UID mismatch fails the same-URL deletion and rotation cases.
  • Dropping the UID drift fails both same-URL rotation cases ("P must create a new key").
  • sameKeySlot without the UID fails the same-URL rotation case ("the record of the old key must stay").
  • Keeping records of the key that the target replacement deleted fails the same-backend case.
  • Skipping the applied key check for status without a UID fails both same-URL legacy cases.
  • Returning on KeyNameInUse before the cleanup fails the replaced-key case ("P must record no create").
  • Setting ProvisionerKeyReady=True only after a replacement fails the revert case.
  • Leaving a UID-less replaced entry unbound fails StatusWithoutUIDReplacedKeyBound.
  • Repeating the delete of an applied key that the ID check already found gone fails status without control plane UID, recreate at the same URL (no delete of the old key expected).
• Existing tests updated to model Coder's state instead of a fresh provisioner that takes over an existing key:
  • RotationOnSecretLoss now loses the Secret of a recorded key.
  • TagsDrift and the KeySecret_OwnedIsReused test use keys with IDs.
  • UnpublishedKeyIsRotated: after a lost create answer, the operator now reports KeyNameInUse until the key is deleted by hand, and then rotates.
  • IDLessRecordKeepsOtherProvisionersKey: Q now creates its key at B after P's failed create at B. Before, P would now refuse B at once.
• Local gates on 90bb3a2: make verify-vendor, make build, make lint, make test, make test-integration, make codegen manifests docs-reference build-installer (no further diff), nix develop -c make docs-check, and cspell and markdownlint-cli2 on the changed docs. All pass.
• An independent pre-PR review (different model family) found the stale KeyNameInUse after a revert (fixed in 5fcce53). It also flagged the current-URL allowance that the first version had, which Codex then raised too (fixed in cd33036), and confirmed the residual race listed under Risks.
• Codex rounds: the automatic review on 5fcce53 (2 findings, fixed in cd33036), @codex review on cd33036 (1 finding, fixed in 94bdf89), @codex review on 9b59a91 (no major issues), @codex security review on 9b59a91 (1 finding, fixed in 0149387), @codex review on 0149387 (1 finding, first declined, then fixed in 1126867 and 90bb3a2 after the verifier showed a lost cleanup), @codex security review on 0149387 (no security issues).

Risks

• A create whose answer was lost (Coder created the key, the operator did not get its ID) now blocks the provisioner with KeyNameInUse until someone deletes that key in Coder. Before, the operator deleted the key by name. That was the takeover of 🤖 CoderProvisioner: a rotation at the target deletes another provisioner's same-named key #414. The docs describe the manual delete.
• Status without key records, for example after a restore without status, also gets KeyNameInUse for a key that may be its own. The provisioner keeps running with its Secret. Deleting the key in Coder lets the operator create a new one.
• After a key name change to a taken name, the operator keeps the old key and Deployment (fail static). Unpublished pending entries are not cleaned while the conflict lasts. 🤖 CoderProvisioner: keys recorded at an older endpoint are not deleted while the current target fails its checks #416 covers cleanup behind the token and entitlement checks.
• DeleteProvisionerKeyByID lists the keys, finds the recorded ID, then deletes by name, because Coder v2.37.3 deletes provisioner keys only by name (httpmw.ExtractProvisionerKeyParam). If someone deletes P's key and creates another key with the same name between those two requests, the delete hits the new key. This race predates this PR and is documented on the function. The operator cannot close it without a Coder API that deletes by ID.
• One extra Coder lookup runs before each rotation, plus one list request per recorded target key with IDs. Each KeyNameInUse retry costs one lookup every 2 minutes.
• A record without a UID at an old URL of the same control plane stays with ControlPlaneChangedKeyKept until someone deletes its key by hand and removes the entry. This is the migration cost above.
• Status without a UID whose applied key the current backend does not list gets a new key, and the operator drops the old record, because Coder's own answer at the recorded organization ID says the key is gone there. If the control plane was created again at the same URL against another backend between the operator upgrade and its first reconcile, the old backend can still hold that key, and no recorded URL reaches it.
• An entry dropped by the older pruning rule (an earlier create at the target) is not deleted by ID again. Each replacement on this head deletes the target entries by ID before its create, so only entries written by an older operator version rely on that rule.
• After a control plane is deleted and created again under the same name, the provisioner rotates its key (a Deployment rollout), and the key of the old control plane stays recorded with ControlPlaneChangedKeyKept until someone deletes it by hand and removes the entry. Deletion of the provisioner before that rotation sends no token for its key and leaves the key in Coder.

---

Generated with xum • Model: anthropic:claude-opus-5-5 • Thinking: xhigh

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-10-30T20:54:46.173190Z	90bb3a2	Manual request
🔒 Security Review	✅ Completed 2026-10-30T20:55:49.095237Z	90bb3a2	Manual request

Security findings

Advisory findings (1)

• 🟡 Revoke off-target keys before returning on name conflicts · Medium

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 5fcce53357

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[ThomasK33]

@codex review

Round 1 findings are addressed in cd33036: a recorded control plane UID is now required for every token use (the current-URL rule applies only to records without a UID), and a UID change rotates the key.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: cd33036074

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[ThomasK33]

@codex review

Round 2 finding is addressed in 94bdf89: status without a control plane UID now gets its applied key checked at the current control plane, and the UID is recorded only when Coder holds that key there.

[chatgpt-codex-connector]

Codex Review: Didn't find any major issues. Already looking forward to the next diff.

Reviewed commit: 9b59a91791

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[ThomasK33]

@codex security review

[chatgpt-codex-connector]

🛡️ Codex Security Review

Here are some automated security review suggestions for this pull request.

Reviewed commit: 9b59a91791

ℹ️ About Codex security reviews in GitHub

This is an experimental Codex feature. Security reviews are triggered when:

• You comment "@codex security review"
• A regular code review gets triggered (for example, "@codex review" or when a PR is opened), and you’re opted in so security review runs alongside code review

Once complete, Codex will leave suggestions, or a comment if no findings are found.

[ThomasK33]

@codex review

The security finding on 9b59a91 is addressed in 0149387: the KeyNameInUse refusal now deletes the published keys recorded outside the target slot before it returns.

[ThomasK33]

@codex security review

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 0149387c47

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

🛡️ Codex Security Review

Security review completed. No security issues were found in this pull request.

Reviewed commit: 0149387c47

View security finding report

Only the user who started this review can view the report in Codex.

ℹ️ About Codex security reviews in GitHub

This is an experimental Codex feature. Security reviews are triggered when:

• You comment "@codex security review"
• A regular code review gets triggered (for example, "@codex review" or when a PR is opened), and you’re opted in so security review runs alongside code review

Once complete, Codex will leave suggestions, or a comment if no findings are found.

[ThomasK33]

@codex review

Head 90bb3a2. New since 0149387: the applied key check of status without a control plane UID looks the key up by its recorded organization and key IDs; a replacement at the target deletes the recorded target keys by ID before its create; a replaced UID-less entry records the UID of its control plane.

[ThomasK33]

@codex security review

[chatgpt-codex-connector]

Codex Review: Didn't find any major issues. Bravo.

Reviewed commit: 90bb3a2686

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

🛡️ Codex Security Review

Security review completed. No security issues were found in this pull request.

Reviewed commit: 90bb3a2686

View security finding report

Only the user who started this review can view the report in Codex.

ℹ️ About Codex security reviews in GitHub

This is an experimental Codex feature. Security reviews are triggered when:

• You comment "@codex security review"
• A regular code review gets triggered (for example, "@codex review" or when a PR is opened), and you’re opted in so security review runs alongside code review

Once complete, Codex will leave suggestions, or a comment if no findings are found.