Skip to content

🤖 fix: revoke a CoderProvisioner key created at an intermediate control plane endpoint - #413

Merged
ThomasK33 merged 11 commits into
mainfrom
provisioner-intermediate-key-revoke
Oct 30, 2026
Merged

ThomasK33 merged 11 commits into
mainfrom
provisioner-intermediate-key-revoke

Conversation

@ThomasK33

@ThomasK33 ThomasK33 commented Oct 30, 2026 •

Copy link
Copy Markdown
Member

Summary

Goal: every provisioner key that the operator creates in Coder is deleted again, also when the control plane target of a CoderProvisioner changes twice before one reconcile finishes.

Before this change, a key created at an intermediate endpoint B (target A to B to C, with the attempt at B failing after Coder created the key) stayed valid in Coder. Neither the next rotation nor the deletion of the provisioner sent a delete to B. The operator also sent the operator token of the new control plane to the URL of the old one.

Now status.pendingKeys records every key that can exist in Coder besides the key in the Secret. Each entry names the Coder URL of the key and the control plane whose operator token deletes it. The operator writes the entry before the create request, and removes it only after Coder confirms that the key is gone.

An entry without key IDs (the create was sent, but its answer was never recorded) deletes no key. Coder returns nothing else that ties a key to this provisioner, and another CoderProvisioner or a person can hold a key with the same name at that endpoint and organization. The operator removes such an entry when Coder has no key with its name, or when that key is the applied key. Otherwise, and while the lookup fails, it leaves the key, keeps the entry, and sets the condition UnverifiedKeyKept=True, whose message names each such entry.

Fixes #259

Background

  • 🤖 fix: delete only the Coder provisioner key recorded in status #248 made deletion target the key IDs in status. Drift rotation still deleted the old key by name at status.controlPlaneURL only.
  • The early status write (before the Secret write) put the new key's IDs into status.provisionerKeyID while status.controlPlaneURL still named the old endpoint. Deletion compensated by also trying the current URL. A second target change lost the intermediate endpoint.
  • The issue scopes this as best-effort cleanup: finalizer removal never waits for Coder.

Implementation

  1. API: new status.pendingKeys (CoderProvisionerKeyRecord: controlPlaneRefName, coderURL, organizationName, keyName, optional organizationID/keyID). Generated deepcopy, CRD, dist/install.yaml and API reference docs are regenerated.
  2. Reconcile, before any step that can create a key (drift, Secret loss, metadata backfill, or a leftover record in the target slot):
    • writes a record without IDs for the target slot (Coder API URL, organization, key name);
    • on drift, adds the replaced applied key as a record at the Coder API URL of its own control plane, unless it is in the target slot (then it is deleted by name at the target, as before);
    • deletes every record with key IDs outside the target slot at its own URL with its own control plane's operator token, and keeps the records that Coder did not confirm (records without IDs: item 5);
    • keeps the newest target record with key IDs until a create at the target proves its key gone (Coder allows one key per name);
    • writes status once, before the create.
  3. After a create, the early status write puts the new key's IDs into its pending record. The applied fields (provisionerKeyID, controlPlaneURL, ...) now change only together, in the status write after the Secret holds the key. That write drops the target record.
  4. A reconcile that ends with entries in status.pendingKeys requeues after 2 minutes, so an unconfirmed delete is retried without an outside event.
  5. An entry without key IDs outside the target slot is only looked up by name (revokeKey, outcome keyKept/keyGone/keyUnverified). No key: the entry goes. The applied key (another URL can reach the same Coder): the entry goes. Any other key, or a failed lookup: the key stays, the entry stays, and UnverifiedKeyKept=True names it. Each reconcile rebuilds the condition from all such entries, and removes it when none is left. Deletion of the provisioner leaves such a key in Coder and logs it.
  6. A record is unpublished from before its create until a key is recorded as applied. An unpublished record left by a failed reconcile (create answer lost, Secret write failed, or the status write after the Secret write failed) forces a rotation on the next reconcile, also after a spec revert. So the applied fields name the key in the Secret, and the Secret never keeps a key that the operator deleted or that belongs to another endpoint.
  7. Deletion deletes every pending record with key IDs and the applied key, each at its recorded URL with its own control plane's token. For status that an older operator wrote (new key IDs under the old URL), it also tries the applied key ID at the current endpoint with the current control plane's token, as 🤖 fix: delete only the Coder provisioner key recorded in status #248 did.
  8. internal/coderbootstrap:
    • ProvisionerKeyByName (new) returns the key with a name in an organization, or nil when Coder answers that the organization or the key does not exist. It deletes nothing.
    • A delete or a lookup counts as "gone" only on Coder's own 404 (Resource not found or you do not have access to this resource, coderd/httpapi/httpapi.go:149 in v2.37.3). A 404 from a proxy or a wrong base URL is an error in DeleteProvisionerKeyByID and ProvisionerKeyByName.
  9. API: new condition type UnverifiedKeyKept. The pendingKeys and organizationID field docs say that an entry without both IDs deletes no key.
  10. Docs: docs/tutorials/deploy-coderprovisioner.md describes status.pendingKeys for deletion and rotation, the retry, how to remove an entry whose control plane is gone, and what to do when UnverifiedKeyKept names a key.

Validation

  • TestCoderProvisionerKey_IntermediateEndpointRevoked (envtest, 4 cases): A to B to C with the attempt at B failing (Secret write fails, or the status write and the revoke fail as after a crash), then a rotation at C or a deletion. It checks that key B is deleted at B with B's token, that every delete uses the token of the control plane whose URL it targets, and that every key create has a record in status first. The first commit adds it; it fails on main:
Before the fix (first commit, b1625aa, on the code of main)
--- FAIL: TestCoderProvisionerKey_IntermediateEndpointRevoked (0.00s)
    --- FAIL: .../key_B_unrecorded_by_ID,_deletion_at_C (2.14s)
        Messages: key B must be deleted at B with the operator token of B, got [{coderURL:https://coder.example.com token:operator-token-b keyName:provisioner keyID:} {coderURL:https://coder.example.com token:operator-token-c keyName: keyID:a22038cc-...} {coderURL:http://coder-c.<ns>.svc.cluster.local:80 token:operator-token-c ...}]
    --- FAIL: .../secret_write_fails_at_B,_deletion_at_C (2.14s)
        Messages: key B must be deleted at B with the operator token of B, got [{coderURL:https://coder.example.com token:operator-token-b ...} ...]
    --- FAIL: .../key_B_unrecorded_by_ID,_rotation_at_C (2.15s)
        Messages: key B must be deleted at B with the operator token of B, got [{coderURL:https://coder.example.com token:operator-token-b keyName:provisioner keyID:} {coderURL:https://coder.example.com token:operator-token-c keyName:provisioner keyID:}]
    --- FAIL: .../secret_write_fails_at_B,_rotation_at_C (2.15s)
        Messages: key B must be deleted at B with the operator token of B, got [...]

After the fix:

--- PASS: TestCoderProvisionerKey_IntermediateEndpointRevoked (0.00s)
    --- PASS: .../key_B_unrecorded_by_ID,_deletion_at_C (2.14s)
    --- PASS: .../secret_write_fails_at_B,_deletion_at_C (2.14s)
    --- PASS: .../key_B_unrecorded_by_ID,_rotation_at_C (2.16s)
    --- PASS: .../secret_write_fails_at_B,_rotation_at_C (2.17s)
  • TestCoderProvisionerKey_IDLessRecordKeepsOtherProvisionersKey (envtest, 2 cases, stateful fake Coder with one key per name): provisioner Q holds key provisioner at endpoint B. Provisioner P (same key name) moves its target to B, records the create without key IDs, and the create fails before it reaches Coder. Then P moves back to A, or P is deleted. Q's key must still exist at B. The final version of the test moves P through B and C (both creates fail) and back to A. P keeps both entries, UnverifiedKeyKept=True names B (same-named key) and C (failed lookup), and P requeues. After Q's key at B is gone, the entry at B and its mention go, while C stays reported. When the lookup at C works again, the entry and the condition go. Commit 488e3f0 adds the test on b25997e, where it fails:
Before the fix (488e3f0, on b25997e)
        	Messages:   	the key of Q at B must still exist
        	Messages:   	the key of Q at B must still exist
--- FAIL: TestCoderProvisionerKey_IDLessRecordKeepsOtherProvisionersKey (0.00s)
    --- FAIL: TestCoderProvisionerKey_IDLessRecordKeepsOtherProvisionersKey/P_is_deleted (2.18s)
    --- FAIL: TestCoderProvisionerKey_IDLessRecordKeepsOtherProvisionersKey/P_moves_back_to_A (2.20s)
FAIL

After the fix (45e6405):

--- PASS: TestCoderProvisionerKey_IDLessRecordKeepsOtherProvisionersKey (0.00s)
    --- PASS: TestCoderProvisionerKey_IDLessRecordKeepsOtherProvisionersKey/P_is_deleted (2.28s)
    --- PASS: TestCoderProvisionerKey_IDLessRecordKeepsOtherProvisionersKey/P_moves_back_to_A (2.38s)

Negative controls on 45e6405: putting a by-name delete back into the ID-less branch fails both cases ("the key of Q at B must still exist"). Freezing the condition after a failed lookup, or dropping failed lookups from it, fails the "P moves back to A" case.

  • The two TestCoderProvisionerKey_IntermediateEndpointRevoked cases where status never holds the ID of key B (status write and revoke fail, as after a crash) now expect key B to stay at B, the entry to stay, and UnverifiedKeyKept=True. Before this round they expected a by-name delete.
  • The same test has a fifth case: the first delete of key B at C fails, the reconcile requeues, the record stays, and the next reconcile deletes the key. It also checks that key A is deleted at A with A's token, and that nothing deletes key C, which the Secret holds.
  • TestCoderProvisionerKey_FailedRotationKeepsKeyIDs: a rotation that fails before its create keeps the IDs of the unpublished key.
  • TestCoderProvisionerKey_UnpublishedKeyIsRotated (2 cases): a create whose answer was lost before a spec revert, and a status write that fails after the Secret write. The next reconcile rotates, status names the key in the Secret, and deletion deletes that key. Both cases fail when a leftover target record does not force the rotation.
  • TestProvisionerKeyByName, TestDeleteProvisionerKeyByID: Coder's 404 for the organization or the list means no key, a missing name means no key, and a 404 that Coder did not send is an error.
  • TestCoderProvisionerReconciler_KeyNameDrift now gives its keys IDs, like Coder does, and checks that the old key is deleted by ID.
  • TestCoderProvisionerKey_RevertAfterFailedRotation (2 cases): a rotation from A to B fails after its create (Secret write fails, or the status write after it fails), and the target goes back to A. The next reconcile deletes key B at B and writes a new key for A. Both cases fail without the unpublished rule.
  • Updated TestCoderProvisionerKey_RecordedBeforePublished and TestCoderProvisionerKey_FailedSecretUpdateRotatesAgain to read the record from status.pendingKeys. TestCoderProvisionerDeletion_TriesCurrentControlPlaneURL now uses HTTP URLs, because an HTTPS URL of the same control plane maps to the same internal URL.
  • Local gates on 45e6405: make verify-vendor, make build, make lint, make test, make test-integration, make codegen manifests docs-reference build-installer (no further diff), nix develop -c make docs-check, cspell and markdownlint-cli2 on the changed docs: all pass.

Risks

  • An entry without key IDs never deletes a key, so the cleanup cannot remove a key of another CoderProvisioner or a person, also with the same name. The cost: when a create reached Coder but its answer was lost (or a crash hit between the create and the status write), and the target then moved away, the key stays in Coder. UnverifiedKeyKept=True names it while the provisioner exists, and the docs say how to delete it. After the provisioner is deleted, only the operator log names it. Matching by Coder's created_at against the attempt time was rejected: a same-named key of another object created after the attempt would still match, and clock skew can make an older key look newer.
  • While another object's same-named key exists at that endpoint, the entry stays and the condition stays (a false alarm in that case), and the operator looks the name up every 2 minutes.
  • Off-target entries are deleted only after the current target passes its checks (control plane, credentials, operator token, entitlement). While the current target fails them, a key recorded at an older endpoint stays valid. Its record stays, and deletion of the provisioner deletes it. Tracked in 🤖 CoderProvisioner: keys recorded at an older endpoint are not deleted while the current target fails its checks #416.
  • A record names its token source by control plane name. If a control plane is deleted and created again under the same name with another URL, a delete sends the new token to the old URL. main has a broader form of this on drift (it sends the current control plane's token to status.controlPlaneURL). Tracked in 🤖 CoderProvisioner: bind each key record's operator token to the control plane UID #417.
  • Pre-existing, not changed here: the rotation paths at the target (drift, Secret loss, metadata backfill) still delete a same-named key by name when Coder returns it without plaintext, so two provisioners with the same key name at one endpoint take over each other's key. Tracked in 🤖 CoderProvisioner: a rotation at the target deletes another provisioner's same-named key #414.
  • An unpublished record costs one extra rotation after a failed reconcile.
  • status.pendingKeys is trusted like status.controlPlaneURL: whoever can write CoderProvisioner status can direct the operator token of a control plane in the namespace to a URL. Writes to these resources are admin-level, as before.
  • A record whose control plane is gone, or whose endpoint keeps failing, stays in status and is retried every 2 minutes. Each retry against a dead endpoint can wait up to the 30-second Coder request timeout. Deletion still removes the finalizer.
  • A failed rotation followed by a revert now costs a second rotation (a new key and a Deployment rollout). Before, the Secret kept a deleted or wrong-endpoint key.

Generated with xum • Model: anthropic:claude-opus-5-5 • Thinking: xhigh

…s never revoked (#259)

Change-Id: Iac0ef846266c12f7aa807a17c973a568476634bc
Signed-off-by: Thomas Kosiewski <tk@coder.com>
…urce, and revoke it there (#259)

Change-Id: I27dde188ca1aa83839b622858523babd03874997
Signed-off-by: Thomas Kosiewski <tk@coder.com>
…y on by-name deletes, and map an old HTTPS URL to the internal one (#259)

Change-Id: I98b10ef7c5fd0e0b25a9022bfb80e6e40fdae5d9
Signed-off-by: Thomas Kosiewski <tk@coder.com>
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 30, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-30T17:22:01.991242Z 45e6405 Manual request
🔒 Security Review ✅ Completed 2026-10-30T17:23:46.260898Z 45e6405 Manual request
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 27c324ffed

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

…, retry unconfirmed deletes, and count a deleted organization as gone (#259)

Change-Id: I1b838ed30fd0452da76e6649fc2d2146e3f60143
Signed-off-by: Thomas Kosiewski <tk@coder.com>
@ThomasK33

Copy link
Copy Markdown
Member Author

@codex review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 54c982fed3

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

…259)

Change-Id: If242534c4c40750c96650d00ada817dc5042c736
Signed-off-by: Thomas Kosiewski <tk@coder.com>
@ThomasK33

Copy link
Copy Markdown
Member Author

@codex review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 77c3441fbc

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

…d keep the current-URL fallback for older status (#259)

Change-Id: I28210e78dede73870e4cfdc4e73afed26558139b
Signed-off-by: Thomas Kosiewski <tk@coder.com>
@ThomasK33

Copy link
Copy Markdown
Member Author

@codex review

@ThomasK33

Copy link
Copy Markdown
Member Author

@codex security review

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Keep it up!

Reviewed commit: b25997ec36

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@chatgpt-codex-connector

Copy link
Copy Markdown

🛡️ Codex Security Review

Security review completed. No security issues were found in this pull request.

Reviewed commit: b25997ec36

View security finding report

Only the user who started this review can view the report in Codex.

ℹ️ About Codex security reviews in GitHub

This is an experimental Codex feature. Security reviews are triggered when:

  • You comment "@codex security review"
  • A regular code review gets triggered (for example, "@codex review" or when a PR is opened), and you’re opted in so security review runs alongside code review

Once complete, Codex will leave suggestions, or a comment if no findings are found.

…er provisioner's same-named key (#259)

Change-Id: Id70605da9dff77139df4da0367fc88be2dcdd155
Signed-off-by: Thomas Kosiewski <tk@coder.com>
…ate-key-revoke

Change-Id: I56664bb7e0b51c03e11b77f7d8f252b8dd3511e6
Signed-off-by: Thomas Kosiewski <tk@coder.com>
…hout IDs, and report a same-named key with UnverifiedKeyKept (#259)

Change-Id: I85db43792ec3049f32ceb5ccc289bd2751f68221
Signed-off-by: Thomas Kosiewski <tk@coder.com>
…h key IDs in the pendingKeys docs (#259)

Change-Id: Ibc2f22637712134a1a7c7b6e352d58a373a14a77
Signed-off-by: Thomas Kosiewski <tk@coder.com>
@ThomasK33

Copy link
Copy Markdown
Member Author

@codex review

@ThomasK33

Copy link
Copy Markdown
Member Author

@codex security review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 3b29933ccb

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@chatgpt-codex-connector

Copy link
Copy Markdown

🛡️ Codex Security Review

Security review completed. No security issues were found in this pull request.

Reviewed commit: 3b29933ccb

View security finding report

Only the user who started this review can view the report in Codex.

ℹ️ About Codex security reviews in GitHub

This is an experimental Codex feature. Security reviews are triggered when:

  • You comment "@codex security review"
  • A regular code review gets triggered (for example, "@codex review" or when a PR is opened), and you’re opted in so security review runs alongside code review

Once complete, Codex will leave suggestions, or a comment if no findings are found.

…thout key IDs on each reconcile (#259)

Change-Id: I0c64a3588a9c25c94fea84e3d8f8c9cf89fb1453
Signed-off-by: Thomas Kosiewski <tk@coder.com>
@ThomasK33

Copy link
Copy Markdown
Member Author

@codex review

@ThomasK33

Copy link
Copy Markdown
Member Author

@codex security review

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Delightful!

Reviewed commit: 45e64056a9

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@chatgpt-codex-connector

Copy link
Copy Markdown

🛡️ Codex Security Review

Security review completed. No security issues were found in this pull request.

Reviewed commit: 45e64056a9

View security finding report

Only the user who started this review can view the report in Codex.

ℹ️ About Codex security reviews in GitHub

This is an experimental Codex feature. Security reviews are triggered when:

  • You comment "@codex security review"
  • A regular code review gets triggered (for example, "@codex review" or when a PR is opened), and you’re opted in so security review runs alongside code review

Once complete, Codex will leave suggestions, or a comment if no findings are found.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

🤖 CoderProvisioner: revoke a key created at an intermediate control plane endpoint

1 participant