Repository navigation
🤖 fix: revoke a CoderProvisioner key created at an intermediate control plane endpoint - #413
Conversation
…s never revoked (#259) Change-Id: Iac0ef846266c12f7aa807a17c973a568476634bc Signed-off-by: Thomas Kosiewski <tk@coder.com>
…urce, and revoke it there (#259) Change-Id: I27dde188ca1aa83839b622858523babd03874997 Signed-off-by: Thomas Kosiewski <tk@coder.com>
…y on by-name deletes, and map an old HTTPS URL to the internal one (#259) Change-Id: I98b10ef7c5fd0e0b25a9022bfb80e6e40fdae5d9 Signed-off-by: Thomas Kosiewski <tk@coder.com>
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 27c324ffed
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
…, retry unconfirmed deletes, and count a deleted organization as gone (#259) Change-Id: I1b838ed30fd0452da76e6649fc2d2146e3f60143 Signed-off-by: Thomas Kosiewski <tk@coder.com>
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 54c982fed3
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
…259) Change-Id: If242534c4c40750c96650d00ada817dc5042c736 Signed-off-by: Thomas Kosiewski <tk@coder.com>
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 77c3441fbc
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
…d keep the current-URL fallback for older status (#259) Change-Id: I28210e78dede73870e4cfdc4e73afed26558139b Signed-off-by: Thomas Kosiewski <tk@coder.com>
|
@codex review |
|
@codex security review |
|
Codex Review: Didn't find any major issues. Keep it up! Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
🛡️ Codex Security ReviewSecurity review completed. No security issues were found in this pull request. Reviewed commit: Only the user who started this review can view the report in Codex. ℹ️ About Codex security reviews in GitHubThis is an experimental Codex feature. Security reviews are triggered when:
Once complete, Codex will leave suggestions, or a comment if no findings are found. |
…er provisioner's same-named key (#259) Change-Id: Id70605da9dff77139df4da0367fc88be2dcdd155 Signed-off-by: Thomas Kosiewski <tk@coder.com>
…ate-key-revoke Change-Id: I56664bb7e0b51c03e11b77f7d8f252b8dd3511e6 Signed-off-by: Thomas Kosiewski <tk@coder.com>
…hout IDs, and report a same-named key with UnverifiedKeyKept (#259) Change-Id: I85db43792ec3049f32ceb5ccc289bd2751f68221 Signed-off-by: Thomas Kosiewski <tk@coder.com>
…h key IDs in the pendingKeys docs (#259) Change-Id: Ibc2f22637712134a1a7c7b6e352d58a373a14a77 Signed-off-by: Thomas Kosiewski <tk@coder.com>
|
@codex review |
|
@codex security review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 3b29933ccb
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
🛡️ Codex Security ReviewSecurity review completed. No security issues were found in this pull request. Reviewed commit: Only the user who started this review can view the report in Codex. ℹ️ About Codex security reviews in GitHubThis is an experimental Codex feature. Security reviews are triggered when:
Once complete, Codex will leave suggestions, or a comment if no findings are found. |
…thout key IDs on each reconcile (#259) Change-Id: I0c64a3588a9c25c94fea84e3d8f8c9cf89fb1453 Signed-off-by: Thomas Kosiewski <tk@coder.com>
|
@codex review |
|
@codex security review |
|
Codex Review: Didn't find any major issues. Delightful! Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
🛡️ Codex Security ReviewSecurity review completed. No security issues were found in this pull request. Reviewed commit: Only the user who started this review can view the report in Codex. ℹ️ About Codex security reviews in GitHubThis is an experimental Codex feature. Security reviews are triggered when:
Once complete, Codex will leave suggestions, or a comment if no findings are found. |
Summary
Goal: every provisioner key that the operator creates in Coder is deleted again, also when the control plane target of a
CoderProvisionerchanges twice before one reconcile finishes.Before this change, a key created at an intermediate endpoint B (target A to B to C, with the attempt at B failing after Coder created the key) stayed valid in Coder. Neither the next rotation nor the deletion of the provisioner sent a delete to B. The operator also sent the operator token of the new control plane to the URL of the old one.
Now
status.pendingKeysrecords every key that can exist in Coder besides the key in the Secret. Each entry names the Coder URL of the key and the control plane whose operator token deletes it. The operator writes the entry before the create request, and removes it only after Coder confirms that the key is gone.An entry without key IDs (the create was sent, but its answer was never recorded) deletes no key. Coder returns nothing else that ties a key to this provisioner, and another
CoderProvisioneror a person can hold a key with the same name at that endpoint and organization. The operator removes such an entry when Coder has no key with its name, or when that key is the applied key. Otherwise, and while the lookup fails, it leaves the key, keeps the entry, and sets the conditionUnverifiedKeyKept=True, whose message names each such entry.Fixes #259
Background
status.controlPlaneURLonly.status.provisionerKeyIDwhilestatus.controlPlaneURLstill named the old endpoint. Deletion compensated by also trying the current URL. A second target change lost the intermediate endpoint.Implementation
status.pendingKeys(CoderProvisionerKeyRecord:controlPlaneRefName,coderURL,organizationName,keyName, optionalorganizationID/keyID). Generated deepcopy, CRD,dist/install.yamland API reference docs are regenerated.provisionerKeyID,controlPlaneURL, ...) now change only together, in the status write after the Secret holds the key. That write drops the target record.status.pendingKeysrequeues after 2 minutes, so an unconfirmed delete is retried without an outside event.revokeKey, outcomekeyKept/keyGone/keyUnverified). No key: the entry goes. The applied key (another URL can reach the same Coder): the entry goes. Any other key, or a failed lookup: the key stays, the entry stays, andUnverifiedKeyKept=Truenames it. Each reconcile rebuilds the condition from all such entries, and removes it when none is left. Deletion of the provisioner leaves such a key in Coder and logs it.unpublishedfrom before its create until a key is recorded as applied. An unpublished record left by a failed reconcile (create answer lost, Secret write failed, or the status write after the Secret write failed) forces a rotation on the next reconcile, also after a spec revert. So the applied fields name the key in the Secret, and the Secret never keeps a key that the operator deleted or that belongs to another endpoint.internal/coderbootstrap:ProvisionerKeyByName(new) returns the key with a name in an organization, or nil when Coder answers that the organization or the key does not exist. It deletes nothing.Resource not found or you do not have access to this resource,coderd/httpapi/httpapi.go:149in v2.37.3). A 404 from a proxy or a wrong base URL is an error inDeleteProvisionerKeyByIDandProvisionerKeyByName.UnverifiedKeyKept. ThependingKeysandorganizationIDfield docs say that an entry without both IDs deletes no key.docs/tutorials/deploy-coderprovisioner.mddescribesstatus.pendingKeysfor deletion and rotation, the retry, how to remove an entry whose control plane is gone, and what to do whenUnverifiedKeyKeptnames a key.Validation
TestCoderProvisionerKey_IntermediateEndpointRevoked(envtest, 4 cases): A to B to C with the attempt at B failing (Secret write fails, or the status write and the revoke fail as after a crash), then a rotation at C or a deletion. It checks that key B is deleted at B with B's token, that every delete uses the token of the control plane whose URL it targets, and that every key create has a record in status first. The first commit adds it; it fails onmain:Before the fix (first commit, b1625aa, on the code of main)
After the fix:
TestCoderProvisionerKey_IDLessRecordKeepsOtherProvisionersKey(envtest, 2 cases, stateful fake Coder with one key per name): provisioner Q holds keyprovisionerat endpoint B. Provisioner P (same key name) moves its target to B, records the create without key IDs, and the create fails before it reaches Coder. Then P moves back to A, or P is deleted. Q's key must still exist at B. The final version of the test moves P through B and C (both creates fail) and back to A. P keeps both entries,UnverifiedKeyKept=Truenames B (same-named key) and C (failed lookup), and P requeues. After Q's key at B is gone, the entry at B and its mention go, while C stays reported. When the lookup at C works again, the entry and the condition go. Commit 488e3f0 adds the test on b25997e, where it fails:Before the fix (488e3f0, on b25997e)
After the fix (45e6405):
Negative controls on 45e6405: putting a by-name delete back into the ID-less branch fails both cases ("the key of Q at B must still exist"). Freezing the condition after a failed lookup, or dropping failed lookups from it, fails the "P moves back to A" case.
TestCoderProvisionerKey_IntermediateEndpointRevokedcases where status never holds the ID of key B (status write and revoke fail, as after a crash) now expect key B to stay at B, the entry to stay, andUnverifiedKeyKept=True. Before this round they expected a by-name delete.TestCoderProvisionerKey_FailedRotationKeepsKeyIDs: a rotation that fails before its create keeps the IDs of the unpublished key.TestCoderProvisionerKey_UnpublishedKeyIsRotated(2 cases): a create whose answer was lost before a spec revert, and a status write that fails after the Secret write. The next reconcile rotates, status names the key in the Secret, and deletion deletes that key. Both cases fail when a leftover target record does not force the rotation.TestProvisionerKeyByName,TestDeleteProvisionerKeyByID: Coder's 404 for the organization or the list means no key, a missing name means no key, and a 404 that Coder did not send is an error.TestCoderProvisionerReconciler_KeyNameDriftnow gives its keys IDs, like Coder does, and checks that the old key is deleted by ID.TestCoderProvisionerKey_RevertAfterFailedRotation(2 cases): a rotation from A to B fails after its create (Secret write fails, or the status write after it fails), and the target goes back to A. The next reconcile deletes key B at B and writes a new key for A. Both cases fail without theunpublishedrule.TestCoderProvisionerKey_RecordedBeforePublishedandTestCoderProvisionerKey_FailedSecretUpdateRotatesAgainto read the record fromstatus.pendingKeys.TestCoderProvisionerDeletion_TriesCurrentControlPlaneURLnow uses HTTP URLs, because an HTTPS URL of the same control plane maps to the same internal URL.make verify-vendor,make build,make lint,make test,make test-integration,make codegen manifests docs-reference build-installer(no further diff),nix develop -c make docs-check,cspellandmarkdownlint-cli2on the changed docs: all pass.Risks
CoderProvisioneror a person, also with the same name. The cost: when a create reached Coder but its answer was lost (or a crash hit between the create and the status write), and the target then moved away, the key stays in Coder.UnverifiedKeyKept=Truenames it while the provisioner exists, and the docs say how to delete it. After the provisioner is deleted, only the operator log names it. Matching by Coder'screated_atagainst the attempt time was rejected: a same-named key of another object created after the attempt would still match, and clock skew can make an older key look newer.mainhas a broader form of this on drift (it sends the current control plane's token tostatus.controlPlaneURL). Tracked in 🤖 CoderProvisioner: bind each key record's operator token to the control plane UID #417.status.pendingKeysis trusted likestatus.controlPlaneURL: whoever can writeCoderProvisionerstatus can direct the operator token of a control plane in the namespace to a URL. Writes to these resources are admin-level, as before.Generated with
xum• Model:anthropic:claude-opus-5-5• Thinking:xhigh