[ThomasK33]

Summary

Goal: a namespace deletion must not leave Coder resources behind that no one can find again. This PR closes the last open items of #353. It does two things:

1. Fixes an inherited bug in provider records. When status.providerID was empty but the cleanup record held the provider ID (the state after a lost status write), the controller rewrote the record without the ID. A later delete then removed the record and released the finalizer without a DELETE, and the provider stayed in Coder. Codex found it on 🤖 fix: finish the control plane hold and cleanup record follow-ups of #353 #410 (🤖 fix: finish the control plane hold and cleanup record follow-ups of #353 #410 (comment)). The record now keeps its ID unless a new create follows.
2. Adds cleanup records and the sweep for CoderMCPServer. MCP server configs now get the same record as providers and keys. A config that a namespace deletion leaves in Coder is deleted by a later sweep through a control plane of the same Coder deployment, when the record holds the config ID. A record without a config ID never deletes anything. While a config in the organization can still be the object's, the record stays and the operator reports it for manual cleanup.

Every other open item is closed by a decision. The reasons are in the Decisions section.

Fixes #353
Refs #156

Background

#353 tracks what a namespace or control plane deletion can leave in Coder. The main parts shipped in v0.1.5. #410 closed 6 follow-ups and left 7 open. This table covers each open item on main at 93d8854:

Item	On main	This PR
Codex finding on #410: a recorded provider ID is dropped when the status has none	Reproduced: the provider stays in Coder, with no DELETE and no record	Done
#384 item 3: cleanup records cover only providers and keys	Open for MCP servers	Done
#362 item 3: queue backlog at the hold deadline	Covered by records for providers and keys only	Done: an MCP server that releases at the hold deadline now keeps its record too
#362 item 1: operator access is not repaired during the hold	Open	Closed by decision
#362 item 4: a foreground deletion seen after its marker is gone	Open	Closed by decision
#362 item 5: the provider delete path reads the wall clock	Open	Closed by decision
#362 item 6: the deployment check and the delete are two requests	Open	Closed by decision
#362 item 8: a record is lost when a namespace is reused with a fake Coder	Open	Closed by decision

#362 items 2, 7 and 9 and #384 items 1, 2, 4 and 5 were done in #362 and #410.

Implementation

Provider fix (coderaiprovider_record.go): writeAIProviderRecord keeps the previous record's provider ID when the status has none and no create follows. A new create still drops the ID. The controller creates only after Coder answers 404 for the provider name, and Coder cannot rename a provider, so the old ID is gone at that point.

MCP server records follow the provider design and the invariants of option B′ (#353 maintainer decision comment):

1. createMCPServer writes the record before the create marker and the create request. The record holds the deployment ID from /api/v2/buildinfo (read in the same reconcile), the temporary display name (nonce), the pinned organization ID, the slug, and the config ID once known. It holds no credential. A failed record write or deployment ID read sends no create.
2. The record gets the config ID before the sync replaces the nonce: right after the create or the read that confirms it, and once more before the create and sync steps of every reconcile. With only a create marker, that write checks the deployment ID first and stalls with CoderDeploymentChanged on a mismatch, before a slug conflict in another deployment can clear the marker.
3. A non-blocking backfill at the start of check() writes the record of a config from before records existed. It runs before the opt-in, organization and credential checks, so an object that a kill switch disables gets its record too. The kill switches never wait on the new buildinfo read.
4. When Coder answers its own 404 for the config ID, the sync (codermcpserver_apply.go) first checks the record's deployment. A control plane that now reaches another deployment stalls the object with CoderDeploymentChanged, keeps the config ID and creates nothing. Before this check, an applied object reported DeletedExternally in that case (Codex review on this PR).
5. The delete path (codermcpserver_delete.go) reads the record once. It checks that the control plane still reaches the recorded deployment, uses the recorded config ID when the status lost it, and removes the record only after Coder confirms the delete. Every release without a delete keeps the record. With only a create marker, the finalizer behavior of main stays. The record also stays when another config holds the slug, or when a config with the nonce has another slug: a person can have renamed the config of a lost create answer.
6. The sweep (cleanup_sweep.go) gets an MCP kind. It deletes the recorded config ID. A record without a config ID deletes nothing, the same rule that 🤖 fix: revoke a CoderProvisioner key created at an intermediate control plane endpoint #413 adopted for provisioner keys. A person can edit the slug and the display name of any config in Coder (UpdateMCPServerConfigRequest has both), so neither proves that the operator created a config. The sweep lists the recorded organization as a site owner. When it holds no config with the recorded slug and none with the nonce as display name, the record goes. Otherwise the record stays: the sweep sets the gauge coder_k8s_cleanup_records_unmatched, writes a ManualCleanupNeeded Warning Event on the ConfigMap that names the config, logs it, and checks again every 10 minutes. A name or display-name mismatch is never proof of absence. The sweep also watches CoderMCPServer deletions. The sweep kinds now return one of three outcomes (retry, gone, kept) instead of a gone flag.
7. The MCP reconciler needs APIReader and OperatorNamespace, and asserts both at setup. mcpServerWithSlug takes a slug instead of the object, so the sweep can use it.

The how-to manage-mcp-servers.md documents the records, the CoderDeploymentChanged reason, the sweep, and the manual cleanup of a record without a config ID.

Two choices in the fix round go beyond keeping a record when a config has the slug. Both only keep more records and delete nothing more:

• The nonce check under any slug. Coder lets a person change a config's slug, so "no config with that slug" alone does not prove that the config of a lost create answer is gone.
• The live delete path keeps the record in the same cases as the sweep. Before, it removed the record when the slug holder's display name differed from the nonce. That treated a mismatch as proof of absence, like the sweep did.

Codex review round on 6474666 (three P2 findings, all fixed in 22aa163):

• The early release in deleteStep (no status ID, no create marker) no longer removes a record without a config ID. recordMCPServer clears the marker on SlugConflict also when a person renamed the config of a lost create answer. The finalizer still goes at once, and the sweep decides.
• deleteStep backfills the record of a config from before records existed, after it has Coder access and before the delete, so a bounded attempt that fails still leaves a record.
• When the organization lookup stalls (OrganizationNotFound, OrganizationChanged), check() reports CoderDeploymentChanged first if the record names another deployment. The buildinfo read runs only in that failing branch.

Decisions

These items stay without code. Each decision states what users see and why.

• 🤖 fix: hold a deleted CoderControlPlane while AI Gateway keys and providers still use it #362 item 1 (no operator access at deletion). The control plane still holds for up to 10 minutes. A status with operatorAccessReady: false does not prove that no dependent cleanup is in flight: a cleanup that read a valid token before the status flip can still call Coder. TestControlPlaneHoldsForCleanupInFlight covers that case, and 🤖 fix: finish the control plane hold and cleanup record follow-ups of #353 #410 rejected two early releases for this reason. The cost is a delay of at most 10 minutes. A dependent that then releases without a delete keeps its cleanup record, so a sweep can still delete its resource.
• 🤖 fix: hold a deleted CoderControlPlane while AI Gateway keys and providers still use it #362 item 4 (foreground deletion after its marker is gone). Same reason: a missing marker or Deployment does not prove that Coder stopped (an orphan delete or a cache miss looks the same, TestControlPlaneHoldsWithoutCoderDeployment). The cost is the same bounded delay, and records cover what stays.
• 🤖 fix: hold a deleted CoderControlPlane while AI Gateway keys and providers still use it #362 item 5 (provider wall clock). No user sees a difference: time.Now() and the real clock return the same time. The hold end lives in shared functions (controlPlaneHolds, controlPlaneHoldEnd, controlPlaneHoldRequeue). TestControlPlaneHoldEnd drives them with a fake clock through the key path, and TestAIProviderDeleteUnderHoldEndsAtHoldEnd covers the provider pass at the hold end. A provider clock field would have no test that needs it.
• 🤖 fix: hold a deleted CoderControlPlane while AI Gateway keys and providers still use it #362 item 6 (deployment check and delete are two requests). No request binds a DELETE answer to a database, so a second buildinfo read after the delete narrows the window but does not close it. That read can reach a pod on the old database too. Closing it needs a control plane rollout change (recreate the Deployment on a database change), which adds downtime and is outside the cleanup code. The window needs a rolling update that changes the control plane database, during a delete. The result is a resource left in the database that the control plane left, without a record. Nothing is deleted by mistake: Coder IDs are UUIDs.
• 🤖 fix: hold a deleted CoderControlPlane while AI Gateway keys and providers still use it #362 item 8 (namespace reused with a fake Coder). To drop a record, someone must create the namespace again with a CoderControlPlane whose Coder answers the recorded deployment ID and then a 404. That person already runs a Coder image of their choice behind the operator's access. The sweep sends that control plane's own token only to that server, and it cannot delete anything in the real Coder. The result is the v0.1.4 behavior: a resource stays in the real Coder without a record. A fix needs a trust anchor that survives a namespace recreation. The normal recovery also recreates the namespace, so a control plane UID check would block it, and the Namespace deletion leaves AI Gateway keys and AI providers in Coder #353 design rules out URL matching.

Validation

• Test first. TestCoderAIProviderKeepsRecordedIDWithoutStatusID landed as its own commit (e0cf501) before the fix. On main the record ID became empty. With the fix, the record keeps the ID and the delete sends DELETE for it.
• New MCP tests, each a behavior that users can observe:
  • TestCoderMCPServerCleanupRecord, TestCoderMCPServerNoCreateWithoutRecord, TestCoderMCPServerRecordKeptWithoutDelete, TestCoderMCPServerPendingCreateAfterDeploymentSwitch, TestCoderMCPServerKeepsRecordedIDWithoutStatusID, TestCoderMCPServerRecordBackfill, TestCoderMCPServerRecordBackfillOnDisable, TestCoderMCPServerRecordAfterLostCreateAnswer, TestCoderMCPServerRecreateDropsGoneID, TestCoderMCPServerDeploymentSwitchKeepsRecord, TestCoderMCPServerDeleteRecordEdgeCases, TestCoderMCPServerSetupNeedsRecordDependencies.
  • TestCleanupSweepMCPServer (settle window, NoMatchingDeployment Event and gauge, sweep through a new control plane of the same deployment, swept counter) and TestCleanupSweepMCPServerLivingObject (living object untouched).
  • TestCleanupSweepMCPServerWithoutConfigID: a record without a config ID never sends a DELETE. The record goes after a create that never committed. It stays with a ManualCleanupNeeded Event for a person's config with the slug, for a person's config that copies the temporary display name, and for the config of a lost create answer after a person edits its display name or its slug. TestCleanupSweepMCPServerManualCleanup: the kept record counts in the gauge, and after a person deletes the config, the next sweep removes the record and the gauge drops to 0. TestCoderMCPServerDeletePendingCreate now also asserts which delete outcomes keep the record.
  • Fix round negative control: the new tests (commit 62da960) run against the exact old head 4c54383. The independent verifier's two probes fail there as their table cases: a person's config with the temporary display name sends 1 DELETE (expected 0), and lost create answer, display name edited removes the record (RequeueAfter 0s, expected 10m). The slug-edit, person's-slug and live-path cases fail too. create never committed passes there and on the fix. All pass on 6474666.
  • Codex round tests (commit 5c4eadd, each failed before 22aa163): TestCoderMCPServerDeleteKeepsRecordAfterSlugConflict, TestCoderMCPServerRecordBackfillOnDelete, and TestCoderMCPServerDeploymentSwitchKeepsRecord/organization not in the new one.
• I ran every new test against main with only the two new reconciler fields added, so that they compile. All fail for behavior reasons (no record, no MCP sweep kind, no setup check), except two that guard new code and pass on main: TestCoderMCPServerDeleteRecordEdgeCases/foreign ConfigMap and TestCoderMCPServerRecreateDropsGoneID (main has no record to drop).
• Negative controls: I removed, one at a time, the code that each review-round test covers. Each test then failed: the backfill call, the record write after a confirmed create, the ID drop on a new create, the organization fallback, the foreign ConfigMap skip, the deployment check before a create, and the deployment check on a 404 in the sync (the applied subtest of TestCoderMCPServerDeploymentSwitchKeepsRecord then sees DeletedExternally). Removing the provider fix fails its test too.
• Changed test setup: pendingCreate in codermcpserver_delete_test.go now clears the record's config ID, because a real pending create has none. mcpRequests leaves out buildinfo reads, which are not MCP requests.
• Two independent pre-PR reviewers found no blocker. I fixed their should-fix items: the backfill for objects that a check stalls, the organization fallback for a status that lost everything, and 5 missing tests.
• Local gates on 22aa163, after the merge of main at 968c155 (🤖 fix: map uncertain aggregated write answers to 504 and list unconfigured namespaces as empty #400): make verify-vendor, make test, make build, make lint, make test-integration, make docs-check (through nix develop), plus markdownlint-cli2 and cspell on the docs.

Risks

• While an MCP create is pending (ConfirmingCreate, every 5 seconds), each reconcile reads buildinfo once more and rewrites an unchanged record. Every MCP reconcile reads the record live once or twice. Providers do the same.
• An object whose status lost its config ID stalls with SlugConflict for its own config, because only the record proves ownership. Its delete still uses the recorded ID. Providers behave the same with NameConflict. A follow-up can accept the recorded ID as proof.
• The sweep requires a site owner before it deletes a recorded config ID, as for providers. A record behind an operator user that is not a site owner waits without the unmatched gauge.
• A record without a config ID now never deletes a config. When a create committed but its answer was lost, and the object then went without a delete (for example in a namespace deletion), the config stays in Coder until a person deletes it. The ManualCleanupNeeded Event and the gauge report it, and the how-to has the steps. This is the cost of the rule that 🤖 fix: revoke a CoderProvisioner key created at an intermediate control plane endpoint #413 adopted for provisioner keys.
• A person's config that holds the slug keeps such a record, its Event every 10 minutes and the gauge count until a person deletes the ConfigMap. This happens only when the object went with a create that the operator could not confirm.
• A deleted object whose create failed for sure (for example CoderRejected) now leaves its record to the sweep instead of removing it at delete time. The sweep removes it after the settle window when no config has the slug or the nonce. Without a ready control plane of that deployment in the namespace, the record stays with NoMatchingDeployment, as any record does.
• Residual: a person edits both the slug and the display name of the config of a lost create answer. No field then links the config to the record. The record goes, and the config stays in Coder without a record. Coder has no immutable creator field for MCP configs that could close this.
• Not changed (Codex round 2 on 22aa163, 🤖 fix: keep recorded provider IDs and add cleanup records for MCP servers #419 (comment)): a config from before records existed can still be left without a record. That needs its first reconcile after the upgrade during a control plane deletion, plus a failed buildinfo read or record write and then a failed DELETE. That pass releases after one bounded attempt by design, so a namespace deletion finishes, and a record needs Coder's deployment ID. releaseWithoutDelete logs the slug and the config ID and counts the release, as on main.
• Choices beyond the brief (see Implementation): the nonce check under any slug, and the record kept on the live delete path. Both keep more records and delete nothing more.
• Inherited from main, not fixed here: the CoderAIProvider sweep without a recorded ID still deletes a provider whose name, display name and type match the record, and the MCP live delete path still deletes a config whose slug and display name match the nonce. The independent verifier says these rely on the same assumption that a person cannot copy the display name. The maintainer will file a follow-up.
• Follow-up, not changed here: an applied CoderAIProvider whose control plane now reaches another deployment still reports DeletedExternally, not CoderDeploymentChanged, as before this PR. Its record and delete path already handle the switch, so nothing leaks. The MCP check above can be ported to providerGone.
• 🤖 fix: show Coder's enabled flag for CoderMCPServer and keep observedGeneration with its conditions #418 (the CoderMCPServer: status-based ENABLED column and observedGeneration only with fresh conditions #415 fix) merged while this PR was open. It changed CoderMCPServer status and printer code. This PR changes no status write or printer column, and the branch merges main at 2054f14 without conflicts.

---

Generated with mux • Model: anthropic:claude-opus-5-5 • Thinking: xhigh

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-10-30T19:59:24.086324Z	22aa163	Manual request
🔒 Security Review	✅ Completed 2026-10-30T20:00:07.085888Z	22aa163	Manual request

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: ece19e1fc9

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[ThomasK33]

@codex review

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 4c543839b9

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[ThomasK33]

@codex review

The finding on codermcpserver_delete.go is answered in its thread: the create marker write persists the cleared config ID before the create, so the stored status cannot keep the old ID.

[ThomasK33]

@codex security review

[chatgpt-codex-connector]

Codex Review: Didn't find any major issues. Can't wait for the next one!

Reviewed commit: 4c543839b9

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

🛡️ Codex Security Review

Security review completed. No security issues were found in this pull request.

Reviewed commit: 4c543839b9

View security finding report

Only the user who started this review can view the report in Codex.

ℹ️ About Codex security reviews in GitHub

This is an experimental Codex feature. Security reviews are triggered when:

• You comment "@codex security review"
• A regular code review gets triggered (for example, "@codex review" or when a PR is opened), and you’re opted in so security review runs alongside code review

Once complete, Codex will leave suggestions, or a comment if no findings are found.

[ThomasK33]

@codex review

[ThomasK33]

@codex security review

[chatgpt-codex-connector]

🛡️ Codex Security Review

Security review completed. No security issues were found in this pull request.

Reviewed commit: 6474666aa5

View security finding report

Only the user who started this review can view the report in Codex.

ℹ️ About Codex security reviews in GitHub

This is an experimental Codex feature. Security reviews are triggered when:

• You comment "@codex security review"
• A regular code review gets triggered (for example, "@codex review" or when a PR is opened), and you’re opted in so security review runs alongside code review

Once complete, Codex will leave suggestions, or a comment if no findings are found.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 6474666aa5

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[ThomasK33]

@codex review

[ThomasK33]

@codex security review

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 22aa163eca

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

🛡️ Codex Security Review

Security review completed. No security issues were found in this pull request.

Reviewed commit: 22aa163eca

View security finding report

Only the user who started this review can view the report in Codex.

ℹ️ About Codex security reviews in GitHub

This is an experimental Codex feature. Security reviews are triggered when:

• You comment "@codex security review"
• A regular code review gets triggered (for example, "@codex review" or when a PR is opened), and you’re opted in so security review runs alongside code review

Once complete, Codex will leave suggestions, or a comment if no findings are found.