fix(placement): a drained member is where maintenance goes, not where it is refused - #317
Merged
rldyourmnd merged 1 commit intoAug 31, 2026
Conversation
… it is refused A build requires its member to be empty, and the only way to empty one is drain. Drain closes that member's pressure gate. The gate then removed it from every candidate list, so the build reported "no fleet member has room" about a member with zero occupants. Observed live: gha-runner-2 drained to zero, and reconcile-image with --apply --stage-only refused with insufficient-memory naming an empty machine. The orchestrator does pass --target, but the placement scriptlet runs anyway and vetoes what the target selected. Two correct guards, jointly unsatisfiable. The gate exists to keep new work off a busy member; an image builder is not new work, it is the maintenance the member was emptied for. The scriptlet now exempts exactly the two instance names imageplan generates -- gha-image-builder- and gha-image-smoke- -- from the gate, and from nothing else. Memory headroom, the storage pool check and the disk reserve still apply, because a build placed where it does not fit fails later and more expensively. The test asserts both halves: the exemption is present, and the capacity checks it must not weaken are still there. Verified by mutation -- removing the maintenance condition fails it.
rldyourmnd
deleted the
fix/maintenance-placement-survives-a-drained-member
branch
August 31, 2026 18:45
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A build requires its member to be empty. The only way to empty one is
drain-member. Drain closes that member's pressure gate. The gate then removes it from every candidate list — so the build reportsno fleet member has roomabout a member with zero occupants.Observed live
gha-runner-2drained to zero occupants in 25 seconds, no job aborted. Then:The orchestrator does pass
--target, with a comment explaining why. Incus runs the placement scriptlet regardless, and the scriptlet vetoes what the target selected.Two correct guards, jointly unsatisfiable
The pressure gate exists to keep new work off a busy member. An image builder is not new work — it is the maintenance the member was emptied for. Each guard is right on its own; together they make rebuilding an image impossible while the fleet serves.
The change
The scriptlet exempts exactly the two instance names
internal/imageplangenerates —gha-image-builder-andgha-image-smoke-— from the gate, and from nothing else.Memory headroom, the storage pool check and the disk reserve still apply. A build placed where it does not fit fails later and more expensively, so the exemption is narrow by construction.
Verification
TestMaintenanceInstancesIgnoreTheClosedGateasserts both halves: the exemption is present, and the capacity checks it must not weaken are still there. Verified by mutation — restoring the unconditional gate check fails it.