Skip to content

fix(placement): a drained member is where maintenance goes, not where it is refused - #317

Merged
rldyourmnd merged 1 commit into
mainfrom
fix/maintenance-placement-survives-a-drained-member
Aug 31, 2026
Merged

fix(placement): a drained member is where maintenance goes, not where it is refused#317
rldyourmnd merged 1 commit into
mainfrom
fix/maintenance-placement-survives-a-drained-member

Conversation

@rldyourmnd

Copy link
Copy Markdown
Contributor

A build requires its member to be empty. The only way to empty one is drain-member. Drain closes that member's pressure gate. The gate then removes it from every candidate list — so the build reports no fleet member has room about a member with zero occupants.

Observed live

gha-runner-2 drained to zero occupants in 25 seconds, no job aborted. Then:

reconcile golden image: initialize image builder: incus command failed:
Failed instance placement scriptlet: Failed to run:
fail: insufficient-memory: no fleet member has room for this worker

The orchestrator does pass --target, with a comment explaining why. Incus runs the placement scriptlet regardless, and the scriptlet vetoes what the target selected.

Two correct guards, jointly unsatisfiable

The pressure gate exists to keep new work off a busy member. An image builder is not new work — it is the maintenance the member was emptied for. Each guard is right on its own; together they make rebuilding an image impossible while the fleet serves.

The change

The scriptlet exempts exactly the two instance names internal/imageplan generates — gha-image-builder- and gha-image-smoke- — from the gate, and from nothing else.

Memory headroom, the storage pool check and the disk reserve still apply. A build placed where it does not fit fails later and more expensively, so the exemption is narrow by construction.

Verification

TestMaintenanceInstancesIgnoreTheClosedGate asserts both halves: the exemption is present, and the capacity checks it must not weaken are still there. Verified by mutation — restoring the unconditional gate check fails it.

… it is refused

A build requires its member to be empty, and the only way to empty one is
drain. Drain closes that member's pressure gate. The gate then removed it
from every candidate list, so the build reported "no fleet member has room"
about a member with zero occupants.

Observed live: gha-runner-2 drained to zero, and reconcile-image with
--apply --stage-only refused with insufficient-memory naming an empty
machine. The orchestrator does pass --target, but the placement scriptlet
runs anyway and vetoes what the target selected.

Two correct guards, jointly unsatisfiable. The gate exists to keep new
work off a busy member; an image builder is not new work, it is the
maintenance the member was emptied for.

The scriptlet now exempts exactly the two instance names imageplan
generates -- gha-image-builder- and gha-image-smoke- -- from the gate, and
from nothing else. Memory headroom, the storage pool check and the disk
reserve still apply, because a build placed where it does not fit fails
later and more expensively.

The test asserts both halves: the exemption is present, and the capacity
checks it must not weaken are still there. Verified by mutation -- removing
the maintenance condition fails it.
@rldyourmnd
rldyourmnd merged commit 43ead2d into main Aug 31, 2026
10 checks passed
@rldyourmnd
rldyourmnd deleted the fix/maintenance-placement-survives-a-drained-member branch August 31, 2026 18:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant