M1 jobs and scheduling: steps 2–6 (fake executor, scheduler core, job wrapper, K3s executor, job API) by BrandonwLii · Pull Request #131 · UTMIST/Mist

BrandonwLii · 2026-10-30T18:31:33Z

Stacked on #130. This is a draft until M1 is finished. It implements steps 2–6 of docs/agents/plans/jobs-and-scheduling.md, one commit per step. The plan has a status entry for each step.

Steps

Step 2: fake clock and fake executor (internal/clock, internal/executor/fake). The fake executor can inject faults: lost create responses, outages, held, dropped or duplicated events, lost nodes and failed cleanup.
Step 3: scheduler core (internal/scheduler).
- Limits, placement, and one transaction per admission.
- Durable operation intents, fenced by the leader epoch.
- Deduplicated event handling and reconciliation.
- Allocations are released only when a node confirms cleanup.
Step 4: mist-job-entry.
- Opens the output file without letting it escape the team folder.
- Forwards signals to the job's process group.
- Detects out-of-memory kills from cgroup data.
- Reports setup failures with exit code 125.
Step 5: K3s executor (internal/executor/k3s, deploy/k3s/).
- Pods built by a pure function, pinned by golden-file tests.
- Strict supplemental groups.
- A network-isolation gate before user code runs. NetworkPolicy is applied about 1 s after a container starts; see executors.md.
- Durable execution records and a node-side cleanup check.
Step 6: job API (internal/api).
- Submit with database defaults and a required Idempotency-Key.
- List, get and cancel.
- Logs: the stored tail, or a live WebSocket stream.
- The queue (anonymised) and partitions.
- An internal endpoint for node cleanup receipts, authenticated with host tokens.
- Production builds reject the development identity.

Testing

go test ./... and go test -tags mistdev ./internal/api ./cmd/... pass. The database tests use embedded Postgres.
scripts/k3d-test.sh passes all 9 tests on k3s v1.36.5 with containerd 2.3.4, running as mistd's own service account. The tests cover:
- isolation;
- SIGTERM forwarding;
- out-of-memory kills;
- time limits;
- setup failures;
- executor restart;
- narrow RBAC;
- two end-to-end runs. The second goes through the HTTP API, and the node posts its own cleanup receipt.
I broke security checks on purpose to confirm the scheduler and API tests catch the failures.
The results are written up in docs/agents/evidence/m1-step5-k3d.md and the plan's status entries.

Needs review from

Lucas (Identity): the internal/identity/identity.go contract. Also how the session and relay authenticators plug into api.Server.Accounts (api.md#implementation-notes-m1). Until they do, mistd api fails closed. Host-token issuing also belongs to Identity.

Not yet done (steps 7–9)

Real-cluster deployment, GPUs, NFS, the host firewall, TLS on the direct link, running the cleanup reporter on nodes, and conformance tests on the real hardware.

🤖 Generated with Claude Code

clock.Fake fires timers and AfterFunc callbacks synchronously in time order during Advance, so executor and scheduler tests are deterministic. executor/fake runs scripted jobs on any clock and injects the failures real executors must survive: lost and late create responses, OOM, launch errors and timeouts, the executor backstop, SIGTERM vs SIGKILL, lost and recovered nodes, failed cleanup and observation outages. Its durable receipts are separate from an event stream that tests can hold, drop, duplicate, reorder or lose on restart. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…step 3) Scheduler tables (partitions, qos, limits, jobs, job_specs, job_events, allocations, nodes, op_intents, consumed_events) with the proposed M1 defaults. The cycle refreshes nodes, applies durable revocations, reconciles List against the database, enforces walltime from started_at and launch deadlines, admits FIFO-within-QoS with user/team/partition limits and partition caps, then dispatches committed intents. Allocations are released only on a cleanup receipt; lost executions keep them quarantined. A leader epoch fences writes from a deposed scheduler. A never-sent launch intent proves nothing exists, so cancelling it frees capacity at once; sent ones are resent with the same ExecutionID. Tests run against Postgres with the fake executor and fake clock. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

The in-job wrapper: umask 0027, symlink-safe private dot-folders, output opened with openat2(RESOLVE_BENEATH) under the team root, header and footer, output copied to the file and stdout, SIGTERM forwarding to the job's process group, a local deadline timer with SIGKILL after grace, OOM detection from the cgroup's memory.events, and a structured termination message. Setup failures exit 125 without running the work. Process tests re-execute the test binary as the wrapper; the OOM test runs inside a 64 MiB systemd user scope. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Pod builder (pure, golden-tested) for plain Pods and immutable ConfigMaps: Mist's nodeName, strict supplemental groups, no service-account token, read-only root, requests = limits, per-job passwd/group, NFS or hostPath team mounts, dnsPolicy None. Executor on client-go informers and a work queue, with durable receipts in k3s_executions: repeat-safe Launch that adopts ambiguous creates and never recreates a recorded Pod, Terminate by shortening activeDeadlineSeconds so the final status survives, results that prefer kubelet observations (OOMKilled, eviction, deadline) over the wrapper, log tail saved before deletion, Lost on outside deletion or 2 min NotReady, and Exited only on a matching node cleanup receipt. k3d testing found that NetworkPolicy is programmed about a second after containers start; every job Pod now waits in a network-gate init container until the node's kubelet is unreachable. Also: deploy/k3s namespace, NetworkPolicy and RBAC, mistd cleanup-report, scripts/k3d-test.sh, and evidence in docs/agents/evidence. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

internal/api serves submit (database defaults, validation, required Idempotency-Key), list, get, cancel, log (tail and WebSocket follow), the anonymised queue and partitions, plus the host-token authenticated cleanup receipt endpoint. mistd cleanup-report -post delivers receipts. Production builds fail closed until Identity's authenticators land, and reject the development user header; -dev-fixture exists only with -tags mistdev. M1-09 is covered by unit tests on the fixture and by a k3d test that runs submit-to-COMPLETED through the API. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

github-actions · 2026-10-30T18:31:49Z

This PR exceeds the recommended size of 1000 lines. Please make sure you are NOT addressing multiple issues with one PR. Note this PR might be rejected due to its size.

BrandonwLii and others added 5 commits October 28, 2026 23:33

github-actions Bot added the size/xl label Oct 30, 2026

Navigation Menu

Repository navigation

M1 jobs and scheduling: steps 2–6 (fake executor, scheduler core, job wrapper, K3s executor, job API) - #131

M1 jobs and scheduling: steps 2–6 (fake executor, scheduler core, job wrapper, K3s executor, job API)#131
BrandonwLii wants to merge 5 commits into
feat/jobs-scheduling-m1from
feat/jobs-fake-executor

BrandonwLii commented Oct 30, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Oct 30, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

BrandonwLii commented Oct 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Steps

Testing

Needs review from

Not yet done (steps 7–9)

Uh oh!

github-actions Bot commented Oct 30, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

BrandonwLii commented Oct 30, 2026 •

edited

Loading