Skip to content

feat: fence stale dispatch results with generations, and prepare the queue for loop attempts - #423

Merged
saucam merged 3 commits into
mainfrom
feat/loops-dispatch-generations
Oct 25, 2026
Merged

saucam merged 3 commits into
mainfrom
feat/loops-dispatch-generations

Conversation

@saucam

@saucam saucam commented Oct 25, 2026

Copy link
Copy Markdown
Collaborator

What this changes

The dispatch queue becomes safe for long-running, interruptible work, and ready to run loop attempts (docs/loop-engineering-design.md §5–§6). It does this by extending the existing queue rather than adding a second one.

  • Stale results can't land. Every claim and reclaim of a task bumps its generation. A worker's completion only applies if it carries the generation it started under. Before this, a worker replaced after a lease expired could still complete the task it had lost, because completion updated by id alone. That fix applies to all dispatch work, not just loops.
  • Parked tasks (held). An attempt waiting on a person (an approval outside its profile) can be held, so its lease doesn't expire underneath it. A hold lapses after dispatch.holdMaxMs (default 2 hours), and any restart settles it. Only attempts can be held, and only by the daemon that owns them.
  • Attempt tasks. A new attempt kind runs on a loop's behalf. It is never retried by the queue (the loop decides), emits no fleet events (no conductor turns), and doesn't tear down its worker. Dispatcher.cancel stops one in any state.
  • Fleet views unchanged. Attempts are kept off the conductor's task list and the fleet board. Loop runs get their own board rows in a later slice. The web board knows the held status.

No loop uses this yet; the controller lands separately.

Review

Two independent audit rounds.

  • Round 1 fixed: a web build break on the new status, holds that could tie up worker capacity forever, holds allowed on any task kind, unscoped unhold, wrong outcomes on unhold after a restart, a destroy on the continue path, a watcher leak, and attempts appearing in conductor views.
  • Round 2: nothing blocking. Its two small items are fixed: unhold reports an explicit outcome, and holdMaxMs is configurable.

Full suite: 3800 pass / 0 fail. Web typecheck and fleet tests pass.

For agents: implementation
  • store.ts:
    • New dispatch_tasks columns run_id, attempt_id, generation INTEGER NOT NULL DEFAULT 0 and held_until, plus the idx_dispatch_run index.
    • The generation bumps on dispatchClaimNext and dispatchReclaimStale.
    • Complete, fail, mark-running and hold are guarded by generation = ? AND status IN ('claimed','running'); touch takes {id, generation}.
    • Hold, unhold, cancel and settleHeld are scoped by owner_daemon. Prior-boot holds and expired holds settle as reclaimed.
    • The conductor/board queries filter kind != 'attempt'.
  • dispatch.ts:
    • Generation threading, with dispatch.stale_result audit.
    • Optional host hooks: spawnAttemptWorker, continueAttempt, onAttemptSettled, onAttemptWaiting, onAttemptOrphanWorker.
    • hold, unhold (returns UnholdOutcome) and cancel.
    • The late-start fresh flag.
  • config.ts: dispatch.holdMaxMs, also in the settings manifest.
  • Protocol FleetTaskWire: the attempt kind, held status, runId/attemptId.
  • Web: fleet-lanes.ts (held maps to awaiting, with an assertNever), attention.ts, and SessionListPane.
  • Tests: src/tests/dispatch-generations.test.ts, 40+ cases.

🤖 Generated with Claude Code

saucam and others added 3 commits October 25, 2026 09:59
…eld tasks, loop attempt routing

Every claim and reclaim of a dispatch task now bumps a generation, and every
state change a worker result can cause (complete, fail, mark-running, hold,
lease renewal) is guarded on that generation and a live claim. Before this,
dispatchComplete updated by id alone, so a worker replaced after a reclaim
could overwrite its successor's outcome or resurrect a task already terminal.
A fenced result is a no-op, logged and audited as dispatch.stale_result.

Adds a non-terminal `held` status that reclaim never selects, with
hold/unhold on the store and dispatcher, and a `attempt` task kind carrying
run_id/attempt_id for Loops (docs/loop-engineering-design.md §5, §6, §13 B).
Attempts route to optional host hooks (spawnAttemptWorker, continueAttempt,
onAttemptSettled, onAttemptWaiting), fail closed on a host without them,
emit no fleet events, are never torn down by the dispatcher, report an
approval wedge to the loop, and are never re-queued: a reclaim settles as
`reclaimed` and any failure as `failed`.

Additive migration: run_id, attempt_id, generation (default 0) via
#addColumnIfMissing, plus a partial run_id index after the ALTERs.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…on, keep attempts off the fleet board

Round-1 audit of the dispatch-generations slice:

- web: classify the new `held` dispatch status (fleet lanes → awaiting, with
  an exhaustive default; attention queue → approval, deduped against the
  session's own; panel badge), fixing the TS2366 build break.
- held_until column + DispatchConfig.holdMaxMs (default 24h). Reclaim settles
  an expired hold, and a hold placed by a previous boot of this daemon, as
  `reclaimed`; held attempts still count toward maxConcurrentWorkers.
- Dispatcher.cancel(taskId, reason) for attempts in any non-terminal state,
  bumping the generation and reporting onAttemptSettled(task, 'failed').
- dispatchHold is attempt-only; hold/unhold/cancel are scoped by owner_daemon.
- unhold: a hold from a previous boot settles `reclaimed` (never reads the
  worker's idle as success); a gone worker fails at once; a worker still
  waiting on an approval re-fires onAttemptWaiting.
- late-start: only a session created in that call is destroyed; a late
  attempt worker is reported via the new onAttemptOrphanWorker hook.
- #execute's catch unwatches before the stale-generation return.
- attempts are excluded from fleet_tasks, the board snapshot, its deltas and
  its counters.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…ts to 2h

- Dispatcher.unhold returns an UnholdOutcome ("resumed" | "waiting" |
  "finished" | "settled" | "refused") instead of a boolean that was true only
  when the attempt ended up running and watched again.
- dispatch.holdMaxMs added to DispatchSchema, the CodeoidConfig type and the
  settings manifest; the default drops from 24h to 2h, since a held attempt
  occupies a worker slot.
- Dispatcher.hold documents that unhold, cancel or expiry is mandatory: a held
  worker is unwatched.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@saucam
saucam force-pushed the feat/loops-dispatch-generations branch from 254ebdd to b392d41 Compare October 25, 2026 02:05
@saucam
saucam merged commit 18f1999 into main Oct 25, 2026
4 checks passed
@saucam
saucam deleted the feat/loops-dispatch-generations branch October 25, 2026 03:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants