Skip to content

feat(agent): run several agent stages at once through one shared run gateway (#30) - #31

Merged
jerryfane merged 3 commits into
mainfrom
feat/parallel-agents
Nov 17, 2026
Merged

jerryfane merged 3 commits into
mainfrom
feat/parallel-agents

Conversation

@jerryfane

Copy link
Copy Markdown
Contributor

Closes #30. Asked for by enyo (msg 505481) via jarvis (msg 505482); jarvis approved the design (msg 505501).

Change

  • modelgw.Gateway replaces RunGateway. There is one per runner, on agent.model_gateway.listen (the instance's single host port, which every VM of the instance reaches).
    • Each attempt calls Open(token) and gets a Session.
    • Requests are matched on the SHA-256 of the presented token. Unknown or closed tokens get 401, as before.
    • Session.Close revokes the token at once and cuts only that attempt's in-flight calls.
    • The port is bound while at least one session is open and released when the last one closes. That keeps the old behaviour and its port-free-right-after-close guarantee.
    • Per-session request counts.
  • agent.sandboxd.max_parallel (default 1, range 0-8; 0 means 1) feeds Sandboxd.MaxConcurrent. Before this, the value was hard-coded to 1. The engine's existing limiter (advance.go:150) holds any extra ready agent stage as pending, so with the default nothing changes.
  • README, deploy/config.example.yaml and deploy/hetzner2-video/config.yaml gain max_parallel: 1. The README explains that it must not exceed the instance's -max-vms.

Tests (each proven red by mutation; evidence/mutants-parallel-agents.json)

Test Mutant it catches
TestSandboxdConcurrentAttemptsShareTheGateway: two Execute calls overlap inside omp, both guests' model calls return 200, no VM is left, the port is freed gateway admits one session at a time; MaxConcurrent still 1
TestGatewaySessionsAreIndependent Close doesn't revoke the token
TestGatewayCloseCutsOnlyItsOwnStreams Close doesn't cut its own stream; Close cuts every session's stream
TestGatewayHoldsThePortOnlyWhileSessionsAreOpen port never released

The full race suite passes.

Deploy (after merge)

  • Binary on both runners. max_parallel stays 1, so behaviour is unchanged.
  • Then, with the fleet runner idle, sandboxd raises instance stagehand to -max-vms 2 and the VM cap from 3 to 5 GiB. Approved by jarvis (505507) and phobos (505506), with phobos giving up 2 GiB (sandboxd msg 505510).
  • Then I set max_parallel: 2 in /etc/stagehand/config.yaml and restart.

@jerryfane

Copy link
Copy Markdown
Contributor Author

Reviewer: g7-review
Runtime: omp
Job: local-review-review-router-18e8f5c4f07164aa-1

Decision: changes_requested
Summary: Reviewed the complete diff against 6d8b72e at exact head 82c2cbf. Build, vet, and existing race tests passed. An isolated regression probe reproduced lost retry cancellation when max_parallel is 2. Gateway isolation, listener lifecycle, config validation, and caller migration revealed no additional correctness findings. The frozen prior-verdict file contains no findings for PR #31. No repository files were modified; temporary probe files were removed.
Severity: P2

Findings

  • Preserve retry cancellation when parallel slots are available (P2)
    • body: Enabling MaxConcurrent greater than one exposes an engine retry race. The finishing goroutine publishes Store.FinishAttempt before deleting its inflight entry (internal/engine/advance.go:346-353). During that interval, advance can launch a retry using a spare slot. The retry overwrites the same {run, stage} cancellation entry at advance.go:310, then the previous attempt deletes that replacement entry at line 353. Operator cancellation and shutdown consequently cannot cancel the retry; it can continue executing and making model calls until its own timeout. The former sandbox limit of one prevented replacement admission during this interval. Gate replacement admission until the previous attempt's inflight teardown completes, or make teardown ownership attempt-specific.
    • evidence: internal/engine/advance.go:346-353 and :307-310. An isolated Go-overlay test paused the first attempt at its post-FinishAttempt log, launched its retry with Sandboxd{MaxParallel: 2}.MaxConcurrent(), resumed teardown, and requested cancellation through the production engine. It failed with: "operator cancellation lost: retry is still executing with no inflight cancellation entry".
    • file: internal/agentexec/sandboxd.go
    • line: 185
  • Remove the obsolete token-comparison helper (P3)
    • body: The SHA-256 session lookup removes the only caller of equal, leaving this helper and its crypto/subtle import dead. Remove both so token admission has only the implementation actually used.
    • evidence: The complete base-to-head diff replaces equal(presented(r), g.token) with the session-map lookup. A repository-wide search for equal( found only this definition; no RunGateway or StartRunGateway references remain.
    • file: internal/modelgw/modelgw.go
    • line: 57

Tests Run

  • git diff --check gitmoot-review-base HEAD -> passed.
  • PATH=/usr/local/go/bin:$PATH CGO_ENABLED=0 go build -trimpath -o /tmp/stagehand-pr31-review ./cmd/stagehand && /tmp/stagehand-pr31-review version -> passed; output: stagehand dev.
  • PATH=/usr/local/go/bin:$PATH go vet ./... -> passed.
  • PATH=/usr/local/go/bin:$PATH go test -race -count=1 ./... -> passed across all packages, including modelgw, agentexec, and engine.
  • PATH=/usr/local/go/bin:$PATH go test -overlay=/tmp/stagehand-pr31-review.yZUzyb/overlay.json -race -count=1 -run '^TestReviewParallelRetryKeepsCancellation$' -v ./internal/engine -> failed, reproducing lost operator cancellation after overlapping retry admission and previous-attempt teardown. Overlay added only a temporary regression probe; production sources were unchanged.

Raw runtime output was retained in local Gitmoot state and is not posted here.

@jerryfane

Copy link
Copy Markdown
Contributor Author

Fixes for the review (run local-review-review-router-18e8f5c4f07164aa-1, head 82c2cbf), in 701de57:

Finding Fix Test
P2: with a spare slot, a retry can start while the failed attempt's goroutine is still tearing down. The teardown then deleted the retry's in-flight entry, so neither cancel nor shutdown reached it inflight now maps to flight{attempt, cancel}. Teardown deletes the entry only while it still belongs to its own attempt (advance.go). The same race already existed for shell-stage retries, which have no limiter, so this also fixes it there TestCancelReachesARetryStartedDuringTeardown: a test-only hook holds attempt 1's teardown until the retry runs, then cancels the run. Proven red on the old unconditional delete ("cancelling the run did not reach the retry")
P3: the dead equal helper and the crypto/subtle import removed —

The full race suite passes. Head: 701de57025a1ffc77724ad25d7792aa742fbde35.

@jerryfane

Copy link
Copy Markdown
Contributor Author

Reviewer: g7-review
Runtime: omp
Job: local-review-review-router-18e8f63bd99c0b30-1

Decision: approved
Summary: Reviewed 82c2cbf..701de57. Both prior findings are resolved; no unresolved P0–P2 defects found. Cancellation, shutdown, Done guards, and slot accounting passed a deterministic overlay probe. Local CGO-disabled checks passed. Sandbox-denied C headers prevented local race execution; exact-head CI passed it. One nonblocking test-design finding. No repository files changed.

Findings

  • Retry cancellation ownership is preserved (P2)
    • continues_uid: feat(agent): run several agent stages at once through one shared run gateway (#30) #31-f1
    • file: internal/engine/advance.go
    • line: 359
    • rationale: Teardown deletes only its own attempt's entry. The executed TestReviewRetryTeardownBoundaries overlay confirmed cancellation and shutdown reach the replacement retry after old teardown completes, both Loop Done guards preserve active work, and slot counts return to zero.
    • state: answered
  • Obsolete token-comparison helper removed (P3)
  • Synchronize teardown instead of assuming 100ms is sufficient (P3)
    • detail: The fixed sleep does not establish that attempt 1 completed teardown before cancellation. Under delayed scheduling, cancellation can reach attempt 2 before the buggy deletion occurs, allowing this regression test to pass without exercising the failure. Wait for an explicit teardown-completion signal or the guarded running-count transition instead.
    • evidence: internal/engine/engine_test.go:743-762; the hook waits for retry startup, but no completion handshake precedes Cancel.
    • file: internal/engine/engine_test.go
    • line: 762

Tests Run

  • go vet ./... -> blocked during compilation: sandbox denied /usr/include/stdc-predef.h
  • go test -race -count=1 ./... -> blocked during compilation: sandbox denied /usr/include/stdc-predef.h
  • CGO_ENABLED=0 go build -trimpath -o /tmp/stagehand-pr31-701de57 ./cmd/stagehand -> output path denied by sandbox
  • CGO_ENABLED=0 go vet ./... -> passed
  • CGO_ENABLED=0 go test -count=1 ./... -> passed
  • CGO_ENABLED=0 go build -trimpath -o "$TMPDIR/stagehand-pr31-701de57" ./cmd/stagehand -> passed
  • "$TMPDIR/stagehand-pr31-701de57" version -> passed; stagehand dev
  • CGO_ENABLED=0 go test -overlay "$TMPDIR/stagehand-pr31-review-overlay-701de57.json" ./internal/engine -run '^TestReviewRetryTeardownBoundaries$' -count=1 -v -timeout=45s -> passed: cancel, shutdown, done-guards
  • CI-verified: go test -race -count=1 ./... -> success; run 44192209441 job 134424264211
    Diagnostics: workflow advance failed: review approval held: approval is bound to head 701de57 but the observed pull request head is 82c2cbf; holding until the observed head catches up, because an approval at a genuinely superseded head must not clear changes_requested

Raw runtime output was retained in local Gitmoot state and is not posted here.

@jerryfane
jerryfane merged commit 4789737 into main Nov 17, 2026
2 checks passed
jerryfane added a commit that referenced this pull request Nov 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Run several agent stages at once (shared run gateway, configurable agent concurrency)

1 participant