Skip to content

feat(harness): seal a stopped call with its partial evidence, flag worker channel bypass and record shared records (TH-10189) - #322

Draft
KarthikAvinashFI wants to merge 19 commits into
feat/TH-8887-worker-permissions-and-timeoutsfrom
feat/TH-10189-worker-outcomes
Draft

KarthikAvinashFI wants to merge 19 commits into
feat/TH-8887-worker-permissions-and-timeoutsfrom
feat/TH-10189-worker-outcomes

Conversation

@KarthikAvinashFI

Copy link
Copy Markdown
Member

Why

TH-10189 and TH-8888, plan tasks T16 and T17 ("Test a multi-agent application"): when the platform stopped a run mid-call, the hosted call workers never saw the stop, so the call in flight kept nothing; a separate worker that reached the world on the shared address was graded as if it had used its own channel; and nothing recorded when two workers touched the same record. The multi-agent fixture could also not fail on a duplicate refund, because its ledger refused one itself.

This PR seals each in-flight call as user_canceled with its partial evidence, flags a worker channel bypass and leaves that worker's checks unmeasured, records shared records, and makes the fixture able to fail. The platform side is future-agi feat/TH-10189-worker-outcomes.

What changed

File Change
src/fi/alk/harness/hosted_scheduler.py, hosted_entrypoint.py A user stop seals each in-flight call as user_canceled, pointing at its partial transcript; user_canceled is never retried; a runner that ignores the cancel is cut at the bound; a runner that fails while being stopped, or a call that fails under the stop, is sealed as the stop with what it kept, not as infrastructure. A scenario not yet started when the stop arrives returns user_canceled with no call. A bypass leaves that worker's checks unmeasured and still grades the task.
src/fi/alk/harness/isolated_process.py, call_worker.py, chat_worker.py The stop reaches the chat and voice call worker subprocesses (SIGUSR1); the parent raises the stop with the worker's sealed call; a stopped worker exits once its result is written, not after a thread still blocked on the agent.
src/fi/alk/harness/call_runner.py Voice: stop_when hangs up at the next turn. Shared records: records two or more participants named in a trial and what the world made of them (write from the contract's effects, unknown is null). Bypass: a separate worker's call on the shared address is flagged; shared tools, shared workers and door rows from before channels are not. A worker's door row and its framework span are one call. scenario_started is sent as the call starts.
src/fi/alk/harness/chat_call_runner.py, src/fi/simulate/agent/wrappers/http.py A stopped chat call seals its partial transcript, and abandons the agent reply it was waiting on (a stopped run's world is never reused). A cancel that is not a stop still propagates.
src/fi/alk/harness/observability.py Tracing keeps the harness's SIGTERM handler, so the platform's Stop seals the calls in flight instead of exiting the guest.
src/fi/alk/harness/world/runtime.py Call.channel: the channel the world saw the call come in on (shared address, a worker's own channel, or none for rows from before channels).
src/fi/alk/harness/bundle_author_v2.py, generated_runtime.py A Compose-selected Python agent is observed for its hand-offs and worker tool calls; a native contract whose tools call the tools-api service the source ships runs it behind the world's door; langgraph dev/up is served through its threads and runs API (assistant = the graph the written interface names), not a model-written chat route; framework instrumentors land in the agent's uv project when the command is rewritten to sh -c for the world's port.
tests/fixtures/harness_agents/returns_multi_agent/ (moved from examples/harness/returns_reference/), tests/fixtures/harness_agents/multi_agent_compatibility/ (moved), .github/workflows/sdk-smoke.yml The multi-agent fixtures are internal CI fixtures, not examples. The returns ledger allows a duplicate refund, and gains fault switches (skip_refund_check, payment_times_out, refund_never_returns, bypass_channel), off by default. The compatibility references test runs in CI.
tests/harness/* Tests below.

Behavior callouts

  • A stopped call now produces a receipt with failure code user_canceled and its partial transcript, hand-offs, tool calls and worker record, where it produced nothing before. A platform without the companion PR does not read it as cancelled.
  • A worker flagged for a channel bypass has its own checks reported unmeasured ("reached the world without their own channel ... their own checks were not graded"); the task is still graded.
  • The returns fixture no longer refuses a second refund: a duplicate-refund scenario can now fail.
  • examples/harness/returns_reference and examples/harness/multi_agent_compatibility moved under tests/fixtures/; nothing outside tests referenced them.
  • No new env var in the harness. The fixture reads RETURNS_FAULT, RETURNS_HANG_SECONDS and RETURNS_TRACING (fixture only).

Tests

  • 40 new tests, including test_cancel_mid_call_seals_a_user_canceled_receipt_pointing_at_the_partial_transcript, test_one_stop_seals_every_call_in_flight_each_with_its_own_transcript, test_a_runner_that_ignores_the_cancel_is_cut_at_the_bound, test_a_stopped_worker_exits_without_waiting_on_a_call_stuck_in_a_thread (60 s thread: exits in seconds, waited 60 s on the base), test_a_stopped_call_abandons_the_agent_reply_it_was_waiting_on (under 2 s, 5 s on the base), test_tracing_never_takes_over_the_harness_s_stop_signal, test_a_separate_worker_call_on_the_shared_address_is_a_bypass, test_shared_records_list_two_participants_on_one_order, test_the_ledger_creates_a_second_refund, test_langgraph_dev_is_served_through_its_threads_and_runs_api.
  • Run module by module: 55 modules, one at a time. All new tests pass; the 32 failures that remain are identical on the base d46d591c. test_bundle_author_v2, test_authoring_runtime_validation and test_runtime_repair green (281, 109, 205). On this branch's final tree (without fix(harness): name a served coordinator's own calls by its run, as its jobs' hand-offs do #318), test_bundle_author_v2 and test_framework_tool_spans: 388 passed.
  • Mutation checks: 29 killed (plan-named tests plus each fix test), plus 3 for the LangGraph serving.
  • Break-it pass: one stop with two calls in flight, a stop landing as the call finishes (normal receipt), a stop before any call starts (every scenario skipped), a worker lost on the way out of a stop, a library replacing the guest's signal handlers.

Live proof

Local stack with production sandbox specs (4 CPU, 8192 MB, restricted egress) and the companion platform branch, on the returns fixture as a hosted agent.

Case Result Proof
Stop mid-call while the refund worker waits on payment (jobs 0cc6e62f, 2bde6f55, f66ff308) Call cancelled, ended_reason user_canceled; transcript, hand-offs, tool calls and the workers record kept Image 1
Stop before the call started (job 38c4153b) Keeps nothing, no crash Not shown
A build whose refund worker uses the shared address "Refund worker reached the world without their own channel ... their own checks were not graded" Image 2
Two workers touch order 1043 Listed as a shared record with both workers Image 3
Payment call times out (payment_times_out) issue_refund returns "payment service timed out"; the hand-off returns; no refund committed; the full-refund check fails as it should Image 4
LangGraph async deep agents, hosted build Before: the builder wired the graph server as an OpenAI chat route (404). After: build 11558728 completed; run c192e6ed shows the coder hand-off start and end with its result and run_code_check under the coder Images 5, 6
Worker hangs past the chat timeout Not reachable live: a new agent version is certified first, and a hang never passes certification. Unit tests only Not shown

Stopped call keeps its evidence

Worker channel bypass in a run

Shared records

Payment timeout

LangGraph hosted: the coder job's hand-off

LangGraph hosted: transcript through threads and runs

The live runs, including the LangGraph ones, used a local build that also had agent-learning-kit #318 applied (a served coordinator's own calls named by its run). #318 is a separate draft PR to dev touching other files; it is not part of this branch.

Playwright e2e

None in this repo. The platform participants flow (stopped call, cleanup, shared records) passed in the platform's flows/simulate 8 of 8 run with this branch mounted.

Scope out

  • A refund hand-off still out at the Stop has no Participants row: its framework span is exported only when it ends.
  • A per-worker stop: workers share the app's one process; only the whole call stops.
  • LangGraph: launches read "parent unknown", and a researcher job that outlives its 30 s call is beyond the call's boundary (TH-9023).
  • MCP, CLI and SDK; T44 evidence access.

Stacked on / merge order

Linear: TH-10189, TH-8888

…hared address and leave its checks unmeasured
…, including a refusal the span saw as output
…eal the partial call, and a call's start event is sent as it starts
…tform's Stop seals the calls in flight instead of exiting
…ritten, not after a thread still blocked on the agent
…source ships as tools-api runs it behind the world's door
…mand rewritten to sh -c for the world's port
@KarthikAvinashFI KarthikAvinashFI self-assigned this Nov 14, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant