Repository navigation
feat(harness): seal a stopped call with its partial evidence, flag worker channel bypass and record shared records (TH-10189) - #322
Draft
KarthikAvinashFI wants to merge 19 commits into
Conversation
…run their tests in CI
…d gains fault switches
…hared address and leave its checks unmeasured
…rial and what the world made of it
… with its partial evidence
…'s hand-back through a spy
…nd a slow call is never cut without a stop
… as the agent the door also names
…as the user's stop, not infrastructure
…ns for its hand-offs and worker tool calls
…, including a refusal the span saw as output
…eal the partial call, and a call's start event is sent as it starts
…tform's Stop seals the calls in flight instead of exiting
…ritten, not after a thread still blocked on the agent
…source ships as tools-api runs it behind the world's door
… reply it was blocked on
…nd runs API, not a model-written chat route
…mand rewritten to sh -c for the world's port
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
TH-10189 and TH-8888, plan tasks T16 and T17 ("Test a multi-agent application"): when the platform stopped a run mid-call, the hosted call workers never saw the stop, so the call in flight kept nothing; a separate worker that reached the world on the shared address was graded as if it had used its own channel; and nothing recorded when two workers touched the same record. The multi-agent fixture could also not fail on a duplicate refund, because its ledger refused one itself.
This PR seals each in-flight call as
user_canceledwith its partial evidence, flags a worker channel bypass and leaves that worker's checks unmeasured, records shared records, and makes the fixture able to fail. The platform side is future-agifeat/TH-10189-worker-outcomes.What changed
src/fi/alk/harness/hosted_scheduler.py,hosted_entrypoint.pyuser_canceled, pointing at its partial transcript;user_canceledis never retried; a runner that ignores the cancel is cut at the bound; a runner that fails while being stopped, or a call that fails under the stop, is sealed as the stop with what it kept, not as infrastructure. A scenario not yet started when the stop arrives returnsuser_canceledwith no call. A bypass leaves that worker's checks unmeasured and still grades the task.src/fi/alk/harness/isolated_process.py,call_worker.py,chat_worker.pysrc/fi/alk/harness/call_runner.pystop_whenhangs up at the next turn. Shared records: records two or more participants named in a trial and what the world made of them (write from the contract'seffects, unknown is null). Bypass: a separate worker's call on the shared address is flagged; shared tools, shared workers and door rows from before channels are not. A worker's door row and its framework span are one call.scenario_startedis sent as the call starts.src/fi/alk/harness/chat_call_runner.py,src/fi/simulate/agent/wrappers/http.pysrc/fi/alk/harness/observability.pysrc/fi/alk/harness/world/runtime.pyCall.channel: the channel the world saw the call come in on (shared address, a worker's own channel, or none for rows from before channels).src/fi/alk/harness/bundle_author_v2.py,generated_runtime.pytools-apiservice the source ships runs it behind the world's door;langgraph dev/upis served through its threads and runs API (assistant = the graph the written interface names), not a model-written chat route; framework instrumentors land in the agent's uv project when the command is rewritten tosh -cfor the world's port.tests/fixtures/harness_agents/returns_multi_agent/(moved fromexamples/harness/returns_reference/),tests/fixtures/harness_agents/multi_agent_compatibility/(moved),.github/workflows/sdk-smoke.ymlskip_refund_check,payment_times_out,refund_never_returns,bypass_channel), off by default. The compatibility references test runs in CI.tests/harness/*Behavior callouts
user_canceledand its partial transcript, hand-offs, tool calls and worker record, where it produced nothing before. A platform without the companion PR does not read it as cancelled.examples/harness/returns_referenceandexamples/harness/multi_agent_compatibilitymoved undertests/fixtures/; nothing outside tests referenced them.RETURNS_FAULT,RETURNS_HANG_SECONDSandRETURNS_TRACING(fixture only).Tests
test_cancel_mid_call_seals_a_user_canceled_receipt_pointing_at_the_partial_transcript,test_one_stop_seals_every_call_in_flight_each_with_its_own_transcript,test_a_runner_that_ignores_the_cancel_is_cut_at_the_bound,test_a_stopped_worker_exits_without_waiting_on_a_call_stuck_in_a_thread(60 s thread: exits in seconds, waited 60 s on the base),test_a_stopped_call_abandons_the_agent_reply_it_was_waiting_on(under 2 s, 5 s on the base),test_tracing_never_takes_over_the_harness_s_stop_signal,test_a_separate_worker_call_on_the_shared_address_is_a_bypass,test_shared_records_list_two_participants_on_one_order,test_the_ledger_creates_a_second_refund,test_langgraph_dev_is_served_through_its_threads_and_runs_api.d46d591c.test_bundle_author_v2,test_authoring_runtime_validationandtest_runtime_repairgreen (281, 109, 205). On this branch's final tree (without fix(harness): name a served coordinator's own calls by its run, as its jobs' hand-offs do #318),test_bundle_author_v2andtest_framework_tool_spans: 388 passed.Live proof
Local stack with production sandbox specs (4 CPU, 8192 MB, restricted egress) and the companion platform branch, on the returns fixture as a hosted agent.
0cc6e62f,2bde6f55,f66ff308)ended_reasonuser_canceled; transcript, hand-offs, tool calls and the workers record kept38c4153b)payment_times_out)issue_refundreturns "payment service timed out"; the hand-off returns; no refund committed; the full-refund check fails as it should11558728completed; runc192e6edshows the coder hand-off start and end with its result andrun_code_checkunder the coderThe live runs, including the LangGraph ones, used a local build that also had agent-learning-kit #318 applied (a served coordinator's own calls named by its run). #318 is a separate draft PR to
devtouching other files; it is not part of this branch.Playwright e2e
None in this repo. The platform participants flow (stopped call, cleanup, shared records) passed in the platform's
flows/simulate8 of 8 run with this branch mounted.Scope out
Stacked on / merge order
feat/TH-8887-worker-permissions-and-timeouts. Merge feat(harness): worker permissions and deadlines at the world's door (TH-8887, TH-8888) #265 first.feat/TH-10189-worker-outcomesreads theuser_canceledseal, shared records and notice workers.dev).Linear: TH-10189, TH-8888