Repository navigation
feat(harness): declared workers in the contract and a returns reference app - #226
Conversation
…ntime says made each call
…t/TH-8886-declared-workers # Conflicts: # src/fi/alk/harness/call_runner.py # src/fi/alk/harness/run/simulation.py # src/fi/alk/harness/world/runtime.py
…all and the worker that made it
…o each keeps its worker
…vation and keep voice on its LiveKit trace
…-off with its return
…uv run project agent runs in
…ll that launched it
…ne still running when the call ends is kept
…and join a job that starts before its launch returns
…de it, as its own span is named only when it ends
…, apart from the call that launched it
…it ran in, its kind and what bounding left out
…s, and the app's top agent hands off at the top level
…built-in transfer from any instrumentation counts once its agent runs
…hapes, including a transfer that outlives its agent
… Agents shape and give an ADK transfer with no target only to a later, different agent
|
Thanks for the review. Each point, with the commit and the test that locks it:
|
hadarishav
left a comment
There was a problem hiding this comment.
I rechecked the current head. The four earlier asks are fixed. Worker IDs now reject blank and duplicate values, delegation relationships reach sealed events, the observer logic moved out of the bootstrap, and the support claim now names the observed framework shapes.
One evidence-correlation bug still blocks approval. Two overlapping successful calls to the same tool can receive each other's runtime event IDs and worker relationship when the span records arrive in a different order from the native records. I reproduced this on this head with search(q=a) at 1, search(q=b) at 2, then span B followed by span A. The merged output assigned event A to q=b and event B to q=a. The pairing needs a stable identity when one exists. A fallback must use compatible payload and timing, and it must leave attribution unknown when the records are ambiguous rather than swap them. Please add the reversed-order, same-tool, same-outcome overlap as a regression test.
The focused ALK suites pass on this head with 1,182 tests. That confirms the existing coverage but does not catch this ordering case.
| # A job's hand-off sits under the call that launched it and is not that call again. | ||
| merged.append(span) | ||
| continue | ||
| same_tool = [c for c in unpaired if _tool_key(c.name) == _tool_key(span.name)] |
There was a problem hiding this comment.
This pairs only by normalized tool name and outcome, then takes the first remaining call. For two concurrent successful calls to the same tool, reversed span completion order swaps their runtime IDs and worker relationships. I reproduced search(q=a) and search(q=b) becoming q=b -> event A and q=a -> event B. Correlate by a stable event ID when available, then compatible payload and time. If the remaining records are ambiguous, keep the attribution unknown instead of attaching the wrong worker evidence.
There was a problem hiding this comment.
Fixed in 66411dd: a span pairs by its event id, else with the one call its arguments, outcome or order among non-overlapping calls single out, and the two tracers' clocks are never compared; overlapping calls nothing tells apart, and two spans with equal claims on one call, stay unknown. Your case is test_two_overlapping_calls_of_one_tool_keep_their_own_spans_when_the_spans_end_in_the_other_order, with the clock-offset, contention and unknown cases beside it.
…d time, else leave it unknown
…d not overlap, and leave contested calls unknown
…ated lines as they were
|
Fixed the overlapping same-tool pairing in 66411dd, with your reversed-order case and the clock-offset, contention and ambiguous cases as regression tests; each rule's test fails when that rule is removed. Could you take another look? |
hadarishav
left a comment
There was a problem hiding this comment.
Re-reviewed the current head 66411dd2. The earlier points are fixed, but one correlation bug still blocks approval.
When one native call and one span remain, _pair_tool merges them without checking arguments or outcome. I reproduced a successful native search(q=a) and a failed span search(q=b). The merged call kept q=a but received worker B's event and parent IDs, while the failed q=b disappeared.
Please apply the compatibility check before the one-to-one shortcut. If the records conflict, keep both or leave attribution unknown. Add this tracer-loss case as a regression test.
The focused suites passed with 197 tests.
| free = [ | ||
| call for call in natives if call.runtime_ids.get("event_id") not in span_ids | ||
| ] | ||
| if len(rest) == 1 and len(free) == 1: |
There was a problem hiding this comment.
This one-span, one-call shortcut bypasses fit(). One tracer can miss call A while the other misses call B, so count alone does not prove identity. With native search(q=a) succeeding and span search(q=b) failing, this merges them, assigns B's worker and relationship to A, and drops B. Require compatibility before merging. Keep both records when arguments or outcome conflict.
There was a problem hiding this comment.
Fixed in 8f020df3, 9a96d57c and e1ed2644. The one-to-one shortcut now goes through the same check as every other pairing path: a span and a call merge only when no recorded argument, outcome, result, worker id, hand-off target or kind differs. Otherwise both are kept with nothing lent across. Your search(q=a) / failed search(q=b) case is a regression.
…every pairing path
…orker, target or kind differs, and keep surplus spans
|
Fixed, with the same check applied on every pairing path: the claim step, pairing by order and interchangeable calls, not only the one-to-one shortcut. Two related cases are fixed too: spans that outnumber the remaining calls are kept rather than dropped, with no worker or parent guessed ( |
|
One more small fix found while live-testing the stacked worker PR: the returns reference app answered every turn on a new event loop, so with an OpenAI model its second turn failed with "Event loop is closed". It now uses one loop for the process ( |
…nd match a worker by any name either tracer recorded
Summary
Records which worker did what in a multi-agent app's calls, for the runtime and framework shapes listed below. Each hand-off from the coordinator to a worker is recorded with its return, and every tool call with the worker that made it. Each sealed worker event names its span, the hand-off it ran in, its kind (
delegated,transferredortool_call), whether that place was observed or is unknown, and whether its input or output was cut. It also declares workers in the contract (a blank or repeated id is an authoring problem) and adds a returns reference app for tests. Platform side: future-agi #3699.Observed for Python agents whose framework spans this sandbox reads: OpenAI Agents SDK, Google ADK (1 and 2), CrewAI, AutoGen (AgentChat and Core), LangGraph, Claude Agent SDK (span attributes). Unobserved is a reported state, never a guess: non-Python agents, apps run through Compose, apps that turn their framework's tracing off, and frameworks whose instrumentor is not installed are written as unobserved. A place the spans do not give reads
unknown.Goal moved: environment quality. A multi-agent app's calls are graded on the tool calls it really made, and the platform can show which worker made each one.
Review points and where each is fixed
Every point from the review, the commit that fixes it, and the test that fails without it (each test was run against the reverted fix and seen failing).
83430e8b,69d85d2a,24dfc23f,a4bea940,0b2f97b0test_two_open_hand_offs_to_one_worker_each_hold_their_own_calls,test_a_framework_s_own_transfer_is_a_control_pass_and_holds_the_leaf_worker_s_calls. Alsotest_a_call_with_no_agent_above_it_has_an_unknown_place,test_a_parent_no_recorded_hand_off_carries_is_unknown_not_kept,test_each_payload_says_whether_part_of_it_was_left_outa097a2a6test_a_worker_without_an_id_of_its_own_is_returned_to_authoring_and_keeps_only_contract_tools(worker[i]:no-id,duplicate-worker-ids)d5601050simulation_worker_events.pyholds all span-to-event reading;test_worker_event_module_imports_only_the_standard_library_and_uses_python_39_syntaxfe44b142,6aa538b0,66411dd2test_two_overlapping_calls_of_one_tool_keep_their_own_spans_when_the_spans_end_in_the_other_order(both span orders). A span pairs by its event id, else with the one call its arguments, outcome or order among non-overlapping calls single out, and the two tracers' clocks are never compared (test_calls_that_did_not_overlap_pair_in_order_whatever_the_offset_between_the_two_clocks,test_calls_the_other_tracer_never_stamped_are_not_paired_by_order). Overlapping calls nothing tells apart, and two spans with equal claims on one call, stay unknown; the stronger of two unequal claims takes the call:test_overlapping_calls_of_one_tool_nothing_tells_apart_keep_their_agent_and_place_unknown,test_two_spans_claiming_one_call_go_to_the_stronger_claim_and_a_tie_leaves_it_unknown. One span and one call of a tool pair only when nothing both recorded differs (arguments, outcome, result, worker, hand-off target, kind):test_one_span_and_one_call_of_a_tool_pair_however_each_tracer_spelled_the_arguments,test_one_span_and_one_call_that_disagree_are_two_calls_each_tracer_missed_one. Consequence: overlapping calls of one tool from an instrumentor that records no arguments read unknown, and a span with no recorded end (a transfer, a job still running) is never paired by order either8f020df3,9a96d57c,e1ed2644,9dac44d4search(q=a)/ failedsearch(q=b)case and its class:test_one_span_and_one_call_that_disagree_are_two_calls_each_tracer_missed_one,test_a_span_and_a_call_that_disagree_on_any_recorded_field_are_two_calls,test_order_pairs_only_calls_that_ended_the_same_way,test_spans_that_outnumber_the_calls_left_are_each_kept_without_a_guessed_owner,test_a_control_pass_made_through_a_tool_is_written_once_as_the_transfer8d3851ectest_spans_count_only_against_the_calls_they_could_be_and_an_unrelated_call_keeps_no_owner,test_one_worker_named_differently_by_two_tracers_is_one_call(a framework's hashed key vs its role name; spelling),test_a_call_named_by_its_worker_label_never_takes_another_worker_s_placeThe later commits read the span shapes found on real calls: OpenAI Agents SDK 0.24 through traceAI, where each run is a name-less agent span with the agents inside it, and Google ADK 2.13, where a transfer may carry no arguments and outlive its agent. They also fix the 6 Nov review findings (
0b2f97b0, below).Two framework-shape rules, stated plainly
graph.node.parent_id), and staysunknownotherwise.TransferToAgentToolcall with no recorded target goes to the next agent that runs in the same workflow, but only to one that started after the transfer ended and is not the agent that made it. Anything else leaves it a transfer with no target. A later agent under the workflow with no transfer into it isunknown.Test matrix
Live rows on
0b2f97b0(an earlier head) with future-agi #3699 at2c9a663f1(before its move todev); the pairing rules in rows 6 and 7 came after it and are covered by the unit tests named there, not re-run live, on the local stack with production's sandbox specs. "Live" means a fresh call on 6 Nov on an existing build; the sandbox read its worker events with that head's code.test_framework_tool_spans.py(73 on this head)defb17c4)21eb5743)Live calls (one each, scenarios generated by the builder, not edited):
391f86e701a7a738807df1e521eb5743defb17c4The local sandbox ran
0b2f97b0's worker-event code. That code was taken from this branch merged into the local release candidate, which adds only unrelated chat fixes. LangGraph was not run live in that round: its local builds fail before the call (oneseed_failedon a fingerprint mismatch, oneresponse_path segment not found: messages), outside this PR.Verified against code on 335b4b6, 10 Nov.
Worker permissions and timeouts (TH-8887, TH-8888): in agent-learning-kit #265
Per-worker tool permissions (each worker held to its own tools, an out-of-permission write refused and shown as an attempted violation, a per-worker verdict) and per-worker deadlines (a worker past its time limit ends as timed out) are written and tested, but are not in this PR. They are built on the world's action log and permission check that arrive with the actors work (agent-learning-kit #257, future-agi #3764), so they are in agent-learning-kit #265 and future-agi #3791, stacked on actors with this PR merged in.
Limitations
unknown); a transfer's target when the framework records none and no later, different agent starts (a transfer with no target).json.dumpsand appends one line. Every table is bounded (open spans 4,096, closed 4,096, launches, waiting jobs and held transfers 256), arguments and outputs are cut at 4,096 characters, and a line at 64 KB. Not bounded, and not changed here: the number of records per call (one per tool span), and the runner reading the whole trace file before merging, which pairs spans to calls in O(spans x calls).Release
The sandbox reads worker events from the code baked into the E2B template. Shipping this needs a template bump pinned in the deployment repo. Local tests ran on production's sandbox specs (4 CPU, 8,192 MB, 23 GB, restricted egress). No migration.
1. Why
To show which worker of a multi-agent app did what, the platform needs two things: which workers the app declares, and, for every tool call, which worker made it and in which hand-off. This PR records declared workers in the agent contract, and makes the sandbox observe each tool call, its worker and its place for the shapes above, without changes to the app. It also adds a small reference app (a returns desk with a coordinator and two workers) to test against.
2. What changed
harness/contract.pyDeclaredWorker(id,name,tools) andAgentContract.workers; a worker keeps only tools the contract has. A blank id (worker[i]:no-id) or a repeated one (duplicate-worker-ids) is returned byvalidate_contractas a problem to fixharness/tools.py,skills/understand-agent/SKILL.mdworkersin the contract submission schema, "an id no other worker has"; how to record workers from the agent's own codeharness/simulation_worker_events.py(new)harness/simulation_mark_bootstrap.pyharness/world/runtime.py,harness/run/simulation.pytrace_rowwrites, per call a runtime attributed,span_id,parent_span_id,kind(delegated,transferred,tool_call),agent,relationship(observed,unknown),input_omitted,output_omitted, plusdelegated_to(a control pass iskind: transferred).graded_callsleaves hand-off records out of checksharness/call_runner.py_merge_span_callsjoins span lines to the calls other sources observed._keep_recorded_parentskeeps a parent only when a recorded hand-off carries it, and places a handed-control agent's calls in that control pass on the record aloneharness/crewai_runtime_observer_bootstrap.pyharness/process_runtime.py,harness/generated_runtime.py,harness/bundle_author_v2.pysource_tool_trace_versionstays 1simulate/agent/wrappers/http.pyexamples/harness/returns_reference/reference.jsondeclares its topologyCHANGELOG.md## Unreleased3. Behavior callouts
name, arguments, result, ok, error, refused, at. A row written now adds, only when set,worker_id,worker_label,task_id,task_label,delegated_to,ended_at,runtime_ids,input_omittedandoutput_omitted, and, for a call a runtime attributed,span_id,parent_span_id,kind,agentandrelationship.autogen.agent.name,claude_agent.subagent_type,graph.node.id,graph.node.parent_id,gen_ai.tool.type,lc_agent_name,langgraph_checkpoint_ns). Two patches are per framework because no public API gives the same result: AutoGen'sBaseTool.run_jsonand the OpenAI Agents SDK'sset_processors.ANTHROPIC_API_KEYorOPENAI_API_KEY); the platform does not supply one.4. Tests
test_framework_tool_spans.py(73 tests, 107 cases)test_contract.pytest_simulation_mark_bootstrap.pytest_call_runner.py,test_chat_readiness.py,test_crewai_runtime_observer.py,test_process_runtime.pyEach test added for the review and the 9 to 10 Nov fixes was run against its reverted fix and seen failing (15 mutations in the review, five more for the 9 to 10 Nov fixes). For the overlapping-call pairing, each of its 15 rules was removed in turn and that removal failed a test: 14 a test added for it, the outcome rule an earlier one. Some of those tests also pass on the code before the pairing fix, because they guard behaviour that already held there.
5. Commands to run the tests
On
fe44b142: 641 passed. On8d3851ecthe touched modules (test_framework_tool_spans.py,test_call_runner.py,test_chat_call_runner.py) give 263 passed. The whole suite was not rerun on this head. On the previous head it gave 7 failures: 5 that fail on the base too (test_dispatch_ack, one grading cap) and 2 intest_isolated_processthat pass when run alone.Latest
devmerged on 10 Nov (335b4b6b, no conflicts):test_framework_tool_spans.py,test_chat_worker.py,test_hosted_conversation.py,test_chat_call_runner.py,test_call_runner.pyandtest_builder_chat.pygive 350 passed.6. Steps to test on local
examples/harness/returns_reference/) with anOPENAI_API_KEYorANTHROPIC_API_KEYsecret.get_orderandlist_refundsand returns.On the platform (recorded on an earlier head, before the 9 to 10 Nov fixes)
Walkthrough (2x): a two-turn returns call, a CrewAI hand-off with no recorded parent, and Open in Traces.
Checked live on the 10 Nov heads (with future-agi #3699)
The local stack ran this branch at
8d3851ecwith future-agi #3699 at645f62abe, production's sandbox specs (4 CPU / 8192 MB, restricted egress,svc-control), and the browser on #3699's frontend.tree_complete: false; the second read merged totree_complete: true. One hand-off per turn, each worker call (get_order,list_refunds) under its hand-off, coordinator "Returns coordinator", nothing doubled. Participants: "2 hand-offs to 1 worker. 2 finished."update_order_statusshows as error under the order-lookup worker, with the world's reason as its outputshakespearean_bard, instance "Shakespearean Bard"); "2 hand-offs to 1 worker", none counted twice. The app's own LLM call failed in the sandbox, so the hand-offs returned nothingplace_agentsits under the control pass toinspiration_agent; "2 hand-offs to 2 workers. 1 finished, 1 passed control"; undeclaredplace_agentlisted as not declaredNot tested live
Merge note
When #257 (actors) and this PR both reach
dev, the second one meets a small conflict incall_runner.py, where the sealed trace is written. Keep both: each row istrace_row(call)plus #257's action marks, computed overgraded_calls(calls), followed by #221's actor rows.Upgrades
ALK_OBSERVE_FRAMEWORKSand the instrumentor step are written by the build itself.7. Scope in / scope out
In: declared workers in the contract; each tool call's worker and place for the observed shapes; the reference app.
Out: the worker timeline and UI (future-agi #3699); per-worker permissions and channels (TH-8887); worker timeouts and cancel states (TH-8888); worker-attributed world state (TH-8921); worker results in the CLI (TH-8847).
8. Design choices
Tickets and plan
delegated,transferred,tool_call) and whether that place was observed or is unknown; hand-off records are never graded. Left: the same record in CLI results (TH-8847); per-worker permissions, verifier and timeout of TH-8884 are in agent-learning-kit feat(harness): worker permissions and deadlines at the world's door (TH-8887, TH-8888) #265 and future-agi #3791.examples/), and "Prove supported transport/runtime and world bindings" in part: worker events observed live on OpenAI Agents SDK, Google ADK, CrewAI and AutoGen apps; LangGraph and Claude Agent SDK by span shape in unit tests only; per-worker world bindings come with feat(harness): worker permissions and deadlines at the world's door (TH-8887, TH-8888) #265.Part of TH-8886