Skip to content

feat(harness): declared workers in the contract and a returns reference app - #226

Merged
KarthikAvinashFI merged 75 commits into
devfrom
feat/TH-8886-declared-workers
Nov 12, 2026
Merged

KarthikAvinashFI merged 75 commits into
devfrom
feat/TH-8886-declared-workers

Conversation

@KarthikAvinashFI

@KarthikAvinashFI KarthikAvinashFI commented Nov 2, 2026 •

Copy link
Copy Markdown
Member

Summary

Records which worker did what in a multi-agent app's calls, for the runtime and framework shapes listed below. Each hand-off from the coordinator to a worker is recorded with its return, and every tool call with the worker that made it. Each sealed worker event names its span, the hand-off it ran in, its kind (delegated, transferred or tool_call), whether that place was observed or is unknown, and whether its input or output was cut. It also declares workers in the contract (a blank or repeated id is an authoring problem) and adds a returns reference app for tests. Platform side: future-agi #3699.

Observed for Python agents whose framework spans this sandbox reads: OpenAI Agents SDK, Google ADK (1 and 2), CrewAI, AutoGen (AgentChat and Core), LangGraph, Claude Agent SDK (span attributes). Unobserved is a reported state, never a guess: non-Python agents, apps run through Compose, apps that turn their framework's tracing off, and frameworks whose instrumentor is not installed are written as unobserved. A place the spans do not give reads unknown.

Goal moved: environment quality. A multi-agent app's calls are graded on the tool calls it really made, and the platform can show which worker made each one.

Review points and where each is fixed

Every point from the review, the commit that fixes it, and the test that fails without it (each test was run against the reverted fix and seen failing).

Review point Commit Test that locks it
1. An explicit delegation relationship on every worker event: a tool event names its delegation, a nested hand-off its parent, a control transfer an explicit type; unknown without evidence 83430e8b, 69d85d2a, 24dfc23f, a4bea940, 0b2f97b0 Your two cases: test_two_open_hand_offs_to_one_worker_each_hold_their_own_calls, test_a_framework_s_own_transfer_is_a_control_pass_and_holds_the_leaf_worker_s_calls. Also test_a_call_with_no_agent_above_it_has_an_unknown_place, test_a_parent_no_recorded_hand_off_carries_is_unknown_not_kept, test_each_payload_says_whether_part_of_it_was_left_out
2. Blank or duplicate worker ids silently dropped a097a2a6 test_a_worker_without_an_id_of_its_own_is_returned_to_authoring_and_keeps_only_contract_tools (worker[i]:no-id, duplicate-worker-ids)
3. Split the observation module d5601050 simulation_worker_events.py holds all span-to-event reading; test_worker_event_module_imports_only_the_standard_library_and_uses_python_39_syntax
4. Narrow the support claim this body The summary above lists the observed shapes and the unobserved states
5. Two overlapping calls of one tool swapped event ids and worker relationships when their spans ended in the other order fe44b142, 6aa538b0, 66411dd2 Your reversed-order case: test_two_overlapping_calls_of_one_tool_keep_their_own_spans_when_the_spans_end_in_the_other_order (both span orders). A span pairs by its event id, else with the one call its arguments, outcome or order among non-overlapping calls single out, and the two tracers' clocks are never compared (test_calls_that_did_not_overlap_pair_in_order_whatever_the_offset_between_the_two_clocks, test_calls_the_other_tracer_never_stamped_are_not_paired_by_order). Overlapping calls nothing tells apart, and two spans with equal claims on one call, stay unknown; the stronger of two unequal claims takes the call: test_overlapping_calls_of_one_tool_nothing_tells_apart_keep_their_agent_and_place_unknown, test_two_spans_claiming_one_call_go_to_the_stronger_claim_and_a_tie_leaves_it_unknown. One span and one call of a tool pair only when nothing both recorded differs (arguments, outcome, result, worker, hand-off target, kind): test_one_span_and_one_call_of_a_tool_pair_however_each_tracer_spelled_the_arguments, test_one_span_and_one_call_that_disagree_are_two_calls_each_tracer_missed_one. Consequence: overlapping calls of one tool from an instrumentor that records no arguments read unknown, and a span with no recorded end (a transfer, a job still running) is never paired by order either
6. The one-span-one-call shortcut merged calls without checking arguments or outcome (round 3) 8f020df3, 9a96d57c, e1ed2644, 9dac44d4 Your search(q=a) / failed search(q=b) case and its class: test_one_span_and_one_call_that_disagree_are_two_calls_each_tracer_missed_one, test_a_span_and_a_call_that_disagree_on_any_recorded_field_are_two_calls, test_order_pairs_only_calls_that_ended_the_same_way, test_spans_that_outnumber_the_calls_left_are_each_kept_without_a_guessed_owner, test_a_control_pass_made_through_a_tool_is_written_once_as_the_transfer
7. Final check (10 Nov): surplus spans were counted against calls they could not be, and worker identity was compared by raw id only 8d3851ec test_spans_count_only_against_the_calls_they_could_be_and_an_unrelated_call_keeps_no_owner, test_one_worker_named_differently_by_two_tracers_is_one_call (a framework's hashed key vs its role name; spelling), test_a_call_named_by_its_worker_label_never_takes_another_worker_s_place

The later commits read the span shapes found on real calls: OpenAI Agents SDK 0.24 through traceAI, where each run is a name-less agent span with the agents inside it, and Google ADK 2.13, where a transfer may carry no arguments and outlive its agent. They also fix the 6 Nov review findings (0b2f97b0, below).

Two framework-shape rules, stated plainly

  • OpenAI Agents SDK. In a run's container span, the first agent is the top of the app's work. A later agent in the same run is placed under a hand-off only when the framework records who handed it control (graph.node.parent_id), and stays unknown otherwise.
  • Google ADK. A TransferToAgentTool call with no recorded target goes to the next agent that runs in the same workflow, but only to one that started after the transfer ended and is not the agent that made it. Anything else leaves it a transfer with no target. A later agent under the workflow with no transfer into it is unknown.

Test matrix

Live rows on 0b2f97b0 (an earlier head) with future-agi #3699 at 2c9a663f1 (before its move to dev); the pairing rules in rows 6 and 7 came after it and are covered by the unit tests named there, not re-run live, on the local stack with production's sandbox specs. "Live" means a fresh call on 6 Nov on an existing build; the sandbox read its worker events with that head's code.

Case Unit E2E Live call Browser (#3699)
Each event names its span, parent hand-off, kind and relationship yes: span-shape tests in test_framework_tool_spans.py (73 on this head) no: ALK has no e2e flow for this yes: five calls (OpenAI Agents SDK twice, ADK, CrewAI, AutoGen Core) yes
Two overlapping invocations of one worker yes no no no
Control transfer then a leaf tool call yes: ADK built-in, plain function and no-argument shapes no yes: AutoGen Core topic transfer (defb17c4) yes
Place not recorded reads unknown yes no yes: CrewAI flow hand-off (21eb5743) yes: "parent unknown" chip
A no-target transfer is not taken by an earlier or the same agent yes: two tests no no no
Blank or repeated worker id yes: contract problem no no no
Input or output cut yes: per-field flags no no: no payload in that round passed 4,096 characters no
Forbidden write, per-worker verdict, timeouts, cancel no no no no: worker permissions and timeouts are TH-8887 and TH-8888

Live calls (one each, scenarios generated by the builder, not edited):

App Framework Call Workers observed Unknown steps
Returns reference, refusal OpenAI Agents SDK 391f86e7 order lookup; refund not observed 0
Returns reference, refund OpenAI Agents SDK 01a7a738 both 0
Travel concierge Google ADK 807df1e5 2 of 8 declared 0
Shakespeare post and verifier CrewAI 21eb5743 bard; verifier not observed 1
Unicycle repair and refund AutoGen Core defb17c4 triage and issues-and-repairs; sales not observed 0

The local sandbox ran 0b2f97b0's worker-event code. That code was taken from this branch merged into the local release candidate, which adds only unrelated chat fixes. LangGraph was not run live in that round: its local builds fail before the call (one seed_failed on a fingerprint mismatch, one response_path segment not found: messages), outside this PR.

Verified against code on 335b4b6, 10 Nov.

Worker permissions and timeouts (TH-8887, TH-8888): in agent-learning-kit #265

Per-worker tool permissions (each worker held to its own tools, an out-of-permission write refused and shown as an attempted violation, a per-worker verdict) and per-worker deadlines (a worker past its time limit ends as timed out) are written and tested, but are not in this PR. They are built on the world's action log and permission check that arrive with the actors work (agent-learning-kit #257, future-agi #3764), so they are in agent-learning-kit #265 and future-agi #3791, stacked on actors with this PR merged in.

Limitations

  • How a worker is identified. From the app's own spans and the rows this sandbox seals. A worker with its own tool-proxy channel, attributed by wiring, and per-worker permissions come with TH-8887. Until then permissions are not enforced.
  • What the record cannot tell: which hand-off a call ran in when the framework records no parent (unknown); a transfer's target when the framework records none and no later, different agent starts (a transfer with no target).
  • Per-span cost in the customer's process. Each span start walks up to the nearest agent and inserts into a dict under a lock. Each span end walks up once. A tool span does one json.dumps and appends one line. Every table is bounded (open spans 4,096, closed 4,096, launches, waiting jobs and held transfers 256), arguments and outputs are cut at 4,096 characters, and a line at 64 KB. Not bounded, and not changed here: the number of records per call (one per tool span), and the runner reading the whole trace file before merging, which pairs spans to calls in O(spans x calls).

Release

The sandbox reads worker events from the code baked into the E2B template. Shipping this needs a template bump pinned in the deployment repo. Local tests ran on production's sandbox specs (4 CPU, 8,192 MB, 23 GB, restricted egress). No migration.

1. Why

To show which worker of a multi-agent app did what, the platform needs two things: which workers the app declares, and, for every tool call, which worker made it and in which hand-off. This PR records declared workers in the agent contract, and makes the sandbox observe each tool call, its worker and its place for the shapes above, without changes to the app. It also adds a small reference app (a returns desk with a coordinator and two workers) to test against.

2. What changed

File Change Preserved
harness/contract.py DeclaredWorker (id, name, tools) and AgentContract.workers; a worker keeps only tools the contract has. A blank id (worker[i]:no-id) or a repeated one (duplicate-worker-ids) is returned by validate_contract as a problem to fix A contract without workers is unchanged
harness/tools.py, skills/understand-agent/SKILL.md workers in the contract submission schema, "an id no other worker has"; how to record workers from the agent's own code
harness/simulation_worker_events.py (new) All span-to-worker-event reading, standard library only and Python 3.9 syntax, never importing a customer module. It covers the delegation and control-transfer records, the two framework-shape rules above, the per-field omitted flags and the bounds above
harness/simulation_mark_bootstrap.py Keeps the hook lifecycle and reaches the event module through one entry; if that module fails to import, the call stamp is still written The mark, the ingest key and the call stamp are unchanged
harness/world/runtime.py, harness/run/simulation.py trace_row writes, per call a runtime attributed, span_id, parent_span_id, kind (delegated, transferred, tool_call), agent, relationship (observed, unknown), input_omitted, output_omitted, plus delegated_to (a control pass is kind: transferred). graded_calls leaves hand-off records out of checks A call with no runtime attribution is written exactly as before
harness/call_runner.py _merge_span_calls joins span lines to the calls other sources observed. _keep_recorded_parents keeps a parent only when a recorded hand-off carries it, and places a handed-control agent's calls in that control pass on the record alone Other sources' calls keep their own arguments and results
harness/crewai_runtime_observer_bootstrap.py Each agent a CrewAI flow runs is a hand-off from the flow, with every cut flagged Tool events are written as before
harness/process_runtime.py, harness/generated_runtime.py, harness/bundle_author_v2.py The hook and the event module are written beside each other for every observed process; every non-voice Python process gets the tool trace and the pinned framework instrumentor build step source_tool_trace_version stays 1
simulate/agent/wrappers/http.py A streamed reply's text is its text deltas joined Every other path reads as before
examples/harness/returns_reference/ A returns desk on the OpenAI Agents SDK with two workers; reference.json declares its topology
CHANGELOG.md Line under ## Unreleased

3. Behavior callouts

  • Declared is not observed: the contract says which workers exist; whether one ran is read from what the sandbox observed.
  • Sealed rows, old vs new: a row written before this PR has name, arguments, result, ok, error, refused, at. A row written now adds, only when set, worker_id, worker_label, task_id, task_label, delegated_to, ended_at, runtime_ids, input_omitted and output_omitted, and, for a call a runtime attributed, span_id, parent_span_id, kind, agent and relationship.
  • Framework-specific attribute names are read only where no shared convention exists (autogen.agent.name, claude_agent.subagent_type, graph.node.id, graph.node.parent_id, gen_ai.tool.type, lc_agent_name, langgraph_checkpoint_ns). Two patches are per framework because no public API gives the same result: AutoGen's BaseTool.run_json and the OpenAI Agents SDK's set_processors.
  • An agents-as-tools hand-off is a real tool call of the coordinator and stays graded; only a worker's run that is not a tool call is left out.
  • The reference app needs a model key (ANTHROPIC_API_KEY or OPENAI_API_KEY); the platform does not supply one.

4. Tests

Test file Scenarios it locks
test_framework_tool_spans.py (73 tests, 107 cases) Each framework's span shape gives the right worker and place; two open hand-offs to one worker; ADK built-in, plain-function and no-argument transfers; OpenAI Agents container runs (first agent top level, a later one unknown, a handed-control agent's calls in its control pass); a no-target transfer not taken by an earlier or the same agent; a later agent under a workflow unknown; an evicted parent reads unknown; held transfers bounded; per-field omitted flags; overlapping calls of one tool keep their own spans or stay unknown, under any clock offset, contention for one call, and 200 identical calls
test_contract.py Workers reach the brief; a blank or repeated id is a contract problem; only contract tools kept; no change for single agents
test_simulation_mark_bootstrap.py The hook and the event module import only the standard library and parse as Python 3.9
test_call_runner.py, test_chat_readiness.py, test_crewai_runtime_observer.py, test_process_runtime.py Sealed rows keep the new fields; hand-offs never graded; the CrewAI flow's hand-offs; the module written beside the hook

Each test added for the review and the 9 to 10 Nov fixes was run against its reverted fix and seen failing (15 mutations in the review, five more for the 9 to 10 Nov fixes). For the overlapping-call pairing, each of its 15 rules was removed in turn and that removal failed a test: 14 a test added for it, the outcome rule an earlier one. Some of those tests also pass on the code before the pairing fix, because they guard behaviour that already held there.

5. Commands to run the tests

uv run pytest tests/harness/test_framework_tool_spans.py tests/harness/test_simulation_mark_bootstrap.py \
  tests/harness/test_call_runner.py tests/harness/test_contract.py tests/harness/test_crewai_runtime_observer.py \
  tests/harness/test_process_runtime.py tests/harness/test_chat_readiness.py -q

On fe44b142: 641 passed. On 8d3851ec the touched modules (test_framework_tool_spans.py, test_call_runner.py, test_chat_call_runner.py) give 263 passed. The whole suite was not rerun on this head. On the previous head it gave 7 failures: 5 that fail on the base too (test_dispatch_ack, one grading cap) and 2 in test_isolated_process that pass when run alone.

Latest dev merged on 10 Nov (335b4b6b, no conflicts): test_framework_tool_spans.py, test_chat_worker.py, test_hosted_conversation.py, test_chat_call_runner.py, test_call_runner.py and test_builder_chat.py give 350 passed.

Touched suites

6. Steps to test on local

  1. Build an environment from the returns reference (examples/harness/returns_reference/) with an OPENAI_API_KEY or ANTHROPIC_API_KEY secret.
  2. Run a refund scenario and open the call's Participants tab (future-agi #3699).
  3. Confirm the coordinator hands work to the order-lookup worker, which calls get_order and list_refunds and returns.
  4. Confirm the refund worker's calls sit under its own hand-off.
  5. Turn on Raw events: each event sits under its recorded hand-off, or reads "parent unknown".

On the platform (recorded on an earlier head, before the 9 to 10 Nov fixes)

Walkthrough (2x): a two-turn returns call, a CrewAI hand-off with no recorded parent, and Open in Traces.

Walkthrough

Returns, two turns, the refund worker not observed

Google ADK: the hand-offs that ran

CrewAI: a hand-off with no recorded parent

Returns raw events: each tool call under its recorded hand-off

Checked live on the 10 Nov heads (with future-agi #3699)

The local stack ran this branch at 8d3851ec with future-agi #3699 at 645f62abe, production's sandbox specs (4 CPU / 8192 MB, restricted egress, svc-control), and the browser on #3699's frontend.

Path How Result
A first span read without the top-level span, then the second read Real spans of returns call 20cdea7e (two turns): the first read replayed on this head without the two root spans, then the real read First read kept with tree_complete: false; the second read merged to tree_complete: true. One hand-off per turn, each worker call (get_order, list_refunds) under its hand-off, coordinator "Returns coordinator", nothing doubled. Participants: "2 hand-offs to 1 worker. 2 finished."
A refused worker call Returns call 4f73a9a9, re-recorded on this head update_order_status shows as error under the order-lookup worker, with the world's reason as its output
A role-named CrewAI worker Live CrewAI call 4fa472e6 (run a3fc85a4) "Shakespearean Bard" is one worker (shakespearean_bard, instance "Shakespearean Bard"); "2 hand-offs to 1 worker", none counted twice. The app's own LLM call failed in the sandbox, so the hand-offs returned nothing
A nested hand-off Live Google ADK call e8557b7d (run 3ee2f0dc) place_agent sits under the control pass to inspiration_agent; "2 hand-offs to 2 workers. 1 finished, 1 passed control"; undeclared place_agent listed as not declared
Browser Participants on all four calls No console errors, no failed API calls, names as recorded
Live early read from export timing Not run New calls send spans to a hosted project, so a live first read without the root cannot happen on this stack; replayed from real spans above

First read without the root, then the full read: one hand-off per turn

A refused worker call shown as error

CrewAI: the role-named worker counted once

Google ADK: a nested hand-off under its parent

Not tested live

  • Two overlapping invocations of one worker, a no-target ADK transfer, cut payloads: unit tests only.
  • LangGraph on this head (local builds fail before the call, see above).
  • Claude Agent SDK, LlamaIndex, Pydantic AI and Strands apps on a hosted build.

Merge note

When #257 (actors) and this PR both reach dev, the second one meets a small conflict in call_runner.py, where the sealed trace is written. Keep both: each row is trace_row(call) plus #257's action marks, computed over graded_calls(calls), followed by #221's actor rows.

Upgrades

  1. No manual step beyond the template bump above.
  2. Old data works unattended: a bundle sealed before this PR runs and reseals as before; a sealed row from before has none of the new keys, and every reader treats them as optional.
  3. to 5. No migration and no table change.
  4. No new required config: ALK_OBSERVE_FRAMEWORKS and the instrumentor step are written by the build itself.
  5. No new stored state beyond optional keys on sealed rows, which travel with the existing trace artefact.
  6. The suites covering the changed modules pass on this head (section 5); CI runs the rest.

7. Scope in / scope out

In: declared workers in the contract; each tool call's worker and place for the observed shapes; the reference app.

Out: the worker timeline and UI (future-agi #3699); per-worker permissions and channels (TH-8887); worker timeouts and cancel states (TH-8888); worker-attributed world state (TH-8921); worker results in the CLI (TH-8847).

8. Design choices

  • Relationships come from the span tree only; nothing is read from names or event order beyond the two framework-shape rules above, each bound to recorded evidence.
  • Workers are named by the name the app gives them, because that is what their spans carry.
  • A worker's run that is not a tool call gets its own record kind, so grading never mistakes a hand-off for work done.
  • A blank or repeated worker id is refused back to authoring, so every worker keeps one identity.
  • The tool-trace version stays 1, since the bindings format did not change.

Tickets and plan

  • Linear: Part of TH-8886 and TH-8884, with future-agi #3699. TH-8886, delivered: declared workers in the agent contract (a blank or repeated id is an authoring problem); each sealed worker event names its worker, its span, the hand-off it ran in, its kind (delegated, transferred, tool_call) and whether that place was observed or is unknown; hand-off records are never graded. Left: the same record in CLI results (TH-8847); per-worker permissions, verifier and timeout of TH-8884 are in agent-learning-kit feat(harness): worker permissions and deadlines at the world's door (TH-8887, TH-8888) #265 and future-agi #3791.
  • Product plan shared on 1 Nov: T17 "Observed parent/worker IDs and delegation/return events are retained" and "Tool actions carry participant identity or Unknown" (with #3699). T16 "Use an internal test app for CI and acceptance, never shown in the product" (the returns reference under examples/), and "Prove supported transport/runtime and world bindings" in part: worker events observed live on OpenAI Agents SDK, Google ADK, CrewAI and AutoGen apps; LangGraph and Claude Agent SDK by span shape in unit tests only; per-worker world bindings come with feat(harness): worker permissions and deadlines at the world's door (TH-8887, TH-8888) #265.
  • Built for: TH-8886 "Two new events in the run's event history: delegated to worker and worker returned, correlated", "Worker attribution on every tool call event", "Evidence comes from what the app emits or what we instrument (the framework's trace spans)"; requirement 1 (a worker counts as observed only when its own events arrive).
  • Considered: relationships come from the span tree only, with two framework-shape rules each bound to recorded evidence; ambiguous calls stay unattributed rather than guessed; a blank or repeated worker id goes back to authoring so every worker keeps one identity; the support claim lists only the shapes observed.

Part of TH-8886

…t/TH-8886-declared-workers

# Conflicts:
#	src/fi/alk/harness/call_runner.py
#	src/fi/alk/harness/run/simulation.py
#	src/fi/alk/harness/world/runtime.py
Base automatically changed from rishav/th-8885-multi-agent-framework-fixtures to dev November 3, 2026 09:12
…it ran in, its kind and what bounding left out
…s, and the app's top agent hands off at the top level
…built-in transfer from any instrumentation counts once its agent runs
…hapes, including a transfer that outlives its agent
… Agents shape and give an ADK transfer with no target only to a later, different agent
@KarthikAvinashFI

Copy link
Copy Markdown
Member Author

Thanks for the review. Each point, with the commit and the test that locks it:

  1. Fixed in 83430e8, with the real OpenAI Agents and ADK 2 shapes in 69d85d2, 24dfc23, a4bea94 and 0b2f97b: every sealed event carries span_id, parent_span_id, kind, agent and relationship, read from the span tree, and unknown when it gives no evidence. Your two cases are test_two_open_hand_offs_to_one_worker_each_hold_their_own_calls and test_a_framework_s_own_transfer_is_a_control_pass_and_holds_the_leaf_worker_s_calls.
  2. Fixed in a097a2a: a blank or repeated worker id is returned to authoring as a contract problem instead of being dropped.
  3. Done in d560105: the span-to-event reading lives in simulation_worker_events.py (standard library only); the bootstrap keeps the hook lifecycle.
  4. Done in the body: the summary lists the observed runtime and framework shapes and reports unobserved and unknown as states, not as a universal claim.

@hadarishav hadarishav left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I rechecked the current head. The four earlier asks are fixed. Worker IDs now reject blank and duplicate values, delegation relationships reach sealed events, the observer logic moved out of the bootstrap, and the support claim now names the observed framework shapes.

One evidence-correlation bug still blocks approval. Two overlapping successful calls to the same tool can receive each other's runtime event IDs and worker relationship when the span records arrive in a different order from the native records. I reproduced this on this head with search(q=a) at 1, search(q=b) at 2, then span B followed by span A. The merged output assigned event A to q=b and event B to q=a. The pairing needs a stable identity when one exists. A fallback must use compatible payload and timing, and it must leave attribution unknown when the records are ambiguous rather than swap them. Please add the reversed-order, same-tool, same-outcome overlap as a regression test.

The focused ALK suites pass on this head with 1,182 tests. That confirms the existing coverage but does not catch this ordering case.

Comment thread src/fi/alk/harness/call_runner.py Outdated
# A job's hand-off sits under the call that launched it and is not that call again.
merged.append(span)
continue
same_tool = [c for c in unpaired if _tool_key(c.name) == _tool_key(span.name)]

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This pairs only by normalized tool name and outcome, then takes the first remaining call. For two concurrent successful calls to the same tool, reversed span completion order swaps their runtime IDs and worker relationships. I reproduced search(q=a) and search(q=b) becoming q=b -> event A and q=a -> event B. Correlate by a stable event ID when available, then compatible payload and time. If the remaining records are ambiguous, keep the attribution unknown instead of attaching the wrong worker evidence.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 66411dd: a span pairs by its event id, else with the one call its arguments, outcome or order among non-overlapping calls single out, and the two tracers' clocks are never compared; overlapping calls nothing tells apart, and two spans with equal claims on one call, stay unknown. Your case is test_two_overlapping_calls_of_one_tool_keep_their_own_spans_when_the_spans_end_in_the_other_order, with the clock-offset, contention and unknown cases beside it.

@KarthikAvinashFI

Copy link
Copy Markdown
Member Author

Fixed the overlapping same-tool pairing in 66411dd, with your reversed-order case and the clock-offset, contention and ambiguous cases as regression tests; each rule's test fails when that rule is removed. Could you take another look?

@hadarishav hadarishav left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-reviewed the current head 66411dd2. The earlier points are fixed, but one correlation bug still blocks approval.

When one native call and one span remain, _pair_tool merges them without checking arguments or outcome. I reproduced a successful native search(q=a) and a failed span search(q=b). The merged call kept q=a but received worker B's event and parent IDs, while the failed q=b disappeared.

Please apply the compatibility check before the one-to-one shortcut. If the records conflict, keep both or leave attribution unknown. Add this tracer-loss case as a regression test.

The focused suites passed with 197 tests.

Comment thread src/fi/alk/harness/call_runner.py Outdated
free = [
call for call in natives if call.runtime_ids.get("event_id") not in span_ids
]
if len(rest) == 1 and len(free) == 1:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This one-span, one-call shortcut bypasses fit(). One tracer can miss call A while the other misses call B, so count alone does not prove identity. With native search(q=a) succeeding and span search(q=b) failing, this merges them, assigns B's worker and relationship to A, and drops B. Require compatibility before merging. Keep both records when arguments or outcome conflict.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 8f020df3, 9a96d57c and e1ed2644. The one-to-one shortcut now goes through the same check as every other pairing path: a span and a call merge only when no recorded argument, outcome, result, worker id, hand-off target or kind differs. Otherwise both are kept with nothing lent across. Your search(q=a) / failed search(q=b) case is a regression.

@KarthikAvinashFI

Copy link
Copy Markdown
Member Author

Fixed, with the same check applied on every pairing path: the claim step, pairing by order and interchangeable calls, not only the one-to-one shortcut. Two related cases are fixed too: spans that outnumber the remaining calls are kept rather than dropped, with no worker or parent guessed (e1ed2644), and a control pass made through a tool is written once as the transfer (9dac44d4). Each has a regression that fails on the previous head.

@KarthikAvinashFI

Copy link
Copy Markdown
Member Author

One more small fix found while live-testing the stacked worker PR: the returns reference app answered every turn on a new event loop, so with an OpenAI model its second turn failed with "Event loop is closed". It now uses one loop for the process (0adb60b4), with a regression test that fails without it. Example app only; no harness change.

@KarthikAvinashFI
KarthikAvinashFI merged commit 8a2f12c into dev Nov 12, 2026
15 checks passed
@KarthikAvinashFI
KarthikAvinashFI deleted the feat/TH-8886-declared-workers branch November 12, 2026 09:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants