Bug Description
AgentResult.tool_calls and AgentResult.events are unreliable after run().
Of the 17 assertions in conductor.ai.agents.testing.assertions, the 8 that read events are
broken unconditionally, and the 5 that read tool_calls are broken for any non-OpenAI provider
and for most tool kinds. The remaining 4 (3 output, 1 status) are fine.
Four break by passing, because they check for absence and find an empty list:
assert_no_errors (assertions.py:198), assert_max_turns (:371),
assert_events_contain(..., expected=False) (:229), and the runner's own private
_assert_no_handoff (eval_runner.py:341, not one of the 17). assert_tool_not_used (:39)
joins them whenever detection fails.
assert_no_errors is the one that matters most, because expect_no_errors defaults to True
(eval_runner.py:98) so it runs on every case. The other always-on check, status
(eval_runner.py:239-241), reads result.status and is a genuine check.
Root Cause
Defects 1 and 2 are in src/conductor/ai/agents/runtime/runtime.py; defect 3 spans that file and
src/conductor/ai/agents/result.py.
1. Detection keys on call_, which is the provider's tool-call ID. runtime.py:5185
(_extract_tool_calls), and again at :3700 and :4229:
if not ref.startswith("call_"):
continue
The server does not add a call_ prefix. It seeds the reference from the provider's
toolCall.id(), falling back to a UUID, then appends the dynamic-fork index and the loop
iteration, giving call_PMnNIdOPvm9EQ8e6tn2kbxPY_0__1 for OpenAI. Anthropic IDs start toolu_
and are not matched.
This SDK writes such references itself. The Claude Agent SDK integration uses the Anthropic
tool-use ID as the reference name by design (frameworks/claude_agent_sdk.py:456-469, injected at
:983-1005), producing tasks like:
taskType = "SIMPLE" taskDefName = "Read" referenceTaskName = "toolu_01PJDP6YvZbhFp3wBnQeC2D3"
_extract_tool_calls drops them. Adding toolu_ is not a fix; the next provider picks its own
format.
2. Five tool kinds are filtered out, and five more are misnamed. _SYSTEM_TASK_TYPES
(:5143-5164) contains HTTP, CALL_MCP_TOOL, SUB_WORKFLOW, HUMAN and
PULL_WORKFLOW_MESSAGES, and every extraction site skips on it (:5181, :3701, :4230). All
five are real tool task types per the server's ToolCompiler.TYPE_MAP, so http, api, mcp,
agent_tool, human and pull_workflow_messages tools never appear at all.
GENERATE_IMAGE, GENERATE_AUDIO, GENERATE_VIDEO, LLM_INDEX_TEXT and LLM_SEARCH_INDEX are
absent from the set, so those are detected and then named from the task type by :5195
(task_type.lower()), giving "generate_image" rather than the tool's name.
Worker tools happen to work, because Conductor sets an executed SIMPLE task's taskType to the
task's own name (SimpleTaskMapper.java:86 in conductor-oss/conductor). The case folding still
corrupts a camelCase name: getWeather becomes getweather, and assert_tool_used compares
exactly.
The correct name is available and discarded. inputData.method is stripped as an internal key at
:5190, five lines before the guess, and neither task_def_name nor inputData._agent_tool_name
is read anywhere on this path.
3. run() never passes events=. runtime.py:2545 builds the result without it, so it stays
at the default_factory=list declared in src/conductor/ai/agents/result.py:123. Same at
:3997, :2631, :2737 and :5045. Passing an on_event callback does populate them
(:2443-2453), and so does stream(). The default path does not.
So the eval runner's own documented example cannot pass: the module docstring at
testing/eval_runner.py:19,26 uses expect_handoff_to, the runner calls run() with no
on_event at :229, and the assertion reads events, finds [], and fails every time.
Steps to Reproduce
Detection, with no server:
from unittest.mock import MagicMock
task = MagicMock()
task.reference_task_name = "toolu_01PJDP6YvZbhFp3wBnQeC2D3"
task.task_type = "SIMPLE"
task.task_def_name = "Read"
task.input_data = {"file_path": "/tmp/x"}
task.output_data = {}
wf = MagicMock()
wf.tasks = [task]
runtime._extract_tool_calls(wf) # [] expected: one call named "Read"
Events: run any agent and check result.events, which is [].
Naming: register an agent with a generate_image tool and check
result.tool_calls[0]["name"], which is "generate_image". Register one with an HTTP or MCP tool
and it is absent from tool_calls entirely.
Expected Behavior
Identify a tool task by task type, allowlisting off TYPE_MAP plus the worker case, and resolve
the name from a field that carries it:
def _is_tool_task(task):
if (task.reference_task_name or "").startswith("_fw_"):
return False
return task.task_type in TOOL_TASK_TYPES or task.task_def_name is not None
def _tool_name(task):
ind = task.input_data or {}
return (ind.get("_agent_tool_name") or ind.get("method")
or task.task_def_name or task.task_type)
No case folding, and nothing reads the reference name for identity, so nothing depends on provider
data. _agent_tool_name is the only key the server sets for every tool kind.
run() should populate events on the default path, or the documented example should stop
relying on it.
Additional Notes
Verified against origin/main @ c99e2cf9. Payloads captured against OSS Conductor 3.32.0 with
--agentspan.embedded=true: one server-compiled worker tool with an OpenAI ID, and the Claude
Agent SDK task quoted above.
Derived from source rather than captured: the five filtered tool kinds, the five misnamed ones,
and the Anthropic case on the server-compiled path, since we have no Anthropic key on hand.
tests/unit/ai/test_runtime.py:819 covers the server-compiled worker case and passes for the
right reason. Nothing covers a non-OpenAI ID, a non-worker tool kind, or a camelCase name.
Same family of defects filed against javascript-sdk, java-sdk and csharp-sdk. A related
server-side issue is filed against conductor-oss/conductor, where the same four tool kinds are
excluded from tool-event emission.
Bug Description
AgentResult.tool_callsandAgentResult.eventsare unreliable afterrun().Of the 17 assertions in
conductor.ai.agents.testing.assertions, the 8 that readeventsarebroken unconditionally, and the 5 that read
tool_callsare broken for any non-OpenAI providerand for most tool kinds. The remaining 4 (3 output, 1 status) are fine.
Four break by passing, because they check for absence and find an empty list:
assert_no_errors(assertions.py:198),assert_max_turns(:371),assert_events_contain(..., expected=False)(:229), and the runner's own private_assert_no_handoff(eval_runner.py:341, not one of the 17).assert_tool_not_used(:39)joins them whenever detection fails.
assert_no_errorsis the one that matters most, becauseexpect_no_errorsdefaults toTrue(
eval_runner.py:98) so it runs on every case. The other always-on check,status(
eval_runner.py:239-241), readsresult.statusand is a genuine check.Root Cause
Defects 1 and 2 are in
src/conductor/ai/agents/runtime/runtime.py; defect 3 spans that file andsrc/conductor/ai/agents/result.py.1. Detection keys on
call_, which is the provider's tool-call ID.runtime.py:5185(
_extract_tool_calls), and again at:3700and:4229:The server does not add a
call_prefix. It seeds the reference from the provider'stoolCall.id(), falling back to a UUID, then appends the dynamic-fork index and the loopiteration, giving
call_PMnNIdOPvm9EQ8e6tn2kbxPY_0__1for OpenAI. Anthropic IDs starttoolu_and are not matched.
This SDK writes such references itself. The Claude Agent SDK integration uses the Anthropic
tool-use ID as the reference name by design (
frameworks/claude_agent_sdk.py:456-469, injected at:983-1005), producing tasks like:_extract_tool_callsdrops them. Addingtoolu_is not a fix; the next provider picks its ownformat.
2. Five tool kinds are filtered out, and five more are misnamed.
_SYSTEM_TASK_TYPES(
:5143-5164) containsHTTP,CALL_MCP_TOOL,SUB_WORKFLOW,HUMANandPULL_WORKFLOW_MESSAGES, and every extraction site skips on it (:5181,:3701,:4230). Allfive are real tool task types per the server's
ToolCompiler.TYPE_MAP, sohttp,api,mcp,agent_tool,humanandpull_workflow_messagestools never appear at all.GENERATE_IMAGE,GENERATE_AUDIO,GENERATE_VIDEO,LLM_INDEX_TEXTandLLM_SEARCH_INDEXareabsent from the set, so those are detected and then named from the task type by
:5195(
task_type.lower()), giving"generate_image"rather than the tool's name.Worker tools happen to work, because Conductor sets an executed SIMPLE task's
taskTypeto thetask's own name (
SimpleTaskMapper.java:86inconductor-oss/conductor). The case folding stillcorrupts a camelCase name:
getWeatherbecomesgetweather, andassert_tool_usedcomparesexactly.
The correct name is available and discarded.
inputData.methodis stripped as an internal key at:5190, five lines before the guess, and neithertask_def_namenorinputData._agent_tool_nameis read anywhere on this path.
3.
run()never passesevents=.runtime.py:2545builds the result without it, so it staysat the
default_factory=listdeclared insrc/conductor/ai/agents/result.py:123. Same at:3997,:2631,:2737and:5045. Passing anon_eventcallback does populate them(
:2443-2453), and so doesstream(). The default path does not.So the eval runner's own documented example cannot pass: the module docstring at
testing/eval_runner.py:19,26usesexpect_handoff_to, the runner callsrun()with noon_eventat:229, and the assertion readsevents, finds[], and fails every time.Steps to Reproduce
Detection, with no server:
Events: run any agent and check
result.events, which is[].Naming: register an agent with a
generate_imagetool and checkresult.tool_calls[0]["name"], which is"generate_image". Register one with an HTTP or MCP tooland it is absent from
tool_callsentirely.Expected Behavior
Identify a tool task by task type, allowlisting off
TYPE_MAPplus the worker case, and resolvethe name from a field that carries it:
No case folding, and nothing reads the reference name for identity, so nothing depends on provider
data.
_agent_tool_nameis the only key the server sets for every tool kind.run()should populateeventson the default path, or the documented example should stoprelying on it.
Additional Notes
Verified against
origin/main@c99e2cf9. Payloads captured against OSS Conductor 3.32.0 with--agentspan.embedded=true: one server-compiled worker tool with an OpenAI ID, and the ClaudeAgent SDK task quoted above.
Derived from source rather than captured: the five filtered tool kinds, the five misnamed ones,
and the Anthropic case on the server-compiled path, since we have no Anthropic key on hand.
tests/unit/ai/test_runtime.py:819covers the server-compiled worker case and passes for theright reason. Nothing covers a non-OpenAI ID, a non-worker tool kind, or a camelCase name.
Same family of defects filed against
javascript-sdk,java-sdkandcsharp-sdk. A relatedserver-side issue is filed against
conductor-oss/conductor, where the same four tool kinds areexcluded from tool-event emission.