Skip to content

Tool-call extraction drops or misnames most tool kinds, and run() returns no events #501

Description

@ambiorix2099

Bug Description

AgentResult.tool_calls and AgentResult.events are unreliable after run().

Of the 17 assertions in conductor.ai.agents.testing.assertions, the 8 that read events are
broken unconditionally, and the 5 that read tool_calls are broken for any non-OpenAI provider
and for most tool kinds. The remaining 4 (3 output, 1 status) are fine.

Four break by passing, because they check for absence and find an empty list:
assert_no_errors (assertions.py:198), assert_max_turns (:371),
assert_events_contain(..., expected=False) (:229), and the runner's own private
_assert_no_handoff (eval_runner.py:341, not one of the 17). assert_tool_not_used (:39)
joins them whenever detection fails.

assert_no_errors is the one that matters most, because expect_no_errors defaults to True
(eval_runner.py:98) so it runs on every case. The other always-on check, status
(eval_runner.py:239-241), reads result.status and is a genuine check.

Root Cause

Defects 1 and 2 are in src/conductor/ai/agents/runtime/runtime.py; defect 3 spans that file and
src/conductor/ai/agents/result.py.

1. Detection keys on call_, which is the provider's tool-call ID. runtime.py:5185
(_extract_tool_calls), and again at :3700 and :4229:

if not ref.startswith("call_"):
    continue

The server does not add a call_ prefix. It seeds the reference from the provider's
toolCall.id(), falling back to a UUID, then appends the dynamic-fork index and the loop
iteration, giving call_PMnNIdOPvm9EQ8e6tn2kbxPY_0__1 for OpenAI. Anthropic IDs start toolu_
and are not matched.

This SDK writes such references itself. The Claude Agent SDK integration uses the Anthropic
tool-use ID as the reference name by design (frameworks/claude_agent_sdk.py:456-469, injected at
:983-1005), producing tasks like:

taskType = "SIMPLE"   taskDefName = "Read"   referenceTaskName = "toolu_01PJDP6YvZbhFp3wBnQeC2D3"

_extract_tool_calls drops them. Adding toolu_ is not a fix; the next provider picks its own
format.

2. Five tool kinds are filtered out, and five more are misnamed. _SYSTEM_TASK_TYPES
(:5143-5164) contains HTTP, CALL_MCP_TOOL, SUB_WORKFLOW, HUMAN and
PULL_WORKFLOW_MESSAGES, and every extraction site skips on it (:5181, :3701, :4230). All
five are real tool task types per the server's ToolCompiler.TYPE_MAP, so http, api, mcp,
agent_tool, human and pull_workflow_messages tools never appear at all.

GENERATE_IMAGE, GENERATE_AUDIO, GENERATE_VIDEO, LLM_INDEX_TEXT and LLM_SEARCH_INDEX are
absent from the set, so those are detected and then named from the task type by :5195
(task_type.lower()), giving "generate_image" rather than the tool's name.

Worker tools happen to work, because Conductor sets an executed SIMPLE task's taskType to the
task's own name (SimpleTaskMapper.java:86 in conductor-oss/conductor). The case folding still
corrupts a camelCase name: getWeather becomes getweather, and assert_tool_used compares
exactly.

The correct name is available and discarded. inputData.method is stripped as an internal key at
:5190, five lines before the guess, and neither task_def_name nor inputData._agent_tool_name
is read anywhere on this path.

3. run() never passes events=. runtime.py:2545 builds the result without it, so it stays
at the default_factory=list declared in src/conductor/ai/agents/result.py:123. Same at
:3997, :2631, :2737 and :5045. Passing an on_event callback does populate them
(:2443-2453), and so does stream(). The default path does not.

So the eval runner's own documented example cannot pass: the module docstring at
testing/eval_runner.py:19,26 uses expect_handoff_to, the runner calls run() with no
on_event at :229, and the assertion reads events, finds [], and fails every time.

Steps to Reproduce

Detection, with no server:

from unittest.mock import MagicMock

task = MagicMock()
task.reference_task_name = "toolu_01PJDP6YvZbhFp3wBnQeC2D3"
task.task_type = "SIMPLE"
task.task_def_name = "Read"
task.input_data = {"file_path": "/tmp/x"}
task.output_data = {}

wf = MagicMock()
wf.tasks = [task]
runtime._extract_tool_calls(wf)     # []   expected: one call named "Read"

Events: run any agent and check result.events, which is [].

Naming: register an agent with a generate_image tool and check
result.tool_calls[0]["name"], which is "generate_image". Register one with an HTTP or MCP tool
and it is absent from tool_calls entirely.

Expected Behavior

Identify a tool task by task type, allowlisting off TYPE_MAP plus the worker case, and resolve
the name from a field that carries it:

def _is_tool_task(task):
    if (task.reference_task_name or "").startswith("_fw_"):
        return False
    return task.task_type in TOOL_TASK_TYPES or task.task_def_name is not None

def _tool_name(task):
    ind = task.input_data or {}
    return (ind.get("_agent_tool_name") or ind.get("method")
            or task.task_def_name or task.task_type)

No case folding, and nothing reads the reference name for identity, so nothing depends on provider
data. _agent_tool_name is the only key the server sets for every tool kind.

run() should populate events on the default path, or the documented example should stop
relying on it.

Additional Notes

Verified against origin/main @ c99e2cf9. Payloads captured against OSS Conductor 3.32.0 with
--agentspan.embedded=true: one server-compiled worker tool with an OpenAI ID, and the Claude
Agent SDK task quoted above.

Derived from source rather than captured: the five filtered tool kinds, the five misnamed ones,
and the Anthropic case on the server-compiled path, since we have no Anthropic key on hand.

tests/unit/ai/test_runtime.py:819 covers the server-compiled worker case and passes for the
right reason. Nothing covers a non-OpenAI ID, a non-worker tool kind, or a camelCase name.

Same family of defects filed against javascript-sdk, java-sdk and csharp-sdk. A related
server-side issue is filed against conductor-oss/conductor, where the same four tool kinds are
excluded from tool-event emission.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions