Spans that end in ERROR don't set error.type (only tool spans do)

🔴 Required Information

Describe the Bug:
When a model call fails, every span the exception passes through ends with status ERROR, but none of them get the error.type attribute. That covers invocation / invoke_workflow, invoke_agent, call_llm and generate_content. Only execute_tool sets it today (via trace_tool_call). For a failing tool, the tool span has error.type but the invoke_agent and invoke_workflow spans above it don't.

The metrics for the same operations already record it. In the inference-error-resource-exhausted functional case the generate_content and invoke_workflow metric points carry error.type: "429", and in inference-error-valueerror they carry "ValueError", while the matching spans have nothing. The GenAI semantic conventions expect error.type on a span that ended in an error, so anything filtering or alerting on it in traces misses these failures.

This is already tracked in tests/unittests/telemetry/test_semconv_conformance.py: test_error_status_implies_error_type is a strict xfail for four cases with the reason "the metric for the same operation records error.type; the span does not".

Steps to Reproduce:

Check out main
Run pytest tests/unittests/telemetry/test_semconv_conformance.py --runxfail -k error_status_implies_error_type
Four cases fail with "ERROR spans without error.type"

Expected Behavior:
Every span that ends in ERROR has error.type, with the same value the metric records for that operation (tracing.resolve_error_type()).

Observed Behavior:

FAILED ...::test_error_status_implies_error_type[agent-inference-error-resource-exhausted-schema-v1]
FAILED ...::test_error_status_implies_error_type[agent-inference-error-resource-exhausted-schema-v2]
FAILED ...::test_error_status_implies_error_type[agent-inference-error-valueerror-schema-v2]
FAILED ...::test_error_status_implies_error_type[agent-tool-error-valueerror-schema-v2]
4 failed, 60 passed

In the inference-error-resource-exhausted-schema-v2 golden, all four spans (invoke_workflow, invoke_agent, call_llm, generate_content) are ERROR without error.type.

Environment Details:

ADK Library Version (pip show google-adk): main at 3c31fa5
Desktop OS: Linux
Python Version (python -V): 3.11.16

Model Information:

Are you using LiteLLM: No
Which model is being used: N/A, telemetry only (the functional cases use a mocked gemini-2.5-flash)

🟡 Optional Information

Additional Context:
Most of these spans get their ERROR status from OpenTelemetry's start_as_current_span when an exception escapes, which records the exception but doesn't know about error.type. The workflow span is a separate path: in node_tracing.py a node hands its failure back as data, and the span is marked ERROR explicitly without setting the attribute.

The fix: a small helper around span creation that sets error.type from resolve_error_type() when an exception passes through, used at the places that create these spans, plus setting it next to the explicit set_status in node_tracing.py. The four xfail entries and their goldens are updated in the same change.

I have a fix ready along these lines and will open a PR for it shortly.

How often has this issue occurred?:

Always (100%)

@ankit2235

Reproduced this on 2.11.0 outside the test suite too @ankit2235, with a plain LlmAgent and a failing model: invoke_agent, call_llm, generate_content and invocation/invoke_workflow all end in ERROR with no error.type on both schema v1 and v2.

A possible workaround in the meantime: wrap tracing.tracer.start_as_current_span so it sets error.type from resolve_error_type() before re-raising (basically the same idea as your PR). With that every ERROR span showed "ValueError" for both the model and the tool failure.

Could you check whether that holds up on your side too especially the node_tracing path where the workflow span is marked ERROR without an exception unwinding?

Please give the PR a full run with the regenerated goldens before review so we know it's solid. Will keep this issue updated.

@surajksharma07

Thanks for reproducing this @surajksharma07.

Yes, it holds up on my side, including the node_tracing path. That's the tool-error-valueerror-schema-v2 case: the tool failure is handed back as data, and with the fix the regenerated golden has error.type: "ValueError" on invoke_workflow as well as on invoke_agent and execute_tool.

Full run on the PR branch with the regenerated goldens:

pytest tests/unittests -n 4
18450 passed, 86 skipped, 20 xfailed, 2 xpassed

The fix is in #7495.

@ankit2235

Thanks for confirming the node_tracing path and for the full run @ankit2235. Merged #7495 onto current main (ca3b490), no conflicts: test_error_status_implies_error_type with --runxfail goes from 4 failed / 60 passed to 64 passed, my standalone repro passes on schema v1 and v2 and the regenerated goldens carry 429/ValueError on invoke_workflow (or invocation), invoke_agent, call_llm and generate_content matching what the metrics record.

The wrapper only catches Exception so cancellations aren't tagged and the node_tracing set_attribute sits next to the explicit set_status which looks right. The other raw tracer spans (compaction, context caching, managed_agent_interaction) still won't carry error.type but that's fine as a follow-up rather than in this PR.

Nothing else needed from your side. @jawoszek could you take a look at #7495 when you get a chance and approve the 2 pending workflows?

Team is looking more into it.

I would like to take the compaction follow-up mentioned above. On current main, _summarize_events_with_trace catches summarizer errors and sets the span to ERROR, but it does not set error.type.

I plan to reuse resolve_error_type in that exception branch and add regression coverage for ordinary exceptions and GenAI API errors, while keeping successful compaction and cancellation behavior unchanged. This would stay separate from #7495 and the other raw tracer spans.

Is this already covered by an internal change, or would a small follow-up PR be welcome?

@Samecho

Thanks for picking this up @Samecho. No compaction change on main or in another open PR so #7532 is welcome. Checked it against today’s main (156f007): applies cleanly, test_compaction.py passes 71/71 and both new error.type cases fail on main without the fix. It doesn’t overlap with #7495 so it can be reviewed on its own.

@ankit2235 heads-up: #7495 no longer applies to main. 10c694f (already in 2.12.0) moved invocation, invoke_agent, call_llm and invoke_workflow off start_as_current_span onto tracer.start_span with errors now recorded in _instrumentation._recording_span and an explicit except in _model_call.py. So the new wrapper only reaches generate_content now.

Could you rebase and set error.type in those two places plus the node_tracing branch then re-record the goldens? test_error_status_implies_error_type still has all four cases xfailed on main so the bug itself is still there.

That also means the start_as_current_span patch suggested earlier only works up to 2.11. On 2.12 it tags only generate_content so anyone on 2.12 should rely on the metric’s error.type until the rebased #7495 lands.

fix(telemetry): set error.type on spans that end in an error

@surajksharma07

Thanks for the heads-up @surajksharma07. I rebased #7495 onto current main (d01d5d8) and reworked it for the new span handling:

_instrumentation._recording_span now sets error.type from resolve_error_type(), which covers invocation, invoke_agent and invoke_workflow
the except in _model_call.py sets it on call_llm
the node_tracing branch sets it next to the explicit set_status
the start_as_current_span wrapper is kept only for the spans still opened that way (generate_content, invoke_node)

Goldens are re-recorded with regenerate (71 recorded, 33 divergences, all explained), and the four xfails in test_error_status_implies_error_type are removed.

pytest tests/unittests/telemetry -n 4
939 passed, 3 xfailed

pytest tests/unittests -n 4
19237 passed, 89 skipped, 15 xfailed, 2 xpassed

@ankit2235

The approach looks right @ankit2235: _recording_span, the call_llm except, the node_tracing branch and the wrapper for generate_content/invoke_node cover every span in the failing chain.

Main has moved again though so #7495 conflicts on today’s main (8ff9941) in _model_call.py, test_semconv_conformance.py, functional_divergences.json and the two inference-error v2 goldens. 5909a0e drops call_llm under schema v2 (the span now comes from _start_call_llm_span and can be None) and the xfail list grew to six with inference-error-valueerror-span-and-event-schema-v2 and tool-error-valueerror-span-only-schema-v2.

Resolved it locally by putting the set_attribute on call_llm_span inside the existing if call_llm_span is not None check and dropping all six xfails. After regenerating test_error_status_implies_error_type passes for every case and both new cases carry ValueError on invoke_workflow, invoke_agent and generate_content/execute_tool. The generate_content error.type entries in functional_divergences.json still marked adk_bug (including the ValueError one) need the same update you made for 429.

Could you rebase once more on current main with those changes and re-record the goldens? Once CI is green it should be ready for @jawoszek to review.

fix(telemetry): set error.type on spans that end in an error

@surajksharma07

Thanks @surajksharma07. Rebased #7495 onto current main (a221ccd): the call_llm attribute is set inside the existing if call_llm_span is not None check, all six xfails are removed, and the goldens are re-recorded.

On the divergences: after regenerating with the fix, the span-level generate_content error.type entry no longer comes up at all, since ADK's span now reports the same value as the instrumentor, so I removed it rather than rewording it. The gen_ai.client.inference.operation.details log entry for error.type still comes up (the inference log doesn't carry it, and this PR doesn't touch logs), so I left it as adk_bug. I also kept the rest of the file as on main, since regenerate on an unchanged main already drops an unrelated compaction entry.

pytest tests/unittests/telemetry/test_semconv_conformance.py -k error_status_implies_error_type
67 passed

pytest tests/unittests/telemetry -n 4
1025 passed, 3 xfailed


pytest tests/unittests -n 4
20728 passed, 90 skipped, 12 xfailed, 2 xpassed (3 failures that also occur on unchanged main: test_release_dependencies and two test_samples cases)

@ankit2235

Went through 406721d @ankit2235 and it merges cleanly onto today’s main (7f3517e). None of the commits since a221ccd touch telemetry. With --runxfail on main test_error_status_implies_error_type still fails the six cases and with #7495 merged all six pass. 2.12.0 is still the latest release so the bug is still in every release. Removing the span-level generate_content divergence and keeping the inference log entry as adk_bug makes sense since this PR doesn’t touch logs.

@Samecho #7532 also still merges cleanly. test_compaction.py passes 77/77 with it and the new error.type cases fail on main without the fix.

Nothing else needed from either of you. @jawoszek could you review #7495 and #7532 and approve the pending workflows on both when you get a chance?

fix: set error.type on spans that end in an error

fix: classify compaction failures with error.type

Navigation Menu

Repository navigation

Spans that end in ERROR don't set error.type (only tool spans do) #7494

🔴 Required Information

🟡 Optional Information

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Spans that end in ERROR don't set error.type (only tool spans do) #7494

Description

🔴 Required Information

🟡 Optional Information

Activity

surajksharma07 commented on Oct 12, 2026

ankit2235 commented on Oct 12, 2026

surajksharma07 commented on Oct 12, 2026

Samecho commented on Oct 14, 2026

surajksharma07 commented on Oct 15, 2026

ankit2235 commented on Oct 15, 2026

surajksharma07 commented on Oct 19, 2026

ankit2235 commented on Oct 21, 2026

surajksharma07 commented on Oct 23, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions