Skip to content

feat(guardrails): group guardrail checks into stage containers - #1222

Open
daniel-joica wants to merge 11 commits into
mainfrom
feat/guardrail-stage-grouping
Open

daniel-joica wants to merge 11 commits into
mainfrom
feat/guardrail-stage-grouping

Conversation

@daniel-joica

@daniel-joica daniel-joica commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

3 of 3 (#1220 and #1221 are merged; this is now based on main):

  1. feat(guardrails): record each guardrail action as a span [AL-712] #1220 (merged): a span for each guardrail action.
  2. feat(guardrails): record deterministic rule checks as spans #1221 (merged): a span for each deterministic rule check.
  3. This PR: grouping checks into stage containers.

Works with the published uipath-platform; no platform change needed. Together with UiPath/uipath-python#1997, which removes the duplicate client evaluate_guardrail span.

What

Each guardrail middleware hook is its own LangGraph node, so seven guardrails gave about seven separate guardrail nodes around each model call. Low-code agents group a stage's checks into one container. This PR does the same for coded agents, without changing how guardrails are written (middleware=[*UiPathPIIDetectionMiddleware(...), ...]):

LangGraph
├─ LLM input guardrail check   (llmPreGuardrails)   ← shared by 4 guardrail nodes
│  ├─ PII Detection             (Guardrails Service)
│  ├─ Log                       (guardrailAction, #1220)
│  ├─ Harmful Content Detection
│  └─ ...
├─ model                        ← its start closes the container
├─ tools
│  ├─ Tool input guardrail check  → Tool PII Detection, London Weather Rule (#1221), Log
│  ├─ get_weather
│  └─ Tool output guardrail check
└─ LLM output guardrail check / Agent output guardrail check

Container names and span types are the low-code ones (agentPre/Post, llmPre/Post, toolPre/PostGuardrails).

How

In guardrails/middlewares/_grouping.py (new):

  • guardrail_stage(scope, stage) wraps each check (evaluation and action). The first check of a stage creates a shared container next to the guardrail nodes (inside tools for tool checks); later guardrail nodes of that stage reuse it. It's lazy: a stage where every check is skipped leaves nothing.
  • guardrail_aware(provider) wraps the runtime's current span provider so a running check reports its container as the current span. @traced spans, the action span and the Guardrails Service's span (through the trace headers) all land in it.
  • GuardrailStageCloser (span processor) ends a container when a sibling that isn't this stage's guardrail node starts (at that sibling's start time, but never before its last check ended: coarse clocks such as Windows before Python 3.13 can stamp the sibling a tick early), or when its parent ends.
  • is_exported (span filter in the factory's trace settings) hides the now-empty guardrail node spans: only OpenInference spans with a registered guardrail node name, only while grouping is on.
  • Containers copy execution.id from the node (their parent is non-recording), so execution-based lookups (evals) include them.
  • A container that closes without an error gets an explicit OK status, like the action and evaluation spans (feat(guardrails): record each guardrail action as a span [AL-712] #1220 review); a failed one keeps ERROR.
  • Creating a stage's container also closes the other stage's container under the same parent, for when no sibling starts in between (a middleware that answers a tool call without running the tool).
  • If a container can't be created, the check runs ungrouped and a warning is logged, so grouping never skips a guardrail.
  • No grouping under a non-recording (sampled-out) span or an unsampled parent, whose end never reaches the closer. A guardrail node in that position stays visible, so its checks keep their parent.
  • The closer returns early, without the lock, while no container is open.

Opt-in by use: runtime/factory.py calls enable_guardrail_grouping(trace_manager, get_current_span), which only registers the provider wrapper and offers the trace manager. A closer is attached, and grouping turns on, once a guardrail middleware is built (guardrails_in_use()). Agents without guardrails, and low-code agents (whose factory subclasses this one), get no closer and no filtering. When the trace manager shuts down, the closer unregisters itself, forgets that manager and ends anything still open; the next run adds a new one.

Other changes:

  • _spans.py keeps only the spans of a single check (action and rule evaluation, feat(guardrails): record each guardrail action as a span [AL-712] #1220/feat(guardrails): record deterministic rule checks as spans #1221); grouping moved to _grouping.py.
  • _span_name.py: SpanType and SpanName, the guardrail span types and container names, with the same member names as low-code's SpanName/SpanType.
  • _base.py: checks run in guardrail_stage; message hooks register their node names; the tool wrapper signals use.
  • deterministic.py: the rule check runs in the tool stage, and building the middleware signals use.
  • test_action_span.py: each test starts with fresh grouping state.
  • Version 0.20.7 → 0.20.8.

Known limits

  • Anything that runs between two guardrail nodes (another middleware) starts a new container for the rest of that stage.
  • LLM checks sit next to model, not inside an LLM call span as in low-code: before_model, model and after_model are separate nodes. Wrapping the model call (wrap_model_call) could add that level later.
  • A custom action that runs a LangChain runnable inside a guardrail hook creates spans under the hidden node, and in a tool stage it ends the container before the check finishes.
  • is_exported hides guardrail node spans by name across the process, so another graph's runnable with exactly that name would be hidden too.
  • If grouping fails for a check, its spans fall back to the guardrail node, which is already hidden (the decision is made when the node starts).
  • Cancelling a run during a tool check leaves the check running in its worker thread; its container can end before the check does.
  • Containers are keyed by parent span id, not trace id; two traces only share a key with a custom span ID generator.
  • While containers are open, each span start and end scans them under one process-wide lock.
  • Once a guardrail middleware is built, grouping and node hiding stay on for the rest of the process.
  • If two trace managers are alive at once, one's shutdown() clears the shared processor list (uipath-core), which also silences the other's closer, and grouping turns off for the rest of the other's run. Today's CLI flows use one at a time.

Tests

  • New test_grouping.py (31):
    • stages: nodes of one stage share a container that ends exactly at model's start, and never before its last check; execution.id copied; a new stage gets its own container; a node in between splits it; LLM output; a block marks the container failed; a grouping failure still runs the check; nothing to check means no container;
    • tools: input and output containers with exact boundaries; the input container closes when the output stage starts without a tool span, stays closable if the output stage can't start, and still ends when the closer never saw the new container start; a container closed during a check stays closed;
    • enabling: off without a closer, under an invalid span, and without guardrail middlewares; one closer per trace manager, back after shutdown, a shut-down manager isn't reused; nothing grouped under a sampled-out node or parent; the closer skips the lock with nothing open; starting a container doesn't deadlock; tool-only guardrails turn it on; only OpenInference guardrail node spans are hidden, and only when they can be grouped;
    • real create_agent + OpenInference: 2 parallel tool calls, 3 parallel tool calls with deliberately overlapping checks, and 2 concurrent runs (no orphans, nothing left open, every check inside its container).
  • Guardrail tests: 469 passed (full suite runs in CI). ruff, ruff format and mypy on the changed code are clean.
  • End to end: jobs on a local robot (an agent with 7 guardrails) show the structure above in LLMOps, with every service evaluation inside its container and no guardrail node spans exported.
  • Reviewed by three independent reviewers before this split; their findings are fixed or listed under Known limits. feat(guardrails): group coded-agent guardrail spans like low-code agents [AL-712] #1219 holds the earlier, single-PR version and its review thread.

🤖 Generated with Claude Code

Development Package

  • Use uipath pack --nolock to get the latest dev build from this PR (requires version range).
  • Add this package as a dependency in your pyproject.toml:
[project]
dependencies = [
  # Exact version:
  "uipath-langchain==0.20.8.dev1012226357",

  # Any version from PR
  "uipath-langchain>=0.20.8.dev1012220000,<0.20.8.dev1012230000"
]

[[tool.uv.index]]
name = "testpypi"
url = "https://test.pypi.org/simple/"
publish-url = "https://test.pypi.org/legacy/"
explicit = true

[tool.uv.sources]
uipath-langchain = { index = "testpypi" }

[tool.uv]
override-dependencies = [
    "uipath-langchain>=0.20.8.dev1012220000,<0.20.8.dev1012230000",
]

Copilot AI balanced review requested due to automatic review settings November 18, 2026 12:27

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

A tracing failure can bypass blocking guardrails, and unresolved span lifecycle and filtering defects can corrupt traces.

5 open issues
What changed in this PR

Completes the guardrail tracing work by grouping coded-agent checks into stage containers aligned with low-code traces, without changing middleware usage.

Changes:

  • Adds shared stage spans, automatic closure, and guardrail-node filtering.
  • Integrates grouping with runtime setup and built-in and deterministic checks.
  • Adds grouping tests, isolates action-span test state, and bumps the version to 0.20.5.
File Description
uv.lock Updates the locked package version to 0.20.5.
tests/​guardrails/​middlewares/​test_grouping.py Tests grouping, boundaries, activation, filtering, and concurrent runs.
tests/​guardrails/​middlewares/​test_action_span.py Resets grouping state for test isolation.
src/​uipath_langchain/​runtime/​factory.py Registers grouping support and the span filter.
src/​uipath_langchain/​guardrails/​middlewares/​deterministic.py Wraps deterministic tool checks in stage containers.
src/​uipath_langchain/​guardrails/​middlewares/​_spans.py Implements container sharing, parenting, closure, and filtering.
src/​uipath_langchain/​guardrails/​middlewares/​_base.py Registers guardrail hooks and wraps built-in checks in stages.
pyproject.toml Bumps the package version to 0.20.5.

Review effort: Balanced

@daniel-joica
daniel-joica force-pushed the feat/guardrail-deterministic-spans branch from 13cf3c6 to 6fa85ec Compare November 19, 2026 13:00
Base automatically changed from feat/guardrail-deterministic-spans to main November 20, 2026 08:38
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@daniel-joica
daniel-joica force-pushed the feat/guardrail-stage-grouping branch from 4e9d992 to a003aa0 Compare November 20, 2026 08:47
@daniel-joica daniel-joica changed the title feat(guardrails): group guardrail checks into stage containers [AL-712] feat(guardrails): group guardrail checks into stage containers Nov 20, 2026
…g goes wrong

- A failure to create a stage container now runs the check ungrouped.
- Creating a stage's container closes the other stage's container under
  the same parent.
- Shutting down a trace manager also forgets it, so it isn't reused.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

…ntainers in grouping

- The closer skips a starting container through a context flag, not only
  its attribute, so it never re-enters the lock.
- Grouping is skipped under a non-recording (sampled-out) span.
- A new stage's container is created before the previous one is removed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Shared parent IDs can mix traces, cancellation can close stages before checks finish, and span tracking can leak containers or serialize callbacks across runs.

No open issues 🎉

4 previously missed issues

In code that hasn't changed since last review

Medium severity Keep guardrail stage open until cancelled worker finishes

src/​uipath_langchain/​guardrails/​middlewares/​_base.py:427

[P2] Cancelling this asyncio.to_thread await exits guardrail_stage, but the worker keeps running with its copied container context. The tools span can then close the container as OK before the service evaluation finishes, leaving a child outside its stage's lifetime. Keep the stage open until the worker actually finishes on cancellation; apply the same handling to the POST check at _base.py:469.

Medium severity Clean up containers whose unsampled parent never ends

src/​uipath_langchain/​guardrails/​middlewares/​_spans.py:235

[P2] A valid parent context does not imply the parent span is recording. With a name-based sampler that records this guardrail node and the container but drops its graph parent and subsequent siblings, the container is cached under a parent that will never trigger on_end, and no sampled sibling triggers on_start; the container and its last-check entry then remain until shutdown. Verify that the parent can report its end, or provide a node-end fallback when it cannot.

Medium severity Scope container keys and bookkeeping by trace ID

src/​uipath_langchain/​guardrails/​middlewares/​_spans.py:238

[P2] Two concurrent traces can legally have the same parent span ID, for example with a custom ID generator. This key then reuses one trace's container for the other, and either parent can close it. Use (trace_id, span_id, span_type) keys and trace-scoped last-check bookkeeping in every creation and close path.

Medium severity Index containers by parent to avoid global scans

src/​uipath_langchain/​guardrails/​middlewares/​_spans.py:327

[P2] Most span-start callbacks and every span-end callback scan all active containers under the process-wide lock, including spans from unrelated runs. With many overlapping checks, total tracing work can become quadratic and callbacks serialize across runs. Index containers by parent context so each callback only examines that parent's stages.

4 resolved since last review

Review effort: Balanced

Don't group under an unsampled parent, don't record a check's end on a
container that already closed, and let the closer return without the
lock while no container is open.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Guardrail nodes are hidden when their parent is unsampled or invalid, leaving recorded evaluations and actions referencing a missing trace parent.

1 open issue

Review effort: Balanced

Grouping and node hiding now share one parent check, so a node whose
parent can't hold a container stays exported and its checks keep their
parent.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Human assessment is needed for sampling, provider integration, and span-lifecycle edge cases that can suppress or misparent telemetry and leave containers cached.

No open issues 🎉

1 resolved since last review

Review effort: Balanced

daniel-joica and others added 5 commits November 20, 2026 18:17
…me.py

SpanType and SpanName replace the string literals in the stage table and
the two span type constants, with the same member names as low-code.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ping.py

_spans.py keeps the spans of a single check. _grouping.py holds the stage
containers, with named container keys and stages, one dict of open
containers (each with its last check's end) instead of two, and sections
for setup, running a check, closing and the export filter.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The closer already skips a container while it starts, so the
uipath.guardrail_container attribute was never needed. The deadlock test
now opens a second container while the first is open, the case it guards.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Containers are keyed by parent span id (a parent has at most one open
container), so closing is a lookup instead of a scan. An RLock replaces
the ContextVar guard, and one closer with its trace manager replaces the
weak dict of closers. A new stage still ends the previous stage's
container itself, outside the lock.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
_container_place finds where a container goes and whether it would close;
_open_container reuses or starts it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@sonarqubecloud

Copy link
Copy Markdown

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants