Skip to content

feat(observability): supervisor decision and latency metrics #5364

Open

Description

@krishicks

User Story

I operate OpenShell gateways and sandboxes. I want alerts on supervisor latency, and I want to see which latency bucket is growing and whether request or denial rates rose with it. Today I'd have to reconstruct that from traces or OCSF records.

Problem Statement

The gateway serves Prometheus metrics, but supervisors export none. There is no rate of egress decisions by outcome, and no latency distribution for policy evaluation, request inspection, or upstream connect.

Impact / Why This Matters

Operators cannot alert on a denial storm or on supervisor-added latency. The workaround is keeping every trace or counting OCSF records in the log pipeline, which pushes rate questions onto traces and SIEMs and still gives no latency distribution.

Proposed Design

  • Counters for egress decisions, labeled by outcome, protocol, and policy.
  • Histograms for policy evaluation, supervisor-added time on inspected requests, and upstream connect.
  • Exemplars linking histogram buckets to sampled trace IDs, so an alert leads to a representative trace.
  • Bounded labels: no destination host labels.
  • Open question: how metrics leave a supervisor. Options include OTLP metrics push to the collector, aggregation through the gateway's Prometheus endpoint, or a per-supervisor scrape target.

Acceptance Criteria

  • Operators can query decision rates by outcome and policy, and latency histograms for the stages above.
  • Histogram buckets carry exemplars when a sampled span is active.
  • Label cardinality stays bounded under any workload behavior.
  • The docs list each metric alongside the gateway metrics page.

Alternatives Considered

  • Derive rates from OCSF records in the log pipeline. This works, but it pushes counting onto every SIEM and gives no latency data.
  • Derive rates and latency from unsampled traces. Traces are for latency and error investigation; exporting all of them to count requests is costly and depends on sampling being off.

Agent Investigation

Activity

  1. aegeiger commented on Nov 12, 2026

    @aegeiger

    @krishicks I would love to take this issue, as working on metrics in OpenShell is exactly what I want to contribute to this project. Could you confirm acceptance and assign to me @aegeiger ?

    I'll start by suggesting here metrics and their bounded labels.

  2. added
    area:runtimeSandbox execution, isolation, supervision, policy enforcement, networking, and provider lifecycle.
    topic:observabilityLogs, metrics, traces, telemetry, dashboards, event correlation, and operational data export.
    on Nov 15, 2026
  3. aegeiger commented on Nov 16, 2026

    @aegeiger

    Proposed metrics:

    Metric Type Labels Measures
    openshell_supervisor_egress_decisions_total Counter scope, outcome, inspection-type Final egress admission decisions.
    openshell_supervisor_policy_evaluation_duration_seconds Histogram scope, result Policy evaluation time, including input construction and waiting for the policy-engine lock.
    openshell_supervisor_request_inspection_duration_seconds Histogram scope, inspection-type, result Supervisor processing time for an inspected request or message, including policy evaluation, middleware, and credential preparation.
    openshell_supervisor_upstream_connect_duration_seconds Histogram route, result Establishing a new upstream connection, including address fallback and proxy negotiation where applicable.
    openshell_supervisor_egress_preparation_duration_seconds Histogram scope, inspection-type, result Elapsed time from the first egress signal/request byte until connection availability and required pre-forward checks are complete—or the operation terminates unsuccessfully.
    Label Applies to Values and meaning
    scope Decisions, evaluation, inspection, preparation connection, request, message. Inspection uses only request and message.
    outcome Decisions allow, deny, audit_allow, fail_open. Records the final admission outcome; a subsequent denial during admission overrides an earlier allow.
    inspection-type Decisions, inspection, preparation tcp, http, rest, graphql, json_rpc, mcp, websocket, unknown. Uses the selected inspector where available; never inferred solely from the port.
    result Policy evaluation allow, deny, error. The evaluator’s verdict before audit mode changes enforcement.
    result Inspection completed, error, timeout, cancelled. A normal policy denial is a completed inspection.
    result Upstream connect success, error, timeout, cancelled.
    result Preparation ready_to_forward, deny, error, timeout, cancelled. Readiness includes requests admitted through audit or fail-open behavior.
    route Upstream connect direct, proxy.

    I considered adding a policy label to the decisions counter, because I think it'll be useful for users, but I don't see a way to make this label bounded and I'm not sure we want unbounded labels.

    As for the question of how to get metrics out of the supervisor, I think we should follow the gateway precedent, of exposing a /metrics endpoint and relying on other mechanism such as Prometheus to scrape it. Prometheus is the de-facto standard and is scrape-based by design. If a user wants to use other solutions which are push-based, they're probably already using otel-collector or a similar tool.

    Would love to have a maintainer validate my thinking here.

  4. TaylorMutch commented on Nov 17, 2026

    @TaylorMutch
    Collaborator

    @aegeiger those look good to me, I can help with reviews when you publish the PR. Thanks!

  5. krishicks commented on Nov 17, 2026

    @krishicks
    CollaboratorAuthor

    Drafted by 🤖, signed off by @krishicks

    The proposed set covers decision rates and the latency stages. I'd add two metrics for volume and load:

    Metric Type Labels Measures
    openshell_supervisor_connections_active Gauge inspection-type In-flight proxied connections.
    openshell_supervisor_sandboxes_active Gauge none Sandboxes currently assigned to this supervisor.

    Notes:

    • Sandbox count: a shared supervisor serves many sandboxes, so this gauge shows how load is spread across supervisors. It also shows when sandboxes move after a supervisor fails. The count comes from the supervisor's assignments, not from a per-sandbox label.
    • Delivery: push over OTLP to start, not a /metrics scrape endpoint. Supervisors expose no listener today, and a scrape target would add one per supervisor. The [otlp.agent] relay can carry metrics, and endpoint, protocol, headers, and TLS settings then apply the same way as for traces and logs. Prometheus users can ingest through the collector.
    • Identity: these metrics need to be tied to a particular supervisor and to a gateway, so they can be aggregated per gateway. I don't see an existing name or ID for a supervisor or a gateway in the supervisor's OTel resource. service.name is the same for every supervisor. The design needs to pick the identifiers, for example a generated supervisor instance ID and a gateway ID. Both should be resource attributes, not metric labels, so label cardinality stays bounded.
    • Exemplars: attach them to every duration histogram so a bucket links to a sampled trace.
    • Shared vocabulary: use the same inspection-type and outcome/result values as the span attributes and OCSF fields, so dashboard panels and trace searches filter on the same terms.
    • Preparation histogram: egress_preparation_duration_seconds overlaps with request inspection and upstream connect. Its definition (first byte to ready-to-forward) should stay clearly distinct, or it can drop from the first pass.
  6. krishicks commented on Nov 17, 2026

    @krishicks
    CollaboratorAuthor

    Also, consider @rhuss's notes on #4489 on OTLP config.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

area:runtimeSandbox execution, isolation, supervision, policy enforcement, networking, and provider lifecycle.topic:observabilityLogs, metrics, traces, telemetry, dashboards, event correlation, and operational data export.

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions