Repository navigation
Open
Labels
area:runtimeSandbox execution, isolation, supervision, policy enforcement, networking, and provider lifecycle.Sandbox execution, isolation, supervision, policy enforcement, networking, and provider lifecycle.topic:observabilityLogs, metrics, traces, telemetry, dashboards, event correlation, and operational data export.Logs, metrics, traces, telemetry, dashboards, event correlation, and operational data export.
Description
User Story
I operate OpenShell gateways and sandboxes. I want alerts on supervisor latency, and I want to see which latency bucket is growing and whether request or denial rates rose with it. Today I'd have to reconstruct that from traces or OCSF records.
Problem Statement
The gateway serves Prometheus metrics, but supervisors export none. There is no rate of egress decisions by outcome, and no latency distribution for policy evaluation, request inspection, or upstream connect.
Impact / Why This Matters
Operators cannot alert on a denial storm or on supervisor-added latency. The workaround is keeping every trace or counting OCSF records in the log pipeline, which pushes rate questions onto traces and SIEMs and still gives no latency distribution.
Proposed Design
- Counters for egress decisions, labeled by outcome, protocol, and policy.
- Histograms for policy evaluation, supervisor-added time on inspected requests, and upstream connect.
- Exemplars linking histogram buckets to sampled trace IDs, so an alert leads to a representative trace.
- Bounded labels: no destination host labels.
- Open question: how metrics leave a supervisor. Options include OTLP metrics push to the collector, aggregation through the gateway's Prometheus endpoint, or a per-supervisor scrape target.
Acceptance Criteria
- Operators can query decision rates by outcome and policy, and latency histograms for the stages above.
- Histogram buckets carry exemplars when a sampled span is active.
- Label cardinality stays bounded under any workload behavior.
- The docs list each metric alongside the gateway metrics page.
Alternatives Considered
- Derive rates from OCSF records in the log pipeline. This works, but it pushes counting onto every SIEM and gives no latency data.
- Derive rates and latency from unsampled traces. Traces are for latency and error investigation; exporting all of them to count requests is costly and depends on sampling being off.
Agent Investigation
- Gateway metrics and their histogram conventions are documented in
docs/observability/gateway-metrics.mdx. - No crate under
crates/openshell-supervisor*records metrics. - Related: feat(observability): correlate HTTP receipts, OCSF and request traces #5214 and feat(ocsf): correlate audit events with sampled OpenTelemetry traces #4789 add request and deny spans; these metrics answer rate questions so those spans can be sampled (feat(observability): operator-controlled sampling for supervisor spans #5362).
Metadata
Metadata
Assignees
Labels
area:runtimeSandbox execution, isolation, supervision, policy enforcement, networking, and provider lifecycle.Sandbox execution, isolation, supervision, policy enforcement, networking, and provider lifecycle.topic:observabilityLogs, metrics, traces, telemetry, dashboards, event correlation, and operational data export.Logs, metrics, traces, telemetry, dashboards, event correlation, and operational data export.
Activity
aegeiger commentedon Nov 12, 2026
@krishicks I would love to take this issue, as working on metrics in OpenShell is exactly what I want to contribute to this project. Could you confirm acceptance and assign to me @aegeiger ?
I'll start by suggesting here metrics and their bounded labels.
aegeiger commentedon Nov 16, 2026
Proposed metrics:
I considered adding a policy label to the decisions counter, because I think it'll be useful for users, but I don't see a way to make this label bounded and I'm not sure we want unbounded labels.
As for the question of how to get metrics out of the supervisor, I think we should follow the gateway precedent, of exposing a
/metricsendpoint and relying on other mechanism such as Prometheus to scrape it. Prometheus is the de-facto standard and is scrape-based by design. If a user wants to use other solutions which are push-based, they're probably already using otel-collector or a similar tool.Would love to have a maintainer validate my thinking here.
TaylorMutch commentedon Nov 17, 2026
@aegeiger those look good to me, I can help with reviews when you publish the PR. Thanks!
krishicks commentedon Nov 17, 2026
The proposed set covers decision rates and the latency stages. I'd add two metrics for volume and load:
openshell_supervisor_connections_activeinspection-typeopenshell_supervisor_sandboxes_activeNotes:
/metricsscrape endpoint. Supervisors expose no listener today, and a scrape target would add one per supervisor. The[otlp.agent]relay can carry metrics, and endpoint, protocol, headers, and TLS settings then apply the same way as for traces and logs. Prometheus users can ingest through the collector.service.nameis the same for every supervisor. The design needs to pick the identifiers, for example a generated supervisor instance ID and a gateway ID. Both should be resource attributes, not metric labels, so label cardinality stays bounded.inspection-typeandoutcome/resultvalues as the span attributes and OCSF fields, so dashboard panels and trace searches filter on the same terms.egress_preparation_duration_secondsoverlaps with request inspection and upstream connect. Its definition (first byte to ready-to-forward) should stay clearly distinct, or it can drop from the first pass.krishicks commentedon Nov 17, 2026
Also, consider @rhuss's notes on #4489 on OTLP config.