Skip to content

feat(metrics): OTLP push export for gateway metrics #4489

Open

Description

@rhuss

User Story

I run OpenShell gateways and feed their telemetry into an existing observability stack with an OTLP pipeline into a trace and metrics backend. I want the gateway to push its operational metrics over OTLP, alongside the current Prometheus scrape endpoint, so that collectors which do not scrape can still receive gateway metrics through the same OTLP path the gateway already uses for traces.

Problem Statement

The gateway emits its metrics families (openshell_server_grpc_requests_total, openshell_server_grpc_request_duration_seconds, the HTTP equivalents, the readiness gauge, and gateway-interceptor counters) only through the Prometheus /metrics scrape endpoint. Gateway trace export already supports OTLP push via the openshell-otel crate and the [openshell.gateway.otlp] config surface, but metrics have no push path. Operators whose collectors are push-only, or who want metrics to travel the same OTLP route as traces, have no supported option today.

Impact / Why This Matters

In push-only or collector-centric deployments, operators must run a separate Prometheus scrape path solely for OpenShell, duplicating transport and trust configuration they already solved for trace OTLP. This is an adoption barrier for environments standardized on OTLP collectors.

Proposed Design

Offer OTLP metric export as an opt-in, in addition to (not replacing) the existing /metrics scrape endpoint. Scrape remains the default, consistent with the "scrape first, push as operator choice" position from #909. Configuration reuses the existing [openshell.gateway.otlp] surface (endpoint, TLS, service name) rather than inventing a parallel one. Default off.

Suggested UX (if applicable)

[openshell.gateway.otlp]
endpoint = "https://collector:4317"
service_name = "production-gateway"
metrics = true   # opt-in; traces unaffected, scrape endpoint still available

Acceptance Criteria

  • Gateway metrics can be exported via OTLP push to a configured endpoint, opt-in, default off.
  • The Prometheus /metrics scrape endpoint remains available and is the default.
  • OTLP metric export reuses the existing [openshell.gateway.otlp] config and TLS surface.
  • Documented in the gateway configuration reference.

Alternatives Considered

Prometheus remote-write: adds a second export protocol and dependency when the gateway already speaks OTLP for traces. A collector-side scrape-to-OTLP bridge: works, but pushes the integration burden onto every operator rather than offering a first-class path.

Agent Investigation

Verified on main: no OTLP MetricExporter/PeriodicReader exists in crates/; metrics are Prometheus scrape-only. Trace OTLP export is present in openshell-otel (OtlpTraceConfig, build_provider) and wired through Helm into gateway.toml.

Context

Re-homed from the now-closed #2507 (sub-issue 4, "OTLP metrics export"), which was auto-closed as stale after its trace-export critical path landed. Part of the #1055 Enterprise Observability umbrella. Related: #3664 (securing the metrics endpoint), #909 (metrics catalog, closed).

cc @krishicks (original #2507 author / observability WG lead)

Activity

  1. added
    state:acceptedA maintainer decided OpenShell should pursue this issue
    and removed
    state:triage-neededOpened without agent diagnostics and needs triage
    on Oct 14, 2026
  2. moved this from Planning to In progress in OpenShell Roadmapon Oct 15, 2026
  3. github-actions commented on Nov 12, 2026

    @github-actions

    This issue has had no activity for 28 days and is now marked stale. It may be closed in 14 days if there is no further activity. Comment or remove the state:stale label to keep it open.

  4. added
    area:platformSDKs, CLIs, APIs, gateway infrastructure, persistence, configuration, and session management.
    topic:observabilityLogs, metrics, traces, telemetry, dashboards, event correlation, and operational data export.
    on Nov 15, 2026
  5. rhuss commented on Nov 15, 2026

    @rhuss
    ContributorAuthor

    still valid, not stale.

  6. aegeiger commented on Nov 16, 2026

    @aegeiger

    Would be happy to get assigned this issue. @rhuss specs looks solid to me. My intuition is using metrics-util's FanoutBuilder to keep using the existing Prometheus metrics and optionally OpenTelemetry's metrics when needed. I think we can rely on metrics-opentelemtery's PeriodicReader to do periodic pushes.

  7. rhuss commented on Nov 17, 2026

    @rhuss
    ContributorAuthor

    @aegeiger thanks for picking this up, glad the spec reads well. The fanout approach sounds right to me: /metrics keeps working exactly as today, and OTLP becomes an extra sink that is off by default.

    One thing changed since I wrote the suggested UX. The OTLP config now has per-signal tables ([openshell.gateway.otlp.traces], [openshell.gateway.otlp.ocsf], [openshell.gateway.otlp.agent]), so instead of a metrics = true flag, an [openshell.gateway.otlp.metrics] table with enabled = true that falls back to the base endpoint, like ocsf and agent do, would fit better. It would also help to say in the PR how the existing names (openshell_server_grpc_requests_total, openshell_server_grpc_request_duration_seconds) look on the OTLP side, so people can keep their dashboards working.

    I can't assign issues myself. @krishicks, could you or another maintainer assign this to @aegeiger and mark it accepted, so a PR can follow?

  8. krishicks commented on Nov 17, 2026

    @krishicks
    Collaborator

    The issue has been marked accepted since 10/14; anyone should feel free to pick it up.

  9. rhuss commented on Nov 17, 2026

    @rhuss
    ContributorAuthor

    sorry, my bad. @aegeiger can you pick it up ?

  10. TaylorMutch commented on Nov 17, 2026

    @TaylorMutch
    Collaborator

    Assigned @aegeiger , I can also help with reviews when the PR comes in. Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

area:platformSDKs, CLIs, APIs, gateway infrastructure, persistence, configuration, and session management.state:acceptedA maintainer decided OpenShell should pursue this issuetopic:observabilityLogs, metrics, traces, telemetry, dashboards, event correlation, and operational data export.

Type

No type

Projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions