feat(hub): add optional OpenTelemetry server tracing by Copilot · Pull Request #36 · github/replica

Copilot · 2026-10-19T19:02:56Z

Why

Companies running their own Hub need to connect their application performance monitoring (APM) provider without rebuilding the published container image. A shared OpenTelemetry integration lets operators choose a provider instead of requiring Hub to bundle each vendor's SDK.

Closes #34.

What changes

Hub can export server-side traces through the OpenTelemetry Protocol (OTLP) over HTTP/protobuf. Tracing is off by default. Operators enable it with environment variables and restart every Hub web and job container. Traces cover Rails requests, background jobs, SQLite queries, and outbound Net::HTTP calls.

By default, Hub exports operational metadata without SQL text, job arguments, or exception messages and stack traces. REPLICA_OTEL_INCLUDE_SENSITIVE_DATA=true enables those diagnostics together. Credential-keyed job fields remain filtered, SQL bind values remain excluded, and request headers and bodies are never attached.

Public requests start fresh traces unless the operator enables REPLICA_OTEL_TRUST_INCOMING_CONTEXT behind an ingress that sanitizes trace headers. Hub discards incoming vendor trace state and baggage while preserving its own job trace continuity. Operators can exclude exact outbound hosts from both HTTP tracing and trace-header propagation.

Export runs in a bounded background queue. Collector or provider outages can drop traces but do not block successful requests. The deployment guide includes a copy-paste local Aspire dashboard setup and explains endpoint authentication, sampling, certificates, restart behavior, and the data impact of each Replica tracing setting.

Operators can send traces directly to a compatible provider or through a company-managed OpenTelemetry Collector. This first version does not add browser monitoring, metric or log export, an admin configuration page, or vendor-native error reporting. Live Sentry and Datadog ingestion has not been verified.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Include filtered job arguments, SQL text, and exception diagnostics, with an operator-controlled identity opt-in. Align exported metadata with bundled instrumentation and isolate public incoming trace context by default. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Document identity opt-in, trusted ingress, outbound exclusions, and the limits of key-based filtering. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Centralize the sensitive-data policy in the exporter while avoiding collection of SQL, job arguments, and exception diagnostics unless operators explicitly opt in. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Add a local Aspire dashboard workflow and show exactly which trace fields change when sensitive diagnostics are enabled. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot

🟡 Changes recommended

Unresolved security, operational, and regression-coverage issues need correction before approval.

7 open issues

_{Review effort: Max}

Sanitize endpoint parse failures, normalize exact outbound host exclusions, suppress unparented database spans, and strengthen propagation and privacy regression coverage. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot

🔵 Needs a closer look

Privacy-sensitive export and worker lifecycle behavior need human verification alongside the unresolved reliability issues.

3 open issues

7 resolved since last review

_{Review effort: Max}

Replace timed-out OTLP connections, reject empty host exclusions, and restore request tracing context inside parallel GitHub workers. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot

🔵 Needs a closer look

Unresolved exporter lifecycle issues and privacy-sensitive tracing boundaries require human review backed by runtime validation.

4 open issues

3 resolved since last review

_{Review effort: Max}

Recreate exporters after forks, validate collector and outbound host settings, and preserve OpenTelemetry service-name precedence. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot

🔵 Needs a closer look

The telemetry privacy boundary and custom exporter’s fork and deadline lifecycle require human verification.

1 open issue

Reject bracketed and prefixed IPv6 addresses · New

4 resolved since last review

_{Review effort: Max}

Require bare IPv6 addresses so outbound host exclusions match the address Net::HTTP uses. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot

🔵 Needs a closer look

Disabled tracing still breaks eager-loaded startup, and the custom exporter lifecycle and privacy guarantees need human validation.

1 open issue

Exclude tracing support from autoloading when tracing is disabled · New

2 resolved since last review

The mutex wait is outside the caller's timeout. If a background export has drained the queue but is…
The bundled OTLP exporter loads the client key with OpenSSL::PKey::RSA.new. An EC key therefore…

_{Review effort: Max}

Exclude manually required tracing support files from Rails eager loading and cover production-style disabled startup. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot

🔵 Needs a closer look

The unresolved credential-logging risk and privacy-sensitive, fork-aware exporter lifecycle require final maintainer review.

1 open issue

Malformed OTLP authentication headers can expose credentials in startup logs. The pinned exporter… · New

1 resolved since last review

Exclude tracing support from autoloading when tracing is disabled

_{Review effort: Max}

Validate selected OTLP headers before exporter construction so malformed authentication values cannot leak through startup exception causes. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot

🔵 Needs a closer look

Privacy-sensitive export and fork/timeout handling need final human review with the pinned Rails and OpenTelemetry runtime.

1 open issue

A trace-specific endpoint such as OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=http://otel-collector:4318… · New

1 resolved since last review

Malformed OTLP authentication headers can expose credentials in startup logs. The pinned exporter…

_{Review effort: Max}

Reject pathless trace-specific OTLP endpoints while preserving standard path expansion for common OTLP endpoints. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot

🔵 Needs a closer look

Privacy-sensitive export and unresolved concurrency issues require human review and dependency-backed validation.

2 open issues

The caller's flush timeout does not cover the batch processor's export lock. In SDK 1.13.1,… · New
Concurrent first export and flush calls after a fork can both pass the PID check while the exporter… · New

1 resolved since last review

A trace-specific endpoint such as OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=http://otel-collector:4318…

_{Review effort: Max}

Enforce force-flush deadlines at the batch processor lock and reset child exporter state once through the processor's fork lifecycle. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot

🔵 Needs a closer look

Custom SDK lifecycle overrides, fork-sensitive locking, and telemetry privacy controls warrant human review with Rails runtime validation, which was unavailable here.

No open issues 🎉

2 resolved since last review

Concurrent first export and flush calls after a fork can both pass the PID check while the exporter…
The caller's flush timeout does not cover the batch processor's export lock. In SDK 1.13.1,…

_{Review effort: Max}

nshki

Thanks for taking this over the finish line, @alliscode!

Keep disabled tracing lazy through an autoloaded enabled integration whose dependencies are declared at the top of the file. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot

🔵 Needs a closer look

Custom SDK lifecycle overrides and sensitive-data boundaries need human review and dependency-backed runtime validation.

2 open issues

Optional: make missing-endpoint errors actionable. With OTEL_TRACES_EXPORTER=otlp and neither… · New
Include disabling or redirecting tracing in the local-viewer cleanup. Stopping the dashboard leaves… · New

_{Review effort: Max}

Name missing and malformed OTLP endpoint settings without exposing values, and document how to stop local trace inspection cleanly. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot

🔵 Needs a closer look

The sensitive-data export boundary and custom fork and deadline handling require final human review and verification in Hub’s supported runtime.

1 open issue

For the Compose bundle, name the command that applies changes to .env. docker restart and… · New

2 resolved since last review

Optional: make missing-endpoint errors actionable. With OTEL_TRACES_EXPORTER=otlp and neither…
Include disabling or redirecting tracing in the local-viewer cleanup. Stopping the dashboard leaves…

_{Review effort: Max}

Name the Compose command that recreates Hub after enabling, redirecting, or disabling OpenTelemetry export. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

feat(hub): add optional OpenTelemetry server tracing

96d978f

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot AI max review requested due to automatic review settings October 19, 2026 19:02

Copilot AI requested review from a team and nshki as code owners October 19, 2026 19:02

This comment was marked as duplicate.

Sign in to view

alliscode reviewed Oct 19, 2026

View reviewed changes

Comment thread hub/docs/deployment.md Outdated

alliscode reviewed Oct 19, 2026

View reviewed changes

Comment thread hub/lib/hub_tracing.rb Outdated

Copilot AI and others added 5 commits October 19, 2026 23:40

docs(hub): explain tracing data and trust controls

286e4d9

Document identity opt-in, trusted ingress, outbound exclusions, and the limits of key-based filtering. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Merge remote-tracking branch 'origin/main' into HEAD

59ab2a6

fix(hub): gate sensitive tracing diagnostics

624f67f

Centralize the sensitive-data policy in the exporter while avoiding collection of SQL, job arguments, and exception diagnostics unless operators explicitly opt in. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

docs(hub): clarify tracing setup and data impact

1c1ad4a

Add a local Aspire dashboard workflow and show exactly which trace fields change when sensitive diagnostics are enabled. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

alliscode requested a max review from Copilot October 21, 2026 20:53

Copilot started reviewing on behalf of alliscode October 21, 2026 20:54

View session

Copilot AI reviewed Oct 21, 2026

View reviewed changes

fix(hub): tighten tracing boundaries

932b9d1

Sanitize endpoint parse failures, normalize exact outbound host exclusions, suppress unparented database spans, and strengthen propagation and privacy regression coverage. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

alliscode requested a max review from Copilot October 21, 2026 22:37

Copilot started reviewing on behalf of alliscode October 21, 2026 22:37

View session

Copilot AI reviewed Oct 21, 2026

View reviewed changes

Comment thread hub/lib/hub_tracing/exporter.rb

Comment thread hub/lib/hub_tracing.rb Outdated

Comment thread hub/lib/hub_tracing.rb Outdated

fix(hub): recover tracing context and exports

ebbcdce

Replace timed-out OTLP connections, reject empty host exclusions, and restore request tracing context inside parallel GitHub workers. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

alliscode requested a max review from Copilot October 22, 2026 00:48

Copilot started reviewing on behalf of alliscode October 22, 2026 00:49

View session