Skip to content

Add optional OpenTelemetry tracing and request metrics - #44

Open
mikelangmayr wants to merge 3 commits into
mainfrom
opentelemetry
Open

mikelangmayr wants to merge 3 commits into
mainfrom
opentelemetry

Conversation

@mikelangmayr

@mikelangmayr mikelangmayr commented Oct 28, 2026 •

Copy link
Copy Markdown
Collaborator
  • New otel extra; libby imports OpenTelemetry only when it is installed and an OTLP endpoint is set
  • Client and server spans per request, linked across peers through the envelope's W3C traceparent; producer and consumer spans for publish
  • Request duration histograms per key with error.type on failures
  • LibbyDaemon turns export on at start and flushes at stop; the SDK lives for the process
  • The CLI exports one trace per command, no metrics
  • Log lines inside a span get trace_id and span_id appended
  • Telemetry docs page

Comment thread libby/telemetry.py

tracer_provider = TracerProvider(resource=resource)
tracer_provider.add_span_processor(BatchSpanProcessor(OTLPSpanExporter(**timeout)))
trace.set_tracer_provider(tracer_provider)

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think OpenTelemetry keeps one global tracer provider per process, and trace.set_tracer_provider() is done the first time when starting the process

We'll probably need to make sure the tracer provider is reinstantiated if the process has to be restarted or stop/start

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Confirmed: a stop/start in one process dropped every span after the restart. stop now only flushes; the SDK is installed once and lives until process exit. test_restarted_daemon_keeps_exporting covers it.

Comment thread libby/telemetry.py
instruments = _instruments
if instruments is None:
return _UNTRACED
return Call.start(instruments, name, instruments.trace.SpanKind.INTERNAL, {})

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Call.start and _Instruments.incoming use the global propagator (opentelemetry.propagate), so they assume it reads and writes traceparent / tracestate. That's only true if W3C is among the configured propagators. If a user sets OpenTelemetry SDK to use the B3 Single Header format inject() writes a b3 key instead, the envelope goes out with traceparent: null I believe

In bamboo the envelope fields are W3C Trace Context by spec, so the format shouldn't depend on deployment config. What about using the TraceContextTextMapPropagator() directly for the envelope?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed, traceparent went out as null with OTEL_PROPAGATORS=baggage. The envelope now uses TraceContextTextMapPropagator directly; test_envelope_context_ignores_the_global_propagator covers it.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants