Skip to content

Pure-Python opentelemetry-proto (drop the protobuf dependency) [1/4 RFC] - #5503

Draft
ocelotl wants to merge 3 commits into
open-telemetry:mainfrom
ocelotl:pure-python-otlp-1-proto
Draft

Pure-Python opentelemetry-proto (drop the protobuf dependency) [1/4 RFC]#5503
ocelotl wants to merge 3 commits into
open-telemetry:mainfrom
ocelotl:pure-python-otlp-1-proto

Conversation

@ocelotl

@ocelotl ocelotl commented Aug 3, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

What

Replace the google.protobuf-generated message classes in opentelemetry-proto with a hand-written pure-Python protobuf implementation, and drop the protobuf dependency (protobuf>=5.0, <8.0) entirely.

The public opentelemetry.proto.* import namespace is preserved: real code lives under a private opentelemetry._proto package, and the existing opentelemetry.proto.* modules become thin re-export shims (from opentelemetry._proto... import *). Downstream code importing opentelemetry.proto.trace.v1.trace_pb2 etc. keeps working unchanged.

Why

protobuf ships a compiled C extension (upb) and enforces a narrow, moving version range. That causes real friction:

  • Injection / zero-code instrumentation conflicts — when the SDK is injected into an arbitrary target process (K8s operator/injector), the bundled protobuf can collide with the version the target app pins, producing hard-to-diagnose ABI/version errors. A pure-Python encoder is safe to inject.
  • Dependency-resolution friction — the >=5.0,<8.0 range regularly clashes with other libraries (gRPC stacks, ML tooling, cloud SDKs).
  • Wheel/ABI portability — no compiled component means it runs anywhere CPython runs.

The OTLP exporters only ever serialize protobuf; they never parse arbitrary messages. The only decode needed is the export-service response, which is empty. So a small, auditable encode-only implementation is sufficient — this is not a general-purpose protobuf runtime.

Scope of this PR

This is the base of a 4-PR stacked series that makes the whole OTLP export path dependency-light. Merge order:

  1. opentelemetry-proto → pure-Python (this PR)
  2. opentelemetry-exporter-otlp-proto-common → pure-Python backend
  3. opentelemetry-exporter-otlp-proto-http → pure-Python + stdlib urllib (drops requests)
  4. opentelemetry-exporter-otlp-proto-grpc → pure-Python gRPC (drops grpcio)

Because this org's fork model can't host intermediate base branches upstream, each PR targets main and its diff is cumulative (this PR's changes appear in all four). Isolated per-package diffs are viewable via the fork compare links noted in PRs 2–4.

Known gaps / discussion points (why this is a draft)

  • profiles signal not ported. The pure-Python impl currently covers common, resource, trace, metrics, logs and the collector service messages, but not opentelemetry.proto.profiles. That namespace is removed here and would need to be added before this could replace the current package for profiles users.
  • Codegen vs hand-maintenance. The message classes should ideally be generated from the same .proto sources the project vendors, so schema changes stay mechanical. This PR carries a hand-written implementation; agreeing on a generator is part of the discussion.
  • Structure. The _proto (real) + proto (shim) split is one option; collapsing into proto directly is another. Happy to go either way.
  • Conformance. Output should be byte-for-byte identical to the protobuf-generated serialization; a differential test against the current encoders would be worth adding.
  • CI will not be green as-is (tests and docs need adaptation) — this is intentionally an RFC to gauge interest in the approach.

The reference implementation this is derived from has been running in a downstream distribution.

Stack (merge in order): #5503 (proto) → #5504 (common) → #5505 (http) → #5506 (grpc)

This was referenced Aug 3, 2026
Draft
Draft
Draft
@ocelotl
ocelotl marked this pull request as ready for review August 7, 2026 23:07
@ocelotl
ocelotl requested a review from a team as a code owner August 7, 2026 23:07
@ocelotl
ocelotl force-pushed the pure-python-otlp-1-proto branch 2 times, most recently from c1d60de to 9989274 Compare August 12, 2026 14:24
@opentelemetry-pr-dashboard opentelemetry-pr-dashboard Bot mentioned this pull request Aug 14, 2026
Open
@opentelemetry-pr-dashboard

opentelemetry-pr-dashboard Bot commented Aug 14, 2026 •
edited
Loading

Copy link
Copy Markdown

Pull request dashboard status

Waiting on the author · refreshed 2026-08-20 16:50 UTC

Move out of draft to request review.

Status above doesn't look right?
  • Just replied or pushed? Anything around or after the refresh time above may not be picked up yet — give it a few minutes.
  • Anything look wrong? Report it with what you expected; it helps us improve the dashboard.

@ocelotl
Replace opentelemetry-proto with a pure-Python protobuf implementation
1da6e50 
Swap the google.protobuf-generated message classes for hand-written pure-Python
encoders under opentelemetry._proto, keeping the public opentelemetry.proto.*
import namespace via thin re-export shims. Removes the protobuf (and native upb)
dependency entirely; only the serialize path used by the OTLP exporters is
implemented (plus the empty export-service response decode).
@ocelotl
ocelotl force-pushed the pure-python-otlp-1-proto branch from 9989274 to 1da6e50 Compare August 20, 2026 13:21
@ocelotl
Make pure-Python proto transparent to downstream OTLP encoders
37335fa 
Restore the parts of the protobuf message API that the unchanged
otlp-proto-common/http/grpc components rely on, so replacing the
generated classes is transparent to them:

- add the top-level SpanFlags enum and nested Span.SpanKind /
  Status.StatusCode enums, with values also lifted onto the parent
  message class as protobuf does
- add a Message base with value equality (by serialized bytes, matching
  proto3 default omission), repr, SerializePartialToString and FromString
- support the metrics encoder's mutable API: as_int/as_double oneof
  setters on NumberDataPoint and Exemplar (with sfixed64 range checking),
  and auto-vivified gauge/sum/histogram/exponential_histogram/summary
  fields on Metric
- accept a mapping for the Span.status message field
- add the gRPC service Servicer and add_*Servicer_to_server helpers
  alongside the existing client Stub

opentelemetry-proto (668 tests, including the byte-for-byte differential
against protobuf) and otlp-proto-common (20 tests) pass.
@tammy-baylis-swi
tammy-baylis-swi marked this pull request as draft August 20, 2026 16:07
@ocelotl
Add opentelemetry-codegen-pyproto protoc plugin (first milestone)
9eafd30 
A protoc plugin that generates the pure-Python, encode-only protobuf-wire
message classes from the .proto files, so the hand-written classes in
opentelemetry._proto can be generated instead of maintained by hand.

Mirrors the architecture of opentelemetry-codegen-json: a plugin entry
point (protoc-gen-pyproto / --pyproto_out), a generic CodeWriter, a
wire-specific type table, and a descriptor-walking generator. Output goes
to opentelemetry/_proto/<signal>/v1/<name>_pb2.py and imports the existing
_pyprotobuf runtime.

Proven for common/v1 and resource/v1: generated classes serialize
byte-identically to both the hand-written classes and real protobuf across
21 constructed messages. Other signals, the mutable API (Metric
auto-vivification, as_int/as_double setters), packed fields, maps, and
gRPC service stubs remain to be covered.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

Python PR digest
Status: No status

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@ocelotl