[Sayan-]

Summary

Serve browser telemetry storage through the metro-api relay instead of baking S2 credentials into VMs. This PR adds the in-VM sink.

• When BTEL_STORAGE_PATH is set, stored telemetry is posted to the metro-api relay instead of appended to S2. Direct S2 (S2_BASIN plus S2_ACCESS_TOKEN) is unchanged and still used when the path is unset. The relay wins when both are set.

• Each record body is json.Marshal(envelope), exactly what the direct sink appended, so S2 contents and the read path are unchanged. No timestamp is sent, so S2 stamps arrival time as it does for the direct sink.

• The sink follows the OTLP sink's semantics:

  • A bounded queue that drops the oldest record when full, so Append never blocks the ring reader.
  • Batches flush at 200 records, 768 KiB metered, or a 100ms linger. One request is in flight at a time, so order is preserved.
  • 429, 502, 503, 504 and temporary transport errors are retried with the OTLP exporter's backoff, honouring Retry-After, for up to 30s per batch.
  • A batch that gets no response because the relay closed a kept-alive connection under it is sent again on a new connection. This happens when a metro-api deploy replaces the process. The relay does not deduplicate, so a batch it appended just before dying without answering is stored twice, the same duplicate a retried 503 can produce.
  • Every other outcome drops the batch and counts it, 403 and 413 included.
  • Close flushes what it can within the shutdown budget.

• The bearer token (instance JWT) and relay host are resolved per batch, so a fork posts with its own identity to the metro-api it was handed. Host order:

  1. the identity's metro-api URL;
  2. BTEL_OTLP_ENDPOINT, with BTEL_OTLP_INSECURE picking the scheme.

  The token goes only on requests to that URL, and redirects are refused.

• The storage controller now takes a storage constructor. The relay controller shares the S2 controller's start/stop/once-only lifecycle and its fork gate: nothing opens until S2_STREAM (or the applied fork identity's stream) resolves. The telemetry handler is unchanged.

• Config: BTEL_STORAGE_PATH, BTEL_STORAGE_MAX_RECORDS (200), BTEL_STORAGE_MAX_BYTES (786432), BTEL_STORAGE_LINGER (100ms), BTEL_STORAGE_MAX_QUEUE (8192). When the path is set, values are validated against S2's batch limits and a relay host is required. Documented in server/README.md.

• Metrics: kernel_btel_relay_exported_total, _dropped_total{reason}, _retries_total, _failed_total{status}, _queue_depth.

Rollout

Release this image before provisioning sets BTEL_STORAGE_PATH. Images without this sink provisioned without S2 credentials store nothing.

Validation

• lib/events relay sink tests against httptest:

  • Batches arrive in order, and bodies match json.Marshal(envelope) byte for byte, including HTML and line-separator escapes. A record on the wire is its body alone, with no timestamp.
  • Flush on record count and on metered bytes, with every batch under the budget.
  • Each of 429/502/503/504 is retried. Retry-After is honoured.
  • 400/401/403/413/500, a refused connection and a redirect are dropped and counted without retry. The redirect target is never contacted.
  • A relay that reads a batch on a reused connection and closes it without answering gets the same batch again on a new connection, and nothing is dropped. Without the change this test fails with the batch dropped on EOF.
  • Token and host are re-read after a simulated fork identity change.
  • Drop-oldest accounting keeps the newest records.
  • Close past its deadline counts a shutdown drop.
  • Relay URL construction.

• Controller contract tests run against both the S2 and relay controllers: lifecycle, the fork gate, and rollback when start is cancelled.

• cmd/api/api/telemetry_test.go: the storage toggle runs against both real controllers. Storage opens once, refuses to turn off (409), and stays closed while no stream resolves.

• Config validation cases, the metrics collector, and the relay base URL fallback.

• e2e, run locally against a freshly built headless image:

  • TestRelayStorage: relay plus direct S2 credentials; records reach a mock relay with the instance JWT and valid envelopes.
  • TestRelayStorageForkIdentity: a fork boots with no JWT and a dead endpoint. After the identity handoff, storage reaches the handed metro-api with the fresh JWT.

  Both pass, as does the existing TestOTLPExport. Locally, testcontainers needed TESTCONTAINERS_RYUK_DISABLED=true and TESTCONTAINERS_HOST_OVERRIDE=127.0.0.1 because of this machine's DNS.

• go test ./... outside e2e passes. One devtoolsproxy test flaked once on Chromium temp-dir cleanup and passed on rerun.

• Not tested: a real metro-api relay and S2.

🤖 Generated with Claude Code

---

Note

Medium Risk
Changes the telemetry persistence and auth path (JWT bearer to metro-api) with batching/drop behavior on failure, though direct S2 remains when the relay path is unset and coverage is extensive.

Overview
Adds an optional metro-api storage relay path so VMs can persist browser telemetry without embedding S2 credentials. When BTEL_STORAGE_PATH is set, the API swaps in a relay storage controller (it wins over direct S2_BASIN/S2_ACCESS_TOKEN) that POSTs json.Marshal(envelope) batches to the relay using the instance JWT, with host resolution preferring the fork/metro identity URL and falling back to BTEL_OTLP_ENDPOINT.

The new relayStorage sink mirrors OTLP semantics: bounded queue with drop-oldest, batching by record/byte count and linger, single-flight ordered sends, retry policy for throttling/outages (honoring Retry-After), no redirect following, and per-batch JWT/base URL resolution for fork handoffs. S2StorageController is refactored behind an injectable open hook so direct S2 and relay share the same start-once / fork stream gate behavior.

Config validation, README env docs, kernel_btel_relay_* metrics, and unit/controller/e2e tests cover the new path (including fork identity and relay-over-S2 precedence).

^{Reviewed by Cursor Bugbot for commit 5aa0895. Bugbot is set up for automated code reviews on this repo. Configure here.}

[Sayan-]

Picked rgarcia to review: keeping one reviewer for the telemetry storage changes.