Skip to content

feature: export session events for post-processing — content-free per-event projection, NDJSON sink, archive-replayable #1810

Open
0 of 2 issues completed

Description

@mrsabath

Summary

Cortex already holds the richest record of what a coding agent did — 30 days of events on
disk, with timings, token counts, hosts and failures that exist nowhere else. Today the only
ways to read it are the session API (one session at a time, for a TUI) and a bespoke Go
program against archive.Page/Earlier. There is no supported way to get a month of events
out of the archive and into a data store for analysis.

This proposes a content-free per-event export: a projection of SessionEvent to a flat
record, an NDJSON sink, and the ability to replay it out of the archive so the existing 30
days can be backfilled rather than only tapped live.

Why a projection is the feature, not an optimisation

core/sessionapi/summary.go already solved a near-identical problem and measured it:

a single inference request carries the whole conversation... 99.5% of that is the
inference field alone. Measured on real traffic, dropping the payloads is a 203x
reduction.

The same ratio governs an export. An analysis of a local 30-day archive (66 sessions,
29,606 events) kept ~30 scalars per event and produced a 29,606-row NDJSON file; the same
corpus at full fidelity is multi-gigabyte (BytesUp alone totals 6.08 GB). Exporting raw
events would be unshippable and would carry every prompt off-box. The projection is what
makes this safe and affordable at once.

summarizeEvent also documents the trap worth copying: its drop rule fails silently —
"a search for completion text simply stops matching." An export schema needs the same
discipline: an explicit rule for what may be dropped, and a test that pins the contract so
it cannot drift.


Tier 1 — the core record, always exported, content-free by construction

Roughly 30 scalars per event. This is the set that an offline analysis of this archive
actually used; each group is here because something could not be answered without it.

Group Fields
Identity sessionId, sessionIncarnation, seq, requestId, at
Routing host, requestedHost, httpMethod, httpPath
Outcome phase, statusCode, errorKind, errorCode
Timing durationMs (and ttfbMs once #951 lands)
Size bytesUp, bytesDown
Attribution client, agentRole
Inference model, requestedModel, stream, streamedResponse, finishReason
Tokens promptTokens, completionTokens, cacheReadTokens, cacheWriteTokens
Counts messageCount, toolCount, toolCallCount, newToolResults, toolResultErrors
Tunnel tunnel, tunnelReason

Notes on the non-obvious ones:

No message content, no completion, no tool arguments, no tool results, no MCP params.
That is what makes a Tier 1 record shippable to an external collector without a per-
destination privacy review.

Tier 2 — opt-in structured detail

  • toolCallNames: []string — the tool names a response asked for. This is what enables
    per-tool latency (below). Names are low-risk; arguments are not — a Bash command
    line is effectively a secret and belongs in Tier 3.
  • invocations: [{plugin, action, reason}] — per-plugin outcomes. summarizeEvent keeps
    these out of the drop set for good reason: the ACTION column and the plugin:<name>
    filter both read them.
  • identity: {subject, clientId, agentId} — the authenticated identity, as distinct
    from client.

Tier 3 — content, off by default

Messages, completions, tool results, MCP params. Proposed position: an export path should
refuse this by default rather than offer it as a flag someone flips under time pressure.
The replay use case for full fidelity is already specified in #1494; this issue is not that.


What this unlocks: tool time vs human time, from Cortex alone

The motivating question was "where did the agent's wall-clock go", with no agent-harness
access. Tier 1 answers it. Classifying each response -> next request gap by whether
proxied traffic landed inside it:

n p50 total share of gap time
Proxied tool (measured from durationMs) 1,910 1,045 ms 11.6 h 18.8%
Local tool (inferred — no proxied traffic) 5,653 158 ms 5.3 h 8.6%
User wait (0 tool calls) 1,442 32,358 ms 44.8 h 72.6%

With toolCallNames (Tier 2), single-tool-call turns yield per-tool latency directly:
Bash p50 201 ms (n=3,754), Edit 89 ms (n=814), Read 82 ms (n=338), and AskUserQuestion
62,602 ms (n=68) — the last being useful validation, since the classifier never saw the
tool's name yet separated the one human-blocking tool from the machine ones by three orders
of magnitude.

Honest limits: the user-wait total is very sensitive to the gap ceiling (24.8 h at 300s,
44.8 h at 1800s, 56.8 h at 3600s, 539 h uncapped), so any consumer publishing it must
publish the cap too.


Three schema decisions worth settling in review

  1. Export durationMs; never expect a consumer to subtract timestamps. The fast-tool
    figures above sit near wall-clock resolution, and Seq's own doc notes same-instant
    events are common on a fast local proxy. A consumer pairing by requestId and
    subtracting at gets noise instead of latency.

  2. sessionIncarnation is the one genuinely new field here. A session id is reused: the
    id default recurs across the entire 2026-10-15 -> 2026-11-06 range of this archive, in
    one directory, spanning many unrelated working periods. Without an incarnation key, every
    per-session aggregate downstream silently averages unrelated work. The archive can derive
    it from segment boundaries (one of these sessions has 34 segments across three weeks).
    Raised independently on feature: agentop observe — group sessions and cost by working directory #1739, where it also affects per-directory cost totals.

  3. httpPath can carry secrets and needs a decision before it ships off-box. Per its
    own documentation: "A secret embedded in a path SEGMENT does survive, because nothing can
    tell it from a resource id — a bot token or a webhook path lands here verbatim." Query
    strings are already stripped; path segments are not. Either redact, or make this field
    opt-out, but do not ship it off-box silently.

Same caveat applies to client, which is derived from a request header: a cost-attribution
and display key, never an authorization subject. Worth restating in the export docs,
since an external collector is exactly where someone will be tempted to treat it as identity.


Format and transport

  • NDJSON, to a file or an HTTP endpoint. This is what post-processing wants: the
    analysis behind this issue was jq and Python over NDJSON, and a full 30-day pass took
    ~10s single-threaded.
  • Archive-replayable, not only a live tap. The archive already holds 30 days;
    backfill is most of the value. Note for whoever implements it: ReplaySources/
    ReplaySince excludes tunnel rows by design (correct for cost replay — no tokens),
    which is 10,493 of 29,606 events here and includes every non-LLM latency figure.
    An export walk wants Page/Earlier, which do not filter.
  • A documented drop rule plus a test pinning the Tier 1 contract, in the spirit of
    TestSummarizeEvent_KeepsEverythingTheFilterSearches.

OTLP deliberately left as a follow-up. The metrics subset maps cleanly onto the GenAI
semantic conventions (gen_ai.* already standardises model, token counts and finish reason)
and belongs with #505, which is about emitting spans for Cortex's own pipeline. Bundling
both into one issue makes it unreviewable.

Relationship to existing issues

Scope of the measurements above

One developer's laptop archive, heavily Claude Code (18,785 of 29,606 events carry
claude-code/2.1.270). The field list generalises — it follows the event schema, not
this traffic. The ratios do not: #1494 measured 59% of replies batching tool calls under
explicit prompting against 7.3% here unprompted, so those numbers move with working style
and prompting.

Assisted-By: Claude (Anthropic AI) noreply@anthropic.com

Activity

  1. mrsabath commented on Nov 11, 2026

    @mrsabath
    MemberAuthor

    Scope decision on the open question in the "Relationship to existing issues" section:

    #898 consumes this layer. The boundary:

    So this issue does not need to grow an aggregate mode, a remote endpoint, retry or
    batching. Those stay on #898, which now reads this export rather than collecting
    independently.

    The practical consequence for Tier 1: it must stay sufficient for #898's aggregates, since
    they will be derived from it rather than from the events. The fields that issue needs —
    model, host, the four token counts, durationMs, client, invocations for per-plugin
    attribution — are already in Tier 1 and Tier 2, so no additions are implied. Worth keeping
    in mind if anything is dropped from the Tier 1 set during review: a field removed here is a
    field #898 cannot aggregate.

    Noted on #898 as well.

  2. mrsabath commented on Nov 13, 2026

    @mrsabath
    MemberAuthor

    Spec drafted — #1854

    This issue, #898, #505, #1740 and #1466 all touch the export, and #1812 shipped part of it,
    but nothing said in one place what a record is or which of the fields above actually landed.
    Drafted that as a spec: #1854, at
    docs/superpowers/specs/2026-11-13-session-event-export-design.md.

    53 numbered requirements, a normative field reference for the 36 fields in core/sessionexport
    today, and an explicit list of what this issue proposed that did not ship.

    What shipped, against this issue's Tier 1

    Group Status
    Identity, routing, outcome, timing, size, attribution, inference, counts, tunnel shipped
    Tokens shipped, but as the modern split — inputTokens/cacheRead/cacheWrite/outputTokens/reasoningTokens + presentKinds, not the four this issue listed
    httpPath not shipped — refused pending the decision this issue raised
    sessionIncarnation not shipped — needs #1739's key
    newToolResults, toolResultErrors not shipped — blocked on #1740

    Two notes on the token change, since this issue specified the other four. promptTokens turned
    out to be the legacy aggregate (Fill sets it to input + cacheRead + cacheWrite), so
    exporting it beside cacheReadTokens double-counted every cached token — 196,096 of them on a
    single response of this archive. And reasoningTokens is a subset of outputTokens, not a
    sibling. The spec states both as rules (R18, R20) so the next consumer does not have to
    rediscover them.

    One field this issue predates

    SessionEvent.Project landed in #1815, after this issue was written — the git-repo-root label
    of the agent process. It is the natural key for per-project cost and latency and is not in
    the record. It is also the same question httpPath raises: a repo path under $HOME names
    directories a user may not want off-box. Flagged Open rather than quietly added.

    The five decisions the spec asks for, not answers

    1. httpPath — redact, opt-out, or never? (this issue's decision 3, still open)
    2. project — is a repo path content?
    3. sessionIncarnation — derive in the archive, or in the export walk? (this issue's
      decision 2)
    4. Does feature: Report statistics to a central collector #898 consume a sink, or share only the Record type? Worth settling before two
      transports exist.
    5. Live tap in version one, or backfill only with the archive as the buffer?

    This issue's decision 1 — export durationMs, never expect a consumer to subtract timestamps
    — is settled and shipped, as R22/R23 plus TestRecord_DurationsAreMilliseconds.

    Review welcome on #1854, particularly §8.1 (#898's relationship) and §7.2 (Tier 2 breaks the
    flat-scalars rule and needs its own schema decision).

  3. 10 remaining items

  4. mrsabath commented on Nov 19, 2026

    @mrsabath
    MemberAuthor

    The Tier 1 field table above is now behind what shipped in #1812 / #1830 / #1831. Noting the differences so nobody builds against a field that does not exist, or re-litigates one that was dropped on purpose.

    Dropped deliberately, and they should come off the table:

    • promptTokens / completionTokens. InferenceExtension.PromptTokens is already Input + CacheRead + CacheWrite. Exporting it beside cacheReadTokens ships a total next to its own components, and a consumer pricing the input rate on one and the cache rate on the other counts every cached token twice — 196k of them on a single response on the development archive. The record carries the four disjoint kinds only, which is what this issue's own "all four token counts separately" note asks for.
    • httpPath. Decision three above asked for this to be settled before it shipped off-box. It was settled by leaving it out: a path segment can hold a bot token or a webhook secret, and nothing can tell that from a resource id. Invocation.Path is a back-filled copy of the same string, so it is out too. httpMethod alone carries the request's shape.
    • newToolResults / toolResultErrors. Still feature: three archive fields for offline session analysis — distinct tool-result counts, interleaved-completion marker, stream mismatch #1740, still unbuilt, so the record carries no tool-result figure at all rather than one that overcounts by ~272x. toolCallCount ships and is unaffected.

    Shipped, not on the table:

    • ttfbMs — feature: latency metrics — time to first token, total response time, session percentiles #951 landed, so this is no longer conditional.
    • servedModel — what the response named, when it differs from what was asked for. requestedModel alone does not show it.
    • reasoningTokens — a subset of outputTokens, never a fifth kind. Adding the two double-counts.
    • presentKinds — the one field that escapes the record's zero-means-absent rule. A set bit with a zero count is "the provider reported zero"; an unset bit is "the provider does not expose this kind". Without it those two are the same absent key.
    • direction, phase — phase was in the Outcome group; direction was not there at all.
    • schema — the record format version, on every record.

    Also unshipped: tool-call names (Tier 2) and sessionIncarnation (Tier 1), the latter now tracked as #1997.

    Net: the shipped record is 36 fields. Decisions two and three of "Three schema decisions worth settling in review" are both now settled — two as #1997, three by omission.

    Assisted-By: Claude (Anthropic AI) noreply@anthropic.com

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions