From 8648e03b6836c198a24e1e906d50d1119d0f42a9 Mon Sep 17 00:00:00 2001 From: Mariusz Sabath Date: Thu, 19 Nov 2026 14:45:47 -0500 Subject: [PATCH 01/12] Docs: Say the timeline already answers per-tool latency The export guide listed per-tool latency under what no export answers, two lines above the sentence that sends a reader to the timeline for names. A reader took the first as the capability being absent, when what is absent is only the names on a record: GET /v1/sessions/{id}/timeline pairs each call with its result by tool-use id and returns named spans with a start and an end, so the question is answered today, without a prompt. Keep both reasons names stay off a record, and state the limit as the shape it is -- a name list on a response record carries no tool-use id, so it could not pair. Add the two rules that cost accuracy when totalling those spans. A batch of parallel calls shares one window rather than carrying a duration each, so charging every member the whole window overcounts; measured at 15.7% of spans (42 of 267) on one 30-day archive. A lost span sets end equal to start and so passes an end != null filter as a 0 ms call. Assisted-By: Claude (Anthropic AI) Signed-off-by: Mariusz Sabath --- docs/session-export.md | 28 +++++++++++++++++++++------- 1 file changed, 21 insertions(+), 7 deletions(-) diff --git a/docs/session-export.md b/docs/session-export.md index 2093db1c8..ddf5a42da 100644 --- a/docs/session-export.md +++ b/docs/session-export.md @@ -140,13 +140,27 @@ names out for now: - A record is a flat list of scalars. A name list would be the first array, which changes the shape every consumer parses. -- Names alone would not give per-tool latency. A tool result arrives in a later request, - and the key that pairs it with its call is the tool-use id, which no record carries. So a - name list says only which tools a response asked for. - -Use `GET /v1/sessions/{id}/timeline` for names today. It pairs each call with its result by -tool-use id, and it returns named spans. -[#1906](https://github.com/rossoctl/cortex/issues/1906) tracks adding names here. +- A name list on a response record would not carry per-tool latency, because a tool result + arrives in a later request and the key that pairs it with its call is the tool-use id. A + record carries no tool-use id, so a name list would say only which tools a response asked + for. **Per-tool latency is not blocked on the names**: the timeline already does the + pairing, below. + +Use `GET /v1/sessions/{id}/timeline` for per-tool timing today. It pairs each call with its +result by tool-use id and returns named spans with a start and an end, so it answers "which +tools ran, and for how long" without a prompt. +[#1906](https://github.com/rossoctl/cortex/issues/1906) tracks bringing names, and the +timing that pairing yields, into a record. + +Two rules before you total those spans, both of which cost real accuracy: + +- **A batch of parallel calls shares one window.** Calls the model issues together carry the + same start and the same end, not a duration each. Measured at 15.7% of spans (42 of 267) + on one 30-day archive. Charging every member of a batch the whole window's time + overcounts; read a shared window once per batch. +- **Exclude `lost` spans.** A span that never got an answer but provably stopped sets `end` + equal to `start`, so it passes an `end != null` filter as a 0 ms call and drags a p50 + toward zero. A span with no `end` is still running. An export is **content-free**. It is **not anonymous**. It keeps hosts, models, session ids and timestamps. Together these show who called what, and when. Read an export before From 4ff75736252c218d439eae200b1c683993eb240a Mon Sep 17 00:00:00 2001 From: Mariusz Sabath Date: Thu, 19 Nov 2026 15:15:55 -0500 Subject: [PATCH 02/12] Docs: Spec the toolCall export record #1906 asks for tool names on the record. Measuring it first changed the shape: a name list on a response record cannot carry per-tool timing, because the pairing key is the tool-use id and no record holds one, and 15.7% of tool spans (42 of 267, on a 30-day archive) share one window with the other calls of their batch rather than carrying a duration each. A list has nowhere to say that, so a consumer dividing a response's time by its toolCallCount is wrong for one call in six, in the direction that inflates. So the record is a sibling on the same NDJSON stream, discriminated by kind, carrying a paired duration with the batch marked explicitly. The tool-use id stays inside the proxy: the export ships the measurement the key yields, never the key. Record what the measurement found, since the design traces to it: inference is 80.3% of measured work (4.1:1 against tools), the heavy tier is 1.9% of calls and 55.4% of work, and Bash is 78% of calls but 97.4% of tool time. The two views reconcile to 2.2%, so a record derived archive-side will not contradict the timeline. Four items are left open for review, not guessed: whether an MCP tool name is gated or hashed, the opt-in switch, which of three places the pairing runs, and the per-session bound it needs. Assisted-By: Claude (Anthropic AI) Signed-off-by: Mariusz Sabath --- ...26-11-19-tool-call-export-record-design.md | 378 ++++++++++++++++++ 1 file changed, 378 insertions(+) create mode 100644 docs/superpowers/specs/2026-11-19-tool-call-export-record-design.md diff --git a/docs/superpowers/specs/2026-11-19-tool-call-export-record-design.md b/docs/superpowers/specs/2026-11-19-tool-call-export-record-design.md new file mode 100644 index 000000000..898f4db0d --- /dev/null +++ b/docs/superpowers/specs/2026-11-19-tool-call-export-record-design.md @@ -0,0 +1,378 @@ +# Tool calls in the export: a paired, content-free `toolCall` record — design + +Status: **draft**. +Date: 2026-11-19. +Issue: rossoctl/cortex#1906 (feature tracker, §2 of three) +Background: #1810 (the export's own spec and tracker), #1830/#1812/#1831 (what shipped), +#1740 (three count fields), #1924 (Claude Code telemetry), #1997 (`sessionIncarnation`), +#1922 (prompt grouping) +Parent spec: [`2026-11-13-session-event-export-design.md`](2026-11-13-session-event-export-design.md). +This one narrows its §7.2 `toolCallNames` item and supersedes the shape proposed there. + +Items marked **Open** must be decided before the work they belong to begins. Nothing here +is merged. + +**22 numbered requirements T1–T22**, phrased as plain declaratives: a requirement says what +the system does once built. The numbering is stable so it can be cited in review and in the +PRs that implement it. + +Written after measuring, not before. §3 is the measurement; every design choice in §5 and §6 +traces to a number in it. The measurement corpus is one developer's 30-day laptop archive: +40,496 export records, 12,104 tool calls, `claude-code/2.1.270` throughout with 46 +`codex/0.163.0` records. **The mechanism generalises; the ratios do not** — see §9. + +--- + +## 1. Goal + +1. Answer "which tools does our community run, and what do they cost in time" from the + export alone, across every session and beyond the archive's 30-day retention. +2. Answer "how does inference time compare with tool time" as a ratio a capacity plan can + use. +3. Keep the export content-free by construction, as #1810 §3.2 requires: no tool arguments, + no tool results, no prompts. + +### 1.1 Why the export, when `/timeline` already pairs tool calls + +`GET /v1/sessions/{id}/timeline` answers per-tool latency today, content-free, by pairing on +tool-use id (#1810 §9.1). It is the right tool for one session. It cannot do this job: + +| Need | Why `/timeline` cannot | +|---|---| +| Fleet totals across every session | one session per HTTP request, one compute each | +| Longitudinal: how p50 moved across a release | no durable artifact; the archive ages out at 30 days | +| Join tool timing to token counts | the timeline carries `outputTokens` and no cache counts | +| Deep sessions | capped at 50,000 events per session | + +Measured cost of doing it the timeline way on the reference install: 82 timeline requests at +~1 s each for a session only on disk, against ~10 s for one `jq` pass over a 29,606-record +NDJSON file (#1810 §9.2). + +--- + +## 2. What exists today + +### 2.1 Shipped + +- `core/sessionexport.Record` — ~36 flat scalars per event, an allowlist, with + `TestRecord_CarriesNoContent` pinning the privacy contract (#1812). +- `Walk` over `archive.Earlier` — newest-first, one event at a time, O(1) state (#1830). +- `GET /v1/export` and `agentop export` (#1830, #1831), with `--gzip` (#1996). + +A record carries `toolCount` (the manifest the request offered) and `toolCallCount` (the +calls the response emitted). It carries no tool name and no per-tool timing. + +### 2.2 The pairing already exists, in `core/session/timeline` + +`timeline.Tool` is `{Agent, ID, Name, Start, End, Preview, Error, Lost, Seq}`, built in +`core/session/timeline/agents.go`. Reading that code is what sized this spec, so its shape is +stated here rather than summarised: + +- It walks **request/response pairs per agent** (`type call struct {req, resp *model.SessionEvent}`), + not single events. +- A span's **start is `responseEnd(req, resp)`** = `req.At + resp.Duration`, falling back to + `resp.At`. So a tool's start is derived from its asking request plus the response's measured + duration — not from any one event's timestamp. +- A span's **end is the first later request of the same agent carrying that `ToolUseID`**, a + forward lookup across events, clamped up to the start when the result lands before the + asking response finished. +- Four further closing cases exist and each one is load-bearing: `endedRun` (a + `SubagentHandback`, `StructuredOutput` or `end_subtask` call never gets a result), the agent + moving on without the result (`lastTurn`), a `` stopping the run + (`stopped`), and background jobs (`backgroundRe`) that extend a `Bash` span past its + acknowledgement. + +### 2.3 The gap, and why it is structural + +`Walk` is newest-first and stateless; pairing needs forward lookup and per-agent state. So +this is **not** a new field on `FromEvent`. It needs either a buffered second pass per session +or a reorganised walk — a design decision with a memory cost, in a package whose premise is +that it is cheap. §6 is that decision. + +--- + +## 3. The measurement + +Method: `/v1/export` for the record view, `/v1/sessions/{id}/timeline` for the paired view, +over the same four sessions. 268 tool spans, 267 usable (one still running, zero `lost`). + +### 3.1 Per-tool timing + +| Tool | n | p50 | p95 | total | share of tool time | +|---|---|---|---|---|---| +| `Bash` | 210 | 473 ms | 7,561 ms | 598.9 s | 97.4% | +| `AskUserQuestion` | 1 | 12,714 ms | — | 12.7 s | 2.1% | +| `Agent` | 1 | 1,093 ms | — | 1.1 s | 0.2% | +| `Read` | 17 | 41 ms | 194 ms | 1.0 s | 0.2% | +| `Edit` | 29 | 24 ms | 66 ms | 0.8 s | 0.1% | +| `Write` | 9 | 16 ms | 224 ms | 0.4 s | 0.1% | + +Errors: `Bash` 11, `Read` 3. + +**`Bash` is 78% of calls and 97.4% of tool time.** One tool dominates, which is why the name +alone nearly answers the workload question — and why the *duration distribution*, not the +name, is the signal worth carrying. + +### 3.2 Inference against tools + +| | calls | total | p50 | +|---|---|---|---| +| Model | 307 | 2,510.7 s | 5,473 ms | +| Tool | 267 | 614.8 s | 243 ms | + +**4.1 : 1** — inference is 80.3% of measured work. For sandbox capacity this is the headline: +the agent waits on the model four times longer than it runs tools. + +### 3.3 Weight tiers, on E15's cuts (1 s / 10 s) + +| Tier | share of calls | share of work | +|---|---|---| +| light (<1 s) | 76.0% | 8.3% | +| medium (1–10 s) | 22.1% | 36.3% | +| **heavy (>10 s)** | **1.9%** | **55.4%** | + +The heaviest 10% of calls carry **77.7%** of tool time; the heaviest single call was a 145.2 s +`Bash`. This reproduces E15's concentration finding (7.4% of calls / 86.5% of work) on our own +traffic, independently. It is the number density planning turns on, and it is the reason a +mean is useless here: a design that reports averages hides the tail that sizes the sandbox. + +### 3.4 Parallel calls share one window — the finding that shapes the record + +**42 of 267 spans (15.7%) carry an identical start and end**: calls the model issued together. +The timeline gives a batch one merged interval, not a duration each. + +So a naive per-call `durationMs` is wrong for one call in six, and wrong in the direction that +inflates: charging every member of a batch the whole window's time overcounts tool time and +drags the heavy tier upward. This is the same class of error as the 272x tool-result overcount +#1810 documents, and the record must make it explicit rather than leave a consumer to discover +it. + +### 3.5 The two views reconcile + +Summed `toolCallCount` over the same sessions is **273** against **267** paired spans — 2.2% +apart, consistent with calls in flight at the archive boundary. A record derived archive-side +will therefore not contradict `/timeline`. + +--- + +## 4. Requirements + +### 4.1 Needs + +T1. The export answers per-tool call counts and duration distributions across every archived +session, without a second endpoint and without a hand-written join. +T2. The export answers the inference-time-to-tool-time ratio per session and in aggregate. +T3. A consumer can compute the §3.3 weight tiers from the export alone. +T4. A consumer can separate built-in tools from MCP tools without parsing a name. +T5. A tool call's record names the session, the agent role and the client, so a fleet view can +split by any of them. + +### 4.2 Privacy and safety + +T6. A `toolCall` record carries **no tool arguments**. A `Bash` command line is a secret and +stays in #1810's Tier 3, permanently. +T7. A `toolCall` record carries **no tool result content**. Result *size* may be carried +(T14); its text may not. +T8. A `toolCall` record carries **no tool-use id**. The id is the pairing key, used inside the +proxy and discarded: the export ships the measurement the key yields, never the key. This is +what keeps a `toolCall` record unjoinable back to a conversation. +T9. `TestRecord_CarriesNoContent` is extended to cover the `toolCall` record, planting a +secret in every content field the pairing reads — tool arguments, result content and the +preview fields `/timeline` builds — and failing if any reaches the encoding. +T10. A built-in tool name is exported verbatim: it comes from a closed vocabulary +(`Bash`, `Read`, `Edit`, …). +T11. An **MCP tool name is operator-defined and namespaced**, so it can carry a vendor +relationship, an internal system name or a project codename — `mcp__acme-billing-prod__refund` +is the worked example. It is therefore **not** treated as a built-in name. **Open:** gate it +off by default, or export a stable hash so distinct MCP tools stay countable without being +readable. Hashing is the author's recommendation: it preserves T1 and T4 for MCP traffic, +which gating does not. +T12. A tool name is bounded on the way out by the same `exportLabel` every wire-sourced +string in a record passes: control runes removed, then truncated to `maxLabelLen` (96) on a +rune boundary. + +### 4.3 Data rules + +T13. A `toolCall` record is **flat scalars only**, as #1810 R2 requires of a record. No array, +no nested object. This is what keeps one line one thing and keeps `jq` usable. +T14. `resultBytes` is the size of the tool's result, as the effort proxy T7 permits instead of +content. Absent means not measured, never zero — the export's zero-means-absent rule. +T15. **A batch of parallel calls is marked as such.** A record whose duration comes from a +window shared with other calls carries `durationShared: true`, the batch's size in +`parallelN`, and a `parallelGroup` key that is stable within the session. A consumer can then +divide, take the window once per batch, or exclude it — explicitly, by its own choice. +T16. A `lost` span — one that never got a result but provably stopped — is marked `lost: true` +and **carries no duration**, rather than the zero-length duration `/timeline` draws it with. A +zero would pass a consumer's `durationMs > 0` filter as a 0 ms call and drag a p50 toward +zero. +T17. A span still open when the walk reads it is not exported. An unfinished call has no +duration to report, and a record that appears with one later would make the export +non-reproducible. +T18. Durations are milliseconds, truncated, matching `Record`. + +### 4.4 Schema and compatibility + +T19. A `toolCall` record is a **sibling record on the same NDJSON stream**, discriminated by a +`kind` field, not a widening of the event record. An existing consumer either filters +`select(.kind != "toolCall")` or ignores the key: an absent `kind` reads as the event record, +so no shipped consumer breaks. +T20. `toolCall` records are **opt-in**, per #1810 R49's rule for Tier 2: a plain +`GET /v1/export` is byte-identical to today's. **Open:** `?include=toolCalls` on the endpoint +and `--with-tool-calls` on the command, or a single Tier-2 switch covering every §7.2 item. +T21. Adding the record does not bump `Record`'s `schema`. A new record *kind* on the stream is +additive; `sessionexport.SchemaVersion` bumps only on a removal or a meaning change. + +### 4.5 Documentation + +T22. `docs/session-export.md` states the §3.4 parallel-window rule and the T16 `lost` rule +beside the record, because both cost real accuracy and neither is guessable from the field +list. The doc fix that precedes this spec already states them for `/timeline`. + +--- + +## 5. The record + +```json +{"schema":1,"kind":"toolCall","sessionId":"45fa31b2-…","seq":2081, + "at":"2026-11-05T15:52:26.102885Z","tool":"Bash","toolSource":"builtin", + "callSeq":2077,"resultSeq":2081,"durationMs":473,"isError":false, + "resultBytes":1432,"agentRole":"main","client":"claude-code/2.1.270", + "parallelGroup":"2077#0","parallelN":3,"durationShared":true} +``` + +| Field | Why it is here | +|---|---| +| `kind` | T19. The discriminator. Always `"toolCall"` on this record. | +| `sessionId` | T5. The session, as the client asserted it. Not a unit of work until #1997. | +| `seq` | Orders the record within its session, as on an event record. The result event's seq. | +| `at` | When the call ended, so a record sorts on one time base with event records. | +| `tool` | T10/T11. The name, bounded by T12. | +| `toolSource` | T4. `builtin`, `mcp` or `unknown` — a closed vocabulary, exported by naming the values. | +| `callSeq` / `resultSeq` | The response that asked and the request that carried the result. Lets a consumer rejoin a `toolCall` to the event records either side **without a tool-use id** (T8). | +| `durationMs` | T18. The paired duration. Absent on a `lost` record (T16). | +| `isError` | Whether the result was an error. The tool failure rate, per tool. | +| `resultBytes` | T14. Effort proxy, never content. | +| `agentRole` | T5. `main` or `subagent`, so subagent tool traffic reads apart. | +| `client` | T5. Cost-attribution and display key. **Never an authorization subject.** | +| `parallelGroup` / `parallelN` / `durationShared` | T15. The §3.4 finding, made explicit. | +| `lost` | T16. Set, with no `durationMs`, on a call that stopped without a result. | + +Not here, deliberately: the tool-use id (T8), arguments (T6), result content (T7), the +`/timeline` `preview` (it is built from argument keys and is explicitly "no prompts, not safe +to share" — #1810 §9.1). + +--- + +## 6. Where the pairing runs + +Three options. The spec records all three because the choice has consequences a reviewer +should weigh, not just the one the author prefers. + +### 6.1 Option A — extract the pairing into a shared package + +Move the span-building out of `core/session/timeline` into something both the timeline and the +export call. + +- **For:** one copy of a subtle algorithm, with §2.2's four closing cases in one place. Both + consumers get a fix at once. +- **Against:** `timeline` is consumed by agentop, so the extraction is a refactor with a live + dependent. It is the largest change of the three. + +### 6.2 Option B — `sessionexport` depends on `timeline` + +The walk calls the timeline builder per session and projects its spans. + +- **For:** smallest diff, no new pairing code. +- **Against:** inherits the 50,000-event cap, which bounds the export by a timeline limit that + has nothing to do with it; and it builds turns, hosts and agents the export discards. + +### 6.3 Option C — a second, bounded pass in `sessionexport` + +The walk makes a pairing pass per session, keeping only tool-use id → (name, start) until the +result lands. + +- **For:** keeps the export's dependencies as they are; the state is one map per session, not + the whole timeline. +- **Against:** a second implementation of §2.2's closing cases, which will drift from the + timeline's. + +**Open: which option.** The author leans **A**, because §2.2's four closing cases are exactly +the kind of logic that must not exist twice, and because the export is the second consumer that +wants it — a third (an OTLP emitter, #505) would make the duplication permanent. + +### 6.4 Memory, whichever option wins + +T-level constraint rather than a numbered requirement, because it binds all three: + +The walk is O(1) per event today. Pairing is O(open calls) per session. On the reference +archive a session held 2,911 events; the largest in §3 held 722 with 204 tool spans. A bound +is needed, with a documented behaviour when it is hit — the export's existing posture is to +degrade loudly (a truncated stream resets the connection) rather than silently, and this +should match. **Open:** the bound and what happens at it. + +--- + +## 7. What this does not answer + +7.1 **Local tool calls are invisible.** `durationMs` here is proxy-observed, so it covers +tools that generate proxied traffic. An `Edit`, `Read` or `Bash` touching nothing on the +network appears only as gap time — #1810 measures that at 5,653 calls and 5.3 h. For sandbox +sizing this may be *correct*, since those calls cost no sandbox time either, but **it +undercounts call volume and any table built from it says so.** Closing the gap needs #1924 +(Claude Code telemetry), not this spec. + +7.2 **Which `Bash`.** The name is `Bash`; which command ran is in the arguments, which T6 +keeps out permanently. So this spec answers "how long do tools take" and never "what did they +do". A workload taxonomy finer than the tool name needs #1810's Tier 3 and a different privacy +posture. + +7.3 **A session is not a unit of work** until #1997 lands `sessionIncarnation`. Per-session +aggregates of `toolCall` records average across unrelated sittings exactly as event records do. + +7.4 **Prompt grouping** is #1922. A `toolCall` record says which turn asked (`callSeq`), not +which user instruction. + +--- + +## 8. Relationship to adjacent work + +| Issue | Relationship | +|---|---| +| #1906 | The tracker. This spec narrows its §2 and **supersedes** its "list on the response record" shape, for the §3.4 reason. | +| #1810 | The parent spec. This is its §7.2 `toolCallNames` item, re-scoped from names to paired calls. | +| #1740 | Adds `newToolResults` and `toolResultErrors` to the event record. Complementary: those count results per event; this counts calls with durations. | +| #1924 | Claude Code telemetry would supply the tool times the proxy cannot see (§7.1), including permission waits. The better long-term answer for local tools. | +| #1997 | `sessionIncarnation`. Needed before per-session aggregates mean anything (§7.3). | +| #505 | An OTLP emitter would be the third consumer of the §6 pairing, and the reason to prefer Option A. | +| rossoctl/trajectory-schema | A `toolCall` record maps closely to that schema's `$defs.sessionToolCall`, which already records a tool call as sizes rather than content (`command_chars`, `result_preview_chars`). Worth aligning field names where the semantics match. | + +--- + +## 9. Scope of the measurements + +One developer's laptop archive over 30 days: 40,496 records, 12,104 tool calls, heavily +`claude-code/2.1.270` with 46 `codex/0.163.0` records. Four sessions had timeline spans. + +The **mechanism** generalises, because it follows the event schema and the pairing rules, not +this traffic. The **ratios do not**: 4.1:1 inference-to-tool and `Bash` at 97.4% of tool time +will move with working style, prompting and agent. Fleet numbers are the entire point of +building this, and they will come from the export rather than from §3. + +A second caveat on §3.4: 15.7% is one archive's parallel rate under one agent's batching +behaviour. #1810 notes Claude Code's calls are sequential in the trace while Codex's overlap, +so the rate is agent-dependent and the 46 Codex records here cannot speak to it. The +requirement T15 derives from it stands regardless — a shared window must be marked whatever +its frequency — but the *rate* is not a constant to design against. + +--- + +## 10. Open items, collected + +| § | Open | +|---|---| +| T11 | MCP tool names: gate off by default, or export a stable hash. Author recommends hashing. | +| T20 | The opt-in switch: per-item (`?include=toolCalls`) or one Tier-2 switch. | +| §6 | Which of the three pairing options. Author leans A. | +| §6.4 | The per-session pairing bound, and the behaviour when it is hit. | + +Assisted-By: Claude (Anthropic AI) From 055942c7f4f5c2be1b97f4113c2996837f748024 Mon Sep 17 00:00:00 2001 From: Mariusz Sabath Date: Thu, 19 Nov 2026 17:21:02 -0500 Subject: [PATCH 03/12] Docs: Add the gap split and the collection/export boundary to the spec Two things the first draft buried, both raised in review. Nothing here needs new capture. Tool arguments and tool result content are already collected AND already on disk: archive/format.go interns ToolResults[i].Content and HistoryToolCalls[i].Arguments, so a 30-day archive holds every command line and every result. The export carries none of it by construction. A new 2.0 says so, because the rest of 2 read like a capability gap and it is not one: this is a projection change. It also separates the two decisions the first draft let blur, since "just export what we collect" is true for names and timing and false for arguments. Content is Tier 3, with redaction (#1921) and its own route (#1925), not a flag on this record. T0 pins that. The first draft also carried a tool duration with no inter-call gap at all, which cannot answer occupancy: a duration says how long the sandbox was busy, never how long it sat idle holding resources. 4.3a adds the split, and rests on the one asymmetry that makes it sound -- model time is MEASURED, so only the leftover gap needs classifying. modelMs copies the asking response's duration, gapMs is the time since the previous call ended, and gapKind names it user, tool or inference from toolCallCount, which is already in the record. The classifier is already validated: on the reference archive it split AskUserQuestion (p50 62.6s) from Bash (p50 201ms) by three orders of magnitude without seeing a tool name, and user wait is 72.6% of gap time -- idle that costs no sandbox occupancy, which is the whole point of separating it. Two limits come with it. An approval wait lands inside a tool gap and cannot be told from execution, which is the largest error in any density figure built from this (T26, 7.1a); and a user gap is meaningless without a stated ceiling, since the same data totals 24.8h at 300s and 539h uncapped (T27). #1924 is the fix for both, so 8 now says the two should be planned together. Correct 2.1 and 2.2 while here: today's record has no per-tool timing of any kind, and the timeline's pairing is derived, not stored. Assisted-By: Claude (Anthropic AI) Signed-off-by: Mariusz Sabath --- ...26-11-19-tool-call-export-record-design.md | 122 +++++++++++++++++- 1 file changed, 115 insertions(+), 7 deletions(-) diff --git a/docs/superpowers/specs/2026-11-19-tool-call-export-record-design.md b/docs/superpowers/specs/2026-11-19-tool-call-export-record-design.md index 898f4db0d..f8b40fa33 100644 --- a/docs/superpowers/specs/2026-11-19-tool-call-export-record-design.md +++ b/docs/superpowers/specs/2026-11-19-tool-call-export-record-design.md @@ -12,9 +12,19 @@ This one narrows its §7.2 `toolCallNames` item and supersedes the shape propose Items marked **Open** must be decided before the work they belong to begins. Nothing here is merged. -**22 numbered requirements T1–T22**, phrased as plain declaratives: a requirement says what +**28 numbered requirements T0–T27**, phrased as plain declaratives: a requirement says what the system does once built. The numbering is stable so it can be cited in review and in the -PRs that implement it. +PRs that implement it. T0 sits in §2.0 because it bounds the whole spec rather than one +section, and T23–T27 in §4.3a. + +Revised 2026-11-19, the same day, after review raised two things the first draft buried. +§2.0 is new: every field here is **already collected and most of it is already on disk**, so +this is a projection change and not an instrumentation one — and arguments and result content +are a separate decision (Tier 3), not a larger version of this one. §4.3a is new: the first +draft carried a tool duration with **no inter-call gap at all**, which cannot answer occupancy +— a duration says how long the sandbox was busy, never how long it sat idle holding resources. +§2.1 and §2.2 are corrected: today's record has no per-tool timing of any kind, and the +timeline's pairing is derived rather than stored. Written after measuring, not before. §3 is the measurement; every design choice in §5 and §6 traces to a number in it. The measurement corpus is one developer's 30-day laptop archive: @@ -52,6 +62,38 @@ NDJSON file (#1810 §9.2). ## 2. What exists today +### 2.0 Nothing here needs new capture — and that is the point + +**Everything this spec exports is already collected, and most of it is already on disk.** The +work is a projection, not an instrumentation change. Stating it plainly because the rest of +§2 reads like a capability gap, and it is not one: + +| Already collected | Where | On disk? | +|---|---|---| +| Tool call name and id | `InferenceToolCall{ID, Name}` on a response | yes | +| Tool call **arguments** | `InferenceToolCall.Arguments` | yes — `archive/format.go` interns `HistoryToolCalls[i].Arguments` | +| Tool **result content** | `InferenceToolResult.Content` | yes — `archive/format.go` interns `ToolResults[i].Content` | +| Result error flag | `InferenceToolResult.IsError` | yes | +| Pairing key | `InferenceToolResult.ToolUseID`, message `ToolUseIDs` | yes, interned | +| Inference duration | `SessionEvent.Duration` | yes | + +So a 30-day archive already holds every tool call's arguments and every tool result's text. +The export carries none of it, by construction (#1810 R7). + +**Two things follow, and they are different decisions, not two sizes of one:** + +1. **Names, per-call timing and result size are a projection change.** Content-free, covered + by this spec, and shippable without touching the privacy posture. +2. **Arguments and result content are NOT.** They are content — a `Bash` command line is the + canonical example (#1810 §4.1) — so exporting them is Tier 3, which needs redaction + (#1921) and its own loopback-only route and command (#1925), not a flag on this one. + `TestRecord_CarriesNoContent` exists to fail the build if they ever arrive here by + accident. + +T0. This spec exports no tool argument and no tool result content, in any tier or mode it +defines. A consumer that needs those uses the Tier 3 path, which is specified elsewhere and +carries a different privacy posture. + ### 2.1 Shipped - `core/sessionexport.Record` — ~36 flat scalars per event, an allowlist, with @@ -60,9 +102,23 @@ NDJSON file (#1810 §9.2). - `GET /v1/export` and `agentop export` (#1830, #1831), with `--gzip` (#1996). A record carries `toolCount` (the manifest the request offered) and `toolCallCount` (the -calls the response emitted). It carries no tool name and no per-tool timing. +calls the response emitted). **Both are counts and nothing else.** Today's export therefore +has: + +- no tool name; +- **no per-tool timing of any kind.** `InferenceToolCall` is three strings — `{ID, Name, + Arguments}` — and `InferenceToolResult` is three more. Neither carries a timestamp or a + duration, and no field anywhere on the event model does per-call timing. The only per-tool + timing in the system is **derived** by the timeline (§2.2), never stored; +- **no gap between calls.** No `gapMs`, think-time or idle field exists on a record, on the + event, or on a timeline span. A consumer can derive a gap between *events* from `at` and + `durationMs`, which is what #1810 §9 did; it cannot derive one between *tool calls*, + because it has no tool call to anchor to. + +The second and third are what make today's record unable to answer occupancy: a count says +how many calls a turn made, never how long the sandbox was busy or idle. -### 2.2 The pairing already exists, in `core/session/timeline` +### 2.2 The pairing is DERIVED by `core/session/timeline`, not stored `timeline.Tool` is `{Agent, ID, Name, Start, End, Preview, Error, Lost, Seq}`, built in `core/session/timeline/agents.go`. Reading that code is what sized this spec, so its shape is @@ -210,6 +266,45 @@ duration to report, and a record that appears with one later would make the expo non-reproducible. T18. Durations are milliseconds, truncated, matching `Record`. +### 4.3a Model time, tool time and user time + +The three quantities a capacity plan needs, and the one asymmetry that makes the split sound: +**model time is measured, not inferred.** `durationMs` on an inference response *is* the +model's time. Only the leftover gap has to be classified. + +T23. A `toolCall` record carries `modelMs`: the measured duration of the inference response +that asked for this call. It is the turn's model time, copied, not derived — so a consumer +gets the inference/tool ratio per call without joining back to the event record. +T24. A `toolCall` record carries `gapMs`: the time from the previous tool call's end to this +call's start, within the same agent. Absent on an agent's first call, where there is no +previous end — absent, never zero. +T25. A `toolCall` record carries `gapKind`, a **closed vocabulary** exported by naming its +values, never by sanitising what arrives: + +| `gapKind` | The preceding response | What the gap is | +|---|---|---| +| `user` | `toolCallCount == 0` | The model stopped and waited for a human. Nothing else can be happening. | +| `tool` | `toolCallCount > 0` | Tool execution, plus any approval wait (T26). | +| `inference` | a model call landed inside the gap | Model time, already in `modelMs`. Not idle. | + +T26. **An approval wait is inside `tool` and cannot be separated from execution.** When a user +sits on a permission prompt, that time lands in the tool gap and is indistinguishable from the +command running. E15's SPEC caps it at 30 s for exactly this reason. This is the largest +single error source in any density figure built from this export, and the docs say so beside +the field. Only #1924 (Claude Code telemetry, which records permission waits explicitly) +removes it. +T27. **A `user` gap is meaningless without a stated ceiling, and the record does not impose +one.** Measured on the reference archive, the same data totals 24.8 h at a 300 s cap, 44.8 h +at 1800 s, 56.8 h at 3600 s and **539 h uncapped** — a 20x spread. So the export carries the +raw `gapMs` and the documentation requires any consumer publishing a duty cycle to publish its +ceiling alongside it (#1810 §9 states the same rule for event gaps). + +Why this split is worth the three fields: on the reference archive, user wait is **72.6% of +all gap time** (44.8 h of 61.7 h) and costs nothing in sandbox occupancy. Without `gapKind`, +a density plan reads that as idle capacity it is paying for. The classifier is already +validated — run on that archive it separated `AskUserQuestion` (p50 62.6 s) from `Bash` +(p50 201 ms) by three orders of magnitude **without ever seeing a tool name** (#1810 §9). + ### 4.4 Schema and compatibility T19. A `toolCall` record is a **sibling record on the same NDJSON stream**, discriminated by a @@ -237,6 +332,7 @@ list. The doc fix that precedes this spec already states them for `/timeline`. "at":"2026-11-05T15:52:26.102885Z","tool":"Bash","toolSource":"builtin", "callSeq":2077,"resultSeq":2081,"durationMs":473,"isError":false, "resultBytes":1432,"agentRole":"main","client":"claude-code/2.1.270", + "modelMs":4279,"gapMs":812,"gapKind":"tool", "parallelGroup":"2077#0","parallelN":3,"durationShared":true} ``` @@ -252,6 +348,9 @@ list. The doc fix that precedes this spec already states them for `/timeline`. | `durationMs` | T18. The paired duration. Absent on a `lost` record (T16). | | `isError` | Whether the result was an error. The tool failure rate, per tool. | | `resultBytes` | T14. Effort proxy, never content. | +| `modelMs` | T23. The asking response's measured duration — model time, per call, no join needed. | +| `gapMs` | T24. Since the previous tool call ended, within the agent. Absent on the first call. | +| `gapKind` | T25. `user`, `tool` or `inference` — what the gap was. The field that turns idle time into a duty cycle. | | `agentRole` | T5. `main` or `subagent`, so subagent tool traffic reads apart. | | `client` | T5. Cost-attribution and display key. **Never an authorization subject.** | | `parallelGroup` / `parallelN` / `durationShared` | T15. The §3.4 finding, made explicit. | @@ -321,8 +420,14 @@ sizing this may be *correct*, since those calls cost no sandbox time either, but undercounts call volume and any table built from it says so.** Closing the gap needs #1924 (Claude Code telemetry), not this spec. -7.2 **Which `Bash`.** The name is `Bash`; which command ran is in the arguments, which T6 -keeps out permanently. So this spec answers "how long do tools take" and never "what did they +7.1a **An approval wait reads as execution.** T26: a permission prompt the user sits on lands +inside a `tool` gap, indistinguishable from the command running. Nobody has quantified it +here, and it inflates exactly the figure a sandbox is sized against. Measuring it against +local Claude Code transcripts — where the harness records the wait — is the cheapest way to +bound the error, and should happen before any density number from this export is published. + +7.2 **Which `Bash`.** The name is `Bash`; which command ran is in the arguments, which T6 and +T0 keep out permanently. So this spec answers "how long do tools take" and never "what did they do". A workload taxonomy finer than the tool name needs #1810's Tier 3 and a different privacy posture. @@ -341,7 +446,8 @@ which user instruction. | #1906 | The tracker. This spec narrows its §2 and **supersedes** its "list on the response record" shape, for the §3.4 reason. | | #1810 | The parent spec. This is its §7.2 `toolCallNames` item, re-scoped from names to paired calls. | | #1740 | Adds `newToolResults` and `toolResultErrors` to the event record. Complementary: those count results per event; this counts calls with durations. | -| #1924 | Claude Code telemetry would supply the tool times the proxy cannot see (§7.1), including permission waits. The better long-term answer for local tools. | +| #1924 | Claude Code telemetry would supply the tool times the proxy cannot see (§7.1) **and the permission waits T26 cannot separate from execution (§7.1a)**. It is the fix for both of this spec's largest error sources, so the two should be planned together even though this one ships first. | +| #1921 / #1925 | Redaction and the content export. Where tool arguments and result content go, per T0 — a different privacy posture, not a flag on this record. | | #1997 | `sessionIncarnation`. Needed before per-session aggregates mean anything (§7.3). | | #505 | An OTLP emitter would be the third consumer of the §6 pairing, and the reason to prefer Option A. | | rossoctl/trajectory-schema | A `toolCall` record maps closely to that schema's `$defs.sessionToolCall`, which already records a tool call as sizes rather than content (`command_chars`, `result_preview_chars`). Worth aligning field names where the semantics match. | @@ -374,5 +480,7 @@ its frequency — but the *rate* is not a constant to design against. | T20 | The opt-in switch: per-item (`?include=toolCalls`) or one Tier-2 switch. | | §6 | Which of the three pairing options. Author leans A. | | §6.4 | The per-session pairing bound, and the behaviour when it is hit. | +| T25 | Whether `gapKind` needs a fourth value for a gap that spans both a model call and tool execution, or whether `inference` winning is the right precedence. | +| §7.1a | Whether to quantify the approval-wait error against local transcripts before publishing any density figure. Author's view: yes, and it is cheap. | Assisted-By: Claude (Anthropic AI) From acf2bb029bd27278b06c4a19b732de3961317394 Mon Sep 17 00:00:00 2001 From: Mariusz Sabath Date: Thu, 19 Nov 2026 19:59:08 -0500 Subject: [PATCH 04/12] Docs: Correct four measurement claims and redefine the gap Addresses @webchang's review on #2009, which read the draft against the code and against a converter written for it. Four claims were wrong. 7.1 contradicted 3.1. A span runs from the asking response's end to the next request carrying the result, and both boundaries are model traffic, so a tool that touches nothing on the network still gets a span -- 3.1's own Edit at p50 35ms is one. The "local calls are invisible" claim was #1810's gap classification misapplied, where no proxied traffic inside a gap is how a local call is INFERRED. 7.1 now states the real limit: the proxy sees the gap around a tool, never inside it. New 7.1b separates sizing the model leg from sizing a sandbox host, which this record cannot do -- a host is bounded by concurrent commands and the memory each holds, and no field here measures either. It carries the join keys for a sandbox-side record instead. 3.2 and 3.3 double-counted shared windows and priced AskUserQuestion as tool time. 3 now states three method rules and every figure is re-derived: the ratio moves 4.1 to 4.2:1, the long band 55.4% to 61.7% of work. The shared- window overcount is 1.9%, smaller than 3.4's span share suggests, because batched calls are short. New 3.5: durationMs is a client round trip including permission waits, which the first draft said only of the gap. So the ratio counts approval waits as tool time for any tool that prompts, and no rule in 3 removes that. 4.3a's gap was undefinable. T24's interval always contained modelMs while T25 classified a different one, so 5's example could not be checked against either. Now one named interval -- from the previous response's end to the next request -- with modelMs outside it by construction and the fourth gapKind dropped as an artifact of the old definition. Record shape follows: startedAt replaces at and is always present, so a call with no result can still be placed (T29); seq is gone, having collided with the event record's and been undefined for an unresolved call; callIndex orders a batch (T30); a closed endReason replaces the lost flag and names the timeline's four closing cases apart (T31). T28 gives event records an explicit kind. T32 says background jobs are out of scope rather than leaving it ambiguous. T33 drops a call whose asking response is below a since floor. Also: T11 takes a keyed hash, since a namespaced name is guessable and an unkeyed digest reversible; T10 exports a name verbatim, as the built-in vocabulary differs per client; T12 records that the 96-byte cut is bytes; T14 adds inputBytes; T9's planted secrets must cover message and result text, which the closing cases read; 1.1 calls /timeline content-light, not content- free, since Tool.Preview carries 80 runes of argument text. 10 splits into what the review settled and what is still open. Assisted-By: Claude (Anthropic AI) Signed-off-by: Mariusz Sabath --- ...26-11-19-tool-call-export-record-design.md | 451 +++++++++++++----- 1 file changed, 329 insertions(+), 122 deletions(-) diff --git a/docs/superpowers/specs/2026-11-19-tool-call-export-record-design.md b/docs/superpowers/specs/2026-11-19-tool-call-export-record-design.md index f8b40fa33..c9805acfa 100644 --- a/docs/superpowers/specs/2026-11-19-tool-call-export-record-design.md +++ b/docs/superpowers/specs/2026-11-19-tool-call-export-record-design.md @@ -12,19 +12,43 @@ This one narrows its §7.2 `toolCallNames` item and supersedes the shape propose Items marked **Open** must be decided before the work they belong to begins. Nothing here is merged. -**28 numbered requirements T0–T27**, phrased as plain declaratives: a requirement says what +**34 numbered requirements T0–T33**, phrased as plain declaratives: a requirement says what the system does once built. The numbering is stable so it can be cited in review and in the -PRs that implement it. T0 sits in §2.0 because it bounds the whole spec rather than one -section, and T23–T27 in §4.3a. - -Revised 2026-11-19, the same day, after review raised two things the first draft buried. -§2.0 is new: every field here is **already collected and most of it is already on disk**, so -this is a projection change and not an instrumentation one — and arguments and result content -are a separate decision (Tier 3), not a larger version of this one. §4.3a is new: the first -draft carried a tool duration with **no inter-call gap at all**, which cannot answer occupancy -— a duration says how long the sandbox was busy, never how long it sat idle holding resources. -§2.1 and §2.2 are corrected: today's record has no per-tool timing of any kind, and the -timeline's pairing is derived rather than stored. +PRs that implement it, and is therefore in the order the requirements were ADDED, not document +order: T0 sits in §2.0 because it bounds the whole spec, T23–T27 in §4.3a, and T28–T33 in §4.4 +and §4.4a. + +Revised twice on 2026-11-19. + +**First revision**, after review raised two things the first draft buried. §2.0 is new: every +field here is **already collected and most of it is already on disk**, so this is a projection +change and not an instrumentation one — and arguments and result content are a separate +decision (Tier 3), not a larger version of this one. §4.3a is new: the first draft carried a +tool duration with **no inter-call gap at all**, which cannot answer occupancy. §2.1 and §2.2 +corrected: today's record has no per-tool timing of any kind, and the timeline's pairing is +derived rather than stored. + +**Second revision**, after @webchang's review on #2009, which read the draft against the code +and against a converter written for it. Four measurement claims were wrong and are corrected: + +- **§7.1 contradicted §3.1.** A span's two boundaries are both model traffic, so a local tool + call *does* get a span — the first draft's "local calls are invisible" was #1810 §9's gap + classification misapplied. §7.1 now states the real limit (the proxy sees the gap around a + tool, never inside it), and §7.1b separates sizing the model leg from sizing a sandbox host. +- **§3.2 and §3.3 double-counted shared windows** and priced `AskUserQuestion` as tool time. + §3 now states three method rules and every figure is re-derived under them: the ratio moves + 4.1 → **4.2 : 1**, the long band 55.4% → **61.7%** of work. +- **§3.5 is new:** `durationMs` is a client round trip including permission waits, which the + first draft stated only for the gap. +- **§4.3a's gap was undefinable** — T24's interval always contained `modelMs` while T25 + classified a different one. Now one named interval, with `modelMs` outside it by + construction and the fourth `gapKind` dropped as an artifact. + +Record shape changed with it: `startedAt` replaces `at` and is always present (T29), `seq` and +`parallelGroup` are gone, `callIndex` (T30) and a closed `endReason` (T31) are added. T28 gives +event records an explicit `kind`. T32 says background jobs are out of scope rather than leaving +it ambiguous; T33 says a call below a `since` floor is dropped. §10 splits into what the review +settled and what is still open. Written after measuring, not before. §3 is the measurement; every design choice in §5 and §6 traces to a number in it. The measurement corpus is one developer's 30-day laptop archive: @@ -38,14 +62,21 @@ traces to a number in it. The measurement corpus is one developer's 30-day lapto 1. Answer "which tools does our community run, and what do they cost in time" from the export alone, across every session and beyond the archive's 30-day retention. 2. Answer "how does inference time compare with tool time" as a ratio a capacity plan can - use. + use — for the **model leg**. Sizing a remote sandbox host needs a sandbox-side record + joined to this one; see §7.1b before using these figures for that. 3. Keep the export content-free by construction, as #1810 §3.2 requires: no tool arguments, no tool results, no prompts. ### 1.1 Why the export, when `/timeline` already pairs tool calls -`GET /v1/sessions/{id}/timeline` answers per-tool latency today, content-free, by pairing on -tool-use id (#1810 §9.1). It is the right tool for one session. It cannot do this job: +`GET /v1/sessions/{id}/timeline` answers per-tool latency today by pairing on tool-use id +(#1810 §9.1). It is the right tool for one session. It cannot do this job: + +It is also **content-light, not content-free**, which matters when comparing it with this +record: `Tool.Preview` carries up to 80 runes of argument text — a command line, a file path, +a query (`agents.go:487-515`) — and `Tool.ID` is the tool-use id itself +(`timeline.go:128-138`). #1810 §9.1 states the same caveat in its own words ("no prompts", +not "safe to share"). A `toolCall` record carries neither (T6, T8). | Need | Why `/timeline` cannot | |---|---| @@ -149,65 +180,124 @@ that it is cheap. §6 is that decision. ## 3. The measurement -Method: `/v1/export` for the record view, `/v1/sessions/{id}/timeline` for the paired view, -over the same four sessions. 268 tool spans, 267 usable (one still running, zero `lost`). +Source: `/v1/export` for the record view, `/v1/sessions/{id}/timeline` for the paired view, +over four sessions of one live install. 348 spans with an end and not `lost`, across **323 +distinct windows** (§3.4). -### 3.1 Per-tool timing +**Three method rules, each of which moves a figure below.** The first draft of this spec +stated none of them, and §3.2 and §3.3 were wrong as a result: -| Tool | n | p50 | p95 | total | share of tool time | -|---|---|---|---|---|---| -| `Bash` | 210 | 473 ms | 7,561 ms | 598.9 s | 97.4% | -| `AskUserQuestion` | 1 | 12,714 ms | — | 12.7 s | 2.1% | -| `Agent` | 1 | 1,093 ms | — | 1.1 s | 0.2% | -| `Read` | 17 | 41 ms | 194 ms | 1.0 s | 0.2% | -| `Edit` | 29 | 24 ms | 66 ms | 0.8 s | 0.1% | -| `Write` | 9 | 16 ms | 224 ms | 0.4 s | 0.1% | +1. **A shared window is counted once**, not once per call in its batch. Summing per span + double-counts a batch's window. Measured: 828.2 s naive against **812.9 s** counted once — + a 1.9% overcount, smaller than §3.4's 14% span share suggests, because batched calls are + short. +2. **An interactive tool is excluded from a tool-time total.** `AskUserQuestion` is a person + answering, not work: 12.7 s on one call here, and 62.6 s p50 over 68 calls on #1810's + larger corpus. Leaving it in prices human think time as tool time. +3. **`durationMs` is a client round trip, including waits** (§3.5). No rule removes that; it + is stated so a reader knows what the figures contain. -Errors: `Bash` 11, `Read` 3. +### 3.1 Per-tool timing -**`Bash` is 78% of calls and 97.4% of tool time.** One tool dominates, which is why the name -alone nearly answers the workload question — and why the *duration distribution*, not the -name, is the signal worth carrying. +Every span, including batch members, since a per-tool p50 wants each call: -### 3.2 Inference against tools +| Tool | n | p50 | p95 | total | +|---|---|---|---|---| +| `Bash` | 272 | 420 ms | 7,860 ms | 811.1 s | +| `AskUserQuestion` | 1 | 12,714 ms | — | 12.7 s | +| `Edit` | 39 | 35 ms | 76 ms | 1.4 s | +| `Agent` | 1 | 1,093 ms | — | 1.1 s | +| `Read` | 18 | 41 ms | 194 ms | 1.0 s | +| `Write` | 17 | 17 ms | 239 ms | 0.9 s | -| | calls | total | p50 | -|---|---|---|---| -| Model | 307 | 2,510.7 s | 5,473 ms | -| Tool | 267 | 614.8 s | 243 ms | +**`Bash` is 78% of calls and over 97% of tool time.** One tool dominates, which is why the +name alone nearly answers the workload question — and why the *duration distribution*, not +the name, is the signal worth carrying. -**4.1 : 1** — inference is 80.3% of measured work. For sandbox capacity this is the headline: -the agent waits on the model four times longer than it runs tools. +### 3.2 Inference against tools -### 3.3 Weight tiers, on E15's cuts (1 s / 10 s) +Windows counted once, `AskUserQuestion` excluded: -| Tier | share of calls | share of work | +| | calls | total | |---|---|---| -| light (<1 s) | 76.0% | 8.3% | -| medium (1–10 s) | 22.1% | 36.3% | -| **heavy (>10 s)** | **1.9%** | **55.4%** | +| Model | 400 | 3,371.6 s | +| Tool | 322 | 800.2 s | -The heaviest 10% of calls carry **77.7%** of tool time; the heaviest single call was a 145.2 s -`Bash`. This reproduces E15's concentration finding (7.4% of calls / 86.5% of work) on our own -traffic, independently. It is the number density planning turns on, and it is the reason a -mean is useless here: a design that reports averages hides the tail that sizes the sandbox. +**4.2 : 1** — inference is 80.8% of measured work. For sandbox capacity this is the headline: +the agent waits on the model four times longer than it runs tools. The ratio is robust to the +method rules — it was 4.1 : 1 before them — because both corrections are small against a gap +this wide. -### 3.4 Parallel calls share one window — the finding that shapes the record +### 3.3 Duration bands (<1 s / 1–10 s / >10 s) -**42 of 267 spans (15.7%) carry an identical start and end**: calls the model issued together. -The timeline gives a batch one merged interval, not a duration each. +Windows counted once, `AskUserQuestion` excluded. Band edges: short `< 1000 ms`, medium +`1000–10000 ms` inclusive, long `> 10000 ms`. -So a naive per-call `durationMs` is wrong for one call in six, and wrong in the direction that -inflates: charging every member of a batch the whole window's time overcounts tool time and -drags the heavy tier upward. This is the same class of error as the 272x tool-result overcount -#1810 documents, and the record must make it explicit rather than leave a consumer to discover -it. - -### 3.5 The two views reconcile - -Summed `toolCallCount` over the same sessions is **273** against **267** paired spans — 2.2% -apart, consistent with calls in flight at the archive boundary. A record derived archive-side -will therefore not contradict `/timeline`. +| Band | share of calls | share of work | +|---|---|---| +| short | 78.9% | 7.4% | +| medium | 18.9% | 31.0% | +| **long** | **2.2%** | **61.7%** | + +The heaviest 10% of calls carry **82.4%** of tool time; the heaviest single call was a 145.2 s +`Bash`. It is the reason a mean is useless here: a design that reports averages hides the tail +that sizes the sandbox. + +These are **per-call duration bands**, named short/medium/long. They are not the +light/medium/heavy *weight tiers* of the TraceLab session-density analysis +(`moca-experiments/experiments/e15-session-density`, cuts at 1 s and 10 s on a call's +`work_ms`; `analysis/tracelab/analysis/tiers.py`), which elsewhere also names terciles of a +task's total time. Same cuts, different unit of analysis, so the tables are not comparable +and the names are kept apart deliberately. That analysis found 7.4% of calls carrying 86.5% +of work; the concentration here is the same shape, independently measured, and nothing more +than that should be read into the agreement. + +### 3.4 A batch of calls shares one window — the finding that shapes the record + +**25 batches, covering 50 of 348 spans (14%), carry an identical start and end.** The timeline +gives such a batch one merged interval, not a duration each. + +A batch is **structural, not inferred from the timestamps**: every call of one response starts +at that response's end (`agents.go:234` sets `Start: end` for each), so they share a start by +construction. They share an *end* when their results arrive in one later request. So the +grouping key is the asking response's seq, and `durationShared` says the results also returned +together — which is why §5 carries both rather than a timestamp match. + +**A shared window is request-boundary granularity, not proof the calls ran in parallel.** Two +calls whose results land in the same request are indistinguishable from each other in time, +whether they ran together, back to back, or overlapped partly. The record therefore says "this +duration is a window shared with n calls", never "these ran concurrently". + +Effect on a total: 828.2 s naive against 812.9 s with each window counted once, a **1.9%** +overcount. Small, because batched calls are short — but it is a bias, not noise, and it always +inflates. §3's method rule 1 applies it. + +### 3.5 What `durationMs` measures, and what it contains + +A span runs from the asking response's end to the arrival of the result, so **it is a client +round trip, not an execution time.** Everything the client did in between is inside it: + +- the tool actually running; +- a **permission prompt and the person's answer** — the single largest contaminant, and the + reason `AskUserQuestion` reads 12,714 ms here (T26, §7.1a); +- any client-side queueing, retry or transport leg before and after the tool ran. + +So §3.1's figures are round trips, §3.2's ratio counts approval waits as tool time for any +tool that prompts, and no rule in §3 removes that. Method rule 2 excludes the one tool whose +*whole* duration is a person; it cannot exclude a permission prompt on a `Bash`. T26 states +the limit on the record and #1924 is the fix. + +### 3.6 The two views reconcile + +Summed `toolCallCount` over the same sessions at the time of the first measurement was **273** +against **267** paired spans — 2.2% apart, consistent with calls in flight at the archive +boundary. A record derived archive-side will therefore not contradict `/timeline`. + +A note on the two span counts in this section: the install is live, so a later re-measurement +read 348 spans where the first read 267. Every figure in §3.1–§3.4 comes from the **later** +snapshot and is internally consistent; §3.6's reconciliation is from the earlier one. The +ratio and the concentration are stable across both, which is the claim being made — not the +absolute counts. --- @@ -218,7 +308,7 @@ will therefore not contradict `/timeline`. T1. The export answers per-tool call counts and duration distributions across every archived session, without a second endpoint and without a hand-written join. T2. The export answers the inference-time-to-tool-time ratio per session and in aggregate. -T3. A consumer can compute the §3.3 weight tiers from the export alone. +T3. A consumer can compute the §3.3 duration bands from the export alone. T4. A consumer can separate built-in tools from MCP tools without parsing a name. T5. A tool call's record names the session, the agent role and the client, so a fleet view can split by any of them. @@ -233,34 +323,49 @@ T8. A `toolCall` record carries **no tool-use id**. The id is the pairing key, u proxy and discarded: the export ships the measurement the key yields, never the key. This is what keeps a `toolCall` record unjoinable back to a conversation. T9. `TestRecord_CarriesNoContent` is extended to cover the `toolCall` record, planting a -secret in every content field the pairing reads — tool arguments, result content and the -preview fields `/timeline` builds — and failing if any reaches the encoding. -T10. A built-in tool name is exported verbatim: it comes from a closed vocabulary -(`Bash`, `Read`, `Edit`, …). +secret in every content field the pairing reads and failing if any reaches the encoding. The +pairing reads more content than the record carries, which is exactly why the test must cover +it: the closing cases scan **message text** for task notifications (`agents.go:147-190`) and +**tool result text** for background-job acknowledgements, so the planted secrets must include +message content and result content, not only tool arguments. +T10. A tool name is exported **verbatim**, whatever its source. The built-in vocabulary is +closed per client but **not shared between clients** — one agent's `Bash` and `Read` are +another's `bash` and `read` — so there is no single list to normalise against, and +normalising would lose which spelling the client actually sent. `toolSource` (T4) is decided +by the client's own list; the name itself is never rewritten. T11. An **MCP tool name is operator-defined and namespaced**, so it can carry a vendor relationship, an internal system name or a project codename — `mcp__acme-billing-prod__refund` -is the worked example. It is therefore **not** treated as a built-in name. **Open:** gate it -off by default, or export a stable hash so distinct MCP tools stay countable without being -readable. Hashing is the author's recommendation: it preserves T1 and T4 for MCP traffic, -which gating does not. +is the worked example. It is therefore **not** treated as a built-in name. It is exported as a +**keyed** hash with a typed prefix (`hmac-sha256:` + hex), under an install-local key. A plain +digest is not enough: the namespace makes the input guessable, so an unkeyed hash of a +known vendor name is reversible by trying candidates. The cost is that counts do not join +across installs, which is the right trade for a name that may identify a customer. Hashing +happens **before** T12's cut, so the cut never splits a digest. T12. A tool name is bounded on the way out by the same `exportLabel` every wire-sourced -string in a record passes: control runes removed, then truncated to `maxLabelLen` (96) on a -rune boundary. +string in a record passes: control runes removed, then truncated to `maxLabelLen` on a rune +boundary. `maxLabelLen` is **96 bytes**, not runes (`record.go:352-355`), so a multi-byte name +is cut sooner than its rune count suggests. ### 4.3 Data rules T13. A `toolCall` record is **flat scalars only**, as #1810 R2 requires of a record. No array, no nested object. This is what keeps one line one thing and keeps `jq` usable. -T14. `resultBytes` is the size of the tool's result, as the effort proxy T7 permits instead of -content. Absent means not measured, never zero — the export's zero-means-absent rule. -T15. **A batch of parallel calls is marked as such.** A record whose duration comes from a -window shared with other calls carries `durationShared: true`, the batch's size in -`parallelN`, and a `parallelGroup` key that is stable within the session. A consumer can then -divide, take the window once per batch, or exclude it — explicitly, by its own choice. -T16. A `lost` span — one that never got a result but provably stopped — is marked `lost: true` -and **carries no duration**, rather than the zero-length duration `/timeline` draws it with. A -zero would pass a consumer's `durationMs > 0` filter as a 0 ms call and drag a p50 toward -zero. +T14. `resultBytes` is the size of the tool's result, and `inputBytes` the size of its +arguments, as the effort proxy T7 permits instead of content. **Both are byte counts, and the +names say so** — a size is not a character count and the two differ for any non-ASCII payload. +Absent means not measured, never zero — the export's zero-means-absent rule. A size is not +content: it is the same concession the trajectory schema's `sessionToolCall` already makes +with `command_chars` and `result_preview_chars`. +T15. **A batch of calls is marked as such.** A record whose duration comes from a window +shared with the other calls of its asking response carries `durationShared: true` and the +batch's size in `parallelN`. The batch's identity is `callSeq` (§3.4), so no separate group key +exists. A consumer can then divide, take the window once per batch, or exclude it — explicitly, +by its own choice. The marker says the window is shared, **never** that the calls ran +concurrently. +T16. A call with no result **carries no duration**, rather than the zero-length span +`/timeline` draws it with. A zero would pass a consumer's `durationMs > 0` filter as a 0 ms +call and drag a p50 toward zero. Which no-result case it was is `endReason` (T31), so the +record carries no separate `lost` flag. T17. A span still open when the walk reads it is not exported. An unfinished call has no duration to report, and a record that appears with one later would make the export non-reproducible. @@ -270,22 +375,41 @@ T18. Durations are milliseconds, truncated, matching `Record`. The three quantities a capacity plan needs, and the one asymmetry that makes the split sound: **model time is measured, not inferred.** `durationMs` on an inference response *is* the -model's time. Only the leftover gap has to be classified. +model's time. Only the leftover interval has to be classified. + +**One named interval, defined once.** The first draft of this section defined `gapMs` from the +previous tool call's end to this call's start, and classified a different interval in T25 — +which cannot be checked, because a call starts when its asking response ends, so that first +interval always contains the asking response's own inference. Corrected: + +> **`gapMs` is the idle interval BEFORE this call's asking response: from the end of the +> previous response of the same agent, to the start of the request that produced this call's +> asking response.** It therefore contains **no** `modelMs` and no tool execution of this +> call. It is the time the agent was neither talking to the model nor running this tool. T23. A `toolCall` record carries `modelMs`: the measured duration of the inference response -that asked for this call. It is the turn's model time, copied, not derived — so a consumer -gets the inference/tool ratio per call without joining back to the event record. -T24. A `toolCall` record carries `gapMs`: the time from the previous tool call's end to this -call's start, within the same agent. Absent on an agent's first call, where there is no -previous end — absent, never zero. +that asked for this call. Copied from that response, not derived — so a consumer gets the +inference/tool ratio per call without joining back to the event record. It sits **after** +`gapMs` and **before** `durationMs` on the timeline, and the three never overlap. +T24. A `toolCall` record carries `gapMs`, the interval defined above. Absent on an agent's +first turn, where there is no previous response — absent, never zero. T25. A `toolCall` record carries `gapKind`, a **closed vocabulary** exported by naming its -values, never by sanitising what arrives: +values, never by sanitising what arrives. It classifies the `gapMs` interval by **the previous +response of the same agent** — the one whose end begins the interval: -| `gapKind` | The preceding response | What the gap is | +| `gapKind` | That previous response | What the interval was | |---|---|---| | `user` | `toolCallCount == 0` | The model stopped and waited for a human. Nothing else can be happening. | -| `tool` | `toolCallCount > 0` | Tool execution, plus any approval wait (T26). | -| `inference` | a model call landed inside the gap | Model time, already in `modelMs`. Not idle. | +| `tool` | `toolCallCount > 0` | The previous turn's tool calls were running. Their own durations are on their own records. | +| `first` | there is none | The agent's first turn. `gapMs` is absent. | + +A fourth value is **not** needed for "a model call landed inside the gap": by the definition +above the interval ends at the next request, so no inference can be inside it. That was an +artifact of the old definition. + +**A consumer with the event records can derive all three fields**, so these definitions are +the deliverable and the fields are the convenience. A definition that cannot be checked +against the data is worse than no field. T26. **An approval wait is inside `tool` and cannot be separated from execution.** When a user sits on a permission prompt, that time lands in the tool gap and is indistinguishable from the @@ -316,6 +440,42 @@ T20. `toolCall` records are **opt-in**, per #1810 R49's rule for Tier 2: a plain and `--with-tool-calls` on the command, or a single Tier-2 switch covering every §7.2 item. T21. Adding the record does not bump `Record`'s `schema`. A new record *kind* on the stream is additive; `sessionexport.SchemaVersion` bumps only on a removal or a meaning change. +T28. An event record carries an explicit `kind: "event"` once `toolCall` records exist, so a +consumer can select on `kind` in both directions rather than on the absence of a key. An +absent `kind` still reads as an event record, so an older file stays readable. + +### 4.4a Placing and ordering a call + +T29. A `toolCall` record carries `startedAt`, the call's start — the asking response's end. It +is **always present**, including on a call that never got a result, so every record can be +placed on a timeline. The end, where there is one, is `startedAt + durationMs`. The record +carries no separate `seq`: that collided with the event record's own `seq` and was undefined +for an unresolved call. +T30. A `toolCall` record carries `callIndex`, the call's position among the calls its asking +response made, from 0, in the order the model emitted them. A batch is otherwise unorderable. +T31. A `toolCall` record carries `endReason`, a closed vocabulary naming how the call ended. +`durationMs` is present only for the reasons that have an end: + +| `endReason` | Duration? | Meaning | +|---|---|---| +| `result` | yes | Its result arrived. | +| `ended_run` | no | By design no result: the call ends the run or hands control back (`SubagentHandback`, `StructuredOutput`, `end_subtask`). | +| `abandoned` | no | The agent moved on without the result (`lastTurn`). | +| `stopped` | no | The run was stopped while the call was open (a ``). | +| `lost` | no | Open with no reason established. | + +These are the timeline's own closing cases (`agents.go:273-295`), named rather than collapsed +into one `lost` flag: "the agent moved on" and "the call ends the run by design" are different +facts, and a consumer filtering latency wants to exclude both but count them apart. +T32. **Background commands are out of scope for this record, and the spec says so rather than +leaving it ambiguous.** A `Bash` call that backgrounds a job ends at the acknowledgement, and +the job's own run is a separate `timeline.Job` (`agents.go:133-137`). So this record undercounts +a backgrounded command's wall time. Carrying jobs needs a second record kind and its own +pairing (`backgroundRe`, the `` match), which this spec does not specify. +T33. A `toolCall` record whose asking response is below a `since` floor is **not exported**. +`Walk` stops at the first event below the floor (`walk.go:243-245`), so such a call has no +start to report, and T29 requires one. A result inside the window whose call is outside it +therefore yields no record, rather than a record with a missing start. ### 4.5 Documentation @@ -328,33 +488,49 @@ list. The doc fix that precedes this spec already states them for `/timeline`. ## 5. The record ```json -{"schema":1,"kind":"toolCall","sessionId":"45fa31b2-…","seq":2081, - "at":"2026-11-05T15:52:26.102885Z","tool":"Bash","toolSource":"builtin", - "callSeq":2077,"resultSeq":2081,"durationMs":473,"isError":false, - "resultBytes":1432,"agentRole":"main","client":"claude-code/2.1.270", +{"schema":1,"kind":"toolCall","sessionId":"45fa31b2-…", + "startedAt":"2026-11-05T15:52:25.629885Z","tool":"Bash","toolSource":"builtin", + "callSeq":2077,"callIndex":0,"resultSeq":2081, + "durationMs":473,"endReason":"result","isError":false,"resultBytes":1432, + "agentRole":"main","client":"claude-code/2.1.270", "modelMs":4279,"gapMs":812,"gapKind":"tool", - "parallelGroup":"2077#0","parallelN":3,"durationShared":true} + "parallelN":3,"durationShared":true} ``` +Four changes from the first draft, all so a record can be placed and ordered on its own: + +- **`startedAt` replaces `at`.** The first draft carried the *end*, so a `lost` call — which + has no duration — could not be placed on a timeline at all. A start exists for every call + (the asking response's end), so it is always present; the end is `startedAt + durationMs` + when there is one. +- **`seq` is gone.** It copied the result event's seq, which collides with the event record's + own `seq` and is undefined for a `lost` call. `callSeq` and `resultSeq` carry the two real + anchors. +- **`callIndex` added.** The call's position among the calls its response asked for, from 0, + in the order the model asked. Without it a batch cannot be put back in order. +- **`parallelGroup` is gone**, replaced by `callSeq` + `callIndex`. The group *is* the asking + response, so `callSeq` already identifies it; a separate composite string was a second + spelling of the same fact. + | Field | Why it is here | |---|---| -| `kind` | T19. The discriminator. Always `"toolCall"` on this record. | +| `kind` | T19. The discriminator. Always `"toolCall"` here; an event record carries `"event"` (T28). | | `sessionId` | T5. The session, as the client asserted it. Not a unit of work until #1997. | -| `seq` | Orders the record within its session, as on an event record. The result event's seq. | -| `at` | When the call ended, so a record sorts on one time base with event records. | -| `tool` | T10/T11. The name, bounded by T12. | +| `startedAt` | T29. When the call started — the asking response's end. **Always present**, so a `lost` call can still be placed. | +| `tool` | T10/T11. The name, verbatim, bounded by T12. | | `toolSource` | T4. `builtin`, `mcp` or `unknown` — a closed vocabulary, exported by naming the values. | -| `callSeq` / `resultSeq` | The response that asked and the request that carried the result. Lets a consumer rejoin a `toolCall` to the event records either side **without a tool-use id** (T8). | -| `durationMs` | T18. The paired duration. Absent on a `lost` record (T16). | +| `callSeq` / `resultSeq` | The response that asked and the request that carried the result. Lets a consumer rejoin a `toolCall` to the event records either side **without a tool-use id** (T8). `resultSeq` is absent when nothing carried a result. | +| `callIndex` | T30. The call's position among its response's calls, from 0, in the order the model asked. | +| `durationMs` | T18. The paired duration. Absent when `endReason` says there was no result (T16, T31). | +| `endReason` | T31. Closed vocabulary: how the call ended. `lost` derives from it, so no separate flag. | | `isError` | Whether the result was an error. The tool failure rate, per tool. | -| `resultBytes` | T14. Effort proxy, never content. | +| `resultBytes` | T14. Effort proxy, never content. Bytes, as the name says. | | `modelMs` | T23. The asking response's measured duration — model time, per call, no join needed. | -| `gapMs` | T24. Since the previous tool call ended, within the agent. Absent on the first call. | -| `gapKind` | T25. `user`, `tool` or `inference` — what the gap was. The field that turns idle time into a duty cycle. | +| `gapMs` | T24. The idle interval before the asking response. Contains no `modelMs`. | +| `gapKind` | T25. `user`, `tool` or `first` — what that interval was. The field that turns idle time into a duty cycle. | | `agentRole` | T5. `main` or `subagent`, so subagent tool traffic reads apart. | | `client` | T5. Cost-attribution and display key. **Never an authorization subject.** | -| `parallelGroup` / `parallelN` / `durationShared` | T15. The §3.4 finding, made explicit. | -| `lost` | T16. Set, with no `durationMs`, on a call that stopped without a result. | +| `parallelN` / `durationShared` | T15. The §3.4 finding. The batch is `callSeq`; these say how many and whether the window is shared. | Not here, deliberately: the tool-use id (T8), arguments (T6), result content (T7), the `/timeline` `preview` (it is built from argument keys and is explicitly "no prompts, not safe @@ -383,7 +559,9 @@ The walk calls the timeline builder per session and projects its spans. - **For:** smallest diff, no new pairing code. - **Against:** inherits the 50,000-event cap, which bounds the export by a timeline limit that - has nothing to do with it; and it builds turns, hosts and agents the export discards. + has nothing to do with it; and it builds turns, hosts and agents the export discards. The cap + lives in the **HTTP handler** (`sessionapi/timeline.go:25`), not the builder, so an in-process + caller could avoid it — but then the export depends on a limit staying where it is. ### 6.3 Option C — a second, bounded pass in `sessionexport` @@ -413,12 +591,31 @@ should match. **Open:** the bound and what happens at it. ## 7. What this does not answer -7.1 **Local tool calls are invisible.** `durationMs` here is proxy-observed, so it covers -tools that generate proxied traffic. An `Edit`, `Read` or `Bash` touching nothing on the -network appears only as gap time — #1810 measures that at 5,653 calls and 5.3 h. For sandbox -sizing this may be *correct*, since those calls cost no sandbox time either, but **it -undercounts call volume and any table built from it says so.** Closing the gap needs #1924 -(Claude Code telemetry), not this spec. +7.1 **The proxy sees the gap around a tool, never inside it.** A correction to the first draft +of this spec, which claimed local tool calls are invisible. They are not: a span runs from the +asking response's end to the next request carrying the result (`agents.go:234`, `:280-287`), +and **both boundaries are model traffic**, so a tool that touches nothing on the network still +gets a span. §3.1's `Edit` at p50 35 ms is such a call. The earlier claim confused this record +with #1810 §9's gap classification, where "no proxied traffic landed inside the gap" is how a +*local* call is inferred — a different measurement answering a different question. + +What the proxy genuinely cannot see is **what happened inside the span**: whether the time went +to the tool running, to a permission prompt, to client-side queueing, or to a transport leg +(§3.5). So the record measures a client round trip per call, with full coverage of calls and +no visibility into their composition. + +7.1b **Sandbox density needs a sandbox-side record; this one sizes the model leg.** The +distinction §3.2's ratio invites a reader to miss. Where an agent's tools run on the +developer's own machine, a tool call costs no sandbox time and the ratio is the whole story. +Where they run in a **remote sandbox**, those calls *are* the sandbox's work, and this record +sees only the round trip around them — which includes a transport leg the sandbox's own timing +excludes, so the two disagree and neither is wrong. A sandbox host is bounded by its concurrent +commands and the memory each holds, and **no field here measures either**. + +So: this record sizes the model leg — tokens, latency, concurrency of model calls — and bounds +the tool leg from outside. A sandbox capacity plan wants it **joined to a sandbox-side record**, +for which the record already carries the keys: `sessionId` plus UTC `startedAt` and +`durationMs`. Specifying that record is not in scope here. 7.1a **An approval wait reads as execution.** T26: a permission prompt the user sits on lands inside a `tool` gap, indistinguishable from the command running. Nobody has quantified it @@ -449,8 +646,8 @@ which user instruction. | #1924 | Claude Code telemetry would supply the tool times the proxy cannot see (§7.1) **and the permission waits T26 cannot separate from execution (§7.1a)**. It is the fix for both of this spec's largest error sources, so the two should be planned together even though this one ships first. | | #1921 / #1925 | Redaction and the content export. Where tool arguments and result content go, per T0 — a different privacy posture, not a flag on this record. | | #1997 | `sessionIncarnation`. Needed before per-session aggregates mean anything (§7.3). | -| #505 | An OTLP emitter would be the third consumer of the §6 pairing, and the reason to prefer Option A. | -| rossoctl/trajectory-schema | A `toolCall` record maps closely to that schema's `$defs.sessionToolCall`, which already records a tool call as sizes rather than content (`command_chars`, `result_preview_chars`). Worth aligning field names where the semantics match. | +| rossoctl/trajectory-schema | **A consumer exists.** That schema added an observed-session kind (`trajectory-schema/session/1`, v2.1) and `tools/from-cortex.py`, which converts a `/v1/export` stream of event and `toolCall` records into it and into OTLP/JSON. Its field-by-field mapping is what surfaced T29 (a start on every record) and T30 (`callIndex`): without them a lost call cannot be placed and a batch cannot be ordered. Both are now required. The converter reports `gapKind` as unmapped and derives it from the event records instead, which is the right division — §4.3a says the definitions are the deliverable. | +| #505 | OpenTelemetry's GenAI conventions name a tool span `execute_tool {gen_ai.tool.name}` with `gen_ai.operation.name=execute_tool`, and `gen_ai.tool.call.id` is only **conditionally** required — so T8's omission of the tool-use id stays spec-compliant. An OTLP emitter would also be the third consumer of §6's pairing. | --- @@ -474,13 +671,23 @@ its frequency — but the *rate* is not a constant to design against. ## 10. Open items, collected +### 10.1 Settled in review (#2009), and now normative + +| § | Decision | +|---|---| +| T11 | **Keyed** hash (`hmac-sha256:` + hex, install-local key), not a plain digest: a namespaced name is guessable, so an unkeyed digest is reversible. Cross-install counts are given up deliberately. | +| T20 | **Per item** (`?include=toolCalls`). One Tier-2 switch would opt a consumer into items it never asked for. | +| §6 | **Option A**, scoped to tool spans and their closing inputs. | +| §6.4 | A fixed cap on open calls per session, and **a per-session marker record** when it is hit — matching the export's degrade-loudly posture. | +| T25 | **No fourth `gapKind`.** The redefined interval ends at the next request, so no inference can fall inside it; the question was an artifact of the first draft's definition. | + +### 10.2 Still open + | § | Open | |---|---| -| T11 | MCP tool names: gate off by default, or export a stable hash. Author recommends hashing. | -| T20 | The opt-in switch: per-item (`?include=toolCalls`) or one Tier-2 switch. | -| §6 | Which of the three pairing options. Author leans A. | -| §6.4 | The per-session pairing bound, and the behaviour when it is hit. | -| T25 | Whether `gapKind` needs a fourth value for a gap that spans both a model call and tool execution, or whether `inference` winning is the right precedence. | -| §7.1a | Whether to quantify the approval-wait error against local transcripts before publishing any density figure. Author's view: yes, and it is cheap. | +| §7.1a | Whether to quantify the approval-wait error against local Claude Code transcripts before publishing any density figure. Author's view: yes, and it is cheap. | +| §7.1b | Whether a sandbox-side record is specified now or later, given that sandbox density was part of this work's motivation and this record cannot answer it alone. | +| T32 | Whether background jobs get a record kind of their own, or stay out. As written their wall time is undercounted. | +| §3.3 | Whether the band names (short/medium/long) should instead match the TraceLab tier names, accepting the collision. Author's view: keep them apart — one measures a call, the other a task. Worth confirming with the analysis's author. | Assisted-By: Claude (Anthropic AI) From 292a96606430af89720def9fe668e69eb956f691 Mon Sep 17 00:00:00 2001 From: Mariusz Sabath Date: Fri, 20 Nov 2026 12:34:00 -0500 Subject: [PATCH 05/12] Docs: Say where a tool runs is unknowable, and mark a backgrounded call Addresses @webchang's three follow-ups on #2009. 7.1b was too narrow. The question is not whether a remote sandbox is involved, it is WHERE a tool runs, and a model proxy cannot see that at all: a built-in Bash, Read or Edit may run on the developer's machine or in a sandbox the agent sends its tools to, and an MCP tool may be served over stdio locally or over HTTP remotely. So the record measures each call as the AGENT saw it and says nothing about the work's location or its cost there. Each candidate host also measures the call differently -- a sandbox's own timing excludes the transport leg this record includes -- so the two disagree and neither is wrong. toolSource says what kind of tool it is, never where it ran, and the proxy must not infer a site from it. Cortex therefore specifies no sandbox-side record. T34 guarantees the join keys instead -- sessionId, startedAt, callSeq + callIndex, agentRole -- so a consumer holding a host's own records joins them on its side. T32 changes from "out of scope" to marked. A backgrounded call now ends background with durationMs measuring the acknowledgement only, so a count of those calls says how much tool time is missing instead of it hiding inside result. A job record is still left for later, with the warning that a backgrounded command overlaps later calls, so summed per-command time can exceed a session's wall-clock tool time. T3 and 3.3: banding is an analyzer's policy, not a property of the record. The export carries each call's raw durationMs and startedAt and names no bands, which also takes the short/medium/long naming question out of this spec. Assisted-By: Claude (Anthropic AI) Signed-off-by: Mariusz Sabath --- ...26-11-19-tool-call-export-record-design.md | 100 ++++++++++++------ 1 file changed, 70 insertions(+), 30 deletions(-) diff --git a/docs/superpowers/specs/2026-11-19-tool-call-export-record-design.md b/docs/superpowers/specs/2026-11-19-tool-call-export-record-design.md index c9805acfa..40559c297 100644 --- a/docs/superpowers/specs/2026-11-19-tool-call-export-record-design.md +++ b/docs/superpowers/specs/2026-11-19-tool-call-export-record-design.md @@ -12,13 +12,13 @@ This one narrows its §7.2 `toolCallNames` item and supersedes the shape propose Items marked **Open** must be decided before the work they belong to begins. Nothing here is merged. -**34 numbered requirements T0–T33**, phrased as plain declaratives: a requirement says what +**35 numbered requirements T0–T34**, phrased as plain declaratives: a requirement says what the system does once built. The numbering is stable so it can be cited in review and in the PRs that implement it, and is therefore in the order the requirements were ADDED, not document -order: T0 sits in §2.0 because it bounds the whole spec, T23–T27 in §4.3a, and T28–T33 in §4.4 -and §4.4a. +order: T0 sits in §2.0 because it bounds the whole spec, T23–T27 in §4.3a, T28–T33 in §4.4 +and §4.4a, and T34 in §7.1b. -Revised twice on 2026-11-19. +Revised twice on 2026-11-19 and once on 2026-11-20. **First revision**, after review raised two things the first draft buried. §2.0 is new: every field here is **already collected and most of it is already on disk**, so this is a projection @@ -44,10 +44,19 @@ and against a converter written for it. Four measurement claims were wrong and a classified a different one. Now one named interval, with `modelMs` outside it by construction and the fourth `gapKind` dropped as an artifact. -Record shape changed with it: `startedAt` replaces `at` and is always present (T29), `seq` and +**Third revision**, after @webchang's follow-ups on #2009. §7.1b is rewritten: the question is +**where a tool runs**, which the proxy cannot see at all — a built-in tool may run locally or in +a remote sandbox, and an MCP tool may be served over stdio or HTTP, so `toolSource` says what +kind of tool it is and never where it ran. Cortex therefore specifies no sandbox-side record and +guarantees stable join keys instead (T34). T32 changes from "out of scope" to **marked**: a +backgrounded call gets `endReason: "background"`, so the undercount is visible per call rather +than hidden inside `result`. T3 and §3.3 now say banding is an **analyzer's** policy — the +export carries raw timings and names no bands — which also removes the naming question from +this spec. + +Record shape changed in the second revision: `startedAt` replaces `at` and is always present (T29), `seq` and `parallelGroup` are gone, `callIndex` (T30) and a closed `endReason` (T31) are added. T28 gives -event records an explicit `kind`. T32 says background jobs are out of scope rather than leaving -it ambiguous; T33 says a call below a `since` floor is dropped. §10 splits into what the review +event records an explicit `kind`. T33 says a call below a `since` floor is dropped. §10 splits into what the review settled and what is still open. Written after measuring, not before. §3 is the measurement; every design choice in §5 and §6 @@ -62,8 +71,8 @@ traces to a number in it. The measurement corpus is one developer's 30-day lapto 1. Answer "which tools does our community run, and what do they cost in time" from the export alone, across every session and beyond the archive's 30-day retention. 2. Answer "how does inference time compare with tool time" as a ratio a capacity plan can - use — for the **model leg**. Sizing a remote sandbox host needs a sandbox-side record - joined to this one; see §7.1b before using these figures for that. + use — **as the agent saw it**. The record cannot say where a tool ran or what it cost + there; see §7.1b before using these figures to size any host. 3. Keep the export content-free by construction, as #1810 §3.2 requires: no tool arguments, no tool results, no prompts. @@ -243,7 +252,11 @@ The heaviest 10% of calls carry **82.4%** of tool time; the heaviest single call `Bash`. It is the reason a mean is useless here: a design that reports averages hides the tail that sizes the sandbox. -These are **per-call duration bands**, named short/medium/long. They are not the +**These cuts and names are this measurement's choice, not the record's.** T3: the export +carries raw `durationMs`, so banding is an analyzer's policy. Nothing downstream should treat +short/medium/long as a Cortex vocabulary. + +They are also not the light/medium/heavy *weight tiers* of the TraceLab session-density analysis (`moca-experiments/experiments/e15-session-density`, cuts at 1 s and 10 s on a call's `work_ms`; `analysis/tracelab/analysis/tiers.py`), which elsewhere also names terciles of a @@ -308,7 +321,9 @@ absolute counts. T1. The export answers per-tool call counts and duration distributions across every archived session, without a second endpoint and without a hand-written join. T2. The export answers the inference-time-to-tool-time ratio per session and in aggregate. -T3. A consumer can compute the §3.3 duration bands from the export alone. +T3. The export carries each call's **raw** timing — `durationMs` and `startedAt` — so any +analyzer can apply its own duration bands or weight tiers. Band names and cut points are an +analyzer's policy, not a property of the record, and the record defines none. T4. A consumer can separate built-in tools from MCP tools without parsing a name. T5. A tool call's record names the session, the agent role and the client, so a fleet view can split by any of them. @@ -459,6 +474,7 @@ T31. A `toolCall` record carries `endReason`, a closed vocabulary naming how the | `endReason` | Duration? | Meaning | |---|---|---| | `result` | yes | Its result arrived. | +| `background` | yes, **to the acknowledgement only** | The result only acknowledged a job that keeps running (`backgroundRe`, `agents.go:37`). The command's real duration is longer and is not here. | | `ended_run` | no | By design no result: the call ends the run or hands control back (`SubagentHandback`, `StructuredOutput`, `end_subtask`). | | `abandoned` | no | The agent moved on without the result (`lastTurn`). | | `stopped` | no | The run was stopped while the call was open (a ``). | @@ -467,11 +483,18 @@ T31. A `toolCall` record carries `endReason`, a closed vocabulary naming how the These are the timeline's own closing cases (`agents.go:273-295`), named rather than collapsed into one `lost` flag: "the agent moved on" and "the call ends the run by design" are different facts, and a consumer filtering latency wants to exclude both but count them apart. -T32. **Background commands are out of scope for this record, and the spec says so rather than -leaving it ambiguous.** A `Bash` call that backgrounds a job ends at the acknowledgement, and -the job's own run is a separate `timeline.Job` (`agents.go:133-137`). So this record undercounts -a backgrounded command's wall time. Carrying jobs needs a second record kind and its own -pairing (`backgroundRe`, the `` match), which this spec does not specify. +T32. **A backgrounded command is MARKED, not silently undercounted.** A `Bash` call that starts +a background job ends at the acknowledgement, and the job's own run is a separate +`timeline.Job` (`agents.go:133-137`), so this record cannot carry the command's real duration. +It therefore sets `endReason: "background"` with `durationMs` measuring only the +acknowledgement. A count of such calls tells a consumer **how much tool time is missing** and +from where, which an undifferentiated `result` would hide. + +A job record is deliberately left for later: its start, its duration or an `endReason` when it +never ended, and its starting call's `callSeq` + `callIndex` (no ids), paired by the +`` match. **Note for whoever specifies it:** a backgrounded command overlaps +later calls, so summed per-command time can exceed a session's wall-clock tool time. A +consumer must not add job time to `durationMs` totals. T33. A `toolCall` record whose asking response is below a `since` floor is **not exported**. `Walk` stops at the first event below the floor (`walk.go:243-245`), so such a call has no start to report, and T29 requires one. A result inside the window whose call is outside it @@ -604,18 +627,30 @@ to the tool running, to a permission prompt, to client-side queueing, or to a tr (§3.5). So the record measures a client round trip per call, with full coverage of calls and no visibility into their composition. -7.1b **Sandbox density needs a sandbox-side record; this one sizes the model leg.** The -distinction §3.2's ratio invites a reader to miss. Where an agent's tools run on the -developer's own machine, a tool call costs no sandbox time and the ratio is the whole story. -Where they run in a **remote sandbox**, those calls *are* the sandbox's work, and this record -sees only the round trip around them — which includes a transport leg the sandbox's own timing -excludes, so the two disagree and neither is wrong. A sandbox host is bounded by its concurrent -commands and the memory each holds, and **no field here measures either**. +7.1b **The record cannot say WHERE a call ran, and therefore what it cost there.** The +question is where a tool runs, not whether a sandbox is involved — and the proxy sees neither: + +- A built-in tool (`Bash`, `Read`, `Edit`) may run on the developer's machine **or** in a + remote sandbox the agent sends its tools to. +- An MCP tool may be served by a **local** server over stdio **or** a **remote** one over HTTP. + +A model proxy sees the model ask for a call and, later, the result arrive. So the record +measures **each call's time as the agent saw it**, and says nothing about where the work +happened. A local call costs the developer's machine; a sandboxed one, the sandbox host; a +remote MCP call, whoever runs that server. Each of those hosts also measures the call +differently — a sandbox's own timing excludes the transport leg this record includes, so the +two disagree and neither is wrong. -So: this record sizes the model leg — tokens, latency, concurrency of model calls — and bounds -the tool leg from outside. A sandbox capacity plan wants it **joined to a sandbox-side record**, -for which the record already carries the keys: `sessionId` plus UTC `startedAt` and -`durationMs`. Specifying that record is not in scope here. +**`toolSource` says what KIND of tool it is, never where it ran.** `builtin` and `mcp` are +properties of the tool, not of its execution site, and the proxy must not infer a site from +them. If a client knows where a call ran, an optional field carrying what the client stated +would be the place for it — never a proxy-side guess. + +T34. **The join keys are stable**, so a consumer holding a host's own records can join on its +side: `sessionId`, `startedAt` (UTC), `callSeq` + `callIndex`, and `agentRole`. Those five are +the contract this record offers a capacity plan; their names and meanings do not change without +a `schema` bump. Cortex specifies no sandbox-side record — a consumer that has one joins it +itself. 7.1a **An approval wait reads as execution.** T26: a permission prompt the user sits on lands inside a `tool` gap, indistinguishable from the command running. Nobody has quantified it @@ -680,14 +715,19 @@ its frequency — but the *rate* is not a constant to design against. | §6 | **Option A**, scoped to tool spans and their closing inputs. | | §6.4 | A fixed cap on open calls per session, and **a per-session marker record** when it is hit — matching the export's degrade-loudly posture. | | T25 | **No fourth `gapKind`.** The redefined interval ends at the next request, so no inference can fall inside it; the question was an artifact of the first draft's definition. | +| T32 | **`endReason: "background"`** now, marking the undercount per call; a job record later. A consumer must not add job time to `durationMs` totals, since a backgrounded command overlaps later calls. | +| §7.1b | **Cortex specifies no sandbox-side record.** It guarantees stable join keys (T34) and a consumer joins its own host's records. | +| §3.3 / T3 | **Banding is the analyzer's policy.** The export carries raw timings; the spec names no bands, so the naming collision is not the export's to settle. | ### 10.2 Still open | § | Open | |---|---| | §7.1a | Whether to quantify the approval-wait error against local Claude Code transcripts before publishing any density figure. Author's view: yes, and it is cheap. | -| §7.1b | Whether a sandbox-side record is specified now or later, given that sandbox density was part of this work's motivation and this record cannot answer it alone. | -| T32 | Whether background jobs get a record kind of their own, or stay out. As written their wall time is undercounted. | -| §3.3 | Whether the band names (short/medium/long) should instead match the TraceLab tier names, accepting the collision. Author's view: keep them apart — one measures a call, the other a task. Worth confirming with the analysis's author. | +| T32 | Whether the job record gets specified in this cycle or a later one. The undercount is marked either way. | +| §7.1b | Whether an optional client-stated execution site is worth a field, for a client that knows where its tools ran. Not a proxy-side inference under any circumstance. | + +Nothing above blocks implementation: §6 Option A, T11's keyed hash, T20's per-item switch and +§6.4's marker record are all settled, and T28–T34 are normative. Assisted-By: Claude (Anthropic AI) From 3471127bedd814cfb149ced693945b55c9f10a32 Mon Sep 17 00:00:00 2001 From: Mariusz Sabath Date: Fri, 20 Nov 2026 19:52:47 -0500 Subject: [PATCH 06/12] Docs: Scope T10 to built-in names, and close the review's loose ends Addresses @huang195's and @clawgenti's reviews on #2009. Both found the same must-fix independently. T10 said a tool name is exported verbatim "whatever its source", which licensed shipping mcp__acme-billing-prod__refund in clear text and disagreed with T11's keyed hash two lines below. It is a privacy rule, so an implementer who reads one requirement and not the next has to land on the safe behaviour: T10 is now about built-in names and defers to T11 for the rest. The widened wording came from answering a different review point about per-client vocabularies; the scope was collateral. The rest were suggestions, all correct: endReason loses `lost`. No case produces it -- a call with none of the four reasons still has End == nil, which means still open, and T17 does not export an open call. A vocabulary entry a producer never emits is a trap for the consumer. The implementation dropped the same constant for the same reason. T35 names the five fields written even at zero. The export's zero-means-absent rule would otherwise drop a real callIndex: 0 -- the first call of every batch, and a T34 join key -- and a duration of 0 on a result clamped to its call's start. T28 is scoped to an opted-in stream. Adding kind to every record of a plain export would make T20's byte-identical promise false. inputBytes was defined in T14 and missing from both the example and the field table. T36 and T37 are new, from @clawgenti: modelMs and gapMs belong to the asking response, so every record of a batch repeats them -- summing modelMs counts a batch's inference parallelN times, in the numerator of T2's ratio, where durationShared does not mark it; and a response that asks for no tool produces no record at all, so an aggregate ratio needs the event records either way. A `tool` gap also overlaps the previous calls' own durations, so adding gap time to tool time double-counts the same wall clock. 2.2 claimed a background job extends a Bash span. It does not: the span ends at the acknowledgement and the job becomes a separate timeline.Job. T32 had it right, so the two contradicted each other, and 6's implementer would have looked for code that does not exist. Stale figures corrected to 3's: 4.2:1 rather than 4.1:1, over 97% rather than 97.4%, 14% rather than 15.7% in both the spec and the guide, and the 44.8 h gap total now names the 1800 s ceiling that T27 itself requires. The guide also said a batch always shares an end; it shares a start always and an end only when the results return together. Assisted-By: Claude (Anthropic AI) Signed-off-by: Mariusz Sabath --- docs/session-export.md | 10 +- ...26-11-19-tool-call-export-record-design.md | 127 ++++++++++++++---- 2 files changed, 104 insertions(+), 33 deletions(-) diff --git a/docs/session-export.md b/docs/session-export.md index ddf5a42da..ab5573124 100644 --- a/docs/session-export.md +++ b/docs/session-export.md @@ -154,10 +154,12 @@ timing that pairing yields, into a record. Two rules before you total those spans, both of which cost real accuracy: -- **A batch of parallel calls shares one window.** Calls the model issues together carry the - same start and the same end, not a duration each. Measured at 15.7% of spans (42 of 267) - on one 30-day archive. Charging every member of a batch the whole window's time - overcounts; read a shared window once per batch. +- **A batch of parallel calls can share one window.** Calls the model issues together always + share a *start*, because that start is the end of the response that asked for them. They + share an *end* only when their results come back in one later request — which is the case + that leaves them with no duration each. Measured at 14% of spans (50 of 348) on one 30-day + archive. Charging every member of such a batch the whole window's time overcounts; read a + shared window once per batch. - **Exclude `lost` spans.** A span that never got an answer but provably stopped sets `end` equal to `start`, so it passes an `end != null` filter as a 0 ms call and drags a p50 toward zero. A span with no `end` is still running. diff --git a/docs/superpowers/specs/2026-11-19-tool-call-export-record-design.md b/docs/superpowers/specs/2026-11-19-tool-call-export-record-design.md index 40559c297..e5b420ee5 100644 --- a/docs/superpowers/specs/2026-11-19-tool-call-export-record-design.md +++ b/docs/superpowers/specs/2026-11-19-tool-call-export-record-design.md @@ -12,13 +12,13 @@ This one narrows its §7.2 `toolCallNames` item and supersedes the shape propose Items marked **Open** must be decided before the work they belong to begins. Nothing here is merged. -**35 numbered requirements T0–T34**, phrased as plain declaratives: a requirement says what +**38 numbered requirements T0–T37**, phrased as plain declaratives: a requirement says what the system does once built. The numbering is stable so it can be cited in review and in the PRs that implement it, and is therefore in the order the requirements were ADDED, not document -order: T0 sits in §2.0 because it bounds the whole spec, T23–T27 in §4.3a, T28–T33 in §4.4 -and §4.4a, and T34 in §7.1b. +order: T0 sits in §2.0 because it bounds the whole spec, T23–T27 and T36–T37 in §4.3a, +T28–T33 and T35 in §4.4 and §4.4a, and T34 in §7.1b. -Revised twice on 2026-11-19 and once on 2026-11-20. +Revised twice on 2026-11-19, once on 2026-11-20 and once on 2026-11-21. **First revision**, after review raised two things the first draft buried. §2.0 is new: every field here is **already collected and most of it is already on disk**, so this is a projection @@ -59,6 +59,19 @@ Record shape changed in the second revision: `startedAt` replaces `at` and is al event records an explicit `kind`. T33 says a call below a `since` floor is dropped. §10 splits into what the review settled and what is still open. +**Fourth revision**, after @huang195's and @clawgenti's reviews on #2009. The must-fix was a +privacy contradiction: T10 said a name is exported verbatim "whatever its source", which +licensed shipping `mcp__acme-billing-prod__refund` in clear text and disagreed with T11's keyed +hash two lines below. T10 is now scoped to built-ins. Also: `lost` is dropped from `endReason` +(no case produces it — a call with no reason is still OPEN, and T17 does not export one); T35 +names the five fields written even at zero, since `omitempty` would drop a real `callIndex: 0` +and a clamped 0 ms duration; T28 is scoped to an opted-in stream, which it has to be or T20 is +false; `inputBytes` reaches the example and the field table; T36 and T37 say that `modelMs` and +`gapMs` repeat across a batch and that a `tool` gap is not idle time, so neither can be summed +naively; §2.2's claim that a background job extends a `Bash` span is removed, because the code +does not do that and T32 says so correctly; and the stale first-draft figures in §9 and the +guide are corrected to §3's. + Written after measuring, not before. §3 is the measurement; every design choice in §5 and §6 traces to a number in it. The measurement corpus is one developer's 30-day laptop archive: 40,496 export records, 12,104 tool calls, `claude-code/2.1.270` throughout with 46 @@ -147,7 +160,7 @@ has: - no tool name; - **no per-tool timing of any kind.** `InferenceToolCall` is three strings — `{ID, Name, - Arguments}` — and `InferenceToolResult` is three more. Neither carries a timestamp or a + Arguments}` — and `InferenceToolResult` is two strings and a bool. Neither carries a timestamp or a duration, and no field anywhere on the event model does per-call timing. The only per-tool timing in the system is **derived** by the timeline (§2.2), never stored; - **no gap between calls.** No `gapMs`, think-time or idle field exists on a record, on the @@ -175,8 +188,11 @@ stated here rather than summarised: - Four further closing cases exist and each one is load-bearing: `endedRun` (a `SubagentHandback`, `StructuredOutput` or `end_subtask` call never gets a result), the agent moving on without the result (`lastTurn`), a `` stopping the run - (`stopped`), and background jobs (`backgroundRe`) that extend a `Bash` span past its - acknowledgement. + (`stopped`), and background jobs (`backgroundRe`): the `Bash` span still **ends at its + acknowledgement** — nothing extends it — and the job becomes a separate `timeline.Job` + (`agents.go:133-137`, closed at `:140-142` and `:192-194`). An earlier draft of this bullet + said the span is extended, which contradicted T32 and would send an implementer of §6 looking + for code that does not exist. ### 2.3 The gap, and why it is structural @@ -343,11 +359,18 @@ pairing reads more content than the record carries, which is exactly why the tes it: the closing cases scan **message text** for task notifications (`agents.go:147-190`) and **tool result text** for background-job acknowledgements, so the planted secrets must include message content and result content, not only tool arguments. -T10. A tool name is exported **verbatim**, whatever its source. The built-in vocabulary is -closed per client but **not shared between clients** — one agent's `Bash` and `Read` are -another's `bash` and `read` — so there is no single list to normalise against, and -normalising would lose which spelling the client actually sent. `toolSource` (T4) is decided -by the client's own list; the name itself is never rewritten. +T10. **A BUILT-IN tool name is exported verbatim.** An MCP name is not: it follows T11. The +built-in vocabulary is closed per client but **not shared between clients** — one agent's +`Bash` and `Read` are another's `bash` and `read` — so there is no single list to normalise +against, and normalising would lose which spelling the client actually sent. Within that +source, then, the name is never rewritten: no case folding, no canonical spelling. +`toolSource` (T4) decides which rule applies, and it is decided by the client's own list. + +"Verbatim" is scoped to built-ins deliberately, and the scope is the privacy rule rather than +a detail of it. An earlier draft of T10 said "whatever its source", which read as a licence to +export `mcp__acme-billing-prod__refund` in clear text and contradicted T11 two lines below. +An implementer who reads one requirement and not the next must still land on the safe +behaviour. T11. An **MCP tool name is operator-defined and namespaced**, so it can carry a vendor relationship, an internal system name or a project codename — `mcp__acme-billing-prod__refund` is the worked example. It is therefore **not** treated as a built-in name. It is exported as a @@ -385,6 +408,19 @@ T17. A span still open when the walk reads it is not exported. An unfinished cal duration to report, and a record that appears with one later would make the export non-reproducible. T18. Durations are milliseconds, truncated, matching `Record`. +T35. **Five `toolCall` fields are always written, even at zero**, because for them zero is a +real value rather than "not recorded" — the opposite of the event record's rule: + +| Always written | Why zero is real | +|---|---| +| `kind` | the discriminator (T19) | +| `sessionId`, `startedAt` | identity and placement (T29) | +| `callIndex` | the FIRST call of every batch is 0, and T34 makes it a join key | +| `durationMs` on a reason that `HasEnd` | a result clamped to its call's start is a real 0 ms call (§3.5), and T31 requires a duration there | +| `endReason` | every exported call has one | + +`isError` stays `omitempty`: absent reads as false, which is what it means. The rest of the +record keeps the export's zero-means-absent rule (#1810 R3). ### 4.3a Model time, tool time and user time @@ -422,6 +458,24 @@ A fourth value is **not** needed for "a model call landed inside the gap": by th above the interval ends at the next request, so no inference can be inside it. That was an artifact of the old definition. +T36. **`modelMs` and `gapMs` belong to the asking RESPONSE, not to the call**, so every record +of a batch repeats the same two values. Two rules follow, and the docs state both beside the +fields: + +- **Take them once per `callSeq`.** Summing `modelMs` over `toolCall` records counts a batch's + inference `parallelN` times — the same inflating bias §3.4 describes for `durationMs`, except + here it is in the NUMERATOR of T2's ratio, and `durationShared` does not mark it. +- **An aggregate inference-to-tool ratio needs the event records.** A response that asks for no + tool — a final answer, or a question to the user — produces no `toolCall` record at all, so + its inference time appears on no record here. `modelMs` answers "what did this call's turn + cost"; it cannot answer T2 on its own. + +T37. **A `tool` gap is not idle time, and must not be added to tool time.** By §4.3a's +definition the interval covers what the previous turn's calls were doing, so for a `gapKind` +of `tool` it overlaps those calls' own `durationMs`: adding the two counts the same wall clock +twice. Only a `user` gap is time no tool and no model was working, which is why it is the one +that answers a duty cycle. + **A consumer with the event records can derive all three fields**, so these definitions are the deliverable and the fields are the convenience. A definition that cannot be checked against the data is worse than no field. @@ -439,7 +493,8 @@ raw `gapMs` and the documentation requires any consumer publishing a duty cycle ceiling alongside it (#1810 §9 states the same rule for event gaps). Why this split is worth the three fields: on the reference archive, user wait is **72.6% of -all gap time** (44.8 h of 61.7 h) and costs nothing in sandbox occupancy. Without `gapKind`, +all gap time** (44.8 h of 61.7 h, at an 1800 s ceiling — T27's own rule) and costs nothing in +sandbox occupancy. Without `gapKind`, a density plan reads that as idle capacity it is paying for. The classifier is already validated — run on that archive it separated `AskUserQuestion` (p50 62.6 s) from `Bash` (p50 201 ms) by three orders of magnitude **without ever seeing a tool name** (#1810 §9). @@ -451,13 +506,17 @@ T19. A `toolCall` record is a **sibling record on the same NDJSON stream**, disc `select(.kind != "toolCall")` or ignores the key: an absent `kind` reads as the event record, so no shipped consumer breaks. T20. `toolCall` records are **opt-in**, per #1810 R49's rule for Tier 2: a plain -`GET /v1/export` is byte-identical to today's. **Open:** `?include=toolCalls` on the endpoint -and `--with-tool-calls` on the command, or a single Tier-2 switch covering every §7.2 item. +`GET /v1/export` is byte-identical to today's. **Settled (§10.1): per item** — +`?include=toolCalls` on the endpoint and `--with-tool-calls` on the command, not a single +Tier-2 switch, which would opt a consumer into items it never asked for. T21. Adding the record does not bump `Record`'s `schema`. A new record *kind* on the stream is additive; `sessionexport.SchemaVersion` bumps only on a removal or a meaning change. -T28. An event record carries an explicit `kind: "event"` once `toolCall` records exist, so a -consumer can select on `kind` in both directions rather than on the absence of a key. An -absent `kind` still reads as an event record, so an older file stays readable. +T28. An event record carries an explicit `kind: "event"` **only on a stream that carries +`toolCall` records** — that is, only when T20's opt-in is on. A plain `GET /v1/export` is +byte-identical to today's, which T20 requires and which adding a key to every record would +break. On an opted-in stream a consumer can then select on `kind` in both directions rather +than on the absence of a key; an absent `kind` still reads as an event record, so a file from +either mode stays readable. ### 4.4a Placing and ordering a call @@ -478,10 +537,16 @@ T31. A `toolCall` record carries `endReason`, a closed vocabulary naming how the | `ended_run` | no | By design no result: the call ends the run or hands control back (`SubagentHandback`, `StructuredOutput`, `end_subtask`). | | `abandoned` | no | The agent moved on without the result (`lastTurn`). | | `stopped` | no | The run was stopped while the call was open (a ``). | -| `lost` | no | Open with no reason established. | + +**No `lost` value.** An earlier draft had one for "open with no reason established", and no +case produces it: a call with none of the four reasons above still has `End == nil`, which +means **still open**, and T17 does not export an open call at all. A vocabulary entry a +producer never emits is a trap for the consumer — it switches on it and ships a dead branch, +or reads it as the catch-all for "no result" and misses the three real reasons. Add one only +alongside the code that returns it. These are the timeline's own closing cases (`agents.go:273-295`), named rather than collapsed -into one `lost` flag: "the agent moved on" and "the call ends the run by design" are different +into one `Lost` bool: "the agent moved on" and "the call ends the run by design" are different facts, and a consumer filtering latency wants to exclude both but count them apart. T32. **A backgrounded command is MARKED, not silently undercounted.** A `Bash` call that starts a background job ends at the acknowledgement, and the job's own run is a separate @@ -514,7 +579,8 @@ list. The doc fix that precedes this spec already states them for `/timeline`. {"schema":1,"kind":"toolCall","sessionId":"45fa31b2-…", "startedAt":"2026-11-05T15:52:25.629885Z","tool":"Bash","toolSource":"builtin", "callSeq":2077,"callIndex":0,"resultSeq":2081, - "durationMs":473,"endReason":"result","isError":false,"resultBytes":1432, + "durationMs":473,"endReason":"result","isError":false, + "inputBytes":64,"resultBytes":1432, "agentRole":"main","client":"claude-code/2.1.270", "modelMs":4279,"gapMs":812,"gapKind":"tool", "parallelN":3,"durationShared":true} @@ -547,7 +613,7 @@ Four changes from the first draft, all so a record can be placed and ordered on | `durationMs` | T18. The paired duration. Absent when `endReason` says there was no result (T16, T31). | | `endReason` | T31. Closed vocabulary: how the call ended. `lost` derives from it, so no separate flag. | | `isError` | Whether the result was an error. The tool failure rate, per tool. | -| `resultBytes` | T14. Effort proxy, never content. Bytes, as the name says. | +| `inputBytes` / `resultBytes` | T14. Effort proxies, never content. Bytes, as the names say. | | `modelMs` | T23. The asking response's measured duration — model time, per call, no join needed. | | `gapMs` | T24. The idle interval before the asking response. Contains no `modelMs`. | | `gapKind` | T25. `user`, `tool` or `first` — what that interval was. The field that turns idle time into a duty cycle. | @@ -596,9 +662,11 @@ result lands. - **Against:** a second implementation of §2.2's closing cases, which will drift from the timeline's. -**Open: which option.** The author leans **A**, because §2.2's four closing cases are exactly -the kind of logic that must not exist twice, and because the export is the second consumer that -wants it — a third (an OTLP emitter, #505) would make the duplication permanent. +**Settled (§10.1): Option A**, scoped to tool spans and their closing inputs. §2.2's four +closing cases are exactly the kind of logic that must not exist twice, and the export is the +second consumer that wants it — a third (an OTLP emitter, #505) would make the duplication +permanent. Note for the implementer: because the closing cases read message and tool-result +text, T9's planted-secret test must plant in both, not only in tool arguments. ### 6.4 Memory, whichever option wins @@ -608,7 +676,8 @@ The walk is O(1) per event today. Pairing is O(open calls) per session. On the r archive a session held 2,911 events; the largest in §3 held 722 with 204 tool spans. A bound is needed, with a documented behaviour when it is hit — the export's existing posture is to degrade loudly (a truncated stream resets the connection) rather than silently, and this -should match. **Open:** the bound and what happens at it. +should match. **Settled (§10.1):** a fixed cap on open calls per session, and a per-session +marker record when it is hit. --- @@ -692,11 +761,11 @@ One developer's laptop archive over 30 days: 40,496 records, 12,104 tool calls, `claude-code/2.1.270` with 46 `codex/0.163.0` records. Four sessions had timeline spans. The **mechanism** generalises, because it follows the event schema and the pairing rules, not -this traffic. The **ratios do not**: 4.1:1 inference-to-tool and `Bash` at 97.4% of tool time -will move with working style, prompting and agent. Fleet numbers are the entire point of +this traffic. The **ratios do not**: 4.2 : 1 inference-to-tool and `Bash` at over 97% of tool +time (811.1 s of 828.2 s) will move with working style, prompting and agent. Fleet numbers are the entire point of building this, and they will come from the export rather than from §3. -A second caveat on §3.4: 15.7% is one archive's parallel rate under one agent's batching +A second caveat on §3.4: 14% is one archive's parallel rate under one agent's batching behaviour. #1810 notes Claude Code's calls are sequential in the trace while Codex's overlap, so the rate is agent-dependent and the 46 Codex records here cannot speak to it. The requirement T15 derives from it stands regardless — a shared window must be marked whatever From f99384874c2b494da9ebf91295faf563fd98255e Mon Sep 17 00:00:00 2001 From: Mariusz Sabath Date: Fri, 20 Nov 2026 20:25:40 -0500 Subject: [PATCH 07/12] Docs: Name the stranded closing case, and hash an unknown tool source Addresses @clawgenti's second pass on #2009. Both must-fixes were places where the fourth revision's own new text claimed more than the code supports. "No case produces lost" was false. endStrandedTools is a FIFTH closing case this spec never mentioned: Build runs it after the pairing, and it gives a tool an End at a failed model-host dial without marking it Lost. So a call can have an end and none of the four reasons -- which T35 says cannot happen on an exported record, and which T17 would drop as still open. endReason gains `stranded`, with the duration it really has. The honest part of that fix is the implementation note: the pairing package cannot see this case today. core/session/toolpair was extracted from buildAgents alone, and endStrandedTools needs the agents, turns and hosts the pairing has none of. Either the host-failure times reach the pairing or the export applies the stranding pass over the records it is about to write; that is left Open rather than guessed, because it changes the pairing's inputs. The T10 fix left a hole. toolSource "unknown" was covered by neither T10 (built-in, verbatim) nor T11 (MCP, hashed), and the field table still said "the name, verbatim" without qualification. T10a says unknown follows T11: the branch is on "provably a built-in", never on "provably MCP", because a hashed built-in costs a reader one unreadable label where a cleartext MCP name costs a customer's identity. T10's own standard -- an implementer who reads one requirement and not the next still lands on the safe behaviour -- is what makes this a separate requirement rather than an inference. Also from that review: the guide's lost-span filter had the same blind spot as the first item, since a stranded span is neither lost nor open, so a reader filtering both still totals an outage as tool time; the #1924 row cited a claim 7.1 stopped making two revisions ago; and T35 said five fields where its table names six. Every code reference now names a symbol instead of a line. All eleven line pins were correct when written and wrong a day later, because #2041 moves every one of them into core/session/toolpair. A reader who checks a stale pin, finds unrelated code and stops trusting the rest is the failure this avoids. Assisted-By: Claude (Anthropic AI) Signed-off-by: Mariusz Sabath --- docs/session-export.md | 10 ++- ...26-11-19-tool-call-export-record-design.md | 83 ++++++++++++++----- 2 files changed, 71 insertions(+), 22 deletions(-) diff --git a/docs/session-export.md b/docs/session-export.md index ab5573124..6b0010a5e 100644 --- a/docs/session-export.md +++ b/docs/session-export.md @@ -160,9 +160,13 @@ Two rules before you total those spans, both of which cost real accuracy: that leaves them with no duration each. Measured at 14% of spans (50 of 348) on one 30-day archive. Charging every member of such a batch the whole window's time overcounts; read a shared window once per batch. -- **Exclude `lost` spans.** A span that never got an answer but provably stopped sets `end` - equal to `start`, so it passes an `end != null` filter as a 0 ms call and drags a p50 - toward zero. A span with no `end` is still running. +- **Exclude `lost` spans, and do not trust `lost` alone.** A span that never got an answer but + provably stopped sets `end` equal to `start`, so it passes an `end != null` filter as a 0 ms + call and drags a p50 toward zero. A span with no `end` is still running. But a third kind + exists: a call whose agent could not reach the model to report back gets an `end` from the + failed attempt and is **not** marked `lost`, so neither filter catches it and its duration is + the outage rather than the tool. Filter `lost`, drop spans with no `end`, and treat a + suspiciously long tail as possibly an outage rather than a slow tool. An export is **content-free**. It is **not anonymous**. It keeps hosts, models, session ids and timestamps. Together these show who called what, and when. Read an export before diff --git a/docs/superpowers/specs/2026-11-19-tool-call-export-record-design.md b/docs/superpowers/specs/2026-11-19-tool-call-export-record-design.md index e5b420ee5..e4af1b717 100644 --- a/docs/superpowers/specs/2026-11-19-tool-call-export-record-design.md +++ b/docs/superpowers/specs/2026-11-19-tool-call-export-record-design.md @@ -12,13 +12,13 @@ This one narrows its §7.2 `toolCallNames` item and supersedes the shape propose Items marked **Open** must be decided before the work they belong to begins. Nothing here is merged. -**38 numbered requirements T0–T37**, phrased as plain declaratives: a requirement says what +**39 numbered requirements T0–T37, plus the lettered T10a that narrows T10**, phrased as plain declaratives: a requirement says what the system does once built. The numbering is stable so it can be cited in review and in the PRs that implement it, and is therefore in the order the requirements were ADDED, not document order: T0 sits in §2.0 because it bounds the whole spec, T23–T27 and T36–T37 in §4.3a, T28–T33 and T35 in §4.4 and §4.4a, and T34 in §7.1b. -Revised twice on 2026-11-19, once on 2026-11-20 and once on 2026-11-21. +Revised twice on 2026-11-19, once on 2026-11-20 and twice on 2026-11-21. **First revision**, after review raised two things the first draft buried. §2.0 is new: every field here is **already collected and most of it is already on disk**, so this is a projection @@ -72,6 +72,30 @@ naively; §2.2's claim that a background job extends a `Bash` span is removed, b does not do that and T32 says so correctly; and the stale first-draft figures in §9 and the guide are corrected to §3's. +**Fifth revision**, after @clawgenti's second pass on #2009 found two more must-fixes, both +places where the fourth revision's new text claimed more than the code supports: + +- **"No case produces `lost`" was false.** `endStrandedTools` is a FIFTH closing case this spec + never mentioned: it runs after the pairing and gives a tool an end at a failed model-host dial + WITHOUT marking it lost. Such a call has an end and none of the four reasons, which T35 says + cannot happen and T17 would drop. `endReason` gains `stranded`, and the implementation note + says plainly that the pairing package cannot see this case today — `endStrandedTools` needs + the agents, turns and hosts the pairing has none of — so how it learns it is **Open**. +- **The T10 fix left `toolSource: "unknown"` uncovered** by either rule. T10a says it follows + T11 and is hashed: the branch is on "provably a built-in", never on "provably MCP", because a + hashed built-in costs a reader one unreadable label where a cleartext MCP name costs a + customer's identity. The field table said "verbatim" without qualification and now does not. + +Also: the guide's lost-span filter had the same blind spot as the first item and now names the +third kind; the #1924 row no longer cites a claim §7.1 stopped making; and T35 says six fields +rather than five. + +**Code references name symbols, not line numbers.** Every `agents.go:NNN` pin in the first +four revisions was correct when written and wrong a day later: #2041 moves all of them into +`core/session/toolpair`. A spec that cites `toolpair.Calls`, `closeCalls`, `preview` or +`endStrandedTools` still reads correctly after the move; one that cites a line is stale the day +it merges, and a reader who checks a stale pin and finds unrelated code stops trusting the rest. + Written after measuring, not before. §3 is the measurement; every design choice in §5 and §6 traces to a number in it. The measurement corpus is one developer's 30-day laptop archive: 40,496 export records, 12,104 tool calls, `claude-code/2.1.270` throughout with 46 @@ -96,8 +120,8 @@ traces to a number in it. The measurement corpus is one developer's 30-day lapto It is also **content-light, not content-free**, which matters when comparing it with this record: `Tool.Preview` carries up to 80 runes of argument text — a command line, a file path, -a query (`agents.go:487-515`) — and `Tool.ID` is the tool-use id itself -(`timeline.go:128-138`). #1810 §9.1 states the same caveat in its own words ("no prompts", +a query (`preview`) — and `Tool.ID` is the tool-use id itself +(`timeline.Tool`). #1810 §9.1 states the same caveat in its own words ("no prompts", not "safe to share"). A `toolCall` record carries neither (T6, T8). | Need | Why `/timeline` cannot | @@ -190,7 +214,7 @@ stated here rather than summarised: moving on without the result (`lastTurn`), a `` stopping the run (`stopped`), and background jobs (`backgroundRe`): the `Bash` span still **ends at its acknowledgement** — nothing extends it — and the job becomes a separate `timeline.Job` - (`agents.go:133-137`, closed at `:140-142` and `:192-194`). An earlier draft of this bullet + (the `Job` the pairing records, closed by its `` or by `TaskStop`). An earlier draft of this bullet said the span is extended, which contradicted T32 and would send an implementer of §6 looking for code that does not exist. @@ -287,7 +311,7 @@ than that should be read into the agreement. gives such a batch one merged interval, not a duration each. A batch is **structural, not inferred from the timestamps**: every call of one response starts -at that response's end (`agents.go:234` sets `Start: end` for each), so they share a start by +at that response's end (`buildAgents` sets a span's `Start` to the asking response's end for each), so they share a start by construction. They share an *end* when their results arrive in one later request. So the grouping key is the asking response's seq, and `durationShared` says the results also returned together — which is why §5 carries both rather than a timestamp match. @@ -356,7 +380,7 @@ what keeps a `toolCall` record unjoinable back to a conversation. T9. `TestRecord_CarriesNoContent` is extended to cover the `toolCall` record, planting a secret in every content field the pairing reads and failing if any reaches the encoding. The pairing reads more content than the record carries, which is exactly why the test must cover -it: the closing cases scan **message text** for task notifications (`agents.go:147-190`) and +it: the closing cases scan **message text** for task notifications (the returns-marker scan) and **tool result text** for background-job acknowledgements, so the planted secrets must include message content and result content, not only tool arguments. T10. **A BUILT-IN tool name is exported verbatim.** An MCP name is not: it follows T11. The @@ -366,11 +390,17 @@ against, and normalising would lose which spelling the client actually sent. Wit source, then, the name is never rewritten: no case folding, no canonical spelling. `toolSource` (T4) decides which rule applies, and it is decided by the client's own list. +T10a. **A name whose source cannot be determined (`toolSource: "unknown"`) follows T11, not +T10** — it is hashed. Neither of the two rules above covers it otherwise, and the safe default +for a name that might be operator-defined is to treat it as if it were: a hashed built-in costs +a consumer one unreadable label, where a cleartext MCP name costs a customer's identity. So +the branch is on "is this provably a built-in", never on "is this provably MCP". + "Verbatim" is scoped to built-ins deliberately, and the scope is the privacy rule rather than a detail of it. An earlier draft of T10 said "whatever its source", which read as a licence to export `mcp__acme-billing-prod__refund` in clear text and contradicted T11 two lines below. An implementer who reads one requirement and not the next must still land on the safe -behaviour. +behaviour — which is why T10a exists rather than leaving the third value to inference. T11. An **MCP tool name is operator-defined and namespaced**, so it can carry a vendor relationship, an internal system name or a project codename — `mcp__acme-billing-prod__refund` is the worked example. It is therefore **not** treated as a built-in name. It is exported as a @@ -381,7 +411,7 @@ across installs, which is the right trade for a name that may identify a custome happens **before** T12's cut, so the cut never splits a digest. T12. A tool name is bounded on the way out by the same `exportLabel` every wire-sourced string in a record passes: control runes removed, then truncated to `maxLabelLen` on a rune -boundary. `maxLabelLen` is **96 bytes**, not runes (`record.go:352-355`), so a multi-byte name +boundary. `maxLabelLen` is **96 bytes**, not runes (`sessionexport.maxLabelLen`), so a multi-byte name is cut sooner than its rune count suggests. ### 4.3 Data rules @@ -408,7 +438,7 @@ T17. A span still open when the walk reads it is not exported. An unfinished cal duration to report, and a record that appears with one later would make the export non-reproducible. T18. Durations are milliseconds, truncated, matching `Record`. -T35. **Five `toolCall` fields are always written, even at zero**, because for them zero is a +T35. **Six `toolCall` fields are always written, even at zero**, because for them zero is a real value rather than "not recorded" — the opposite of the event record's rule: | Always written | Why zero is real | @@ -533,10 +563,25 @@ T31. A `toolCall` record carries `endReason`, a closed vocabulary naming how the | `endReason` | Duration? | Meaning | |---|---|---| | `result` | yes | Its result arrived. | -| `background` | yes, **to the acknowledgement only** | The result only acknowledged a job that keeps running (`backgroundRe`, `agents.go:37`). The command's real duration is longer and is not here. | +| `background` | yes, **to the acknowledgement only** | The result only acknowledged a job that keeps running (`backgroundRe`, in the pairing package). The command's real duration is longer and is not here. | | `ended_run` | no | By design no result: the call ends the run or hands control back (`SubagentHandback`, `StructuredOutput`, `end_subtask`). | | `abandoned` | no | The agent moved on without the result (`lastTurn`). | | `stopped` | no | The run was stopped while the call was open (a ``). | +| `stranded` | **yes** | The agent could not reach the model to report back: its next attempt on that host failed to dial, and the call ended there (`endStrandedTools`). | + +**`stranded` is the fifth closing case, and this spec missed it until review.** `Build` runs +`endStrandedTools` AFTER the pairing (`buildAgents` then `endStrandedTools`, both from `Build`), and it sets a tool's +`End` on a non-`Lost` span without setting `Lost` — so the timeline has a call with an end and +none of the four reasons above. A pairing that stops at those four leaves such a call with an +empty reason, which T35 says cannot happen on an exported record, and T17 would then drop a +call that did in fact finish. + +Implementation note: the pairing package must therefore learn this case, which it does **not** +do today — `core/session/toolpair` was extracted from `buildAgents` only, and +`endStrandedTools` needs `t.Agents`, `t.Turns` and `t.Hosts`, which the pairing has none of. +Two ways out, and this is **Open**: carry the host-failure times into the pairing, or let the +export apply the stranding pass itself over the records it is about to write. The second keeps +the pairing's inputs as they are; the first keeps the reason where the other four live. **No `lost` value.** An earlier draft had one for "open with no reason established", and no case produces it: a call with none of the four reasons above still has `End == nil`, which @@ -545,12 +590,12 @@ producer never emits is a trap for the consumer — it switches on it and ships or reads it as the catch-all for "no result" and misses the three real reasons. Add one only alongside the code that returns it. -These are the timeline's own closing cases (`agents.go:273-295`), named rather than collapsed +These are the timeline's own closing cases (the closing pass), named rather than collapsed into one `Lost` bool: "the agent moved on" and "the call ends the run by design" are different facts, and a consumer filtering latency wants to exclude both but count them apart. T32. **A backgrounded command is MARKED, not silently undercounted.** A `Bash` call that starts a background job ends at the acknowledgement, and the job's own run is a separate -`timeline.Job` (`agents.go:133-137`), so this record cannot carry the command's real duration. +`timeline.Job` (a `Job` of its own), so this record cannot carry the command's real duration. It therefore sets `endReason: "background"` with `durationMs` measuring only the acknowledgement. A count of such calls tells a consumer **how much tool time is missing** and from where, which an undifferentiated `result` would hide. @@ -561,7 +606,7 @@ never ended, and its starting call's `callSeq` + `callIndex` (no ids), paired by later calls, so summed per-command time can exceed a session's wall-clock tool time. A consumer must not add job time to `durationMs` totals. T33. A `toolCall` record whose asking response is below a `since` floor is **not exported**. -`Walk` stops at the first event below the floor (`walk.go:243-245`), so such a call has no +`Walk` stops at the first event below the floor (`Walk`'s `since` stop), so such a call has no start to report, and T29 requires one. A result inside the window whose call is outside it therefore yields no record, rather than a record with a missing start. @@ -606,8 +651,8 @@ Four changes from the first draft, all so a record can be placed and ordered on | `kind` | T19. The discriminator. Always `"toolCall"` here; an event record carries `"event"` (T28). | | `sessionId` | T5. The session, as the client asserted it. Not a unit of work until #1997. | | `startedAt` | T29. When the call started — the asking response's end. **Always present**, so a `lost` call can still be placed. | -| `tool` | T10/T11. The name, verbatim, bounded by T12. | -| `toolSource` | T4. `builtin`, `mcp` or `unknown` — a closed vocabulary, exported by naming the values. | +| `tool` | T10/T10a/T11. A built-in name verbatim; an `mcp` or `unknown` name as a keyed hash. Bounded by T12. | +| `toolSource` | T4. `builtin`, `mcp` or `unknown` — a closed vocabulary, exported by naming the values. Only `builtin` exports its name in clear (T10a). | | `callSeq` / `resultSeq` | The response that asked and the request that carried the result. Lets a consumer rejoin a `toolCall` to the event records either side **without a tool-use id** (T8). `resultSeq` is absent when nothing carried a result. | | `callIndex` | T30. The call's position among its response's calls, from 0, in the order the model asked. | | `durationMs` | T18. The paired duration. Absent when `endReason` says there was no result (T16, T31). | @@ -649,7 +694,7 @@ The walk calls the timeline builder per session and projects its spans. - **For:** smallest diff, no new pairing code. - **Against:** inherits the 50,000-event cap, which bounds the export by a timeline limit that has nothing to do with it; and it builds turns, hosts and agents the export discards. The cap - lives in the **HTTP handler** (`sessionapi/timeline.go:25`), not the builder, so an in-process + lives in the **HTTP handler** (`sessionapi`'s `timelineMaxEvents`), not the builder, so an in-process caller could avoid it — but then the export depends on a limit staying where it is. ### 6.3 Option C — a second, bounded pass in `sessionexport` @@ -685,7 +730,7 @@ marker record when it is hit. 7.1 **The proxy sees the gap around a tool, never inside it.** A correction to the first draft of this spec, which claimed local tool calls are invisible. They are not: a span runs from the -asking response's end to the next request carrying the result (`agents.go:234`, `:280-287`), +asking response's end to the next request carrying the result (the span-emission loop and the closing pass in `buildAgents`, now `toolpair.Calls` and `toolpair.closeCalls`), and **both boundaries are model traffic**, so a tool that touches nothing on the network still gets a span. §3.1's `Edit` at p50 35 ms is such a call. The earlier claim confused this record with #1810 §9's gap classification, where "no proxied traffic landed inside the gap" is how a @@ -747,7 +792,7 @@ which user instruction. | #1906 | The tracker. This spec narrows its §2 and **supersedes** its "list on the response record" shape, for the §3.4 reason. | | #1810 | The parent spec. This is its §7.2 `toolCallNames` item, re-scoped from names to paired calls. | | #1740 | Adds `newToolResults` and `toolResultErrors` to the event record. Complementary: those count results per event; this counts calls with durations. | -| #1924 | Claude Code telemetry would supply the tool times the proxy cannot see (§7.1) **and the permission waits T26 cannot separate from execution (§7.1a)**. It is the fix for both of this spec's largest error sources, so the two should be planned together even though this one ships first. | +| #1924 | Claude Code telemetry records what happens INSIDE a span, which §7.1 says the proxy cannot see: the permission waits T26 cannot separate from execution (§7.1a), and where a call actually ran (§7.1b). It is the fix for this spec's largest error source, so the two should be planned together even though this one ships first. | | #1921 / #1925 | Redaction and the content export. Where tool arguments and result content go, per T0 — a different privacy posture, not a flag on this record. | | #1997 | `sessionIncarnation`. Needed before per-session aggregates mean anything (§7.3). | | rossoctl/trajectory-schema | **A consumer exists.** That schema added an observed-session kind (`trajectory-schema/session/1`, v2.1) and `tools/from-cortex.py`, which converts a `/v1/export` stream of event and `toolCall` records into it and into OTLP/JSON. Its field-by-field mapping is what surfaced T29 (a start on every record) and T30 (`callIndex`): without them a lost call cannot be placed and a batch cannot be ordered. Both are now required. The converter reports `gapKind` as unmapped and derives it from the event records instead, which is the right division — §4.3a says the definitions are the deliverable. | From 044a9de9706a36d28a5cf77bb1984dde8b633aac Mon Sep 17 00:00:00 2001 From: Mariusz Sabath Date: Fri, 20 Nov 2026 20:43:03 -0500 Subject: [PATCH 08/12] Docs: Finish wiring stranded, and fix its duration in the guide Addresses @clawgenti's third pass on #2009. The fifth revision added the stranded closing case and left it half-wired; this finishes it. The guide had the duration backwards. It said a stranded span's duration is the outage rather than the tool. It is the opposite: endStrandedTools exists to cut the outage off, and its own doc records that the dial it cuts at "fell within 0.1s of the tool's real end every time", after one 9-minute Bash drew as 8 hours. A reader following the old sentence would have discarded the one class of span the proxy has already corrected. What the guide should warn about is the RESULT, not the duration: a stranded span may have none, or one that arrived hours later, so "has an end and is not lost" does not mean "its result arrived". It also now says that only a failed dial is caught this way, so a long tail can still be an outage of another kind. T16 said "a call with no result carries no duration", which stranded contradicts. The rule is "no END, no duration": durationMs is present exactly when endReason.HasEnd(). A note keeps #2041 in step, where HasEnd is result || background today and must gain stranded, or omitempty drops the duration of a stranded call that measured 0ms. The late-result case is specified rather than left hanging. When the result does arrive, buildAgents ends the call on it and the stranding pass moves End back to the dial, so both reasons apply. stranded wins, because it is the one that explains the duration; resultSeq stays present, since a request did carry the result; and durationMs ends at the dial, not at resultSeq. That is the one place where those two disagree, and the callSeq/resultSeq row implied otherwise. Also: "a call with none of the four reasons still has End == nil" was the sentence the previous commit message itself called false, still sitting 15 lines below the paragraph that disproves it -- four to five, and the closing count of reasons-without-a-result recounted with stranded among them. The remaining `lost` wording in the record section is gone, since T31 has no such reason, and the toolpair references now say #2041's, because that package does not exist on main. Assisted-By: Claude (Anthropic AI) Signed-off-by: Mariusz Sabath --- docs/session-export.md | 18 ++-- ...26-11-19-tool-call-export-record-design.md | 83 ++++++++++++++----- 2 files changed, 74 insertions(+), 27 deletions(-) diff --git a/docs/session-export.md b/docs/session-export.md index 6b0010a5e..99818370f 100644 --- a/docs/session-export.md +++ b/docs/session-export.md @@ -160,13 +160,17 @@ Two rules before you total those spans, both of which cost real accuracy: that leaves them with no duration each. Measured at 14% of spans (50 of 348) on one 30-day archive. Charging every member of such a batch the whole window's time overcounts; read a shared window once per batch. -- **Exclude `lost` spans, and do not trust `lost` alone.** A span that never got an answer but - provably stopped sets `end` equal to `start`, so it passes an `end != null` filter as a 0 ms - call and drags a p50 toward zero. A span with no `end` is still running. But a third kind - exists: a call whose agent could not reach the model to report back gets an `end` from the - failed attempt and is **not** marked `lost`, so neither filter catches it and its duration is - the outage rather than the tool. Filter `lost`, drop spans with no `end`, and treat a - suspiciously long tail as possibly an outage rather than a slow tool. +- **Exclude `lost` spans.** A span that never got an answer but provably stopped sets `end` + equal to `start`, so it passes an `end != null` filter as a 0 ms call and drags a p50 + toward zero. A span with no `end` is still running. Filter `lost`, and drop a span with no + `end`. + + A third kind needs no filter: a call whose agent finished the tool and then could not reach + the model to report back — a lid closed, a VPN dropped — gets its `end` from that first + failed attempt and is **not** marked `lost`. **Keep it.** Its duration is the tool's, not the + outage's: the proxy cuts the span at the failed dial precisely so the outage is excluded, and + against Claude Code's own transcripts that dial landed within 0.1 s of the tool's real end + every time. Without it, one 9-minute `Bash` drew as 8 hours. An export is **content-free**. It is **not anonymous**. It keeps hosts, models, session ids and timestamps. Together these show who called what, and when. Read an export before diff --git a/docs/superpowers/specs/2026-11-19-tool-call-export-record-design.md b/docs/superpowers/specs/2026-11-19-tool-call-export-record-design.md index e4af1b717..a1788db9e 100644 --- a/docs/superpowers/specs/2026-11-19-tool-call-export-record-design.md +++ b/docs/superpowers/specs/2026-11-19-tool-call-export-record-design.md @@ -18,7 +18,7 @@ PRs that implement it, and is therefore in the order the requirements were ADDED order: T0 sits in §2.0 because it bounds the whole spec, T23–T27 and T36–T37 in §4.3a, T28–T33 and T35 in §4.4 and §4.4a, and T34 in §7.1b. -Revised twice on 2026-11-19, once on 2026-11-20 and twice on 2026-11-21. +Revised twice on 2026-11-19, once on 2026-11-20 and three times on 2026-11-21. **First revision**, after review raised two things the first draft buried. §2.0 is new: every field here is **already collected and most of it is already on disk**, so this is a projection @@ -77,7 +77,7 @@ places where the fourth revision's new text claimed more than the code supports: - **"No case produces `lost`" was false.** `endStrandedTools` is a FIFTH closing case this spec never mentioned: it runs after the pairing and gives a tool an end at a failed model-host dial - WITHOUT marking it lost. Such a call has an end and none of the four reasons, which T35 says + WITHOUT marking it lost. Such a call has an end and none of the other four reasons, which T35 says cannot happen and T17 would drop. `endReason` gains `stranded`, and the implementation note says plainly that the pairing package cannot see this case today — `endStrandedTools` needs the agents, turns and hosts the pairing has none of — so how it learns it is **Open**. @@ -90,6 +90,23 @@ Also: the guide's lost-span filter had the same blind spot as the first item and third kind; the #1924 row no longer cites a claim §7.1 stopped making; and T35 says six fields rather than five. +**Sixth revision**, finishing what the fifth started: it added `stranded` and left it +half-wired, which is the whole of this round. + +- **The guide had the duration backwards.** It said a stranded span's duration is the outage. + It is the opposite: `endStrandedTools` exists to CUT the outage off, and the dial it cuts at + measured within 0.1 s of the tool's real end on every session checked against Claude Code's + transcripts — one 9-minute `Bash` had drawn as 8 hours. A reader following the old text would + have discarded the one class of span already corrected. What the guide should warn about, and + now does, is the **result**: a stranded span may have none, or one that arrived hours later. +- **T16 said "no result, no duration"**, which `stranded` contradicts. The rule is "no END, no + duration": `durationMs` is present exactly when `endReason.HasEnd()`. +- **The late-result case is specified**: `stranded` wins over `result`, because it is the reason + that explains the duration; `resultSeq` stays present; and `durationMs` ends at the dial, not + at `resultSeq`. That is the one case where the two disagree. +- **"None of the four reasons ... has `End == nil`"** was the sentence the fifth revision's own + commit message called false, still sitting 15 lines below the paragraph disproving it. + **Code references name symbols, not line numbers.** Every `agents.go:NNN` pin in the first four revisions was correct when written and wrong a day later: #2041 moves all of them into `core/session/toolpair`. A spec that cites `toolpair.Calls`, `closeCalls`, `preview` or @@ -430,10 +447,21 @@ batch's size in `parallelN`. The batch's identity is `callSeq` (§3.4), so no se exists. A consumer can then divide, take the window once per batch, or exclude it — explicitly, by its own choice. The marker says the window is shared, **never** that the calls ran concurrently. -T16. A call with no result **carries no duration**, rather than the zero-length span -`/timeline` draws it with. A zero would pass a consumer's `durationMs > 0` filter as a 0 ms -call and drag a p50 toward zero. Which no-result case it was is `endReason` (T31), so the -record carries no separate `lost` flag. +T16. A call with no result and **no end** carries no duration, rather than the zero-length +span `/timeline` draws it with. A zero would pass a consumer's `durationMs > 0` filter as a +0 ms call and drag a p50 toward zero. Which case it was is `endReason` (T31), so the record +carries no separate `lost` flag. + +**`stranded` is the exception, and it is not a contradiction**: that call HAS an end, so it has +a duration. Its agent finished the tool and then could not reach the model to report back, and +the proxy cuts the span at that first failed dial — which measured within 0.1 s of the tool's +real end on every session checked against Claude Code's transcripts. So the rule is "no end, no +duration", not "no result, no duration": `durationMs` is present exactly when +`endReason.HasEnd()`, which is `result`, `background` and `stranded`. + +**Note for #2041:** `toolpair.EndReason.HasEnd()` there is `result || background`. Whichever of +§2.2's two options lands, `HasEnd` has to include `stranded`, or `omitempty` drops the duration +of a stranded call that measured 0 ms. T17. A span still open when the walk reads it is not exported. An unfinished call has no duration to report, and a record that appears with one later would make the export non-reproducible. @@ -551,8 +579,8 @@ either mode stays readable. ### 4.4a Placing and ordering a call T29. A `toolCall` record carries `startedAt`, the call's start — the asking response's end. It -is **always present**, including on a call that never got a result, so every record can be -placed on a timeline. The end, where there is one, is `startedAt + durationMs`. The record +is **always present**, including one that never got a result, so every record can be placed +on a timeline. The end, where there is one, is `startedAt + durationMs`. The record carries no separate `seq`: that collided with the event record's own `seq` and was undefined for an unresolved call. T30. A `toolCall` record carries `callIndex`, the call's position among the calls its asking @@ -563,19 +591,34 @@ T31. A `toolCall` record carries `endReason`, a closed vocabulary naming how the | `endReason` | Duration? | Meaning | |---|---|---| | `result` | yes | Its result arrived. | -| `background` | yes, **to the acknowledgement only** | The result only acknowledged a job that keeps running (`backgroundRe`, in the pairing package). The command's real duration is longer and is not here. | +| `background` | yes, **to the acknowledgement only** | The result only acknowledged a job that keeps running (`backgroundRe`, in the pairing). The command's real duration is longer and is not here. | | `ended_run` | no | By design no result: the call ends the run or hands control back (`SubagentHandback`, `StructuredOutput`, `end_subtask`). | | `abandoned` | no | The agent moved on without the result (`lastTurn`). | | `stopped` | no | The run was stopped while the call was open (a ``). | -| `stranded` | **yes** | The agent could not reach the model to report back: its next attempt on that host failed to dial, and the call ended there (`endStrandedTools`). | +| `stranded` | **yes** | The agent finished the tool and could not reach the model to report back: its next attempt on that host failed to dial, and the call ends there (`endStrandedTools`). The duration is the tool's, not the outage's. `resultSeq` is absent when no result ever came, and set when one arrived late (see below). | **`stranded` is the fifth closing case, and this spec missed it until review.** `Build` runs `endStrandedTools` AFTER the pairing (`buildAgents` then `endStrandedTools`, both from `Build`), and it sets a tool's `End` on a non-`Lost` span without setting `Lost` — so the timeline has a call with an end and -none of the four reasons above. A pairing that stops at those four leaves such a call with an +none of the other four reasons. A pairing that stops at those four leaves such a call with an empty reason, which T35 says cannot happen on an exported record, and T17 would then drop a call that did in fact finish. +**`stranded` WINS over `result` when both apply.** `endStrandedTools`'s own doc says the result +"reaches the proxy only once the network is back, hours later, or never". When it does arrive, +`buildAgents` first ends the call on it and the stranding pass then moves `End` back to the +dial — so the call satisfies both reasons, and the one that EXPLAINS the duration is the one to +export. Concretely, for a stranded call whose result arrived late: + +- `endReason` is `stranded`, not `result`. +- `resultSeq` **stays present**: a request did carry the result, and a consumer joining on it + is right to. +- `durationMs` ends at the **dial**, not at `resultSeq`. This is the one case where the two + disagree, and the `callSeq`/`resultSeq` row would otherwise imply the latter. + +So whichever option below is chosen must apply the stranding cut to a call **with** a result as +well as one without. **Open** with the rest of this case. + Implementation note: the pairing package must therefore learn this case, which it does **not** do today — `core/session/toolpair` was extracted from `buildAgents` only, and `endStrandedTools` needs `t.Agents`, `t.Turns` and `t.Hosts`, which the pairing has none of. @@ -584,7 +627,7 @@ export apply the stranding pass itself over the records it is about to write. Th the pairing's inputs as they are; the first keeps the reason where the other four live. **No `lost` value.** An earlier draft had one for "open with no reason established", and no -case produces it: a call with none of the four reasons above still has `End == nil`, which +case produces it: a call with none of the five reasons above still has `End == nil`, which means **still open**, and T17 does not export an open call at all. A vocabulary entry a producer never emits is a trap for the consumer — it switches on it and ships a dead branch, or reads it as the catch-all for "no result" and misses the three real reasons. Add one only @@ -612,8 +655,8 @@ therefore yields no record, rather than a record with a missing start. ### 4.5 Documentation -T22. `docs/session-export.md` states the §3.4 parallel-window rule and the T16 `lost` rule -beside the record, because both cost real accuracy and neither is guessable from the field +T22. `docs/session-export.md` states the §3.4 parallel-window rule and T16's end-versus-result +rule beside the record, because both cost real accuracy and neither is guessable from the field list. The doc fix that precedes this spec already states them for `/timeline`. --- @@ -633,12 +676,12 @@ list. The doc fix that precedes this spec already states them for `/timeline`. Four changes from the first draft, all so a record can be placed and ordered on its own: -- **`startedAt` replaces `at`.** The first draft carried the *end*, so a `lost` call — which +- **`startedAt` replaces `at`.** The first draft carried the *end*, so a call with no result — which has no duration — could not be placed on a timeline at all. A start exists for every call (the asking response's end), so it is always present; the end is `startedAt + durationMs` when there is one. - **`seq` is gone.** It copied the result event's seq, which collides with the event record's - own `seq` and is undefined for a `lost` call. `callSeq` and `resultSeq` carry the two real + own `seq` and is undefined for a call that never got a result. `callSeq` and `resultSeq` carry the two real anchors. - **`callIndex` added.** The call's position among the calls its response asked for, from 0, in the order the model asked. Without it a batch cannot be put back in order. @@ -650,13 +693,13 @@ Four changes from the first draft, all so a record can be placed and ordered on |---|---| | `kind` | T19. The discriminator. Always `"toolCall"` here; an event record carries `"event"` (T28). | | `sessionId` | T5. The session, as the client asserted it. Not a unit of work until #1997. | -| `startedAt` | T29. When the call started — the asking response's end. **Always present**, so a `lost` call can still be placed. | +| `startedAt` | T29. When the call started — the asking response's end. **Always present**, so a call with no result can still be placed. | | `tool` | T10/T10a/T11. A built-in name verbatim; an `mcp` or `unknown` name as a keyed hash. Bounded by T12. | | `toolSource` | T4. `builtin`, `mcp` or `unknown` — a closed vocabulary, exported by naming the values. Only `builtin` exports its name in clear (T10a). | | `callSeq` / `resultSeq` | The response that asked and the request that carried the result. Lets a consumer rejoin a `toolCall` to the event records either side **without a tool-use id** (T8). `resultSeq` is absent when nothing carried a result. | | `callIndex` | T30. The call's position among its response's calls, from 0, in the order the model asked. | -| `durationMs` | T18. The paired duration. Absent when `endReason` says there was no result (T16, T31). | -| `endReason` | T31. Closed vocabulary: how the call ended. `lost` derives from it, so no separate flag. | +| `durationMs` | T18. The paired duration. Absent exactly when `endReason.HasEnd()` is false (T16, T31) — so present for `stranded`, which has an end without a result. | +| `endReason` | T31. Closed vocabulary: how the call ended, and the only thing that says whether a result arrived. No separate flag derives from it. | | `isError` | Whether the result was an error. The tool failure rate, per tool. | | `inputBytes` / `resultBytes` | T14. Effort proxies, never content. Bytes, as the names say. | | `modelMs` | T23. The asking response's measured duration — model time, per call, no join needed. | @@ -730,7 +773,7 @@ marker record when it is hit. 7.1 **The proxy sees the gap around a tool, never inside it.** A correction to the first draft of this spec, which claimed local tool calls are invisible. They are not: a span runs from the -asking response's end to the next request carrying the result (the span-emission loop and the closing pass in `buildAgents`, now `toolpair.Calls` and `toolpair.closeCalls`), +asking response's end to the next request carrying the result (the span-emission loop and the closing pass in `buildAgents`; #2041's `toolpair.Calls` and `closeCalls`), and **both boundaries are model traffic**, so a tool that touches nothing on the network still gets a span. §3.1's `Edit` at p50 35 ms is such a call. The earlier claim confused this record with #1810 §9's gap classification, where "no proxied traffic landed inside the gap" is how a From deeb12cbb70666a5a74854cec412755ff914883a Mon Sep 17 00:00:00 2001 From: Mariusz Sabath Date: Fri, 20 Nov 2026 20:52:54 -0500 Subject: [PATCH 09/12] Docs: Add the stranded caveats the last commit claimed were already there The sixth revision's message said the guide now warns about a stranded span's missing result and about only a failed dial being caught. It did not. The edit landed on the duration and stopped there, and the message described the intent rather than the result -- the second time in this PR, so: verified by reading the rendered sections back this time, not by grepping for a phrase. The guide now carries both caveats. An end and no `lost` does not mean a result came back, so an error rate or a result-size join over those spans is counting something else; and only a failed dial is cut, so an upstream 502 or a client hanging up mid-handshake leaves its outage inside the span, and a long tail can still be an outage rather than a slow tool. Four smaller corrections, all from the same review: The stranding cut is matched by HOST, not by agent. Its documented shortcut is that a blip in any other agent, a parent included, ends a tool whose own branch had no success first -- so a subagent's long Bash can be cut where the main agent retried, and two agents stranded by one outage both end at its first failure. That bounds what `stranded` claims: cut at an outage this agent could not be proven to have survived, not at this agent's own failure. Worth naming, since the reason exists to explain a duration. The "every time" and "its next attempt on that host" wording overstated both and is gone. The reasons-that-can-occur-without-a-result count is recounted with `stranded` among them -- also claimed done last time and not done. `stranded` wins over `background` as well as over `result`: the cut can land before an acknowledgement too, and the reason that explains the duration wins. The durationMs row no longer says `stranded` has "an end without a result", which contradicted the late-result paragraph above it. T11a settles the HMAC key's lifecycle, which T11 left unsaid (CodeRabbit). Generated once on first use, 32 random bytes, 0600, read back unchanged, and never rotated: a key that changes re-labels every MCP tool from that point and splits one tool's history into two series a consumer cannot rejoin. A lost key is the same event, and the old labels are unrecoverable -- which is the property that made a keyed hash safe to ship. Assisted-By: Claude (Anthropic AI) Signed-off-by: Mariusz Sabath --- docs/session-export.md | 17 ++++-- ...26-11-19-tool-call-export-record-design.md | 56 +++++++++++++++---- 2 files changed, 57 insertions(+), 16 deletions(-) diff --git a/docs/session-export.md b/docs/session-export.md index 99818370f..2554d9355 100644 --- a/docs/session-export.md +++ b/docs/session-export.md @@ -166,11 +166,18 @@ Two rules before you total those spans, both of which cost real accuracy: `end`. A third kind needs no filter: a call whose agent finished the tool and then could not reach - the model to report back — a lid closed, a VPN dropped — gets its `end` from that first - failed attempt and is **not** marked `lost`. **Keep it.** Its duration is the tool's, not the - outage's: the proxy cuts the span at the failed dial precisely so the outage is excluded, and - against Claude Code's own transcripts that dial landed within 0.1 s of the tool's real end - every time. Without it, one 9-minute `Bash` drew as 8 hours. + the model to report back — a lid closed, a VPN dropped — gets its `end` from the first failed + dial and is **not** marked `lost`. **Keep it.** Its duration is the tool's, not the outage's: + Cortex cuts the span at that dial precisely so the outage is excluded, and against Claude + Code's own transcripts the dial landed within a fraction of a second of the tool's real end. + Without the cut, one 9-minute `Bash` drew as 8 hours. + + Two things to know about that kind before you join on it. **Its result may be missing, or may + have arrived much later** — an `end` and no `lost` does not mean a result came back, so an + error rate or a result-size join over these spans is counting something else. And **only a + failed dial is caught this way**: an upstream 502, or a client that hung up mid-handshake, + leaves its outage inside the span. So a long tail can still be an outage of another kind + rather than a slow tool. An export is **content-free**. It is **not anonymous**. It keeps hosts, models, session ids and timestamps. Together these show who called what, and when. Read an export before diff --git a/docs/superpowers/specs/2026-11-19-tool-call-export-record-design.md b/docs/superpowers/specs/2026-11-19-tool-call-export-record-design.md index a1788db9e..5c389b9c6 100644 --- a/docs/superpowers/specs/2026-11-19-tool-call-export-record-design.md +++ b/docs/superpowers/specs/2026-11-19-tool-call-export-record-design.md @@ -12,13 +12,13 @@ This one narrows its §7.2 `toolCallNames` item and supersedes the shape propose Items marked **Open** must be decided before the work they belong to begins. Nothing here is merged. -**39 numbered requirements T0–T37, plus the lettered T10a that narrows T10**, phrased as plain declaratives: a requirement says what +**40 numbered requirements T0–T37, plus the lettered T10a and T11a that narrow T10 and T11**, phrased as plain declaratives: a requirement says what the system does once built. The numbering is stable so it can be cited in review and in the PRs that implement it, and is therefore in the order the requirements were ADDED, not document order: T0 sits in §2.0 because it bounds the whole spec, T23–T27 and T36–T37 in §4.3a, T28–T33 and T35 in §4.4 and §4.4a, and T34 in §7.1b. -Revised twice on 2026-11-19, once on 2026-11-20 and three times on 2026-11-21. +Revised twice on 2026-11-19, once on 2026-11-20 and four times on 2026-11-21. **First revision**, after review raised two things the first draft buried. §2.0 is new: every field here is **already collected and most of it is already on disk**, so this is a projection @@ -107,6 +107,17 @@ half-wired, which is the whole of this round. - **"None of the four reasons ... has `End == nil`"** was the sentence the fifth revision's own commit message called false, still sitting 15 lines below the paragraph disproving it. +**Seventh revision.** The sixth said the guide now warned about a stranded span's missing +result and about only a failed dial being caught. It did not — the edit landed on the duration +and not on the caveats, and the commit message described the intent rather than the result. The +guide now carries both warnings. Also: the stranding cut is matched by HOST, not by agent, so +its documented shortcut can cut a subagent's running tool at another agent's blip — named now, +because `stranded` claims to explain a duration; the reasons-without-a-result count is +recounted with `stranded` among them; `stranded` wins over `background` as well as `result`; +and T11a settles the HMAC key's lifecycle, which T11 left open — generated once, never +rotated, because a key that changes re-labels every MCP tool and splits one tool's history into +two series a consumer cannot rejoin. + **Code references name symbols, not line numbers.** Every `agents.go:NNN` pin in the first four revisions was correct when written and wrong a day later: #2041 moves all of them into `core/session/toolpair`. A spec that cites `toolpair.Calls`, `closeCalls`, `preview` or @@ -426,6 +437,16 @@ digest is not enough: the namespace makes the input guessable, so an unkeyed has known vendor name is reversible by trying candidates. The cost is that counts do not join across installs, which is the right trade for a name that may identify a customer. Hashing happens **before** T12's cut, so the cut never splits a digest. +T11a. **The key is generated once and kept**, because a key that changes makes T1 and T4 fail +for MCP traffic: the same tool would carry a different label in every export, and no consumer +could total its time across two of them. Concretely — created on first use by an export that +needs it, 32 bytes from `crypto/rand`, written `0600` beside the install's other state, read +back unchanged thereafter, following the read-or-create pattern `cmd/cortex`'s gateway token +already uses. **It does not rotate.** Rotation would silently re-label every MCP tool from that +point, splitting one tool's history into two series that a consumer cannot rejoin; anyone who +needs new labels deletes the key and knows the old exports no longer compare. A lost key is the +same event, and the honest answer to it is that the old labels are unrecoverable — which is the +property that made a keyed hash safe to ship in the first place. T12. A tool name is bounded on the way out by the same `exportLabel` every wire-sourced string in a record passes: control runes removed, then truncated to `maxLabelLen` on a rune boundary. `maxLabelLen` is **96 bytes**, not runes (`sessionexport.maxLabelLen`), so a multi-byte name @@ -454,8 +475,8 @@ carries no separate `lost` flag. **`stranded` is the exception, and it is not a contradiction**: that call HAS an end, so it has a duration. Its agent finished the tool and then could not reach the model to report back, and -the proxy cuts the span at that first failed dial — which measured within 0.1 s of the tool's -real end on every session checked against Claude Code's transcripts. So the rule is "no end, no +the proxy cuts the span at that first failed dial — which measured within a fraction of a +second of the tool's real end on the sessions checked against Claude Code's transcripts. So the rule is "no end, no duration", not "no result, no duration": `durationMs` is present exactly when `endReason.HasEnd()`, which is `result`, `background` and `stranded`. @@ -595,7 +616,7 @@ T31. A `toolCall` record carries `endReason`, a closed vocabulary naming how the | `ended_run` | no | By design no result: the call ends the run or hands control back (`SubagentHandback`, `StructuredOutput`, `end_subtask`). | | `abandoned` | no | The agent moved on without the result (`lastTurn`). | | `stopped` | no | The run was stopped while the call was open (a ``). | -| `stranded` | **yes** | The agent finished the tool and could not reach the model to report back: its next attempt on that host failed to dial, and the call ends there (`endStrandedTools`). The duration is the tool's, not the outage's. `resultSeq` is absent when no result ever came, and set when one arrived late (see below). | +| `stranded` | **yes** | The agent finished the tool and could not reach the model to report back: the first failed dial to a model host after the call started ends it (`endStrandedTools`). The duration is the tool's, not the outage's. `resultSeq` is absent when no result ever came, and set when one arrived late (see below). | **`stranded` is the fifth closing case, and this spec missed it until review.** `Build` runs `endStrandedTools` AFTER the pairing (`buildAgents` then `endStrandedTools`, both from `Build`), and it sets a tool's @@ -604,13 +625,15 @@ none of the other four reasons. A pairing that stops at those four leaves such a empty reason, which T35 says cannot happen on an exported record, and T17 would then drop a call that did in fact finish. -**`stranded` WINS over `result` when both apply.** `endStrandedTools`'s own doc says the result -"reaches the proxy only once the network is back, hours later, or never". When it does arrive, +**`stranded` WINS over `result` — and over `background` — when both apply.** +`endStrandedTools`'s own doc says the result "reaches the proxy only once the network is back, +hours later, or never". When it does arrive, `buildAgents` first ends the call on it and the stranding pass then moves `End` back to the dial — so the call satisfies both reasons, and the one that EXPLAINS the duration is the one to export. Concretely, for a stranded call whose result arrived late: -- `endReason` is `stranded`, not `result`. +- `endReason` is `stranded`, not `result`. The same holds for a `background` acknowledgement + the cut lands before: the reason that explains the duration wins. - `resultSeq` **stays present**: a request did carry the result, and a consumer joining on it is right to. - `durationMs` ends at the **dial**, not at `resultSeq`. This is the one case where the two @@ -619,6 +642,16 @@ export. Concretely, for a stranded call whose result arrived late: So whichever option below is chosen must apply the stranding cut to a call **with** a result as well as one without. **Open** with the rest of this case. +**The cut is matched by host, not by agent, and that bounds how precisely `stranded` explains a +duration.** `endStrandedTools` names no agent: its documented `shortcut:` is that a blip in any +other agent — a parent included — ends a tool whose own branch had no success first, so a +subagent's long-running `Bash` can be cut at the moment the main agent retried. Two agents +stranded by one outage therefore both end at its first failure. For a fleet latency figure this +is the right trade, since the alternative is a span that runs to the session's last unrelated +event; for a single call it means `stranded` says "cut at an outage this agent could not be +proven to have survived", not "cut at this agent's own failure". The upstream note says to +revisit if CONNECTs ever carry the agent id. + Implementation note: the pairing package must therefore learn this case, which it does **not** do today — `core/session/toolpair` was extracted from `buildAgents` only, and `endStrandedTools` needs `t.Agents`, `t.Turns` and `t.Hosts`, which the pairing has none of. @@ -630,8 +663,9 @@ the pairing's inputs as they are; the first keeps the reason where the other fou case produces it: a call with none of the five reasons above still has `End == nil`, which means **still open**, and T17 does not export an open call at all. A vocabulary entry a producer never emits is a trap for the consumer — it switches on it and ships a dead branch, -or reads it as the catch-all for "no result" and misses the three real reasons. Add one only -alongside the code that returns it. +or reads it as the catch-all for "no result" and misses the four reasons that can occur +without one — `ended_run`, `stopped`, `abandoned`, and `stranded` where no late result +arrived. Add one only alongside the code that returns it. These are the timeline's own closing cases (the closing pass), named rather than collapsed into one `Lost` bool: "the agent moved on" and "the call ends the run by design" are different @@ -698,7 +732,7 @@ Four changes from the first draft, all so a record can be placed and ordered on | `toolSource` | T4. `builtin`, `mcp` or `unknown` — a closed vocabulary, exported by naming the values. Only `builtin` exports its name in clear (T10a). | | `callSeq` / `resultSeq` | The response that asked and the request that carried the result. Lets a consumer rejoin a `toolCall` to the event records either side **without a tool-use id** (T8). `resultSeq` is absent when nothing carried a result. | | `callIndex` | T30. The call's position among its response's calls, from 0, in the order the model asked. | -| `durationMs` | T18. The paired duration. Absent exactly when `endReason.HasEnd()` is false (T16, T31) — so present for `stranded`, which has an end without a result. | +| `durationMs` | T18. The paired duration. Absent exactly when `endReason.HasEnd()` is false (T16, T31) — so present for `stranded`, whose end comes from the failed dial whether or not a result ever arrived. | | `endReason` | T31. Closed vocabulary: how the call ended, and the only thing that says whether a result arrived. No separate flag derives from it. | | `isError` | Whether the result was an error. The tool failure rate, per tool. | | `inputBytes` / `resultBytes` | T14. Effort proxies, never content. Bytes, as the names say. | From 29f8b8bade27cbe548b62a330d8aaa06602bcc0e Mon Sep 17 00:00:00 2001 From: Mariusz Sabath Date: Fri, 20 Nov 2026 21:09:11 -0500 Subject: [PATCH 10/12] Docs: Say resultSeq answers whether a result arrived, and that T31 is not ready Two must-fixes carried forward from the pass that landed 84 seconds before the previous push. The endReason field-table row said it was "the only thing that says whether a result arrived". Four other places in the same revision contradict it, and they are right: stranded occurs both with a late result and without one, so resultSeq is what answers that question. The row was introduced while fixing an unrelated nit about a derived flag, and it was the last place still disagreeing -- in the table an implementer actually builds from. Section 10 claimed nothing blocks implementation while T31's stranding route sat Open in two paragraphs and in no 10.2 row. It has the row now, names T31 as the one requirement not ready, and states the interim behaviour: a stranded call matches abandoned and therefore exports with no duration, and since a closed lid strands the long call rather than the short one, that undercount lands in the heavy tail. So 10.2 now says not to publish a tier table from these records until #2046 closes. Three smaller ones from the same review. The stranded row dropped the "no success in between" condition, which is what makes the dial a tool's end rather than any later failure. T11a's read-or-create is last-writer-wins for an export where it is not for the gateway -- two exports racing on first use would each generate a key and re-label the earlier one's MCP tools -- so the implementation note says to create under a lock or at service start. And the guide said a stranded span's duration IS the tool's, where the spec now correctly says usually: the cut is matched by host, so another agent's failure can end a call that was still running. The PR body is rewritten. It still carried the first draft's numbers -- 22 requirements, 4.1:1, 15.7% -- and listed four items as open that 10.1 settled, so a squash-merge from it would have shipped the oldest figures in the PR. Assisted-By: Claude (Anthropic AI) Signed-off-by: Mariusz Sabath --- docs/session-export.md | 10 +++-- ...26-11-19-tool-call-export-record-design.md | 41 +++++++++++++++---- 2 files changed, 38 insertions(+), 13 deletions(-) diff --git a/docs/session-export.md b/docs/session-export.md index 2554d9355..4dfc65878 100644 --- a/docs/session-export.md +++ b/docs/session-export.md @@ -167,10 +167,12 @@ Two rules before you total those spans, both of which cost real accuracy: A third kind needs no filter: a call whose agent finished the tool and then could not reach the model to report back — a lid closed, a VPN dropped — gets its `end` from the first failed - dial and is **not** marked `lost`. **Keep it.** Its duration is the tool's, not the outage's: - Cortex cuts the span at that dial precisely so the outage is excluded, and against Claude - Code's own transcripts the dial landed within a fraction of a second of the tool's real end. - Without the cut, one 9-minute `Bash` drew as 8 hours. + dial and is **not** marked `lost`. **Keep it.** Its duration is usually the tool's rather than + the outage's: Cortex cuts the span at that dial precisely so the outage is excluded, and + against Claude Code's own transcripts the dial landed within a fraction of a second of the + tool's real end. Without the cut, one 9-minute `Bash` drew as 8 hours. "Usually" because the + cut is matched by host and not by agent, so another agent's failure can end a call that was + still running — a subagent's long `Bash` cut where the main agent retried. Two things to know about that kind before you join on it. **Its result may be missing, or may have arrived much later** — an `end` and no `lost` does not mean a result came back, so an diff --git a/docs/superpowers/specs/2026-11-19-tool-call-export-record-design.md b/docs/superpowers/specs/2026-11-19-tool-call-export-record-design.md index 5c389b9c6..d0266bc9b 100644 --- a/docs/superpowers/specs/2026-11-19-tool-call-export-record-design.md +++ b/docs/superpowers/specs/2026-11-19-tool-call-export-record-design.md @@ -9,8 +9,8 @@ Background: #1810 (the export's own spec and tracker), #1830/#1812/#1831 (what s Parent spec: [`2026-11-13-session-event-export-design.md`](2026-11-13-session-event-export-design.md). This one narrows its §7.2 `toolCallNames` item and supersedes the shape proposed there. -Items marked **Open** must be decided before the work they belong to begins. Nothing here -is merged. +Items marked **Open** must be decided before the work they belong to begins; §10.2 collects +them, and says which requirement is not ready because of one. Nothing here is merged. **40 numbered requirements T0–T37, plus the lettered T10a and T11a that narrow T10 and T11**, phrased as plain declaratives: a requirement says what the system does once built. The numbering is stable so it can be cited in review and in the @@ -18,7 +18,7 @@ PRs that implement it, and is therefore in the order the requirements were ADDED order: T0 sits in §2.0 because it bounds the whole spec, T23–T27 and T36–T37 in §4.3a, T28–T33 and T35 in §4.4 and §4.4a, and T34 in §7.1b. -Revised twice on 2026-11-19, once on 2026-11-20 and four times on 2026-11-21. +Revised twice on 2026-11-19, once on 2026-11-20 and five times on 2026-11-21. **First revision**, after review raised two things the first draft buried. §2.0 is new: every field here is **already collected and most of it is already on disk**, so this is a projection @@ -118,6 +118,16 @@ and T11a settles the HMAC key's lifecycle, which T11 left open — generated onc rotated, because a key that changes re-labels every MCP tool and splits one tool's history into two series a consumer cannot rejoin. +**Eighth revision**, two carried-forward must-fixes. The `endReason` field-table row said it +was "the only thing that says whether a result arrived" — which four other places in the same +revision correctly contradict, because `stranded` occurs both with a late result and without +one. `resultSeq` answers that question; the row was the last place still disagreeing, and it is +the place an implementer builds from. And §10 claimed nothing blocks implementation while T31's +stranding route sat **Open** in two paragraphs and in no §10.2 row: it now has the row, names +T31 as the one requirement not ready, and states the interim behaviour — a stranded call +exports as `abandoned` with no duration, biasing the heavy tail — so nobody builds a tier table +on it before #2046 closes. + **Code references name symbols, not line numbers.** Every `agents.go:NNN` pin in the first four revisions was correct when written and wrong a day later: #2041 moves all of them into `core/session/toolpair`. A spec that cites `toolpair.Calls`, `closeCalls`, `preview` or @@ -442,7 +452,10 @@ for MCP traffic: the same tool would carry a different label in every export, an could total its time across two of them. Concretely — created on first use by an export that needs it, 32 bytes from `crypto/rand`, written `0600` beside the install's other state, read back unchanged thereafter, following the read-or-create pattern `cmd/cortex`'s gateway token -already uses. **It does not rotate.** Rotation would silently re-label every MCP tool from that +already uses (`writeGatewayFiles`). One difference from that precedent to settle in +implementation: the gateway writes its token once at boot, where two exports racing on first +use could each generate a key and the later write would win, re-labelling the earlier export's +MCP tools. Create under a lock, or create it at service start rather than on first use. **It does not rotate.** Rotation would silently re-label every MCP tool from that point, splitting one tool's history into two series that a consumer cannot rejoin; anyone who needs new labels deletes the key and knows the old exports no longer compare. A lost key is the same event, and the honest answer to it is that the old labels are unrecoverable — which is the @@ -616,7 +629,7 @@ T31. A `toolCall` record carries `endReason`, a closed vocabulary naming how the | `ended_run` | no | By design no result: the call ends the run or hands control back (`SubagentHandback`, `StructuredOutput`, `end_subtask`). | | `abandoned` | no | The agent moved on without the result (`lastTurn`). | | `stopped` | no | The run was stopped while the call was open (a ``). | -| `stranded` | **yes** | The agent finished the tool and could not reach the model to report back: the first failed dial to a model host after the call started ends it (`endStrandedTools`). The duration is the tool's, not the outage's. `resultSeq` is absent when no result ever came, and set when one arrived late (see below). | +| `stranded` | **yes** | The agent finished the tool and could not reach the model to report back: the first failed dial to a model host after the call started ends it, when no turn of that agent or an agent below it succeeded in between (`endStrandedTools`). The duration is the tool's, not the outage's. `resultSeq` is absent when no result ever came, and set when one arrived late (see below). | **`stranded` is the fifth closing case, and this spec missed it until review.** `Build` runs `endStrandedTools` AFTER the pairing (`buildAgents` then `endStrandedTools`, both from `Build`), and it sets a tool's @@ -733,7 +746,7 @@ Four changes from the first draft, all so a record can be placed and ordered on | `callSeq` / `resultSeq` | The response that asked and the request that carried the result. Lets a consumer rejoin a `toolCall` to the event records either side **without a tool-use id** (T8). `resultSeq` is absent when nothing carried a result. | | `callIndex` | T30. The call's position among its response's calls, from 0, in the order the model asked. | | `durationMs` | T18. The paired duration. Absent exactly when `endReason.HasEnd()` is false (T16, T31) — so present for `stranded`, whose end comes from the failed dial whether or not a result ever arrived. | -| `endReason` | T31. Closed vocabulary: how the call ended, and the only thing that says whether a result arrived. No separate flag derives from it. | +| `endReason` | T31. Closed vocabulary: how the call ended, and what its duration means. It does **not** say whether a result arrived — `resultSeq` does, because `stranded` occurs both with a late result and without one. No separate flag derives from it. | | `isError` | Whether the result was an error. The tool failure rate, per tool. | | `inputBytes` / `resultBytes` | T14. Effort proxies, never content. Bytes, as the names say. | | `modelMs` | T23. The asking response's measured duration — model time, per call, no join needed. | @@ -917,8 +930,18 @@ its frequency — but the *rate* is not a constant to design against. | §7.1a | Whether to quantify the approval-wait error against local Claude Code transcripts before publishing any density figure. Author's view: yes, and it is cheap. | | T32 | Whether the job record gets specified in this cycle or a later one. The undercount is marked either way. | | §7.1b | Whether an optional client-stated execution site is worth a field, for a client that knows where its tools ran. Not a proxy-side inference under any circumstance. | - -Nothing above blocks implementation: §6 Option A, T11's keyed hash, T20's per-item switch and -§6.4's marker record are all settled, and T28–T34 are normative. +| T31 / §2.2 | **Where the stranding cut runs** — carry the host-failure times into the pairing, or apply the cut in the export over the records it is about to write. Author leans the former, for §6's reason: a closing case that exists twice drifts. Tracked as #2046 so it does not block the record. | + +**T31 is the one requirement above that is NOT ready to implement**, and §10.2's row above says +why: `stranded` is specified, but where its cut runs is not, and the pairing package cannot see +the case at all today. Everything else is settled — §6 Option A, T10a/T11/T11a's hashing, +T20's per-item switch, §6.4's marker record — and T28–T30 and T32–T37 are normative. + +**Interim behaviour, until T31's route is decided:** a stranded call matches `abandoned`, +because the agent's later request is after the call's start, and an `abandoned` call carries no +duration (T16). So the export silently drops the timing of a call that did finish and was +measured accurately. A closed laptop lid strands the long call rather than the short one, so +that undercount lands in the heavy tail — the part §3.3's bands and any density figure turn on. +Do not publish a tier table from `toolCall` records until this is closed. Assisted-By: Claude (Anthropic AI) From c30740f8087e740d44289b5a226421a33bd17714 Mon Sep 17 00:00:00 2001 From: Mariusz Sabath Date: Sat, 21 Nov 2026 09:17:30 -0500 Subject: [PATCH 11/12] =?UTF-8?q?Docs:=20State=20all=20three=20stranded=20?= =?UTF-8?q?outcomes,=20and=20fix=20the=20=C2=A72.2=20route=20pointers?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The eighth revision's interim-behaviour paragraph said a stranded call exports as `abandoned` with no duration. That is one of three outcomes, and the smallest. Probed against #2041's `closeCalls` at `410966e`: - `closeCalls` checks for a result before every no-result reason, so a stranded call whose result arrived late exports as `result` with the outage inside `durationMs` — the 9-minute `Bash` that drew as 8 hours. That INFLATES the heavy tail, the opposite direction from the undercount the paragraph described. - A failed dial is a CONNECT with no `Inference`, which `pairCalls` skips, so it never advances `lastAsk`. A stranded call whose session never resumed keeps the zero reason and T17 drops the record whole. - Only the middle case — the agent asked again without the result — reads as `abandoned`. §10 now states all three in a table, and names the first as the one that matters for a tier table. Also this round: - §2.2 names `endStrandedTools` as a fifth case that runs after the pairing, so "reading that code is what sized this spec" covers the case that later broke it. The two references that pointed at §2.2 for the stranding route now point at §4.4a, where the two options are. - The `stranded` row drops the last unqualified "the duration is the tool's": the cut is matched by host, so another agent's blip can end a call early. The guide already says "usually"; the row now agrees. - "fifth closing case" is scoped to §2.2's count, since #2041's `toolpair` counts five of its own and `stranded` would be its sixth. - T35 keeps a measured zero for `inputBytes`, `resultBytes` and `gapMs` distinguishable from an unmeasured field. T14 reserves absence for "not measured" and T24 for an agent's first turn, so `omitempty` would turn an empty result into "not measured" and a sub-millisecond gap into "first turn". - T37's `user` gap is qualified: a background job from an earlier `Bash` can span one, so a duty cycle from `user` gaps is an upper bound on idleness. T27's sandbox-occupancy claim is scoped to foreground work. - "T31 is the one requirement NOT ready" is scoped to T28–T37 and says why T32's open row does not count. Assisted-By: Claude (Anthropic AI) Signed-off-by: Mariusz Sabath --- ...26-11-19-tool-call-export-record-design.md | 81 +++++++++++++++---- 1 file changed, 65 insertions(+), 16 deletions(-) diff --git a/docs/superpowers/specs/2026-11-19-tool-call-export-record-design.md b/docs/superpowers/specs/2026-11-19-tool-call-export-record-design.md index d0266bc9b..860e5928a 100644 --- a/docs/superpowers/specs/2026-11-19-tool-call-export-record-design.md +++ b/docs/superpowers/specs/2026-11-19-tool-call-export-record-design.md @@ -124,9 +124,23 @@ revision correctly contradict, because `stranded` occurs both with a late result one. `resultSeq` answers that question; the row was the last place still disagreeing, and it is the place an implementer builds from. And §10 claimed nothing blocks implementation while T31's stranding route sat **Open** in two paragraphs and in no §10.2 row: it now has the row, names -T31 as the one requirement not ready, and states the interim behaviour — a stranded call -exports as `abandoned` with no duration, biasing the heavy tail — so nobody builds a tier table -on it before #2046 closes. +T31 as the one requirement not ready, and states the interim behaviour, so nobody builds a tier +table on it before #2046 closes. + +**Ninth revision**, and the interim behaviour the eighth revision added was itself wrong — it +described one of three outcomes as the whole loss. Probed against #2041's `closeCalls` at +`410966e`: a result wins over every no-result reason and a failed dial is a CONNECT that +`pairCalls` skips, so a stranded call whose result arrives late exports as `result` with the +**outage inside `durationMs`** — the 9-minute `Bash` that drew as 8 hours — one the agent moved +on from exports as `abandoned` with no duration, and one whose session never resumed stays open +and is not exported at all (T17). The first *inflates* the heavy tail, which is the opposite +direction from the undercount the eighth revision described, and the one that matters for a +tier table. §10 now states all three. Also this round: §2.2 names `endStrandedTools` as a fifth +case that runs after the pairing, so the two places that pointed at §2.2 for its route now point +at §4.4a where the options actually live; the `stranded` row drops the last unqualified "the +duration is the tool's"; T35 keeps a measured zero for `inputBytes`, `resultBytes` and `gapMs` +distinguishable from an unmeasured field, which `omitempty` would not; and T37's `user` gap is +qualified for a background job still running behind it. **Code references name symbols, not line numbers.** Every `agents.go:NNN` pin in the first four revisions was correct when written and wrong a day later: #2041 moves all of them into @@ -255,6 +269,10 @@ stated here rather than summarised: (the `Job` the pairing records, closed by its `` or by `TaskStop`). An earlier draft of this bullet said the span is extended, which contradicted T32 and would send an implementer of §6 looking for code that does not exist. +- **A fifth case runs after the pairing, not inside it.** `Build` calls `buildAgents` and then + `endStrandedTools`, which closes a tool whose agent could not reach the model to report back + (§4.4a, T31). It is the one case this spec missed until review, and the one whose route is + still open — §4.4a holds both options and §10.2 tracks it as #2046. ### 2.3 The gap, and why it is structural @@ -494,13 +512,13 @@ duration", not "no result, no duration": `durationMs` is present exactly when `endReason.HasEnd()`, which is `result`, `background` and `stranded`. **Note for #2041:** `toolpair.EndReason.HasEnd()` there is `result || background`. Whichever of -§2.2's two options lands, `HasEnd` has to include `stranded`, or `omitempty` drops the duration +§4.4a's two options lands, `HasEnd` has to include `stranded`, or `omitempty` drops the duration of a stranded call that measured 0 ms. T17. A span still open when the walk reads it is not exported. An unfinished call has no duration to report, and a record that appears with one later would make the export non-reproducible. T18. Durations are milliseconds, truncated, matching `Record`. -T35. **Six `toolCall` fields are always written, even at zero**, because for them zero is a +T35. **Nine `toolCall` fields are always written, even at zero**, because for them zero is a real value rather than "not recorded" — the opposite of the event record's rule: | Always written | Why zero is real | @@ -510,6 +528,13 @@ real value rather than "not recorded" — the opposite of the event record's rul | `callIndex` | the FIRST call of every batch is 0, and T34 makes it a join key | | `durationMs` on a reason that `HasEnd` | a result clamped to its call's start is a real 0 ms call (§3.5), and T31 requires a duration there | | `endReason` | every exported call has one | +| `inputBytes`, `resultBytes` when measured | an empty result or an argument-free call is a real 0, and T14 reserves absence for "not measured" | +| `gapMs` when measured | a gap under half a millisecond rounds to a real 0, and T24 reserves absence for an agent's first turn | + +The last two rows are why these fields cannot use `omitempty`: T14 and T24 each give absence a +meaning other than zero, so a measured zero has to be distinguishable from an unmeasured field. +An implementation that reaches for `omitempty` on them silently turns an empty result into "not +measured" and a sub-millisecond gap into "first turn". `isError` stays `omitempty`: absent reads as false, which is what it means. The rest of the record keeps the export's zero-means-absent rule (#1810 R3). @@ -565,8 +590,17 @@ fields: T37. **A `tool` gap is not idle time, and must not be added to tool time.** By §4.3a's definition the interval covers what the previous turn's calls were doing, so for a `gapKind` of `tool` it overlaps those calls' own `durationMs`: adding the two counts the same wall clock -twice. Only a `user` gap is time no tool and no model was working, which is why it is the one -that answers a duty cycle. +twice. Only a `user` gap is time no *foreground* tool and no model was working, which is why it +is the one that answers a duty cycle. + +One exception, and it is T32's: a background job started by an earlier `Bash` keeps running +after the acknowledgement that closed that call, so it can span a `user` gap. Such a gap is +still classified `user` — the pairing has no job state to consult, and reclassifying it would +need the job record T32 defers — so a duty cycle computed from `user` gaps is an upper bound on +idleness wherever background jobs are in use. A consumer that needs the exact figure joins the +job records when they exist (§10.2). The same qualification applies to T27's "costs nothing in +sandbox occupancy": it is true of the agent's foreground work, not of a job still running +behind it. **A consumer with the event records can derive all three fields**, so these definitions are the deliverable and the fields are the convenience. A definition that cannot be checked @@ -629,9 +663,12 @@ T31. A `toolCall` record carries `endReason`, a closed vocabulary naming how the | `ended_run` | no | By design no result: the call ends the run or hands control back (`SubagentHandback`, `StructuredOutput`, `end_subtask`). | | `abandoned` | no | The agent moved on without the result (`lastTurn`). | | `stopped` | no | The run was stopped while the call was open (a ``). | -| `stranded` | **yes** | The agent finished the tool and could not reach the model to report back: the first failed dial to a model host after the call started ends it, when no turn of that agent or an agent below it succeeded in between (`endStrandedTools`). The duration is the tool's, not the outage's. `resultSeq` is absent when no result ever came, and set when one arrived late (see below). | +| `stranded` | **yes** | The agent finished the tool and could not reach the model to report back: the first failed dial to a model host after the call started ends it, when no turn of that agent or an agent below it succeeded in between (`endStrandedTools`). Its duration is usually the tool's rather than the outage's; the cut is matched by host, so another agent's blip can end it early (§4.4a). `resultSeq` is absent when no result ever came, and set when one arrived late (see below). | -**`stranded` is the fifth closing case, and this spec missed it until review.** `Build` runs +**`stranded` is the fifth case beyond a plain result that §2.2 lists, and this spec missed it +until review.** (#2041's `toolpair` doc comment counts five of its own, because it treats the +clamped early result as a case rather than as part of the end definition; `stranded` would be +its sixth.) `Build` runs `endStrandedTools` AFTER the pairing (`buildAgents` then `endStrandedTools`, both from `Build`), and it sets a tool's `End` on a non-`Lost` span without setting `Lost` — so the timeline has a call with an end and none of the other four reasons. A pairing that stops at those four leaves such a call with an @@ -930,18 +967,30 @@ its frequency — but the *rate* is not a constant to design against. | §7.1a | Whether to quantify the approval-wait error against local Claude Code transcripts before publishing any density figure. Author's view: yes, and it is cheap. | | T32 | Whether the job record gets specified in this cycle or a later one. The undercount is marked either way. | | §7.1b | Whether an optional client-stated execution site is worth a field, for a client that knows where its tools ran. Not a proxy-side inference under any circumstance. | -| T31 / §2.2 | **Where the stranding cut runs** — carry the host-failure times into the pairing, or apply the cut in the export over the records it is about to write. Author leans the former, for §6's reason: a closing case that exists twice drifts. Tracked as #2046 so it does not block the record. | +| T31 / §4.4a | **Where the stranding cut runs** — carry the host-failure times into the pairing, or apply the cut in the export over the records it is about to write. Author leans the former, for §6's reason: a closing case that exists twice drifts. Tracked as #2046 so it does not block the record. | -**T31 is the one requirement above that is NOT ready to implement**, and §10.2's row above says +**T31 is the one requirement in T28–T37 that is NOT ready to implement** (T32's open row is the +job record it deliberately defers, not an unimplementable requirement), and §10.2's row says why: `stranded` is specified, but where its cut runs is not, and the pairing package cannot see the case at all today. Everything else is settled — §6 Option A, T10a/T11/T11a's hashing, T20's per-item switch, §6.4's marker record — and T28–T30 and T32–T37 are normative. -**Interim behaviour, until T31's route is decided:** a stranded call matches `abandoned`, -because the agent's later request is after the call's start, and an `abandoned` call carries no -duration (T16). So the export silently drops the timing of a call that did finish and was -measured accurately. A closed laptop lid strands the long call rather than the short one, so -that undercount lands in the heavy tail — the part §3.3's bands and any density figure turn on. +**Interim behaviour, until T31's route is decided:** the pairing has no stranding pass, so a +stranded call exports as whichever of its other cases applies — and there are three, not one. +`closeCalls` checks for a result before every no-result reason ("a result wins over every +no-result reason"), and a failed dial is a CONNECT with no `Inference`, which `pairCalls` skips, +so it never advances `lastAsk` either: + +| The stranded call | `toolpair` today | What the export would ship | +|---|---|---| +| Its result arrived late | `result`, with `End` at that late result | `durationMs` **including the outage** — the 9-minute `Bash` that drew as 8 hours. The heavy tail is **overcounted**. | +| Its result never arrived, and the agent asked again without it | `abandoned` | No duration (T16). The timing of a call that did finish is dropped. | +| Its result never arrived, and the session never resumed | the zero reason, still open | **Nothing**: T17 does not export an open call, so it is not even counted as an `abandoned` one. | + +The first is the dangerous one for a tier table, because it inflates the tail rather than +shrinking it. A closed laptop lid strands the long call rather than the short one, so all three +errors land in the heavy tail — the part §3.3's bands and any density figure turn on — one +inflating it, one dropping a duration from it, and one hiding the call entirely. Do not publish a tier table from `toolCall` records until this is closed. Assisted-By: Claude (Anthropic AI) From 0cfec9636a507bc4ed514f51b1ad3a65eb7c6641 Mon Sep 17 00:00:00 2001 From: Mariusz Sabath Date: Sat, 21 Nov 2026 09:29:44 -0500 Subject: [PATCH 12/12] Docs: Scope the user gap to its agent, and keep a backgrounded call marked MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two consequences of the last commit, both caught in review. Each is a case where one field was carrying two facts. T37's `user` gap is AGENT-LOCAL. T25 classifies a gap from the previous response of the same agent, so a parent waiting on a human can have a subagent running a tool beside it, and the parent's gap is `user` either way. T25's `user` row said "Nothing else can be happening", which is false in that case and was false before this PR. Two things can run inside a `user` gap: a background job from an earlier `Bash` (T32), and another agent's tool or inference. So a `user` gap answers "was THIS agent idle", and a sandbox duty cycle has to union the busy intervals across agents instead; summed `user` gaps are an upper bound on sandbox idleness, not the figure. T25, T27 and T37 now agree on this. T32a adds `backgrounded`. §4.4a makes `stranded` win over `background` when both apply, which is right — `stranded` explains the duration — but `endReason` holds one value, so that rule silently deleted T32's per-call marking of a backgrounded command, for exactly the calls whose timing is already least trustworthy. A backgrounded `Bash` now sets `backgrounded: true` whatever its `endReason`, and a consumer counting missing job time filters on the field rather than the reason. The `endReason` field-table row no longer claims no separate flag derives from it. Also: the requirement count is 41 (T32a joins T10a and T11a), and §10's normative list names T32a. Assisted-By: Claude (Anthropic AI) Signed-off-by: Mariusz Sabath --- ...26-11-19-tool-call-export-record-design.md | 72 ++++++++++++++----- 1 file changed, 55 insertions(+), 17 deletions(-) diff --git a/docs/superpowers/specs/2026-11-19-tool-call-export-record-design.md b/docs/superpowers/specs/2026-11-19-tool-call-export-record-design.md index 860e5928a..74f838d15 100644 --- a/docs/superpowers/specs/2026-11-19-tool-call-export-record-design.md +++ b/docs/superpowers/specs/2026-11-19-tool-call-export-record-design.md @@ -12,7 +12,7 @@ This one narrows its §7.2 `toolCallNames` item and supersedes the shape propose Items marked **Open** must be decided before the work they belong to begins; §10.2 collects them, and says which requirement is not ready because of one. Nothing here is merged. -**40 numbered requirements T0–T37, plus the lettered T10a and T11a that narrow T10 and T11**, phrased as plain declaratives: a requirement says what +**41 numbered requirements T0–T37, plus the lettered T10a, T11a and T32a that narrow T10, T11 and T32**, phrased as plain declaratives: a requirement says what the system does once built. The numbering is stable so it can be cited in review and in the PRs that implement it, and is therefore in the order the requirements were ADDED, not document order: T0 sits in §2.0 because it bounds the whole spec, T23–T27 and T36–T37 in §4.3a, @@ -142,6 +142,16 @@ duration is the tool's"; T35 keeps a measured zero for `inputBytes`, `resultByte distinguishable from an unmeasured field, which `omitempty` would not; and T37's `user` gap is qualified for a background job still running behind it. +**Tenth revision**, two consequences of the ninth that review caught. Both are cases where one +field was carrying two facts. **T37's `user` gap is agent-local**, which T25's own "nothing else +can be happening" denied: a parent waiting on a human can have a subagent running a tool beside +it, and the parent's gap is `user` either way. A sandbox duty cycle therefore has to union the +busy intervals across agents, and summed `user` gaps are only an upper bound on its idleness — +T25, T27 and T37 now all say so. And **T32a adds `backgrounded`**, because §4.4a makes +`stranded` win over `background` and that silently deleted T32's per-call marking for exactly +the calls whose timing is least trustworthy. The background fact now has its own field and +`endReason` keeps explaining the duration. + **Code references name symbols, not line numbers.** Every `agents.go:NNN` pin in the first four revisions was correct when written and wrong a day later: #2041 moves all of them into `core/session/toolpair`. A spec that cites `toolpair.Calls`, `closeCalls`, `preview` or @@ -567,7 +577,7 @@ response of the same agent** — the one whose end begins the interval: | `gapKind` | That previous response | What the interval was | |---|---|---| -| `user` | `toolCallCount == 0` | The model stopped and waited for a human. Nothing else can be happening. | +| `user` | `toolCallCount == 0` | The model stopped and waited for a human. Nothing else of THIS agent's foreground work is happening — see T37 for the two things that still can be. | | `tool` | `toolCallCount > 0` | The previous turn's tool calls were running. Their own durations are on their own records. | | `first` | there is none | The agent's first turn. `gapMs` is absent. | @@ -590,17 +600,30 @@ fields: T37. **A `tool` gap is not idle time, and must not be added to tool time.** By §4.3a's definition the interval covers what the previous turn's calls were doing, so for a `gapKind` of `tool` it overlaps those calls' own `durationMs`: adding the two counts the same wall clock -twice. Only a `user` gap is time no *foreground* tool and no model was working, which is why it -is the one that answers a duty cycle. - -One exception, and it is T32's: a background job started by an earlier `Bash` keeps running -after the acknowledgement that closed that call, so it can span a `user` gap. Such a gap is -still classified `user` — the pairing has no job state to consult, and reclassifying it would -need the job record T32 defers — so a duty cycle computed from `user` gaps is an upper bound on -idleness wherever background jobs are in use. A consumer that needs the exact figure joins the -job records when they exist (§10.2). The same qualification applies to T27's "costs nothing in -sandbox occupancy": it is true of the agent's foreground work, not of a job still running -behind it. +twice. Only a `user` gap is time the asking agent's own foreground tool and model were both +idle, which is why it is the one a duty cycle starts from — with the scope qualification +below. + +**A `user` gap is AGENT-LOCAL**, because T25 classifies it from the previous response of the +same agent. Two things can therefore be running inside one, and neither is visible on the +record that carries it: + +- **A background job** started by an earlier `Bash` (T32) keeps running past the + acknowledgement that closed that call, so it can span the gap. The gap is still classified + `user`: the pairing has no job state to consult, and reclassifying it would need the job + record T32 defers. +- **Another agent's work.** While a parent waits on a human, a subagent can be running a tool + or an inference of its own. The parent's gap is `user` regardless — its own previous response + made no tool calls — and nothing on that record says the sandbox was busy. + +So a `user` gap answers "was THIS agent idle", and a **sandbox** duty cycle is a different +question: it must exclude every interval in which any agent in that sandbox was running a tool +or a model, which means unioning the intervals across agents rather than summing one agent's +`user` gaps. Summed `user` gaps are an upper bound on sandbox idleness, not the figure itself. +A consumer that needs the exact figure unions by `sessionId` across `agentRole` (T34's join +keys) and joins the job records for the background case when they exist (§10.2). The same +qualification applies to T27's "costs nothing in sandbox occupancy": it is true of the waiting +agent's own foreground work, not of a job or a sibling agent still working behind it. **A consumer with the event records can derive all three fields**, so these definitions are the deliverable and the fields are the convenience. A definition that cannot be checked @@ -620,7 +643,9 @@ ceiling alongside it (#1810 §9 states the same rule for event gaps). Why this split is worth the three fields: on the reference archive, user wait is **72.6% of all gap time** (44.8 h of 61.7 h, at an 1800 s ceiling — T27's own rule) and costs nothing in -sandbox occupancy. Without `gapKind`, +the waiting agent's own foreground occupancy — though not necessarily the sandbox's, since +T37 names a background job and a sibling agent as two things that can run inside such a gap. +Without `gapKind`, a density plan reads that as idle capacity it is paying for. The classifier is already validated — run on that archive it separated `AskUserQuestion` (p50 62.6 s) from `Bash` (p50 201 ms) by three orders of magnitude **without ever seeing a tool name** (#1810 §9). @@ -683,7 +708,8 @@ dial — so the call satisfies both reasons, and the one that EXPLAINS the durat export. Concretely, for a stranded call whose result arrived late: - `endReason` is `stranded`, not `result`. The same holds for a `background` acknowledgement - the cut lands before: the reason that explains the duration wins. + the cut lands before: the reason that explains the duration wins. The background fact is not + lost with it — T32a puts it on `backgrounded`, precisely because this rule takes the reason. - `resultSeq` **stays present**: a request did carry the result, and a consumer joining on it is right to. - `durationMs` ends at the **dial**, not at `resultSeq`. This is the one case where the two @@ -727,6 +753,16 @@ It therefore sets `endReason: "background"` with `durationMs` measuring only the acknowledgement. A count of such calls tells a consumer **how much tool time is missing** and from where, which an undifferentiated `result` would hide. +T32a. **The background fact also rides on its own field, `backgrounded: true`, because +`endReason` cannot always carry it.** `endReason` holds one value and §4.4a makes `stranded` +win over `background` when both apply — correctly, since `stranded` is what explains the +duration. But then the only marker of a backgrounded call is gone, and T32's "how much tool +time is missing" becomes uncountable for exactly the calls whose timing is already least +trustworthy. So a backgrounded `Bash` sets `backgrounded: true` whatever its `endReason`, and +`endReason: "background"` means "backgrounded AND nothing later overrode the reason". A +consumer counting missing job time filters on the field, never on the reason. It is written +only when true (`omitempty`), like `isError`: absent reads as false, which is what it means. + A job record is deliberately left for later: its start, its duration or an `endReason` when it never ended, and its starting call's `callSeq` + `callIndex` (no ids), paired by the `` match. **Note for whoever specifies it:** a backgrounded command overlaps @@ -783,7 +819,8 @@ Four changes from the first draft, all so a record can be placed and ordered on | `callSeq` / `resultSeq` | The response that asked and the request that carried the result. Lets a consumer rejoin a `toolCall` to the event records either side **without a tool-use id** (T8). `resultSeq` is absent when nothing carried a result. | | `callIndex` | T30. The call's position among its response's calls, from 0, in the order the model asked. | | `durationMs` | T18. The paired duration. Absent exactly when `endReason.HasEnd()` is false (T16, T31) — so present for `stranded`, whose end comes from the failed dial whether or not a result ever arrived. | -| `endReason` | T31. Closed vocabulary: how the call ended, and what its duration means. It does **not** say whether a result arrived — `resultSeq` does, because `stranded` occurs both with a late result and without one. No separate flag derives from it. | +| `endReason` | T31. Closed vocabulary: how the call ended, and what its duration means. It does **not** say whether a result arrived — `resultSeq` does, because `stranded` occurs both with a late result and without one — nor whether the call was backgrounded, which `backgrounded` says, because `stranded` wins over `background`. | +| `backgrounded` | T32a. The call started a background job, whatever its `endReason`. Count missing job time on this, never on `endReason: "background"`. Omitted when false. | | `isError` | Whether the result was an error. The tool failure rate, per tool. | | `inputBytes` / `resultBytes` | T14. Effort proxies, never content. Bytes, as the names say. | | `modelMs` | T23. The asking response's measured duration — model time, per call, no join needed. | @@ -973,7 +1010,8 @@ its frequency — but the *rate* is not a constant to design against. job record it deliberately defers, not an unimplementable requirement), and §10.2's row says why: `stranded` is specified, but where its cut runs is not, and the pairing package cannot see the case at all today. Everything else is settled — §6 Option A, T10a/T11/T11a's hashing, -T20's per-item switch, §6.4's marker record — and T28–T30 and T32–T37 are normative. +T20's per-item switch, §6.4's marker record — and T28–T30 and T32–T37, T32a included, are +normative. **Interim behaviour, until T31's route is decided:** the pairing has no stranding pass, so a stranded call exports as whichever of its other cases applies — and there are three, not one.