Skip to content

Keep running session totals out of the token columns - #3015

Merged
HypForge merged 2 commits into
masterfrom
additive-usage-columns
Nov 17, 2026
Merged

HypForge merged 2 commits into
masterfrom
additive-usage-columns

Conversation

@platypii

Copy link
Copy Markdown
Contributor

Problem

The token columns from LLP 0590 should be safe to sum: sum(output_tokens) over any rows should count each token once. Two adapters break that. Droid and Mastra Code write a session_usage row holding a session's lifetime totals, and re-send it as the totals grow. Locally each revision replaces the last, but every revision is exported, so a server-side sum counts the session once per revision. The server usage rollup works around this with Droid-specific code that picks the newest revision from attributes, and it ignores Mastra Code entirely.

Fix

withUsageColumns and the column backfill leave the token columns NULL on any provider_subtype = 'session_usage' row. The counts stay in attributes.usage, unchanged.

This lets readers sum the columns with no role filters or per-client code. A follow-up server PR rewrites the usage rollup as a single GROUP BY.

Turning these totals into increments was considered. It needs durable per-session state the adapters do not keep (Droid's live cursors are in memory, capped at 64 files), so the totals stay out of the columns instead. Droid and Mastra Code usage drop out of column sums.

Changes

  • ai-gateway/src/dataset.js: rowUsage returns nothing for session_usage rows.
  • LLP 0590 gains #additive, which states the rule. The Draft specs LLP 0598 (Droid) and LLP 0603 (Mastra Code) and the two integration docs are updated to match.
  • Mastra Code tests now read the normalized counts from attributes.usage and assert that the columns are NULL.

Not changed

  • Partitions the column backfill already rewrote keep their old column values for these rows. On the hyperparam org there are none.
  • The local hyp overview still reads attributes.usage (LLP 0590 rollout step 3).

Checks

  • npm test: 9,731 pass, 0 fail.
  • npm run typecheck: clean.

Droid and Mastra Code write lifetime usage snapshots that are re-sent as they grow. Locally each replaces the last, but every revision is exported, so summing the token columns on the server counted a session once per revision. Their counts now stay in attributes.usage only, so the columns hold only counts that can be summed.
@platypii platypii added the neutral:adopt Foreign PR adopted into neutral's reconcile scope label Nov 17, 2026
@HypForge HypForge added the neutral:adopted Adoption completion record: merged while carrying neutral:adopt (LLP 0031) label Nov 17, 2026
The droid_capture smoke still read the snapshot's output_tokens column,
which is NULL now, so it failed. It reads attributes.usage and checks the
columns are NULL. Add a Droid test for the columns and a backfill test
that leaves session_usage rows NULL while it fills an ordinary row.
@HypForge

Copy link
Copy Markdown
Contributor

Final review at head 11708ca: LOW, clean.

Pushed (tests only, normal push 0004979..11708ca). The droid_capture smoke failed at 0004979. Its usage step still expected the snapshot's totals in the token columns. It now asserts NULL columns and reads the totals from attributes.usage. Also added a Droid test (withUsageColumns leaves the columns NULL) and a backfill test (Droid and Mastra Code session_usage rows stay NULL; a normal row is still filled). Both new tests fail on master's code and pass here.

Writers. Every path that fills the columns goes through rowUsage: the settle step, the Droid listener, the backfill reconcile, the column backfill and the server mover. No other row kind re-sends a running total. Cursor stops are one generation each and a re-sent stop is dropped at ingest. Codex uses last_token_usage. Hermes writes its session-end row once, and that row has no total_tokens.

Readers.

  • These are unchanged: the local overview, which sums attributes.usage; the server usage rollup, which reads attributes and picks the newest Droid revision; and the usage summary, the spike email and the hypscope usage page, which all read usage_rollup_daily. Droid numbers on the dashboard are the same before the server follow-up. Mastra Code was not in the rollup before and is not now.
  • Two agent prompts, which I have not changed, still say to read Droid usage from the columns. They are hypaware-core/skills/hypaware-query/SKILL.md:72-80 and hypscope src/chat/instructions.ts:36-38. After this change they would find NULL. They should change with the server follow-up.

Already-written 1.53.0 rows (author's choice, not blocking). A local row keeps its filled columns until a newer revision of the session replaces it. Revisions already exported keep theirs. A column sum over 1.53.0 data counts such a session once per revision. Leaving them costs nothing and the overcount ages out. Nulling them once would need migration code on both client and server.

LLPs. 0590, 0598 and 0603 are all Draft. llp-numbers.js check reports no collision. One wording point in 0590 #additive: Hermes message rows do carry total_tokens. The rule still holds, so it is wording only.

Release note (next release). Droid and Mastra Code session totals now stay in attributes.usage and no longer fill the token columns, so a sum over the columns counts each token once. The local overview and the HypAware Cloud usage dashboard are unchanged.

Privacy: LOW. The counts are still kept and exported in attributes.usage. Nothing collected, kept, exported or shown changes, so no legal-page check is needed.

CPU and memory: one string compare per row on the write path and in the backfill. No allocation or growth. No concern.

Checks: full npm test at 0004979: 9740 tests, 0 fail. Typecheck is clean at 11708ca. The droid_capture and mastracode_capture smokes pass. CI is green at 11708ca, test-windows included. The first test-windows attempt failed on an unrelated timing test (github-source.test.js "source never overlaps slow ticks"). Its one re-run passed.

Server-suite gate: hypaware-server master 3a01876 against this client head 11708ca: typecheck clean, vitest 329 files and 704 tests, all pass.

@HypForge

Copy link
Copy Markdown
Contributor

Landing note (HypForge orchestrator): enqueuing at 11708ca. Kenny handed this PR over with neutral:adopt at 09:24Z on 2026-11-17. It lands on a clean LOW review with green checks, so there is no merge ask.

  • Review: LOW, clean, at this exact head (final comment above, 6944697474). Server-suite gate: pass (server master 3a01876 against this head, 704 of 704). CPU and memory: no concern. Privacy: LOW, no legal-page check needed.
  • What HypForge added: tests only, one normal push. The droid_capture smoke failed at the author's head because it still expected totals in the columns; it now expects NULL. A Droid test and a backfill test were added.
  • Guard: open, not a draft, CLEAN, ten checks green including test-windows (one re-run, an unrelated timing test), 0 behind master 10f1003, no release PR open.

Two points left as they are unless the author says otherwise:

  1. Rows already written by 1.53.0 keep their filled columns, so a column sum over that data counts such a session once per exported revision. Default taken: leave them; it costs nothing and ages out. The other way is a one-time migration on client and server.
  2. LLP 0590 #additive says Hermes message rows carry no usage; they do carry total_tokens. The rule still holds. Default taken: leave the wording, or fold it into the server follow-up.

To do before the next client release, not in this PR:

  • hypaware-core/skills/hypaware-query/SKILL.md:72-80 and the hypscope Ask agent prompt (src/chat/instructions.ts:36-38) still tell an agent to read Droid usage from the token columns. After this change they find NULL. Both are prompt text, so HypForge asks before changing them.
  • The v1.53.0 notes say Droid and Mastra Code counts are in the standard token columns. The next notes carry the reviewer's line above.

@HypForge
HypForge added this pull request to the merge queue Nov 17, 2026
Merged via the queue into master with commit 19f3cb0 Nov 17, 2026
16 of 17 checks passed
@HypForge
HypForge deleted the additive-usage-columns branch November 17, 2026 10:02
@HypForge

Copy link
Copy Markdown
Contributor

Merged (HypForge orchestrator) as 19f3cb0 at 10:02:42Z on 2026-11-17, on the first pass through the merge queue. CI on the merge commit is green.

  • This is on master only. It reaches users with the next client release; 1.53.0 still fills the columns.
  • Docs: checked after the merge. No other client, site or hypscope sentence puts Droid or Mastra Code counts in the token columns, so there is no docs PR.
  • For the server follow-up: docs/runbooks/usage-columns-backfill.md in hypaware-server says the mover fills every row (lines 6-7 and 129-130). That stops being true at the first server deploy that picks up this client commit.
  • Two prompt texts still read Droid usage from the columns: hypaware-core/skills/hypaware-query/SKILL.md:79 (and sql.md:6) and hypscope src/chat/instructions.ts:36,38. Brendan has been asked who should change them. They need to change before the next client release, or an agent reads NULL for Droid and reports zero.
  • The v1.53.0 notes say Mastra Code counts are "in the standard token columns". Those notes are not on the site yet; that phrase should be dropped when they are.
  • Not verified: nobody ran a real Droid or Mastra Code session against this commit. The evidence is the unit tests and the two capture smokes.

@HypForge

Copy link
Copy Markdown
Contributor

Correction (HypForge orchestrator) to point 1 of the landing note above. I wrote that leaving the rows already written with filled token columns "costs nothing and ages out". That is wrong for a server once hypaware-server #1724 deploys.

What a server holds:

  • A client on 1.53.0, or on a master build between Capture Droid session token snapshots and fix hook metadata #2971/Capture Mastra Code session usage in token columns #2979 and this PR, exports every revision of a Droid or Mastra Code session_usage row with the token columns filled.
  • The server keeps every revision. Ingest drops a row only when its part_id repeats, and each revision's part_id is the message id plus a hash of the usage values (Droid projector.js:85,92; Mastra Code projector.js:174,179). The archive keeps them past the 7-day cache.
  • A later revision from a client with this PR adds a row with NULL columns. It does not change the earlier filled rows.

What #1724 does with them (read at its head aec6ad7a): aggregatePartition sums every row that has any non-NULL token column, with no role or subtype filter. So a session with N filled revisions is counted N times, on the day the session started. Master today counts a Droid session once and does not count Mastra Code. Every boot rebuilds every day, so the deploy itself rebuilds the affected days.

What is not known: whether production holds any such rows. No HypForge seat can read production. This read-only query answers it; any row where revisions is greater than sessions is a day #1724 would overcount:

SELECT org, date, conversation_source, COUNT(*) AS revisions, COUNT(DISTINCT message_id) AS sessions
FROM ai_gateway_messages
WHERE provider_subtype = 'session_usage' AND output_tokens IS NOT NULL
GROUP BY org, date, conversation_source

Options, the author's choice:

  1. Run the query before hypaware-privacy Step 1 ambiguity guard passes a reroute that names the gateway-only destination in words the file never uses for it #1724 deploys. If it returns nothing above one revision per session, nothing more is needed. Cost: one query.
  2. Have hypaware-privacy Step 1 ambiguity guard passes a reroute that names the gateway-only destination in words the file never uses for it #1724 skip provider_subtype = 'session_usage' rows. Cost: one per-row rule in a query written to have none, and its speed is not measured.
  3. Null the columns once on the stored rows. Cost: migration code on the server.
  4. Leave it. Cost: those sessions are overcounted on the usage dashboard for their start days, and it does not age out.

Nothing in this PR needs to change for any of these.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

neutral:adopt Foreign PR adopted into neutral's reconcile scope neutral:adopted Adoption completion record: merged while carrying neutral:adopt (LLP 0031)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants