[khushalsonawat]

Description

The kit's record of what a run used gets two fixes.

• TH-9039: a Claude stage that sends a second message stops counting its earlier turns again.
• TH-9057: a cost the kit can't price is counted as unpriced instead of taking the Claude CLI's own estimate, and the Gemini price table stops going silent on 31 December.

The platform bills the simulated customer's tokens and authoring tokens from these figures. Before this change, a 10-line simulated conversation recorded 5.5× its true tokens. No company has been invoiced for environments yet, so there are no past bills to correct.

Type of Change

• Bug fix

Changes

• Each Claude result records its own turn (backends/claude.py).
  • The CLI reports running totals within one connection. Each result now records the difference from the connection's last totals.
  • On a resumed connection's first result, it records the SDK's per-query usage. SDK 0.2.176 and later carry the old process's totals over a resume, while the locked 0.2.156 restarts them.
  • A fall in any count, or in fresh input (input minus cached), means an unseen restart, so the counts are taken whole.
  • A result with no counts at all is "no report": it records 0 tokens and $0, and the baseline is kept.
• Pricing.
  • The cost is priced from the turn's own tokens. A model the table doesn't price records cost_usd=None (counted in unpriced_turns), never the CLI's total_cost_usd. For a Gemini model the CLI's figure was about 7× our price.
  • With a gateway alias on the wire, the turn is priced by the reported model.
• A dated price table (backends/vertex_gemini.py).
  • Rows are (input, output, first_day, last_day), and CHECK_BY = "2027-03-31".
  • New current_rates(name, today=None) returns today's rates, whatever the row order. Code that read the table by position should use it.
  • Sources, named in the header:
    • today's rates come from the gateway's agentcc-gateway/internal/modeldb/litellm.json (future-agi 9ec5f9fb1, 2026-11-02);
    • gemini-3.6, 3.7 and 3.8 Flash get a second row, $1.50/$7.50 with cache reads at $0.15 from 2027-01-01, from Google's Vertex pricing page (Global tier, read 2026-11-10).
  • transcribe-preview's input is corrected from $2.50 to $2.00, which changes that model's Gemini spend.
  • Past CHECK_BY, a model keeps its latest row's price and logs a WARNING once per model per UTC day, so a running release keeps counting cost. A model with no row, or a day before its first row, stays unpriced and counts in unpriced_turns.
  • Prices are keyed to the UTC day, through vertex_gemini._today(), as the platform's dated rows are.
  • A committed fixture (tests/harness/fixtures/litellm_vertex_prices.json, the gateway's 18 vertex rows at future-agi 9ec5f9fb1) pins today's rates in CI.
• A per-query usage guard (claude.py): when a turn's running-total count falls below the query's own usage, the totals are taken whole.
• CI now runs these tests: sdk-smoke.yml adds test_spend.py and test_claude_cost.py, and test_ci_runs_these_tests fails if either is dropped. A second step reruns test_claude_cost.py on the SDK version uv.lock pins (0.2.156), the one the hosted image installs.
• The re-check date runs weekly, not on PRs: a new workflow, price-check-by.yml (Mondays 06:00 UTC), runs the check-by test and goes red from 2027-03-01. It fails if the test is skipped, and a separate notify job, the only one with issues: write, opens or updates one issue on a scheduled failure. A PR test pins the cron, the env var, the node id, the skip condition and the notify job. GitHub runs schedules on the default branch only, so it starts after this reaches main.
• Tests:
  • test_claude_cost.py (new):
    • per-turn counting, restarts, a second connection, a resume, no-report results, aliases, two model keys, cache-rate pricing;
    • the CLI-shape test, which runs the bundled CLI against a local stub through Stage, a resume and run/conversation.py's customer;
    • a tripwire for feat(harness): build an agent from a repo's subfolder #225 (below).
  • test_spend.py:
    • the check-by test (skipped on PRs, run by the weekly workflow), its boundary test, and the workflow-shape test;
    • latest-row pricing past CHECK_BY, and a model with no row staying unpriced;
    • row choice by date;
    • all three Flash models on both sides of 1 Jan and after CHECK_BY, including the "no current price" warning;
    • a two-call Gemini stage that records the plain sum.
  • test_vertex_gemini_backend.py: one test that priced gemini-3.7-flash without a date now pins it.

The integration with #225 (TH-8804), now in this PR

This PR targets #225's branch, so the double subtraction is fixed here (be64c352, after merging the base c0e51b7e in 67b52516):

• agent_reading.py: _ReadingStage._own, the running_totals flag and their baselines are removed. A turn is added as the backend reports it, since this PR's Claude backend already reports per-turn figures.
• test_agent_reading.py: the three tests that fed running totals now feed per-turn figures, with the same expected totals. _gemini_priced uses current_rates.
• The tripwire, test_a_claude_reading_adds_per_turn_figures_as_they_come, now runs. It failed on the merge before the fix (0.07875 against 0.1575) and passes after.
• Plants:
  • feat(harness): build an agent from a repo's subfolder #225's subtraction put back: fails 4 tests, including the tripwire;
  • the same for Gemini: fails 1 test;
  • a double count: fails 26;
  • the backend reporting running totals again: fails 9.

Fixed after the review rounds (named as asked)

• Plan review round 1 (not clear). Fixes:

  • the resume rule, for SDK 0.2.176 and later;
  • the Flash prices that double on 1 Jan;
  • the billed path, traced to the simulated customer's and authoring tokens.

• Plan review round 2 (clear). Fixed with no further round:

  • the check-by rule now checks each model's latest row, with more than 30 days left and no gaps between rows;
  • the 2027 Vertex rows were added, after Google's page was read;
  • current_rates;
  • all five feat(harness): build an agent from a repo's subfolder #225 tests named;
  • a no-report turn records $0;
  • new planted bugs for these.

• Change review round 1 (not clear: CI ran none of these tests). Fixes:

  • the CI list, with its guard test;
  • all three Flash models tested on both sides of 1 Jan;
  • the baseline checked after a restart;
  • the fresh-input restart rule;
  • a warning when a resumed result has no usage;
  • small code tidy-ups.

• Change review round 2 (clear). Fixed with no further round:

  • a cache-rate pricing test;
  • the CI lines moved so the list merges cleanly with feat(harness): build an agent from a repo's subfolder #225;
  • the CI-list guard now matches whole lines;
  • an input-only turn is still a report;
  • a header comment.

  These were checked by planted bugs, not by another round.

Known limits and open items

• The open complete: false item (feat(harness): build an agent from a repo's subfolder #225). On feat(harness): build an agent from a repo's subfolder #225, a reading whose turns include an unpriced one writes complete: false in cost.json. After this change, that happens only for a model missing from the price table. The CLI's estimate no longer counts as priced, which is what the ticket asks. How the platform bills an incomplete reading isn't on future-agi dev, so this is left for feat(harness): build an agent from a repo's subfolder #225's landing.
• A resumed connection's first turn leaves out helper-agent and CLI side-call tokens. The SDK's per-query usage doesn't carry them: up to a third of that turn in a probe. Only hosted chat resumes, and chat usage isn't billed. If that result has no usage at all, it records 0 and logs a warning.
• Today's rates are pinned in CI by the committed fixture. The live agreement test against litellm.json runs only beside a future-agi checkout, so it skips in CI. From 2027-01-01 it is red locally for the three Flash models until the gateway's table gets the same 2027 rows (TH-9508). Both tables must change together.
• The CLI-shape test has run on macOS with SDK 0.2.139, 0.2.156 and 0.2.178, and in a CI-like environment on Python 3.12 (scrubbed environment, the workflow's livekit pins). Watch its first ubuntu run.
• An interrupted query's tokens don't reach the spend ledger. This predates the change: on the real CLI stub, the ledger summed 4,400 tokens against a true 6,600. Whether the provider bills an aborted stream isn't known. It's recorded in the coordination register as a Medium (money).
• The gateway's table deprecates gemini-3.6-flash on 2026-11-19 and gemini-3.7-flash, the default model, on 2027-01-28. Choosing the next default is TH-9510.

Testing

Run locally under a memory cap, after the last pushed change (ffa61b78):

• Python 3.12.3 (SDK 0.2.176): test_spend.py and test_claude_cost.py: 43 passed, 3 skipped (the tripwire, the local-only agreement test, and the check-by test, which runs only in the weekly workflow). With ALK_PRICE_CHECK_BY=1, the check-by test passes.

• The locked SDK 0.2.156: test_claude_cost.py: 19 passed, 1 skipped.

• test_vertex_gemini_backend.py (19 passed): not in CI. Its test_provider_429_exhausts_bounded_retries fails the same way on the base (c0e51b7e, Python 3.12.3), so the file isn't added to sdk-smoke.yml.

• CI's whole pytest list, in a CI-like environment on Python 3.12 (a reviewer's run, before the weekly-workflow change): green.

• The agreement test, run by hand against litellm.json at 9ec5f9fb1: all 18 rows match.

• Planted bugs:

  • 17 from the plan, re-run on the final code: all caught;
  • 6 more for the review fixes: all caught;
  • the change reviewers' own 64 across two rounds: 54 caught, and every survivor fixed after.

• Reviews: two plan review rounds and two change review rounds, each with two fresh reviewers. Dileep's review fixes and the CHECK_BY change had their own change review, with two reviewers (both clear). A fix-only check of the commits made after those clears (a232700b, and the Lows in ffa61b78), with two reviewers: clear, no High. It found one Medium and five Lows, all guards no test pinned (the locked-SDK rerun step, two weaker forms of the per-query guard, a non-UTC zone, and the price-check-by workflow's checkout and if:). They are fixed in 5e45ac81.

• The last fix-only check, two reviewers, covered be64c352 and 5e45ac81: clear, no High. It covered the integration fix and those guards. Owl's six plants are caught, and the locked SDK 0.2.156 gives 23 passed. It found 2 Mediums and 4 Lows, left for a follow-up commit with its own check:

  • the notify job's no-installs assertion is looser than before;
  • the locked-SDK rerun could be made non-blocking (continue-on-error, if, shell) with the test still green;
  • _gemini_priced breaks after 2027-04-01;
  • the tripwire's docstring is stale, and its importorskip could hide an import fault;
  • no single test chains the real ClaudeSession into the reading;
  • a few rationale comments and ticket ids remain.

• CI clean-install on the pull_request run fails one speed budget, test_contract_model.py::test_a_crafted_file_of_block_comments_does_not_hold_up_the_check[a-file-of-openers] (0.5 s), at about 0.54–0.66 s. That file isn't in this PR. The base fails it on its own push run, 477efe88 (https://github.com/future-agi/agent-learning-kit/actions/runs/43169044737, 0.84 s, plus test_a_quote_that_grows_when_normalised_costs_little). This PR's own push run passes (https://github.com/future-agi/agent-learning-kit/actions/runs/43171616440).

• Failures that predate this change:

  • in this machine's kit venv only: two test_terminal_save_closes_adk_invocation_in_its_original_context cases, and tests/test_harness.py::test_hosted_claude_session_uses_durable_resume_and_restricted_tools;
  • on 0.2.178: test_provider_429_exhausts_bounded_retries.

  All of them fail the same way on untouched dev in the same environment.

Related Issues

Refs: TH-9057
Refs: TH-9039
Related: #225 (TH-8804), TH-9508, TH-9510

[cdileep23]

Please check the comments.