Repository navigation
Conversation
Red on main: runstore.execute takes no max_calls/max_usd, EvolveSpec has neither knob, and RunStatus carries no stop detail. The counting factory makes each model call observable, so "the second call did not happen" is an assertion rather than an inference.
…963) A run-level cap charged inside the run: CallLimit.reserve() runs before every model call, so 'the call did not happen' is structural. BudgetStop is caught by name in the worker wrapper, at the round barrier and on the main thread, so the round closes without a partial merge or a traceback and the run ends 'done' with stop_reason=budget and a stop_detail naming the limit and the observed value. max_calls on evolve() stays the round-barrier budget; the same number is also enforced per call, which can only be stricter. Partial: #963
…1166) The allowlist entry and the evolve() docstring are what keep a flat kwarg deliberate rather than drift; the two status.json tests pin the additive contract (no key when unset, stop_detail when the cap fires).
The barrier-free runtime has no per-call guard yet; a cap that reached it would be silently unbounded, so it is refused by name rather than warned about. Sync and async stay two contracts (#809).
|
Tip SHA: 3b49b89 Ran:
Not run:
Non-blocking:
Verdict: PASS |
|
Advisory read at Blocking1. A spec-level
Offline reproduction, using the counting-agent spec in The run is recorded 2. A cap that trips mid-round ends the run
Same shape with Non-blocking
Verified at this tip
Not run: |
|
Re-read only the two paths and the regression rows, in a fresh clone at the named sha. 1. A spec-level 2. A refusal inside an executor-dispatched rollout ends
The previous nine rows did not move. The nine original Gates: Non-blocking, documentation edge. A spec-level Not run: |
) Two contract gaps found after the first tip, both reproduced before the fix: - A spec-level `max_usd` could not run: `_limit_from_knobs` popped `max_usd` but left `usd_per_call`, which `compose` then handed to `evolve()`, so every valid dollar cap died with `TypeError: evolve() got an unexpected keyword argument 'usd_per_call'` and the run was recorded `failed` with calls=0. - A cap refused inside an executor-dispatched rollout was recorded as a worker failure: `ThreadExecutor._one`'s generic `except Exception` turned the `BudgetStop` into `Result(kind="model")`, `_rollout_failure` rebuilt it as `RuntimeError("BudgetStop: ...")`, and the round body's generic handler set `run_error` -- `state=failed`, `stop_reason=error`, `stop_detail` present. `max_calls=2` under four workers hit it; so did `max_usd=0.05` at $0.02/call. The refusal now travels as its own `Result.kind="budget"` (report, never raise: `map_rollouts` delivers through a queue, and a bare raise would hang the reader), and `_rollout_failure` re-raises the stop the `CallLimit` already recorded -- numbers included -- so the round closes `done` with `stop_reason=budget`. `CallLimit.stop` exposes that recorded refusal. Also drops the contradictory `RuntimeWarning` before the async `ValueError`: the async path refuses a cap, it does not ignore one.
|
Tip SHA: Base: Ran — fresh clone at this sha,
Not run:
No UI surface: terminal text only — the Non-blocking observation: spec-level Verdict: PASS — advisory QA verdict at |
|
Advisory read at T3 by path ( Tip SHA: 3300d55
Findings:
Two body nits for your decision, not defects:
No UI surface: run-loop stop behaviour plus two docs pages; nothing rendered changes. |
|
Self-review (Birfy) Tip SHA: 3300d55
|
Partial: #963
What changed
A run-level budget checked only between rounds is not a cap: a four-worker round on
max_calls=1still makes four calls, andbudget_usdonly warns. This addsmax_calls/max_usdat run level, charged per call inside the run: when the next call would exceed either limit the call is not made, the round winds down, and the run recordsstop_reason="budget"with astop_detailnaming the limit and the observed value.status.jsonandagentdescent statusboth carry it.budget.py:CallLimitcounter (a counter, not a budget engine),BudgetStop, andCallLimit.stop— the refusal the limit recorded, so a later handler re-raises the same stop instead of rebuilding it from a message. Imports nothing fromcli/runstore.metrics.measured(guard=...): new optional keyword, defaultNone; existing call sites pass positionally and are unchanged.runstore.execute: builds the limit from the run settings, passes it explicitly to each worker'smeasured(), catchesBudgetStopby name for a cleandone+stop_reason="budget", and closes the round on the main thread too.evolvespec.compose: validates both knobs, naming the field and the offending value asRunStoreError/SpecErrorbefore any run dir exists;_limit_from_knobsconsumesmax_usdandusd_per_call(both are compose-only) whilemax_callsstays in the knob dict, so the barrier budget of the same name keeps its meaning.evolution.evolve:max_callskeeps its round-barrier meaning and is additionally enforced per call (strictly tighter, never looser);call_limitis in the parameter allowlist and documented.evolve(asynchronous=True)refuses a call cap (ValueError, noRuntimeWarningfirst) — an ignored cap is an unbounded run that looks bounded.executor.py: one narrow addition —ThreadExecutor._onecatchesBudgetStopby name and reportsResult(kind="budget", error=e.detail()). It reports rather than raises:_one's contract is to report, andmap_rolloutsdelivers through a queue, so a bare raise would strand its readers.RunStatus.stop_detailis additive and optional: absent fromto_dict()whenNone, so an uncapped run'sstatus.jsonkeeps exactly its old key set.0means zero calls: clean budget stop before the first call, not a validation error.cli:--max-calls/--max-usd/--usd-per-call, and the budget line inagentdescent status.docs/api.mdregenerated;docs/start.mddocuments the knobs;changelog.d/1166-run-call-cap.md.Out of scope and untouched:
max_concurrency, cross-run accounting, nested refusal, MCP write confinement, CLI write dirs.What the first tip got wrong (both reproduced at
3b49b89a, both fixed here)1. A spec-level
max_usdcould not run._limit_from_knobspoppedmax_usdbut leftusd_per_call, whichcomposethen handed toevolve():2. A cap refused inside an executor-dispatched rollout ended the run
failed._one's genericexcept Exceptionturned the stop intoResult(kind="model"),_rollout_failurerebuilt it as aRuntimeError, and the round body's generic handler recorded a run error:The refusal now travels as
Result.kind="budget"and_rollout_failurere-raises the stopCallLimitrecorded, so the round closesdonewith a named budget reason. Both rows are new tests intests/test_budget_guard.pyand both are red at3b49b89a, green here.Verification at the tip
Tip SHA: 3300d55
Ran:
pytest -q tests/test_budget_guard.py→12 passed(7 original rows + 2status.jsonshape rows + 3 regression rows for the two defects above)pytest -q tests/test_budget_guard.py tests/test_executor.py tests/test_async_early_stop.py tests/test_runstore_cli.py tests/test_spec_usage.py tests/test_run_panel.py tests/test_evolve_settings.py tests/test_api_docs.py→124 passed3b49b89a(worktree at that sha, onlytests/test_budget_guard.pyswapped in): the three new rows fail —assert 'usd_per_call' not in comp.kwargs, and twostate='failed'rows with theRuntimeError: BudgetStop: …above. Red on pristinemainf8efc731too (TypeError: execute() got an unexpected keyword argument 'max_calls').22 failed, 7745 passed, 61 skipped, 524 warnings, 10 errors in 293.07s (0:04:53). Breakdown: 20 of the 22 failures reproduce on pristinemainf8efc731(test_apply_noop.py×5,test_dir_evolution.py::test_write_to_backs_up_before_overwriting,test_era_srbench.py×6,test_genesis_example.py×2,test_integrations.py::test_the_launcher_install_writes_ignores_a_shadowing_cwd,test_skill_dir_example.py::test_write_to_installs_the_evolved_skill,test_skill_library_apply_install.py×2,test_skill_library_example.py::test_unread_or_wrong_library_scores_zero_then_write_to_installs,test_wheel_smoke.py::test_wheel_smoke_removes_inherited_python_overrides), the other 2 are docker rows (test_sandbox_container.py), and all 10errorsaretest_sandbox_container.pytoo — no docker daemon on this box. The failure list is byte-identical to the one at3b49b89a.python -m tools.gen_api_docs --check→docs/api.md is up to datepython -m tools.changelog_fragments check→80 changelog fragment(s) OKagentdescent status <run>after a cap that trips inside a rollout (four workers,rounds=3):--max-calls 2:20261119-122430-a59656 done round 0/3 calls=2 best=0.000 text:You are a helpful assistant. [budget: max_calls: 2 call(s) made, cap 2]—errorempty,status.jsonstate=done,calls=2,stop_reason=budget.--max-usd 0.05 --usd-per-call 0.02:20261119-122505-351442 done round 0/3 calls=2 best=0.000 $0.04 text:You are a helpful assistant. [budget: max_usd: $0.0400 spent, cap $0.0500]—errorempty,state=done,calls=2,usd=0.04.Not run:
ERROR - Error building page 'index.md': HTTPSConnectionPool(host='fonts.google.com', port=443) … Connection to fonts.google.com timed out. CIbuildis the line.Acceptance rows (fake worker named in the spec, spec
tests.test_budget_guard:counting_agent):max_calls=1, worker would call moreCALLS == ["agent"],state=done,error=None,stop_reason=budget,stop_detail="max_calls: 1 call(s) made, cap 1"max_usd=0.015,usd_per_call=0.01usd=0.01,stop_reason=budget,stop_detail="max_usd: $0.0100 spent, cap $0.0150"max_usd=0.015+usd_per_call=0.01(realcompose→evolvepath)usd_per_callabsent fromcomp.kwargs; 1 call,done,budget, noTypeErrormax_calls=0done,budget, not a validation errormax_calls=-1evolve.max_calls: must be a non-negative integer, got -1max_usd="cheap"stop_reason=rounds,stop_detailkey absent, key set identical to pristinemainmax_calls=2, four workers, refusal inside a rolloutdone,error=None,calls=2,budget,stop_detail="max_calls: 2 call(s) made, cap 2"max_usd=0.05at $0.02/call, four workersdone,error=None,calls=2,budget,stop_detail="max_usd: $0.0400 spent, cap $0.0500"--max-calls 1--max-calls -1{"ok": false, "error": "evolve.max_calls: must be a non-negative integer, got -1"}evolve(asynchronous=True)+ capValueError, no run dir, no precedingRuntimeWarning