Skip to content

feat(run): stop a worker run on max_calls / max_usd at the call (#1166) - #1167

Open
HugoXK wants to merge 5 commits into
mainfrom
agent/1166-budget-guard
Open

HugoXK wants to merge 5 commits into
mainfrom
agent/1166-budget-guard

Conversation

@HugoXK

@HugoXK HugoXK commented Nov 19, 2026 •

Copy link
Copy Markdown
Collaborator

Partial: #963

What changed

A run-level budget checked only between rounds is not a cap: a four-worker round on max_calls=1 still makes four calls, and budget_usd only warns. This adds max_calls / max_usd at run level, charged per call inside the run: when the next call would exceed either limit the call is not made, the round winds down, and the run records stop_reason="budget" with a stop_detail naming the limit and the observed value. status.json and agentdescent status both carry it.

  • budget.py: CallLimit counter (a counter, not a budget engine), BudgetStop, and CallLimit.stop — the refusal the limit recorded, so a later handler re-raises the same stop instead of rebuilding it from a message. Imports nothing from cli/runstore.
  • metrics.measured(guard=...): new optional keyword, default None; existing call sites pass positionally and are unchanged.
  • runstore.execute: builds the limit from the run settings, passes it explicitly to each worker's measured(), catches BudgetStop by name for a clean done + stop_reason="budget", and closes the round on the main thread too.
  • evolvespec.compose: validates both knobs, naming the field and the offending value as RunStoreError/SpecError before any run dir exists; _limit_from_knobs consumes max_usd and usd_per_call (both are compose-only) while max_calls stays in the knob dict, so the barrier budget of the same name keeps its meaning.
  • evolution.evolve: max_calls keeps its round-barrier meaning and is additionally enforced per call (strictly tighter, never looser); call_limit is in the parameter allowlist and documented.
  • evolve(asynchronous=True) refuses a call cap (ValueError, no RuntimeWarning first) — an ignored cap is an unbounded run that looks bounded.
  • executor.py: one narrow addition — ThreadExecutor._one catches BudgetStop by name and reports Result(kind="budget", error=e.detail()). It reports rather than raises: _one's contract is to report, and map_rollouts delivers through a queue, so a bare raise would strand its readers.
  • RunStatus.stop_detail is additive and optional: absent from to_dict() when None, so an uncapped run's status.json keeps exactly its old key set.
  • 0 means zero calls: clean budget stop before the first call, not a validation error.
  • cli: --max-calls / --max-usd / --usd-per-call, and the budget line in agentdescent status.
  • docs/api.md regenerated; docs/start.md documents the knobs; changelog.d/1166-run-call-cap.md.

Out of scope and untouched: max_concurrency, cross-run accounting, nested refusal, MCP write confinement, CLI write dirs.

What the first tip got wrong (both reproduced at 3b49b89a, both fixed here)

1. A spec-level max_usd could not run. _limit_from_knobs popped max_usd but left usd_per_call, which compose then handed to evolve():

execute RAISED TypeError: evolve() got an unexpected keyword argument 'usd_per_call'
state='failed' error="TypeError: evolve() got an unexpected keyword argument 'usd_per_call'" calls=0 stop_reason=None

2. A cap refused inside an executor-dispatched rollout ended the run failed. _one's generic except Exception turned the stop into Result(kind="model"), _rollout_failure rebuilt it as a RuntimeError, and the round body's generic handler recorded a run error:

EarlyStopWarning: evolve() stopped early after 2 round(s): RuntimeError: BudgetStop: max_calls=2 reached
EarlyStopWarning: evolve() stopped early after 2 round(s): RuntimeError: BudgetStop: max_usd=0.05 reached

The refusal now travels as Result.kind="budget" and _rollout_failure re-raises the stop CallLimit recorded, so the round closes done with a named budget reason. Both rows are new tests in tests/test_budget_guard.py and both are red at 3b49b89a, green here.

Verification at the tip

Tip SHA: 3300d55
Ran:

  • pytest -q tests/test_budget_guard.py → 12 passed (7 original rows + 2 status.json shape rows + 3 regression rows for the two defects above)
  • pytest -q tests/test_budget_guard.py tests/test_executor.py tests/test_async_early_stop.py tests/test_runstore_cli.py tests/test_spec_usage.py tests/test_run_panel.py tests/test_evolve_settings.py tests/test_api_docs.py → 124 passed
  • red at 3b49b89a (worktree at that sha, only tests/test_budget_guard.py swapped in): the three new rows fail — assert 'usd_per_call' not in comp.kwargs, and two state='failed' rows with the RuntimeError: BudgetStop: … above. Red on pristine main f8efc731 too (TypeError: execute() got an unexpected keyword argument 'max_calls').
  • full suite at the tip → 22 failed, 7745 passed, 61 skipped, 524 warnings, 10 errors in 293.07s (0:04:53). Breakdown: 20 of the 22 failures reproduce on pristine main f8efc731 (test_apply_noop.py ×5, test_dir_evolution.py::test_write_to_backs_up_before_overwriting, test_era_srbench.py ×6, test_genesis_example.py ×2, test_integrations.py::test_the_launcher_install_writes_ignores_a_shadowing_cwd, test_skill_dir_example.py::test_write_to_installs_the_evolved_skill, test_skill_library_apply_install.py ×2, test_skill_library_example.py::test_unread_or_wrong_library_scores_zero_then_write_to_installs, test_wheel_smoke.py::test_wheel_smoke_removes_inherited_python_overrides), the other 2 are docker rows (test_sandbox_container.py), and all 10 errors are test_sandbox_container.py too — no docker daemon on this box. The failure list is byte-identical to the one at 3b49b89a.
  • python -m tools.gen_api_docs --check → docs/api.md is up to date
  • python -m tools.changelog_fragments check → 80 changelog fragment(s) OK
  • CLI, agentdescent status <run> after a cap that trips inside a rollout (four workers, rounds=3):
    • --max-calls 2: 20261119-122430-a59656 done round 0/3 calls=2 best=0.000 text:You are a helpful assistant. [budget: max_calls: 2 call(s) made, cap 2] — error empty, status.json state=done, calls=2, stop_reason=budget.
    • --max-usd 0.05 --usd-per-call 0.02: 20261119-122505-351442 done round 0/3 calls=2 best=0.000 $0.04 text:You are a helpful assistant. [budget: max_usd: $0.0400 spent, cap $0.0500] — error empty, state=done, calls=2, usd=0.04.
      Not run:
  • mkdocs build --strict — did not complete on this box: ERROR - Error building page 'index.md': HTTPSConnectionPool(host='fonts.google.com', port=443) … Connection to fonts.google.com timed out. CI build is the line.

Acceptance rows (fake worker named in the spec, spec tests.test_budget_guard:counting_agent):

row reading
max_calls=1, worker would call more CALLS == ["agent"], state=done, error=None, stop_reason=budget, stop_detail="max_calls: 1 call(s) made, cap 1"
max_usd=0.015, usd_per_call=0.01 1 call, usd=0.01, stop_reason=budget, stop_detail="max_usd: $0.0100 spent, cap $0.0150"
spec-level max_usd=0.015 + usd_per_call=0.01 (real compose → evolve path) usd_per_call absent from comp.kwargs; 1 call, done, budget, no TypeError
max_calls=0 0 calls, done, budget, not a validation error
max_calls=-1 refused before the run dir exists: evolve.max_calls: must be a non-negative integer, got -1
max_usd="cheap" refused naming field and value
both knobs unset stop_reason=rounds, stop_detail key absent, key set identical to pristine main
max_calls=2, four workers, refusal inside a rollout done, error=None, calls=2, budget, stop_detail="max_calls: 2 call(s) made, cap 2"
max_usd=0.05 at $0.02/call, four workers done, error=None, calls=2, budget, stop_detail="max_usd: $0.0400 spent, cap $0.0500"
CLI --max-calls 1 exit 0, status line above
CLI --max-calls -1 exit 2, {"ok": false, "error": "evolve.max_calls: must be a non-negative integer, got -1"}
evolve(asynchronous=True) + cap ValueError, no run dir, no preceding RuntimeWarning

@HugoXK HugoXK added enhancement New feature or request area:evolve The evolve loop: evolution, async_evolve, aggregator, population, policies, strategies area:runstore Run store, ledger, checkpoint, resume and replay area:cli The agentdescent command line (cli.py, doctor, plan/evolve/status/apply) labels Nov 19, 2026
Red on main: runstore.execute takes no max_calls/max_usd, EvolveSpec has
neither knob, and RunStatus carries no stop detail. The counting factory makes
each model call observable, so "the second call did not happen" is an
assertion rather than an inference.
…963)

A run-level cap charged inside the run: CallLimit.reserve() runs before every
model call, so 'the call did not happen' is structural. BudgetStop is caught by
name in the worker wrapper, at the round barrier and on the main thread, so the
round closes without a partial merge or a traceback and the run ends 'done'
with stop_reason=budget and a stop_detail naming the limit and the observed
value. max_calls on evolve() stays the round-barrier budget; the same number is
also enforced per call, which can only be stricter.

Partial: #963
@HugoXK
HugoXK marked this pull request as ready for review November 19, 2026 11:24
…1166)

The allowlist entry and the evolve() docstring are what keep a flat kwarg
deliberate rather than drift; the two status.json tests pin the additive
contract (no key when unset, stop_detail when the cap fires).
The barrier-free runtime has no per-call guard yet; a cap that reached it
would be silently unbounded, so it is refused by name rather than warned
about. Sync and async stay two contracts (#809).
@HugoXK

HugoXK commented Nov 19, 2026

Copy link
Copy Markdown
Collaborator Author

Tip SHA: 3b49b89
Base: 76223fd (merge-base, ahead 4 / behind 0)

Ran:

  • Fresh clone at the tip, clean tree; git diff 76223fda...HEAD = 11 files, +567/−28 (budget.py +82/−0, cli.py +37/−5, evolution.py +133/−17, evolve_settings.py +10/−0, evolvespec.py +88/−1, metrics.py +10/−1, runstore.py +55/−4, changelog +11, docs/api.md +3, docs/start.md +14, tests/test_budget_guard.py +124).
  • pytest -o addopts="" -q tests/test_budget_guard.py at the tip → 9 passed in 3.69s.
  • Red first: base 76223fda worktree + only the new test file → 7 failed, 2 passed in 3.75s; the seven: test_max_calls_stops_before_the_next_call, test_max_usd_stops_before_the_call_that_would_overspend, test_the_status_line_names_the_limit_that_fired, test_unset_limits_keep_todays_run, test_a_negative_max_calls_is_refused_with_the_field_named, test_zero_is_a_limit_not_an_absence, test_status_json_carries_the_limit_that_fired. The two green on base are test_status_json_keeps_its_shape_when_no_cap_is_set (passes by construction: no stop_detail key exists there) and test_a_non_numeric_max_usd_is_refused_with_the_field_named — see non-blocking 1.
  • CLI at the tip, real spec: evolve spec.json --max-calls 1 → exit 0, calls=1, state=done, error=None, stop_reason=budget, stop_detail="max_calls: 1 call(s) made, cap 1", status line … done round 0/4 calls=1 … [budget: max_calls: 1 call(s) made, cap 1]; --max-calls -1 → exit 2, spec error: max_calls: must be a non-negative integer, got -1; spec evolve.max_calls=1 (compose-built cap) → 1 call, same budget stop and detail; uncapped run → 52 calls, stop_reason=rounds, stop_detail key absent.
  • Independent checks, from the tip tree:
    • BudgetStop is caught only by name. The worker _run_unit generic except Exception is unchanged from base — the diff inserts the named handler above it, and _propose_one, _score_seed_held_out, the _measure retry loop, the merge/held-out scoring retry and the _bounded round body re-raise or catch it by name, before any generic handler.
    • Main thread closes cleanly too: budget_stop[0] is checked at the top of each round (:4104) and right after the future join (:4558, before any merge); the round body's except BudgetStop sets stop_reason="budget" and breaks without touching run_error/first_error, so the run ends done, error=None.
    • stop_detail is additive: RunStatus.from_dict of a key set without it yields None, and to_dict() omits the key when unset — an old status.json reads as before.
    • budget.py imports only __future__, math, threading, dataclasses, typing (AST check; the runstore occurrences are docstrings, not imports).
    • measured(fn, meter, *, phase=None, guard=None): the two positional parameters are unchanged; guard is keyword-only with a None default.
    • evolve(asynchronous=True) + CallLimit(max_calls=1) raises ValueError and creates no run — called directly, not inferred from the diff.
  • Docs/changelog: python -m tools.gen_api_docs --check → docs/api.md is up to date; python -m tools.changelog_fragments check → 80 changelog fragment(s) OK; mkdocs build --strict (social plugin excluded locally: it needs cairo) → success in 22.60 s; python tests/test_site_navigation.py site → 149 rendered pages pass.
  • CI at this sha, read from the API, all 10 settled: test (3.9) success 11:36:24Z, test (3.10) success 11:33:41Z, test (3.11) success 11:30:22Z, test (3.12) success 11:33:39Z, test (3.13) success 11:28:40Z; build success 11:18:21Z, otel-wire success 11:14:43Z, reviewed tip (report only) success 11:13:24Z, risk tier (report only) success 11:13:23Z, deploy skipped. 9 success + 1 skipped, none red, none pending.
  • Labels at read: enhancement, area:evolve, area:runstore, area:cli. No risk:* label on the PR at posting time (the risk tier job is success).
  • No UI surface: CLI flags plus one status-line / status.json field; the before/after CLI text is quoted in the body, there is no rendered interface to screenshot.

Not run:

  • Full pytest -q locally: this box has no egress, so some rows sit on network timeouts and my run had reached 20% when this verdict was posted. The full suite is green at this sha on CI's five pytest -q legs (3.9–3.13), which is the line.
  • Social cards and Playwright capture (cairo missing); Docker rows (no daemon) — the build/otel-wire jobs cover the wheel and exporter paths.

Non-blocking:

  1. test_a_non_numeric_max_usd_is_refused_with_the_field_named is green on base for the wrong reason: base's "evolve() has no such parameter" message lists cheap_eval_tasks, so "cheap" in message is satisfied by a substring. At the tip the message is evolve.max_usd: must be a non-negative number, got 'cheap', so the behaviour is verified by a direct call above; the row is weak, not wrong.
  2. Diffstat: the dispatch said +524 additions; the tree and the API both say +567 (4 commits, 11 files). No code consequence.
  3. mergeable_state: blocked at read, PR open and non-draft — the required review has not landed.

Verdict: PASS

@HugoXK

HugoXK commented Nov 19, 2026

Copy link
Copy Markdown
Collaborator Author

Advisory read at 3b49b89ac31da1f5a2707c3633675963b346084e — same API read: open, merged: false, non-draft, base 76223fda, 11 files +567/−28. Two blocking findings; the per-call accounting itself is right where it is reached.

Blocking

1. A spec-level max_usd cannot run: usd_per_call leaks into evolve().

agentdescent/evolvespec.py::_limit_from_knobs (the knobs.pop("max_usd", None) line, ~:1160) pops max_usd but leaves usd_per_call in the knob dict, and its own docstring says both must go. compose then hands usd_per_call= to evolve(**kwargs), which has no such parameter.

Offline reproduction, using the counting-agent spec in tests/test_budget_guard.py:

spec = _spec(max_usd=0.015, usd_per_call=0.01)
comp = compose(spec)          # "usd_per_call" in comp.kwargs
runstore.execute(rd)          # TypeError: evolve() got an unexpected keyword argument 'usd_per_call'

The run is recorded failed with that TypeError. So the spec path the changelog advertises ("also max_calls / max_usd in a spec's evolve block") is unusable for dollars — every valid spec-level max_usd needs usd_per_call, and that is exactly the key that crashes. Fix: pop usd_per_call alongside max_usd, and add one row that composes and runs that spec (CALLS == ["agent"], stop_reason="budget").

2. A cap that trips mid-round ends the run failed, not done.

agentdescent/executor.py:142-144 catches every exception from a rollout and reports it as Result(kind="model", error=describe(e)); agentdescent/evolution.py:4329 then does raise _rollout_failure(outcome) and _rollout_failure (:1103-1113) turns that report into RuntimeError("BudgetStop: max_calls=2 reached"), which the round's generic handler at :4285 records as a run error.

agentdescent evolve spec.json --max-calls 2
-> exit 1
-> 20261119-…  failed … calls=2 … [error: max_calls: 2 call(s) made, cap 2]
   error: RuntimeError: BudgetStop: max_calls=2 reached

Same shape with --max-usd 0.05 --usd-per-call 0.02: failed, error: RuntimeError: BudgetStop: max_usd=0.05 reached, exit 1. Reproduced three times out of three at max_calls=2; max_calls=1/3/5/9/17 happen to end clean because the refusal lands in the seed score or the propose wrapper, which is why the nine rows do not catch it. The promised contract — done, stop_reason="budget", no traceback — does not hold as soon as the refusal happens inside an executor-dispatched rollout. A side effect of the same path: status.json carries stop_detail with stop_reason="error" (27 keys), so the detail is no longer only on budget stops. Fix: let BudgetStop escape the executor's catch-all (except BudgetStop: raise before executor.py:142, or carry kind="budget" and return the BudgetStop from _rollout_failure), plus one row that trips the cap mid-round with four workers.

Non-blocking

  • evolution.py:3893-3905: evolve(asynchronous=True) with a cap emits RuntimeWarning: … ignores call_limit= and then raises ValueError: … cannot enforce. Behaviour is the right refusal; only the wording contradicts itself.
  • test_a_non_numeric_max_usd_is_refused_with_the_field_named passes at base for the wrong reason (max_usd is an unknown evolve() kwarg there), so it is not a fail-on-main row; the base run of the new file is 7 failed, 2 passed.

Verified at this tip

  • Run-option charging: max_calls=1 -> exactly one call, done/budget; max_calls=0 -> zero calls, clean stop before the first call; max_usd=0.015/usd_per_call=0.01 -> one call. stop_detail names the limit and the observed value.
  • Caught by name, not by widening: _run_unit (:4275) before its generic :4285; the retry wrapper (:676); seed score (:3146); propose (:4386); merge gate (:4589, :4645); final/test score (:4743, :4761).
  • Seams: metrics.measured gained only the keyword guard (positional call sites unchanged); budget.py imports math/threading/dataclasses/typing only; no new dependency; nested refusal, MCP limits and CLI write dirs untouched.
  • Shape: to_dict() omits stop_detail when None (uncapped status.json stays at its old key set); an old dict without the key reads as None.
  • CLI: bad values refused before any run dir with field and value named — max_calls: must be a non-negative integer, got -1, max_usd needs usd_per_call, both exit 2; a budget stop exits 0 when it lands cleanly.
  • Gates: python -m tools.gen_api_docs --check clean; python -m tools.changelog_fragments check 80 OK.
  • Tests: base 76223fda with only the new file 7 failed, 2 passed; tip tests/test_budget_guard.py 9 passed; engine/budget/runstore/CLI subsets 255 + 91 passed.

Not run: mkdocs build --strict (no mkdocs in this environment) and the full pytest -q (the engine, budget, runstore and CLI subsets above were run instead).

@HugoXK

HugoXK commented Nov 19, 2026

Copy link
Copy Markdown
Collaborator Author

3300d557a4cbb170fd52682dbce6cf82662b6c5d — both blocking findings from my review at 3b49b89a are resolved; no blocking finding at this tip.

Re-read only the two paths and the regression rows, in a fresh clone at the named sha.

1. A spec-level max_usd now runs. _limit_from_knobs pops max_usd and usd_per_call (evolvespec.py:1153-1154, exactly two pops). compose → runstore.execute with evolve={max_usd: 0.015, usd_per_call: 0.01}: exit 0, state=done, calls=1, stop_reason=budget, stop_detail="max_usd: $0.0100 spent, cap $0.0150", error empty. No TypeError.

2. A refusal inside an executor-dispatched rollout ends done. executor.py:143-150 reports Result(kind="budget", error=e.detail()); evolution.py:1111-1116 returns the stop CallLimit recorded (budget.py:127-135); _propose_one re-raises at evolution.py:4332, and _run_unit's except BudgetStop at :4278 sits before the generic :4288 (try: _run_unit_inner at :4271-4272; _propose_one is called at :4500). Behaviour with four workers: --max-calls 2 → exit 0, done, calls=2, budget, "max_calls: 2 call(s) made, cap 2"; --max-usd 0.05 --usd-per-call 0.02 → exit 0, done, calls=2, budget, "max_usd: $0.0400 spent, cap $0.0500".

done.get() regression check (a check, not a blocking reason): a direct ThreadExecutor(max_workers=2).map_rollouts over three specs whose run raises BudgetStop returns in ~1 ms with three kind="budget" results; nothing blocks.

The previous nine rows did not move. The nine original tests/test_budget_guard.py node ids → 9 passed; file plus tests/test_async_early_stop.py → 18 passed; the wider budget/executor/async/runstore/evolution/evolvespec/evolve-settings subset → 257 passed. The three new rows are non-vacuous: with only tests/test_budget_guard.py swapped into a worktree at 3b49b89a, 3 failed, 9 passed (exactly the three new rows); the same file at the tip is green. Unset limits: calls=52, stop_reason=rounds, no stop_detail key. max_calls=0: 0 calls, budget. -1 and max_usd without usd_per_call: exit 2, field named, no run dir. evolve(asynchronous=True) with a cap: ValueError, no preceding RuntimeWarning.

Gates: python -m tools.gen_api_docs --check → docs/api.md is up to date; python -m tools.changelog_fragments check → 80 changelog fragment(s) OK.

Non-blocking, documentation edge. A spec-level max_calls equal to the held-out count (2 here) ends done with stop_reason="max_calls" and no stop_detail: the seed evaluation spends exactly the cap, so the round barrier at evolution.py:4129-4134 stops before any per-call refusal happens (2 calls, no overspend). max_calls=1/5/9 all end budget with the detail. The cap holds in every case; only the changelog sentence "also max_calls … in a spec's evolve block … ends done with stop_reason=budget" reads as if budget were always the reason.

Not run: mkdocs build --strict (no mkdocs in this environment), full pytest -q (the subsets above only).

)

Two contract gaps found after the first tip, both reproduced before the fix:

- A spec-level `max_usd` could not run: `_limit_from_knobs` popped `max_usd`
  but left `usd_per_call`, which `compose` then handed to `evolve()`, so every
  valid dollar cap died with `TypeError: evolve() got an unexpected keyword
  argument 'usd_per_call'` and the run was recorded `failed` with calls=0.
- A cap refused inside an executor-dispatched rollout was recorded as a worker
  failure: `ThreadExecutor._one`'s generic `except Exception` turned the
  `BudgetStop` into `Result(kind="model")`, `_rollout_failure` rebuilt it as
  `RuntimeError("BudgetStop: ...")`, and the round body's generic handler set
  `run_error` -- `state=failed`, `stop_reason=error`, `stop_detail` present.
  `max_calls=2` under four workers hit it; so did `max_usd=0.05` at $0.02/call.

The refusal now travels as its own `Result.kind="budget"` (report, never raise:
`map_rollouts` delivers through a queue, and a bare raise would hang the
reader), and `_rollout_failure` re-raises the stop the `CallLimit` already
recorded -- numbers included -- so the round closes `done` with
`stop_reason=budget`. `CallLimit.stop` exposes that recorded refusal.

Also drops the contradictory `RuntimeWarning` before the async `ValueError`:
the async path refuses a cap, it does not ignore one.
@HugoXK

HugoXK commented Nov 19, 2026 •

Copy link
Copy Markdown
Collaborator Author

Tip SHA: 3300d557a4cbb170fd52682dbce6cf82662b6c5d

Base: 76223fda44de1accf0c718764b0ccf9e53edb3f9 (merge-base; 5 commits, behind 1 of origin/main f8efc731; 3b49b89a..3300d557 is exactly one commit — no force-push). Diff 76223fda..3300d557 = 13 files +647/−32, matching the PR. PR at read time: open, non-draft, mergeable: true, mergeable_state: clean (GitHub Date 12:29:17Z), labels enhancement, area:evolve, area:runstore, area:cli.

Ran — fresh clone at this sha, PYTHONPATH=<clone>, pytest -o addopts="" -q -p no:cacheprovider:

  • tests/test_budget_guard.py → 12 passed, 10 warnings in 2.89s.
  • tests/test_executor.py tests/test_async_early_stop.py tests/test_runstore_cli.py tests/test_spec_usage.py tests/test_run_panel.py → 100 passed, 9 warnings in 43.07s.
  • Red-first, worktree at 3b49b89a with only tests/test_budget_guard.py swapped in → 3 failed, 9 passed, 11 warnings in 4.04s. The three ids are exactly the new rows; test_a_spec_level_dollar_cap_runs fails on assert 'usd_per_call' not in {... 'call_limit': CallLimit(max_calls=None, max_usd=0.015, usd_per_call=0.01)}.kwargs, the two rollout rows on assert 'failed' == 'done', with the base warnings RuntimeError: BudgetStop: max_calls=2 reached and RuntimeError: BudgetStop: max_usd=0.05 reached.
  • Base repro at 3b49b89a, spec-level max_usd=0.015 + usd_per_call=0.01, status read back: state='failed', calls=0, stop_reason=None, error="TypeError: evolve() got an unexpected keyword argument 'usd_per_call'", 26 keys. Same-sha run option max_calls=2 on the four-worker spec → state='failed', stop_reason='error', error="RuntimeError: BudgetStop: max_calls=2 reached"; max_usd=0.05 at $0.02/call → state='failed', error="RuntimeError: BudgetStop: max_usd=0.05 reached".
  • Row 1, spec-level dollar cap end to end (compose → runstore.execute): usd_per_call absent from comp.kwargs (['usd_per_call'] is empty among usd keys), comp.call_limit = CallLimit(max_calls=None, max_usd=0.015, usd_per_call=0.01); run → 1 call, state='done', stop_reason='budget', stop_detail="max_usd: $0.0100 spent, cap $0.0150", error=None, calls=1, 27 keys, no TypeError.
  • Row 2, n_workers=4 + max_calls=2 refused inside a rollout: state='done', error=None, calls=2, stop_reason='budget', stop_detail="max_calls: 2 call(s) made, cap 2", 27 keys. map_rollouts checked directly, not only through status.json: 8 specs on 4 workers under one budget, limit refusing the third call → 8 results yielded, one per spec, in 0.002s wall, no raise, kinds {'budget', 'ok'}, 2 ok + 6 budget; every spec puts exactly one result, so done.get() has nothing to block on.
  • Row 3, max_usd=0.05 at $0.02/call, four workers: state='done', error=None, calls=2, stop_detail="max_usd: $0.0400 spent, cap $0.0500". The same pair as a run option also reads usd=0.04.
  • Row 4, the original rows unchanged: uncapped → 52 calls, stop_reason='rounds', 26 keys, no stop_detail key; max_calls=0 → 0 calls, done, budget, "max_calls: 0 call(s) made, cap 0"; max_calls=1 → 1 call, done, budget; max_calls=-1 → evolve.max_calls: must be a non-negative integer, got -1; max_usd="cheap" → evolve.max_usd: must be a non-negative number, got 'cheap'; max_usd without usd_per_call → max_usd needs usd_per_call: a dollar cap cannot be charged without a per-call price (pass --usd-per-call on the command line); the run option execute(max_calls=-1) is refused before running (RunStoreError, state='failed', no calls).
  • Row 5, agentdescent status <run>, verbatim: 20261119-124043-73cda0 done round 0/4 calls=2 best=0.000 text:You are a helpful assistant. [budget: max_calls: 2 call(s) made, cap 2] — limit and observed value on the same line, error: segment empty. The spec-level dollar run reads [budget: max_usd: $0.0100 spent, cap $0.0150].
  • Gates: python -m tools.gen_api_docs --check → docs/api.md is up to date; python -m tools.changelog_fragments check → 80 changelog fragment(s) OK; mkdocs build --strict (social plugin excluded locally) → exit 0, built in 20.17 s.
  • CI at this sha (GitHub Date 12:28:56Z): 10 check runs, all settled — test (3.9) success 12:28:49Z, (3.10) 12:20:19Z, (3.11) 12:16:13Z, (3.12) 12:16:29Z, (3.13) 12:20:26Z, build 12:02:46Z, otel-wire 12:02:15Z, reviewed tip 12:01:16Z, risk tier 12:01:13Z; deploy skipped 12:02:46Z.

Not run:

  • Local full pytest -q to completion — stopped at [ 56%]; CI's five pytest -q legs are the gate and all five are green above.
  • Anything needing a model or the network: the suite is offline by design and no live call was made.
  • tests/test_sandbox_container.py — no docker daemon on this box; untouched by this change.

No UI surface: terminal text only — the agentdescent status line and the status.json keys. There is no rendered interface to screenshot, which is why the body carries no before/after images.

Non-blocking observation: spec-level max_calls keeps its round-barrier meaning as well, so with a seed phase that has already spent 2 calls, max_calls: 2 on four workers ends stop_reason="max_calls" with no stop_detail, while max_calls: 3..6 end budget with the detail and exactly N calls. Both make exactly N calls; the barrier winning at its own boundary is the behaviour _limit_from_knobs documents, so I do not read it as a regression.

Verdict: PASS — advisory QA verdict at 3300d557a4cbb170fd52682dbce6cf82662b6c5d only. No merge action from QA: this report is advisory, the merge gate is the repo's review decision on the head sha, and the merge itself is executed by the repo's merge process.

@HugoXK

HugoXK commented Nov 19, 2026

Copy link
Copy Markdown
Collaborator Author

Advisory read at 3300d557a4cbb170fd52682dbce6cf82662b6c5d — from the QA seat, not a counting PASS (.github/agent/review-policy.json counts Birfy only), and I take no merge action.

T3 by path (agentdescent/runstore.py, agentdescent/evolution*.py), and over the T2 file cap at 13 files, so T3 either way. Live in the same read as the head: open, draft: false, mergeable_state: clean, base 76223fda, body Tip SHA: == head.

Tip SHA: 3300d55
Ran:

  • pytest -q -n 8 at the tip → 21 failed, 7744 passed, 63 skipped, 10 errors (281.78s). The 31 red ids (21 failed + 10 errors: docker sandboxes, apply-noop, era_srbench, genesis/skill examples, wheel smoke) are the same set, id for id, as on unmodified main f8efc731 under the same command: 0 new red, 0 vanished.
  • fail-on-main, both halves: with only the tip's two test files copied onto f8efc731, tests/test_budget_guard.py → 10 failed, 2 passed; tests/test_async_early_stop.py → 6 passed.
  • python -m tools.gen_api_docs --check → docs/api.md is up to date.
  • python -m tools.changelog_fragments check → 80 fragments OK.
  • mkdocs build --strict at the tip (docs/api.md, docs/start.md touched; social plugin removed for the offline box) → pass, 11.00s.
    Not run:
  • Docker sandboxes and the wheel smoke — the 10 errors and 1 failure above are host state, not this change.
  • Ark real-model run — no model path changed.
  • risk_gate.py — no gh on this box; the tier above is computed from .github/agent/risk-rules.json.

Findings:

  • none on the code.

Two body nits for your decision, not defects:

  1. The body's own suite line reads 22 failed / 7745 passed; I read 21 failed / 7744 passed at the same tree. One row is flaky under -n 8; the red set equals main's and is what counts.
  2. The body opens with Partial: #963 and carries no Fixes #N, while the title names #1166. The template's line is Fixes #N, and #1166 is open, so it would not close on merge as written.

No UI surface: run-loop stop behaviour plus two docs pages; nothing rendered changes.

@Birfy

Birfy commented Nov 19, 2026

Copy link
Copy Markdown
Collaborator

Self-review (Birfy)

Tip SHA: 3300d55
Ran:

  • .github/scripts/risk_gate.py --pr 1167 → T3 (evolution.py, runstore.py; 676 lines / 12 files, 1 generated not counted)
  • pytest -q tests/test_budget_guard.py tests/test_async_early_stop.py → 18 passed
  • python -m tools.changelog_fragments check → fragments OK
  • python -m tools.gen_api_docs --check → docs/api.md is up to date
  • Red first: tests/test_budget_guard.py on origin/main (f8efc73) → failed (execute() has no max_calls / max_usd)
  • Required CI at this tip → green (build, test 3.9–3.13, otel-wire)
    Not run:
  • Full offline suite locally — the CI test matrix is the line
  • Ark real-model run — no model path changed; the new rows use an in-process counting agent
    Findings:
  • No actionable defects. The call is reserved before it is made; a cap ends the run done with stop_reason=budget and a stop_detail, including mid-rollout refusals. Spec-level max_usd + usd_per_call are popped before evolve(). Async + call cap is refused by name. Prior blocking findings at 3b49b89 are resolved at this tip.
    Verdict: PASS

@Birfy

Birfy commented Nov 19, 2026

Copy link
Copy Markdown
Collaborator

This now conflicts with main after #1169 and #1170 landed (both also touch agentdescent/cli.py). Please merge origin/main into the branch (no rebase) so it can be re-checked and landed. Tip at the conflict: 3300d557a4cbb170fd52682dbce6cf82662b6c5d.

@github-actions github-actions Bot added risk:T3 Risk tier T3 (core loop, release, process; needs Danyang's click), set by the controller agent-pr Pull request opened by an agent labels Nov 19, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

agent-pr Pull request opened by an agent area:cli The agentdescent command line (cli.py, doctor, plan/evolve/status/apply) area:evolve The evolve loop: evolution, async_evolve, aggregator, population, policies, strategies area:runstore Run store, ledger, checkpoint, resume and replay enhancement New feature or request risk:T3 Risk tier T3 (core loop, release, process; needs Danyang's click), set by the controller

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants