Skip to content

feat(mcp): let the plan tool run the preflight probe (#1094) - #1170

Merged
Birfy merged 2 commits into
mainfrom
agent/1094-mcp-plan-probe
Nov 19, 2026
Merged

Birfy merged 2 commits into
mainfrom
agent/1094-mcp-plan-probe

Conversation

@HugoXK

@HugoXK HugoXK commented Nov 19, 2026 •

Copy link
Copy Markdown
Collaborator

Fixes #1094

What changed

The CLI has had plan --probe since #1048, but a host driving the MCP server could only call plan, which runs nothing: an expired subscription or a wrong model name surfaced only after start, as a failed run. This gives the tool the same preflight and teaches every installed host to ask for it.

  • agentdescent/mcp.py: Tools.plan(spec, usd_per_call=None, budget_usd=None, probe: bool = False) (line 665) and the registered plan wrapper (line 1389) take probe. When true _with_probe (line 405) adds probe: the same probe_payload(spec) object the CLI prints. A refusal sets ok: false and next to the repair action, never to start.
  • agentdescent/probe.py: first_failed_next() (line 157) moves here from cli.py (_first_failed_probe_next), so the CLI (cli.py:58, call sites 2577/4835) and the tool pick a refusal's next from one place.
  • agentdescent/integrations/SKILL.md (line 143): the plan step now says to run the first use of a model/endpoint with the probe on (plan --probe, MCP plan(spec, probe=true)), and that a refusal's next goes to the user with no start.
  • docs/plugins.md (line 248 flow, line 261 bullet): the Path A flow and the plan bullet describe the probe on both surfaces.
  • Regenerated integrations/claude-code/skills/agentdescent/SKILL.md and integrations/dsh-agentdescent/lib/index.js; changelog.d/1094-mcp-plan-probe.md.

Out of scope: no second probe implementation in mcp.py, no MCP tool wraps algorithms run (none exists — algorithm_spec writes a spec).

Red first (at f8efc731, only tests/test_mcp_plan_probe.py swapped in)

TypeError: Tools.plan() got an unexpected keyword argument 'probe'
AssertionError: assert 'probe' in '---\nname: agentdescent\n…'

3 failed, 1 passed. The pass is the default-path row (test_omitting_the_probe_returns_exactly_the_old_payload), which pins the 9-key payload so the new parameter cannot change what a plain plan returns.

Verification

Tip SHA: b7dfd33
Ran:

  • pytest -q tests/test_mcp_plan_probe.py → 4 passed
  • focused: tests/test_mcp.py tests/test_mcp_meta.py tests/test_api_docs.py tests/test_mcp_plan_probe.py tests/test_plan_probe.py tests/test_integrations.py → 181 passed, 8 skipped
  • python3 -m tools.gen_api_docs --check → docs/api.md is up to date
  • python3 -m tools.changelog_fragments check → 81 changelog fragment(s) OK
  • registered plan tool schema at the tip, read through the SDK (installed on this box) — probe is in it:
"probe": {"default": false, "title": "Probe", "type": "boolean"}
  • base repro for the skill row: every generated SKILL.md had probe 0 times at f8efc731.

Not run:

  • local full pytest -q — stopped at 73%; CI's five test (3.9–3.13) legs are the gate.
  • mkdocs build --strict — did not complete on this box: fonts.google.com connect timeout. CI build is the line.

Acceptance rows (where they are implemented / pinned):

row where
A. 400 InvalidSubscription stub → ok is False, probe.ok is False, next has no start mcp.py:405 + probe.py:157; test at test_mcp_plan_probe.py:105
B. 200 stub → probe.ok is True, plan ok stays true mcp.py:405; test at test_mcp_plan_probe.py:116
C. probe omitted → payload key set unchanged (9 keys, baseline captured at f8efc731) test_mcp_plan_probe.py:41,125
D. install of all 5 hosts → every generated SKILL.md mentions the probe, host_status ok integrations/SKILL.md:143; test at test_mcp_plan_probe.py:131
E. tests/test_plan_probe.py still green (CLI side of #1048) in the focused run above

No UI surface: one MCP tool argument plus its JSON payload and the docs/host-skill text; nothing is rendered.

@HugoXK HugoXK added area:plugins Host plugins and installers: Claude Code, Codex, DSH, OpenCode, MCP server enhancement New feature or request labels Nov 19, 2026
Red first: on f8efc73 Tools.plan() has no probe parameter (TypeError) and
every generated SKILL.md has zero occurrences of "probe". The default-path
row pins the old key set so adding the parameter cannot change what plain
plan returns.
A host driving the MCP server could only call plan, which runs nothing, so
an expired subscription surfaced after start as a failed run. plan now takes
probe: bool = False and, when true, returns the same probe_payload rows the
CLI's plan --probe prints; a refusal makes the result ok=false and points
next at the repair action, never at start. The helper that picks that sentence
moves to probe.py so both surfaces share one source, and the shared host
SKILL.md teaches the probe step on a first model. Default plan is unchanged.
@HugoXK

HugoXK commented Nov 19, 2026

Copy link
Copy Markdown
Collaborator Author

Tip SHA: b7dfd336e90d16e1ae35e37e73fd6e881b3fbb9c

Base: f8efc7315497191ad35b87105f9d88c1e5119e4e (merge-base; ahead 2 / behind 0; 2 commits, 9 files +226/−17). PR at read: open, non-draft, mergeable: true, mergeable_state: clean, labels enhancement, area:plugins.

Ran — fresh clone at this sha, PYTHONPATH=<clone>, pytest -o addopts="" -q -p no:cacheprovider:

  • tests/test_mcp_plan_probe.py tests/test_plan_probe.py → 20 passed in 14.81s.
  • Red first, worktree at f8efc731 with only tests/test_mcp_plan_probe.py added → 3 failed, 1 passed in 2.58s: TypeError: Tools.plan() got an unexpected keyword argument 'probe' on rows A and B, assert 'probe' in '…' on the host-skill row; the pass is the deliberate default-path row. Matches the author's count.
  • tests/test_mcp.py tests/test_mcp_meta.py tests/test_mcp_algorithm_spec.py → 58 passed, 6 skipped.
  • tests/test_integrations.py -k "skill or plugin or dsh or claude or drift" → 22 passed, 2 skipped, 83 deselected.
  • Gates: python -m tools.gen_api_docs --check → docs/api.md is up to date; python -m tools.changelog_fragments check → 81 changelog fragment(s) OK; mkdocs build --strict (social plugin excluded locally: it needs cairo) → exit 0, 12.20 s.

Independently, with my own stub server and my own 8-task spec (agent and reflect both openai_compatible):

  • A. 400 Ark InvalidSubscription → ok is False, probe.ok is False, exactly one failed probe row, exactly one HTTP hit (nothing else ran), next = renew the subscription at the provider, or point api_key_env / model at another plan — no start; no run-store directory is created at all.
  • B. 200 → ok is True, probe.ok is True, rows ['ok'], one hit, no next.
  • C. probe omitted → the 9-key payload, and byte-identical to what the same spec returns at f8efc731 in a worktree there (same store path, same env) — the default path is unchanged, not merely the same key set.
  • D. Install into a clean home for all five hosts (dsh, claude-code, codex, opencode, grok-bot) → five SKILL.md, every one contains "probe", and every host_status reads state: ok, skill: ok.
  • E. tests/test_plan_probe.py is green (16 of the 20 above), and the CLI side is unchanged: agentdescent plan <spec> --probe --json against the same 400 stub is byte-identical between base f8efc731 and this tip (4255 B stdout, rc 1, empty stderr), so moving first_failed_next changed nothing a CLI user sees.
  • Edge rows: probe=True on a spec whose plan is already ok: False (bad kind) returns that shape error with no probe key and sends nothing; the caller's spec dict is not mutated by a probe; keyword and positional call forms agree; probe exists on no other tool and mcp.py holds no second probe implementation (_with_probe → probe_payload).
  • Surface: Tools.plan(spec, usd_per_call=None, budget_usd=None, probe: bool = False); the registered wrapper forwards it positionally in the same order; TOOL_DESCRIPTIONS["plan"] documents probe=true and the ok: false refusal.
  • CI at this sha, my own read, all 10 settled: test (3.9) success 13:42:21Z, (3.10) 13:40:09Z, (3.11) 13:38:41Z, (3.12) 13:39:04Z, (3.13) 13:38:45Z, build 13:25:09Z, otel-wire 13:24:32Z, reviewed tip 13:23:31Z, risk tier 13:23:29Z; deploy skipped 13:25:09Z — 9 success + 1 skipped, none red.

Not run:

  • Full pytest -q locally — CI's five pytest -q legs at this sha are the line and all five are green above.
  • The SDK-built plan JSON schema: no mcp package on this box (mergedvenv, the mkdocs venv and system python all fail import mcp), so I checked the signature, the wrapper and the description in source instead; the schema Dev read on a box with the SDK is not independently confirmed by me.
  • Docker rows — no daemon on this box.

No UI surface: one MCP tool argument and its JSON payload, plus docs and host-skill text; nothing is rendered, so no screenshot applies.

Non-blocking:

  1. The PR body's last line is No UI surface. without the reason; the owner's rule asks for No UI surface: <why>.
  2. No risk:* label on the PR at read time while the risk tier job is green (report only).
  3. probe=true spends one real provider call per model-backed agent by design; the default is verified off and the docs/skill text say so.

Verdict: PASS — advisory QA verdict at b7dfd336e90d16e1ae35e37e73fd6e881b3fbb9c only. No merge action from QA: this report is advisory, the merge gate is the repo's review decision on the head sha, and the merge itself is executed by the repo's merge process.

@HugoXK

HugoXK commented Nov 19, 2026

Copy link
Copy Markdown
Collaborator Author

b7dfd336e90d16e1ae35e37e73fd6e881b3fbb9c — no blocking code finding at this tip.

Fresh clone at the named sha (f8efc731..b7dfd336, fast-forward, 2 commits, 9 files, +226/−17). I read the diff, then drove the probe against a local HTTP stub rather than relying on the new tests alone.

Contract. Tools.plan(spec, usd_per_call=None, budget_usd=None, probe=False) (mcp.py:665) and the registered wrapper (mcp.py:1388-1391) take probe; the built server's plan schema now has probe: {"default": false, "type": "boolean"}, required stays ["spec"], and the tool count stays 13 (base had 3 properties, tip 4). _with_probe (mcp.py:405-424) reuses probe_payload and puts its object under probe; a refusal sets ok: false and next from the first failed row. first_failed_next (probe.py:157-166) is the single source now; _first_failed_probe_next is gone and both CLI call sites (cli.py:2577, :4835) use it. Against the stub: 400 InvalidSubscription → 1 hit, ok=false, probe.ok=false, row failed/agent, next = the renewal sentence (no start); 200 → 1 hit, ok=true, probe.ok=true, row ok; a bad spec (SpecError) → {ok:false,error} with no probe key and 0 HTTP hits; the embedded probe object equals a direct probe_payload(spec) (minus row seconds).

Default path unchanged. Tools().plan(spec) is byte-identical between f8efc731 and the tip (same 9 keys and same values). first_failed_next returns the same sentence as the old cli._first_failed_probe_next for every payload I fed both (failed-with-next, failed-without-next, skipped-only, empty rows, rows: None). The CLI plan --probe --json output (minus row seconds) is identical at base and tip for both the 400 and 200 stubs.

Generated artefacts are exact, not just "regenerated". _skill_body() (26390 bytes, probe ×4) equals the body of integrations/claude-code/skills/agentdescent/SKILL.md; dsh_plugin_source() equals integrations/dsh-agentdescent/lib/index.js byte for byte. Those are the only two checked-in files that embed the skill (git grep), so nothing else needed regeneration.

Tests and gates. At the tip: tests/test_mcp_plan_probe.py → 4 passed; the wider focused set (test_mcp, test_mcp_meta, test_api_docs, test_plan_probe, test_integrations) → 182 passed, 2 skipped, 1 failed — the failure is test_the_launcher_install_writes_ignores_a_shadowing_cwd, which fails identically at f8efc731 (pre-existing .local/bin stub). Red first at f8efc731 with only the new file → 3 failed, 1 passed (the default-path row). python -m tools.gen_api_docs --check → docs/api.md is up to date; python -m tools.changelog_fragments check → 81 fragment(s) OK.

Non-blocking — MCP refusal wording (probe.py:68-79, mcp.py:405-424). _with_probe calls first_failed_next(result) with the default retry="plan --probe", and each failed row's next comes from _next_step(error) with the same default, so a ProviderConfigError refusal in an MCP reply says "…re-run plan --probe". The fallback sentence itself is not reachable today: every failed row probe_payload produces carries a non-empty next, so this only bites if row construction changes.

Non-blocking — docs (docs/mcp-path-a.md:15, :27). The Path A tool table and flow line still say plan (show the user), while docs/plugins.md:248,261 and the shared skill teach the probe. The Path A list is what a host operator reads, so a clause there would match the rest of the PR.

Not run: mkdocs build --strict (no mkdocs in this environment; docs/plugins.md changed); full pytest -q (the focused set above only).

Merge path: the owner's PASS at this tip and the explicit click; nothing here approves or merges.

@HugoXK

HugoXK commented Nov 19, 2026

Copy link
Copy Markdown
Collaborator Author

Advisory read at b7dfd336e90d16e1ae35e37e73fd6e881b3fbb9c — from the QA seat, not a counting PASS (.github/agent/review-policy.json counts Birfy only), and I take no merge action.

T3 by path (integrations/**, agentdescent/integrations/**). Live in the same read as the head: open, draft: false, mergeable_state: clean, base f8efc731, body Tip SHA: == head, 9 files +226/−17.

Tip SHA: b7dfd33
Ran:

  • pytest -q -n 8 at the tip → 21 failed, 7738 passed, 63 skipped, 10 errors (273.31s). Same 31 red ids as on unmodified main f8efc731 under the same command: 0 new red, 0 vanished.
  • fail-on-main: tests/test_mcp_plan_probe.py only, onto f8efc731 → 3 failed, 1 passed, exactly the body's row.
  • python -m tools.gen_api_docs --check → docs/api.md is up to date (mcp.py changed a public signature).
  • python -m tools.changelog_fragments check → 81 fragments OK.
  • mkdocs build --strict at the tip (docs/plugins.md touched; social plugin removed for the offline box) → pass, 11.16s.
    Not run:
  • Docker sandboxes and the wheel smoke — the red ids above are host state, not this change.
  • Ark real-model run — no model path changed.
  • risk_gate.py — no gh on this box; the tier above is computed from .github/agent/risk-rules.json.

Findings:

  • none. Probe at the tip: Tools.plan params are ['self','spec','usd_per_call','budget_usd','probe']; first_failed_next lives in probe.py, and _first_failed_probe_next is gone from cli.py.
  • The body's Not run: full pytest line is closed by the run above.

No UI surface: an MCP tool argument plus plugin docs; nothing rendered changes.

@Birfy

Birfy commented Nov 19, 2026

Copy link
Copy Markdown
Collaborator

Self-review (Birfy)

Tip SHA: b7dfd33
Ran:

  • .github/scripts/risk_gate.py --pr 1170 → T3 (integrations/SKILL.md, generated host skill, dsh lib; 243 lines / 9 files)
  • pytest -q tests/test_mcp_plan_probe.py tests/test_plan_probe.py tests/test_mcp.py tests/test_mcp_meta.py tests/test_api_docs.py → 76 passed, 6 skipped
  • python -m tools.changelog_fragments check → 81 fragment(s) OK
  • python -m tools.gen_api_docs --check → docs/api.md is up to date
  • Red first: the new tests on origin/main (f8efc73) → 3 failed, 1 passed (TypeError on plan(probe=…); skill text has no probe; the omit-probe payload row already passed)
  • Required CI at this tip → green (build, test 3.9–3.13, otel-wire)
    Not run:
  • Full offline suite locally — the CI test matrix is the line
  • Ark real-model run — no model path changed; the new probe rows use a local HTTP stub
    Findings:
  • No actionable defects. MCP plan(..., probe=True) reuses probe_payload / first_failed_next; a refusal sets ok: false and next to the repair, never start. Plain plan keeps the old key set.
    Verdict: PASS

@Birfy
Birfy merged commit 367d140 into main Nov 19, 2026
10 checks passed
@Birfy
Birfy deleted the agent/1094-mcp-plan-probe branch November 19, 2026 16:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:plugins Host plugins and installers: Claude Code, Codex, DSH, OpenCode, MCP server enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

plugin/MCP: the #1048 preflight probe is CLI-only — MCP plan has no probe, and no host skill mentions it

2 participants