Skip to content

snapshot eval: a skill_dir candidate is staged under "artifact", so the dest agent never reads it and a successful demo re-eval scores 0 #1164

Closed

Description

@Birfy

现象

agentdescent demo 离线跑通后,held-out 从 0.000 升到 1.000,候选里的 references/rules.md 已是 COLUMN: amount。把该 run 导出为可移植快照再对同一份 demo spec 做 snapshot eval,文档说 dest 的 target 会被忽略、只借用任务和打分器,候选是快照本身。

实际得到的却是 ok: true、failed: []、reward: 0.0,12 条任务全是 0。这不是打分器坏了,也不是快照内容错了:快照文件确实是 COLUMN: amount。根因是 #1150 之后 _write_target 把候选固定物化到临时目录下名为 artifact 的目录,compose 把 artifact_id 设成这个目录名,claude_skill 布局把树写到 .claude/skills/artifact/。demo 的离线 agent(以及 Agent Skills「目录名必须等于 SKILL.md name」的约定)去读 .claude/skills/csv-total/references/rules.md,找不到规则就回退到列 id,把 1+2+…+6=21 当成答案,对不上 gold 321。

用户沿文档最短路径(demo → snapshot export → snapshot eval)看不到已经学到的 reward,可移植快照的再评估是错的。

复现

环境:ThinkFlowLab/agentdescent main f2eda2e7a4b3b87a7251e39e4a7680669ddc0906(#1160,0.10.0),pip install -e ".[mcp]"。全程离线,不需要 API key。

agentdescent demo --dir ./demo --json
# ok: true, run_id 例如 20261119-093843-182ed6
# held-out: initial_reward 0.0 -> final_reward 1.0, learned: rules.md -> 'COLUMN: amount'

agentdescent snapshot export 20261119-093843-182ed6 --out ./skill.json
# skill.json 里 files["references/rules.md"] == "COLUMN: amount\n",id 为 csv-total@v2

agentdescent snapshot eval ./skill.json --spec ./demo/spec.json

输出要点(有无 --json 都一样):

{
  "snapshot_id": "csv-total@v2",
  "n": 12,
  "reward": 0.0,
  "scores": {"0": 0.0, "...": 0.0, "11": 0.0},
  "failed": [],
  "kind": "skill_dir",
  "ok": true
}

对照:同一快照物化后,tree_runner 的 workspace 里是 .claude/skills/artifact/references/rules.md(内容已是 COLUMN: amount),而 agentdescent.demo:offline_agent 读的是 .claude/skills/csv-total/references/rules.md。对第一条任务直接调用 compose 得到的 run,输出是 '21',gold 是 '321'。

代码:agentdescent/artifact_snapshot.py 的 _write_target 在 #1150 后把目录名写死为 artifact(os.path.join(tmp, "artifact")),evaluate_snapshot_spec 再 replace(spec, target=target)。EvolveSpec.artifact_id() 优先用 name,否则用 target 的 basename,因此这里变成 "artifact"。tests/test_snapshot.py 的 evaluate_snapshot_spec 只覆盖了 kind=text + echo agent,没有覆盖 skill_dir + claude_skill 布局。

预期

对这份已学到 COLUMN: amount 的 demo 快照,snapshot eval --spec ./demo/spec.json 应给出接近 1.0 的 reward(与 demo 的 final_reward 一致),而不是全 0。候选必须按快照的 artifact_id(这里是 csv-total)出现在 dest agent 会读的布局路径下。#1150 的约束仍然有效:临时路径只能是 tmp 下经过净化的单层名字,不能把未净化的 artifact_id 直接拼进路径。

建议修法(单 PR)

不要为了封闭路径而丢掉名字。复用已有的 EvolveSpec.artifact_id() 净化规则([^A-Za-z0-9_.-]+ → -,空则回退 artifact),让 _write_target / evaluate_snapshot_spec 把候选写到 tmp/<sanitized_id>,或在 replace 时同时设置 name=snapshot.artifact_id(artifact_id() 优先读 name),使 claude_skill 前缀成为 .claude/skills/csv-total/。两种都可以,但目录名和 skill name 应对齐。在 tests/test_snapshot.py 用 demo 的 skill_dir 快照 + agentdescent.demo:offline_agent dest spec 钉住 reward(例如 COLUMN: amount → 1.0;错误种子 → 0.0),并断言 staging 前缀带的是净化后的 csv-total 而不是 artifact。现有的 artifact_id 逃逸测试(#1150 / #1153)必须继续失败于越界路径。

验收

  • 上述 demo → export → snapshot eval --spec ./demo/spec.json 得到 ok: true、failed: []、reward == 1.0(或与 demo final_reward 一致)。
  • 含 /、.. 或不安全 artifact_id 的快照在 eval 物化时仍不能写出 tmp 之外。
  • tests/test_snapshot.py 里现有的 text / kind 不匹配用例仍然通过。
  • 不必改 evolve 本身,也不必依赖真实模型。

Activity

  1. added
    bugSomething isn't working
    area:evolveThe evolve loop: evolution, async_evolve, aggregator, population, policies, strategies
    area:cliThe agentdescent command line (cli.py, doctor, plan/evolve/status/apply)
    on Nov 19, 2026
  2. HugoXK commented on Nov 19, 2026

    @HugoXK
    Collaborator

    Verified independently at 76223fda (main after #1162 / #1165), offline, no API key.

    demo → run 20261119-050528-6758c3, initial_reward 0.0 → final_reward 1.0, rules.md -> 'COLUMN: amount'; snapshot export → csv-total@v2 with SKILL.md + references/rules.md = COLUMN: amount\n; snapshot eval ./skill.json --spec ./demo/spec.json → n 12, reward 0.0, kind skill_dir, ok true, failed []. Matches the report exactly.

    Root cause confirmed line by line: _write_target stages tmp/artifact (agentdescent/artifact_snapshot.py:401); replace(spec, target=…) leaves name=None, so EvolveSpec.artifact_id() falls back to the basename → artifact (agentdescent/evolvespec.py:337); tree_runner(name=aid) then writes .claude/skills/artifact/ (agentdescent/runners.py:67); demo:offline_agent reads the fixed .claude/skills/csv-total/references/rules.md (agentdescent/demo.py:94), misses it, and sums column id → 21 against gold 321.

    The name= route is the smaller of the two, and it keeps the #1150 guarantee: in evaluate_snapshot_spec, add name=snapshot.artifact_id to the existing replace(...). Staging still uses the fixed tmp/artifact path; only the destination spec's layout prefix changes to .claude/skills/csv-total (the runner resolves the prefix from the sanitized artifact_id()). Same snapshot, same dest spec, one probe:

    name=artifact_id (today)   artifact_id=artifact   reward=0.000 failed=()
    name=csv-total (fix)       artifact_id=csv-total  reward=1.000 failed=()
    

    Test gap also confirmed: tests/test_snapshot.py:146 is the only evaluate_snapshot_spec case, and it is kind="text" + agent="echo"; no skill_dir row exercises the staged workspace shape, which is why CI is green on this.

    One edge worth stating in the slice: the directory name then follows snapshot.artifact_id, not the name: frontmatter inside SKILL.md. They agree here (csv-total), so the slice is correct for this path; a snapshot whose frontmatter name differs from its artifact_id would need a separate card.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area:cliThe agentdescent command line (cli.py, doctor, plan/evolve/status/apply)area:evolveThe evolve loop: evolution, async_evolve, aggregator, population, policies, strategiesbugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions