Repository navigation
Description
现象
agentdescent demo 离线跑通后,held-out 从 0.000 升到 1.000,候选里的 references/rules.md 已是 COLUMN: amount。把该 run 导出为可移植快照再对同一份 demo spec 做 snapshot eval,文档说 dest 的 target 会被忽略、只借用任务和打分器,候选是快照本身。
实际得到的却是 ok: true、failed: []、reward: 0.0,12 条任务全是 0。这不是打分器坏了,也不是快照内容错了:快照文件确实是 COLUMN: amount。根因是 #1150 之后 _write_target 把候选固定物化到临时目录下名为 artifact 的目录,compose 把 artifact_id 设成这个目录名,claude_skill 布局把树写到 .claude/skills/artifact/。demo 的离线 agent(以及 Agent Skills「目录名必须等于 SKILL.md name」的约定)去读 .claude/skills/csv-total/references/rules.md,找不到规则就回退到列 id,把 1+2+…+6=21 当成答案,对不上 gold 321。
用户沿文档最短路径(demo → snapshot export → snapshot eval)看不到已经学到的 reward,可移植快照的再评估是错的。
复现
环境:ThinkFlowLab/agentdescent main f2eda2e7a4b3b87a7251e39e4a7680669ddc0906(#1160,0.10.0),pip install -e ".[mcp]"。全程离线,不需要 API key。
agentdescent demo --dir ./demo --json
# ok: true, run_id 例如 20261119-093843-182ed6
# held-out: initial_reward 0.0 -> final_reward 1.0, learned: rules.md -> 'COLUMN: amount'
agentdescent snapshot export 20261119-093843-182ed6 --out ./skill.json
# skill.json 里 files["references/rules.md"] == "COLUMN: amount\n",id 为 csv-total@v2
agentdescent snapshot eval ./skill.json --spec ./demo/spec.json输出要点(有无 --json 都一样):
{
"snapshot_id": "csv-total@v2",
"n": 12,
"reward": 0.0,
"scores": {"0": 0.0, "...": 0.0, "11": 0.0},
"failed": [],
"kind": "skill_dir",
"ok": true
}对照:同一快照物化后,tree_runner 的 workspace 里是 .claude/skills/artifact/references/rules.md(内容已是 COLUMN: amount),而 agentdescent.demo:offline_agent 读的是 .claude/skills/csv-total/references/rules.md。对第一条任务直接调用 compose 得到的 run,输出是 '21',gold 是 '321'。
代码:agentdescent/artifact_snapshot.py 的 _write_target 在 #1150 后把目录名写死为 artifact(os.path.join(tmp, "artifact")),evaluate_snapshot_spec 再 replace(spec, target=target)。EvolveSpec.artifact_id() 优先用 name,否则用 target 的 basename,因此这里变成 "artifact"。tests/test_snapshot.py 的 evaluate_snapshot_spec 只覆盖了 kind=text + echo agent,没有覆盖 skill_dir + claude_skill 布局。
预期
对这份已学到 COLUMN: amount 的 demo 快照,snapshot eval --spec ./demo/spec.json 应给出接近 1.0 的 reward(与 demo 的 final_reward 一致),而不是全 0。候选必须按快照的 artifact_id(这里是 csv-total)出现在 dest agent 会读的布局路径下。#1150 的约束仍然有效:临时路径只能是 tmp 下经过净化的单层名字,不能把未净化的 artifact_id 直接拼进路径。
建议修法(单 PR)
不要为了封闭路径而丢掉名字。复用已有的 EvolveSpec.artifact_id() 净化规则([^A-Za-z0-9_.-]+ → -,空则回退 artifact),让 _write_target / evaluate_snapshot_spec 把候选写到 tmp/<sanitized_id>,或在 replace 时同时设置 name=snapshot.artifact_id(artifact_id() 优先读 name),使 claude_skill 前缀成为 .claude/skills/csv-total/。两种都可以,但目录名和 skill name 应对齐。在 tests/test_snapshot.py 用 demo 的 skill_dir 快照 + agentdescent.demo:offline_agent dest spec 钉住 reward(例如 COLUMN: amount → 1.0;错误种子 → 0.0),并断言 staging 前缀带的是净化后的 csv-total 而不是 artifact。现有的 artifact_id 逃逸测试(#1150 / #1153)必须继续失败于越界路径。
验收
- 上述 demo → export →
snapshot eval --spec ./demo/spec.json得到ok: true、failed: []、reward == 1.0(或与 demofinal_reward一致)。 - 含
/、..或不安全artifact_id的快照在 eval 物化时仍不能写出tmp之外。 tests/test_snapshot.py里现有的 text / kind 不匹配用例仍然通过。- 不必改
evolve本身,也不必依赖真实模型。
Activity
HugoXK commentedon Nov 19, 2026
Verified independently at
76223fda(main after #1162 / #1165), offline, no API key.demo→ run20261119-050528-6758c3,initial_reward 0.0 → final_reward 1.0,rules.md -> 'COLUMN: amount';snapshot export→csv-total@v2withSKILL.md+references/rules.md=COLUMN: amount\n;snapshot eval ./skill.json --spec ./demo/spec.json→n 12, reward 0.0, kind skill_dir, ok true, failed []. Matches the report exactly.Root cause confirmed line by line:
_write_targetstagestmp/artifact(agentdescent/artifact_snapshot.py:401);replace(spec, target=…)leavesname=None, soEvolveSpec.artifact_id()falls back to the basename →artifact(agentdescent/evolvespec.py:337);tree_runner(name=aid)then writes.claude/skills/artifact/(agentdescent/runners.py:67);demo:offline_agentreads the fixed.claude/skills/csv-total/references/rules.md(agentdescent/demo.py:94), misses it, and sums columnid→21against gold321.The
name=route is the smaller of the two, and it keeps the #1150 guarantee: inevaluate_snapshot_spec, addname=snapshot.artifact_idto the existingreplace(...). Staging still uses the fixedtmp/artifactpath; only the destination spec's layout prefix changes to.claude/skills/csv-total(the runner resolves the prefix from the sanitizedartifact_id()). Same snapshot, same dest spec, one probe:Test gap also confirmed:
tests/test_snapshot.py:146is the onlyevaluate_snapshot_speccase, and it iskind="text"+agent="echo"; noskill_dirrow exercises the staged workspace shape, which is why CI is green on this.One edge worth stating in the slice: the directory name then follows
snapshot.artifact_id, not thename:frontmatter insideSKILL.md. They agree here (csv-total), so the slice is correct for this path; a snapshot whose frontmatter name differs from itsartifact_idwould need a separate card.