Skip to content

fix(snapshot): expose a skill_dir eval under its artifact_id - #1168

Merged
Birfy merged 1 commit into
mainfrom
fix/1164-snapshot-eval-skill-name
Nov 19, 2026
Merged

Birfy merged 1 commit into
mainfrom
fix/1164-snapshot-eval-skill-name

Conversation

@Birfy

@Birfy Birfy commented Nov 19, 2026

Copy link
Copy Markdown
Collaborator

Closes #1164

What changed

snapshot eval still stages a skill_dir candidate under the fixed #1150 path tmp/artifact, so an unsanitized artifact_id cannot steer the write. The destination spec's name was left unset, so EvolveSpec.artifact_id() fell back to that basename (artifact) and claude_skill wrote .claude/skills/artifact/.

The demo offline agent reads .claude/skills/csv-total/references/rules.md. After a successful demo → snapshot export, snapshot eval --spec therefore scored 0 even when the snapshot already contained COLUMN: amount.

This PR sets name=snapshot.artifact_id on the dest spec used for re-eval, so the layout prefix becomes .claude/skills/csv-total/. Staging is unchanged.

Verification at the tip

Tip SHA: 07380bd

Red first, this tip's test on origin/main (72eabc6) without the name= line: 1 failed — assert recorded["name"] == DEMO_SKILL (artifact vs csv-total).

Ran:

  • pytest -q tests/test_snapshot.py tests/test_snapshot_artifact_id.py at this tip → 37 passed
  • python -m tools.changelog_fragments check → fragments OK

Not run:

  • full pytest -q — the tests workflow runs it on this PR
  • a live demo → snapshot export → snapshot eval --spec cycle — the new test uses the demo skill, offline_agent, and the same amount-vs-id CSV shape

`snapshot eval` kept the dest spec's name unset, so a skill_dir
candidate staged at tmp/artifact was visible as
`.claude/skills/artifact/` instead of `.claude/skills/<artifact_id>/`.
The demo offline agent then missed the learned rules and scored 0.
Staging stays on the fixed #1150 path; only the dest spec's name is
set from the snapshot id.

Closes #1164
@Birfy

Birfy commented Nov 19, 2026

Copy link
Copy Markdown
Collaborator Author

Self-review (Birfy)

Tip SHA: 07380bd
Ran:

@Birfy
Birfy enabled auto-merge (squash) November 19, 2026 10:26
@Birfy
Birfy merged commit f8efc73 into main Nov 19, 2026
10 checks passed
@Birfy
Birfy deleted the fix/1164-snapshot-eval-skill-name branch November 19, 2026 10:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

snapshot eval: a skill_dir candidate is staged under "artifact", so the dest agent never reads it and a successful demo re-eval scores 0

1 participant