Repository navigation
Release: bring the qwen3.5-9b Command eval to threshold #47
Description
Problem
The first valid post-roadmap live-model Command eval completed against local qwen/qwen3.5-9b, but it does not meet Lucid's release bars.
- 22/35 canonical Command cases passed (63%; target 90%)
- no pipeline errors or cancellations
- non-destructive and destructive-confirmation cases passed
- diagram editing, reference resolution, block references, ambiguity handling, and mode separation remain below threshold
- one unrelated block was modified
The report stays local because eval artifacts are not release source files.
Work
- classify the 13 failures as model, prompt/schema, compiler, or matcher defects
- fix product defects without weakening safety checks or accepting incorrect edits
- add or tighten regression coverage for every confirmed defect
- rerun the canonical Command eval and then the full live suite
Done when
- overall live eval meets 90%
- each category meets its documented bar
- no unrelated-block edits, pipeline errors, or cancellations
- normal CI and focused desktop E2E remain green
Metadata
Metadata
Assignees
Labels
No labels
Activity
triggeredcode commentedon Oct 20, 2026
Fix pushed as
e52e3d1in PR #48. Local CI passes 1,519 tests with 22 skipped. The exact regression passes, the canonical Command eval passes 35/35, and the strict full live suite passes 98/99 (99%) with every category above its bar, zero pipeline errors or cancellations, and zero unrelated-block edits. The only miss safely refused nonexistent block 9 without changing the document.