Repository navigation
communication-structure: fix the judge's top loss causes, add reader-accuracy check - #1967
Merged
Merged
Conversation
…accuracy check The skill loses to humanizer at 0.225 (comms-r4-1017). Across 174 rival-win verdicts the judge reasons cluster (keyword count, overlapping) as: not send-ready (placeholders, sender notes, emoji, outline fragments) 129; no stated recommendation, rationale or trade-offs 59; omits or over-condenses source detail 30. Replace the 150-word default with a send-ready, complete rule, keep source trade-offs and reasons, drop the three-findings cut. Add reader_check.py with 5 scenarios of 3 regex-keyed questions: verbatim source facts with paraphrase alternations, consequence-phrase polarity, and a word cap per target so the source notes alone score 0. Replace the eval that asserted the word cap, route the checker from SKILL.md, and make the CLI exit 2 on bad input. Tests: 10/10 Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014QNFFPWVcPUbeuvwM5fk2Z
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Reworks communication-structure against the judge's own stated loss reasons, and adds an outcome check (reader accuracy) so the row is no longer scored only on preference. The row currently loses to humanizer at 0.225 (comms-r4-1017).
Across 174 rival-win verdicts, the judge's reasons cluster as (keyword count, overlapping):
Changes:
SKILL.md: the 150-word default becomes a send-ready, complete rule. Messages carry no placeholders, no notes to the sender and no emoji beyond ✅/⏳. They keep every trade-off, cost and reason the source states, and cutting never drops a source fact. The three-findings cut and the placeholder example are removed. The reference budgets stay as lint ceilings, not targets.scripts/reader_check.py+evals/reader-checks.json: 5 scenarios, each with 2-3 regex-keyed reader questions. The keys accept paraphrase, check polarity (a consequence such as "lose the rebate", not the bare word) and need no template labels. Bad input exits 2.Not yet measured: the effect on the judged row. The next paid comms run should report reader-check accuracy next to preference.
Tests: 10/10 (test_comms_reader_check.py), run on the commit's own tree; comms_lint and comms_guard pass. Independent review: 0🔴; the 9🟡 are fixed here, and the possibly-real product names were swapped for placeholders.
🤖 Generated with Claude Code
https://claude.ai/code/session_014QNFFPWVcPUbeuvwM5fk2Z