runTests scrapes ast-grep test's human output, and fixture text can corrupt the counts

[thecodedrift]

runTests in packages/cli/src/rules/verify.ts recovers the pass/fail counts of ast-grep test by regex-scanning its human-readable output:

const output = stdoutChunks.join("") + stderrChunks.join("");

// sg test outputs: "test result: ok. 3 passed; 0 failed;"
const passedMatch = /(\d+)\s+passed/i.exec(output);
const failedMatch = /(\d+)\s+failed/i.exec(output);

The regexes are unanchored and run against stdout and stderr concatenated, taking the first match anywhere in the stream. ast-grep test echoes the source of a failing test case, so fixture content can be read as the summary.

Reproduction

Measured against the vendored ast-grep 0.41.0. Rule and fixture:

# rules/no-eval.yml
id: no-eval
language: javascript
severity: warning
message: Avoid eval
rule:
  pattern: eval($$$A)

# rule-tests/no-eval-test.yml
id: no-eval
valid: []
invalid:
  - "const msg = '7 passed; 0 failed';"

ast-grep test -c sgconfig.yml --skip-snapshot-tests --filter '^no-eval$' prints:

  const msg = '7 passed; 0 failed';

FAIL no-eval  M

Error: test failed. 0 passed; 1 failed;

exit code 4. Applying the two regexes above to that output yields:

CLI would report: passed = 7   failed = 0
truth from the summary line:   0 passed; 1 failed

The echoed fixture matched before the summary line did.

Severity — what this does and does not do

It corrupts the reported counts. It does not cause a false pass. Validity is decided by the exit code, not the counts, and a passing run never echoes fixture source — so the only match on a clean run is the genuine summary. A rule cannot be made to verify green this way.

What it does produce is a failure reported with nonsense numbers: Tests: ✗ failed (7 passed, 0 failed). That output is not only read by humans — the improve-rule loop feeds it back to an agent iterating on a rule, so wrong counts can steer the next edit.

Two related defects in the same function

• UTF-8 chunk boundaries. Both handlers do chunk.toString() per chunk and join afterwards, so a multi-byte character split across a chunk boundary is corrupted. This is the same bug fixed for Vale in vale/run.ts (replaced with a StringDecoder per stream, flushed on close); the ast-grep path never got the same treatment.
• Format drift is silent-ish. @ast-grep/cli is pinned exact at 0.41.0. If a future version reworded the summary, both regexes would miss, counts would read 0/0, and the code would fall through to the exit-code snippet — degraded rather than wrong, but nothing would announce that the parser had stopped working.

Why there is no trivial fix

ast-grep test has no structured output option at 0.41.0 — --help offers --test-dir, --snapshot-dir, --skip-snapshot-tests, --update-all, --interactive, --filter, --config, --include-off, and nothing for JSON or machine-readable reporting. So "use --json" is not available, and this cannot simply be deleted the way dist/prompts.js scanning was in the styleguide's worked example.

That makes it a narrower problem: keep parsing, but parse the one line that is a summary rather than the whole stream.

Suggested direction

• Anchor to the summary line specifically — test result: ok. N passed; M failed; on success and Error: test failed. N passed; M failed; on failure — rather than scanning all output. Match on the last such line, not the first token that looks numeric.
• Strip ANSI escapes before matching; the output is colorized (test result: \e[32mok\e[0m.).
• Use a StringDecoder per stream, as vale/run.ts does.
• Pin the summary format in the ast-grep vendor-contract test proposed in Add an ast-grep vendor-contract test #108, so a version bump fails loudly instead of quietly returning 0/0.

Related: #108 (ast-grep vendor-contract test). See also the "Verify Build Output In The Build, Not By Parsing It" section of .conventions/STYLEGUIDE-CODE.md — the same failure shape (prose read as structure) as the which engine case documented there.