Repository navigation
P1: core engine + CLI - #2
Merged
Merged
Conversation
Extract claims with the FAST model, verify each with STRONG (AbortController timeouts, zod retry, FALLBACK failover), backstop verdicts with quote/citation/line guards + deterministic signals, grade into a GitHub-ready report, and drive it all from a streaming CLI with fixtures from three real PRs.
- Extraction only emits outcome claims; problem statements become one bug_fixed outcome claim. Claims carry scope (ci | local | unspecified). - CI checks are classified by kind; tests_pass is only contradicted by a failed check of the kind the claim talks about, and a VERIFIED one gets the passing checks of that kind attached as evidence. - Skipped/neutral/stale checks are ignored instead of reading as pending. - sanitizePrBody() strips bot sections, HTML comments and tags. - Low-confidence (< 0.75) CONTRADICTED is downgraded to UNVERIFIED. - npm run eval: live-model regression eval over fixtures/real-*.json. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
CodeGovindz
added a commit
that referenced
this pull request
Oct 26, 2026
* P1: core engine + CLI Extract claims with the FAST model, verify each with STRONG (AbortController timeouts, zod retry, FALLBACK failover), backstop verdicts with quote/citation/line guards + deterministic signals, grade into a GitHub-ready report, and drive it all from a streaming CLI with fixtures from three real PRs. * P1 precision fixes: no false CONTRADICTED verdicts - Extraction only emits outcome claims; problem statements become one bug_fixed outcome claim. Claims carry scope (ci | local | unspecified). - CI checks are classified by kind; tests_pass is only contradicted by a failed check of the kind the claim talks about, and a VERIFIED one gets the passing checks of that kind attached as evidence. - Skipped/neutral/stale checks are ignored instead of reading as pending. - sanitizePrBody() strips bot sections, HTML comments and tags. - Low-confidence (< 0.75) CONTRADICTED is downgraded to UNVERIFIED. - npm run eval: live-model regression eval over fixtures/real-*.json.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
P1: core engine + CLI
Receipts' whole engine now exists in
src/core/(zero Next.js imports, TypeScript strict, noany): feed it any public GitHub PR URL and it extracts the author's claims with the FAST model, verifies each one with STRONG (Kimi K3) against real diff hunks, deterministic signals, and CI check runs, guards every verdict, and prints a GitHub-comment-ready report.What works (validated on real PRs, zero network in tests)
extract.ts) viaqwen.qwen3-coder-next, max 12 claims, plus the quote guard: any claim whosequoteisn't a (whitespace-normalized, case-insensitive) substring of its declared source is dropped and counted indroppedClaims.verify.ts) viacallStrong()— STRONG with a 45 s AbortController timeout (the HTTP request really is cancelled; typedTimeoutError), on timeout/throttle-after-SDK-retries/5xx it fails over straight to FALLBACK (30 s); concurrency 4; per-claimmodelUsed+latencyMsrecorded.signals.ts): test files/cases, skipped/focused markers, CI state incl. pending (correction 5), removedExports vs SIGNATURE_CHANGE (correction 4), dependency major bumps, docs. Every signal has a stable key the guard validates.guard.ts): citation guard (evidence must exist in the bundle), line guard (evidence line must fall inside a changed hunk's new-file range, else it degrades to a file link at head SHA), evidence URLs (blob at headSHA with#L, check-run URLs), and a tests_pass CI-pending override to UNVERIFIED "CI still running".report.ts): grading truth table (FAIL on any CONTRADICTED, PASS if all VERIFIED, else MIXED; 0 claims → MIXED with trust "—"), trust score,modelsUsedwith friendly names and the "(2 via GPT-OSS 120B)" fallback note, markdown with the<!-- receipts:report -->marker,toJSON()that round-trips through the zodReportschema.pipeline.ts): typed events (fetching,fetched,extracting,extracted,verifying(i/n),fallback,done,error) and the correction-7 in-memory report cache keyedowner/repo#number@headSha+ model IDs (--freshbypasses).scripts/verify.ts):--json,--fresh,--dump-fixture <name>; progress to stderr, report to stdout, timing/token summary.src/core/__tests__/(hunk parsing, skip-lists/budgets, hunk selection, guards incl. a hostile-PR prompt-injection fixture, grading truth table, llm parser incl. reasoningContent + fenced/prose JSON, and callJson/callStrong retry + failover with a stubbed client).npm test,npm run lint,npm run buildall green.Real-PR validation (fixtures under
fixtures/real-*.json, records in docs/PLAN.md §5)All three beat the < 60 s target. The Cap PR is the money shot: the agent's "Validation: all 41 SDK tests pass" was CONTRADICTED by the failing
Format (Biome)/Vercel – cap-webchecks; dispenso's "1287/1287 pass" claims were held UNVERIFIED by the CI-pending rule because its head-SHA CI was mid-run.Known weaknesses (honest)
{"claims":[]}where a human would find one checkable claim. On the three recorded runs extraction was solid (12/12/3 verbatim-quoted claims); keeping qwen as FAST per plan and monitoring.public); exotic languages or indirect public-API removals won't be caught.commitsTruncatedeven when the list is complete (can't distinguish without an extra API call).npm run verify -- <url> --jsondrops unknown flags in npm's arg forwarding — usenpx tsx scripts/verify.ts <url> --json ...(documented in PLAN.md for the P4 README).Also in this PR: P0-review fixes (env.ts with typed
getConfig()+ productionGITHUB_TOKENcheck shared by smoke + CLI; llm.ts Promise.race → realAbortControllercancellation), the PLAN.md §5 note thatcacheComponents: trueforces a<Suspense>boundary around anything readingsearchParamsin P2, and fixture-based offline test data.Next: P2 (web UI + streaming API + Vercel) will import
src/core/*unchanged.Precision fixes (review round)
The first Cap#3412 run graded the PR FAIL on two false CONTRADICTED verdicts. Both are fixed:
Format (Biome)check and a Vercel deploy. Checks are now classified by kind, andtests_passcan only be contradicted by a failed test check.New
npm run evalruns the recorded PRs through the live models againstfixtures/expectations.json: 11/11 pass. Cap#3412 is now MIXED · 4/8 verified, docs#46488 PASS, dispenso#129 MIXED, with no CONTRADICTED anywhere.Known weakness: the extraction model sometimes still emits fragment claims ("five fail before the fix.") and the quote guard drops 1–2 paraphrased claims per long PR.