chatgpt-review: pr mode returns a stale-head review after a push (restates prior findings, reports the old SHA) in long conversations

[BorisTyshkevich]

Summary

In pr mode, chatgpt-review.mjs sometimes returns a review of a stale PR head: the response restates findings from an earlier pass verbatim and self-reports reported_reviewed_sha as an older commit, even though (a) the fixes for those findings were pushed before the call, and (b) gh pr view <n> --json headRefOid confirmed the live head was the newer commit at call time. The independent fact-check stage downstream caught every instance (all stale findings were rejected as "already fixed at HEAD"), so no bad fix was applied — but each occurrence burns one of the session's three hard-capped pr-mode passes on a review of code that no longer exists.

Observed 3 times across 2 PRs in one /ship run (Altinity/altinity-oauth-helper), each time in a long-running ChatGPT conversation with many prior turns (plan authoring + prior review passes in the same thread), and each time immediately after a fix had been pushed between passes.

Reproduction (as observed)

1. Run a plan-author loop, then start pr mode with --seed-from-session <plan-session> on the PR. Pass 1 reviews the correct head and finds real issues.
2. Fix them, push, confirm gh pr view --json headRefOid shows the new head.
3. Re-invoke the same PR with --session <pr-session> (pass 2), or --seed-from-session <exhausted-session> to start a new pr session (pass 1 of session 2).
4. Response: the same findings as the prior pass, byte-for-byte in substance; reported_reviewed_sha = the pre-fix SHA. The published PR comment for that pass likewise cites the old SHA.

Instances (all in https://github.com/Altinity/altinity-oauth-helper):

PR	Pass	Live head at call time	reported_reviewed_sha returned	Outcome
#21	session 1 pass 2	e1a434f	c0c495f (pre-fix)	2 stale findings, both rejected by fact-check
#21	session 2 pass 1 (seeded)	eb3d02f	e1a434f (pre-fix)	2 stale findings, both rejected
#22	session 1 pass 2	e577d06	8f3e303 (pre-fix)	4 stale findings, all rejected

Conversations: https://chatgpt.com/c/6a9098f3-a038-83eb-b5e2-852a4f1cd2bc (PR #21; sessions c16ce96d-caee-48ae-894b-7a77c59b0670, dd05fae0-8a2c-4c26-bce1-9f481fd10b46) and https://chatgpt.com/c/6a910d15-c1a0-83eb-aa16-d926786d07ad (PR #22; session 55cd505c-ece3-4547-9942-c614259682dd).

What made it go away

Appending an explicit instruction to the question file worked every time it was tried:

Your previous response reviewed a stale snapshot (self-reported reviewed_sha <old>; the live head is <new>, confirmed via gh pr view). Navigate to https://github.com/<owner>/<repo>/pull/<n>/files fresh and, BEFORE writing any finding, state the exact head SHA GitHub shows you and confirm it is <new>; if it differs, say so rather than proceeding.

The next pass then reported the correct SHA and produced genuinely new findings (PR #21 passes 5–6; PR #22 second-session pass 1).

Likely cause

scripts/lib/prompt.mjs:27 — the fix-review pass prompt reads:

This is fix-review pass N. Reuse your earlier analysis, fetch the new PR head, compare it with the previously reviewed SHA <previousSha>, reassess every earlier finding, and inspect the complete updated PR for regressions. Report both old and new exact SHAs.

It leads with "reuse your earlier analysis", and it tells the model the old SHA but never the expected new one — so there is nothing the model must verify against, and in a long conversation it can satisfy the prompt from memory ("new head" = whatever it last saw) and report the old SHA as "reviewed". The --seed-from-session first pass uses the fresh "Browse the canonical PR, clone or fetch the repository…" prompt (prompt.mjs:26) and still went stale once (PR #21 session 2 pass 1), so conversation memory dominates even the fresh-review wording when the thread is long. The fact that a plain-language "state the head SHA you see and confirm it equals <new>" instruction fixed it every time supports this.

Suggested fixes

1. Make the pass prompt state the expected head SHA and require confirmation. The CLI already knows (or can cheaply fetch via gh pr view --json headRefOid) the live head. Put it in every pr-mode prompt: "The PR head is <sha>. Re-open the Files view and confirm you are reviewing <sha> before writing findings; if GitHub shows a different SHA, stop and report it." This is exactly the manual workaround, automated.
2. Fail closed on SHA mismatch in the CLI. After the response, compare reported_reviewed_sha to the live headRefOid; if they differ, return a distinct status (e.g. stale_review) instead of completed, so callers (the /ship code-review-pass workflow) don't spend fact-check agents and a pass slot on it. Ideally the CLI retries once with the re-navigate instruction before giving up.
3. Don't count a stale review against the 3-pass cap (or at least surface it), since the cap exists to bound real review rounds. Note the interaction with the cap is what made this costly: on PR Editor enhancements without CodeMirror — design reference and tracking #22 a stale pass 2 plus pass 1's internal retries exhausted the session before pass 3 could run at all (session-cap-exhausted).
4. (Nice to have) In --seed-from-session starts, explicitly tell the model the conversation is being continued for a new review of the current head and that earlier findings may already be fixed.

Environment

• Skill at skills/chatgpt-review (this repo), consumed via ~/.claude/skills/chatgpt-review; Agent Chrome over CDP; Node 20+.
• Observed 2026-08-27/28 during /ship 19 in Altinity/altinity-oauth-helper; the downstream fact-check that caught it is altinity-oauth-helper/.claude/skills/ship/references/code-review-pass.workflow.mjs.

[BorisTyshkevich]

Second variant observed 2026-08-28 (altinity-oauth-helper PR #25, pr mode, --session pass 2 after a push):

• ChatGPT did review the live head (c1f40c5) and the CLI did publish a fresh comment (#issuecomment-5454087136, "Reviewed head: c1f40c5…", two new MAJOR findings, VERDICT: REVISE).
• But the JSON the caller received was pass 1's result replayed: reviewedSha = the previous head (972504f), commentUrl = the pass-1 comment (#issuecomment-5453870447), findings = the two pass-1 findings (already fixed at the reviewed head), attemptsUsed: 1.
• Downstream, the verification loop "rejected" the stale findings as already-fixed and the two genuinely new findings were never surfaced — silent loss of a REVISE review's content, which is worse than the stale-head-review variant above because the published comment looks correct.

Suspect the resume/"already-finished uncollected response" path returning the prior pass's collected result, or the result cache keyed by session rather than by (session, pass). Repro shape: pr --session <h> on the same PR after pushing a new head, in a long seeded conversation.

[BorisTyshkevich]

Third occurrence of variant 2 (PR #25, 2026-08-28 ~17:10Z) — this time in a brand-new conversation (--session absent, no --seed-from-session), so it is not a long-thread effect. ChatGPT reviewed the live head (a72a021) and the CLI published a correct fresh comment (#issuecomment-5455474278, one new MAJOR, VERDICT: REVISE), but the JSON returned to the caller carried the previous pass's reviewedSha (01e8164), the previous pass's commentUrl (#issuecomment-5454883248) and the previous pass's two findings. Likely keyed on the PR rather than the (session, pass): a per-PR result cache or the output-file path chatgpt-review-pr-<n>-pass<k>.json being reused/reread. Note the workflow that consumed it also passed --output-file pointing at a path that already existed from the earlier rate-limited attempt.