fix(review): preserve counterexamples in Noema reviews - #2035
Conversation
|
Warning Review limit reachedNext included review available in 59 minutes. View limit detailsLimit details: You’ve used the included review currently available. You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository. Review configuration: ⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Advanced Run ID: 📒 Files selected for processing (7)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
seonghobae
left a comment
seonghobae
left a comment
There was a problem hiding this comment.
Exact-head source review of 6324f4dce523039c992956d478750d74f55e52e3: the production change preserves the already-validated attack_or_counterexample field only when present, keeps historical inputs compatible, and does not alter verdict, reviewed-commit, credential, routing, or publication-event authority. The APPROVE / REQUEST_CHANGES / COMMENT serialization regression exercises the actual review payload boundary. The additional small-context and mixed-DNS cases are tests of existing fail-closed behavior and do not broaden production scope. I found no new source-backed substantive defect in this delta. This COMMENT is not independent approval or merge authority; CodeQL settlement for this exact head remains nonterminal, and protected integration still requires current-head checks and authorized review.
seonghobae
added
bug
ChatGPT Codex Connector
![opencode-agent[bot]](https://avatars.githubusercontent.com/in/1549082?s=60&v=4){kind=small}
There was a problem hiding this comment.
Pull request overview
OpenCode could not approve from deterministic current-head evidence because GitHub Checks have failed.
Findings
1. HIGH Current-head GitHub Checks - Fix failed required checks before approval
- Problem: Failed same-head checks remain for
6324f4dce523039c992956d478750d74f55e52e3. - Root cause: The model-unavailable evidence fallback is allowed only when peer GitHub Checks are complete and clean.
- Fix: Read and fix the failed check logs below, then rerun the current-head checks.
- Regression test: Keep the model-unavailable fallback gated on an empty failed-check rollup.
Failed checks:
- CodeQL PR/CodeQL compatibility analysis (python): FAILURE (https://github.com/ContextualWisdomLab/.github/actions/runs/34183705274/job/101954672753)
- CodeQL compatibility analysis (python) check run: failure (https://github.com/ContextualWisdomLab/.github/actions/runs/34183705274/job/101954672753)
- Required Noema Review/noema-review: FAILURE (https://github.com/ContextualWisdomLab/.github/actions/runs/34183702988/job/101940582259)
- noema-review check run: failure (https://github.com/ContextualWisdomLab/.github/actions/runs/34183702988/job/101940582259)
Changed-File Evidence Map
flowchart LR
PR["PR changed files"] --> Evidence["OpenCode bounded evidence"]
Evidence --> S1["Repository file: AGENTS.md"]
S1 --> I1["repository behavior"]
I1 --> R1["Review risk: Repository file: AGENTS.md"]
R1 --> V1["required checks"]
Evidence --> S2["Repository file: CHANGELOG.md"]
S2 --> I2["repository behavior"]
I2 --> R2["Review risk: Repository file: CHANGELOG.md"]
R2 --> V2["required checks"]
Evidence --> S3["Repository file: CLAUDE.md"]
S3 --> I3["repository behavior"]
I3 --> R3["Review risk: Repository file: CLAUDE.md"]
R3 --> V3["required checks"]
Evidence --> S4["Docs: noema-counterexample-publication.md (2 files)"]
S4 --> I4["operator or user guidance"]
I4 --> R4["Review risk: Docs: noema-counterexample-publication.md (2 files)"]
R4 --> V4["docs review"]
Evidence --> S5["CI script: noema_review_gate.py"]
S5 --> I5["review and security gate shell path"]
I5 --> R5["Review risk: CI script: noema_review_gate.py"]
R5 --> V5["bash -n plus Strix self-test"]
Evidence --> S6["Test: test_noema_review_gate.py"]
S6 --> I6["regression suite"]
I6 --> R6["Review risk: Test: test_noema_review_gate.py"]
R6 --> V6["targeted test run"]
OpenCode Review Overview
|
|
Current-head RCA for |
|
Current exact-head review admission update for |
|
Lifecycle correction at unchanged exact head |
1 participant
Noema validates an attack/counterexample for each adversarial probe but discarded it when constructing the published review. Preserve that scenario under its probe so readers can follow the hypothesis, evidence, and outcome.
The renderer uses the existing field and publication path; verdicts, current-head binding, validation, credentials, and routing are unchanged. Historical renderer inputs without the field remain compatible.
Validation: the publication regression fails on all three review events before the fix and passes 3/3 after it. The expanded Noema suite passes 250 tests with 100% statement and branch coverage of the publication module; docstring coverage is 100%. Two additional boundary tests cover missing context, untruncated diffs, and mixed public/private DNS answers. An independent agent review found no production blocker; its contradictory fixture example was corrected. This is unit-level publication evidence; actual bot output and protected merge remain unverified.
Quality reference: BerriAI/litellm#20421 (comment). Reproduction, limits, and follow-up:
docs/doctoring/noema-counterexample-publication.md.