Skip to content

fix(panicbot): carry unsuppressed test-code findings, as the direct lane does - #677

Merged
hyperpolymath merged 1 commit into
mainfrom
fix/panicbot-carry-test-context-findings
Nov 14, 2026
Merged

hyperpolymath merged 1 commit into
mainfrom
fix/panicbot-carry-test-context-findings

Conversation

@hyperpolymath

@hyperpolymath hyperpolymath commented Nov 14, 2026 •

Copy link
Copy Markdown
Owner

What this is

Second finding from the phase 1 shadow (standards#1562, run 43602190815): the direct panic-attack row read critical in a changed test script, panicbot read clean.

Root cause (measured)

Same panic-attack binary (410a8f8, built locally), same tree. Both invocations (assail . --output and assail . --headless) give byte-identical output, so the invocation is not the cause. Of 75 weak points: 51 production, 6 test_only not suppressed, 18 test_only suppressed. The direct lane keeps 51 + 6 = 57; panicbot's translate_all dropped every test_only/doc finding, so it reported 51. The 6 include the critical eval in scripts/tests/dismiss-code-scanning-alerts-test.sh.

Change (owner ruling 2026-11-14: carry at full severity)

translate_all drops only what panic-attack itself suppressed. Unsuppressed test_only/doc findings are carried at their own severity, with test_context in the finding metadata. Tests updated: carries unsuppressed test_only and doc, still drops suppressed.

Verified

  • Against the real tree with the real binaries: panicbot assail now reports 57 findings (was 51), equal to the direct lane.
  • cargo test --locked in bots/panicbot all pass (107 unit + integration), cargo fmt --check and cargo clippy --locked --all-targets -- -D warnings clean.
  • tests/fleet-report-back-test.sh: 212 passed, 0 failed (also with STANDARDS_DIR set, build test 69 passed). With a real panic-attack on PATH two tests fail; that setup is not what the suite assumes.

Not verified

A live CI shadow run. After this and #676 merge, standards' FLEET_REF needs bumping; the next shadow run is the test.



Generated by Claude Code

Summary by CodeRabbit

  • Bug Fixes
    • Unsuppressed findings classified as test-only or documentation-related can now appear in results when they pass severity filtering. Suppressed findings remain excluded.
    • Finding details now include test context when available, providing additional information about where a weak point was identified.

…ane does

First shadow run (standards#1562): the direct panic-attack row read critical
in a changed test script; panicbot read clean. Same binary, same tree. Of 75
weak points, 51 are production, 6 test_only (not suppressed) and 18 test_only
(suppressed). The direct lane keeps the first 57; panicbot dropped the
test_only ones, so it reported 51.

Owner ruling 2026-11-14: carry them at full severity. translate_all now drops
only what panic-attack itself suppressed; test_context is recorded in the
finding metadata. Against the real tree panicbot now reports 57, equal to the
direct lane.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0136P7ZkEdznGjgTixZgjhB6
@coderabbitai

coderabbitai Bot commented Nov 14, 2026 •

Copy link
Copy Markdown
Contributor

Explain Change →

No actionable comments were generated in the recent review.

Walkthrough

The translator adds test_context to finding metadata. It retains unsuppressed test_only and doc findings while continuing to drop suppressed findings. Tests cover these cases.

Changes: Finding translation

Layer / File(s) Summary
Add test context to finding metadata
bots/panicbot/src/translator.rs
Finding metadata now includes the weak point’s test_context alongside its category, severity, and recommended attacks.
Retain unsuppressed test-only and documentation findings
bots/panicbot/src/translator.rs
The translator continues to drop suppressed findings but no longer filters unsuppressed test_only or doc findings by test context. Tests verify that unsuppressed findings are retained and suppressed test-only findings are dropped.

Estimated code review effort: 2 (Simple) | ~10 minutes

Change: Bug fix

Merge Risk: Minimal · up to 0369b

Panicbot now reports unsuppressed test-code and documentation findings at their own severity, with test context recorded in the metadata, so its output matches the direct lane. No actionable merge risk remains.

Pre-merge checks: Passed 4

Check name Status Explanation
Description Check Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check Passed The title names the panicbot fix: unsuppressed test-code findings are now carried, matching the direct lane. The raw summary confirms translate_all keeps unsuppressed test_only and doc findings and dr…
Linked Issues check Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check Passed Check skipped because no linked issues were found for this pull request.
Full details: Title check

Explanation

The title names the panicbot fix: unsuppressed test-code findings are now carried, matching the direct lane. The raw summary confirms translate_all keeps unsuppressed test_only and doc findings and drops only suppressed ones, and that test_context is recorded in metadata. The title states the main behavioural change and its reason. It is conventional and specific, so it passes.

Finishing Touches

Generate unit tests (beta)

  • Commit to this branch
  • Create a new PR
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

Recent review info

Run configuration

Commits

Reviewing files that changed from the base of the PR and between b97e4e1 and 0369b4d.

Files selected for processing: 1

  • bots/panicbot/src/translator.rs

Included review availability: This review used your included allowance. Your plan provides up to 1 included review per hour; 0 remain after this review.

@sonarqubecloud

Copy link
Copy Markdown

@hyperpolymath
hyperpolymath marked this pull request as ready for review November 14, 2026 14:25
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 14, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ❌ Failed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-14T14:27:45.808630Z 0369b4d Draft marked ready
🔒 Security Review ❌ Failed 2026-11-14T14:25:49.406791Z
We couldn’t complete this review. Comment @codex security review to try again.
0369b4d Draft marked ready
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector

Copy link
Copy Markdown

This security review couldn’t start because the paying account has exhausted its Codex allowance and has insufficient available credits. The payer can check their balance and add credits in the Codex usage dashboard, then try again. If you do not manage the paying account, contact this repository's admins.

@hyperpolymath
hyperpolymath merged commit 6a3a227 into main Nov 14, 2026
71 checks passed
@hyperpolymath
hyperpolymath deleted the fix/panicbot-carry-test-context-findings branch November 14, 2026 14:31
hyperpolymath added a commit to metadatastician/standards that referenced this pull request Nov 14, 2026
…adow fixes (#1567)

## What this is

Follow-up to #1562 (phase 1 shadow enrolment). One line: `FLEET_REF` in
`.github/workflows/fork-governance.yml` goes from `7fc27784` to
gitbot-fleet `14892a53`, which includes the two fixes the first shadow
run exposed:

- hyperpolymath/gitbot-fleet#676: ponsbot accepts the `source_dirty:
"true"` stamp that a `cargo install --git --rev` pons carries on the
exact pinned commit. Before it the ponsbot row read orange (`error: pons
was built from a dirty tree`).
- hyperpolymath/gitbot-fleet#677: panicbot carries unsuppressed
test-code findings at full severity, as the direct lane does. Before it
panicbot reported 51 findings and read clean where the direct row read
critical (57 findings).

No line moves, so the line-keyed code-scanning dismissal for
`fork-governance.yml` is unaffected (checked: `security-events: write`
stays at line 289).

## What to check
The Hypatia grid on this PR's own run is the first shadow comparison
with both fixes: panicbot should report the same count as the
`panic-attack assail` row, and ponsbot should read a real result (not
orange), both advisory.

---

---
_Generated by [Claude
Code](https://claude.ai/code/session_0136P7ZkEdznGjgTixZgjhB6)_

Co-authored-by: Claude <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants