Repository navigation
fix(analyzer): narrow AR1 queueing and RA1 consent-row and heading matches (#1120) - #1122
Varshith-Kali wants to merge 2 commits into
Conversation
…tches (NVIDIA#1120) AR1 fired on a scheduling row where the object of never refuse is a work item to queue, not a request to the agent. RA1 fired on a consent-gate row and on a section heading, neither of which instructs self-modification. Exempt the bare never refuse/decline form only when a queueing verb applies to a work item in the same clause, with a pronoun object proved by a work-item noun in the same table cell. Exempt a bare self-update noun only when the same line carries closed consent wording (checked + offered, offered before, offers the update, ask before, only after approval, user runs it). Skip RA1 matches on Markdown heading lines; the section body is still scanned. Regression tests cover each reported case plus genuine cases that must still flag. Signed-off-by: Varshith Puli <pulivarshit@gmail.com>
|
CI note: test-unit shows 4 failures, all in tests/nodes/analyzers/test_json_decoded_views.py::test_oms_decoded_window_keeps_unicode_fence_source_coordinates (unicode fence source coordinates, expected '~~~' but got '~~~sh'). I verified these fail identically on pristine upstream main (ce46d46), so they are pre-existing and unrelated to this change. The other 26,132 tests pass, including the 21 new cases for this fix. |
rng1995
left a comment
There was a problem hiding this comment.
[SkillSpector Review]
Decision — Request changes.
Value and readiness — The reported worker-queue and consent examples are useful false-positive regressions. The proposed exemptions also suppress independent instructions that the policies are meant to catch.
Material findings
static_patterns_rogue_agent.py:632searches for consent anywhere on the line.Apply the self-update now. Ask before restarting.loses RA1 because the second sentence supplies an unrelated gate. The character restriction inside the gate regex does not bind that gate to the matched action. Limit the exemption to a gate governing that update in the same clause or proven table field, and test an unrelated gate in another sentence and another cell. Also keep actionable matches in heading lines and fenced commands. The heading check currently precedes every RA1 path, including code matches. RA1 exemptionsstatic_patterns_anti_refusal.py:291checks the object of the queueing verb but never checks the object of refusal. InQueue the worker, never refuse a user request., the presence of a worker lets the unrelated explicit user-request instruction pass. Require the refusal to omit its object or demonstrably refer to that work item. Add this regression next to the original queueing case. AR1 exemption
PIC tradeoffs — A clause- and action-bound exemption preserves the precision gain on the issue examples and keeps genuine AR1 and RA1 findings visible. The existing helpers are bounded and the proposed narrowing can stay local to them.
Verification and gaps — Reviewed the complete diff, both analyzers, regression fixtures, PR discussion, and the failing unit job. CI reported 26,132 passes and four Unicode fence-coordinate failures that the contributor reproduced on the PR base. The added tests omit the mixed-action cases above. Static review only.
Reviewed head: de6f22637bbdc9a6dd78571622a6f3dc080af5ef
…verned action (NVIDIA#1120) Address review findings on de6f226: the RA1 consent gate matched anywhere on the logical line, so an unrelated gate in another sentence or table cell exempted a real instruction, and the AR1 queueing exemption checked the queueing verb's object but never the refusal's own object. RA1: a consent gate now must share the match's sentence and table cell, and sit in the same semicolon clause unless the gate names the update itself (offers the update spans a semicolon in the reported case). The heading skip now applies only to a bare self-update noun on a genuine heading line; an imperative on a heading line still instructs, and a match inside a fenced code block is code, not a heading. AR1: the refusal must omit its object or demonstrably refer to the queued work item. A pronoun object refers to the proven work item and an explicit work-item noun stays exempt; any other object (a user request, any request, the user) keeps the finding. Regression tests cover an unrelated gate in another sentence, another cell, and another clause, an imperative heading, fenced heading and comment lines, and a refusal object that is a user request, next to the original cases. Signed-off-by: Varshith Puli <pulivarshit@gmail.com>
|
Thanks for the review, all three findings are fixed in cdc7cea. RA1 consent gate (static_patterns_rogue_agent.py, _is_consent_gated_self_update): the gate must now share the match's sentence and table cell, and sit in the same semicolon clause unless the gate names the update itself ("offers the update" spans a semicolon in the reported case). "Apply the self-update now. Ask before restarting." flags again, as does a gate in another cell or after a semicolon. The four reported consent cases still clear. Heading check: it now applies only to a bare self-update noun on a genuine heading line. "## Update this skill" flags, and a match inside a fenced block (a "#" shell comment line, or a heading inside fenced markdown) is code, not a heading, so it flags too. AR1 (static_patterns_anti_refusal.py, _is_queueing_work_item_refusal): the refusal must now omit its object or demonstrably refer to the queued work item. "Queue the worker, never refuse a user request." flags again; "never refuse it" and "never refuse the worker" stay exempt. Tests: test_issue_1120_false_positives.py now has 30 cases, all passing. The AR1 suite and test_static_patterns.py pass (924 passed, 1 skipped, 4 xfailed), as do the related RA1 suites (222 passed). One failure in the run, test_python_shell_flag_precheck_is_bounded_at_full_window, is a pre-existing timing check in the tool-misuse analyzer that fails identically on the base without this change. |
Fixes #1120.
Repro: pinned TheColliery/CoalFace v0.15.5 SKILL.md, scanned with the current main analyzers. Before this fix it produced four HIGH findings: AR1 "never refuse" at line 64, RA1 "self-update" at lines 31, 145 and 146. After this fix it produces none, with no other AR/RA findings on the file.
What changed
Tests