Skip to content

fix(tp4): anchor findings to the assessed file, align severity, and drop absence-only mismatches - #1069

Open
Varshith-Kali wants to merge 9 commits into
NVIDIA:mainfrom
Varshith-Kali:fix/tp4-workflow-fences-1066
Open

Varshith-Kali wants to merge 9 commits into
NVIDIA:mainfrom
Varshith-Kali:fix/tp4-workflow-fences-1066

Conversation

@Varshith-Kali

Copy link
Copy Markdown
Contributor

Fixes #1066.

Three parts, all in the TP4 path:

  1. Anchoring. _tp4_finding hardcoded file="SKILL.md" with no line, so a finding on references/briefing-authoring.md:27-28 reported at SKILL.md:1. Findings now carry the assessed batch file and lines, which the evidence already had. The meta-analyzer formatter needed no change, it already renders file and start_line.

  2. Severity. TP4 escalated to HIGH at confidence 0.7, while the README pattern table lists TP4 as MEDIUM. TP4 is now always MEDIUM, matching the documented pattern.

  3. Partial-snippet criterion. A fence implementing one declared workflow step was flagged for omitting the rest of the workflow, with mismatched_capabilities phrased as absences ("No evidence of querying videos"). The TP4 prompt now states a chunk may implement only one declared workflow step and each entry must name what the code does. Deterministically, an assessment is suppressed when every mismatched_capabilities entry is phrased as an absence, since that carries no affirmative evidence of a mismatch. Mixed or affirmative assessments still emit.

Tests: 13 new regression tests in tests/test_mcp_tool_poisoning.py (12 failed pre-fix, 13 pass post-fix), including the issue verbatim anchoring case and the video-intel-search absence items. tests/test_mcp_tool_poisoning.py 174 passed, tests/unit/test_tp4_openai_http.py 51 passed, full suite 20964 passed with 8 failures reproduced on clean main (pre-existing Anthropic provider issues, unrelated). Ruff check and format clean. The prompt addition was kept compact (206 to 238 tokens) so the batch budget tests still pass.

Limits, stated plainly: the deterministic guard covers only the pure-absence class. The issue EPUB and PDF findings named affirmative behaviors, and the real pytest finding was mixed, so those still emit and rely on the prompt criterion plus meta-review, which are nondeterministic. No live-LLM verification was run; the LLM-judged part was verified with stubbed inputs only.

…rop absence-only mismatches

TP4 reported every finding at SKILL.md:1 even when the assessed fence
lived in another file, escalated severity to HIGH at confidence 0.7 while
the README pattern table lists TP4 as MEDIUM, and flagged partial
workflow snippets for omitting the rest of the declared workflow.

Anchor each TP4 finding to the batch file and lines, which the evidence
already carried. TP4 severity is now always MEDIUM, matching the README.
Add a partial-snippet criterion to the TP4 prompt, and suppress an
assessment deterministically when every mismatched_capabilities entry is
phrased as an absence, since that carries no affirmative evidence of a
mismatch. Mixed and affirmative assessments still emit.

Fixes NVIDIA#1066

Signed-off-by: Varshith Puli <pulivarshit@gmail.com>

@rng1995 rng1995 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[SkillSpector Review]

Decision: Changes Requested (reviewed head e739a9a59f8f022f5a3782b85f6c818b0073d5be).

Value and readiness: Anchoring TP4 to the assessed file and lines fixes actionable evidence, and MEDIUM matches the documented severity. The partial-workflow prompt change is sensible. The deterministic absence guard, however, also discards affirmative evidence of undeclared behavior.

Material finding — blocker, P2: In src/skillspector/nodes/analyzers/mcp_tool_poisoning.py:1172-1185, words such as without, missing, and lacks anywhere in a capability make the entire item count as an absence. _tp4_finding then returns None at lines 1201-1207 if all items satisfy that test. For example, an assessment with is_mismatch=True, confidence 0.9, and one item Uploads source files to an undeclared endpoint without user consent is suppressed because it contains without. That item describes an affirmative undeclared capability; neither an affirmative actual-behavior summary nor explanation can rescue it. The old code emitted the finding.

Please make suppression conservative: only discard an assessment when every item actually states absence of a declared workflow capability, retaining any item that also describes affirmative behavior. Add regressions for an affirmative capability containing without and for an item containing both an affirmative action and a negated clause. The current mixed test puts the affirmative evidence in a separate list item, so it does not cover this failure.

PIC tradeoffs: None requiring a separate decision for this fix.

Verification and gaps: Reviewed the complete three-file diff and pinned source, tracing the new guard through finding construction. The regression assertions cover anchoring, severity, pure absence, separate mixed entries, affirmative entries, and HTTP graph anchoring; they omit affirmative and absence wording within the same entry. All six hosted CI checks passed for this exact head (run). No contributor code, tests, imports, builds, setup, hooks, regex probes, or live model were executed. The finding above follows directly from the literal without matcher and unconditional suppression branch; it is not a claimed runtime reproduction.

…DIA#1069)

An absence marker anywhere in a mismatched capability (without, missing,
lacks) suppressed the whole assessment, even when the item described an
affirmative undeclared capability such as uploading source files to an
undeclared endpoint without user consent. Make suppression conservative:
an item counts as an absence only when no clause of it describes
affirmative behavior, with capability verbs under a marker treated as
governed by it.

Add regressions for an affirmative capability containing without and for
an item holding both an affirmative action and a negated clause.

Signed-off-by: Varshith Puli <pulivarshit@gmail.com>
@Varshith-Kali

Copy link
Copy Markdown
Contributor Author

Thanks for the catch. Fixed: _tp4_mismatched_item_is_absence is now conservative. An item counts as an absence only when no clause of it describes affirmative behavior. A capability verb that precedes the first absence marker in its clause, or sits in a clause with no marker, is affirmative evidence, so Uploads source files to an undeclared endpoint without user consent is now emitted, as are items holding both an affirmative action and a negated clause (Uploads files but does not encrypt them, No support for TLS, but sends credentials in plaintext). Pure absence phrasings (No evidence of querying videos, The code does not implement search, Lacks any corpus query, Missing authentication) are still suppressed. Clauses split only on contrast boundaries, so enumerations joined by "and" or commas stay under one leading marker.

Added regressions: _tp4_finding level tests for the exact without example (is_mismatch, confidence 0.9) and for a mixed affirmative plus negated clause item, plus four new cases in the absence-phrasing detection table.

Evidence: tests/test_mcp_tool_poisoning.py 180 passed (6 deselected); analyzer suite 12,133 passed, 4 skipped, 4 xfailed. New head dd3b48a.

Varshith-Kali added a commit to Varshith-Kali/SkillSpector that referenced this pull request Nov 9, 2026
Signed-off-by: Varshith Puli <pulivarshit@gmail.com>
@Varshith-Kali
Varshith-Kali force-pushed the fix/tp4-workflow-fences-1066 branch from 017b3a9 to 4f87c4c Compare November 9, 2026 22:11

@yashrajp22 yashrajp22 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One source-confirmed behavior bug remains in the absence-only guard, and the detailed TP4 severity guide needs to match the new MEDIUM rule.

Reviewed authored base e48d771a26f7398b36256ceb8d8f94adce3d5c44 through head 4f87c4c4caf406664d3c0e2e930c3227292ba444. The independent static assessment and exact-comparison Greptile review are complete, and both Greptile suggestions have been checked against source.

No builds, installs, tests, source/wheel scans, or live-provider checks were performed. The prompt changes and the reporter’s affirmative/mixed examples remain unverified; this is not a complete behavioral-verification result.

Comment on lines +1232 to +1235
for verb in _TP4_AFFIRMATIVE_VERB_PATTERN.finditer(clause):
if not marker_positions or verb.start() < min(marker_positions):
return False
return True

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we keep affirmative behavior when an absence phrase comes first? For a valid is_mismatch=True, confidence 0.9 assessment containing only Without user consent, uploads source files to a remote endpoint, the marker is at position zero and every verb follows it, so this returns True. No support for TLS. Sends credentials in plaintext is also treated as one negated clause. The finite verb list additionally drops Persists credentials without user consent. These describe actual behavior, but _tp4_finding discards them and the caller records a completed batch with no TP4 finding. Please suppress only unambiguously absence-only entries, retain affirmative or uncertain phrasing, and cover these cases with regressions.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in cc0165e. All three shapes are now retained: a leading absence phrase closed by a comma ("Without user consent, uploads source files ...") only modifies the main clause, so the marker no longer governs the verbs after the comma; clauses now split on sentence boundaries too, so "No support for TLS. Sends credentials in plaintext" keeps its second sentence affirmative; and persist joins the affirmative verb list, so "Persists credentials without user consent" is retained. The guard stays conservative: pure absence entries are still suppressed, including "No evidence of querying, storing videos" (the verb sits inside the leading phrase, before its comma) and "Without user consent, no encryption is performed". Added regression tests for all three examples plus the pure-absence controls: module 187 passed, analyzer suite 12,140 passed.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Follow-up in 56ad39c: the remaining gap in this thread was the subject-first clause after a comma (No support for TLS, the code sends credentials in plaintext), which the list fix did not cover because the remainder opens with a subject, not a verb. The strip now recognizes that shape as a separate main clause. The three shapes fixed earlier (leading without, second sentence, persists) still pass, verified in the same repro run. Repro: 18/18 pass.

Comment on lines +1271 to +1273
# TP4 is documented as MEDIUM in the README pattern table; the
# confidence-based escalation to HIGH was never documented (issue #1066).
severity="MEDIUM",

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please update the detailed TP4 guide alongside this severity change. docs/B.3.2-mcp-tool-poisoning.md:212 currently says MEDIUM or HIGH (based on LLM confidence), so escalation was documented there and users following that guide will still expect HIGH findings. The new fixed MEDIUM behavior is consistent with the README; the detailed guide and this comment should describe the same rule.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in cc0165e. docs/B.3.2-mcp-tool-poisoning.md now states the TP4 severity as fixed MEDIUM, matching the README pattern table. I also updated the code comment in _tp4_finding, which previously said the escalation was never documented; it now states the fixed MEDIUM rule and points at the guide. Line 212 was the only place in the guide stating the old MEDIUM or HIGH rule.

…VIDIA#1069)

The absence guard still suppressed affirmative capabilities in three
shapes: a leading absence phrase closed by a comma (Without user consent,
uploads source files ...), an affirmative second sentence (No support for
TLS. Sends credentials in plaintext), and verbs missing from the finite
list (Persists credentials without user consent).

Split clauses on sentence boundaries as well as contrast boundaries,
drop a leading absence phrase only when its closing comma precedes the
first capability verb, and add persist to the affirmative verb list.
Pure absence entries stay suppressed. The detailed TP4 guide now states
the fixed MEDIUM severity, matching the README and the code comment.

Signed-off-by: Varshith Puli <pulivarshit@gmail.com>

@yashrajp22 yashrajp22 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The three previously cited phrases and the severity documentation are corrected in source. The new comma handling introduces an absence-only list regression, detailed inline. The broader affirmative-preservation request in the existing suppression thread also remains open; I have not duplicated that finding.

This review covers authored base e48d771a26f7398b36256ceb8d8f94adce3d5c44 through head cc0165e8421b34fb16a8178c39d4a58aa895e11d, using the independent corrective source assessment and completed exact-comparison Greptile review, with suggestions checked against source.

No builds, installs, tests, source/wheel scans, or live-provider checks were performed. Author-reported test results and broader workflow accuracy remain unverified; this does not complete behavioral verification.

Comment on lines +1226 to +1229
first_verb = _TP4_AFFIRMATIVE_VERB_PATTERN.search(clause)
if first_verb is not None and first_verb.start() < comma:
return clause
return clause[comma + 1 :].strip()

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we preserve the leading negation for comma-separated lists of missing capabilities? With is_mismatch=True, confidence 0.9, and only No evidence of network activity, file writes, or remote uploads, the first recognized verb is writes, after the comma. This helper therefore strips the negative prefix and leaves file writes, or remote uploads. The caller sees a verb with no absence marker, so _tp4_finding emits a MEDIUM finding even though the entire original entry describes absent behavior. The previous head suppressed this input. Please keep negation across these lists and add a regression alongside the affirmative leading-without case.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 8ef3c59. The regression was in _tp4_strip_leading_absence_modifier: it dropped a leading absence phrase whenever the first capability verb fell after the comma, so "No evidence of network activity, file writes, or remote uploads" was stripped to "file writes, or remote uploads" and read as affirmative. The strip now only applies when the text after the comma reads as a main clause: a "without" modifier, or a remainder that opens with a capability verb (so "No support for TLS, sends credentials in plaintext" stays retained). Noun-first list continuations keep their leading negation.

Test evidence: the repro script confirmed the failure before the fix (a MEDIUM finding was emitted for the reviewer example) and 7/7 cases pass after, including the affirmative leading-"without" case and the three shapes from the earlier suppression thread, which I re-verified and did not regress. Module tests: 190 passed, 6 deselected. Analyzer suite plus module: 12,143 passed, 4 skipped, 11 deselected, 4 xfailed, 0 failed. Ruff check and format clean on both changed files.

New head: 8ef3c59

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Follow-up in 56ad39c: the remaining gap in this thread was noun-first absence phrasing (Search functionality is missing, Network access is not implemented), which the list fix could not address because there is no comma. Those statements are now recognized as absences while affirmative controls still emit TP4. The list case itself still passes. Repro: 18/18 pass.

…ilities (NVIDIA#1069)

The leading-absence strip dropped 'No evidence of ...' whenever the first
capability verb fell after the comma, so 'No evidence of network activity,
file writes, or remote uploads' was read as an affirmative finding. The
strip now only applies when the text after the comma reads as a main clause:
a 'without' modifier, or a remainder that opens with a capability verb.
Noun-first list continuations keep their leading negation, while the
affirmative leading-'without' case and the sentence-split cases stay
retained. Adds a _tp4_finding regression for the reviewer example plus
parametrize rows for the list and the verb-first remainder.

Signed-off-by: Varshith Puli <pulivarshit@gmail.com>

@yashrajp22 yashrajp22 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The new prefix check regresses affirmative clauses with explicit subjects. The absence guard also retains simple noun-first omission statements; both are detailed inline. The earlier affirmative-suppression and negated-list requests remain incompletely addressed, while their specific patched examples are corrected in source.

Reviewed authored base e48d771a26f7398b36256ceb8d8f94adce3d5c44 through head 8ef3c595a78edec3e30c87ebb27aaf47281b3023 using the independent corrective source assessment and completed exact-comparison Greptile review. Both vendor suggestions were adjudicated against source; one duplicates an existing finding.

No builds, installs, tests, source/wheel scans, or live-provider checks were performed. Author-reported tests and broader workflow accuracy remain unverified; behavioral verification and lifecycle actions are still pending.

Comment on lines +1237 to +1240
is_without_modifier = clause.casefold().startswith("without ")
remainder_opens_with_verb = _TP4_AFFIRMATIVE_VERB_PATTERN.match(remainder) is not None
if not (is_without_modifier or remainder_opens_with_verb):
return clause

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we retain affirmative clauses that start with a subject? For a valid is_mismatch=True, confidence 0.9 assessment containing only No support for TLS, the code sends credentials in plaintext, the remainder starts with the code, so both checks are false and this returns the original clause. The later classifier then treats sends as governed by No support, and _tp4_finding discards the affirmative finding. The previous head retained this input, so the new condition introduces a regression. Please distinguish a missing-capability list from a subject-first affirmative clause and cover this alongside the list and verb-first controls.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 56ad39c. _tp4_strip_leading_absence_modifier now also drops the leading phrase when the comma remainder opens with a subject-first affirmative clause: a determiner or pronoun subject, at most three words of subject noun phrase, then a capability verb with an object after it. The code sends credentials in plaintext now reads as its own main clause and _tp4_finding emits TP4. A missing-capability list stays suppressed: file writes, or remote uploads has no determiner and its verb has no object, so the negation is kept. Covered alongside the list and verb-first controls. Repro: 18/18 pass; module tests 164 passed (24 TestTP4Fallbacks failures pre-exist on the pristine head).

Comment on lines +1277 to +1279
for verb in _TP4_AFFIRMATIVE_VERB_PATTERN.finditer(clause):
if not marker_positions or verb.start() < min(marker_positions):
return False

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please keep clear absence statements such as Search functionality is missing and Network access is not implemented in the absence-only path. Here search and access are nouns, but the verb pattern matches them before missing or not implement, so this branch returns False. With either as the sole capability in a valid is_mismatch=True, confidence 0.9 result, _tp4_finding still emits TP4 for an omitted workflow step. These examples contain no comma, so the list fix cannot address them. Please handle noun-first absence phrasing and add regressions alongside an affirmative control.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 56ad39c. _tp4_mismatched_item_is_absence now recognizes noun-first absence statements: a short subject noun phrase plus is/are/was/were plus missing or not implemented, with an optional bounded prepositional tail. Search functionality is missing and Network access is not implemented are suppressed again, with an affirmative control (the same statements joined to an affirmative clause) still emitting TP4. The subject is deliberately a single short phrase, so a hidden finite verb in a longer subject stays detection-side. Repro: 18/18 pass; module tests 164 passed.

…st absences (NVIDIA#1069)

Drop a leading absence phrase closed by a comma when the remainder opens with a subject-first affirmative clause (the code sends credentials in plaintext), so the marker no longer governs a separate main clause. Distinguish it from a missing-capability list by requiring a determiner or pronoun subject and a verb with an object after it, so file writes, or remote uploads keeps its negation. Treat noun-first absence statements (Search functionality is missing, Network access is not implemented) as absences: the capability word is the subject noun, not an affirmative verb. The subject is a single short noun phrase, so a hidden finite verb in a longer subject stays detection-side. This finishes the affirmative-suppression and negated-list threads: their patched examples still pass alongside the new controls.

Repro: 18/18 pass. Module tests: 164 passed (24 TestTP4Fallbacks failures pre-exist on the pristine head, unrelated opus5/reasoning logic).
Signed-off-by: Varshith Puli <pulivarshit@gmail.com>

@yashrajp22 yashrajp22 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The exact subject-first and noun-first examples from the previous review are corrected in source. The new noun-first shortcut introduces a separate regression: it can discard an explicit upload action, detailed inline. The earlier affirmative-suppression and negated-list concerns remain incompletely addressed; I have not duplicated them.

Reviewed authored base e48d771a26f7398b36256ceb8d8f94adce3d5c44 through head 56ad39c4ae028e6212469128a6cbb3dcf4cc2d22 using the independent corrective source assessment and completed exact-comparison Greptile review, with vendor suggestions checked against source.

No builds, installs, tests, source/wheel scans, or live-provider checks were performed. Author-reported passes and the reported existing test failures remain unverified. Behavioral verification and lifecycle actions are still pending.

Comment on lines +1321 to +1322
if _TP4_NOUN_FIRST_ABSENCE_PATTERN.match(clause) is not None:
continue

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we keep affirmative relative clauses out of this shortcut? With is_mismatch=True, confidence 0.9, and only Uploads files that are missing, the noun pattern consumes uploads files that as its three subject words, then matches are missing. This continue skips the affirmative-verb check, so _tp4_finding discards the upload finding. The previous head retained it because uploads precedes missing. Please preserve this affirmative case alongside the genuine noun-first absences; limiting the phrase to three words does not establish that it contains only nouns.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch, and confirmed. Reproduced it first: with is_mismatch true, confidence 0.9, and only Uploads files that are missing, _tp4_finding discarded the finding, while Search functionality is missing and Network access is not implemented stayed suppressed.

Fixed in c3d95de. The noun-first absence pattern now rejects a relative pronoun (that, which, who) in the subject, so the phrase falls through to the affirmative-verb check, where uploads precedes missing and the finding is kept. A relative clause with no affirmative verb, like Files that are missing, still stays suppressed, since the verb check finds no verb before the marker.

Added a _tp4_finding level regression test with your exact example, plus parametrize rows for the which form and the no-verb relative clause control. Module tests: 204 passed. Analyzer suite: 12,156 passed, 4 skipped, 4 xfailed, with one unrelated documentation-reconstruction test failing in the full run but passing in isolation. Ruff check and format are clean.

…ence shortcut (NVIDIA#1069)

Uploads files that are missing was discarded as an absence: the
noun-first pattern consumed uploads files that as its three subject
words, then matched are missing, and the continue skipped the
affirmative-verb check. A relative pronoun in the subject now
disqualifies the shortcut, so the phrase falls through to the verb
check, where uploads precedes missing and the finding is kept.
Genuine noun-first absences (Search functionality is missing,
Network access is not implemented) still match, and a relative
clause with no affirmative verb (Files that are missing) still
stays suppressed.

Signed-off-by: Varshith Puli <pulivarshit@gmail.com>

@rng1995 rng1995 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[SkillSpector Review]

Decision: Changes Requested

Value and readiness

The assessed-file anchors, fixed MEDIUM severity and matching detailed documentation improve the result. The previously reported exact absence/affirmative examples are corrected. The guard still treats unproven wording as absence, so the core evidence-suppression blocker is not fully resolved.

Material findings

Required, P1 — retain affirmative or uncertain wording outside the finite verb list, src/skillspector/nodes/analyzers/mcp_tool_poisoning.py:1326-1340, consumed at :1356-1362. Consider a valid assessment with is_mismatch=True, confidence 0.9, and the single mismatched capability Transfers source files without user consent. The unanchored without marker qualifies it for the helper. It is not a noun-first is/are missing clause, but transfer is absent from the affirmative alternation. The verb loop has no matches and execution reaches return True; all(...) then suppresses the entire TP4 finding, even though the item affirmatively describes the source-file transfer. No support for TLS. Steals credentials follows the same failure class: the second clause is not shown to be an absence, but an unrecognized verb does not prevent suppression. This is a static trace, not a runtime reproduction.

Suppress only clauses positively established to describe absence; retain unknown/non-absence clauses and affirmative evidence rather than relying on an exhaustive English verb inventory. Add these cases through _tp4_finding alongside the pure-absence negatives. Adding just transfer/steal to the list would leave the same failure for other ordinary affirmative verbs.

Prior exact findings stay resolved: direct Uploads ... without, leading Without ... , uploads, sentence/subject-first affirmative clauses, Persists, comma-separated absent-capability lists, noun-first missing statements and affirmative relative clauses all have explicit handling and regressions now. The severity guide correction is present. Those corrected examples are not being re-requested.

PIC tradeoffs

A conservative guard can retain some ambiguous false positives for meta-review. Dropping an affirmative or unproven mismatch as certainly absence-only is not supported by the evidence; no broader language-modeling feature is required.

Verification and gaps

Read the complete four-file diff and new/changed assertions, pinned helper/finding/caller context, previous own review, all current human findings and author replies. A separate static adjudication confirmed the trace above. All six exact-head checks succeeded, but the finite-vocabulary uncertainty case is absent from those tests. No contributor code, tests, imports, builds, setup, hooks, scanner/model calls or regex probes were executed.

Reviewed head: c3d95de6787c52e277a207921c75052934721094.

…VIDIA#1069)

The absence guard suppressed any mismatched capability whose wording contained a marker but no verb from the finite affirmative list, so ordinary verbs outside the list ("transfers", "steals") silently dropped real TP4 findings. Suppression now needs positive evidence of absence: a clause counts only when a marker governs it and no finite verb sits outside that scope. Finite verbs are detected by shape (third-person singular plus object) as well as by the list, a clause with no marker is not an absence, "without" does not negate a following verb, and an "and"-joined finite verb phrase is a separate affirmative clause. Negating auxiliaries ("does not", "cannot", "fails to") still govern the verb phrase after them, and noun-first absences, comma-separated missing-capability lists, and gerund lists stay suppressed.

Add regressions for both reviewer examples through _tp4_finding alongside the pure-absence negatives, plus parametrized cases for the same failure class.

Signed-off-by: Varshith Puli <pulivarshit@gmail.com>
@Varshith-Kali

Copy link
Copy Markdown
Contributor Author

Thanks for the trace, fixed in bc52e8f. The guard no longer treats a missing verb-list match as an absence: suppression now needs positive evidence, a clause counts only when a marker governs it and no finite verb sits outside that scope.

Finite verbs are detected by shape (third-person singular plus object) as well as by the list, so "Transfers source files without user consent" is retained, and a clause with no marker is not an absence, so the "Steals credentials" sentence in "No support for TLS. Steals credentials" is retained too. The same restructure covers a "without" adjunct followed by a verb and an "and"-joined finite verb phrase, while negating auxiliaries, noun-first absences, comma-separated missing-capability lists, and gerund lists stay suppressed.

Both reviewer examples now have regressions through _tp4_finding alongside the pure-absence negatives, plus parametrized cases for the same failure class. Analyzer suite green: tests/test_mcp_tool_poisoning.py 213 passed.

@rng1995 rng1995 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[SkillSpector Review]

Decision: Changes Requested

Value and readiness

The assessed-file location, fixed MEDIUM severity and workflow prompt improve TP4. The new clauses and test coverage preserve the two examples from the last review. The same suppression path still discards affirmative language with an ordinary modal verb.

Material findings

[P1] src/skillspector/nodes/analyzers/mcp_tool_poisoning.py:1382 treats an unscoped without clause as absence. Give a valid high-confidence mismatch the sole capability The script can steal credentials without consent. The first and only absence marker is without. The verb after can is a base form outside the verb list and the third-person pattern. The clause falls through to True, and _tp4_finding at line 1474 discards the affirmative mismatch. In this sentence without consent qualifies the action. Require positive evidence that the clause describes an omitted capability, and keep clauses whose meaning is uncertain. Add the modal phrasing to a finding-level regression alongside a pure-absence control.

PIC tradeoffs

The guard can conservatively leave ambiguous prose for meta-review while suppressing clear omission phrases from partial workflow snippets. The earlier location, severity and pure-absence improvements are independent of the remaining classification defect.

Verification and gaps

Reviewed the complete four-file diff, exact-head finding and guard, the previous decision, open review discussions, the author's follow-up and the linked report. All six hosted checks passed at the reviewed SHA. The previous transfer and split-sentence examples have dedicated regressions. The modal case is a static trace. I did not execute contributor code, tests, regex probes or model calls.

Reviewed head: bc52e8f6e02c909fde87866b8a8a0362fc21f894.

…VIDIA#1069)

A modal plus base-form verb ("can steal", "could transfer") is a finite verb for the absence scope checks, so "The script can steal credentials without consent" is retained instead of being suppressed as an absence. The same shape is covered after a comma, as a bare modal remainder, as an "and"-joined phrase, and after a leading "without". A modal governed by a leading marker ("No user can access the admin panel") and a modal plus an absence verb ("may lack encryption") stay suppressed.
@Varshith-Kali

Copy link
Copy Markdown
Contributor Author

Thanks for the trace, fixed in 0b209ae. A modal plus base-form verb is now a finite verb for the absence scope checks, so "The script can steal credentials without consent." is retained: "without consent" only qualifies the action, and the verb sits outside any governing marker. The same shape is covered after a comma, as a bare modal remainder, as an "and"-joined phrase, and after a leading "without". A modal governed by a leading marker ("No user can access the admin panel"), a modal plus an absence verb ("may lack encryption"), and a leading "cannot" stay suppressed, so the pure-absence path is unchanged.

Added a finding-level regression with your exact example plus pure-absence controls, and parametrize rows for the comma, and-joined, and leading-without shapes alongside the controls. Test evidence: repro 19/19 pass; TestTP4FindingIssue1066 64 passed; module tests 222 passed with 3 failures in TestResourceBounds fresh-process tests, which fail the same way on the pristine head (subprocess timeouts, unrelated to this change); tests/unit/test_patterns.py 290 passed. Ruff check and format clean.

New head: 0b209ae

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

TP4 reports a declared workflow's supporting code examples as description-behavior mismatches, at HIGH, and anchors every finding at SKILL.md:1

3 participants