Repository navigation
fix(tp4): anchor findings to the assessed file, align severity, and drop absence-only mismatches - #1069
fix(tp4): anchor findings to the assessed file, align severity, and drop absence-only mismatches#1069Varshith-Kali wants to merge 9 commits into
Conversation
…rop absence-only mismatches TP4 reported every finding at SKILL.md:1 even when the assessed fence lived in another file, escalated severity to HIGH at confidence 0.7 while the README pattern table lists TP4 as MEDIUM, and flagged partial workflow snippets for omitting the rest of the declared workflow. Anchor each TP4 finding to the batch file and lines, which the evidence already carried. TP4 severity is now always MEDIUM, matching the README. Add a partial-snippet criterion to the TP4 prompt, and suppress an assessment deterministically when every mismatched_capabilities entry is phrased as an absence, since that carries no affirmative evidence of a mismatch. Mixed and affirmative assessments still emit. Fixes NVIDIA#1066 Signed-off-by: Varshith Puli <pulivarshit@gmail.com>
rng1995
left a comment
There was a problem hiding this comment.
[SkillSpector Review]
Decision: Changes Requested (reviewed head e739a9a59f8f022f5a3782b85f6c818b0073d5be).
Value and readiness: Anchoring TP4 to the assessed file and lines fixes actionable evidence, and MEDIUM matches the documented severity. The partial-workflow prompt change is sensible. The deterministic absence guard, however, also discards affirmative evidence of undeclared behavior.
Material finding — blocker, P2: In src/skillspector/nodes/analyzers/mcp_tool_poisoning.py:1172-1185, words such as without, missing, and lacks anywhere in a capability make the entire item count as an absence. _tp4_finding then returns None at lines 1201-1207 if all items satisfy that test. For example, an assessment with is_mismatch=True, confidence 0.9, and one item Uploads source files to an undeclared endpoint without user consent is suppressed because it contains without. That item describes an affirmative undeclared capability; neither an affirmative actual-behavior summary nor explanation can rescue it. The old code emitted the finding.
Please make suppression conservative: only discard an assessment when every item actually states absence of a declared workflow capability, retaining any item that also describes affirmative behavior. Add regressions for an affirmative capability containing without and for an item containing both an affirmative action and a negated clause. The current mixed test puts the affirmative evidence in a separate list item, so it does not cover this failure.
PIC tradeoffs: None requiring a separate decision for this fix.
Verification and gaps: Reviewed the complete three-file diff and pinned source, tracing the new guard through finding construction. The regression assertions cover anchoring, severity, pure absence, separate mixed entries, affirmative entries, and HTTP graph anchoring; they omit affirmative and absence wording within the same entry. All six hosted CI checks passed for this exact head (run). No contributor code, tests, imports, builds, setup, hooks, regex probes, or live model were executed. The finding above follows directly from the literal without matcher and unconditional suppression branch; it is not a claimed runtime reproduction.
…DIA#1069) An absence marker anywhere in a mismatched capability (without, missing, lacks) suppressed the whole assessment, even when the item described an affirmative undeclared capability such as uploading source files to an undeclared endpoint without user consent. Make suppression conservative: an item counts as an absence only when no clause of it describes affirmative behavior, with capability verbs under a marker treated as governed by it. Add regressions for an affirmative capability containing without and for an item holding both an affirmative action and a negated clause. Signed-off-by: Varshith Puli <pulivarshit@gmail.com>
|
Thanks for the catch. Fixed: Added regressions: Evidence: tests/test_mcp_tool_poisoning.py 180 passed (6 deselected); analyzer suite 12,133 passed, 4 skipped, 4 xfailed. New head dd3b48a. |
Signed-off-by: Varshith Puli <pulivarshit@gmail.com>
017b3a9 to
4f87c4c
Compare
yashrajp22
left a comment
There was a problem hiding this comment.
One source-confirmed behavior bug remains in the absence-only guard, and the detailed TP4 severity guide needs to match the new MEDIUM rule.
Reviewed authored base e48d771a26f7398b36256ceb8d8f94adce3d5c44 through head 4f87c4c4caf406664d3c0e2e930c3227292ba444. The independent static assessment and exact-comparison Greptile review are complete, and both Greptile suggestions have been checked against source.
No builds, installs, tests, source/wheel scans, or live-provider checks were performed. The prompt changes and the reporter’s affirmative/mixed examples remain unverified; this is not a complete behavioral-verification result.
| for verb in _TP4_AFFIRMATIVE_VERB_PATTERN.finditer(clause): | ||
| if not marker_positions or verb.start() < min(marker_positions): | ||
| return False | ||
| return True |
There was a problem hiding this comment.
Could we keep affirmative behavior when an absence phrase comes first? For a valid is_mismatch=True, confidence 0.9 assessment containing only Without user consent, uploads source files to a remote endpoint, the marker is at position zero and every verb follows it, so this returns True. No support for TLS. Sends credentials in plaintext is also treated as one negated clause. The finite verb list additionally drops Persists credentials without user consent. These describe actual behavior, but _tp4_finding discards them and the caller records a completed batch with no TP4 finding. Please suppress only unambiguously absence-only entries, retain affirmative or uncertain phrasing, and cover these cases with regressions.
There was a problem hiding this comment.
Fixed in cc0165e. All three shapes are now retained: a leading absence phrase closed by a comma ("Without user consent, uploads source files ...") only modifies the main clause, so the marker no longer governs the verbs after the comma; clauses now split on sentence boundaries too, so "No support for TLS. Sends credentials in plaintext" keeps its second sentence affirmative; and persist joins the affirmative verb list, so "Persists credentials without user consent" is retained. The guard stays conservative: pure absence entries are still suppressed, including "No evidence of querying, storing videos" (the verb sits inside the leading phrase, before its comma) and "Without user consent, no encryption is performed". Added regression tests for all three examples plus the pure-absence controls: module 187 passed, analyzer suite 12,140 passed.
There was a problem hiding this comment.
Follow-up in 56ad39c: the remaining gap in this thread was the subject-first clause after a comma (No support for TLS, the code sends credentials in plaintext), which the list fix did not cover because the remainder opens with a subject, not a verb. The strip now recognizes that shape as a separate main clause. The three shapes fixed earlier (leading without, second sentence, persists) still pass, verified in the same repro run. Repro: 18/18 pass.
| # TP4 is documented as MEDIUM in the README pattern table; the | ||
| # confidence-based escalation to HIGH was never documented (issue #1066). | ||
| severity="MEDIUM", |
There was a problem hiding this comment.
Please update the detailed TP4 guide alongside this severity change. docs/B.3.2-mcp-tool-poisoning.md:212 currently says MEDIUM or HIGH (based on LLM confidence), so escalation was documented there and users following that guide will still expect HIGH findings. The new fixed MEDIUM behavior is consistent with the README; the detailed guide and this comment should describe the same rule.
There was a problem hiding this comment.
Done in cc0165e. docs/B.3.2-mcp-tool-poisoning.md now states the TP4 severity as fixed MEDIUM, matching the README pattern table. I also updated the code comment in _tp4_finding, which previously said the escalation was never documented; it now states the fixed MEDIUM rule and points at the guide. Line 212 was the only place in the guide stating the old MEDIUM or HIGH rule.
…VIDIA#1069) The absence guard still suppressed affirmative capabilities in three shapes: a leading absence phrase closed by a comma (Without user consent, uploads source files ...), an affirmative second sentence (No support for TLS. Sends credentials in plaintext), and verbs missing from the finite list (Persists credentials without user consent). Split clauses on sentence boundaries as well as contrast boundaries, drop a leading absence phrase only when its closing comma precedes the first capability verb, and add persist to the affirmative verb list. Pure absence entries stay suppressed. The detailed TP4 guide now states the fixed MEDIUM severity, matching the README and the code comment. Signed-off-by: Varshith Puli <pulivarshit@gmail.com>
yashrajp22
left a comment
There was a problem hiding this comment.
The three previously cited phrases and the severity documentation are corrected in source. The new comma handling introduces an absence-only list regression, detailed inline. The broader affirmative-preservation request in the existing suppression thread also remains open; I have not duplicated that finding.
This review covers authored base e48d771a26f7398b36256ceb8d8f94adce3d5c44 through head cc0165e8421b34fb16a8178c39d4a58aa895e11d, using the independent corrective source assessment and completed exact-comparison Greptile review, with suggestions checked against source.
No builds, installs, tests, source/wheel scans, or live-provider checks were performed. Author-reported test results and broader workflow accuracy remain unverified; this does not complete behavioral verification.
| first_verb = _TP4_AFFIRMATIVE_VERB_PATTERN.search(clause) | ||
| if first_verb is not None and first_verb.start() < comma: | ||
| return clause | ||
| return clause[comma + 1 :].strip() |
There was a problem hiding this comment.
Could we preserve the leading negation for comma-separated lists of missing capabilities? With is_mismatch=True, confidence 0.9, and only No evidence of network activity, file writes, or remote uploads, the first recognized verb is writes, after the comma. This helper therefore strips the negative prefix and leaves file writes, or remote uploads. The caller sees a verb with no absence marker, so _tp4_finding emits a MEDIUM finding even though the entire original entry describes absent behavior. The previous head suppressed this input. Please keep negation across these lists and add a regression alongside the affirmative leading-without case.
There was a problem hiding this comment.
Fixed in 8ef3c59. The regression was in _tp4_strip_leading_absence_modifier: it dropped a leading absence phrase whenever the first capability verb fell after the comma, so "No evidence of network activity, file writes, or remote uploads" was stripped to "file writes, or remote uploads" and read as affirmative. The strip now only applies when the text after the comma reads as a main clause: a "without" modifier, or a remainder that opens with a capability verb (so "No support for TLS, sends credentials in plaintext" stays retained). Noun-first list continuations keep their leading negation.
Test evidence: the repro script confirmed the failure before the fix (a MEDIUM finding was emitted for the reviewer example) and 7/7 cases pass after, including the affirmative leading-"without" case and the three shapes from the earlier suppression thread, which I re-verified and did not regress. Module tests: 190 passed, 6 deselected. Analyzer suite plus module: 12,143 passed, 4 skipped, 11 deselected, 4 xfailed, 0 failed. Ruff check and format clean on both changed files.
New head: 8ef3c59
There was a problem hiding this comment.
Follow-up in 56ad39c: the remaining gap in this thread was noun-first absence phrasing (Search functionality is missing, Network access is not implemented), which the list fix could not address because there is no comma. Those statements are now recognized as absences while affirmative controls still emit TP4. The list case itself still passes. Repro: 18/18 pass.
…ilities (NVIDIA#1069) The leading-absence strip dropped 'No evidence of ...' whenever the first capability verb fell after the comma, so 'No evidence of network activity, file writes, or remote uploads' was read as an affirmative finding. The strip now only applies when the text after the comma reads as a main clause: a 'without' modifier, or a remainder that opens with a capability verb. Noun-first list continuations keep their leading negation, while the affirmative leading-'without' case and the sentence-split cases stay retained. Adds a _tp4_finding regression for the reviewer example plus parametrize rows for the list and the verb-first remainder. Signed-off-by: Varshith Puli <pulivarshit@gmail.com>
yashrajp22
left a comment
There was a problem hiding this comment.
The new prefix check regresses affirmative clauses with explicit subjects. The absence guard also retains simple noun-first omission statements; both are detailed inline. The earlier affirmative-suppression and negated-list requests remain incompletely addressed, while their specific patched examples are corrected in source.
Reviewed authored base e48d771a26f7398b36256ceb8d8f94adce3d5c44 through head 8ef3c595a78edec3e30c87ebb27aaf47281b3023 using the independent corrective source assessment and completed exact-comparison Greptile review. Both vendor suggestions were adjudicated against source; one duplicates an existing finding.
No builds, installs, tests, source/wheel scans, or live-provider checks were performed. Author-reported tests and broader workflow accuracy remain unverified; behavioral verification and lifecycle actions are still pending.
| is_without_modifier = clause.casefold().startswith("without ") | ||
| remainder_opens_with_verb = _TP4_AFFIRMATIVE_VERB_PATTERN.match(remainder) is not None | ||
| if not (is_without_modifier or remainder_opens_with_verb): | ||
| return clause |
There was a problem hiding this comment.
Could we retain affirmative clauses that start with a subject? For a valid is_mismatch=True, confidence 0.9 assessment containing only No support for TLS, the code sends credentials in plaintext, the remainder starts with the code, so both checks are false and this returns the original clause. The later classifier then treats sends as governed by No support, and _tp4_finding discards the affirmative finding. The previous head retained this input, so the new condition introduces a regression. Please distinguish a missing-capability list from a subject-first affirmative clause and cover this alongside the list and verb-first controls.
There was a problem hiding this comment.
Fixed in 56ad39c. _tp4_strip_leading_absence_modifier now also drops the leading phrase when the comma remainder opens with a subject-first affirmative clause: a determiner or pronoun subject, at most three words of subject noun phrase, then a capability verb with an object after it. The code sends credentials in plaintext now reads as its own main clause and _tp4_finding emits TP4. A missing-capability list stays suppressed: file writes, or remote uploads has no determiner and its verb has no object, so the negation is kept. Covered alongside the list and verb-first controls. Repro: 18/18 pass; module tests 164 passed (24 TestTP4Fallbacks failures pre-exist on the pristine head).
| for verb in _TP4_AFFIRMATIVE_VERB_PATTERN.finditer(clause): | ||
| if not marker_positions or verb.start() < min(marker_positions): | ||
| return False |
There was a problem hiding this comment.
Please keep clear absence statements such as Search functionality is missing and Network access is not implemented in the absence-only path. Here search and access are nouns, but the verb pattern matches them before missing or not implement, so this branch returns False. With either as the sole capability in a valid is_mismatch=True, confidence 0.9 result, _tp4_finding still emits TP4 for an omitted workflow step. These examples contain no comma, so the list fix cannot address them. Please handle noun-first absence phrasing and add regressions alongside an affirmative control.
There was a problem hiding this comment.
Fixed in 56ad39c. _tp4_mismatched_item_is_absence now recognizes noun-first absence statements: a short subject noun phrase plus is/are/was/were plus missing or not implemented, with an optional bounded prepositional tail. Search functionality is missing and Network access is not implemented are suppressed again, with an affirmative control (the same statements joined to an affirmative clause) still emitting TP4. The subject is deliberately a single short phrase, so a hidden finite verb in a longer subject stays detection-side. Repro: 18/18 pass; module tests 164 passed.
…st absences (NVIDIA#1069) Drop a leading absence phrase closed by a comma when the remainder opens with a subject-first affirmative clause (the code sends credentials in plaintext), so the marker no longer governs a separate main clause. Distinguish it from a missing-capability list by requiring a determiner or pronoun subject and a verb with an object after it, so file writes, or remote uploads keeps its negation. Treat noun-first absence statements (Search functionality is missing, Network access is not implemented) as absences: the capability word is the subject noun, not an affirmative verb. The subject is a single short noun phrase, so a hidden finite verb in a longer subject stays detection-side. This finishes the affirmative-suppression and negated-list threads: their patched examples still pass alongside the new controls. Repro: 18/18 pass. Module tests: 164 passed (24 TestTP4Fallbacks failures pre-exist on the pristine head, unrelated opus5/reasoning logic). Signed-off-by: Varshith Puli <pulivarshit@gmail.com>
yashrajp22
left a comment
There was a problem hiding this comment.
The exact subject-first and noun-first examples from the previous review are corrected in source. The new noun-first shortcut introduces a separate regression: it can discard an explicit upload action, detailed inline. The earlier affirmative-suppression and negated-list concerns remain incompletely addressed; I have not duplicated them.
Reviewed authored base e48d771a26f7398b36256ceb8d8f94adce3d5c44 through head 56ad39c4ae028e6212469128a6cbb3dcf4cc2d22 using the independent corrective source assessment and completed exact-comparison Greptile review, with vendor suggestions checked against source.
No builds, installs, tests, source/wheel scans, or live-provider checks were performed. Author-reported passes and the reported existing test failures remain unverified. Behavioral verification and lifecycle actions are still pending.
| if _TP4_NOUN_FIRST_ABSENCE_PATTERN.match(clause) is not None: | ||
| continue |
There was a problem hiding this comment.
Could we keep affirmative relative clauses out of this shortcut? With is_mismatch=True, confidence 0.9, and only Uploads files that are missing, the noun pattern consumes uploads files that as its three subject words, then matches are missing. This continue skips the affirmative-verb check, so _tp4_finding discards the upload finding. The previous head retained it because uploads precedes missing. Please preserve this affirmative case alongside the genuine noun-first absences; limiting the phrase to three words does not establish that it contains only nouns.
There was a problem hiding this comment.
Good catch, and confirmed. Reproduced it first: with is_mismatch true, confidence 0.9, and only Uploads files that are missing, _tp4_finding discarded the finding, while Search functionality is missing and Network access is not implemented stayed suppressed.
Fixed in c3d95de. The noun-first absence pattern now rejects a relative pronoun (that, which, who) in the subject, so the phrase falls through to the affirmative-verb check, where uploads precedes missing and the finding is kept. A relative clause with no affirmative verb, like Files that are missing, still stays suppressed, since the verb check finds no verb before the marker.
Added a _tp4_finding level regression test with your exact example, plus parametrize rows for the which form and the no-verb relative clause control. Module tests: 204 passed. Analyzer suite: 12,156 passed, 4 skipped, 4 xfailed, with one unrelated documentation-reconstruction test failing in the full run but passing in isolation. Ruff check and format are clean.
…ence shortcut (NVIDIA#1069) Uploads files that are missing was discarded as an absence: the noun-first pattern consumed uploads files that as its three subject words, then matched are missing, and the continue skipped the affirmative-verb check. A relative pronoun in the subject now disqualifies the shortcut, so the phrase falls through to the verb check, where uploads precedes missing and the finding is kept. Genuine noun-first absences (Search functionality is missing, Network access is not implemented) still match, and a relative clause with no affirmative verb (Files that are missing) still stays suppressed. Signed-off-by: Varshith Puli <pulivarshit@gmail.com>
rng1995
left a comment
There was a problem hiding this comment.
[SkillSpector Review]
Decision: Changes Requested
Value and readiness
The assessed-file anchors, fixed MEDIUM severity and matching detailed documentation improve the result. The previously reported exact absence/affirmative examples are corrected. The guard still treats unproven wording as absence, so the core evidence-suppression blocker is not fully resolved.
Material findings
Required, P1 — retain affirmative or uncertain wording outside the finite verb list, src/skillspector/nodes/analyzers/mcp_tool_poisoning.py:1326-1340, consumed at :1356-1362. Consider a valid assessment with is_mismatch=True, confidence 0.9, and the single mismatched capability Transfers source files without user consent. The unanchored without marker qualifies it for the helper. It is not a noun-first is/are missing clause, but transfer is absent from the affirmative alternation. The verb loop has no matches and execution reaches return True; all(...) then suppresses the entire TP4 finding, even though the item affirmatively describes the source-file transfer. No support for TLS. Steals credentials follows the same failure class: the second clause is not shown to be an absence, but an unrecognized verb does not prevent suppression. This is a static trace, not a runtime reproduction.
Suppress only clauses positively established to describe absence; retain unknown/non-absence clauses and affirmative evidence rather than relying on an exhaustive English verb inventory. Add these cases through _tp4_finding alongside the pure-absence negatives. Adding just transfer/steal to the list would leave the same failure for other ordinary affirmative verbs.
Prior exact findings stay resolved: direct Uploads ... without, leading Without ... , uploads, sentence/subject-first affirmative clauses, Persists, comma-separated absent-capability lists, noun-first missing statements and affirmative relative clauses all have explicit handling and regressions now. The severity guide correction is present. Those corrected examples are not being re-requested.
PIC tradeoffs
A conservative guard can retain some ambiguous false positives for meta-review. Dropping an affirmative or unproven mismatch as certainly absence-only is not supported by the evidence; no broader language-modeling feature is required.
Verification and gaps
Read the complete four-file diff and new/changed assertions, pinned helper/finding/caller context, previous own review, all current human findings and author replies. A separate static adjudication confirmed the trace above. All six exact-head checks succeeded, but the finite-vocabulary uncertainty case is absent from those tests. No contributor code, tests, imports, builds, setup, hooks, scanner/model calls or regex probes were executed.
Reviewed head: c3d95de6787c52e277a207921c75052934721094.
…VIDIA#1069) The absence guard suppressed any mismatched capability whose wording contained a marker but no verb from the finite affirmative list, so ordinary verbs outside the list ("transfers", "steals") silently dropped real TP4 findings. Suppression now needs positive evidence of absence: a clause counts only when a marker governs it and no finite verb sits outside that scope. Finite verbs are detected by shape (third-person singular plus object) as well as by the list, a clause with no marker is not an absence, "without" does not negate a following verb, and an "and"-joined finite verb phrase is a separate affirmative clause. Negating auxiliaries ("does not", "cannot", "fails to") still govern the verb phrase after them, and noun-first absences, comma-separated missing-capability lists, and gerund lists stay suppressed. Add regressions for both reviewer examples through _tp4_finding alongside the pure-absence negatives, plus parametrized cases for the same failure class. Signed-off-by: Varshith Puli <pulivarshit@gmail.com>
|
Thanks for the trace, fixed in bc52e8f. The guard no longer treats a missing verb-list match as an absence: suppression now needs positive evidence, a clause counts only when a marker governs it and no finite verb sits outside that scope. Finite verbs are detected by shape (third-person singular plus object) as well as by the list, so "Transfers source files without user consent" is retained, and a clause with no marker is not an absence, so the "Steals credentials" sentence in "No support for TLS. Steals credentials" is retained too. The same restructure covers a "without" adjunct followed by a verb and an "and"-joined finite verb phrase, while negating auxiliaries, noun-first absences, comma-separated missing-capability lists, and gerund lists stay suppressed. Both reviewer examples now have regressions through _tp4_finding alongside the pure-absence negatives, plus parametrized cases for the same failure class. Analyzer suite green: tests/test_mcp_tool_poisoning.py 213 passed. |
rng1995
left a comment
There was a problem hiding this comment.
[SkillSpector Review]
Decision: Changes Requested
Value and readiness
The assessed-file location, fixed MEDIUM severity and workflow prompt improve TP4. The new clauses and test coverage preserve the two examples from the last review. The same suppression path still discards affirmative language with an ordinary modal verb.
Material findings
[P1] src/skillspector/nodes/analyzers/mcp_tool_poisoning.py:1382 treats an unscoped without clause as absence. Give a valid high-confidence mismatch the sole capability The script can steal credentials without consent. The first and only absence marker is without. The verb after can is a base form outside the verb list and the third-person pattern. The clause falls through to True, and _tp4_finding at line 1474 discards the affirmative mismatch. In this sentence without consent qualifies the action. Require positive evidence that the clause describes an omitted capability, and keep clauses whose meaning is uncertain. Add the modal phrasing to a finding-level regression alongside a pure-absence control.
PIC tradeoffs
The guard can conservatively leave ambiguous prose for meta-review while suppressing clear omission phrases from partial workflow snippets. The earlier location, severity and pure-absence improvements are independent of the remaining classification defect.
Verification and gaps
Reviewed the complete four-file diff, exact-head finding and guard, the previous decision, open review discussions, the author's follow-up and the linked report. All six hosted checks passed at the reviewed SHA. The previous transfer and split-sentence examples have dedicated regressions. The modal case is a static trace. I did not execute contributor code, tests, regex probes or model calls.
Reviewed head: bc52e8f6e02c909fde87866b8a8a0362fc21f894.
…VIDIA#1069) A modal plus base-form verb ("can steal", "could transfer") is a finite verb for the absence scope checks, so "The script can steal credentials without consent" is retained instead of being suppressed as an absence. The same shape is covered after a comma, as a bare modal remainder, as an "and"-joined phrase, and after a leading "without". A modal governed by a leading marker ("No user can access the admin panel") and a modal plus an absence verb ("may lack encryption") stay suppressed.
|
Thanks for the trace, fixed in 0b209ae. A modal plus base-form verb is now a finite verb for the absence scope checks, so "The script can steal credentials without consent." is retained: "without consent" only qualifies the action, and the verb sits outside any governing marker. The same shape is covered after a comma, as a bare modal remainder, as an "and"-joined phrase, and after a leading "without". A modal governed by a leading marker ("No user can access the admin panel"), a modal plus an absence verb ("may lack encryption"), and a leading "cannot" stay suppressed, so the pure-absence path is unchanged. Added a finding-level regression with your exact example plus pure-absence controls, and parametrize rows for the comma, and-joined, and leading-without shapes alongside the controls. Test evidence: repro 19/19 pass; TestTP4FindingIssue1066 64 passed; module tests 222 passed with 3 failures in TestResourceBounds fresh-process tests, which fail the same way on the pristine head (subprocess timeouts, unrelated to this change); tests/unit/test_patterns.py 290 passed. Ruff check and format clean. New head: 0b209ae |
Fixes #1066.
Three parts, all in the TP4 path:
Anchoring.
_tp4_findinghardcodedfile="SKILL.md"with no line, so a finding onreferences/briefing-authoring.md:27-28reported atSKILL.md:1. Findings now carry the assessed batch file and lines, which the evidence already had. The meta-analyzer formatter needed no change, it already rendersfileandstart_line.Severity. TP4 escalated to HIGH at confidence 0.7, while the README pattern table lists TP4 as MEDIUM. TP4 is now always MEDIUM, matching the documented pattern.
Partial-snippet criterion. A fence implementing one declared workflow step was flagged for omitting the rest of the workflow, with
mismatched_capabilitiesphrased as absences ("No evidence of querying videos"). The TP4 prompt now states a chunk may implement only one declared workflow step and each entry must name what the code does. Deterministically, an assessment is suppressed when everymismatched_capabilitiesentry is phrased as an absence, since that carries no affirmative evidence of a mismatch. Mixed or affirmative assessments still emit.Tests: 13 new regression tests in
tests/test_mcp_tool_poisoning.py(12 failed pre-fix, 13 pass post-fix), including the issue verbatim anchoring case and the video-intel-search absence items.tests/test_mcp_tool_poisoning.py174 passed,tests/unit/test_tp4_openai_http.py51 passed, full suite 20964 passed with 8 failures reproduced on clean main (pre-existing Anthropic provider issues, unrelated). Ruff check and format clean. The prompt addition was kept compact (206 to 238 tokens) so the batch budget tests still pass.Limits, stated plainly: the deterministic guard covers only the pure-absence class. The issue EPUB and PDF findings named affirmative behaviors, and the real pytest finding was mixed, so those still emit and rely on the prompt criterion plus meta-review, which are nondeterministic. No live-LLM verification was run; the LLM-judged part was verified with stubbed inputs only.