FIX: package hallucination scorer misses all but the first package in import a, b - #2454
Merged
Roman Lutz (romanlutz) merged 2 commits intoAug 21, 2026
Conversation
… `import a, b`
PackageHallucinationScorer's Python extraction used
`^import\s+([a-zA-Z0-9_][a-zA-Z0-9\-\_]*)`, which captures only the first module on
the line. `import a, b` is a single valid Python statement naming two packages, so
for a response containing
import os, hallucinated_pkg
the scorer extracted only `os`, never compared `hallucinated_pkg` against
known_packages, and returned False.
That is a false negative in a security scorer. The point of this scorer is catching
hallucinated dependencies, which are the attack surface for slopsquatting, and a
hallucinated package hidden as the second or later item on a comma list evades it
entirely. Nothing about the response looks unusual, so the miss is silent.
The first Python pattern now captures the whole import clause and
_split_python_import_clause reduces it to top-level package names. Each
comma-separated item drops any `as` alias so the alias is not mistaken for a package,
dotted paths reduce to the top-level module as garak does, and hyphenated
distributions survive. The `from ... import` pattern is unchanged, as is extraction
for Ruby, JavaScript and Rust, none of which have a comma-separated form.
Adds six tests covering the hidden package, alias handling, dotted paths, hyphenated
names, the end-to-end flag, and an all-real comma import that must not false-positive.
All six fail on main.
Same root cause as NVIDIA/garak#1991, which these regexes were ported from; that fix
also covered a double-escaped character class in the `from` pattern which is not
present here.
All 1634 tests in tests/unit/score pass.
Signed-off-by: WatchTree-19 <119982314+WatchTree-19@users.noreply.github.com>
Roman Lutz (romanlutz)
approved these changes
Aug 21, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
PackageHallucinationScorer's Python extraction uses:which captures only the first module on the line.
import a, bis a single valid Python statement naming two packages, so for a response containing:the scorer extracts only
os, never compareshallucinated_pkgagainstknown_packages, and returnsFalse.This is a false negative in a security scorer. The purpose of this scorer is catching hallucinated dependencies, which are the attack surface for slopsquatting. A hallucinated package hidden as the second or later item on a comma list evades it entirely, and nothing about the response looks unusual, so the miss is silent.
Before / after
import os, hallucinated_pkg{os}{os, hallucinated_pkg}import os, sys, evilpkg{os}{os, sys, evilpkg}import numpy as np, ghostlib{numpy}{numpy, ghostlib}import os.path, a.b.c{os}{os, a}import scikit-learn, ghost-lib{scikit-learn}{scikit-learn, ghost-lib}import os{os}{os}from foo-bar import x{foo-bar}{foo-bar}Fix
The first Python pattern now captures the whole import clause, and
_split_python_import_clausereduces it to top-level package names:asalias, so the alias is not mistaken for a package nameThe
from ... importpattern is unchanged, and so is extraction for Ruby, JavaScript and Rust — none of which have a comma-separated form with this problem.Tests
Six new tests: the hidden package, alias handling, dotted paths, hyphenated names, the end-to-end flag, and an all-real comma import that must not false-positive. All six fail on
main. All 1634 tests intests/unit/scorepass,ruff format --checkandruff checkclean.Notes
Same root cause as NVIDIA/garak#1991, which these regexes were ported from. That fix also covered a double-escaped character class in the
frompattern; that one is not present here, PyRIT's version is single-escaped and correct.#2446 is open against this same file, but it is additive (new ecosystems) and does not touch the Python patterns or
_extract_package_references, so this should be independent of it. Happy to rebase whichever way suits the merge order.