Skip to content

fix(sc4): read dependency manifests that start with a UTF-8 BOM - #1163

Open
YusefSyed wants to merge 1 commit into
NVIDIA:mainfrom
YusefSyed:fix/json-manifest-bom
Open

YusefSyed wants to merge 1 commit into
NVIDIA:mainfrom
YusefSyed:fix/json-manifest-bom

Conversation

@YusefSyed

Copy link
Copy Markdown

Closes #1162

Behaviour change

decode_text() keeps a leading UTF-8 byte-order mark, and json.loads() / tomllib.loads() reject it. A package.json, package-lock.json, npm-shrinkwrap.json, Pipfile, Pipfile.lock, pyproject.toml, uv.lock or poetry.lock that starts with a BOM therefore got a dependency_parse_error and the scan was marked incomplete. The lockfiles, Pipfile and pyproject.toml sent no versions to OSV, a single-line package.json yielded no dependencies from its line-scan fallback, and a BOM-prefixed lockfile no longer pinned its manifest's ranges. npm (json-parse-even-better-errors strips the BOM before JSON.parse), pipenv (utf-8-sig) and uv (uv lock --check, uv export --frozen) each read their own files with a BOM. poetry.lock goes through the same TOML lockfile parser in SkillSpector as uv.lock.

The requirements.txt strip from #963 moves into _strip_leading_bom(), which the JSON and TOML manifest parsers and npm lock-pin collection now call before parsing. Only leading marks are removed; a U+FEFF inside the file is kept. decode_text() stays unchanged, and line numbers do not move because the mark sits before the first line break.

Verification

  • Real CLI scans (--no-llm, OSV recorded instead of queried) on main 48e0f94, each file with and without a BOM: without it the file's entry (lodash 4.17.20 or pyyaml 5.3.1) reached OSV; with it nothing did, and the scan reported a dependency_parse_error. On this branch both send the entry and the scan is complete.
  • New tests in tests/nodes/analyzers/test_static_patterns_supply_chain_lockfiles.py write the BOM bytes to disk and scan through the CLI: one per file type, one per lockfile (package-lock.json, npm-shrinkwrap.json, Pipfile.lock, uv.lock) that must still pin its manifest's range, and one that a U+FEFF inside package.json stays part of the package name. All 13 fail on main and pass here.
  • pytest -m "not integration and not provider" on every test file that names the supply-chain analyzer module passes apart from test_build_context_empty_directory_is_valid_empty_scan, which fails the same way on main on this machine (it passes in main's CI).
  • ruff check src tests and ruff format --check src tests pass.
  • Not run locally: the full make test-ci suite, the Docker smoke test and the Node tests.

decode_text() keeps a leading byte-order mark, so json.loads() and
tomllib.loads() rejected a BOM-prefixed package.json, package-lock.json,
npm-shrinkwrap.json, Pipfile, Pipfile.lock, pyproject.toml, uv.lock or
poetry.lock, and the scan recorded a dependency parse error for the
file. The lockfiles, Pipfile and pyproject.toml sent no versions to
OSV, a package.json fell back to its line scan (which finds nothing in
a single-line manifest), and a BOM-prefixed lockfile no longer pinned
its manifest's ranges. npm, pipenv and uv each read their own files
when they start with a BOM.

Move the requirements.txt BOM strip into a shared helper and call it
in SC4's JSON and TOML dependency parsers, including npm lock-pin
collection. Only leading marks are removed.

Signed-off-by: YusefSyed <211442445+YusefSyed@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

SC4 skips dependency manifests and lockfiles that start with a UTF-8 BOM

1 participant