Repository navigation
Conversation
decode_text() keeps a leading byte-order mark, so json.loads() and tomllib.loads() rejected a BOM-prefixed package.json, package-lock.json, npm-shrinkwrap.json, Pipfile, Pipfile.lock, pyproject.toml, uv.lock or poetry.lock, and the scan recorded a dependency parse error for the file. The lockfiles, Pipfile and pyproject.toml sent no versions to OSV, a package.json fell back to its line scan (which finds nothing in a single-line manifest), and a BOM-prefixed lockfile no longer pinned its manifest's ranges. npm, pipenv and uv each read their own files when they start with a BOM. Move the requirements.txt BOM strip into a shared helper and call it in SC4's JSON and TOML dependency parsers, including npm lock-pin collection. Only leading marks are removed. Signed-off-by: YusefSyed <211442445+YusefSyed@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #1162
Behaviour change
decode_text()keeps a leading UTF-8 byte-order mark, andjson.loads()/tomllib.loads()reject it. Apackage.json,package-lock.json,npm-shrinkwrap.json,Pipfile,Pipfile.lock,pyproject.toml,uv.lockorpoetry.lockthat starts with a BOM therefore got adependency_parse_errorand the scan was marked incomplete. The lockfiles,Pipfileandpyproject.tomlsent no versions to OSV, a single-linepackage.jsonyielded no dependencies from its line-scan fallback, and a BOM-prefixed lockfile no longer pinned its manifest's ranges. npm (json-parse-even-better-errorsstrips the BOM beforeJSON.parse), pipenv (utf-8-sig) and uv (uv lock --check,uv export --frozen) each read their own files with a BOM.poetry.lockgoes through the same TOML lockfile parser in SkillSpector asuv.lock.The requirements.txt strip from #963 moves into
_strip_leading_bom(), which the JSON and TOML manifest parsers and npm lock-pin collection now call before parsing. Only leading marks are removed; a U+FEFF inside the file is kept.decode_text()stays unchanged, and line numbers do not move because the mark sits before the first line break.Verification
--no-llm, OSV recorded instead of queried) on main 48e0f94, each file with and without a BOM: without it the file's entry (lodash 4.17.20orpyyaml 5.3.1) reached OSV; with it nothing did, and the scan reported adependency_parse_error. On this branch both send the entry and the scan is complete.tests/nodes/analyzers/test_static_patterns_supply_chain_lockfiles.pywrite the BOM bytes to disk and scan through the CLI: one per file type, one per lockfile (package-lock.json,npm-shrinkwrap.json,Pipfile.lock,uv.lock) that must still pin its manifest's range, and one that a U+FEFF insidepackage.jsonstays part of the package name. All 13 fail on main and pass here.pytest -m "not integration and not provider"on every test file that names the supply-chain analyzer module passes apart fromtest_build_context_empty_directory_is_valid_empty_scan, which fails the same way on main on this machine (it passes in main's CI).ruff check src testsandruff format --check src testspass.make test-cisuite, the Docker smoke test and the Node tests.