Repository navigation
Conversation
BASE_ANALYSIS_PROMPT embedded the scanned skill with no adversarial framing and a precision-over-recall bias, so a skill could talk the semantic analyzers out of SSD findings, which have no deterministic backstop. Add an explicit untrusted-input section and wrap the file in markers carrying a nonce derived from its own content (content_nonce), keeping prompts deterministic for caching and seeded runs while a file cannot forge its own closing marker. Adds tests for the framing, nonce determinism, and a forged closing marker. Signed-off-by: oherbawi <oherbawi@gmail.com>
The builder ran pip install ., resolving the >= ranges in pyproject.toml afresh on every build and ignoring uv.lock, and the runtime image ran as root. Sync with uv 0.10.10 (the release workflow's version, pinned by image digest) using --frozen, and run as uid 10001 with HOME=/tmp so any uid works. Writing reports into a bind mount now needs --user "$(id -u):$(id -g)": the README examples and tests/docker/smoke.sh pass it, and the smoke test fails if the image defaults to root. Also pin the commented-out codeql-action upload-sarif step by commit so uncommenting it is safe. Signed-off-by: oherbawi <oherbawi@gmail.com>
PER_FILE_ANALYSIS_PROMPT wrapped raw file content in a fixed triple-backtick fence, so a file containing a backtick fence could close it and continue the prompt, for example with a fake findings section. Size the fence one longer than the longest backtick run in the content (the rule nodes/report.py already uses). Signed-off-by: oherbawi <oherbawi@gmail.com>
_is_git_url tested 'allowed in host', so github.com.evil.example was classified as a Git URL. Not exploitable, since _validate_url_host enforces the strict suffix allowlist before any clone, but it misclassified the input and produced a misleading error. Use the same rule as _validate_url_host. Signed-off-by: oherbawi <oherbawi@gmail.com>
Both integrations pass the target to the CLI as an argv element after 'scan', so a target such as --output=... was parsed as a CLI option. Reject a target starting with '-' and point to the ./-name form for a real path. Signed-off-by: oherbawi <oherbawi@gmail.com>
ruff I001: place the content_nonce import in the order ruff's isort rules expect. Signed-off-by: oherbawi <oherbawi@gmail.com>
yashrajp22
left a comment
There was a problem hiding this comment.
One introduced prompt-budget regression remains in the meta analyzer; the inline comment includes a reproduction.
Validation: fresh base/HEAD wheels; selected Python tests (122 base and 127 HEAD passed in each source/wheel lane, one platform-specific skip per lane); OpenCode and Pi tests; targeted prompt/URL/integration probes; 48 offline regression scans with matching normalized results and existing partial-analysis outcomes preserved. Existing Docker build/smoke CI ran on a tree identical to HEAD. Provider context rejection was simulated; no live-model resistance claim is made.
The backtick fence was sized to outlast the longest backtick run in the content, so its length grew with the file while get_batches budgets only the template and raw content. A file of 300,000 backticks fit one 96,000-token batch but framed to about 225,000 tokens and could exceed the model context. Use the untrusted-file markers with content_nonce, as BASE_ANALYSIS_PROMPT does: their size is constant and sits in the template get_batches already budgets, and a file cannot forge its own closing marker. Adds a long-backtick regression case: 300,000 backticks add the same framing overhead as a one-character file. Signed-off-by: oherbawi <oherbawi@gmail.com>
yashrajp22
left a comment
There was a problem hiding this comment.
Rechecked 7808dc6. The fixed-size nonce framing resolves the earlier long-backtick regression: both reported cases now fit in source and fresh-wheel runs (3,478 / 4,096 and 75,478 / 96,000 estimated tokens), and the simulated budget checks complete without failed batches.
One Docker smoke-test permission issue still needs a follow-up: the report readers use a different UID from the scans. The inline comment describes the private-checkout case.
Validation: selected Python tests passed (122 at base, 128 at HEAD, each in source and wheel installs; one macOS-specific skip per run), OpenCode/Pi checks passed, and 48 offline scans across 12 samples retained normalized parity and their existing partial-analysis states. Production Docker CI also passed on an identical HEAD tree. The permission finding is based on the source; the offline checks do not establish live-provider prompt-injection resistance.
The scans already run with --user "$(id -u):$(id -g)", but validate_json_report and assert_report_contains_component used the image's default uid 10001. On Linux, with a private (0700) checkout owned by another uid, the scan could write its report but the reader could not traverse /scan, so make docker-smoke failed during validation. Signed-off-by: oherbawi <oherbawi@gmail.com>
rng1995
left a comment
There was a problem hiding this comment.
[SkillSpector Review]
@oherbawi thank you for this careful hardening pass, especially the content-hash nonce framing and the quick turnaround on both review threads.
Value and readiness
Each change is sound on its own:
- A file cannot embed the 64-bit nonce of its own content, so it cannot forge its closing marker.
- The meta framing is now fixed-size, so
get_batches()budgets it. _is_git_urlnow uses the same suffix rule as_validate_url_host.- Both integrations refuse dash-led targets.
- The image builds from
uv.lockwith a digest-pinned uv and runs as uid 10001.
Two small items block merge, and the bundle raises scope questions for the PIC.
Material findings
- Blocker: the branch conflicts with current
mainintests/nodes/test_meta_analyzer.py, because #796 (7ee43713) changed the same imports and appended tests at the end of that file. This is a fork PR, so automation cannot update it. Please mergemainand keep both sets of tests. - Blocker (test),
extensions/skillspector.ts:175-178: the Pi extension's new dash-led refusal has no test; only the OpenCode path is covered (tests/opencode/skillspector_scan_lib.test.ts).tests/unit/test_pi_extension.mjsalready mockspi.exec. Please add a case asserting that a--output=...target throws beforeexecruns and that./-skillpasses.
PIC tradeoffs
- Scope: this PR bundles six unrelated changes. The Docker one breaks every existing
docker run -v "$PWD:/scan" … --output …invocation: writing into the mount now needs--user, and private (0700) trees cannot be read without it. I'd recommend moving it to its own PR with a CHANGELOG/release note, so it can be announced and reverted on its own. At minimum, add that note here. - Detection behavior:
src/skillspector/llm_analyzer_base.py:886-909does more than add framing. It also drops "It is far better to miss an edge case than to report a false positive" and tells every semantic analyzer to report steering attempts under a matchingrule_id. This may raise the semantic false-positive rate, and the offline parity runs cited in review do not exercise the LLM. The PIC should either accept this or ask for a live-providerscripts/compare_scan_accuracy.pycomparison first.
Verification and gaps
- Read the full diff at
fdaabc00. The new smoke-test commit does pass--user "${HOST_USER}"to both report readers, as @yashrajp22 asked. - Traced
build_promptand theget_batchesbudget, thecontent_noncecall sites, the PiapproveScanInputs→buildScanArgsflow, and the Dockerfile. - Verified that
ghcr.io/astral-sh/uv:0.10.10resolves to index digestsha256:cbe0a44b…and that codeql-action9f759ee6…isv3.38.3. The image still installs only core dependencies, as before. - Both review threads are resolved, but @yashrajp22's CHANGES_REQUESTED still needs their re-review.
- No CI has run on
fdaabc00yet (7808dc65was green). I did not build the image or run tests.
|
|
||
| function buildScanArgs(params: SkillSpectorScanParams, output?: string): string[] { | ||
| // The CLI would parse a dash-led target as one of its own options. | ||
| if (params.target.startsWith("-")) { |
There was a problem hiding this comment.
Blocker (test): this refusal has no Pi-extension test; only the OpenCode path is covered. tests/unit/test_pi_extension.mjs already mocks pi.exec. Please add a case where a --output=/tmp/x target throws before exec is called, and ./-skill passes through unchanged.
Resolve the conflict in tests/nodes/test_meta_analyzer.py by keeping upstream's pytest import and new tests and this branch's content_nonce framing tests. Keep both the hashlib and json imports in llm_analyzer_base.py, and the multi-line docker run form in the README. Signed-off-by: oherbawi <oherbawi@gmail.com>
Add a Pi extension case alongside the OpenCode one: a --output=/tmp/x target is rejected before pi.exec runs, and ./-skill reaches the CLI as the workspace path of the -skill file. Signed-off-by: oherbawi <oherbawi@gmail.com>
|
@rng1995 @yashrajp22 thanks again for the reviews. Both blockers are addressed:
One note on the test: through the tool, Locally, |
rng1995
left a comment
There was a problem hiding this comment.
[SkillSpector Review]
Decision — Request changes
Value and readiness — The branch update resolves the previous conflict, the Pi tool now has an end-to-end check that a dash-led target never reaches exec, and both Docker report readers use the scan UID. The nonce framing for source text is constant-size. The prompt still exposes other skill-controlled strings outside that framing.
Material findings
- [P1]
src/skillspector/nodes/meta_analyzer.py:178–189,214–227,433–443inserts manifest-controlled text before the framed file. A different file's manifest can reproduce that file's deterministic nonce because the hash covers only its label and content. The prompt can therefore contain duplicate exact-nonce markers or forged headings before the genuine block. Both prompt builders also interpolate the raw batch path in a heading (src/skillspector/llm_analyzer_base.py:941–950,1486,1893–1897). A POSIX path can contain a newline and inject another heading outside the markers. Encode path and manifest fields as labeled untrusted data, frame skill-derived finding snippets as data, and derive the delimiter from all rendered untrusted fields or verify that those fields cannot contain the selected markers. Add prompt-structure regressions with multiline metadata and filenames.
PIC tradeoffs — The Docker change makes a host UID override necessary for private bind mounts and report writes. The README documents the updated commands. The prompt change also asks the semantic analyzers to classify steering under an applicable rule. Hosted tests establish prompt shape and scanner behavior exercised by the suite. Evidence about live-provider findings and false positives remains open.
Verification and gaps — Reviewed the complete diff, both prompt assembly and budgeting paths, integration argument flow, and Docker changes against the prior review. The Pi test exercises the tool boundary, where input resolution rejects the target before argument construction. Exact-head hosted lint, unit, integration-tool, DCO, and Docker smoke checks succeeded. A human Changes Requested review remains active. Local review was static. Hosted CI
Reviewed head: af508e970982846c78a6b3800b777eec45aeb754
Summary
Changes from an external review, one commit each:
fix(llm): frame scanned files as untrusted, nonce-delimited input.BASE_ANALYSIS_PROMPTembedded the scanned skill with no adversarial framing, followed by a precision-over-recall instruction, in the semantic analyzers whose SSD findings have no deterministic backstop. Adds an explicit untrusted-input section and wraps the file in markers carrying a nonce from the newcontent_nonce(). The nonce is a SHA-256 of the file's own content, so prompts stay deterministic for caching and seeded runs, while a file cannot contain its own closing marker.meta_analyzerandmcp_tool_poisoningoverridebuild_promptand are unaffected.fix(meta-analyzer): frame scanned content with fixed-size nonce markers.PER_FILE_ANALYSIS_PROMPTused a fixed triple-backtick fence that a file could close and continue in its own words. It now wraps the file in the sameuntrusted-filemarkers asBASE_ANALYSIS_PROMPT, carryingcontent_nonce, so a file cannot forge its closing marker. The markers are constant-size and sit in the template thatget_batches()already budgets. (An earlier revision sized a backtick fence to the content instead; review showed that let a long backtick run grow the prompt past the batch budget, so it was replaced.) Impact was limited to annotations, since the LLM cannot suppress deterministic findings.fix(input): exact or subdomain host match in_is_git_url. It testedallowed in host, sogithub.com.evil.examplewas classified as Git. Not exploitable, because_validate_url_hostenforces the strict allowlist before any clone, but it misclassified input.fix(integrations): refuse dash-led scan targets in the OpenCode tool and the Pi extension. Both pass the target as argv afterscan, so a target like--output=...was parsed as a CLI option.build(docker): install fromuv.lock, run as non-root. The builder ranpip install ., resolving the>=ranges afresh and ignoringuv.lock; the runtime ran as root. It now syncs with uv 0.10.10 (the release workflow's version, pinned by image digest) using--frozen, and runs as uid 10001 withHOME=/tmp. Behavior change: writing reports into a bind mount now needs--user "$(id -u):$(id -g)". The README examples andtests/docker/smoke.shpass it, and the smoke test fails if the image ever defaults to root. Also pins the commented-outcodeql-action/upload-sarifstep by commit, so uncommenting it is safe.Testing
On Windows 11, Python 3.14, dependencies from
uv.lock(uv sync --frozen --extra dev), default markers (no live-LLM tests):EPERMwithout Windows symlink privilege), Windows upper-casing environment variable names, and release/accuracy tooling. The identical set of 28 fails onmainwhen run withmain's ownsrcon the same machine.main(the new test); the 5 failing on both are Windows symlink tests.Not tested: the Docker image build and
smoke.sh(no Docker on this machine). CI's docker job covers both.