Repository navigation
fix(skill-inspector): resolve the installed artefact, not the default-branch clone (#1112) - #1119
Conversation
…-branch clone Step 1 resolved a URL target by cloning the default branch and step 2 scanned the whole tree, so a verdict could describe a different version, a superset, a different install channel, or skill text generated at run time. Step 1 now names the install channel (git tag or commit, publish branch, package registry tarball, release binary) and fetches from that channel at the exact pinned version, for example git clone --branch <tag> or npm pack <pkg>@<version>, and records the resolved version or commit in the report Source line. Runtime-generated skill text is reviewed through its generator, above all the trigger description, and code executed outside the installed path is listed and either reviewed or stated as not covered by the verdict. Step 2 now scans an installed subset as its own target and greps its bodies for directives to run project-local files (./*.sh, NEVER SKIP, always run). Adds a regression test pinning the new step 1 and step 2 wording. Fixes NVIDIA#1112 Signed-off-by: Varshith Puli <pulivarshit@gmail.com>
…n tests The CI lint job failed on ruff format --check for the SKILL_PATH assignment. Collapse it to one line. No behavior change; the 5 regression tests still pass. Signed-off-by: Varshith Puli <pulivarshit@gmail.com>
|
Fixed the CI lint failure in 2b4169b: ruff format --check flagged the SKILL_PATH assignment in the new test file, collapsed to one line. No behavior change, the 5 regression tests still pass and ruff check is clean. |
rng1995
left a comment
There was a problem hiding this comment.
[SkillSpector Review]
Decision: Changes Requested
Value and readiness
Resolving a pinned install channel and scanning the installed subset directly address #1112. The workflow still needs a safe, explicit outcome when the installed instructions are generated at runtime and the reviewer cannot inspect their resulting text.
Material findings
[P1] skills/skill-inspector/SKILL.md:40 directs the reviewer to inspect rendered instructions, but gives no path for a generator whose output depends on runtime inputs. The operating rules forbid running target code. For example, if the trigger description is assembled on first run from selected configuration, inspection can reach the generator but stop before the final description. The verdict rubric at lines 95–101 can still lead to APPROVE based on the visible source. Specify static reconstruction or a supplied rendered artifact and record the missing coverage in the verdict if the final text cannot be obtained within the operating rules. Cover that fallback in the wording regression.
PIC tradeoffs
The exact-channel workflow is useful on its own. A qualified verdict remains available when runtime-generated content is unavailable, and the reviewer can identify the specific artifact needed for full coverage.
Verification and gaps
Reviewed the complete two-file diff, exact-head operating rules and verdict rubric, five wording-test assertions, linked issue and lint follow-up. Exact-head lint, unit tests, DCO, change detection and OpenCode tests succeeded. Docker smoke was skipped. The new tests assert the required generated-text instruction but give no example of unavailable output. I did not execute the skill, target installers or contributor tests.
Reviewed head: 2b4169b14e4953480c1cf1700e83a9c3181aed67.
…ot obtain as missing coverage Step 1 told the reviewer to inspect rendered instructions from the generator but gave no compliant path when that text depends on runtime inputs: the operating rules forbid running target code, so a description assembled on first run from selected configuration could never be seen, and the verdict rubric still allowed APPROVE on the visible source alone. Step 1 now says not to run the generator, and to reconstruct the rendered text statically by listing every runtime input it branches on and reviewing each resulting variant, or to review a rendered artifact produced outside the review. If neither path yields the final text, the report must record the missing coverage and the verdict is capped at CAUTION. The APPROVE rubric now requires runtime-generated instruction text to have been reconstructed statically or reviewed as a supplied rendered artifact. Adds two wording regression tests pinning the fallback and the rubric gate. Signed-off-by: Varshith Puli <pulivarshit@gmail.com>
|
Thanks for the catch, you are right. Step 1 told the reviewer to inspect the rendered text but gave no compliant way to get it when the output depends on runtime inputs, since the operating rules forbid running target code, and the rubric still allowed APPROVE on the visible source alone. Fixed in bca599f:
Docs and test wording only, no behavior change. |
Fixes #1112.
Gap
The skill-inspector companion skill resolved a URL target by cloning or downloading the default branch and then scanned the whole tree. A verdict could therefore describe a different version than the one installed, a superset of what gets installed, a different install channel, or skill text that only exists after the first run. The issue lists five observed instances of exactly that.
Change
skills/skill-inspector/SKILL.md:git clone --branch <tag>ornpm pack <pkg>@<version>, never a default-branch clone. If no version is pinned, the reviewer asks which tag, commit, or release will be installed before reviewing. The resolved version or commit goes in the report Source line, and line references must point at that artefact.description../*.sh, "NEVER SKIP", "always run").No production code changes; the fix is wording in the companion skill.
Tests
tests/test_skill_inspector_target_resolution.pypins the new step 1 and step 2 wording (5 tests, all passing; the tests fail against the previous wording).tests/test_skill_inspector_target_resolution.py,tests/test_structured_skill.py,tests/test_multi_skill.py,tests/test_documentation_image_references.py,tests/nodes/analyzers/test_cli_companion_documentation.py,tests/nodes/analyzers/test_documentation_reconstruction.py(634 passed).Signed-off-by: Varshith Puli pulivarshit@gmail.com