Skip to content

fix(pe3): stop flagging credential words in Python docstrings and comments - #1145

Open
Varshith-Kali wants to merge 3 commits into
NVIDIA:mainfrom
Varshith-Kali:fix-1144-pe3-python-docstrings-comments
Open

Varshith-Kali wants to merge 3 commits into
NVIDIA:mainfrom
Varshith-Kali:fix-1144-pe3-python-docstrings-comments

Conversation

@Varshith-Kali

Copy link
Copy Markdown
Contributor

Fixes #1144.

Problem

PE3 raised HIGH findings on credential words mentioned in Python docstrings, comments, and message strings that read nothing. The demo skill from the issue reports six HIGH findings on main (a09091e), all from a module docstring, two guard docstrings, a comment, and a message string. Mentioning a credential path is not credential access, and the two guards are the clearest cases: rejecting /etc/passwd and refusing a symlink to ~/.ssh/id_rsa is what PE3 wants skills to do.

Fix

In the PE3 loop, Python matches are now checked against proven string and comment spans before they become findings:

  • Matches inside a docstring, a bare string statement, or a comment are skipped. These never execute, so they cannot be credential access. Docstring spans resolve through the shared Python AST parse, using the same byte-offset conversion as the existing setting-literal exemption. Comment and string spans come from the shared python_literal_spans tokenizer proof.
  • A prose phrase (access the credentials and kin) inside any other string literal is skipped. It is a message, not an instruction to read a credential file.
  • Code patterns still fire on the path itself, so a literal that reaches a file API stays visible: open("~/.ssh/id_rsa"), Path("credentials.json"), config_dir() / "credentials.json", and a path assigned to a variable.
  • Unproven source (a windowed fragment, a malformed module, a polyglot shebang) keeps every finding.

Tests

  • New tests/nodes/analyzers/test_pe3_python_docstring_comment.py: 19 tests, including the issue's demo skill verbatim (0 PE3 findings, was 6 HIGH) plus docstring, comment, message-string, f-string, non-ASCII, and CRLF cases, and stay-visible cases for real file access.
  • Two existing tests updated to the new behavior, keeping their original intent: the credential-type label exemption stays markdown-only (now pinned on a YAML file), and ignore-list fences still buy no exemption in a script (now pinned on a fenced string literal instead of a docstring).
  • Full tests/nodes/analyzers suite: 16065 passed with only those 2 tests failing before the test updates; the affected PE3 and static-pattern suites re-ran green after (1616 passed, 0 failed).
  • End to end: skillspector scan of the demo skill reports 0 issues (was 6 PE3 HIGH). A control skill with open("~/.ssh/id_rsa") and config_dir() / "credentials.json" still reports both as PE3 HIGH.

Notes

The issue also asks whether a skill can declare its own credential file (for example in SKILL.md permissions) so PE3 can tell it apart from reading someone else's credentials. That is a separate design question and is left out of this fix.

…ments

PE3 raised HIGH findings on credential words mentioned in Python
docstrings, comments, and message strings that read nothing: the
demo skill in NVIDIA#1144 reported six HIGH findings, all from a module
docstring, two guard docstrings, a comment, and a message string.
Mentioning a credential path is not credential access, and the two
guards are the clearest cases: rejecting /etc/passwd and refusing a
symlink to ~/.ssh/id_rsa is what PE3 wants skills to do.

PE3 now skips matches inside Python docstrings, bare string
statements, and comments, and skips prose phrases (access the
credentials and kin) inside any other string literal. Code patterns
still fire on the path itself, so a literal that reaches a file API
stays visible, as does a path assigned to a variable. Unproven
source (windowed fragments, malformed modules) keeps every finding.

Fixes NVIDIA#1144.

Signed-off-by: Varshith Puli <pulivarshit@gmail.com>
The NVIDIA#1144 fix deliberately silences PE3 for credential paths named
only in Python comments, so test_pe3_surrounding_warning_cannot_hide_
sensitive_path now fails: its fixture is two comment lines and nothing
reads the path. That expectation is outdated, not the fix: the same
warning comment followed by code that opens the path still flags PE3,
and the new test_pe3_python_docstring_comment suite already pins the
comment-only case as zero findings.

Split the test in two. The comment-only case now asserts no PE3
finding, and the original intent (a surrounding warning cannot hide a
sensitive path) is kept under its original name with a code line that
actually reads ~/.ssh/id_rsa after the warning comment.

Signed-off-by: Varshith Puli <pulivarshit@gmail.com>

@rng1995 rng1995 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[SkillSpector Review]

Decision — Request changes.

Value and readiness — Suppressing documented path mentions in genuine Python comments and docstrings addresses the reported PE3 false positives. The AST suppression must share the same source-ownership requirement as comment suppression.

Material findings

  • static_patterns_privilege_escalation.py:1250–1263 calls _python_inert_string_spans even when python_literal_spans returns None. The fallback calls ast.parse, which treats the shebang as a comment. For a .py artifact with a shell shebang and a shell/Python polyglot body, a shell operation can lie inside what Python sees as a triple-quoted string, so the AST branch suppresses the credential-path match. The runner can still pass file_type="python" from the extension. Gate every suppression path on successful ownership proof for the complete artifact, and add a runner-level shell-shebang polyglot regression alongside the genuine docstring control. Suppression · Interpreter check

PIC tradeoffs — Keeping PE3 findings on unproven code preserves coverage of real executable surfaces. The tokenizer proof still permits the precision improvement for genuine Python without requiring credential-flow analysis.

Verification and gaps — Reviewed the complete diff, the static runner's file-type selection, tokenizer/AST helpers, and the PE3 fixtures. All six recorded checks succeeded for this head. The added malformed-Python test exercises a parse failure, while the polyglot case requires syntactically valid Python that fails the interpreter-ownership check. Static review only.

Reviewed head: cae25c57a61f5b5f27673dfe7bdea64c2c9ae35a

Move the _python_inert_string_spans call inside the branch where
python_literal_spans is not None, so a .py artifact with a shell
shebang and shell/Python polyglot content no longer gets its shell
lines suppressed as inert strings. The fallback ast.parse treats
the shebang as a comment, so unproven source wrongly hid real
credential-path findings. Unproven source now keeps every finding,
matching the comment suppression path and the class docstring.

Add regression tests: a shell-shebang polyglot keeps its PE3
finding, and the same body under a Python shebang stays
suppressed.

Signed-off-by: Varshith Puli <pulivarshit@gmail.com>
@Varshith-Kali

Copy link
Copy Markdown
Contributor Author

Thanks, good catch, this was real. Verified on the previous head: for a .py artifact with a shell shebang and shell/Python polyglot content, python_literal_spans returns None (ownership unproven) but _python_inert_string_spans still returned a span covering the shell line, because its fallback ast.parse treats the shebang as a comment. The PE3 credential-path finding was wrongly suppressed.

Fixed in 05661c0: the _python_inert_string_spans call in _PythonPe3Prose.suppresses (static_patterns_privilege_escalation.py) now runs only inside the branch where python_literal_spans is not None, so inert-string suppression shares the same source-ownership requirement as comment suppression. Unproven source keeps every finding, matching the class docstring.

Added two regression tests in tests/nodes/analyzers/test_pe3_python_docstring_comment.py: a shell-shebang polyglot keeps its PE3 finding (fails without the fix, verified), and the same body under a Python shebang stays suppressed. Test file: 21 passed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

PE3 flags credential words in Python docstrings, comments and messages that access nothing

2 participants