[khushalsonawat]

Stacked on feat/TH-8804-source-subfolder (#225's branch, at cf3ebb44). The diff shows only TH-9490.

Summary

The browser scan and the instrumentation scan look for imports by name, and the import walk decides which files outside the agent's folder both of them read. All three searched a Python file's raw bytes, or decoded it the wrong way, before looking for a name. A file whose coding line names a codec that spells ASCII letters in other bytes could therefore import playwright or traceai_openai unseen. Python runs each of these as the import:

• raw_unicode_escape (playwright);
• utf-7 (+AHA-laywright);
• the seven iso2022_* codecs (play ESC (B wright);
• hz (play~{~}wright, or play~ then a line end);
• mac-arabic and mac-farsi (traceai 0xDF openai, where 0xDF decodes to _).

After this PR, such a file is decoded first, as Python decodes it, and the name is looked for in what it decodes to.

Ticket: TH-9490. TH-9448 item 7 (import \x70laywright under unicode-escape) was already fixed by #225. This PR adds a no-final-LF case to its test.

How it works

• Which codecs. python_source._RAW_SAFE_CODECS lists the 81 codecs Python accepts on a coding line in which no byte form spells a name character, NUL, CR or LF except its own single byte. A test proves each one with a sweep of every byte form, and fails if a new Python adds a codec that isn't classified. Every other text codec is decoded first: today the 13 that can spell a name, plus 13 that Python refuses on a coding line. idna, punycode, undefined and non-text codecs keep today's handling.

• How it decodes (_decoded_first, as Python reads source):

  1. CRLF and CR become LF before decoding, so hz's ~ before a CRLF joins the lines.
  2. A final LF is added.
  3. The bytes are decoded; a decode error gives None, since Python refuses the file.
  4. The text is cut at its first NUL.

  A CR the decode makes is left in place, exactly as today's code hands text to the parser.

• One total. Every decoded-first file is charged to the scan's existing 4 MiB EscapeBudget, which unicode-escape already used. Past it, a file takes today's path. That caps the extra work at about 0.05 s per scan.

• The gate. A decoded-first file passes when a needle is in the decoded text, or in its raw bytes when that text isn't ASCII. Base gates on the raw bytes, so no file base lets in is stopped.

• The import walk (credentials.imported_by_agent) uses the same helper for .py files, with its own total, and skips a file Python refuses.

  • A unicode-escape file whose decoded text holds a CR is also read the way the walk read it before (as UTF-8, where \r is two characters), and the union of both readings' imports is followed. So the walk never reaches less than it did.
  • Both readings are charged to the parse total once, as the larger one.
  • At bundling, that second parse is bounded by the walk's 4 MiB escape total.

Known misses, pinned by tests

• Decoded CRs are read as the parser reads them, not as Python 3.12+'s tokenizer does: inside a comment, inside a string, and at a line's start. That is today's behaviour for every file today's code decodes. TH-9520.
• A cookie after 65 failed cookie places on the first line is read as a refused codec by _coding's cap, which stops a crafted-head stall. TH-9524, for TH-8804's owner.
• Padding past the parse totals (older than this PR). TH-9512.
• A fullwidth name with no raw needle (older than this PR, a literal match only).
• SyntaxWarnings print to stderr (older than this PR): the shared reader parses customer files without catching warnings, so each file with an invalid escape prints one line, quoting only the escape, outside logging.

Review

Each round used two fresh Opus reviewers.

• Design: r1 not clear (one High: a cost path when decoding every codec but UTF-8), then r2 clear.
• Plan: three rounds.
  • r1 and r2 were not clear, both times because of the rewrite of decoded CRs. The rewrite was then dropped, so this never reads worse than base.
  • r3 was clear, with one Medium, the raw gate, folded in.
• Change:
  • r1 was not clear, with one High: the walk lost a unicode-escape file with an escaped \r. That's fixed by the union.
  • r2 was clear: the 66 decoy cases from plan r3 read 66/66 on 3.12, 3.13 and 3.14, a 60k-case fuzz finds 0 regressions against base, and the plan's planted bugs were all caught.
• Fixed after change review r2, with no further round, as the rule allows for a round with nothing above Medium:
  • the union is charged once, as the larger reading, when both readings are within the limit;
  • four test rows: the union keeps the decoded reading, a latin-1 byte, the union being for unicode-escape only, and the single charge;
  • comment wording.

Verification

• ruff check and ruff format --check on the new test file; no new findings in the edited files against cf3ebb44; compileall.

• One test run, a file at a time:

  file	passed
  test_python_source_codecs (new)	156
  test_agent_type_refusal	652
  test_instrumentation_evidence	48
  test_source_tool_evidence	51
  test_contract_tools_subfolder	180
  test_scan_walk	92
  test_hosted_authoring_entrypoint	12
  test_harness_credentials	80
  test_bundle_author_v2	323
  test_harness_sandbox_server	35; 2 fail the same way on the base cf3ebb44 (HarnessJob validation in sandbox_server.py:541), so they're older than this change

• 49 bugs planted one at a time; 46 fail a named test. Three survive:

  • Always appending the final LF changes nothing.
  • Leaving the walk's two readings unfolded is not equivalent, though the change review said so. TH-9543's change review found that the walk sizes each reading against its 128 KB per-file limit after folding line ends. A file whose CR count straddles the limit is followed when folded and refused when not. fix(harness): decode a Python file before searching it when its codec can spell a name (TH-9490) #276 has no test for it; TH-9543 adds test_a_reading_is_sized_with_its_line_ends_folded[decoded, utf8], and each case fails only under its own mutant.

• TypeScript: not touched.

Checklist

• Public API names unchanged
• No new release claim
• No hosted-service dependency added
• Security-sensitive output is redacted or justified: no new log lines

[cdileep23]

Please check the comments.