Skip to content

fix(harness): the save's import walk says when it leaves code unread, and reads more of it (TH-9543) - #283

Merged
khushalsonawat merged 3 commits into
feat/TH-8804-source-subfolderfrom
feat/TH-9543-walk-cuts
Nov 11, 2026
Merged

khushalsonawat merged 3 commits into
feat/TH-8804-source-subfolderfrom
feat/TH-9543-walk-cuts

Conversation

@khushalsonawat

@khushalsonawat khushalsonawat commented Nov 11, 2026 •

Copy link
Copy Markdown
Member

Review only commits after 73e9170. This PR carries #276 (TH-9490, 73e9170), which isn't in feat/TH-8804-source-subfolder yet.

Summary

When the contract is saved, the kit follows the agent's imports to the code it runs, so the browser check and instrumentation see shared code outside the agent's folder. That walk skipped some code without a word, and ran out of reading on ordinary agents.

  • The walk says when it leaves code unread (stopped, which the save logs as a WARNING and bundling records as truncated in credentials.json). This covers:
    • a .py holding import whose decoded reading isn't parsed: too big, past the total, or newer syntax;
    • a code file over the 1,000,000-byte read limit, or a link to one;
    • a folder link in the agent's folder that leads inside the repository;
    • an import name dropped for its length or depth, scripts included;
    • a link target past 128 parts;
    • a file in a codec decoded first, left undecoded once the walk's decode total ran out.
  • Nothing else changes stopped: a file with no import, a standard-library name, a file Python itself refuses, a data file a script imports, and a folder link out of the repository.
  • The save's walk gets its own 512 KB parse total. It was the 256 KB the other scans share, and those keep 256 KB.
  • Each reading of a file read two ways is charged as it's parsed. This replaces fix(harness): decode a Python file before searching it when its codec can spell a name (TH-9490) #276's charge-once rule, which let a crafted pair cost twice the total. The parse total never refuses a file the old 256 KB let through.

Why: an import Python runs could hide behind a file the walk couldn't parse, and every ordinary subfolder agent over 256 KB of Python was cut on every save.

TH-9543 lands before TH-9512, which re-points its walk-stop test when it merges this.

Known limits:

  • The four real subfolder runs measured stay cut: futureagi's ai_tools and agentic_eval, and the kit's src/fi/alk/harness and examples/frameworks. They reach 4-32 MB of Python and more than 5,000 import names. Reading them fully is TH-9445's (one capped subprocess and one deadline for the save).
  • At the save, any reached .py over 128 KB that holds an import now says the walk stopped: 53 tracked files in the futureagi backend, 18 in the kit's src/.
  • Honest agents up to about 500 KB of reached Python aren't stopped: measured, 305 and 401 KB are not; 517 KB is.
  • The walk reads a decoded file only from its bytes, the way Python imports it, not also as text, the way it runs a script. A file in a codec outside the raw-safe set whose decoded text holds a CR can read differently the two ways, so an import only the script reading sees isn't followed. TH-9520 reads both ways for the browser scan, instrumentation and tool evidence; the walk doesn't yet.

Pre-existing, for Khushal's read (TH-9445):

  • On 3.11, the combined worst crafted layout is over the 4 s line before this change: 6.4 s on feat/TH-8804-source-subfolder, 6.3-8.0 s with fix(harness): decode a Python file before searching it when its codec can spell a name (TH-9490) #276, and 7.9-8.3 s with this change. On 3.12 it's 3.6-4.4 s before, and 4.2-4.8 s with this change, under the 5 s line Khushal set for feat(harness): build an agent from a repo's subfolder #225.
  • The walk alone with its other limits spent (1,500-4,500 missing names through a link chain, plus files only the walk parses): 1.7-2.6 s on 3.12 (base 1.1-1.9 s), and 3.4-3.9 s on 3.11 (base 1.8-2.1 s; 3.75-3.90 s on base's unicode-escape shape). The Coordinator sent this to TH-9445 under the same 5 s line: an extension of Khushal's ruling.
  • Not measured on 3.13, the hosted image's Python: no environment here has 3.13 with the kit's dependencies, and installing one waits for Khushal.

Verification

  • uv run ruff check .: not run on the whole repo.
  • ruff check on the changed files passes. Focused pytest target, the seven changed-area files: 1532 passed on 3.12.14 and 3.12.3, and 1506 passed plus 26 skipped on 3.11 (25 of fix(harness): decode a Python file before searching it when its codec can spell a name (TH-9490) #276's are 3.12-only).
  • sdk-smoke.yml's list on 3.12.14: 3689 passed, 3 skipped, 2 deselected. The named list on 3.11: 4179 passed, 32 skipped and 16 failed, the same 16 that fail on feat/TH-8804-source-subfolder, none about this walk.
  • TypeScript build/test: not touched.
  • agent-learn release-proof: not run.

Checklist

  • Public API names stay under agent_learning.*, agent-learn, or @future-agi/agent-learning-kit
  • New user-facing workflow has docs or an example: no user-facing workflow
  • New release claim has executable gate coverage
  • No hosted-service dependency was added to release-gated examples
  • Security-sensitive output is redacted or justified

… can spell a name

A coding line can name a codec that spells ASCII letters in other bytes
(raw_unicode_escape, utf-7, the iso2022 codecs, hz, mac-arabic and
mac-farsi), so a file could import playwright or traceai_openai while its
raw bytes never held the name. The browser and instrumentation scans
searched those raw bytes, and the import walk decoded such files before
ending their lines, or read unicode-escape as UTF-8.

Every codec not proven safe to search as bytes is now decoded first, as
Python reads source (lines ended, a final line end, NUL cut), under the
scan's one 4 MiB decode total. The 81 safe codecs are proven by a sweep
test. A decoded-first file passes the gate when a needle is in its text,
or in its raw bytes when the text isn't ASCII, so no file base lets in is
stopped. The walk reads the same way, with its own total, and follows the
union of both readings for a unicode-escape file whose text holds a CR,
charged once.

Refs: TH-9490
Signed-off-by: khushalsonawat <khushalsonawat@gmail.com>
… and reads more of it (TH-9543)

- The walk sets stopped when it leaves code it reached unfollowed, beyond its limits:
  - a .py holding import whose decoded reading isn't parsed (too big, past the total, or no
    tree);
  - a code file over the read limit, or a link to one;
  - an import name dropped for its length or depth, scripts included;
  - a link target past 128 parts;
  - a file in a codec decoded first, left undecoded once the walk's decode total ran out.
- The save walk gets its own 512 KB parse total (the other scans keep 256 KB), and each reading
  of a file read two ways is charged as it's parsed.
- One flag replaces the walk's two.

Signed-off-by: khushalsonawat <khushalsonawat@gmail.com>
The change review cleared with nothing above Medium. Its findings:
- A folder link in the agent's folder that leads inside the repository says the walk stopped:
  the walk doesn't enter it, and the agent may run code there. One that leads out of the
  repository, such as a data folder, doesn't.
- Tests for a refused link target before a candidate's last folder, a reading sized with its line
  ends folded (decoded and UTF-8), a script with more names than the walk tries, and a decode
  total spent exactly.
- The save's warning says the walk stopped at its limits or left code unread; two test comments
  say which Python versions they describe.

Signed-off-by: khushalsonawat <khushalsonawat@gmail.com>
khushalsonawat added a commit that referenced this pull request Nov 11, 2026
Signed-off-by: khushalsonawat <khushalsonawat@gmail.com>

@cdileep23 cdileep23 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please check the comments.

Comment thread src/fi/alk/harness/credentials.py Resolved
Comment thread src/fi/alk/harness/credentials.py Resolved
@khushalsonawat
khushalsonawat merged commit 53a54d8 into feat/TH-8804-source-subfolder Nov 11, 2026
11 checks passed
@khushalsonawat
khushalsonawat deleted the feat/TH-9543-walk-cuts branch November 11, 2026 05:52
khushalsonawat added a commit that referenced this pull request Nov 11, 2026
…288) into TH-9491

- tests/test_harness_sandbox_server.py: both sides appended tests; kept both.
- test_contract_writer's allow-list gains #269's two contract.json copies:
  reading_result.document's replace_if_unchanged (the saved contract's own bytes
  plus the reading record) and seed_contract.use_seed's _write_whole (the
  platform's seed, as written).
- sandbox_server.py imports SecretResolutionError, which #284's resolver raises
  but the base never imports: seven sandbox-server tests fail with NameError on
  the base itself.

Signed-off-by: khushalsonawat <khushalsonawat@gmail.com>
khushalsonawat added a commit that referenced this pull request Nov 11, 2026
…total

Brings in #269, #276 (through #283), #283 and #288. Three hunks against
#288 (TH-9448) were resolved by hand, keeping both intents:
- a skipped file logs its type once at WARNING, and its path and trace
  at DEBUG (#288); it is also counted as the scan's `skipped` cut, and
  that count is in the cut line, so #288's separate "skipped N files"
  line is not repeated (TH-9512);
- #288's skip tests and its NUL-byte test now expect the cut line, which
  holds names and counts only.

Tests on 3.12.3: test_agent_type_refusal 720, test_browser_scan_cut 59,
test_contract_tools_subfolder 180, all passed.

Signed-off-by: khushalsonawat <khushalsonawat@gmail.com>
khushalsonawat added a commit that referenced this pull request Nov 11, 2026
- Comments in this branch's code and tests no longer name reviewers, rounds or people; the
  reason and the measurement stay.
- A uv workspace member past the 256 read for names is read when a requirement names its
  folder, until one declares that name, at most 32 such reads a walk; the walk still says it
  was cut. A pyproject.toml whose project isn't a table is passed over instead of raising.
- The save's import walk: a folder link back into the agent's folder isn't counted as code
  left unread, unless it leads into a part the walk skips (compared case-folded). Past 32 folder
  links it says the walk stopped without resolving more. A link to a code file, dangling or
  not, still counts: telling them apart would follow its chain uncharged.
- The walk's docstring says any reached .py holding import that doesn't parse counts as
  unread, whatever the reason.

TH-8804

Signed-off-by: khushalsonawat <khushalsonawat@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants