[khushalsonawat]

Description

Before anything else, the agent scan checks that the source holds a regular file. That look was an os.walk with no
bound. A crafted tree with no file in it, such as many empty folders or folders holding only links, grew the walk's
list of folders still to visit without end, and could kill the 4 GB sandbox before the scan wrote anything. Because
os.walk lists a folder whole first, even one folder of millions of empty folders was held before anything could stop
it.

The look is now the scan's own short loop. It's bounded by what its list holds, and by the scan's clock. Past either
with no file found, the scan ends the way a walk stopped with nothing found already ends.

Type of Change

• Bug fix
• Tests

For the platform (Dileep)

Two rows of the scan's results table are widened. No new rows and no new codes:

what happened	exit	files	result
The walk's limits, or the source look's (64 MiB of folders still to visit, before any regular file), ran out before finding any	0	agents.json, cost.json	error, scan_limit_reached
The clock ran out, in the walk or in the source look, before finding any	0	agents.json, cost.json	error, scan_timed_out

• Both are written exactly as a walk stopped with nothing found writes them: status: "error", no agents,
  truncated: false, and no model step.
• The reader on feat/TH-8812-agent-layer accepts both. ScanErrorSerializer takes any ^[a-z0-9_]+$ code, and
  scan_limit_reached is already in its tests.
• Its own SCAN_TIMED_OUT = "scan_timed_out" in agent_reading_sandbox.py is a failure code, for an abandoned
  worker. It's not this report's error.code. They share a string, but not a meaning.

Changes

• agent_scan.py: _holds_a_file becomes _look, which returns (word, folders, held).

  • It reads each folder once, entry by entry, depth first in name order.
  • It takes each real subfolder (not a link) onto its list, counted at what its path costs as a str, and takes it off
    when it moves into it.
  • A regular file (lstat + S_ISREG, today's test) wins wherever it is in a folder.
  • Past 64 MiB held, nothing more is taken on, and with no file in that folder, the result is limit.
  • The clock is read before each folder after the source's own, so an empty source is still "source missing".
  • A folder or entry that can't be read is passed over, as before.

• _scan: limit or clock logs one line of counts, then writes through _result with a walk stopped that way.

• Tests: 18 new in test_agent_scan.py.

  • The bound, flat and nested.
  • The list watched through a scandir wrapper, never past the bound under ASCII, long and wide-character paths.
  • The count held exact at every entry.
  • The file found first, a deep chain, links not followed, the clock and its first read, and unreadable folders and
    entries.
  • Name order, and the log line.

  test_the_clock_ends_in_a_file gains a root file, so it still drives the walk's clock.

Testing

• One file at a time, on Python 3.12.14 and 3.12.3, all passed:

  test file	passed
  test_agent_scan	81
  test_scan_walk	92
  test_scan_folders	7
  test_authoring_entrypoint_find_agents	5
  test_reading_entry	61
  test_harness_architecture	55

• Planted breaks: 17, one per guard, each caught and restored. Both change reviewers ran every one.

• Crafted trees, through the real scan, each under a 400-character path so it passes 64 MiB:

  tree	this change	before
  130,000 empty folders in one folder	scan_limit_reached, in under a second	scan_source_missing, after walking it all
  a nested broom	scan_limit_reached	scan_source_missing
  130,000 folders holding only links	scan_limit_reached	scan_source_missing

  At this size the old walk doesn't crash the laptop either, so this shows the bound and the outcome, not a prevented
  kill.

• Real repositories (the kit, and a monorepo checkout): the look finds a file in its first folder, as before.

• Review, on Lane N:

  • the design review cleared in round 1;
  • the plan review cleared in round 1;
  • the change review cleared in round 1, with 0 High, 4 Medium and 11 Low between the two reviewers. One found nothing
    wrong in the code.

What wasn't reviewed

d0b0fbf5 fixes the change review's findings without another round. It's tests, plus the stop line's level, now info.

Not changed here

A tree with a file found early and a huge empty subtree behind it gets past the look. On this branch, the scan's main
walk then holds that subtree's folders with no bound. #273 bounds that walk.

Related Issues

Refs: TH-9525

[cdileep23]

Please check the comments.