[khushalsonawat]

Description

The framework check (folder_findings) walks an agent's folder with os.walk. Its limits count Python files, bytes
read and parsing work. A deep tree with many sibling folders and no Python reaches none of them. So os.walk's list of
folders still to visit grew without end, and in the agent scan's 4 GB sandbox it could take the room the scan's own
walk keeps free (#273).

The walk now keeps a running total of every folder it keeps to walk into, at what that folder's full path costs as a
str in that list. Past 64 MiB it stops as at its other limits, and the word is unknown. A crafted tree then ends the
scan with a finished row, not scan_crashed. That holds together with #273, which bounds the scan's own walk.

Type of Change

• Bug fix
• Tests

Changes

• framework_detection.py:
  • _MAX_FOLDER_BYTES (64 MiB) and _TOP_FLOOR (184);
  • _folder_bytes(top, below): width × (max(len(top), 184) + 1 + len(below)) + 120, where width is 1 if the whole
    path is ASCII and 4 otherwise;
  • in folder_findings, a running total that is never reduced. Past the bound, over_limit is set before that
    folder's files are read.
• What the count promises:
  • It's never below what os.walk holds, under pymalloc and glibc.
  • While the checked folder's path is ASCII and at most 184 characters, the scan, a build and a reading count the same
    tree the same, so they stop at the same folder.
• Tests: 14 new cases in test_framework_detection.py. Every tree is sized from fixed numbers, and every expected
  stop is worked out by hand.
  • One test reads os.walk's own list from its generator while it walks, and holds it to the bound under short, long,
    wide-character, wide-top and deep paths.
  • Others hold the exact totals at the bound, below depth 1 and for an installed tree.

Testing

• One file at a time, on Python 3.12.14 and 3.12.3:

  test file	passed
  test_framework_detection	276
  test_scan_folders	7
  test_scan_walk	92
  test_agent_scan	63
  test_reading_entry	61
  test_scenario_source_provision_agent	24
  test_hosted_entrypoint	139
  test_harness_architecture	55

  On 3.12.3, 6 test_hosted_entrypoint tests fail with No module named 'livekit'. That's the local venv, and the same
  6 fail on the base, cf3ebb44.

• Planted breaks: 12, one per guard, each caught and restored.

• Real trees: the most the total reaches is 3.15 MiB, on a monorepo checkout. That's about 5% of the bound.

• A crafted tree at the real 64 MiB (60 levels × 2,000 folders) stops at level 54.

• End to end, through the scan with its real folder-check child, on this branch merged with fix(harness): bound the scan walk by what it holds, merge its two readers, test its untested branches (TH-9449, TH-9450, TH-9451) #273:

  • with the bound, a finished row with unknown;
  • with fix(harness): bound the scan walk by what it holds, merge its two readers, test its untested branches (TH-9449, TH-9450, TH-9451) #273 alone, the same tree gives crewai, which shows the bound tripped;
  • with the scan's own walk budget lowered so it stops at its limit (truncated: true), still a finished row with
    unknown.

  Nothing is killed in any run. The ticket's full 700-level tree can't be built under macOS's path limit, so "Done
  when" on that tree rests on the sandbox memory sum.

• Review, on Lane N:

  • The design review cleared in round 1.
  • The plan review cleared in round 2. Round 1 found a count that ignored the checked folder's path too low.
  • The change review by two cold reviewers cleared in round 1, with 0 High, 2 Medium (the same one, found by both) and
    13 Low.

What wasn't reviewed

• ae5364d2 fixes the change review's findings without another round. It's tests plus one docstring:
  • a non-ASCII top;
  • exact totals below depth 1 and for an installed tree;
  • cases resized so they can fail on a Linux tmp_path.
• The plan review's round 2 fixes are in the code the change review read.

Known, not changed here

These have their own tickets:

• TH-9516: one huge folder's listing.
• TH-9517: a build's other walks over the source.
• TH-9525: the scan's first look for a file.

Related Issues

Refs: TH-9511

[cdileep23]

Please check the comments.