[khushalsonawat]

Description

These are the agent scan's three follow-ups from round 2 of the review of #251.

• TH-9449. The walk used to stop at a file count. Now it stops at what it holds: one 1.5 GB budget, charged as
  things are kept, rounded to the allocator's blocks. It covers everything the walk and its join keep. The file limit
  now counts only files the join works on (40,000), so a repository of many small files isn't cut early.
• TH-9450. The walk and the agent description read through one capped reader, with the same stop order as before.
• TH-9451. Tests for the five branches no test held, and the masked-trace comments fixed.

Type of Change

• Bug fix
• Refactor
• Tests

Changes

• scan_walk.py, the budget (_MAX_HELD = 1,536 MiB).
  • It charges facts, keys, import lists and the join's share for each file.
  • It also charges templates, graphs, prompt paths, project lookups with nearest-project links, module sets, and the
    folders os.walk holds to enter.
  • Past the budget, the walk stops at limit and keeps what it found. A cap reached while joining ends the same way.
• scan_walk.py, the join.
  • _roots is gone. Imports are looked for through the nearest-project links.
  • The join's two loops and each row's pass over the files are linear. The row pass uses bisect ranges and checks
    the clock.
  • Rows keep at most 50 setting names, the most the answer file writes, so the file is unchanged.
    ENV_NAMES_KEPT now lives in scan_walk.
  • The hazard look gives up past 64 Mi characters of listed paths.
• scan_walk.py, one _CappedReader for the walk and describe_entry.
• scan_ai.py, scan_folders.py, scan_walk.py: comments say only the platform's secret values are masked.
• Tests:
  • 16 memory shapes, where the charge is never below what is really held, under tracemalloc and with an independent
    model of the allocator;
  • a check per kept structure, and the stop tests;
  • the join's equivalence;
  • the reader's order;
  • the TH-9451 branches.

Testing

• Run on this branch merged into the target's head (c1685ecb), on Python 3.12.14 and 3.12.3, one file at a time,
  all passed on both:

  test file	passed
  test_scan_walk	147
  test_scan_folders	8
  test_scan_document	17
  test_scan_ai	59
  test_agent_scan	63
  test_authoring_entrypoint_find_agents	5
  test_reading_entry	61
  test_framework_detection	262
  test_harness_architecture	55

• Planted breaks: 46, one per guard, each caught by a test and restored.

• Walk only, no model, base (22bcbb87) against this branch at b1633742: the agent learning kit (20 rows) and a
  monorepo checkout (43 rows, stopped=limit).

  • The rows are identical on both.
  • Peak memory is about the same, and varies from run to run. No saving is claimed. These trees are far below the
    budget.

• Review. A plan review took three rounds, and round 3 cleared. A change review of 22bcbb87..b1633742 by two
  cold reviewers cleared in round 1, with 0 High, 4 Medium and 18 Low between them. One reviewer's differential over
  1,400 generated repositories found no row that differs from the base.

What wasn't reviewed

• Round 3's plan fixes. They are in the code the change review read.

• 811a918c, which fixes the change review's Mediums and Lows without another round.

  • In src/, that's walk_repo mapping a cap raised while joining to limit, a row's pass skipping the name merge
    for files with no settings, and two comments.
  • The rest is tests.

• f82106fc is titled test(harness), but it also removes one redundant 50-name slice in describe_entry. The
  slice changed nothing, because _env_names already returns at most 50.

• bc3dd187, after CI: test_a_hazard_look_gives_up_past_its_listed_paths assumed an order in which folders are listed, and failed on Linux. os.walk doesn't sort, so the look found both agent.py files first. The long folders now sit at the root, so the root's own listing passes the bound on any file system. This is test-only.

  The other failure in that first CI run, test_post_terminal_wire_block_is_bounded_by_the_remaining_flush_window, is a timing test in the hosted entrypoint, which this PR doesn't touch. It passed on the second CI run and three times here.

Known, not changed here

The folder check's child process (framework_detection.py) has no memory bound. Its limit counts only Python files, so
a crafted tree of many folders with no Python files could grow it after the walk has stopped. That's pre-existing and
outside this change. It's filed as TH-9511.

Related Issues

Refs: TH-9449, TH-9450, TH-9451

[cdileep23]

Please check the comments.