[khushalsonawat]

Description

Answers Dileep's nine P3 comments on #273, #275, #277 and #285, which merged before the fixes were ready. Everything is in the scan's bounds and its tests.

Type of Change

• Bug fix
• Tests

Changes, comment by comment

#273 (scan walk memory budget)

1. scan_walk.py:932, folders charged twice. The side pass now charges only the folders the first pass never entered: side folders, and folders below one. Each folder is charged once (883ace6a). On 50,200 empty folders the charge halves, from 14.46 MB to 7.23 MB. Tested by test_each_folder_is_charged_once_across_the_two_passes, which includes dot folders.

2. :215, sizes from the wrong Python.

   • The comment now says the sizes are 3.12's, the hosted kit venv's (_SIZES_MEASURED_ON), and that the str header is 3.11's, so it runs high on 3.12 and 3.13.
   • test_the_estimates_sizes_were_measured_on_the_hosted_kits_python fails if the Dockerfile's one line that makes /opt/alk-venv names another Python.
   • test_the_estimate_rounds_each_object_up_to_its_block already fails on 3.14, where the tuple is larger.
   • test_scan_walk.py is in sdk-smoke.yml's list, so CI runs it.

3. :1476, a budget stop while joining logged "edge budget". It now has its own except, which logs "passed its memory budget" and lets the facts go before the empty result; the rows found are dropped there.

4. :228, _JOIN_PER_FILE untested. The constant stays at 64. Two fan-out shapes, each about 22,000 files whose leaves import one shared module, check the join's real rise against its per-file charges without _JOIN_BASE:

   • with one entry, the import table has just grown;
   • with two entries, the second search's seen set grows while that table is full.

   The two-entry shape needs about 59.5 B a file, so the test passes at 64 and fails at 56, 48, 32 and 0. A first attempt dropped the constant, and the change review showed that under-charged the join.

5. :1579, memory outside the budget. Recorded under "Known limits" below.

#275
6. test_framework_detection.py:1759, the watcher read nothing on 3.11. It now reads os.walk's stack by key, so a missing one fails loudly. Both stack tests are skipped below 3.12, with the reason, because before 3.12 os.walk recurses and has no list to watch.

#277
7. agent_scan.py:182, no clock inside one folder.

• The source look reads the clock every 4,096 entries (_LOOK_CLOCK_EVERY). In a 250,000-entry folder past the deadline, the overrun drops from 1.1 s to 18 ms for dangling links, and from 95 ms to 2 ms for empty folders.
• New tests cover a huge folder of empty folders (with the exact count held when the clock stops it), a huge folder of links, and a folder past the 64 MiB bound.
• One behaviour change: a source whose own folder holds 4,096 or more entries and no regular file, past the deadline, now ends scan_timed_out (return 0) rather than scan_source_missing (return 1).

8. agent_scan.py:150, a cost with no floor. As Dileep allowed, this is left as a follow-up: one shared folder-cost helper with fix(harness): bound the framework check's walk by the folders it lists (TH-9511) #275's floor, for fix(harness): bound the scan walk by what it holds, merge its two readers, test its untested branches (TH-9449, TH-9450, TH-9451) #273's hazard look, fix(harness): bound the framework check's walk by the folders it lists (TH-9511) #275 and this look. Recorded in "Known limits".

#285
9. scan_walk.py:1547, files.sort() changed nothing. Dropped; the comment says only the folders need an order.

Contract: scan-parts.md 1.3m (#273's folder charge, and the per-file join charge named) and 1.3n (#277's clock).

Testing

All runs on Python 3.12.3, the hosted image's minor version.

• Seven scan test files: test_scan_walk, test_agent_scan, test_framework_detection, test_scan_ai, test_scan_folders, test_scan_document and test_authoring_entrypoint_find_agents. 605 passed.
• Planted bugs, each run on its test:
  • all caught: the per-fix plants, both rounds' reviewer plants for fix(harness): bound the scan walk by what it holds, merge its two readers, test its untested branches (TH-9449, TH-9450, TH-9451) #273, and all ten of fix(harness): bound the scan's first look for a file (TH-9525) #277's reviewer plants;
  • by design, re-adding files.sort() to _hazard passes, since the order can't matter.
• Reviews:
  • fix(harness): bound the scan walk by what it holds, merge its two readers, test its untested branches (TH-9449, TH-9450, TH-9451) #273's part: two rounds with two reviewers each. Round 1 was not clear: dropping _JOIN_PER_FILE under-charged the join, so the constant is kept. Round 2 cleared.
  • fix(harness): bound the scan's first look for a file (TH-9525) #277's part: two reviewers, cleared.
  • fix(harness): bound the framework check's walk by the folders it lists (TH-9511) #275's and fix(harness): take the build-hazard look's folders in name order (TH-9514) #285's parts: one reviewer, cleared.
• Not reviewed: each round's Lows, fixed afterwards: round 2's two-entry shape and venv-line test (edf7bea6), fix(harness): bound the scan's first look for a file (TH-9525) #277's added tests (c8883737), and the contract wording.

Known limits

• describe_entry's reader has no held budget. Only _DESCRIBE_BYTES bounds it: 700 files of 200 unique dotted imports each peaked at 349 MB in one description. This isn't a regression.
• The folder being listed isn't charged. os.walk's file and subfolder names for the current folder aren't charged; only the folders still to visit are. On 50,200 sibling folders the walk is charged 7.23 MB against a traced peak of 8.82 MB. This isn't a regression, and the contract already lists one folder's listing as not bounded.
• The three walks price a folder three ways. fix(harness): bound the scan walk by what it holds, merge its two readers, test its untested branches (TH-9449, TH-9450, TH-9451) #273's hazard look, fix(harness): bound the framework check's walk by the folders it lists (TH-9511) #275's framework check and fix(harness): bound the scan's first look for a file (TH-9525) #277's source look each price a folder still to visit differently, and fix(harness): bound the scan's first look for a file (TH-9525) #277's has no floor, so where a crafted tree flips between file and limit depends on the checkout root's length. One shared helper with fix(harness): bound the framework check's walk by the folders it lists (TH-9511) #275's floor is a follow-up (comment 8).

Related Issues

Refs: TH-9449, TH-9511, TH-9514, TH-9525

[cdileep23]

Please check the comments.