[shehab-ali]

Rationale for this change

On a cache miss, DataFusion buffers the entire object-store file listing before rejecting entries that exceed the cache budget. Large directories therefore increase memory use and delay queries that only need a few files, such as LIMIT queries.

What changes are included in this PR?

Stop building the cache entry once its key and metadata exceed the cache budget. Return the buffered prefix followed by the remaining listing stream, so consumers can start processing or stop early. Only complete, successful listings are cached.

Add regression tests and a Criterion benchmark through the public ListingTableUrl API.

Related work: #25858 changes cache identity and object-store isolation; this PR addresses buffering on cache misses.

Local performance testing

Measured locally on November 6, 2026: main (a61683112) versus this change (6194d58e9), on an M3 Max with 64 GiB RAM/macOS, using matched release-nonlto builds and a 1 MiB listing cache:

Workload / metric	Before	After	Result
SQL SELECT id, value FROM events LIMIT 10	510.00 ms	29.01 ms	17.6× faster
Same SQL query, peak process RSS	70.81 MiB	42.72 MiB	39.7% lower
Metadata-only listing, 1 million files: peak process RSS	139.50 MiB	16.76 MiB	88.0% lower

The SQL workload uses 100,000 local Parquet files × 32 rows, four target partitions, and medians of seven alternating fresh-process pairs after warm-up. Statistics collection is disabled during table registration, then enabled before the query. Timings include planning and execution; RSS covers the whole process. These gains describe this early-stop workload, not default registration costs. The metadata-only memory benchmark uses ten fresh processes per version and performs no file-data I/O.

Full-scan SQL and the existing pruning, SQLIO, and narrow-schema controls showed no reliable gain; the additional local measurements are below.

Standard TPC-H SF10: all 22 queries ran in two reversed-order pairs with default runtime settings. Overall performance was inconclusive because of substantial timing drift. These single-file table paths bypass the changed listing-cache code.

Potential regression: Q12 was slower in both full-suite pairs. Four additional paired runs reproduced a 4.8% slowdown (102.41 → 107.31 ms). Its cause remains unconfirmed and needs investigation before claiming no regressions.

Additional local performance results and measurement method

Workload / metric	Before	After	Observed change
Metadata-only listing: 100,000 files, consume first 10	22.12 ms	1.81 ms	−91.8%
SQL full scan: SUM(id)	27.62 s	27.45 s	Median −0.6%; mean +0.7%
Same full scan, peak process RSS	1,733.22 MiB	1,742.33 MiB	+0.5%
parquet_pruning_setup_cache	9.186 ms	8.950 ms	−2.6%
sql_query_with_io: all-partitions EXPLAIN	0.1845 ms	0.1837 ms	−0.4%
wide_schema: narrow Q02	1.614 ms	1.586 ms	−1.7%
TPC-H SF10: sum of 22 averaged query means	7.056 s	6.336 s	−10.2%; inconclusive
TPC-H Q12: four follow-up pairs	102.41 ms	107.31 ms	+4.8%; all four pairs slower

Method: SQL time/RSS values are medians of seven alternating fresh-process pairs. Metadata listing time is the median of three Criterion run means (30 samples/run). Stock controls use medians of Criterion run means (10 samples/run): three pairs for pruning and SQLIO, seven for narrow Q02. Small control differences are within observed variation.

TPC-H used four full passes in baseline/fixed/fixed/baseline order, with default runtime settings and 10 samples/query/pass. The TPC-H total averages two run means per query, then sums all 22 queries; it excludes setup and warm-up, so it is not total suite wall time. The geometric-mean query-time change was −9.9%, but the same binaries varied by 13–28% between repeats and 14/22 queries reversed direction. These figures do not establish an overall speedup.

Q12 follow-up values average four Criterion run means per version. Those extra pairs were selected after Q12 slowed in both full-suite pairs, and remain separate from the full-suite totals. No measured samples were discarded from these comparisons. All results are from a shared workstation, without flushing filesystem caches; the standard suite did not measure process RSS or CPU consumption.

What is the testing strategy for this PR?

• Six regression tests cover bounded prefetch, ordered output, cancellation, cache-admission boundaries, oversized entries, and errors.
• Passed: 13,535 Rust tests, 546 SQL logic test files, and 116 CLI tests.
• Passed: formatting checks and cargo clippy --all-targets --all-features -- -D warnings.
• SQL benchmark results were validated; all 22 fixed TPC-H queries matched baseline outputs.

Are there any user-facing changes?

Oversized listings can be consumed or cancelled incrementally. Errors beyond the prefetched prefix surface during stream consumption. The cache budget limits prefetch accounting, not total query memory; public APIs are unchanged.