[Muneer320]

Which issue does this PR close?

Closes #27134.

Rationale for this change

For a table partitioned by year and month, year=2024/month=02.parquet is incorrectly read as month 02.parquet. The filename supplies a partition value even though the month=.../ directory is missing.

What changes are included in this PR?

Require a following path component for each partition directory, so the final filename cannot supply a partition value. Preserve percent-decoding and the handling of incomplete paths. A filename containing = is still valid inside a complete partition directory structure.

Document this contract on the public parse_partitions_for_path function. Add direct assertions for a root mypartition=v1.csv file and a two-level mypartition=v1/otherpartition=v2.csv path, with a complete-directory control. SQL regressions cover string and integer partition columns, returned rows, filtering and a key=value filename inside complete directories.

Disabling the listing cache rebuilds session state and changes the default-catalog setting; the test resets that setting for the runner's configuration check.

What is the testing strategy for this PR?

The hosted verification passed formatting, the focused partition SQL file and the repository-required extended workspace test command against main 34eb6dc5bf057a35c471a4fbfd2eece2a7937afe. All-target/all-feature Clippy passed in the earlier run; the final run verified that every Rust input is identical before reusing that result. Only the SQL fixture changed between those runs.

Both ablation checks passed: restoring the original parser while retaining the new tests fails the direct assertion and reproduces the integer cast error for 2025.parquet. Restoring the fix produces the exact verified patch again. These are contributor-controlled hosted checks, separate from upstream PR CI. No existing microbenchmark calls this helper and this is not a performance claim.

Are there any user-facing changes?

Yes. The public parser treats the final component as the filename and returns None when a required partition directory is missing, even if the filename looks like name=value. Its Rust signature does not change, but its accepted-path behavior does.

For a non-string partition column, for example PARTITIONED BY (year INT), a newly listed root file year=2025.parquet arriving after registration previously caused a cast error for 2025.parquet. It is now skipped, so a query can succeed with fewer rows instead of failing. Complete partition paths retain their values. This matters for an existing table that later receives an incorrectly located file, or a consumer relying on the query failure to detect it. Creating a new table over a pre-existing mixture of root files and partition directories is still subject to the separate layout-inference check.

Related #26943 proposes datafusion.execution.listing_table_strict_partitions to report invalid listed paths as errors. That PR is still open at this update; the option is not introduced or assumed available here. Coordinating the two changes would give users an opt-in error path alongside the corrected default behavior.

AI assistance: Codex assisted with the implementation, regression tests and this description.

[mbutrovich]

@xanderbailey do you have any thoughts on this since you just worked on #26310?

[xanderbailey]

Thanks @Muneer320, this looks right to me. It's the same rule the opt-in strict mode in #26943 uses (it drops the last segment before matching directories), so once both land the default and strict paths agree on which files are invalid which is nice!