Skip to content

Exact reverse parquet scan counts shared buffers once per batch in its memory reservation #27254

Open

Description

@zhuqi-lucas

With datafusion.execution.parquet.enable_exact_reverse_scan (#27079), the scan buffers one decoded row group and charges each batch's get_array_memory_size() to the memory pool. Batches of the same row group can share buffers, such as a page dictionary, so a shared buffer is counted once per batch. The scan cannot spill, so a query can fail with a resources error well below its real memory use.

Example from the #27079 review: 1M rows with 4000 distinct strings (about 25 MB). A 100 MB pool fails in the reverse scan, while the plan with the option off finishes at 30 MB by spilling in the sort.

The reservation should count each distinct buffer once across the batches it holds. This should be fixed before the option is enabled by default.

Related: #27079, #19048

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions