Repository navigation
feat: exact reverse scan for parquet sort pushdown - #27079
Conversation
When a query asks for the reverse of the files' declared ordering, the parquet scan already reverses the row group order, but rows inside each row group stay in on-disk order, so the pushdown is `Inexact` and a `SortExec` (usually `TopK`) stays above the scan. Add `datafusion.execution.parquet.enable_exact_reverse_scan` (default false). With it set, `ParquetSource::try_pushdown_sort` returns `Exact` for a reversed request: the opener reverses row groups and also reverses the rows of each row group by draining its reader, reversing each batch and emitting the batches last to first. `FileScanConfig` reverses the files of each group, declares the reversed orderings, and re-validates them, falling back to `Inexact` when they do not hold (e.g. overlapping files). The `SortExec` is removed and a `LIMIT` is pushed into the scan. A limit is counted in decode order, which the reversal does not follow, so a reversing scan does not pass it to the decoder or use it for row group and page pruning; the file stream applies it to reversed output. Instead, under a limit: - a row group whose output is known up front (no predicate, or fully matched by statistics) is cut to its last `limit` rows with a row selection, and the offset index is loaded so the pages ahead of them are not fetched; - for other row groups only the last `limit` decoded rows are kept. The buffered rows are charged to the task's memory pool through a new `FileSource::with_memory_pool` hook (default: no accounting), so an oversized row group fails the scan with a resources error rather than growing the process. Row groups are not reordered by statistics in this mode, since that is not order-preserving for row groups that share a `min`.
github-actions
Bot
added
documentation
|
run benchmark sort_pushdown_sorted changed: |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing exact-reverse-rows (aa7bfa9) to ea33cce (merge-base) diff Run configurationrun benchmark sort_pushdown_sorted
changed:
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_EXACT_REVERSE_SCAN: "true"Results will be posted here when complete File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing exact-reverse-rows (aa7bfa9) to ea33cce (merge-base) diff Run configurationrun benchmark sort_pushdown_sorted
changed:
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_EXACT_REVERSE_SCAN: "true"CPU Details (lscpu)Details
Resource Usagesort_pushdown_sorted — base (merge-base)
sort_pushdown_sorted — branch
File an issue against this benchmark runner |
There was a problem hiding this comment.
Warning
Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.
Copilot review overview
1 open issue
What changed in this PR
Adds an opt-in “exact reverse scan” mode for Parquet sort-pushdown so ORDER BY ... DESC LIMIT ... can be satisfied directly by the scan (removing SortExec) by reversing both row-group order and row order within each row group, with memory usage accounted against the task memory pool.
Changes:
- Introduces
datafusion.execution.parquet.enable_exact_reverse_scan(defaultfalse) and wires it through configs, information_schema, and proto (includingreverse_rowsonParquetScanExecNode). - Implements
reverse_rowsexecution: buffer/decode a row group, reverse rows in each batch, emit batches last-to-first; avoid decode-order limit pruning and add tail-selection optimization under limit. - Adds
FileSource::with_memory_poolhook and uses it to account buffered row groups to the task memory pool; adds extensive unit + sqllogictest coverage.
| File | Description |
|---|---|
| docs/source/user-guide/configs.md | Documents the new enable_exact_reverse_scan option and its memory behavior. |
| datafusion/sqllogictest/test_files/sort_pushdown_exact_reverse.slt | End-to-end sqllogictest coverage for exact reverse scan mode and fallbacks. |
| datafusion/sqllogictest/test_files/information_schema.slt | Exposes the new config option in information_schema expectations. |
| datafusion/proto/tests/cases/plans/sources.rs | Adds proto round-trip test ensuring reverse_rows is preserved. |
| datafusion/proto-models/src/generated/prost.rs | Adds reverse_rows to ParquetScanExecNode prost model. |
| datafusion/proto-models/src/generated/pbjson.rs | Adds JSON (de)serialization support for reverseRows. |
| datafusion/proto-models/src/generated/datafusion_proto_common.rs | Adds enable_exact_reverse_scan to ParquetOptions proto model. |
| datafusion/proto-models/src/from_proto.rs | Maps enable_exact_reverse_scan from proto into runtime options. |
| datafusion/proto-models/proto/datafusion.proto | Extends ParquetScanExecNode with reverse_rows. |
| datafusion/proto-common/src/to_proto/mod.rs | Maps enable_exact_reverse_scan into protobuf ParquetOptions. |
| datafusion/proto-common/src/generated/prost.rs | Adds enable_exact_reverse_scan to common prost-generated models. |
| datafusion/proto-common/src/generated/pbjson.rs | Adds JSON (de)serialization for enableExactReverseScan. |
| datafusion/proto-common/src/from_proto/mod.rs | Maps enable_exact_reverse_scan from common protobuf into runtime options. |
| datafusion/proto-common/proto/datafusion_common.proto | Adds enable_exact_reverse_scan to shared ParquetOptions proto. |
| datafusion/datasource/src/file_scan_config/sort_pushdown.rs | Handles “Exact reverse” by reversing output ordering and re-validating against reversed file groups. |
| datafusion/datasource/src/file_scan_config/mod.rs | Calls FileSource::with_memory_pool when opening scans; adds tests for exact reverse semantics. |
| datafusion/datasource/src/file.rs | Extends FileSource with with_memory_pool hook (default no-op). |
| datafusion/datasource-parquet/src/source.rs | Adds reverse_rows + memory-pool plumbing; returns Exact when option enabled and reverse satisfies. |
| datafusion/datasource-parquet/src/push_decoder.rs | Implements row-group buffering + row reversal + tail-keeping under limit; adds memory reservation support. |
| datafusion/datasource-parquet/src/opener/mod.rs | Disables decode-order limit pruning under row-reversal; adds tail-selection + offset-index enablement + memory reservation wiring; adds tests. |
| datafusion/datasource-parquet/src/file_format.rs | Includes enable_exact_reverse_scan in serialized TableParquetOptions. |
| datafusion/datasource-parquet/src/access_plan.rs | Adds PreparedAccessPlan::select_tail_for_limit helper + tests. |
| datafusion/common/src/file_options/parquet_writer.rs | Propagates new option through ParquetOptions struct (writer props ignore it). |
| datafusion/common/src/config.rs | Registers config option and documentation string. |
Review effort: Lite
|
cc @alamb @adriangb Follow-up for exact reverse scan for parquet sort pushdown, the benchmark result is pretty good: ┏━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query ┃ HEAD ┃ exact-reverse-rows ┃ Change ┃
┡━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ Q1 │ 151.53 / 152.77 ±1.86 / 156.43 ms │ 150.45 / 151.51 ±1.21 / 153.78 ms │ no change │
│ Q2 │ 2.36 / 2.50 ±0.21 / 2.91 ms │ 2.34 / 2.50 ±0.16 / 2.82 ms │ no change │
│ Q3 │ 512.90 / 516.28 ±2.10 / 518.72 ms │ 508.42 / 512.09 ±3.48 / 517.57 ms │ no change │
│ Q4 │ 5.36 / 5.56 ±0.36 / 6.28 ms │ 5.36 / 5.63 ±0.30 / 6.19 ms │ no change │
│ Q5 │ 10.66 / 10.78 ±0.17 / 11.12 ms │ 2.41 / 2.51 ±0.11 / 2.68 ms │ +4.30x faster │
│ Q6 │ 14.28 / 14.39 ±0.12 / 14.62 ms │ 2.41 / 2.73 ±0.54 / 3.81 ms │ +5.26x faster │
│ Q7 │ 19.13 / 19.43 ±0.29 / 19.81 ms │ 4.78 / 5.00 ±0.13 / 5.17 ms │ +3.89x faster │
│ Q8 │ 22.82 / 22.90 ±0.08 / 23.04 ms │ 4.75 / 5.54 ±1.41 / 8.35 ms │ +4.14x faster │
└───────┴───────────────────────────────────┴───────────────────────────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━┓
┃ Benchmark Summary ┃ ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━┩
│ Total Time (HEAD) │ 744.62ms │
│ Total Time (exact-reverse-rows) │ 687.51ms │
│ Average Time (HEAD) │ 93.08ms │
│ Average Time (exact-reverse-rows) │ 85.94ms │
│ Queries Faster │ 4 │
│ Queries Slower │ 0 │
│ Queries with No Change │ 4 │
│ Queries with Failure │ 0 │
└───────────────────────────────────┴──────────┘ |
jayzhan211
left a comment
jayzhan211
left a comment
There was a problem hiding this comment.
@zhuqi-lucas thanks for this, the LIMIT wins reproduce here. In file_scan_config/mod.rs around line 1075, one problem: removing the SortExec leaves the limit in the scan, so the post-phase filter pushdown can now put a join or TopK dynamic filter below it and rows outside the limit get through. The forward ASC shape already has this bug on main, but DESC was safe until now. Can we decline filters when the scan has a limit, and add these two queries to the slt?
SET datafusion.execution.target_partitions = 1;
SET datafusion.execution.parquet.enable_exact_reverse_scan = true;
SET datafusion.execution.parquet.pushdown_filters = true;
SET datafusion.execution.parquet.max_row_group_size = 4;
COPY (SELECT value AS id, CASE WHEN value % 2 = 0 THEN 'a' ELSE 'b' END AS category FROM generate_series(1, 40) ORDER BY id) TO 't/data.parquet';
COPY (SELECT * FROM (VALUES (1), (2), (39)) v(id)) TO 'small/s.parquet';
CREATE EXTERNAL TABLE t(id BIGINT, category VARCHAR) STORED AS PARQUET LOCATION 't/' WITH ORDER (id ASC);
CREATE EXTERNAL TABLE small(id BIGINT) STORED AS PARQUET LOCATION 'small/';
-- returns 1, 2, 39; should be 39
SELECT s.id FROM small s JOIN (SELECT * FROM t ORDER BY id DESC LIMIT 5) x ON s.id = x.id ORDER BY 1;
-- returns 35; should be 37
SELECT id FROM (SELECT * FROM t ORDER BY id DESC LIMIT 5) ORDER BY category DESC, id LIMIT 1; ) -> Result<FilterPushdownPropagation<Arc<dyn DataSource>>> {
+ // The scan applies its limit after any filter it holds, so a filter from
+ // above the limit would let rows past it that the limit should have cut.
+ if self.limit.is_some() {
+ return Ok(FilterPushdownPropagation::with_parent_pushdown_result(vec![
+ datafusion_physical_plan::filter_pushdown::PushedDown::No;
+ filters.len()
+ ]));
+ }| // Re-validate against the reversed groups before keeping them; | ||
| // if the ordering does not hold, clear it and the caller falls | ||
| // back to `Inexact`, which keeps the `SortExec`. | ||
| if reverse_file_groups && is_exact { |
There was a problem hiding this comment.
This branch can reverse a scan that already has a limit, which then keeps rows from the wrong end of the file. With the setup from my review comment, this returns 40 39 38 37 36 instead of 5 4 3 2 1. Main also gets this query wrong through the Inexact path, so this guard fixes both. Can you add it, with this query in the slt?
SELECT id FROM (SELECT * FROM t ORDER BY id ASC LIMIT 5) ORDER BY id DESC;--- a/datafusion/datasource/src/file_scan_config/mod.rs
+++ b/datafusion/datasource/src/file_scan_config/mod.rs
) -> Result<SortOrderPushdownResult<Arc<dyn DataSource>>> {
+ // Reordering the scan changes which rows its limit keeps.
+ if self.limit.is_some() {
+ return Ok(SortOrderPushdownResult::Unsupported);
+ }There was a problem hiding this comment.
Added in ec8e932, with this query as test 13 in the slt and a unit test.
| new_source.reverse_row_groups = true; | ||
| new_source.reverse_rows = true; |
There was a problem hiding this comment.
A second pushdown on an already reversed scan flips the files and the ordering back but leaves these flags on: I got output_ordering=[id ASC] with rows still in DESC order. A single PushdownSort pass can't reach this, but a second optimizer pass or a direct try_pushdown_sort call can. Toggling fixes it, and a unit test that pushes down twice would cover it.
| new_source.reverse_row_groups = true; | |
| new_source.reverse_rows = true; | |
| new_source.reverse_row_groups = !self.reverse_rows; | |
| new_source.reverse_rows = !self.reverse_rows; |
There was a problem hiding this comment.
Done, with try_pushdown_sort_exact_reverse_twice_reads_forward.
| continue; | ||
| } | ||
| if let Some(reservation) = &self.reversed_reservation { | ||
| reservation.try_grow(batch.get_array_memory_size())?; |
There was a problem hiding this comment.
Each batch charges the full size of buffers it shares with other batches, so a large page dictionary is counted once per batch, and this scan can't spill. With 1M rows and 4000 distinct strings (about 25 MB) a 100 MB pool fails here, while the option-off plan finishes at 30 MB by spilling. It's fine to handle this in a follow-up since it needs shared buffers counted once across batches, but could you file an issue so it isn't lost before this option goes on by default?
There was a problem hiding this comment.
Agreed, filed #27254 to fix this before the option goes on by default.
| /// emits its rows in reverse, so the scan is exactly ordered and the | ||
| /// `SortExec` above it can be removed. Peak memory per scanned file is | ||
| /// one decoded row group (after projection and filtering). If false, | ||
| /// only the row group order is reversed and the sort is kept. |
There was a problem hiding this comment.
Optional cleanup: a later SET has no effect on an existing table in either direction, so someone hitting the resources error can't turn this off that way. Worth saying so here.
| /// only the row group order is reversed and the sort is kept. | |
| /// only the row group order is reversed and the sort is kept. The value | |
| /// is captured when the table is created, so a later `SET` does not | |
| /// affect existing tables. |
| ) -> Result<()> { | ||
| let mut kept: VecDeque<RecordBatch> = VecDeque::new(); | ||
| let mut kept_rows = 0; | ||
| for batch in reader { |
There was a problem hiding this comment.
Optional cleanup: this decodes the whole row group inside one poll, about 150 ms per 1M rows on SELECT * here, and a query can't be cancelled until it finishes. Making this function async and yielding between batches would keep it cancellable.
tokio::task::yield_now().await;There was a problem hiding this comment.
Done. The stream now pushes one batch per loop iteration and yields between them, with the compute timer paused across the yield.
| /// Emit each row group's rows in reverse order. The active reader is | ||
| /// drained into [`Self::reversed_batches`] before anything is returned, | ||
| /// so one decoded row group is held in memory at a time. | ||
| pub(crate) reverse_rows: bool, |
There was a problem hiding this comment.
Optional cleanup: these four fields are only meaningful together, so one optional struct would remove the .filter(|_| prepared.reverse_rows) calls in the opener and the self.reverse_rows check here.
pub(crate) reverser: Option<RowGroupReverser>,There was a problem hiding this comment.
Done: Option<RowGroupReverser> with push, finish and pop.
…everse scans - Decline filter pushdown into a scan that holds a limit, since the filter would run before the limit. - Do not push a sort into a scan that holds a limit, since reordering it changes which rows the limit keeps. - Reverse files and orderings when the reversed ordering satisfies the request under the scan's equivalences, not only when it is the literal reverse, so these requests become exact instead of leaving a row reversing scan under a TopK. - When an exact claim cannot be kept, continue with the original source so no row reversal is left under the sort. - Toggle the reverse flags when a reverse is pushed onto a reversed scan. - Group the row reversal state into RowGroupReverser and yield between the batches of a buffered row group so the query stays cancellable. - Document that the option is captured when a table is created.
|
Thanks a lot @jayzhan211 for the thorough review and the repro queries, they made each of these easy to confirm. All addressed in ec8e932. For the filter case in your review: the scan now declines filters when it holds a limit, and both queries are tests 11 and 12 in the slt. |
jayzhan211
left a comment
jayzhan211
left a comment
There was a problem hiding this comment.
Optional cleanup: under user-facing changes, could the description say that queries over a scan with a limit now return different (correct) rows at default config, as in #27194? Readers of the release notes will look there.
| // The scan applies its limit after any filter it holds, so a filter | ||
| // from above the limit would let through rows the limit should have | ||
| // cut. | ||
| if self.limit.is_some() { |
There was a problem hiding this comment.
@zhuqi-lucas thanks, everything from the last round checks out here. One change, and it reverses my own suggestion, sorry: the two self.limit.is_some() checks make the scan the third node with its own "a fetch blocks pushdown" guard (after #27060 and #23599), while one node.fetch() check in the PushdownSort and FilterPushdown rules covers every node. With the diff below in place of the two checks, the full sqllogictest suite passes, tests 11–13 included, and two filter_pushdown.rs snapshots change, correctly, since they merged an outer filter into a FilterExec with fetch=5.
This also fixes #27194 at default config, so could you add Closes #27194 and its three queries, with the option off, to the slt?
Rule-level check (replaces both scan checks and `sort_pushdown_with_limit_unsupported`; clippy clean)
--- a/datafusion/physical-optimizer/src/filter_pushdown.rs
+++ b/datafusion/physical-optimizer/src/filter_pushdown.rs
@@ fn push_down_filters(
phase: FilterPushdownPhase,
) -> Result<FilterPushdownPropagation<Arc<dyn ExecutionPlan>>> {
+ // A node with a fetch picks its rows before a filter from above applies, so
+ // that filter can't move below or into it. The node's own filters still can.
+ if node.fetch().is_some() && !parent_predicates.is_empty() {
+ let result = push_down_filters(node, vec![], config, phase)?;
+ return Ok(FilterPushdownPropagation {
+ filters: vec![PushedDown::No; parent_predicates.len()],
+ updated_node: result.updated_node,
+ });
+ }
+
--- a/datafusion/physical-optimizer/src/pushdown_sort.rs
+++ b/datafusion/physical-optimizer/src/pushdown_sort.rs
+/// Whether a node below this sort has a fetch. A fetch keeps the first rows in the
+/// input's current order, so changing that order would change which rows it keeps.
+fn input_has_fetch(input: &Arc<dyn ExecutionPlan>) -> Result<bool> {
+ input.exists(|node| Ok(node.fetch().is_some()))
+}
@@ (SortPreservingMergeExec arm)
let sort_input = Arc::clone(sort_child.input());
+ if input_has_fetch(&sort_input)? {
+ return Ok(Transformed::no(plan));
+ }
@@ (SortExec arm)
let sort_input = Arc::clone(sort_exec.input());
+ if input_has_fetch(&sort_input)? {
+ return Ok(Transformed::no(plan));
+ }New snapshot for test_filter_collapse_inner_fetch_preserved (the both_fetch one changes the same way):
- FilterExec: b@1 = bar
- FilterExec: a@0 = foo, fetch=5
- DataSourceExec: ...
There was a problem hiding this comment.
Done in 71beb85: the two scan checks and sort_pushdown_with_limit_unsupported are replaced by your rule-level check. The full sqllogictest suite passes, and the two filter_pushdown.rs snapshots now keep the filters separate (I renamed those tests to match). The three queries from #27194 are test 16 in the slt with the option off.
Replace the scan-level `limit` checks with one check per rule, as suggested in review: a node with a fetch keeps the first rows in its input's current order, so `PushdownSort` must not reorder the input below it and `FilterPushdown` must not move a filter from above into or below it (the node's own filters still push down). This also fixes apache#27194 at the default configuration. The two `filter_pushdown.rs` snapshots that merged an outer filter into a `FilterExec` with a fetch now keep the filters separate, which keeps the rows the inner fetch picks.
|
Thanks @jayzhan211 for the review again, everything is addressed in 71beb85. The description now says Closes #27194 and notes the changed (correct) results under user-facing changes. |
|
Thanks @zhuqi-lucas ! |
Which issue does this PR close?
Inexactreverse scan) and Phase 2: make sort pushdown support exact mode which can support sort elimination when ordering is guaranteed #19329 (Exactfor the natural ordering).ORDER BY ... LIMITreads a sorted Parquet file #27194.Rationale for this change
When a query asks for the reverse of the files' declared ordering (for example
ORDER BY ts DESC LIMIT 10on files written ints ASC), the parquet scan already reverses the row group order, but rows inside each row group stay in on-disk order. The pushdown is thereforeInexactand aSortExec(usuallyTopK) stays above the scan, which has to consume at least one full row group per file before it can emit anything.Reading each row group back to front makes the scan exactly ordered, so the
SortExeccan go and theLIMITcan be pushed into the scan.What changes are included in this PR?
New option
datafusion.execution.parquet.enable_exact_reverse_scan(defaultfalse).ParquetSource::try_pushdown_sortreturnsExactwhen the reversed declared ordering satisfies the request and the option is set. The source sets a newreverse_rowsflag next toreverse_row_groups, and does not reorder row groups by statistics, since that is not order-preserving for row groups that share amin.The parquet stream drains each row group's reader, reverses the rows of every batch, and emits the batches last to first. At most one decoded row group (after projection and filtering) is held at a time.
A limit is counted in decode order, which the reversal does not follow, so a reversing scan does not pass it to the decoder or use it for row group / page pruning; the file stream applies it to the reversed output. Under a limit the scan reads less instead:
limitrows with a row selection, and the offset index is loaded so the pages ahead of them are not fetched;limitdecoded rows are kept, so at mostlimitrows plus one batch are held.The buffered rows are charged to the task's memory pool through a new
FileSource::with_memory_poolhook (default: no accounting;FileScanConfigcalls it when the scan is opened). An oversized row group fails the scan with a resources error instead of growing the process.FileScanConfig::rebuild_with_sourcereverses the files of each group, declares the reversed orderings, and re-validates them, falling back toInexactwhen they do not hold (for example overlapping files). For a source that isExactbut whose request is not literally the reverse of the declared ordering, the forwardExactpath now also checks that the kept ordering satisfies the request, so such a claim is downgraded instead of keeping an ordering that does not describe the scan.reverse_rowsis carried inParquetScanExecNodeand the option inParquetOptionsproto.PushdownSortandFilterPushdowntreat a node with a fetch as a barrier: a fetch keeps the first rows in its input's current order, so the sort rule does not reorder the input below it, and the filter rule does not move a filter from above into or below it (the node's own filters still push down). This replaces scan-specific limit checks and fixes Wrong rows when a subquery'sORDER BY ... LIMITreads a sorted Parquet file #27194.Are these changes tested?
PreparedAccessPlan::select_tail_for_limit: tail selection, spanning row groups, stopping at a row group whose output is unknown or that already has a selection.ParquetSource::try_pushdown_sort:Exactwith the option,Inexactwithout.FileScanConfig: files and ordering reversed, overlapping files downgraded, anExactclaim without file reversal downgraded.reverse_rows.sort_pushdown_exact_reverse.slt: single and multiple files, filter, overlapping files, multi-column prefix, NULLs, NOT NULLDESC NULLS LAST, two partitions, option off.ORDER BY ... LIMITreads a sorted Parquet file #27194 insort_pushdown_exact_reverse.slt(test 16), with the option off.Benchmark
sort_pushdown_sorted(TPC-H SF1lineitem, 3 files sorted byl_orderkey,WITH ORDER), median of 5 iterations, this branch with the option off vs on (run withdfbench sort-pushdown --sortedas inbench.sh run sort_pushdown_sorted):ORDER BY l_orderkeyORDER BY l_orderkey LIMIT 100*,ORDER BY l_orderkey*,ORDER BY l_orderkey LIMIT 100DESC LIMIT 100DESC LIMIT 1000*,DESC LIMIT 100*,DESC LIMIT 1000The reversed
LIMITqueries now cost about the same as their forward counterparts (q5 vs q2, q7 vs q4). Thesort_pushdown_inexact*benchmarks are unaffected: their files overlap, so the exact path does not apply.Are there any user-facing changes?
At the default configuration, a query whose
ORDER BY ... LIMITover a sorted Parquet file ends up as a scan limit, with a reversing sort or a dynamic filter above it, now returns different (correct) rows (#27194). Twofilter_pushdown.rssnapshots change accordingly: an outer filter is no longer merged into aFilterExecwith a fetch.A new config option, off by default, and a new
FileSource::with_memory_poolmethod with a default implementation. With the option on, plans for reversed requests drop theSortExecand showreverse_rows=trueon the scan. A scan without a limit still holds one decoded row group per partition (charged to the memory pool), which is why the option is off by default; page-level reverse decoding (#19486) would bound it further.