Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
38 commits
Select commit Hold shift + click to select a range
ac76b0d
fix: only apply SingleDistinctToGroupBy rewrite when it pays (closes …
mohitgurav20 Oct 17, 2026
09f9d96
Merge branch 'main' into fix/single-distinct-groupby-memory-usage
mohitgurav20 Oct 17, 2026
bbf4ccb
test: cover rewrite_pays for unsupported distinct handling in SingleD…
mohitgurav20 Oct 17, 2026
222ff4e
feat: add single_distinct_aggregation_prefer_memory optimizer option
mohitgurav20 Oct 19, 2026
cca6cd2
Merge branch 'main' into fix/single-distinct-groupby-memory-usage
mohitgurav20 Oct 19, 2026
f9c72ea
feat(optimizer): refine single distinct rewrite decision tree
mohitgurav20 Oct 19, 2026
3739116
Merge branch 'main' into fix/single-distinct-groupby-memory-usage
mohitgurav20 Oct 19, 2026
df3a448
test(optimizer): add branch coverage tests for is_single_distinct_agg
mohitgurav20 Oct 19, 2026
7f375a3
Merge branch 'main' into fix/single-distinct-groupby-memory-usage
mohitgurav20 Oct 19, 2026
1609f89
fix(docs): escape angle brackets in is_single_distinct_agg doc comment
mohitgurav20 Oct 19, 2026
d3e37ce
Merge branch 'main' into fix/single-distinct-groupby-memory-usage
mohitgurav20 Oct 19, 2026
4587fde
fix(optimizer): preserve count rollup guard and update tpch q16 plan
mohitgurav20 Oct 20, 2026
341df0f
Merge branch 'main' into fix/single-distinct-groupby-memory-usage
mohitgurav20 Oct 20, 2026
51f1775
test(sqllogictest): update plan snapshots for single distinct to grou…
mohitgurav20 Oct 20, 2026
38468c5
Merge branch 'main' into fix/single-distinct-groupby-memory-usage
mohitgurav20 Oct 20, 2026
7e4a93c
Merge branch 'main' into fix/single-distinct-groupby-memory-usage
mohitgurav20 Oct 21, 2026
5d437a3
test(optimizer): eliminate unexercised test branches and add avg comp…
mohitgurav20 Oct 21, 2026
5aa5906
Merge remote-tracking branch 'apache/main' into fix/single-distinct-g…
mohitgurav20 Oct 28, 2026
095b8c7
Merge remote-tracking branch 'apache/main' into fix/single-distinct-g…
mohitgurav20 Oct 28, 2026
2a091fa
ci: trigger mergeability recheck after sync with apache/main
mohitgurav20 Oct 28, 2026
1682661
Merge remote-tracking branch 'apache/main' into fix/single-distinct-g…
mohitgurav20 Oct 29, 2026
03dbe3e
Merge branch 'main' into fix/single-distinct-groupby-memory-usage
mohitgurav20 Oct 30, 2026
051a9d2
fix(optimizer): gate single distinct rewrite with single_distinct_agg…
mohitgurav20 Nov 1, 2026
eb2cd8a
Merge branch 'main' into fix/single-distinct-groupby-memory-usage
mohitgurav20 Nov 1, 2026
8024c82
fix: resolve intra-doc link for DistinctHandling in single_distinct_t…
mohitgurav20 Nov 1, 2026
a077c0c
Merge branch 'main' into fix/single-distinct-groupby-memory-usage
mohitgurav20 Nov 1, 2026
8071b00
Merge remote-tracking branch 'apache/main' into fix/single-distinct-g…
mohitgurav20 Nov 2, 2026
b4f3d08
feat(optimizer): refine prefer_memory guard to avoid regressions on s…
mohitgurav20 Nov 2, 2026
16000fe
Merge remote-tracking branch 'apache/main' into fix/single-distinct-g…
mohitgurav20 Nov 2, 2026
2341fe3
style: format single_distinct_to_groupby.rs
mohitgurav20 Nov 2, 2026
f45925c
docs: fix doc comment placement in single_distinct_to_groupby.rs
mohitgurav20 Nov 2, 2026
93e3746
Merge branch 'main' into fix/single-distinct-groupby-memory-usage
mohitgurav20 Nov 2, 2026
c8df6bc
Merge remote-tracking branch 'apache/main' into fix/single-distinct-g…
mohitgurav20 Nov 3, 2026
24155f6
style: add trailing newline to single_distinct_to_groupby.slt
mohitgurav20 Nov 7, 2026
c413a0e
Merge remote-tracking branch 'apache/main' into fix/single-distinct-g…
mohitgurav20 Nov 8, 2026
defd182
Merge remote-tracking branch 'apache/main' into fix/single-distinct-g…
mohitgurav20 Nov 9, 2026
1f7c26b
fix: address review feedback on float distinct semantics and document…
mohitgurav20 Nov 9, 2026
1b4e42b
refactor(optimizer): consolidate single distinct rewrite settings int…
mohitgurav20 Nov 11, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
23 changes: 15 additions & 8 deletions datafusion/common/src/config.rs
Original file line number Diff line number Diff line change
Expand Up @@ -1941,15 +1941,22 @@ config_namespace! {
pub enable_unions_to_filter: bool, default = false

/// Prefer native grouped DISTINCT aggregation when target_partitions is 1
/// and every aggregate explicitly supports a specialized GroupsAccumulator
/// for its DISTINCT argument types. Defaults to false, preserving the
/// two-phase GROUP BY rewrite.
/// and every DISTINCT argument is a fixed-width, non-float type (such as
/// integers, decimals, dates), instead of rewriting into a two-phase `GROUP BY`.
/// Defaults to false, preserving the two-phase GROUP BY rewrite.
///
/// The native plan can avoid materializing distinct (group, value) pairs,
/// but can be slower for many groups with duplicate values. It must fit
/// each group's distinct values in memory, so queries that succeed by
/// spilling with the rewrite may fail with this option enabled. One target
/// partition can still have multiple input partitions and merge native states.
/// The native plan avoids materializing intermediate `(group, distinct value)`
/// pairs, lowering peak memory when no memory limit is set. However,
/// native DISTINCT must fit each group's distinct values in memory: under
/// a memory limit, the rewrite can spill those pairs, but the native
/// accumulator cannot, so queries that succeed by spilling with the rewrite
/// may fail with this option enabled.
///
/// Queries with multiple target partitions, non-distinct companion aggregates,
/// floating-point arguments, and aggregates with unsupported distinct handling
/// (such as `approx_median` or `stddev`) continue with the rewrite.
///
/// See <https://github.com/apache/datafusion/issues/25932> and <https://github.com/apache/datafusion/pull/25988>
pub single_distinct_prefer_native: bool, default = false
}
}
Expand Down
Loading
Loading