Repository navigation
fix: SanityCheckPlan error when a per-partition constant leads an ordering
#27191
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Changes from all commits
b42a16b
6e6a8ae
c777864
6425ba0
9c7ee55
9210b51
17c73e6
79d9e8f
96a7399
434a127
5f70669
a8ec6f4
28a60df
2ceaba5
7f87bff
5045608
27ed6c6
f2e397e
8789680
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
The walk does not derive it because nothing is pinned. This is the order dependence the new comment says must not happen.
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I couldn't get an order dependence from |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -379,7 +379,8 @@ impl EquivalenceProperties { | |
| } | ||
| } | ||
|
|
||
| /// Adds new orderings into the existing ordering equivalence class. | ||
| /// Adds new orderings into the existing ordering equivalence class, along | ||
| /// with the orderings they imply through known equalities. | ||
| pub fn add_orderings( | ||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Now that With |
||
| &mut self, | ||
| orderings: impl IntoIterator<Item = impl IntoIterator<Item = PhysicalSortExpr>>, | ||
|
|
@@ -391,14 +392,32 @@ impl EquivalenceProperties { | |
| .cloned() | ||
| .filter_map(|o| self.normalize_sort_exprs(o)) | ||
| .collect(); | ||
| if !normal_orderings.is_empty() { | ||
| self.oeq_class.extend(orderings); | ||
| // Normalize given orderings to update the cache: | ||
| self.oeq_cache.normal_cls.extend(normal_orderings); | ||
| // TODO: If no ordering is found to be redundant during extension, we | ||
| // can use a shortcut algorithm to update the leading map. | ||
| self.oeq_cache.update_map(); | ||
| if normal_orderings.is_empty() { | ||
| return; | ||
| } | ||
| // `add_constants` and `add_equal_conditions` discover implied orderings | ||
| // too, and whether an ordering is satisfied must not depend on which of | ||
| // them ran last. Deriving one fails only on an expression that does not | ||
| // match the schema; that ordering is then not implied, which can cost a | ||
| // sort but not a wrong result. | ||
| let mut implied: Vec<LexOrdering> = vec![]; | ||
| for ordering in &normal_orderings { | ||
| // Two orderings can imply the same one; expand it only once: | ||
| for ordering in self.implied_by(ordering).unwrap_or_default() { | ||
| if !implied.contains(&ordering) { | ||
| implied.push(ordering); | ||
| } | ||
| } | ||
| } | ||
| self.oeq_class.extend(orderings); | ||
| // Normalize given orderings to update the cache: | ||
| self.oeq_cache.normal_cls.extend(normal_orderings); | ||
| // TODO: If no ordering is found to be redundant during extension, we | ||
| // can use a shortcut algorithm to update the leading map. | ||
| self.oeq_cache.update_map(); | ||
| // Each implied ordering is shorter than the one implying it, so this | ||
| // terminates: | ||
| self.add_orderings(implied); | ||
|
Comment on lines
+418
to
+420
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. can we use a deduplicated worklist and avoid expanding already-known candidates?
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Each round's implied orderings are now deduplicated before they are expanded. I didn't skip candidates that are already in the class:
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
|
||
| } | ||
|
|
||
| /// Adds a single ordering to the existing ordering equivalence class. | ||
|
|
@@ -568,15 +587,76 @@ impl EquivalenceProperties { | |
| let mut new_orderings = vec![]; | ||
| for idx in ordering_idxs { | ||
| let ordering = &self.oeq_cache.normal_cls[*idx]; | ||
| new_orderings | ||
| .extend(self.implied_ordering(&ordering[0], ordering[1..].iter())?); | ||
| } | ||
| if !new_orderings.is_empty() { | ||
| self.add_orderings(new_orderings); | ||
| // As in `add_orderings`, an ordering that cannot be derived is | ||
| // not implied, so the result does not depend on the call order: | ||
| new_orderings.extend(self.implied_by(ordering).unwrap_or_default()); | ||
| } | ||
| self.add_orderings(new_orderings); | ||
| Ok(()) | ||
| } | ||
|
|
||
| /// Returns the orderings that the normal ordering `ordering` implies | ||
| /// through known equalities: for each key that is a function of the keys | ||
| /// after it (see [`Self::implied_ordering`]), `ordering` without that key. | ||
| /// | ||
| /// # Example: a key that is a function of the keys after it | ||
| /// | ||
| /// * Ordering: `[k ASC, a ASC, c ASC]` | ||
| /// * Equality: `a = CAST(c AS DATE)` | ||
| /// * Returns: `[k ASC, c ASC]` | ||
| /// | ||
| /// Among the rows that tie on `k`, the rows are sorted by `a`, and the | ||
| /// rows that also tie on `a` are sorted by `c`. `a` is the date of `c`, so | ||
| /// a row with an earlier `a` also has an earlier `c`, and the rows that | ||
| /// tie on `k` are sorted by `c` alone: | ||
| /// | ||
| /// ```text | ||
| /// ┌───┬────────────┬──────────────────┐ | ||
| /// │ k │ a │ c │ | ||
| /// ├───┼────────────┼──────────────────┤ | ||
| /// │ x │ 2026-01-01 │ 2026-01-01 09:00 │ | ||
| /// │ x │ 2026-01-02 │ 2026-01-02 01:00 │ | ||
| /// │ x │ 2026-01-02 │ 2026-01-02 05:00 │ | ||
| /// │ y │ 2026-01-01 │ 2026-01-01 03:00 │ | ||
| /// └───┴────────────┴──────────────────┘ | ||
| /// ``` | ||
| /// | ||
| /// # Example: a constant that differs between partitions | ||
| /// | ||
| /// * Ordering: `[k ASC, a ASC, c ASC]`, where `k` is a constant whose value | ||
| /// differs between partitions | ||
| /// * Equality: `a = CAST(c AS DATE)` | ||
| /// * Returns: `[c ASC]` | ||
| /// | ||
| /// A normal ordering can keep such a constant (see | ||
| /// `EquivalenceGroup::normalize_sort_exprs`). Within a partition `k` has | ||
| /// one value, so it is left out of the keys and of the result. | ||
| /// | ||
| /// # Example: directions that differ | ||
| /// | ||
| /// * Ordering: `[a ASC, c DESC]` | ||
| /// * Equality: `a = CAST(c AS DATE)` | ||
| /// * Returns: nothing | ||
| /// | ||
| /// `c` descends within a date but ascends across dates, so the rows are | ||
| /// not sorted by `c` in either direction. | ||
| fn implied_by(&self, ordering: &LexOrdering) -> Result<Vec<LexOrdering>> { | ||
| let keys: Vec<_> = ordering | ||
| .iter() | ||
| .filter(|sort_expr| self.eq_group.is_expr_constant(&sort_expr.expr).is_none()) | ||
| .collect(); | ||
| let mut implied = vec![]; | ||
| for (idx, key) in keys.iter().enumerate() { | ||
| if let Some(rest) = | ||
| self.implied_ordering(key, keys[idx + 1..].iter().copied())? | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. it seems like the fact that Since the only error that is actually possible is if the type can't be found from the schema, which seems like a safe thing to ignore (
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. We could switch back to
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I also vote to return This will make it less disruptive to backport Then we can make a PR to main perhaps to propagate the error |
||
| { | ||
| let prefix = keys[..idx].iter().copied().cloned(); | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. IMO the Simpler fix would be: keep the constants in the stored implied ordering (store wdyt?
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This handles a leading constant, but not one between a key and its arguments, as in
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I don't understand the implications of this comment Specifically, if we didn't do this change, would we get wrong plans / errors? Or is it just a missed optimization opportunity?
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I'd keep
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I think the issue is not critical but the change is quite small
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. It was a wrong-result risk, not a missed optimization: before the reorder, the merged output could claim |
||
| implied.extend(LexOrdering::new(prefix.chain(rest))); | ||
| } | ||
| } | ||
| Ok(implied) | ||
| } | ||
|
|
||
| /// Returns the ordering that the normal ordering `[leading, rest..]` | ||
| /// implies through an equality on its leading expression, if any. | ||
| /// | ||
|
|
||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
datafusion/datafusion/physical-expr/src/equivalence/properties/mod.rs
Line 590 in c777864
discover_new_orderingsnow duplicates work: it computesimplied_orderingfor each leading ordering and passes the results toadd_orderings, which immediately recomputesimplied_orderingon those results and recurses again, each level re-normalizing and callingupdate_map.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
These aren't the same computation:
discover_new_orderingsderives from an ordering, andadd_orderingsthen derives from what that ordering implied. The recursion is what finds chains. Withd = CAST(a),a = CAST(c)and ordering[d, a, c],[c]is found with it and missed with one level, andimplied_orderings_chaincovers that. It only runs when something was implied.