fix(physical-expr): stop false infeasibility report from Eq+FALSE in cp_solver - #24464
Draft
Smallfu666 wants to merge 1 commit into
Draft
fix(physical-expr): stop false infeasibility report from Eq+FALSE in cp_solver#24464Smallfu666 wants to merge 1 commit into
Smallfu666 wants to merge 1 commit into
Conversation
…cp_solver propagate_comparison for Operator::Eq with parent == Interval::FALSE returned Ok(None), which ExprIntervalGraph::propagate_constraints interprets as infeasible. The correct semantics is that a = b being certainly false means a != b, which excludes at most a single point from each operand. Interval arithmetic cannot represent a hole, so the children cannot be refined further, but the expression is not infeasible. This caused analyze() to return None for the column interval when the predicate was NOT (a = 0.0) and the input domain contained 0.0 but was not equal to it (e.g. a in [-1, 1]). The None is documented in ExprBoundaries::interval to mean "evaluating the given column results in an empty set", but NOT (a = 0.0) is true for every a != 0, so the correct result is the full input domain. This is a contract violation of the public analyze() API, as reported in apache#19264 by an external user calling analyze() to infer bounds for pushdown into another library. Eq + FALSE is infeasible when equality is provably true for the two singleton operands under the comparison semantics. This must use comparison-semantics equality, not structural ScalarValue::PartialEq: the latter is bit-wise for floats (to_bits), under which -0.0 != +0.0 even though SQL/IEEE-754 comparison treats them as equal. The guard uses singleton_values_equal, which normalizes signed zero via normalize_float_zero_scalar (the same normalization DataFusion applies in physical-expr-common/src/datum.rs for runtime comparison) before structural equality. This matters in nested boolean contexts (e.g. NOT(a = +0.0) AND b) where the bottom-up pass does not short-circuit and top-down propagation forces Eq to FALSE, reaching this arm with [-0.0,-0.0] vs [+0.0,+0.0]. This revives the fix from apache#20138, which was closed as stale after review. The singleton guard was agreed on during apache#20138 review by berkaysynnada and pepijnve; this version strengthens it to use comparison-semantics equality. Direct unit tests isolate the ordinary identical-singleton case that the [0,0] analyze() test cannot reach because update_ranges short-circuits. The nested signed-zero E2E test separately proves that analyze() can reach the production arm through the pre-existing interval/runtime equality divergence. Tests: - analysis.rs: E2E tests via analyze(). Input [-1,1] is the reported case and does reach the fixed branch; input [0,0] is the nearest-invalid counterpart and must stay infeasible; the nested NOT(a = +0.0) AND b case proves the case-(1) short-circuit does not protect the production arm and exercises the signed-zero guard. - cp_solver.rs: direct propagate_comparison unit tests for Eq + FALSE covering identical singletons (infeasible), signed-zero singletons (infeasible under comparison semantics), distinct singletons (feasible), and overlapping intervals including the [-1,1] vs [0,0] case from the issue. Negative control on baseline 6eaca8b with the production arm reverted to Ok(None): 3 of the 7 new tests fail. The 4 that pass on baseline are boundary guards (baseline returns None for every Eq + FALSE input, so it agrees by accident). A second, independent control covers those guards: replacing singleton_values_equal with structural == makes both signed-zero tests fail, confirming they exercise the comparison-semantics guard rather than passing vacuously. Closes apache#19264. Co-authored-by: evangelisilva <silvaevangeli@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
physical_expr::analyze#19264.Rationale for this change
physical_expr::analyzeis a public API. As reported in #19264, an externaluser calling
analyze()to infer bounds for pushdown into another libraryreceives
None(infeasible) for the column interval when the expression isNOT (a = 0.0)and the input domain ofacontains0.0but is not equalto it (e.g.
a ∈ [-1, 1]).The
ExprBoundaries::intervaldoc states thatNonemeans "evaluating thegiven column results in an empty set" — i.e. the predicate is always false
over the input interval. But
NOT (a = 0.0)is true for everya ≠ 0, soover
[-1, 1]the predicate is satisfiable and the correct result is thefull input domain
[-1, 1]. The currentNoneis a contract violation.This revives the fix from #20138, which was closed as stale after review. It
preserves the reviewed identical-singleton infeasibility case (identified by
@berkaysynnada and @pepijnve during #20138 review) and adds direct unit
coverage that isolates the ordinary identical-singleton case. The
[0,0]analyze()test cannot reach that arm becauseupdate_rangesshort-circuits;the nested signed-zero
analyze()test does reach it through the pre-existinginterval/runtime equality divergence (see "Are these changes tested?" below).
Scope: the
FilterExecstatistics path does not currently reachanalyze()for a predicate containing a
NotExpr.check_support(intervals/utils.rs)accepts only
BinaryExpr,Column,Literal,CastExprandNegativeExpr,so such a predicate falls through to
falseandFilterExec::statistics_helpertakes its default-selectivity branch instead. This PR is about the public
analyze()contract, which #19264 reports against directly.What changes are included in this PR?
datafusion/physical-expr/src/intervals/cp_solver.rs:propagate_comparisonforOperator::Eqwithparent == Interval::FALSE:the arm previously returned
Ok(None)for all cases, which the caller(
ExprIntervalGraph::propagate_constraints) interprets as infeasible.The correct semantics is that
a = bbeing certainly false meansa != b,which excludes at most a single point from each operand's interval.
A single interval cannot represent that excluded point, so returning the
children unchanged is a safe over-approximation — except when equality is
provably true for the two singleton operands under the comparison
semantics, in which case
a = bis certainly true andNOT(a = b)isgenuinely infeasible.
singleton_values_equalhelper: compares two singletonScalarValues using SQL comparison semantics.ScalarValue::PartialEqis bit-wise for floats (
to_bits), under which-0.0 != +0.0; butSQL/IEEE-754 comparison treats them as equal. The helper uses
normalize_float_zero_scalar(the same normalization DataFusion appliesin
physical-expr-common/src/datum.rsfor runtime comparison) beforestructural equality, so the guard correctly identifies
[-0.0,-0.0]and[+0.0,+0.0]as certainly-equal singletons.propagate_comparisondirectly,forcing the
Eq + FALSEbranch:test_propagate_eq_false_identical_singletons:[0,0]vs[0,0]→None(nearest-invalid boundary).test_propagate_eq_false_signed_zero_singletons:[-0.0,-0.0]vs[+0.0,+0.0]→None(Float32 and Float64). This case is reachablein nested boolean contexts (see E2E test below).
test_propagate_eq_false_distinct_singletons:[0,0]vs[1,1]→Some(nearest-valid boundary).test_propagate_eq_false_overlapping_intervals:[-1,1]vs[0,0]→Some(the incorrect results when using NOT physical expression inphysical_expr::analyze#19264 case),[-1,1]vs[0,2]→Some.datafusion/physical-expr/src/analysis.rs:analyze():test_analyze_not_eq_around_zero(input[-1, 1], expectsSome([-1, 1])),test_analyze_not_eq_clamped_at_zero(input[0, 0], expectsNone),and
test_analyze_not_eq_nested_signed_zero_infeasible(nestedNOT(a = +0.0) AND bwitha ∈ [-0.0,-0.0],b ∈ [false,true]→expects infeasible). The nested test proves that the case-(1)
short-circuit in
update_rangesdoes not protect the production arm:the bottom-up pass evaluates
a = +0.0over[-0.0,-0.0]asFALSE(because
Interval::equaluses structural equality), soNOTbecomesTRUE, the root becomesTRUE_OR_FALSE, and top-down propagationforces
a = +0.0toFALSE, reachingpropagate_comparison(Eq, FALSE, [-0.0,-0.0], [+0.0,+0.0]).The guarded production logic:
Are these changes tested?
Yes, at two levels.
analysis.rscovers the user-visibleanalyze()behavior. The[-1, 1]casefrom the issue does reach the fixed branch: its bounds evaluate to
TRUE_OR_FALSE, soExprIntervalGraph::update_rangesruns propagation andNotExprhandsEqaFALSEparent. Its[0, 0]counterpart does not —those bounds evaluate to
FALSE, soupdate_rangesshort-circuits toInfeasiblebeforepropagate_constraintsis called.cp_solver.rstherefore adds unit tests callingpropagate_comparisondirectly. These isolate the ordinary identical-singleton guard that the
[0,0]analyze()test cannot reach, while the nested signed-zero E2E testseparately proves that the production arm is reachable from
analyze().Negative control, run on the pinned code baseline (
6eaca8bfe) with theproduction arm temporarily reverted to
Ok(None). The two modified files'pre-patch production blobs are identical to the final parent (
19f2e85fa);intervening upstream changes do not touch them. Each test is split so that its
baseline verdict is unambiguous — 3 of the 7 fail:
test_analyze_not_eq_around_zeroNone, expected[-1, 1]test_analyze_not_eq_nested_signed_zero_infeasibletest_propagate_eq_false_overlapping_intervals[-1,1]vs[0,0]returnsNone, expectedSometest_propagate_eq_false_distinct_singletons[0,0]vs[1,1]returnsNone, expectedSometest_propagate_eq_false_signed_zero_singletonstest_propagate_eq_false_identical_singletonstest_analyze_not_eq_clamped_at_zeroAll 7 pass on this branch.
test_analyze_not_eq_around_zerofailing on baseline is the direct evidencethat this PR fixes the reported behavior.
The four tests that pass on baseline are boundary guards, not evidence: the
baseline returns
Nonefor everyEq + FALSEinput, so it agrees with themby accident rather than by identifying the singleton-equal case. They are
included to pin the cases that must stay infeasible. The mutation control
below is what gives the two signed-zero guards their teeth.
test_analyze_not_eq_nested_signed_zero_infeasibledeserves special note:it passes on baseline (which reports
Nonefor allEq + FALSE), but aguard using structural
ScalarValue::PartialEqinstead ofsingleton_values_equalwould make it fail — the nestedANDdefeatsthe case-(1) short-circuit in
update_ranges, so propagation reaches theEq + FALSEarm with[-0.0,-0.0]vs[+0.0,+0.0], and structural equalitytreats them as distinct, producing a false feasible result. This was verified
by temporarily replacing
singleton_values_equalwith==and confirmingboth this test and
test_propagate_eq_false_signed_zero_singletonsfail.Note on signed zero:
Interval::equal/Interval::intersectuse structuralScalarValue::PartialEq, which is bit-wise for floats (to_bits), so[-0.0,-0.0]and[+0.0,+0.0]are distinct at the interval-algebra leveleven though SQL comparison treats them as equal. This divergence is
pre-existing and not fixed by this PR. The guard added here uses
singleton_values_equal(which normalizes signed zero vianormalize_float_zero_scalar, the same function used inphysical-expr-common/src/datum.rsfor runtime comparison) so that theEq + FALSEinfeasibility check is consistent with theEqoperator'sactual comparison semantics. The standalone
NOT(a = 0.0)overa ∈ [-0.0,-0.0]case remains unchanged before and after this PR (thebottom-up pass short-circuits before reaching this arm); only the nested
case, where top-down propagation forces
EqtoFALSE, is affected.Full local verification:
cargo fmt --all -- --check: cleancargo clippy --all-targets --all-features -- -D warnings: clean./dev/rust_lint.sh: exit 0AGENTS.md(withulimit -n 8192): passcargo test -p datafusion-physical-expr --lib: 1602 passed, 0 failed, 2 ignoredcargo test -p datafusion-physical-optimizer --lib: 33 passed, 0 failedcargo test -p datafusion-physical-plan --lib pruning: 120 passed, 0 failedcargo test -p datafusion-sqllogictest --test sqllogictests -- simplify_predicates: passAll listed lint and test gates, including the extended workspace suite, were
rerun on SHA
03e6d0ad0. A subsequent upstream rebase added only the dfbenchstatistics command and changed neither modified production blob; fmt and the
full physical-expr suite were rerun on final SHA
04acb8d40.Are there any user-facing changes?
No public API changes. Behavior fix:
analyze()no longer reportsNone(infeasible) for a satisfiable
NOT(a = b)predicate over an interval thatcontains
bbut is not equal to it. The only case that remains infeasibleunder
Eq + FALSEis when both children are singletons equal under thecomparison semantics (including
-0.0and+0.0).