fix(isthmus): convert extended expressions with the provider's converter - #1168
fix(isthmus): convert extended expressions with the provider's converter#1168alexandrefimov wants to merge 5 commits into
Conversation
ebd729c to
f26ef1a
Compare
nielspardon
left a comment
There was a problem hiding this comment.
Does an execution-context variable belong in an ExtendedExpression at all? algebra.proto ties it to ExecutionBehavior, which only Plan carries, so CURRENT_DATE now converts into a message with nowhere to record whether it is evaluated once or per record — and the same widening makes ROW_NUMBER() OVER (...) convert, untested.
The description needs a rewrite before merge: its subquery paragraph describes the javadoc that commit 2 replaced, and i + (SELECT 1) is not rejected by the validator — it converts, to a field reference.
The half-honoured provider getters behind getCallConverters() are #1048; this also unblocks dropping #1166's javadoc caveat that nothing configured on the provider reaches this path.
SqlExpressionToSubstrait built its own RexExpressionConverter from the scalar function converter alone. That varargs constructor takes the whole call-converter list, so the list on this path was just that one converter and CallConverters.defaults never reached it: CAST, CASE, the execution-context variables and the array and map constructors were unconvertible as extended expressions while converting fine inside a plan. Taking the converter from ConverterProvider.getRexExpressionConverter instead of assembling a second one here also means a provider that overrides that method is honoured on this path, as it already is when converting a plan, and that the window function converter and type converter are the provider's rather than null and TypeConverter.DEFAULT. Closes substrait-io#1165.
The extended-expression path builds its converter without a relation visitor, since an expression there stands alone. Saying that a subquery cannot appear is a claim about the input; dereferencing the missing visitor if one does is a NullPointerException. Reject it with a message instead, and let the Javadoc say what the converter does rather than what the input is.
The subquery path says so where the visitor is missing; the outer-reference path dereferenced it and threw a NullPointerException instead. Both are reachable on a converter built to convert expressions that stand alone.
callConversionFailureMessage converted each operand again to render its type, so on a converter with no relation visitor a subquery operand reported its own failure and the call the message is about was never named. The types come off the nodes now. visitOver dereferenced the window function converter the same way three of the four constructors leave null; it says so instead. The tests for both, and for the two guards the previous commits added, moved to SubqueryPlanTest, which has a RelBuilder to build the nodes with. The expressions the scalar function converter does not claim are pinned by what they convert into rather than by a proto round trip, which compares a message against itself; they also no longer feed the duplicate-column test, where they threw before the expression was parsed.
f26ef1a to
044c6cb
Compare
|
On the execution-context variable: the gap you point at is there. I would rather not refuse it here, though. The message the spec defines admits an Happy to raise the behaviour question upstream, or to refuse both here if you would rather this path stay to what an extended expression can fully describe -- your call, and I have no strong second preference. |
nielspardon
left a comment
There was a problem hiding this comment.
Thanks — the rest all look good.
Both arms of the conditional were the subquery's own type. The javadoc's account of what SQL reaches the guard was wrong twice over: a bare scalar subquery converts, to a reference to the base schema's first field, and an embedded one throws a raw NullPointerException rather than failing validation.
SqlExpressionToSubstraitbuilt its ownRexExpressionConverterfrom the scalar function converter alone. That varargs constructor takes the whole call-converter list, so the list on this path was just that one converter andCallConverters.defaults(...)never reached it —CAST,CASE, the execution-context variables and the array and map constructors were unconvertible as extended expressions while converting fine inside a plan.Taking the converter from
ConverterProvider.getRexExpressionConverterrather than assembling a second one here also means a provider that overrides that method is honoured on this path, as it already is when converting a plan, and that the window function converter and type converter are the provider's rather thannullandTypeConverter.DEFAULT. Only the call-converter half and the provider hook are pinned by tests; a markerTypeConvertercannot isolate the third, becauseSqlConverterBasealready uses the provider's type converter for the schema, so such a test passes against the old constructor too.getRexExpressionConverteris passednullfor the relation visitor: an expression that stands alone has no relation to visit. Nothing in this path produces a subquery to convert -- validating an extended expression that holds one throws inside Calcite first, a scalar subquery andIN/EXISTSalike -- so the converter names what is missing instead of dereferencing it.visitFieldAccessdid the same dereference for an outer reference, andvisitOverfor the window function converter that three of the four constructors leave null; both say so now.Naming what is missing has to stay out of the way of the message that names the call.
callConversionFailureMessageconverted each operand again to render its type, so an unconvertible call with a subquery operand reported the operand's failure rather than the call's; the types come off the nodes now.The expressions the scalar function converter does not claim are pinned by what they convert into --
CAST,IF_THEN,EXECUTION_CONTEXT_VARIABLE, twoLITERALs and, since the same wiring admits it, theWINDOW_FUNCTIONofROW_NUMBER() OVER (...). The round trip they used to share compares a proto against the one converting it back produces, which holds whatever the conversion made of them, and they no longer feed the duplicate-column test, where they threw before the expression was parsed.Closes #1165.