Skip to content

fix(isthmus): convert extended expressions with the provider's converter - #1168

Open
alexandrefimov wants to merge 5 commits into
substrait-io:mainfrom
alexandrefimov:issue-1165-extended-expression-converters
Open

fix(isthmus): convert extended expressions with the provider's converter#1168
alexandrefimov wants to merge 5 commits into
substrait-io:mainfrom
alexandrefimov:issue-1165-extended-expression-converters

Conversation

@alexandrefimov

@alexandrefimov alexandrefimov commented Aug 26, 2026

Copy link
Copy Markdown
Contributor

SqlExpressionToSubstrait built its own RexExpressionConverter from the scalar function converter alone. That varargs constructor takes the whole call-converter list, so the list on this path was just that one converter and CallConverters.defaults(...) never reached it — CAST, CASE, the execution-context variables and the array and map constructors were unconvertible as extended expressions while converting fine inside a plan.

Taking the converter from ConverterProvider.getRexExpressionConverter rather than assembling a second one here also means a provider that overrides that method is honoured on this path, as it already is when converting a plan, and that the window function converter and type converter are the provider's rather than null and TypeConverter.DEFAULT. Only the call-converter half and the provider hook are pinned by tests; a marker TypeConverter cannot isolate the third, because SqlConverterBase already uses the provider's type converter for the schema, so such a test passes against the old constructor too.

getRexExpressionConverter is passed null for the relation visitor: an expression that stands alone has no relation to visit. Nothing in this path produces a subquery to convert -- validating an extended expression that holds one throws inside Calcite first, a scalar subquery and IN/EXISTS alike -- so the converter names what is missing instead of dereferencing it. visitFieldAccess did the same dereference for an outer reference, and visitOver for the window function converter that three of the four constructors leave null; both say so now.

Naming what is missing has to stay out of the way of the message that names the call. callConversionFailureMessage converted each operand again to render its type, so an unconvertible call with a subquery operand reported the operand's failure rather than the call's; the types come off the nodes now.

The expressions the scalar function converter does not claim are pinned by what they convert into -- CAST, IF_THEN, EXECUTION_CONTEXT_VARIABLE, two LITERALs and, since the same wiring admits it, the WINDOW_FUNCTION of ROW_NUMBER() OVER (...). The round trip they used to share compares a proto against the one converting it back produces, which holds whatever the conversion made of them, and they no longer feed the duplicate-column test, where they threw before the expression was parsed.

Closes #1165.

@alexandrefimov
alexandrefimov force-pushed the issue-1165-extended-expression-converters branch from ebd729c to f26ef1a Compare August 28, 2026 07:37

@nielspardon nielspardon left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does an execution-context variable belong in an ExtendedExpression at all? algebra.proto ties it to ExecutionBehavior, which only Plan carries, so CURRENT_DATE now converts into a message with nowhere to record whether it is evaluated once or per record — and the same widening makes ROW_NUMBER() OVER (...) convert, untested.

The description needs a rewrite before merge: its subquery paragraph describes the javadoc that commit 2 replaced, and i + (SELECT 1) is not rejected by the validator — it converts, to a field reference.

The half-honoured provider getters behind getCallConverters() are #1048; this also unblocks dropping #1166's javadoc caveat that nothing configured on the provider reaches this path.

Comment thread isthmus/src/test/java/io/substrait/isthmus/SimpleExtendedExpressionsTest.java Outdated
Comment thread isthmus/src/test/java/io/substrait/isthmus/SimpleExtendedExpressionsTest.java Outdated
Comment thread isthmus/src/test/java/io/substrait/isthmus/SimpleExtendedExpressionsTest.java Outdated
Comment thread isthmus/src/test/java/io/substrait/isthmus/SimpleExtendedExpressionsTest.java Outdated
Comment thread isthmus/src/main/java/io/substrait/isthmus/ConverterProvider.java Outdated
alexandrefimov and others added 4 commits August 28, 2026 12:17
SqlExpressionToSubstrait built its own RexExpressionConverter from the scalar
function converter alone. That varargs constructor takes the whole
call-converter list, so the list on this path was just that one converter and
CallConverters.defaults never reached it: CAST, CASE, the execution-context
variables and the array and map constructors were unconvertible as extended
expressions while converting fine inside a plan.

Taking the converter from ConverterProvider.getRexExpressionConverter instead
of assembling a second one here also means a provider that overrides that
method is honoured on this path, as it already is when converting a plan, and
that the window function converter and type converter are the provider's
rather than null and TypeConverter.DEFAULT.

Closes substrait-io#1165.
The extended-expression path builds its converter without a relation
visitor, since an expression there stands alone. Saying that a subquery
cannot appear is a claim about the input; dereferencing the missing visitor
if one does is a NullPointerException. Reject it with a message instead, and
let the Javadoc say what the converter does rather than what the input is.
The subquery path says so where the visitor is missing; the outer-reference
path dereferenced it and threw a NullPointerException instead. Both are
reachable on a converter built to convert expressions that stand alone.
callConversionFailureMessage converted each operand again to render its type,
so on a converter with no relation visitor a subquery operand reported its own
failure and the call the message is about was never named. The types come off
the nodes now.

visitOver dereferenced the window function converter the same way three of the
four constructors leave null; it says so instead. The tests for both, and for
the two guards the previous commits added, moved to SubqueryPlanTest, which has
a RelBuilder to build the nodes with.

The expressions the scalar function converter does not claim are pinned by what
they convert into rather than by a proto round trip, which compares a message
against itself; they also no longer feed the duplicate-column test, where they
threw before the expression was parsed.
@alexandrefimov
alexandrefimov force-pushed the issue-1165-extended-expression-converters branch from f26ef1a to 044c6cb Compare August 28, 2026 09:25
@alexandrefimov

Copy link
Copy Markdown
Contributor Author

On the execution-context variable: the gap you point at is there. ExtendedExpression declares version, extension_urns, extensions, referred_expr, base_schema, advanced_extensions and expected_type_urls -- execution_behavior is on Plan alone -- so a CURRENT_DATE in an extended expression carries no statement of whether it is evaluated once or per record.

I would rather not refuse it here, though. The message the spec defines admits an Expression, and this conversion has no rule the plan path does not; refusing would make the extended-expression path narrower than the message it produces, on a question the spec has not answered. What the wiring can do is stop being silent about it, which is what the split test now does: CURRENT_DATE is pinned as EXECUTION_CONTEXT_VARIABLE and ROW_NUMBER() OVER (...) as WINDOW_FUNCTION, so neither arrives untested and both are visible to whoever settles the semantics.

Happy to raise the behaviour question upstream, or to refuse both here if you would rather this path stay to what an extended expression can fully describe -- your call, and I have no strong second preference.

@nielspardon nielspardon left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks — the rest all look good.

Comment thread isthmus/src/test/java/io/substrait/isthmus/SubqueryPlanTest.java Outdated
Both arms of the conditional were the subquery's own type. The javadoc's account
of what SQL reaches the guard was wrong twice over: a bare scalar subquery
converts, to a reference to the base schema's first field, and an embedded one
throws a raw NullPointerException rather than failing validation.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

isthmus: SqlExpressionToSubstrait skips the default call converters, so CAST and CASE do not convert as extended expressions

2 participants