fix(isthmus): build a character literal from the type its conversion produced - #1171
Open
alexandrefimov wants to merge 2 commits into
Open
fix(isthmus): build a character literal from the type its conversion produced#1171alexandrefimov wants to merge 2 commits into
alexandrefimov wants to merge 2 commits into
Conversation
…produced LiteralConverter.convert derives the Substrait type twice: once through typeConverter.toSubstrait, which consults the UserTypeMapper, and again in the switch over the Calcite type name, which does not. The second derivation is the one the literal gets, so a mapped character column ended up with a schema of the mapped type and literals of the unmapped one, and VirtualTableScan rejected the relation. A null literal took its type from the first derivation and so already agreed, which is why only rows carrying a value diverged. The character branches now build from the type already derived, covering the three forms a Calcite character type can map to. A fixedchar literal carries no length of its own -- FixedCharLiteral derives it from the text -- so the text is padded to the declared width, which is also what CHAR(n) means. That half fixes a case needing no mapper at all: a LogicalValues row field wider than its literal, the shape substrait-io#1064 was reported with, was still rejected for character types. A mapping to anything with no character literal form is reported where it happens rather than as a schema mismatch further down. Closes substrait-io#1170.
…ength The fixedchar branch rejects a value wider than the type it is declared as, because the literal carries no length of its own and the text is the length. A varchar literal does carry one, and a value longer than it goes unchecked -- by this conversion and by the POJO.
alexandrefimov
force-pushed
the
issue-1170-literal-user-type
branch
from
August 28, 2026 07:41
026c0e3 to
cc63cda
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
LiteralConverter.convertderives the Substrait type twice. Once throughtypeConverter.toSubstrait(resultType), which consults theUserTypeMapper, and again in theswitchoverresultType.getSqlTypeName(), which does not — and the second derivation is the one the literal gets. So a mapped character column produced a schema of the mapped type and literals of the unmapped one, andVirtualTableScanrejected the relation:A null literal takes its type straight from the first derivation, so it already agreed; only rows carrying a value diverged. That asymmetry is what the
nullAndNonNullLiteralsCarryTheSameMappedTypetest pins.The character branches now build from the type already derived, which covers the three forms a Calcite character type can map to —
fixedchar,varcharandstring. It also removes a copy of thePRECISION_NOT_SPECIFIEDrule thatTypeConverteralready applies, so the two cannot drift apart again for these types.A
fixedcharliteral needed one thing more: it carries no length of its own, sinceExpression.FixedCharLiteralderives the type from its text. Building one from the declared type therefore means padding the text to the declared width, which is also whatCHAR(n)means —'a'in aCHAR(3)is'a '— and whatpadRightIfNeededalready does forBINARYa few lines down. A value longer than the width it is declared as is rejected rather than truncated.That half fixes a case needing no mapper at all. A
LogicalValuesrow field wider than its literal — the shape #1064 was reported with — was still rejected for character types: aCHAR(3)field holding'a'producedRow field type (FixedChar{length=1}) does not match schema field type (FixedChar{length=3}). Isthmus' own SQL path does not reach it, because Calcite pads the literal itself; a plan arriving from another planner does.What this does not cover, deliberately. A mapper that maps some other family still diverges the same way — mapping
INTEGERtoi64givesRow field type (I32) does not match schema field type (I64). Closing that generally means dispatching every branch on the Substrait type rather than the Calcite type name, which is a much larger change to this method, and it still would not reach a genuineType.UserDefined: there is no way to build a literal of one from aRexLiteralwithout knowing its encoding, so that needs a literal-side hook the interface does not have. The character family is the part that can be closed without deciding either of those. A mapping to anything with no character literal form is now reported where it happens, naming the type, rather than surfacing as a schema mismatch further down.Worth saying, since it bears on whether this use is meant to be supported at all:
UserTypeMapper.toSubstraitreturns aType, so mapping to a built-in is expressible, while its reverse takes aType.UserDefined. The consumer this came from maps Impala'sstring— reaching Calcite asVARCHAR(2147483647)— onto Substrait'sstring, which is a natural reading of the forward signature. If you would rather the mapper were restricted to user-defined types on both sides, that is a different fix and I am happy to do that one instead.Closes #1170.