Skip to content

fix(isthmus): build a character literal from the type its conversion produced - #1171

Open
alexandrefimov wants to merge 2 commits into
substrait-io:mainfrom
alexandrefimov:issue-1170-literal-user-type
Open

fix(isthmus): build a character literal from the type its conversion produced#1171
alexandrefimov wants to merge 2 commits into
substrait-io:mainfrom
alexandrefimov:issue-1170-literal-user-type

Conversation

@alexandrefimov

Copy link
Copy Markdown
Contributor

LiteralConverter.convert derives the Substrait type twice. Once through typeConverter.toSubstrait(resultType), which consults the UserTypeMapper, and again in the switch over resultType.getSqlTypeName(), which does not — and the second derivation is the one the literal gets. So a mapped character column produced a schema of the mapped type and literals of the unmapped one, and VirtualTableScan rejected the relation:

SELECT * FROM (VALUES ('a'), ('b')) AS v(c)     -- with CHAR/VARCHAR mapped to string
-- Row field type (FixedChar{nullable=false, length=1})
--   does not match schema field type (Str{nullable=false})

A null literal takes its type straight from the first derivation, so it already agreed; only rows carrying a value diverged. That asymmetry is what the nullAndNonNullLiteralsCarryTheSameMappedType test pins.

The character branches now build from the type already derived, which covers the three forms a Calcite character type can map to — fixedchar, varchar and string. It also removes a copy of the PRECISION_NOT_SPECIFIED rule that TypeConverter already applies, so the two cannot drift apart again for these types.

A fixedchar literal needed one thing more: it carries no length of its own, since Expression.FixedCharLiteral derives the type from its text. Building one from the declared type therefore means padding the text to the declared width, which is also what CHAR(n) means — 'a' in a CHAR(3) is 'a ' — and what padRightIfNeeded already does for BINARY a few lines down. A value longer than the width it is declared as is rejected rather than truncated.

That half fixes a case needing no mapper at all. A LogicalValues row field wider than its literal — the shape #1064 was reported with — was still rejected for character types: a CHAR(3) field holding 'a' produced Row field type (FixedChar{length=1}) does not match schema field type (FixedChar{length=3}). Isthmus' own SQL path does not reach it, because Calcite pads the literal itself; a plan arriving from another planner does.

What this does not cover, deliberately. A mapper that maps some other family still diverges the same way — mapping INTEGER to i64 gives Row field type (I32) does not match schema field type (I64). Closing that generally means dispatching every branch on the Substrait type rather than the Calcite type name, which is a much larger change to this method, and it still would not reach a genuine Type.UserDefined: there is no way to build a literal of one from a RexLiteral without knowing its encoding, so that needs a literal-side hook the interface does not have. The character family is the part that can be closed without deciding either of those. A mapping to anything with no character literal form is now reported where it happens, naming the type, rather than surfacing as a schema mismatch further down.

Worth saying, since it bears on whether this use is meant to be supported at all: UserTypeMapper.toSubstrait returns a Type, so mapping to a built-in is expressible, while its reverse takes a Type.UserDefined. The consumer this came from maps Impala's string — reaching Calcite as VARCHAR(2147483647) — onto Substrait's string, which is a natural reading of the forward signature. If you would rather the mapper were restricted to user-defined types on both sides, that is a different fix and I am happy to do that one instead.

Closes #1170.

alexandrefimov and others added 2 commits August 28, 2026 10:39
…produced

LiteralConverter.convert derives the Substrait type twice: once through
typeConverter.toSubstrait, which consults the UserTypeMapper, and again in the
switch over the Calcite type name, which does not. The second derivation is the
one the literal gets, so a mapped character column ended up with a schema of the
mapped type and literals of the unmapped one, and VirtualTableScan rejected the
relation. A null literal took its type from the first derivation and so already
agreed, which is why only rows carrying a value diverged.

The character branches now build from the type already derived, covering the
three forms a Calcite character type can map to. A fixedchar literal carries no
length of its own -- FixedCharLiteral derives it from the text -- so the text is
padded to the declared width, which is also what CHAR(n) means. That half fixes
a case needing no mapper at all: a LogicalValues row field wider than its
literal, the shape substrait-io#1064 was reported with, was still rejected for character
types. A mapping to anything with no character literal form is reported where it
happens rather than as a schema mismatch further down.

Closes substrait-io#1170.
…ength

The fixedchar branch rejects a value wider than the type it is declared as,
because the literal carries no length of its own and the text is the length.
A varchar literal does carry one, and a value longer than it goes unchecked --
by this conversion and by the POJO.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

isthmus: a UserTypeMapper is applied to a relation's schema but dropped from its literals, so the two disagree

1 participant