Skip to content

feat(core)!: support offset_expr on window bounds (spec v0.102.0) - #1156

Open
anasik wants to merge 5 commits into
substrait-io:mainfrom
anasik:bump-102
Open

feat(core)!: support offset_expr on window bounds (spec v0.102.0)#1156
anasik wants to merge 5 commits into
substrait-io:mainfrom
anasik:bump-102

Conversation

@anasik

@anasik anasik commented Aug 25, 2026

Copy link
Copy Markdown
Contributor
  • Preceding/Following now prefer offset_expr over the deprecated offset when set, per the spec's precedence rule
  • A literal offset still populates the legacy offset field too, for producers that need backward compatibility
  • Reject BOUNDS_TYPE_UNSPECIFIED when a bound is CurrentRow, Preceding, or Following; still allowed when both bounds are Unbounded (needed for Spark's UnspecifiedFrame)
  • Update isthmus and Spark conversions, since both only support literal offsets in their own window frame APIs
  • Bumping to spec v0.102.0 also exposes negate(decimal<P,S>) via FunctionMappings's UNARY_MINUSnegate mapping, so SELECT -decimal_col may now resolve a different function; untested by any isthmus test

Closes #1143

BREAKING CHANGE: WindowBound.Preceding/Following.offset() now returns Expression instead of long. ExpressionProtoConverter.BoundConverter's public convert(WindowBound) is now an instance method (reachable via ExpressionProtoConverter.toProto(WindowBound)) instead of a static utility. A plan with bounds_type unset (BOUNDS_TYPE_UNSPECIFIED) alongside a bound that isn't Unbounded no longer parses; it must set bounds_type to ROWS or RANGE, or use Unbounded on both sides.

@nielspardon

Copy link
Copy Markdown
Member

Can you check ExpressionCopyOnWriteVisitor, RelCopyOnWriteVisitor and OuterReferenceConverter? I think they need to be changed to handle copying / handling references in the expression trees in offset_expr.

@anasik

anasik commented Aug 26, 2026

Copy link
Copy Markdown
Contributor Author

Can you check ExpressionCopyOnWriteVisitor, RelCopyOnWriteVisitor and OuterReferenceConverter? I think they need to be changed to handle copying / handling references in the expression trees in offset_expr.

check now, @nielspardon

@nielspardon nielspardon left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the copy-on-write fix — that part looks good.

The main thing left is on the producer side, in isthmus WindowBoundConverter. It still turns every Calcite offset into a long, so RANGE BETWEEN 1.5 PRECEDING silently becomes 1 PRECEDING (DECIMAL is in SqlTypeName.EXACT_TYPES), and it can never emit an offset_expr for a decimal or interval offset. Since the deprecated offset field is an int64 and has to stay one, I'd build the bound from the offset's own type: integral literals become an i64 offset_expr, which the writer still mirrors into offset; everything else sets only offset_expr. The same place should map a zero offset to WindowBound.CURRENT_ROW, since the spec says zero isn't an offset. One caveat for RANGE: an integral offset needs the ordering column's type, not a blanket i64, or add(T, D) -> T isn't defined.

Three body-level ones: please add an offset expression to the RelSamples ConsistentPartitionWindow sample so the shared round-trip tests cover one; the BREAKING CHANGE: footer should also mention the removed public BoundConverter.convert(WindowBound) and that plans with bounds_type unset no longer parse; and ExpressionRexConverter.isRowMode (:760) still rejects UNSPECIFIED even when both bounds are unbounded, which checkBoundsType now allows.

The rest is inline. Suggestions are written against 62e2aa9e but not compiled — worth a local build before committing them.

Comment thread core/src/main/java/io/substrait/expression/WindowBound.java
Comment thread core/src/main/java/io/substrait/expression/proto/ProtoExpressionConverter.java Outdated
Comment thread core/src/main/java/io/substrait/expression/WindowBound.java
Comment thread isthmus/src/main/java/io/substrait/isthmus/expression/ExpressionRexConverter.java Outdated
Comment thread gradle/libs.versions.toml

@nielspardon nielspardon left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks good — the producer side is better than what I suggested; keying the offset type off the ordering expression is the right call, and WindowBoundConverterTest covers the cases individually.

Three follow-ups, none of which I'd block on — see inline.

Two notes outside the diff: the bullet "Update isthmus and Spark conversions, since both only support literal offsets in their own window frame APIs" isn't true for isthmus any more, and the body becomes the commit message. And WindowBoundConverter.toWindowBound is public and gained three parameters — probably fine to leave out of the footer given it's effectively isthmus-internal, but worth a thought.

@anasik
anasik requested a review from nielspardon August 28, 2026 11:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Update to Substrait v0.102.0

2 participants