Repository navigation
[lake] Let the lake tiering factory declare record batch write support - #5223
Open
mikepapadim wants to merge 1 commit into
Open
mikepapadim wants to merge 1 commit into
mikepapadim wants to merge 1 commit into
Conversation
The tiering job wrote ARROW log tables batch by batch only when the table's lake format was Paimon, by comparing DataLakeFormat.PAIMON in TieringSplitReader. A lake writer of another format that implements SupportsRecordBatchWrite was never given batches. LakeTieringFactory#supportsRecordBatchWrite(TableInfo), false by default, now declares it, the same way supportsPartitionMarkDone() does for mark-done. PaimonLakeTieringFactory returns true, so Paimon tiering is unchanged; other lakes keep the row path until they opt in. fluss-flink-common gets unshaded Arrow as a test dependency, so the record batch path can be tested there.
This was referenced Nov 13, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Generated-by: Claude Code (Claude Opus 5.5) following the guidelines
Purpose
Linked issue: close #5222
The tiering job sends an append-only ARROW log table to the lake batch by batch only when the table's lake format is Paimon.
TieringSplitReader#useRecordBatchPathcomparesDataLakeFormat.PAIMONdirectly. Because of that, a lake writer of another format that implementsSupportsRecordBatchWritenever receives batches. One example is the Iceberg writer in #4048.This PR has the lake tiering factory declare the capability instead, the same way
LakeTieringFactory#supportsPartitionMarkDone()already does for partition mark-done.Brief change log
LakeTieringFactory(@PublicEvolving): adddefault boolean supportsRecordBatchWrite(TableInfo tableInfo), which returnsfalse. When a factory returnstruefor a table, every lake writer it creates for that table must implementSupportsRecordBatchWrite.PaimonLakeTieringFactory: returnstrue, so Paimon tiering behaves as before.TieringSplitReader#useRecordBatchPath: asks the factory instead of comparing the lake format. The other conditions stay the same: unshaded Arrow is available, the table is append-only, and the log format is ARROW.fluss-flink-commonpom: addsarrow-vectorandarrow-memory-netty(${arrow.version}, the versionsfluss-commonandfluss-clientalready declare) with test scope only, so the record batch path can be tested in this module. Nothing that ships changes.Iceberg, Lance and Hudi keep the row path. Each can opt in by overriding the method once its writer supports batches. For Iceberg that would be after #4048.
Tests
New cases in
TieringSplitReaderTest, both using a testing factory whose writer accepts rows and record batches:testArrowLogTableUsesRecordBatchWriteWhenFactorySupportsIt: the factory returnstrue, and every bucket is written throughwrite(RecordBatch)with no row writes. This test fails on main, because the writer receives rows there.testArrowLogTableWritesRowsWhenFactoryDoesNotSupportRecordBatchWrite: the factory keeps the default, and the table is written row by row.Also run:
org.apache.fluss.flink.tiering.**tests influss-flink-common: 99 tests, 0 failuresorg.apache.fluss.lake.paimon.tiering.**tests andPaimonTieringITCaseinfluss-lake-paimon(Paimon's record batch path): 72 tests, 0 failuresAPI and Format
The change adds one default method to
LakeTieringFactory(@PublicEvolving). Existing implementations compile and behave unchanged. There is no storage format change.Documentation
None. The behaviour is unchanged for all existing lake formats.