[qzyu999]

Summary

Implements SupportsRecordBatchWrite for the Iceberg lake writer, enabling direct Arrow batch writing for append-only log tables during tiering. This closes the performance gap with Paimon's batch write path (landed in #3417/#3418).

Closes #4047

Changes

• IcebergLakeWriter now implements SupportsRecordBatchWrite
• New IcebergArrowBatchHelper reads Arrow FieldVector columns directly and produces Iceberg GenericRecord objects enriched with system columns (__bucket, __offset, __timestamp), then writes through the existing TaskWriter<Record> pipeline
• Added arrow-vector and arrow-memory-netty as provided dependencies (same pattern as fluss-lake-paimon)
• Only enabled for append-only tables (same restriction as Paimon)

Motivation

The Paimon lake writer implements SupportsRecordBatchWrite which takes raw Arrow batches from Fluss internal format and writes them to Parquet via paimon-arrow. The Iceberg path previously wrote row-by-row through GenericRecord objects wrapping LogRecord, requiring per-record binary deserialization.

With this change, the Iceberg tiering path can read directly from Arrow columnar memory, avoiding the LogRecord -> InternalRow -> FlussRecordAsIcebergRecord conversion chain for each record.

Design

The helper creates typed ArrowFieldExtractor lambdas per column based on the Fluss DataType, then iterates rows reading from the pre-fetched FieldVector array. This preserves the existing TaskWriter pipeline (file rolling, partition key generation, Iceberg metrics tracking) while eliminating the input-side deserialization cost.

Supported types: boolean, tinyint, smallint, int, bigint, float, double, string, char, binary, bytes, decimal, date, time, timestamp, timestamp_ltz. Complex types (array, map, row) have placeholder extractors and are left as follow-up work.

Test Plan

• Compilation verified: mvn compile -pl fluss-lake/fluss-lake-iceberg -am -DskipTests
• Formatting: mvn spotless:check passes
• Checkstyle: mvn checkstyle:check passes
• Existing tests: IcebergLakeCatalogTest, IcebergWriteResultSerializerTest, FlussToIcebergPredicateConverterTest, IcebergConfigurationTest all pass
• Integration test coverage needed: a dedicated test writing ArrowRecordBatch through the new path (follow-up)

[github-actions]

This PR has been marked as stale because it has had no activity for 60 days. It will be closed in 10 days unless it receives a new comment, commit or other update.

[qzyu999]

bump

[mikepapadim]

@qzyu999 Heads-up: with this PR alone, Iceberg tables would still be tiered row by row. TieringSplitReader#useRecordBatchPath only takes the SupportsRecordBatchWrite route when the lake format is Paimon.

I've opened #5223 (issue #5222) to let the tiering factory declare it instead, through LakeTieringFactory#supportsRecordBatchWrite(TableInfo). It's about 20 lines and leaves Paimon unchanged. Once that's in, IcebergLakeTieringFactory can return true for the tables this PR supports. Separately, this PR targets fluss-lake-iceberg, which has since been split into fluss-lake-iceberg-common and fluss-lake-iceberg-1.12, so it will need a rebase.

[qzyu999]

Hi @mikepapadim, thanks for taking a look, unfortunately this looks heavily bottlenecked by apache/parquet-java#3734, not sure if you're willing to take a look to help push it forward.

[mikepapadim]

Thanks for the pointer. I benchmarked apache/parquet-java#3734 on the data from the tiering profile: TPC-H lineitem-like, 19 columns with Fluss's system columns, 16384-row Arrow batches, ZSTD. I built it against parquet 1.18.2. As is, it writes 512 MB in 2.57 s, versus 7.3 s for Fluss's Iceberg row path and 3.2 s for the Paimon writer. I also found a page-sizing issue in it. With that fixed, it takes 1.73 s and the files are 19% smaller. I've left the details on apache/parquet-java#3734 and opened the fix against your branch (qzyu999/parquet-java#1). The Fluss-side numbers are in #5227.

On the Fluss side, #5223 makes the batch path reachable for Iceberg, and #5226 proposes a way to plug in an Arrow file writer. apache/parquet-java#3734 would be the obvious CPU implementation for it, so #4048 wouldn't have to wait on a parquet-java release to get the columnar path wired up.

[qzyu999]

Thanks for benchmarking this, @mikepapadim! I merged your page-sizing fix into the branch for apache/parquet-java#3734. For this PR, I’ll wait for #5223 to land, then rebase and wire up the Iceberg factory so append-only Arrow tables can use the batch path.