Skip to content

MINOR: Reuse RLE decoder buffers for packed runs - #3749

Open
divjotarora wants to merge 2 commits into
apache:masterfrom
divjotarora:rle-decoder-buffer-reuse
Open

MINOR: Reuse RLE decoder buffers for packed runs#3749
divjotarora wants to merge 2 commits into
apache:masterfrom
divjotarora:rle-decoder-buffer-reuse

Conversation

@divjotarora

@divjotarora divjotarora commented Aug 26, 2026

Copy link
Copy Markdown
Contributor

Rationale for this change

RunLengthBitPackingHybridDecoder allocated integer and byte buffers for every packed run. This
decoder is used for repetition and definition levels and dictionary IDs, so repeated packed runs
created avoidable allocation and GC pressure.

What changes are included in this PR?

The decoder now grows and reuses its packed-run buffers, tracks the active value position
independently of buffer capacity, and reads directly from the input stream. It preserves existing
short-input zero padding and EOF behavior. A focused JMH benchmark and regression tests cover
smaller runs following larger runs and truncated final groups.

Are these changes tested?

Yes. The existing parquet-column tests pass. The new benchmark shows 27-35% speedup and reduced allocations:

> ./parquet-benchmarks/run.sh all \
    org.apache.parquet.benchmarks.RunLengthBitPackingHybridDecodingBenchmark \
    -wi 3 -i 5 -f 2 -prof gc

   Bit width                       Before ops/s                        After ops/s    Change    Before B/value    After B/value
  ━━━━━━━━━━━  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━  ━━━━━━━━  ━━━━━━━━━━━━━━━━  ━━━━━━━━━━━━━━━
           1    125,889,985.415 ± 1,443,739.608      173,074,464.937 ± 470,051.407    +37.5%    13.501 ± 0.001    0.002 ± 0.001
  ───────────  ─────────────────────────────────  ─────────────────────────────────  ────────  ────────────────  ───────────────
           8      130,416,053.717 ± 754,362.833      175,155,689.891 ± 499,245.748    +34.3%    13.751 ± 0.001    0.002 ± 0.001
  ───────────  ─────────────────────────────────  ─────────────────────────────────  ────────  ────────────────  ───────────────
          16    132,141,608.282 ± 5,365,801.318    167,817,248.033 ± 1,318,258.836    +27.0%    13.751 ± 0.797    0.002 ± 0.001

Are there any user-facing changes?

No. This is an internal performance improvement with no API or format changes.

@divjotarora
divjotarora marked this pull request as ready for review August 26, 2026 05:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant