Skip to content

perf(parquet/internal/encoding): speed up ALP decoding - #1552

Open
zeroshade wants to merge 2 commits into
apache:mainfrom
zeroshade:alp-decode-perf
Open

zeroshade wants to merge 2 commits into
apache:mainfrom
zeroshade:alp-decode-perf

Conversation

@zeroshade

@zeroshade zeroshade commented Nov 13, 2026 •

Copy link
Copy Markdown
Member

Rationale for this change

Decoding an ALP vector makes four passes over its values, and copies the packed bytes once before the first:

  • BitReader.GetBatch unpacks the deltas. The vectorized unpackers behind it read through an io.Reader, so the packed bytes are copied into a pooled buffer first, and the unpacked uint32s are then widened to uint64.
  • A loop generic over the float type adds the frame of reference to each delta and scales it into the decoder's buffer.
  • The exceptions are patched in through a generic helper that looks the type up for every value.
  • The values are copied to the caller.

Deltas wider than 32 bits never reach the vectorized unpackers: GetBatch reads them one value at a time.

On amd64 the AVX2 unpacker is itself slow at most widths. At any width that does not divide 32, it builds each register from scalar inserts (vpinsrd), which Intel runs on one port: unpacking 1,024 deltas of 14 bits takes about 0.69µs, against 0.13µs at 8 bits. Scaling them, one CVTSI2SDQ and two MULSD at a time, takes about 1µs more.

What changes are included in this PR?

  • utils.Unpack32Vectorized runs the NEON or AVX2 unpacker on a byte slice directly. It unpacks nothing on a platform without one.
  • Deltas of up to 32 bits are unpacked with it straight from the page, and scaled in a second pass written for each type. Wider deltas, and every delta on a platform without a vectorized unpacker, are unpacked and scaled in one pass, with one 8-byte load per delta.
  • On amd64 with AVX2, deltas of up to 25 bits are decoded in one pass by a kernel in assembly. A group of 8 deltas spans whole bytes, so the kernel loads it with two 16-byte loads, moves the 4 bytes holding each delta into its own 32-bit lane with a shuffle, shifts and masks it, adds the frame of reference, converts, scales, and stores the 8 values. The shuffles, shifts and masks for each width are built once.
  • AVX2 cannot convert an int64 to a double. A DOUBLE vector whose integers all lie in [-2^51, 2^51) converts with one integer addition and one subtraction. Any other converts the upper 48 bits and the lower 16 apart, both exactly, and rounds once in the final addition. Both give the double CVTSI2SDQ gives, and the scaling is two multiplications, as in Go, so a page decodes to the same bits on every platform.
  • Wider deltas, and every delta on arm64, go through the loops above.
  • A read that starts at a vector and covers all of it is decoded into the caller's slice rather than the decoder's buffer.
  • Exceptions are patched by a loop written for each type.
  • A vector of zero bit width is filled with its one value.

The unpacking loops load a few bytes past a vector's packed deltas. Those bytes are the vector's exceptions or the next vector. Only the last vector of a page can be without them, and it is unpacked from a padded copy. The AVX2 kernel loads up to 16 bytes past the middle of a group, and decodes only the groups whose loads stay inside the page; the loops above decode the rest.

Timings on an Apple M4 (darwin/arm64), which this PR's AVX2 kernel does not change. The machine was busy with other work, so main and this branch were run in alternating short turns, 50 samples each. The spread is wide, and the absolute times are slower than on an idle machine, but both sides ran under the same conditions:

                                               │     main      │                  PR                  │
                                               │    sec/op     │    sec/op     vs base                │
ALPDecodingFloat32/no_exceptions/len_1024-10     1258.0n ± 12%   599.8n ± 10%  -52.32% (p=0.000 n=50)
ALPDecodingFloat32/no_exceptions/len_65536-10     84.73µ ± 16%   35.62µ ± 17%  -57.96% (p=0.000 n=50)
ALPDecodingFloat32/10%_exceptions/len_1024-10    1607.0n ± 11%   593.0n ± 14%  -63.10% (n=50)
ALPDecodingFloat32/10%_exceptions/len_65536-10   111.59µ ±  6%   40.05µ ± 11%  -64.12% (p=0.000 n=50)
ALPDecodingFloat32/all_exceptions/len_1024-10    4287.0n ± 15%   809.5n ± 12%  -81.12% (p=0.000 n=50)
ALPDecodingFloat32/all_exceptions/len_65536-10   260.66µ ± 13%   51.08µ ± 14%  -80.40% (p=0.000 n=50)
ALPDecodingFloat64/no_exceptions/len_1024-10     1335.5n ± 13%   591.8n ± 16%  -55.69% (p=0.000 n=50)
ALPDecodingFloat64/no_exceptions/len_65536-10     93.38µ ±  9%   36.95µ ± 10%  -60.43% (p=0.000 n=50)
ALPDecodingFloat64/10%_exceptions/len_1024-10    1741.5n ±  8%   651.2n ± 17%  -62.61% (n=50)
ALPDecodingFloat64/10%_exceptions/len_65536-10   103.73µ ± 13%   42.27µ ± 15%  -59.25% (p=0.000 n=50)
ALPDecodingFloat64/all_exceptions/len_1024-10    4158.5n ± 14%   736.5n ± 12%  -82.29% (n=50)
ALPDecodingFloat64/all_exceptions/len_65536-10   255.89µ ± 21%   52.04µ ± 14%  -79.66% (p=0.000 n=50)
ALPDecodingFloat64Batched/len_1024-10            1732.5n ± 16%   857.5n ± 13%  -50.50% (p=0.000 n=50)
ALPDecodingFloat64Batched/len_65536-10           109.50µ ± 13%   55.60µ ± 13%  -49.23% (p=0.000 n=50)
ALPDecodingFloat64Wide/len_1024-10                8.127µ ± 15%   1.096µ ±  8%  -86.51% (n=50)
ALPDecodingFloat64Wide/len_65536-10              519.79µ ±  9%   63.99µ ± 10%  -87.69% (p=0.000 n=50)
geomean                                           19.42µ         5.795µ        -70.16%

Wide is a benchmark this PR adds, for deltas of about 50 bits.

Timings on an Intel Core Ultra 7 155H (linux/amd64, AVX2), Go 1.27.4. main is this PR's base, fc5d5ce, with this PR's benchmark file, and PR is this PR without the AVX2 kernel. Each binary was pinned to one performance core, and the three were run in alternating turns, 30 samples each. Other work was running on the machine, with a load average that ranged from about 2 to 18 on 22 threads, so the spread is wide:

                                            │     main      │                  PR                   │                 AVX2                 │
                                            │    sec/op     │    sec/op      vs base                │    sec/op     vs base                │
ALPDecodingFloat32/no_exceptions/len_1024     1542.0n ± 19%    892.2n ± 18%  -42.14% (p=0.000 n=30)   212.8n ± 15%  -86.20% (p=0.000 n=30)
ALPDecodingFloat32/no_exceptions/len_65536    101.57µ ± 17%    56.07µ ± 12%  -44.80% (p=0.000 n=30)   11.63µ ± 16%  -88.55% (p=0.000 n=30)
ALPDecodingFloat32/10%_exceptions/len_1024    1989.0n ± 15%    945.0n ± 13%  -52.49% (p=0.000 n=30)   227.5n ± 12%  -88.56% (p=0.000 n=30)
ALPDecodingFloat32/10%_exceptions/len_65536   128.25µ ±  9%    59.54µ ± 12%  -53.58% (p=0.000 n=30)   13.82µ ± 19%  -89.22% (p=0.000 n=30)
ALPDecodingFloat32/all_exceptions/len_1024    4901.0n ± 14%    646.5n ± 11%  -86.81% (p=0.000 n=30)   636.9n ± 17%  -87.00% (p=0.000 n=30)
ALPDecodingFloat32/all_exceptions/len_65536   312.37µ ±  8%    42.89µ ± 13%  -86.27% (p=0.000 n=30)   39.93µ ± 20%  -87.22% (p=0.000 n=30)
ALPDecodingFloat64/no_exceptions/len_1024     1462.0n ± 10%    906.2n ± 13%  -38.02% (p=0.000 n=30)   323.6n ± 17%  -77.86% (p=0.000 n=30)
ALPDecodingFloat64/no_exceptions/len_65536    103.54µ ±  9%    56.68µ ± 13%  -45.25% (p=0.000 n=30)   19.08µ ± 18%  -81.57% (p=0.000 n=30)
ALPDecodingFloat64/10%_exceptions/len_1024    1896.0n ±  8%    958.4n ± 11%  -49.45% (p=0.000 n=30)   360.8n ± 17%  -80.97% (p=0.000 n=30)
ALPDecodingFloat64/10%_exceptions/len_65536   124.38µ ±  8%    59.26µ ± 11%  -52.36% (p=0.000 n=30)   22.56µ ± 15%  -81.87% (p=0.000 n=30)
ALPDecodingFloat64/all_exceptions/len_1024    4784.0n ± 12%    734.0n ± 15%  -84.66% (p=0.000 n=30)   685.0n ± 15%  -85.68% (p=0.000 n=30)
ALPDecodingFloat64/all_exceptions/len_65536   307.59µ ± 14%    48.23µ ± 11%  -84.32% (p=0.000 n=30)   45.14µ ± 13%  -85.32% (p=0.000 n=30)
ALPDecodingFloat64Batched/len_1024            2011.0n ± 11%   1138.5n ± 14%  -43.39% (p=0.000 n=30)   551.3n ± 11%  -72.59% (p=0.000 n=30)
ALPDecodingFloat64Batched/len_65536           130.90µ ± 12%    70.85µ ± 13%  -45.87% (p=0.000 n=30)   32.67µ ± 15%  -75.04% (p=0.000 n=30)
ALPDecodingFloat64Wide/len_1024                8.655µ ±  8%    1.700µ ± 14%  -80.37% (p=0.000 n=30)   1.619µ ± 18%  -81.30% (p=0.000 n=30)
ALPDecodingFloat64Wide/len_65536               589.9µ ± 12%    109.6µ ± 17%  -81.42% (p=0.000 n=30)   101.4µ ± 13%  -82.81% (p=0.000 n=30)
geomean                                        22.50µ          7.589µ        -66.27%                  3.630µ        -83.87%

The benchmarks' DOUBLE values take the short conversion. On 1,024 values the long one took about 0.70µs, against 1.1µs to 1.7µs without the kernel, in a separate run.

all_exceptions vectors have a bit width of 0, so the kernel does not change them, nor Wide, whose deltas are about 50 bits.

Are these changes tested?

Yes, by new tests of the loops and of the decoder, and by the existing ALP tests. TestALPDecodePacked compares the decoded bits with Go's own conversion at every width, at lengths either side of the groups of 4, 8 and 32 the loops work in, and with frames of reference either side of each end of the range the short DOUBLE conversion takes. TestALPDecodePackedGuardPage decodes deltas placed against a page that cannot be read, at every width. Both also pass with -tags noasm and on 386.

Are there any user-facing changes?

No.

Unpack deltas of up to 32 bits with the vectorized unpackers straight
from the page, and wider ones with one load per delta; scale them in
loops written for each type; decode whole vectors into the caller's
slice; and patch exceptions with a typed loop.
@zeroshade
zeroshade requested review from lidavidm, raulcd and rok November 13, 2026 21:50
@lidavidm

Copy link
Copy Markdown
Member

I have no amd64 timings: the AVX2 path is covered by the tests, not by a benchmark run.

It would be useful to benchmark on amd64 as well

@zeroshade

Copy link
Copy Markdown
Member Author

Ran them on an Intel Core Ultra 7 155H (linux/amd64, AVX2), Go 1.27.4. main is this PR's base, fc5d5ce, with this PR's benchmark file. Each binary was pinned to one performance core, and the two were run in alternating turns, 30 samples each:

                                            │     main     │                 PR                  │
                                            │    sec/op    │   sec/op     vs base                │
ALPDecodingFloat32/no_exceptions/len_1024      2.242µ ± 2%   1.241µ ± 2%  -44.65% (p=0.000 n=30)
ALPDecodingFloat32/no_exceptions/len_65536    147.28µ ± 3%   76.44µ ± 3%  -48.10% (p=0.000 n=30)
ALPDecodingFloat32/10%_exceptions/len_1024     2.844µ ± 9%   1.269µ ± 3%  -55.38% (p=0.000 n=30)
ALPDecodingFloat32/10%_exceptions/len_65536   182.17µ ± 4%   79.47µ ± 3%  -56.37% (p=0.000 n=30)
ALPDecodingFloat32/all_exceptions/len_1024    7000.5n ± 4%   869.6n ± 2%  -87.58% (p=0.000 n=30)
ALPDecodingFloat32/all_exceptions/len_65536   457.65µ ± 3%   57.11µ ± 2%  -87.52% (p=0.000 n=30)
ALPDecodingFloat64/no_exceptions/len_1024      2.158µ ± 3%   1.224µ ± 4%  -43.27% (p=0.000 n=30)
ALPDecodingFloat64/no_exceptions/len_65536    144.36µ ± 3%   76.83µ ± 3%  -46.78% (p=0.000 n=30)
ALPDecodingFloat64/10%_exceptions/len_1024     2.751µ ± 2%   1.264µ ± 3%  -54.04% (p=0.000 n=30)
ALPDecodingFloat64/10%_exceptions/len_65536   183.14µ ± 2%   80.77µ ± 1%  -55.89% (p=0.000 n=30)
ALPDecodingFloat64/all_exceptions/len_1024    6755.5n ± 3%   983.0n ± 3%  -85.45% (p=0.000 n=30)
ALPDecodingFloat64/all_exceptions/len_65536   436.78µ ± 3%   66.94µ ± 7%  -84.67% (p=0.000 n=30)
ALPDecodingFloat64Batched/len_1024             2.959µ ± 6%   1.591µ ± 5%  -46.21% (p=0.000 n=30)
ALPDecodingFloat64Batched/len_65536            191.6µ ± 5%   100.0µ ± 5%  -47.81% (p=0.000 n=30)
ALPDecodingFloat64Wide/len_1024               13.265µ ± 8%   2.336µ ± 5%  -82.39% (p=0.000 n=30)
ALPDecodingFloat64Wide/len_65536               871.9µ ± 4%   149.8µ ± 5%  -82.81% (p=0.000 n=30)
geomean                                        32.65µ        10.33µ       -68.37%

The description now includes them as well.

…ith AVX2 in one pass

On amd64 the two passes cost about the same: the AVX2 unpacker builds a
register from scalar inserts at any width that does not divide 32, and
the scaling loop converts one value at a time.

A kernel now unpacks a group of 8 deltas with two loads, a shuffle, a
shift and a mask, adds the frame of reference, converts and scales, and
stores the 8 values. AVX2 cannot convert an int64 to a double, so a
DOUBLE vector converts its upper 48 and lower 16 bits apart and rounds
once, or, when its integers all lie in [-2^51, 2^51), with one addition
and one subtraction. Both round as CVTSI2SDQ does.
@zeroshade

Copy link
Copy Markdown
Member Author

It would be useful to benchmark on amd64 as well

The amd64 run showed the AVX2 path gained less than NEON. At most widths the AVX2 unpacker builds each register from scalar inserts, and the scaling pass converts one value at a time. b40e7ac adds an AVX2 kernel that unpacks, converts and scales 8 deltas of up to 25 bits at a time, and decodes to the same bits as Go's conversion.

Against this PR without it, on the same Intel Core Ultra 7 155H, 30 alternating samples each:

without with
Float32/no_exceptions/len_65536 56.07µs 11.63µs -79%
Float64/no_exceptions/len_65536 56.68µs 19.08µs -66%
Float64Batched/len_65536 70.85µs 32.67µs -54%
geomean, all 16 7.589µs 3.630µs -52%

Deltas wider than 25 bits, zero-width vectors and arm64 are unchanged. The description has the full table and how the kernel works.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants