[WIP] feat: elias-fano encoding

[rapour]

Rationale for this change

This PR introduces a new encoding: Non-partitioned Elias-Fano for weakly monotonic sequences. The changes include the encoding itself plus a set of Reduce rules and kernels.

What changes are included in this PR?

The following decisions, subject to further discussion, have affected the implementation:

• Non-partitioned only. The partitioned/quasi-succinct variant is left out entirely for the sake of simplicity. It can be added later.

• Non-nullable. Currently the implementation only accepts non-nullable sequences. We can probably switch to accept them and persist a third validity buffer.

• Sampled select rather than a full rank/select directory. The encoder stores the absolute position of every 256th set bit and every 512th unset bit, so a lookup jumps to the nearest signpost and finishes with a popcount walk bounded to one or two 64-byte blocks, for well under half a bit per value. We can switch to more complex rand/select directories based solutions like RsVec but the performance trade-offs needs investigation.

• An Elias-Fano array is encoded once, over a whole sequence, and every position inside it is absolute: element 500's bits sit at a fixed place in the buffers no matter which part of the array you are looking at. first_rank is just the offset saying which element of that original sequence this array's row 0 corresponds to. So slicing 1000..2000 out of a million-element array copies nothing. It sets first_rank = 1000 and the length to 1000, and reuses the original buffers unchanged; reading row 0 of the slice reads element 1000 underneath, row 1 reads element 1001, and so on. The trade-off is that the slice still carries the entire encoded sequence, in memory and on disk. Those unused rows only go away when the array is written out again.

• The change implements a stateful cursor. The cursor remembers where it last stopped: the bit position, rank, and value of the element it just read, plus the last next_geq answer and, once enough reads land in one FastLanes block, that block unpacked. When the next probe is the same element or the one after it, reading is a step to the next set bit rather than a sampled select, so an ascending stream of probes (a list-offset scan, a merge join, a sorted selection vector) touches neither sample table, and clustered low-bits reads switch to one bulk unpack after the ninth hit in a block instead of paying a single-value unpack each time. A probe that lands somewhere unrelated just falls back to the normal sampled select, so the state never costs anything, which is why take, filter, and compare all open one cursor for the whole batch rather than one per row.

• One 64-bit path serves every integer width. Each value widens to u64 on the way in and has the reference subtracted; on the way out the reference is added back and the result narrows to the column's own width. Both steps use wrapping arithmetic, which is what makes them exact for signed columns as well as unsigned, so i8 through u64 share one encoder, cursor, and decoder instead of getting one each. The column's type is consulted only at those two edges, so nothing in the per-element loop branches on it.

What APIs are changed? Are there any user-facing changes?

Additive only. New crate vortex-elias-fano (encodings/elias-fano), re-exported as
vortex::encodings::elias_fano, exporting EliasFano, EliasFanoArray, EliasFanoData,
EliasFanoMetadata, EliasFanoSlots, EliasFanoArraySlotsExt, EliasFanoCursor,
elias_fano_encode, and initialize. vortex-buffer gains BitBuffer::select_range and
BitBuffer::select_zero_range; existing select behaviour is unchanged and the shared internals
stay crate-private.

No behaviour changes. register_default_encodings registers the encoding, so vortex.elias_fano
is readable with no feature flag, but nothing selects it: there is no compressor scheme yet and the
encoding is not a member of any edition, so files written by default sessions are byte-identical to
before. Applying it means calling elias_fano_encode explicitly. Compressor selection and edition
membership follow separately.

Disclosure:
Claude code has implemented parts of the code, specially vortex-specific handles, comments, and tests, and improved others and has reviewed the implementation multiple times.

[joseph-isaacs]

can we try and make a mod out of all EF specific stuff so that I it would be easy to pull it into a external lib