Repository navigation
Move array serde off VTable onto explicit ArrayPlugin impls - #11298
Closed
connortsui20 wants to merge 3 commits into
Closed
connortsui20 wants to merge 3 commits into
connortsui20 wants to merge 3 commits into
Conversation
`VTable` no longer has `serialize` or `deserialize`, and the blanket `impl<V: VTable> ArrayPlugin for V` is gone. Each encoding with a wire form now implements `ArrayPlugin` directly on its vtable type. Encodings with no wire form (Filter, Slice, Interleave, ScalarFn, Shared, Widen, PiecewiseSequence, ForeignArray, and the vtables whose serde lives in a separate plugin: Decimal, BitPacked, FoR, DecimalByteParts) do not implement it, so they can no longer be registered by mistake. `ArrayPlugin::read_ids` and `write_ids` now default to the plugin's own ID. The trait docs state the caller contracts that the session registry upholds, so single-ID plugins do not re-check `serialized_id`. PiecewiseSequence is no longer registered in the default session, since it never had a wire form. The tests that checked that registering a bare BitPacked, FoR, or DecimalByteParts vtable gave no serde are removed: that now fails to compile. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01J3PQtA2vX4uh3U3QwFKPZf Signed-off-by: Connor Tsui <connor@spiraldb.com>
ScalarFnArrayPlugin, DecimalPlugin, and ALPPatchedPlugin each returned their own ID from both methods, which is now the trait default. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01J3PQtA2vX4uh3U3QwFKPZf Signed-off-by: Connor Tsui <connor@spiraldb.com>
PiecewiseSequence no longer implements ArrayPlugin, so the default session does not register it and serialization fails at the registry lookup instead of inside the encoding. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01J3PQtA2vX4uh3U3QwFKPZf Signed-off-by: Connor Tsui <connor@spiraldb.com>
Merging this PR will regress 1 benchmark1 warning
Warning Please fix the performance issues or acknowledge them on CodSpeed. Performance Changes
Tip Investigate this regression by commenting Comparing Footnotes
|
Contributor
|
Superseded by #11285 |
mhk197
added a commit
that referenced
this pull request
Nov 6, 2026
## Summary Removes the blanket `impl<V: VTable> ArrayPlugin for V` and the `serialize` / `deserialize` methods on `VTable`. Every serializable encoding now implements `ArrayPlugin` explicitly on its vtable type, in a `plugin.rs` child module next to its `impl VTable`. Redo of #10111 on current develop. ## Changes - `VTable` covers only in-memory behaviour; serde is `impl ArrayPlugin for X`. - Serde bodies are ported verbatim, so wire formats are unchanged. - `read_ids` / `write_ids` default to `vec![self.id()]`, what the blanket impl declared. Plugins with one wire ID no longer spell them out, including `ALPPatchedPlugin`, `ScalarFnArrayPlugin` and the Python stub. - Inside an impl the wire ID is `ArrayPlugin::id(self)`; only the `id` method itself delegates to `VTable::id`. - `session.arrays().register(X)` is unchanged for every encoding that was serializable. - In-memory-only vtables lose the two stub methods and get no plugin: Filter, Slice, Interleave, ScalarFn, ScalarFnValidity, ForeignArray, PiecewiseSequence, Widen, Shared, test-harness vtables. - `DecimalPlugin` (#10995) becomes `impl ArrayPlugin for Decimal`: one wire ID, same legacy-node reads, no separate type. Its module, `StoredDecimal` and tests stay where they are. - Multi-ID plugins (`BitPackedPlugin`, `FoRPlugin`, `DecimalBytePartsPlugin`) and `WidenPlugin` are untouched; `ALPPatchedPlugin` now delegates to `ArrayPlugin::deserialize(&ALP, ..)`. - `PythonVTable` keeps a stub plugin so the Python `register` API still compiles. - Vtable files that gained a `plugin.rs` child are directory modules (`array.rs` becomes `array/mod.rs`, 14 pure renames). ALP's own impl and `ALPPatchedPlugin` share `alp/array/plugin.rs`. - Tests that called the vtable serde directly (ALP, ALPRD, Dict, List, RunEnd, VarBin, RLE, serde.rs, unary, zstd buffers) go through `ArrayPlugin` instead. | | count | |---|---| | new `impl ArrayPlugin` (vortex-array) | 20 | | new `impl ArrayPlugin` (encodings + python) | 17 | | vtables that only lose the stub methods | 14 | ## API changes - `VTable` loses `serialize` and `deserialize`. An external vtable that was registered through the blanket impl must now `impl ArrayPlugin` to be registrable. - `ArrayPlugin::read_ids` / `write_ids` gain defaults of `vec![self.id()]`. - `vortex_array::arrays::DecimalPlugin` is removed; register `Decimal` instead. ## Behaviour changes - `PiecewiseSequence` is no longer registered by `ArraySession::default()`; it never serialized, and its error is now "is not registered for serialization". - The writer seeds the footer's encoding list from the session registry, so dropping `vortex.piecewise-sequence` makes every default-session file 48 bytes smaller. `test_stock_ticker_file_size` expectations move accordingly (70060 → 70012, 69996 → 69948, 55144 → 55096). - `DecimalByteParts`, `BitPacked` and `FoR` can no longer be registered bare, so `bare_vtable_refuses_serde` and `vtable_serde_requires_plugin` are removed. ## Testing - `cargo clippy --all-targets --all-features -- -D warnings` and `cargo +nightly-2026-09-10 fmt --check` on every touched crate (vortex-array, vortex-file, vortex-layout, vortex-python and the 13 encodings crates). - `cargo nextest run` on all touched crates plus vortex-ipc and vortex-scan: 8981 passed, 0 failed; `cargo test --doc` on vortex-array, fastlanes, decimal-byte-parts and alp. - `cargo nextest run -p vortex-file -E 'test(test_stock_ticker_file_size)'` with and without `--features zstd`, to derive the new sizes. ## Follow-up - Pre-existing, unchanged here: the Patched, RLE, OnPair, ALPRD, FSST and Zstd deserializers index children or buffers before checking their counts, so a malformed node panics in `SerializedArray::child` instead of returning an error. Worth hoisting the count checks in a separate PR. - Fold `FoRPlugin` / `BitPackedPlugin` / `DecimalBytePartsPlugin` into `impl ArrayPlugin for FoR` etc. if wanted. - Overlaps with #11298, which makes the same change with inline impls and trait defaults for `read_ids` / `write_ids`. --------- Signed-off-by: Matt Katz <mhkatz97@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Array serde lived in two places:
serializeanddeserializeonVTable, and theArrayPlugintrait that the session registry actually calls, bridged by a blanketimpl<V: VTable> ArrayPlugin for V. Because of the blanket impl, every vtable could be registered as a plugin, so every encoding had to carry serde methods even when it had no wire form. About 15 vtables (Filter, Slice, ScalarFn, Shared, and others) carriedOk(None)andbail!stubs only to satisfy the trait. Vtables whose wire form lives in a separate plugin (BitPacked,FoR,DecimalByteParts,Decimal) could still be registered directly and silently lose serde, which the oldvtable_serde_requires_plugintests guarded at runtime. Removing the blanket impl turns that mistake into a compile error and leaves one serde path.Changes
Each encoding with a wire form implements
ArrayPluginon its existing vtable type rather than on a newFooPluginstruct. This keeps the number of types the same and leaves everyregister(Primitive)call site unchanged. The cost is thatX.id()is ambiguous where both traits are imported, so a few call sites now spellVTable::id(&X).VTabledoes not getArrayPluginas a supertrait. That would force in-memory-only encodings to keep the stubs this PR removes, and it does not fit encodings whose wire form belongs to a separate plugin.Single-ID plugins no longer check
serialized_id, which the blanket impl used to do. TheArrayPlugin::deserializedocs now state that callers must pass an ID fromread_ids, which the registry already guarantees.read_idsandwrite_idsdefault to the plugin's own ID, and the second commit removes three overrides that matched the default.PythonVTablekeeps an explicit stub impl, sovortex.registry.registerstill works.PiecewiseSequenceis no longer registered in the default session, since it never had a wire form, and serializing it now fails at the registry lookup with a different message. The three tests that registered a bare vtable and expected no serde are removed, because that no longer compiles.zoned::writer::tests::zone_maps_hold_no_variant_state::case_2_variant_in_structfails in the fullvortex-layoutsuite on this branch and on its base commit alike, so it is unrelated.API Changes
VTable::serializeandVTable::deserializeare removed, so third-party encodings must implementArrayPluginto be serializable. Vtables no longer implementArrayPluginautomatically, soArraySession::registerno longer accepts in-memory-only vtables such asFilter,Slice,ScalarFn,PiecewiseSequence,BitPacked, orFoR.ArrayPlugin::read_idsandArrayPlugin::write_idsnow have default implementations.Generated by Claude Code