Skip to content

Move array serde off VTable onto explicit ArrayPlugin impls - #11298

Closed
connortsui20 wants to merge 3 commits into
developfrom
ct/upbeat-ride-if0rb0
Closed

connortsui20 wants to merge 3 commits into
developfrom
ct/upbeat-ride-if0rb0

Conversation

@connortsui20

Copy link
Copy Markdown
Member

Summary

Array serde lived in two places: serialize and deserialize on VTable, and the ArrayPlugin trait that the session registry actually calls, bridged by a blanket impl<V: VTable> ArrayPlugin for V. Because of the blanket impl, every vtable could be registered as a plugin, so every encoding had to carry serde methods even when it had no wire form. About 15 vtables (Filter, Slice, ScalarFn, Shared, and others) carried Ok(None) and bail! stubs only to satisfy the trait. Vtables whose wire form lives in a separate plugin (BitPacked, FoR, DecimalByteParts, Decimal) could still be registered directly and silently lose serde, which the old vtable_serde_requires_plugin tests guarded at runtime. Removing the blanket impl turns that mistake into a compile error and leaves one serde path.

Changes

Each encoding with a wire form implements ArrayPlugin on its existing vtable type rather than on a new FooPlugin struct. This keeps the number of types the same and leaves every register(Primitive) call site unchanged. The cost is that X.id() is ambiguous where both traits are imported, so a few call sites now spell VTable::id(&X).

VTable does not get ArrayPlugin as a supertrait. That would force in-memory-only encodings to keep the stubs this PR removes, and it does not fit encodings whose wire form belongs to a separate plugin.

Single-ID plugins no longer check serialized_id, which the blanket impl used to do. The ArrayPlugin::deserialize docs now state that callers must pass an ID from read_ids, which the registry already guarantees. read_ids and write_ids default to the plugin's own ID, and the second commit removes three overrides that matched the default.

PythonVTable keeps an explicit stub impl, so vortex.registry.register still works. PiecewiseSequence is no longer registered in the default session, since it never had a wire form, and serializing it now fails at the registry lookup with a different message. The three tests that registered a bare vtable and expected no serde are removed, because that no longer compiles. zoned::writer::tests::zone_maps_hold_no_variant_state::case_2_variant_in_struct fails in the full vortex-layout suite on this branch and on its base commit alike, so it is unrelated.

API Changes

VTable::serialize and VTable::deserialize are removed, so third-party encodings must implement ArrayPlugin to be serializable. Vtables no longer implement ArrayPlugin automatically, so ArraySession::register no longer accepts in-memory-only vtables such as Filter, Slice, ScalarFn, PiecewiseSequence, BitPacked, or FoR. ArrayPlugin::read_ids and ArrayPlugin::write_ids now have default implementations.


Generated by Claude Code

connortsui20 and others added 3 commits November 6, 2026 12:24
`VTable` no longer has `serialize` or `deserialize`, and the blanket
`impl<V: VTable> ArrayPlugin for V` is gone. Each encoding with a wire
form now implements `ArrayPlugin` directly on its vtable type. Encodings
with no wire form (Filter, Slice, Interleave, ScalarFn, Shared, Widen,
PiecewiseSequence, ForeignArray, and the vtables whose serde lives in a
separate plugin: Decimal, BitPacked, FoR, DecimalByteParts) do not
implement it, so they can no longer be registered by mistake.

`ArrayPlugin::read_ids` and `write_ids` now default to the plugin's own
ID. The trait docs state the caller contracts that the session registry
upholds, so single-ID plugins do not re-check `serialized_id`.

PiecewiseSequence is no longer registered in the default session, since
it never had a wire form. The tests that checked that registering a
bare BitPacked, FoR, or DecimalByteParts vtable gave no serde are
removed: that now fails to compile.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01J3PQtA2vX4uh3U3QwFKPZf
Signed-off-by: Connor Tsui <connor@spiraldb.com>
ScalarFnArrayPlugin, DecimalPlugin, and ALPPatchedPlugin each returned
their own ID from both methods, which is now the trait default.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01J3PQtA2vX4uh3U3QwFKPZf
Signed-off-by: Connor Tsui <connor@spiraldb.com>
PiecewiseSequence no longer implements ArrayPlugin, so the default
session does not register it and serialization fails at the registry
lookup instead of inside the encoding.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01J3PQtA2vX4uh3U3QwFKPZf
Signed-off-by: Connor Tsui <connor@spiraldb.com>
@codspeed

codspeed Bot commented Nov 6, 2026

Copy link
Copy Markdown

Merging this PR will regress 1 benchmark

1 warning

⚠️ Walltime results may be unreliable

Some benchmarks used the Walltime instrument outside CodSpeed Macro Runners, which can make the results inconsistent.

Learn more about Macro Runners

See affected benchmarks

⚡ 2 improved benchmarks
❌ 1 regressed benchmark
✅ 1914 untouched benchmarks
⏩ 47 skipped benchmarks1

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

Mode Benchmark BASE HEAD Efficiency
❌ CPU Simulation take_filter_primitive_small_uncached_random_mask_random_indices[4096, 10] 26.6 µs 29.8 µs -10.5%
⚡ CPU Simulation missing_stats[16] 91.8 µs 74.5 µs +23.25%
⚡ CPU Simulation max_f64 317.1 µs 283.8 µs +11.76%

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.


Comparing ct/upbeat-ride-if0rb0 (26e275a) with develop (5852636)

Open in CodSpeed

Footnotes

  1. 47 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

@mhk197

mhk197 commented Nov 6, 2026

Copy link
Copy Markdown
Contributor

Superseded by #11285

@mhk197 mhk197 closed this Nov 6, 2026
@connortsui20
connortsui20 deleted the ct/upbeat-ride-if0rb0 branch November 6, 2026 15:40
mhk197 added a commit that referenced this pull request Nov 6, 2026
## Summary

Removes the blanket `impl<V: VTable> ArrayPlugin for V` and the
`serialize` / `deserialize` methods on `VTable`. Every serializable
encoding now implements `ArrayPlugin` explicitly on its vtable type, in
a `plugin.rs` child module next to its `impl VTable`. Redo of #10111 on
current develop.

## Changes

- `VTable` covers only in-memory behaviour; serde is `impl ArrayPlugin
for X`.
- Serde bodies are ported verbatim, so wire formats are unchanged.
- `read_ids` / `write_ids` default to `vec![self.id()]`, what the
blanket impl declared. Plugins with one wire ID no longer spell them
out, including `ALPPatchedPlugin`, `ScalarFnArrayPlugin` and the Python
stub.
- Inside an impl the wire ID is `ArrayPlugin::id(self)`; only the `id`
method itself delegates to `VTable::id`.
- `session.arrays().register(X)` is unchanged for every encoding that
was serializable.
- In-memory-only vtables lose the two stub methods and get no plugin:
Filter, Slice, Interleave, ScalarFn, ScalarFnValidity, ForeignArray,
PiecewiseSequence, Widen, Shared, test-harness vtables.
- `DecimalPlugin` (#10995) becomes `impl ArrayPlugin for Decimal`: one
wire ID, same legacy-node reads, no separate type. Its module,
`StoredDecimal` and tests stay where they are.
- Multi-ID plugins (`BitPackedPlugin`, `FoRPlugin`,
`DecimalBytePartsPlugin`) and `WidenPlugin` are untouched;
`ALPPatchedPlugin` now delegates to `ArrayPlugin::deserialize(&ALP,
..)`.
- `PythonVTable` keeps a stub plugin so the Python `register` API still
compiles.
- Vtable files that gained a `plugin.rs` child are directory modules
(`array.rs` becomes `array/mod.rs`, 14 pure renames). ALP's own impl and
`ALPPatchedPlugin` share `alp/array/plugin.rs`.
- Tests that called the vtable serde directly (ALP, ALPRD, Dict, List,
RunEnd, VarBin, RLE, serde.rs, unary, zstd buffers) go through
`ArrayPlugin` instead.

| | count |
|---|---|
| new `impl ArrayPlugin` (vortex-array) | 20 |
| new `impl ArrayPlugin` (encodings + python) | 17 |
| vtables that only lose the stub methods | 14 |

## API changes

- `VTable` loses `serialize` and `deserialize`. An external vtable that
was registered through the blanket impl must now `impl ArrayPlugin` to
be registrable.
- `ArrayPlugin::read_ids` / `write_ids` gain defaults of
`vec![self.id()]`.
- `vortex_array::arrays::DecimalPlugin` is removed; register `Decimal`
instead.

## Behaviour changes

- `PiecewiseSequence` is no longer registered by
`ArraySession::default()`; it never serialized, and its error is now "is
not registered for serialization".
- The writer seeds the footer's encoding list from the session registry,
so dropping `vortex.piecewise-sequence` makes every default-session file
48 bytes smaller. `test_stock_ticker_file_size` expectations move
accordingly (70060 → 70012, 69996 → 69948, 55144 → 55096).
- `DecimalByteParts`, `BitPacked` and `FoR` can no longer be registered
bare, so `bare_vtable_refuses_serde` and `vtable_serde_requires_plugin`
are removed.

## Testing

- `cargo clippy --all-targets --all-features -- -D warnings` and `cargo
+nightly-2026-09-10 fmt --check` on every touched crate (vortex-array,
vortex-file, vortex-layout, vortex-python and the 13 encodings crates).
- `cargo nextest run` on all touched crates plus vortex-ipc and
vortex-scan: 8981 passed, 0 failed; `cargo test --doc` on vortex-array,
fastlanes, decimal-byte-parts and alp.
- `cargo nextest run -p vortex-file -E
'test(test_stock_ticker_file_size)'` with and without `--features zstd`,
to derive the new sizes.

## Follow-up

- Pre-existing, unchanged here: the Patched, RLE, OnPair, ALPRD, FSST
and Zstd deserializers index children or buffers before checking their
counts, so a malformed node panics in `SerializedArray::child` instead
of returning an error. Worth hoisting the count checks in a separate PR.
- Fold `FoRPlugin` / `BitPackedPlugin` / `DecimalBytePartsPlugin` into
`impl ArrayPlugin for FoR` etc. if wanted.
- Overlaps with #11298, which makes the same change with inline impls
and trait defaults for `read_ids` / `write_ids`.

---------

Signed-off-by: Matt Katz <mhkatz97@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants