Add shared conversion to canonical decimal arrays or Widen by connortsui20 · Pull Request #11005 · vortex-data/vortex

connortsui20 · 2026-10-29T21:33:16Z

Tracking Issue: #10701

Stacked on #11004.

Summary

Legacy decimal nodes and DecimalByteParts decoding can produce a DecimalArray whose stored width differs from the canonical width of its dtype. Those arrays cannot pass through unchanged once DecimalArray requires canonical storage. Rejecting them would break existing data, while widening every buffer would discard the narrow storage that Widen is meant to preserve.

The migration therefore needs a shared conversion from these stored buffers into either a canonical decimal array or a Widen over a canonical decimal child. This PR provides that conversion for the reader and decoder follow-ups, preserving the logical dtype and retaining narrow buffers where possible. Sharing child selection with decimal encoding keeps the resulting child dtypes consistent.

Changes

Shares child selection between decimal encoding and stored-value conversion. Already-canonical buffers are reused without scanning, including device buffers. Other widths are handled on the host, selecting a valid child dtype or copying to the logical canonical width when no narrower child fits.

The reader integration is in #10995, which uses this helper when deserializing vortex.decimal nodes. #10905 uses it for DecimalByteParts decoding. This PR provides the common conversion without changing either read path yet.

Tests cover conversion, buffer reuse, range statistics, and malformed inputs. Formatting, DCO, and clippy hooks passed. Tests were not run.

API Changes

Adds WidenArray::from_stored_decimal(parts, known_range, ctx) for converting stored decimal buffers. A supplied range must contain every valid unscaled value, and an exact range must give the actual extrema.

codspeed · 2026-10-29T21:43:33Z

Merging this PR will improve performance by 21.13%

1 warning

⚠️ Walltime results may be unreliable

Some benchmarks used the Walltime instrument outside CodSpeed Macro Runners, which can make the results inconsistent.

Learn more about Macro Runners

See affected benchmarks

⚡ 2 improved benchmarks
✅ 1763 untouched benchmarks
⏩ 611 skipped benchmarks¹

Performance Changes

	Mode	Benchmark	`BASE`	`HEAD`	Efficiency
⚡	CPU Simulation	`missing_stats[16]`	84.8 µs	67 µs	+26.57%
⚡	CPU Simulation	`encode_primitives[f32, (2000, 2)]`	204.5 µs	176.4 µs	+15.92%

Tip

Curious why performance improved? Comment @codspeedbot explain why performance improved on this PR, or directly use the CodSpeed MCP with your agent.

_{Comparing ct/decimal-widen-stored (d2a9738) with develop (22e02f1)}

611 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

connortsui20 · 2026-11-03T20:23:47Z

this is effectively the old vortex-array/src/arrays/widen/encode.rs

Tracking Issue: #10701 Stacked on #11003. #11005 builds on it. No release gate. ## Summary Decimal widening preserves order, equality and validity, so ordered aggregates can use the child instead of copying it first. Sum also needs the logical result precision: using the child's sum dtype could overflow a result that should fit. ## Changes Ordered aggregates widen their child partials, and MinMax reuses exact cached child extrema before scanning. Decimal `Sum` runs in the logical sum dtype only when the legacy `Stat::Sum` path combines independent batch totals. `SumV2` and `Sum` without `skip_nans` keep the fallback because their running sum can cancel a batch total that would overflow on its own. Tests compare running partials against fresh canonical batches so cached statistics cannot hide kernel work. <details> <summary>Historical local benchmark results</summary> These retained measurements predate this style pass and rebase. They do not measure the current head. `cargo bench -p vortex-array --bench widen_decimal`, Apple M4, medians. The numbers were measured at the top of this stack before the split, so they cover the whole stack, not this layer alone. The baseline is the canonical input, and the candidate is the same values as a `Widen` (child precision 5 under precision 10, 9 under 18, and 18 under 38). CodSpeed on this PR measures the layer itself. | bench | p10 canonical | p10 Widen | p18 canonical | p18 Widen | p38 canonical | p38 Widen | | --- | ---: | ---: | ---: | ---: | ---: | ---: | | `min_max` | 14.93 µs | 15.12 µs | 14.62 µs | 15.12 µs | 21.58 µs | 14.95 µs | `sum` medians are within 8% of the canonical input at every precision. </details> ## API Changes None. Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>

Storage width alone cannot determine a valid decimal child precision. For example, unscaled 127 fits i8 but requires precision 3 and an i16 canonical child. Share child selection between encoding and stored-value reads so comparisons and aggregates can trust the selected child dtype. Reuse canonical buffers without scanning, including device buffers. Resolve other widths on the host from valid-value bounds and scale, retaining the range exactness in child statistics. Validate buffer layout and validity length before typed access, and use checked constructors. Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>

Remove the local signed/unsigned conversion helper. Its callers already establish non-null integer values, and PValue::as_i128 holds every integer PValue exactly. The scalar type-selection helper is now infallible. Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>

Explain why legacy decimal nodes and DecimalByteParts output need canonical conversion, including why only noncanonical widths require host access. Keep range resolution and statistic exactness together, share canonical array construction, and use vortex_ensure_eq for validity length. Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>

Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>

robert3005 · 2026-11-04T13:48:54Z

I think you can reduce some of the redundant test coverage here

Tracking Issue: #10701 Top of the decimal `Widen` stack, on #11005. No release gate. ## Summary A same-scale precision increase copied every value into a wider buffer even when the caller did not need the values yet. Returning a `Widen` defers that copy and lets subsequent kernels keep working on the narrow child. Basically, we might as well keep things in `Widen` form instead of casting immediately. ## Changes The child retains the source dtype and statistics because its values are unchanged. A cast of a decimal `Widen` passes to that child. Execution copies into the logical width directly, since yielding another widening cast would return a `Widen` again. The default-writer test compares round-trip values. Known cost: the retained benchmarks show slower filtering at precision 38 (see #10946). <details> <summary>Historical local benchmark results</summary> These retained measurements predate this style pass and rebase. They do not measure the current head. `cargo bench -p vortex-array --bench widen_decimal`, Apple M4, medians. The numbers were measured at the top of this stack before the split, so they cover the whole stack, not this layer alone. The baseline is the canonical input, and the candidate is the same values as a `Widen` (child precision 5 under precision 10, 9 under 18, and 18 under 38). CodSpeed on this PR measures the layer itself. | bench | p10 canonical | p10 Widen | p18 canonical | p18 Widen | p38 canonical | p38 Widen | | --- | ---: | ---: | ---: | ---: | ---: | ---: | | `cast_widen` | 12.08 µs | 6.958 µs | 12.33 µs | 6.958 µs | 16.54 µs | 14.58 µs | </details> ## API Changes A same-scale decimal cast returns a `Widen` when the target precision grows the canonical width. --------- Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>

connortsui20 added this pull request to stack #11007 October 29, 2026 21:33

connortsui20 added the changelog/break A breaking API change label Oct 29, 2026

connortsui20 mentioned this pull request Oct 30, 2026

Tracking Issue: Fix Canonical Decimal #10701

Open

20 of 34 tasks

connortsui20 force-pushed the ct/decimal-widen-stored branch from eb6bfc5 to b3b006c Compare October 30, 2026 17:50

connortsui20 removed this pull request from stack #11007 October 30, 2026 18:02

connortsui20 added this pull request to stack #11058 October 30, 2026 18:04

connortsui20 force-pushed the ct/decimal-widen-stored branch from b3b006c to e50341e Compare October 30, 2026 19:28

connortsui20 force-pushed the ct/decimal-widen-stored branch from e50341e to d030931 Compare November 2, 2026 13:50

connortsui20 force-pushed the ct/decimal-widen-stored branch from d030931 to 0c45ef9 Compare November 2, 2026 16:57

connortsui20 removed this pull request from stack #11058 November 2, 2026 16:59

connortsui20 added this pull request to stack #11104 November 2, 2026 16:59

connortsui20 mentioned this pull request Nov 2, 2026

Tracking Issue: Add Widen encoding for integers and decimals #10733

Open

12 of 23 tasks

connortsui20 force-pushed the ct/decimal-widen-stored branch 2 times, most recently from 73a9038 to bf6bc81 Compare November 2, 2026 17:24

connortsui20 force-pushed the ct/decimal-widen-stored branch from bf6bc81 to 0892933 Compare November 2, 2026 19:24

connortsui20 force-pushed the ct/decimal-widen-stored branch from 0892933 to a1837f2 Compare November 2, 2026 19:52

connortsui20 force-pushed the ct/decimal-widen-stored branch from 7aa8c56 to 9997931 Compare November 3, 2026 15:12

connortsui20 force-pushed the ct/decimal-widen-stored branch 2 times, most recently from 5f17d7b to 9f36e20 Compare November 3, 2026 16:46

connortsui20 force-pushed the ct/decimal-widen-stored branch from 9f36e20 to 754e893 Compare November 3, 2026 19:36

connortsui20 changed the title ~~Read stored decimal values into the canonical array or a Widen~~ Read old "incorrect" stored decimal values into Widen Nov 3, 2026

connortsui20 changed the title ~~Read old "incorrect" stored decimal values into Widen~~ Add shared conversion to canonical decimal arrays or Widen Nov 3, 2026

connortsui20 force-pushed the ct/decimal-widen-stored branch 2 times, most recently from 7102a1e to a19248e Compare November 3, 2026 20:12

connortsui20 commented Nov 3, 2026

View reviewed changes

connortsui20 force-pushed the ct/decimal-widen-stored branch from d2db6b0 to fe14932 Compare November 3, 2026 20:49

Base automatically changed from ct/decimal-widen-aggregates to develop November 3, 2026 21:17

connortsui20 added 3 commits November 3, 2026 16:17

connortsui20 force-pushed the ct/decimal-widen-stored branch from fe14932 to fe24fc7 Compare November 3, 2026 21:17

connortsui20 added 2 commits November 3, 2026 16:19

Clarify decimal conversion contracts and share integer extrema

1154a33

Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>

Make decimal conversion comments self-contained

d2a9738

Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>

connortsui20 marked this pull request as ready for review November 3, 2026 21:39

robert3005 approved these changes Nov 4, 2026

View reviewed changes

joseph-isaacs merged commit 58ceb89 into develop Nov 4, 2026
128 of 129 checks passed

joseph-isaacs deleted the ct/decimal-widen-stored branch November 4, 2026 13:53

Navigation Menu

Repository navigation

Add shared conversion to canonical decimal arrays or Widen - #11005

Add shared conversion to canonical decimal arrays or Widen#11005
joseph-isaacs merged 5 commits into
developfrom
ct/decimal-widen-stored

connortsui20 commented Oct 29, 2026 •

edited

Loading

Uh oh!

codspeed Bot commented Oct 29, 2026 •

edited

Loading

Uh oh!

connortsui20 Nov 3, 2026

Uh oh!

robert3005 commented Nov 4, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

connortsui20 commented Oct 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Changes

API Changes

Uh oh!

codspeed Bot commented Oct 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Merging this PR will improve performance by 21.13%

Performance Changes

Footnotes

Uh oh!

connortsui20 Nov 3, 2026

Choose a reason for hiding this comment

Uh oh!

robert3005 commented Nov 4, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

connortsui20 commented Oct 29, 2026 •

edited

Loading

codspeed Bot commented Oct 29, 2026 •

edited

Loading