Skip to content

Push decimal aggregates through Widen - #11004

Merged
connortsui20 merged 1 commit into
developfrom
ct/decimal-widen-aggregates
Nov 3, 2026
Merged

connortsui20 merged 1 commit into
developfrom
ct/decimal-widen-aggregates

Conversation

@connortsui20

@connortsui20 connortsui20 commented Oct 29, 2026 •

Copy link
Copy Markdown
Member

Tracking Issue: #10701

Stacked on #11003. #11005 builds on it. No release gate.

Summary

Decimal widening preserves order, equality and validity, so ordered aggregates can use the child instead of copying it first. Sum also needs the logical result precision: using the child's sum dtype could overflow a result that should fit.

Changes

Ordered aggregates widen their child partials, and MinMax reuses exact cached child extrema before scanning. Decimal Sum runs in the logical sum dtype only when the legacy Stat::Sum path combines independent batch totals. SumV2 and Sum without skip_nans keep the fallback because their running sum can cancel a batch total that would overflow on its own. Tests compare running partials against fresh canonical batches so cached statistics cannot hide kernel work.

Historical local benchmark results

These retained measurements predate this style pass and rebase. They do not measure the current head.

cargo bench -p vortex-array --bench widen_decimal, Apple M4, medians. The numbers were measured at the top of this stack before the split, so they cover the whole stack, not this layer alone. The baseline is the canonical input, and the candidate is the same values as a Widen (child precision 5 under precision 10, 9 under 18, and 18 under 38). CodSpeed on this PR measures the layer itself.

bench p10 canonical p10 Widen p18 canonical p18 Widen p38 canonical p38 Widen
min_max 14.93 µs 15.12 µs 14.62 µs 15.12 µs 21.58 µs 14.95 µs

sum medians are within 8% of the canonical input at every precision.

API Changes

None.

@connortsui20
connortsui20 added this pull request to stack #11007 October 29, 2026 21:33
@connortsui20 connortsui20 changed the title ct/decimal widen aggregates Push decimal aggregates through Widen Oct 29, 2026
@codspeed

codspeed Bot commented Oct 29, 2026 •

Copy link
Copy Markdown

Merging this PR will regress 3 benchmarks

1 warning

⚠️ Walltime results may be unreliable

Some benchmarks used the Walltime instrument outside CodSpeed Macro Runners, which can make the results inconsistent.

Learn more about Macro Runners

See affected benchmarks

⚡ 3 improved benchmarks
❌ 3 regressed benchmarks
✅ 1876 untouched benchmarks
⏩ 469 skipped benchmarks1

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

Mode Benchmark BASE HEAD Efficiency
❌ CPU Simulation encode_primitives[f32, (2000, 2)] 176.9 µs 204.5 µs -13.5%
❌ CPU Simulation take_filter_primitive_small_uncached_random_mask_random_indices[4096, 10] 21.9 µs 24.4 µs -10.45%
❌ Walltime lt_i64_nullable_avx512 7.4 µs 8.2 µs -10.22%
⚡ CPU Simulation bench_compare_varbin[(50000, 2)] 162.4 µs 132.3 µs +22.71%
⚡ CPU Simulation bench_compare_sliced_dict_varbinview[(1000, 10000)] 90.3 µs 78.7 µs +14.68%
⚡ CPU Simulation bench_compare_sliced_dict_varbinview[(9999, 10000)] 233.6 µs 205.9 µs +13.46%

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.


Comparing ct/decimal-widen-aggregates (65ea070) with develop (1ec5a1b)

Open in CodSpeed

Footnotes

  1. 469 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

@connortsui20 connortsui20 mentioned this pull request Oct 30, 2026
20 of 34 tasks
@connortsui20
connortsui20 force-pushed the ct/decimal-widen-aggregates branch from 384deaa to 5d992ff Compare October 30, 2026 17:50
@connortsui20
connortsui20 removed this pull request from stack #11007 October 30, 2026 18:02
@connortsui20
connortsui20 added this pull request to stack #11058 October 30, 2026 18:04
@connortsui20
connortsui20 force-pushed the ct/decimal-widen-aggregates branch 3 times, most recently from 357188f to 28413bf Compare November 2, 2026 16:57
@connortsui20
connortsui20 removed this pull request from stack #11058 November 2, 2026 16:59
@connortsui20
connortsui20 added this pull request to stack #11104 November 2, 2026 16:59
@connortsui20
connortsui20 force-pushed the ct/decimal-widen-aggregates branch from 28413bf to 1fccc4c Compare November 2, 2026 17:08
@connortsui20
connortsui20 force-pushed the ct/decimal-widen-aggregates branch 2 times, most recently from 590c518 to e32c0f0 Compare November 2, 2026 19:24
@connortsui20
connortsui20 force-pushed the ct/decimal-widen-aggregates branch from e32c0f0 to 9b794f6 Compare November 2, 2026 19:52
@connortsui20 connortsui20 added the changelog/performance A performance improvement label Nov 2, 2026 — with ChatGPT Codex Connector
@connortsui20
connortsui20 force-pushed the ct/decimal-widen-aggregates branch 2 times, most recently from 10622b2 to d2b541c Compare November 3, 2026 13:36
@connortsui20
connortsui20 force-pushed the ct/decimal-widen-aggregates branch 2 times, most recently from 8e64f4b to ee32bc3 Compare November 3, 2026 14:52
@connortsui20
connortsui20 force-pushed the ct/decimal-widen-aggregates branch from ee32bc3 to d8739cd Compare November 3, 2026 15:12
@connortsui20
connortsui20 force-pushed the ct/decimal-widen-aggregates branch from d8739cd to 3298bef Compare November 3, 2026 16:31
Base automatically changed from ct/decimal-widen-compare to develop November 3, 2026 16:46
connortsui20 added a commit that referenced this pull request Nov 3, 2026
Tracking Issue: #10701

Based on `develop`, including the merged #10946. #11004 builds on it. No
release gate.

## Summary

Comparisons, bounds checks and fills otherwise widen every decimal value
before doing their work. The child's precision already bounds its
values, so these operations can often stay narrow or answer a comparison
without reading the buffer.

## Changes

Constants that fit use the child dtype, and constants outside its
precision answer comparisons from the range bound. Two decimal children
compare at their larger precision. Dtype matches select the compatible
types, then integer and decimal kernels share casts and execution. A
decimal fill outside the child precision falls back to canonical
execution. These shortcuts rely on the documented requirement that every
valid value fits the child precision.

Shares the canonical comparison assertion and decimal fixtures, removes
duplicate scalar precision coverage, and reduces the new decimal test
cases from 193 to 99. Keeps the four paired benchmark cases for in-range
comparisons and the out-of-range shortcut.

<details>
<summary>Historical local benchmark results</summary>

These retained measurements predate this style pass and rebase. They do
not measure the current head.

`cargo bench -p vortex-array --bench widen_decimal`, Apple M4, medians.
The numbers were measured at the top of this stack before the split, so
they cover the whole stack, not this layer alone. The baseline is the
canonical input, and the candidate is the same values as a `Widen`
(child precision 5 under precision 10, 9 under 18, and 18 under 38).
CodSpeed on this PR measures the layer itself.

| bench | p10 canonical | p10 Widen | p18 canonical | p18 Widen | p38
canonical | p38 Widen |
| --- | ---: | ---: | ---: | ---: | ---: | ---: |
| `compare_constant` | 4.166 µs | 5.041 µs | 4.124 µs | 5.041 µs | 10.7
µs | 4.207 µs |
| `compare_constant_out_of_range` | 3.791 µs | 374.5 ns | 3.79 µs |
374.5 ns | 9.874 µs | 397.9 ns |

</details>

## API Changes

None.

## Validation

Source review and downstream rebase range-diffs. No local tests, builds,
formatting, linting, or benchmarks were run, per repository
instructions.

Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
@connortsui20
connortsui20 force-pushed the ct/decimal-widen-aggregates branch 4 times, most recently from 151fb07 to f0c5782 Compare November 3, 2026 18:56
@connortsui20
connortsui20 marked this pull request as ready for review November 3, 2026 18:58
@connortsui20
connortsui20 force-pushed the ct/decimal-widen-aggregates branch from f0c5782 to f7b5b1d Compare November 3, 2026 19:11
Decimal widening preserves order, equality, and validity, so ordered
aggregates can use child partials without widening every input value.
Reuse exact cached child extrema for MinMax.

Compute decimal Sum in the logical sum dtype to avoid overflow at the
child precision. Serve it only through the legacy Stat::Sum slot, whose
independent batch totals preserve canonical overflow behavior. Keep the
running-sum variants on the fallback path.

Cover batch equivalence, logical precision, overflow, and child statistics
without repeating partial-combination tests or adding benchmark cases.

Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
@connortsui20
connortsui20 force-pushed the ct/decimal-widen-aggregates branch from f7b5b1d to 65ea070 Compare November 3, 2026 19:18
connortsui20 added a commit that referenced this pull request Nov 3, 2026
## Summary

Follow-up to #11110. Keep the remaining noisy cases out of automatic
CodSpeed runs while preserving them for local benchmarks.

The Python scalar exclusions in #11110 never took effect because the
runner exports simulation as `instrumentation`, while the fixture
checked for `simulation`. Use the plugin's resolved mode so these cases
are actually skipped.

## Changes

- Allow a trailing comma in `#[divan::bench(...)]` when the CPU-feature
macro appends its options. The multiline bitpacked take definition
covers this case in CI builds.
- Share the resolved simulation check between the scalar and filter
fixtures. Also skip the 2 integer one-shot filtered scans that still
report system calls under simulation.
- Keep AVX2 bitpacked take with 80 selections local after batching still
produced a regression alert. Retain the 64-selection case and both
AVX-512 cases in CI.
- Keep AVX-512 nullable integer comparison local because its minimum
changes by more than 10% while its median stays stable.
- Keep the small random power-law boolean filter and the large
correlated power-law boolean filter local under simulation. The former
has repeated unexplained changes, and the latter repeatedly makes system
calls that simulation cannot measure.

<details>
<summary>Evidence from recent PR reports</summary>

Reviewed 40 CodSpeed report comments updated between November 2 at 19:28
UTC and November 3 at 19:28 UTC, including their comparison revisions
and the affected measurement modes.

| Case | Evidence |
| --- | --- |
| Python integer scalar scans |
[#11154](#11154 (comment))
still reports the repeated scan's 10 system calls. [A develop run after
#11110](https://github.com/vortex-data/vortex/actions/runs/41677647530/job/126062463160)
executes both integer scalar cases instead of skipping them. The [runner
exports the legacy `instrumentation`
value](https://github.com/CodSpeedHQ/codspeed/blob/main/crates/runner-shared/src/measurement_mode.rs),
which pytest-codspeed 5.0.3 normalizes to simulation. |
| Python integer filtered scans |
[#11130](#11130 (comment))
reports 30 system calls totaling 14.3 ms in `test_scan_filter[int]`.
[#11138](#11138 (comment))
reports 30 calls totaling 16.3 ms in
`test_scan_filter_to_arrow_array[int]`. |
| AVX2 `threshold[80]` |
[#11004](#11004 (comment))
compares `f7b5b1d864` with `e59b236d7b`: 2.183 to 2.531 µs, reported as
a 13.75% regression. Both use 1000 samples of 16 iterations. The head
median is 2.556 µs, so this is not just a single outlying minimum. The
PR does not change the fastlanes code or benchmark. |
| AVX-512 `lt_i64_nullable` |
[#11168](#11168 (comment))
reports 7.342 to 8.217 µs. Several unrelated CUDA PRs report the
opposite direction, including
[#10872](#10872 (comment)),
at 8.208 to 7.348 µs. Their head medians remain around 8.3 to 8.5 µs. |
| Boolean `filter_powerlaw_by_random[10000]` |
[#11167](#11167 (comment))
reports a 16.2% regression,
[#11168](#11168 (comment))
a 23.56% improvement, and
[#11169](#11169 (comment))
a 13.06% regression. The cause is unknown. |
| Boolean `filter_powerlaw_by_correlated_runs[100000]` |
[#10871](#10871 (comment))
and
[#11165](#11165 (comment))
each report 2 system calls totaling about 44 µs. |

The NEON chunked-take alerts on #11090 are on a PR that changes
execution substantially, so those cases remain enabled. Other dictionary
simulation alerts need more attribution before excluding them.

</details>

CI passes formatting, Python lint and script tests, Rust lint,
default-feature and all-feature Rust builds, all 3 CPU walltime
benchmark jobs, and Python benchmarks (36 passed, 4 intentionally
skipped). The previously failed simulation shard also passes. Some
remaining CI jobs and CodSpeed performance analysis are still pending.

No local checks were run. The pre-commit and pre-push hooks were
bypassed as requested.

---------

Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
@connortsui20
connortsui20 merged commit 22e02f1 into develop Nov 3, 2026
96 of 101 checks passed
@connortsui20
connortsui20 deleted the ct/decimal-widen-aggregates branch November 3, 2026 21:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

changelog/performance A performance improvement

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants