Repository navigation
Push decimal aggregates through Widen - #11004
Merged
Merged
Conversation
This was referenced Oct 29, 2026
This was referenced Oct 29, 2026
Merging this PR will regress 3 benchmarks1 warning
Warning Please fix the performance issues or acknowledge them on CodSpeed. Performance Changes
Tip Investigate this regression by commenting Comparing Footnotes
|
20 of 34 tasks
connortsui20
force-pushed
the
ct/decimal-widen-aggregates
branch
from
October 30, 2026 17:50
384deaa to
5d992ff
Compare
connortsui20
force-pushed
the
ct/decimal-widen-aggregates
branch
3 times, most recently
from
November 2, 2026 16:57
357188f to
28413bf
Compare
12 of 23 tasks
connortsui20
force-pushed
the
ct/decimal-widen-aggregates
branch
from
November 2, 2026 17:08
28413bf to
1fccc4c
Compare
connortsui20
force-pushed
the
ct/decimal-widen-aggregates
branch
2 times, most recently
from
November 2, 2026 19:24
590c518 to
e32c0f0
Compare
connortsui20
force-pushed
the
ct/decimal-widen-aggregates
branch
from
November 2, 2026 19:52
e32c0f0 to
9b794f6
Compare
connortsui20
force-pushed
the
ct/decimal-widen-aggregates
branch
2 times, most recently
from
November 3, 2026 13:36
10622b2 to
d2b541c
Compare
connortsui20
force-pushed
the
ct/decimal-widen-aggregates
branch
2 times, most recently
from
November 3, 2026 14:52
8e64f4b to
ee32bc3
Compare
connortsui20
force-pushed
the
ct/decimal-widen-aggregates
branch
from
November 3, 2026 15:12
ee32bc3 to
d8739cd
Compare
connortsui20
force-pushed
the
ct/decimal-widen-aggregates
branch
from
November 3, 2026 16:31
d8739cd to
3298bef
Compare
connortsui20
added a commit
that referenced
this pull request
Nov 3, 2026
Tracking Issue: #10701 Based on `develop`, including the merged #10946. #11004 builds on it. No release gate. ## Summary Comparisons, bounds checks and fills otherwise widen every decimal value before doing their work. The child's precision already bounds its values, so these operations can often stay narrow or answer a comparison without reading the buffer. ## Changes Constants that fit use the child dtype, and constants outside its precision answer comparisons from the range bound. Two decimal children compare at their larger precision. Dtype matches select the compatible types, then integer and decimal kernels share casts and execution. A decimal fill outside the child precision falls back to canonical execution. These shortcuts rely on the documented requirement that every valid value fits the child precision. Shares the canonical comparison assertion and decimal fixtures, removes duplicate scalar precision coverage, and reduces the new decimal test cases from 193 to 99. Keeps the four paired benchmark cases for in-range comparisons and the out-of-range shortcut. <details> <summary>Historical local benchmark results</summary> These retained measurements predate this style pass and rebase. They do not measure the current head. `cargo bench -p vortex-array --bench widen_decimal`, Apple M4, medians. The numbers were measured at the top of this stack before the split, so they cover the whole stack, not this layer alone. The baseline is the canonical input, and the candidate is the same values as a `Widen` (child precision 5 under precision 10, 9 under 18, and 18 under 38). CodSpeed on this PR measures the layer itself. | bench | p10 canonical | p10 Widen | p18 canonical | p18 Widen | p38 canonical | p38 Widen | | --- | ---: | ---: | ---: | ---: | ---: | ---: | | `compare_constant` | 4.166 µs | 5.041 µs | 4.124 µs | 5.041 µs | 10.7 µs | 4.207 µs | | `compare_constant_out_of_range` | 3.791 µs | 374.5 ns | 3.79 µs | 374.5 ns | 9.874 µs | 397.9 ns | </details> ## API Changes None. ## Validation Source review and downstream rebase range-diffs. No local tests, builds, formatting, linting, or benchmarks were run, per repository instructions. Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
connortsui20
force-pushed
the
ct/decimal-widen-aggregates
branch
4 times, most recently
from
November 3, 2026 18:56
151fb07 to
f0c5782
Compare
connortsui20
marked this pull request as ready for review
November 3, 2026 18:58
connortsui20
force-pushed
the
ct/decimal-widen-aggregates
branch
from
November 3, 2026 19:11
f0c5782 to
f7b5b1d
Compare
Decimal widening preserves order, equality, and validity, so ordered aggregates can use child partials without widening every input value. Reuse exact cached child extrema for MinMax. Compute decimal Sum in the logical sum dtype to avoid overflow at the child precision. Serve it only through the legacy Stat::Sum slot, whose independent batch totals preserve canonical overflow behavior. Keep the running-sum variants on the fallback path. Cover batch equivalence, logical precision, overflow, and child statistics without repeating partial-combination tests or adding benchmark cases. Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
connortsui20
force-pushed
the
ct/decimal-widen-aggregates
branch
from
November 3, 2026 19:18
f7b5b1d to
65ea070
Compare
connortsui20
added a commit
that referenced
this pull request
Nov 3, 2026
## Summary Follow-up to #11110. Keep the remaining noisy cases out of automatic CodSpeed runs while preserving them for local benchmarks. The Python scalar exclusions in #11110 never took effect because the runner exports simulation as `instrumentation`, while the fixture checked for `simulation`. Use the plugin's resolved mode so these cases are actually skipped. ## Changes - Allow a trailing comma in `#[divan::bench(...)]` when the CPU-feature macro appends its options. The multiline bitpacked take definition covers this case in CI builds. - Share the resolved simulation check between the scalar and filter fixtures. Also skip the 2 integer one-shot filtered scans that still report system calls under simulation. - Keep AVX2 bitpacked take with 80 selections local after batching still produced a regression alert. Retain the 64-selection case and both AVX-512 cases in CI. - Keep AVX-512 nullable integer comparison local because its minimum changes by more than 10% while its median stays stable. - Keep the small random power-law boolean filter and the large correlated power-law boolean filter local under simulation. The former has repeated unexplained changes, and the latter repeatedly makes system calls that simulation cannot measure. <details> <summary>Evidence from recent PR reports</summary> Reviewed 40 CodSpeed report comments updated between November 2 at 19:28 UTC and November 3 at 19:28 UTC, including their comparison revisions and the affected measurement modes. | Case | Evidence | | --- | --- | | Python integer scalar scans | [#11154](#11154 (comment)) still reports the repeated scan's 10 system calls. [A develop run after #11110](https://github.com/vortex-data/vortex/actions/runs/41677647530/job/126062463160) executes both integer scalar cases instead of skipping them. The [runner exports the legacy `instrumentation` value](https://github.com/CodSpeedHQ/codspeed/blob/main/crates/runner-shared/src/measurement_mode.rs), which pytest-codspeed 5.0.3 normalizes to simulation. | | Python integer filtered scans | [#11130](#11130 (comment)) reports 30 system calls totaling 14.3 ms in `test_scan_filter[int]`. [#11138](#11138 (comment)) reports 30 calls totaling 16.3 ms in `test_scan_filter_to_arrow_array[int]`. | | AVX2 `threshold[80]` | [#11004](#11004 (comment)) compares `f7b5b1d864` with `e59b236d7b`: 2.183 to 2.531 µs, reported as a 13.75% regression. Both use 1000 samples of 16 iterations. The head median is 2.556 µs, so this is not just a single outlying minimum. The PR does not change the fastlanes code or benchmark. | | AVX-512 `lt_i64_nullable` | [#11168](#11168 (comment)) reports 7.342 to 8.217 µs. Several unrelated CUDA PRs report the opposite direction, including [#10872](#10872 (comment)), at 8.208 to 7.348 µs. Their head medians remain around 8.3 to 8.5 µs. | | Boolean `filter_powerlaw_by_random[10000]` | [#11167](#11167 (comment)) reports a 16.2% regression, [#11168](#11168 (comment)) a 23.56% improvement, and [#11169](#11169 (comment)) a 13.06% regression. The cause is unknown. | | Boolean `filter_powerlaw_by_correlated_runs[100000]` | [#10871](#10871 (comment)) and [#11165](#11165 (comment)) each report 2 system calls totaling about 44 µs. | The NEON chunked-take alerts on #11090 are on a PR that changes execution substantially, so those cases remain enabled. Other dictionary simulation alerts need more attribution before excluding them. </details> CI passes formatting, Python lint and script tests, Rust lint, default-feature and all-feature Rust builds, all 3 CPU walltime benchmark jobs, and Python benchmarks (36 passed, 4 intentionally skipped). The previously failed simulation shard also passes. Some remaining CI jobs and CodSpeed performance analysis are still pending. No local checks were run. The pre-commit and pre-push hooks were bypassed as requested. --------- Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
robert3005
approved these changes
Nov 3, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Tracking Issue: #10701
Stacked on #11003. #11005 builds on it. No release gate.
Summary
Decimal widening preserves order, equality and validity, so ordered aggregates can use the child instead of copying it first. Sum also needs the logical result precision: using the child's sum dtype could overflow a result that should fit.
Changes
Ordered aggregates widen their child partials, and MinMax reuses exact cached child extrema before scanning. Decimal
Sumruns in the logical sum dtype only when the legacyStat::Sumpath combines independent batch totals.SumV2andSumwithoutskip_nanskeep the fallback because their running sum can cancel a batch total that would overflow on its own. Tests compare running partials against fresh canonical batches so cached statistics cannot hide kernel work.Historical local benchmark results
These retained measurements predate this style pass and rebase. They do not measure the current head.
cargo bench -p vortex-array --bench widen_decimal, Apple M4, medians. The numbers were measured at the top of this stack before the split, so they cover the whole stack, not this layer alone. The baseline is the canonical input, and the candidate is the same values as aWiden(child precision 5 under precision 10, 9 under 18, and 18 under 38). CodSpeed on this PR measures the layer itself.min_maxsummedians are within 8% of the canonical input at every precision.API Changes
None.