Skip to content

Exclude remaining noisy CodSpeed cases from CI - #11176

Merged
connortsui20 merged 2 commits into
developfrom
ct/codspeed-followup
Nov 3, 2026
Merged

connortsui20 merged 2 commits into
developfrom
ct/codspeed-followup

Conversation

@connortsui20

@connortsui20 connortsui20 commented Nov 3, 2026 •

Copy link
Copy Markdown
Member

Summary

Follow-up to #11110. Keep the remaining noisy cases out of automatic CodSpeed runs while preserving them for local benchmarks.

The Python scalar exclusions in #11110 never took effect because the runner exports simulation as instrumentation, while the fixture checked for simulation. Use the plugin's resolved mode so these cases are actually skipped.

Changes

  • Allow a trailing comma in #[divan::bench(...)] when the CPU-feature macro appends its options. The multiline bitpacked take definition covers this case in CI builds.
  • Share the resolved simulation check between the scalar and filter fixtures. Also skip the 2 integer one-shot filtered scans that still report system calls under simulation.
  • Keep AVX2 bitpacked take with 80 selections local after batching still produced a regression alert. Retain the 64-selection case and both AVX-512 cases in CI.
  • Keep AVX-512 nullable integer comparison local because its minimum changes by more than 10% while its median stays stable.
  • Keep the small random power-law boolean filter and the large correlated power-law boolean filter local under simulation. The former has repeated unexplained changes, and the latter repeatedly makes system calls that simulation cannot measure.
Evidence from recent PR reports

Reviewed 40 CodSpeed report comments updated between November 2 at 19:28 UTC and November 3 at 19:28 UTC, including their comparison revisions and the affected measurement modes.

Case Evidence
Python integer scalar scans #11154 still reports the repeated scan's 10 system calls. A develop run after #11110 executes both integer scalar cases instead of skipping them. The runner exports the legacy instrumentation value, which pytest-codspeed 5.0.3 normalizes to simulation.
Python integer filtered scans #11130 reports 30 system calls totaling 14.3 ms in test_scan_filter[int]. #11138 reports 30 calls totaling 16.3 ms in test_scan_filter_to_arrow_array[int].
AVX2 threshold[80] #11004 compares f7b5b1d864 with e59b236d7b: 2.183 to 2.531 µs, reported as a 13.75% regression. Both use 1000 samples of 16 iterations. The head median is 2.556 µs, so this is not just a single outlying minimum. The PR does not change the fastlanes code or benchmark.
AVX-512 lt_i64_nullable #11168 reports 7.342 to 8.217 µs. Several unrelated CUDA PRs report the opposite direction, including #10872, at 8.208 to 7.348 µs. Their head medians remain around 8.3 to 8.5 µs.
Boolean filter_powerlaw_by_random[10000] #11167 reports a 16.2% regression, #11168 a 23.56% improvement, and #11169 a 13.06% regression. The cause is unknown.
Boolean filter_powerlaw_by_correlated_runs[100000] #10871 and #11165 each report 2 system calls totaling about 44 µs.

The NEON chunked-take alerts on #11090 are on a PR that changes execution substantially, so those cases remain enabled. Other dictionary simulation alerts need more attribution before excluding them.

CI passes formatting, Python lint and script tests, Rust lint, default-feature and all-feature Rust builds, all 3 CPU walltime benchmark jobs, and Python benchmarks (36 passed, 4 intentionally skipped). The previously failed simulation shard also passes. Some remaining CI jobs and CodSpeed performance analysis are still pending.

No local checks were run. The pre-commit and pre-push hooks were bypassed as requested.

Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
@codspeed

codspeed Bot commented Nov 3, 2026

Copy link
Copy Markdown

Merging this PR will improve performance by ×2.5

1 warning

⚠️ Walltime results may be unreliable

Some benchmarks used the Walltime instrument outside CodSpeed Macro Runners, which can make the results inconsistent.

Learn more about Macro Runners

See affected benchmarks

⚡ 2 improved benchmarks
✅ 1872 untouched benchmarks
⏩ 477 skipped benchmarks1

Performance Changes

Mode Benchmark BASE HEAD Efficiency
⚡ CPU Simulation filter_random_by_random[10000] 32 µs 7 µs ×4.6
⚡ CPU Simulation filter_random_by_correlated_runs[100000] 86 µs 62.4 µs +37.77%

Tip

Curious why performance improved? Comment @codspeedbot explain why performance improved on this PR, or directly use the CodSpeed MCP with your agent.


Comparing ct/codspeed-followup (e3f4e31) with develop (8bada73)

Open in CodSpeed

Footnotes

  1. 477 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

@connortsui20
connortsui20 merged commit 24cd0ea into develop Nov 3, 2026
85 of 86 checks passed
@connortsui20
connortsui20 deleted the ct/codspeed-followup branch November 3, 2026 20:20
connortsui20 added a commit that referenced this pull request Nov 3, 2026
## Summary

Remove third-party timing baselines, old decode-first comparison paths,
and one exact duplicate. Keep benchmarks for distinct Vortex operations
and representations.

## Changes

- Reduce the integer `Widen` target from 46 cases to 17. Keep the
operations on `Widen` arrays and measure decoding separately, instead of
repeating every operation on canonical and explicitly decoded inputs.
- Remove the expanded-index take baselines, decode-then-compare
baselines, and the benchmark that manually reifies constant run lengths.
Keep the maintained entry points and their distinct input
representations.
- Remove `patterns_i128(Random)`, which duplicates `random_i128(0.5)`
with the same values and mask.
- Remove Arrow and std timing baselines and their unused dependencies.
Keep the `row_encode --output-sizes` comparison and the documented local
allocation baselines.
- Update the affected sharding estimates in
`scripts/codspeed_bench_counts.json`, preserving the new decimal Widen
target. These estimates are not a fresh inventory of every simulation
benchmark.

The restored integer arithmetic, Widen slice/chunked, bitpacked `Lt`,
and mask iteration cases remain. The exclusions from #11176 also remain
in effect.

## Validation

The original revision reported targeted Clippy and simulation builds,
shard-planner tests, spelling checks, and pinned-nightly formatting.
Those checks have not been rerun after rebasing onto current `develop`.
No local checks were run during this cleanup. The pre-commit and
pre-push hooks were bypassed as requested. CI validation is pending.

Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants