Repository navigation
Exclude remaining noisy CodSpeed cases from CI - #11176
Merged
Merged
Conversation
Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
Merging this PR will improve performance by ×2.51 warning
Performance Changes
Tip Curious why performance improved? Comment Comparing Footnotes
|
robert3005
approved these changes
Nov 3, 2026
connortsui20
added a commit
that referenced
this pull request
Nov 3, 2026
## Summary Remove third-party timing baselines, old decode-first comparison paths, and one exact duplicate. Keep benchmarks for distinct Vortex operations and representations. ## Changes - Reduce the integer `Widen` target from 46 cases to 17. Keep the operations on `Widen` arrays and measure decoding separately, instead of repeating every operation on canonical and explicitly decoded inputs. - Remove the expanded-index take baselines, decode-then-compare baselines, and the benchmark that manually reifies constant run lengths. Keep the maintained entry points and their distinct input representations. - Remove `patterns_i128(Random)`, which duplicates `random_i128(0.5)` with the same values and mask. - Remove Arrow and std timing baselines and their unused dependencies. Keep the `row_encode --output-sizes` comparison and the documented local allocation baselines. - Update the affected sharding estimates in `scripts/codspeed_bench_counts.json`, preserving the new decimal Widen target. These estimates are not a fresh inventory of every simulation benchmark. The restored integer arithmetic, Widen slice/chunked, bitpacked `Lt`, and mask iteration cases remain. The exclusions from #11176 also remain in effect. ## Validation The original revision reported targeted Clippy and simulation builds, shard-planner tests, spelling checks, and pinned-nightly formatting. Those checks have not been rerun after rebasing onto current `develop`. No local checks were run during this cleanup. The pre-commit and pre-push hooks were bypassed as requested. CI validation is pending. Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Follow-up to #11110. Keep the remaining noisy cases out of automatic CodSpeed runs while preserving them for local benchmarks.
The Python scalar exclusions in #11110 never took effect because the runner exports simulation as
instrumentation, while the fixture checked forsimulation. Use the plugin's resolved mode so these cases are actually skipped.Changes
#[divan::bench(...)]when the CPU-feature macro appends its options. The multiline bitpacked take definition covers this case in CI builds.Evidence from recent PR reports
Reviewed 40 CodSpeed report comments updated between November 2 at 19:28 UTC and November 3 at 19:28 UTC, including their comparison revisions and the affected measurement modes.
instrumentationvalue, which pytest-codspeed 5.0.3 normalizes to simulation.test_scan_filter[int]. #11138 reports 30 calls totaling 16.3 ms intest_scan_filter_to_arrow_array[int].threshold[80]f7b5b1d864withe59b236d7b: 2.183 to 2.531 µs, reported as a 13.75% regression. Both use 1000 samples of 16 iterations. The head median is 2.556 µs, so this is not just a single outlying minimum. The PR does not change the fastlanes code or benchmark.lt_i64_nullablefilter_powerlaw_by_random[10000]filter_powerlaw_by_correlated_runs[100000]The NEON chunked-take alerts on #11090 are on a PR that changes execution substantially, so those cases remain enabled. Other dictionary simulation alerts need more attribution before excluding them.
CI passes formatting, Python lint and script tests, Rust lint, default-feature and all-feature Rust builds, all 3 CPU walltime benchmark jobs, and Python benchmarks (36 passed, 4 intentionally skipped). The previously failed simulation shard also passes. Some remaining CI jobs and CodSpeed performance analysis are still pending.
No local checks were run. The pre-commit and pre-push hooks were bypassed as requested.