Repository navigation
test: ignore tests that flake on wall-clock search budgets (#1067) - #1166
Merged
Merged
Conversation
These tests fail intermittently on CI across unrelated branches (main included) because the search stops on wall-clock budgets. Ignore them until #1067 replaces budget-based stopping with optimal-only stopping. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
zzylol
added a commit
that referenced
this pull request
Nov 2, 2026
…r deadline (#1067) (#1180) Follows #1166. A search budget (wall_time, time_limit_ms) or a short real deadline (Instant::now() + a few ms or s) decides these tests' outcome on a slow or loaded machine; e.g. structural_seed_timeout_preserves_validated_incumbent sleeps 1.1 s against a 1 s budget and fails under load. Stage 3 is being replaced (#1095), and #1067 moves to optimal-only stopping; until then they are ignored. Tests whose deadline is already expired (Instant::now(), Duration::ZERO) or whose fake evaluator never reaches its budget stay.
milindsrivastava1997
added a commit
that referenced
this pull request
Nov 2, 2026
…1175) #1166 and #1180 ignored the tests seen failing on deadlines; every other test that reaches the time-limited search (plan_installable, e2e_plan, optimize, run_planning, or the planner binary) can fail the same way under CI load. Ignore them as a stopgap; #1175 tracks re-enabling or rewriting each once Stage 3's design (#1095, #1067) is final. Tests that only use select_among_plans or installable_plan_of (no deadline) still run. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
milindsrivastava1997
added a commit
that referenced
this pull request
Nov 2, 2026
…1175) #1166 and #1180 ignored the tests seen failing on deadlines; every other test that reaches the time-limited search (plan_installable, e2e_plan, optimize, run_planning, or the planner binary) can fail the same way under CI load. Ignore them as a stopgap; #1175 tracks re-enabling or rewriting each once Stage 3's design (#1095, #1067) is final. Tests that only use select_among_plans or installable_plan_of (no deadline) still run. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
milindsrivastava1997
added a commit
that referenced
this pull request
Nov 2, 2026
…1175) (#1181) #1166 and #1180 ignored the tests seen failing on deadlines; every other test that reaches the time-limited search (plan_installable, e2e_plan, optimize, run_planning, or the planner binary) can fail the same way under CI load. Ignore them as a stopgap; #1175 tracks re-enabling or rewriting each once Stage 3's design (#1095, #1067) is final. Tests that only use select_among_plans or installable_plan_of (no deadline) still run. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
3 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of #1067.
Why
The planner search stops on wall-clock budgets: phase caps, pane-width round caps and the HiGHS time limit. Under CI load it stops at a different point, so tests that assert a selected plan fail intermittently, on a different test each run and on unrelated branches,
mainincluded. The decision recorded on #1067 is to park the real fix (stopping only on a proven-optimal plan, with no time or count budgets) and skip these tests until then.What
Mark 9 tests
#[ignore = "flaky: depends on wall-clock search budgets (#1067)"]. Each failed in the last 25 failed CI runs withexecution-scope expansion exceeded the deadline, or with the round-dependentCost { .. Overflow }that #1164 traces:planner_layering_example2::production_memo_selects_a_validated_example2_planjoint_search_equivalence::{custom_accuracy_policy_is_respected, different_windows_and_raw_retention_policies_produce_valid_plans, seeded_small_mixed_workloads_return_validated_plans}sharing_issue_regressions::{grouped_filtered_quantile_plans, nested_exact_sum_reuses_newest_panes, slower_repeat_reuses_faster_sum, sql_sum_shares_nested_windows_and_slower_readers}rollup_controlled::production_distinct_rollup_finds_shared_kmvNot ignored, because their failures look like real bugs rather than timing:
stage_pipeline::workload_examples_preserve_data_and_valid_materialized_views(direct-cost objective disagrees with the evaluator) andmemo_enumeration::choices_whose_candidates_read_each_other_are_left_out.Before this PR
CI fails intermittently on one of the tests above, for example
joint planning failed: plan selection: MILP solver: execution-scope expansion exceeded the deadline.After this PR
Those tests are skipped. In those four modules, 48 tests pass and 13 are ignored (9 new, 4 already ignored).
cargo test -- --ignoredstill runs them.Key code interfaces added / modified
None.
Major algorithm changes
None.
🤖 Generated with Claude Code