[zzylol]

Why

Part 1 of #1185. The group selection MILP (#1095), Stage 3's only search since #1187, stopped on wall-clock deadlines and costed some candidates inexactly behind hardcoded caps. On a loaded machine HiGHS returned an unproven incumbent, and the warm start tried a different number of query orders on each run, so the selected plan depended on machine speed (the flakes behind #1067/#1175/#1180). The caps silently approximated cost or dropped options, so the returned plan was not the optimum of the model we meant to solve.

What

• select_plan has no deadline parameter; the MILP always runs to proven optimality.
• The warm start is bounded by count only (MORE_ORDERS), not by time.
• The inexact-costing caps are removed; every case is costed exactly: MAX_CONTEXTS, MAX_PRODUCER_CHOICES, MAX_WINDOW_KINDS, MAX_COMPOSITIONS.
• HiGHS threads are a crate-internal option, SelectionMilpOptions::highs_threads, default 1 (was a constant 4), so tie-breaking is deterministic. No public API or CLI sets it yet; every caller uses the default.
• Loop guards stay: MOST_PASSES, MAX_SHIFTS, MOST_STEPS_PER_QUERY.
• Out of scope (part 2, next PR): the facade's 30 s planning budget and round caps in planner/src/run.rs, and Pass 2's generation deadline. The facade simply stops passing its deadline into Stage 3 here.

How

• solve_selection_milp / solve_part lose their deadline, the per-part time split and the HiGHS time_limit; warm_start loses its budget.
• Since a solve now ends only at Optimal or Infeasible, the status enum and the deadline outcomes go away. Any other HiGHS status (iteration, memory or solution limit) is a Solver error.
• In groups.rs every context combination and producer choice is costed, and add_unions encodes every union of a scan's window kinds; the "widest charged" fallback in cost_terms.rs is deleted.
• In accuracy_positions.rs, combinations/roots return every composition (no Option).
• The LP diagnosis (feature lp-diagnosis) loses LP_TIME_LIMIT_S: its LP solves and one-node root solve finish on their own.

Key code interfaces added / modified

select.rs

/// Stage 3's only entry: select one plan with the group selection MILP,
/// solved to proven optimality.
pub fn select_plan(
    physical: &PhysicalASAPMemoCandidates,
    entries: &[QueryWorkloadEntry],
    data: &DataWorkload,
    accuracy_checks: &AccuracyCheckInputs<'_>,
    cost_inputs: &CostInputs,
    realizer: &dyn AssignmentRealizer,
    // deadline: Option<Instant>,   // removed
) -> Result<Selection, ErrorSelectingPlan>;

report.rs

/// How the search ended with a validated plan; stays (JSON field kept stable).
pub enum SearchStatus {
    /// The search proved the plan optimal over the catalog it was given.
    Optimal,          // unchanged
    // DeadlineReached, // removed: the solve no longer stops early
}

/// Why Stage 3 selected no plan; stays.
pub enum ErrorSelectingPlan {
    // NoPlanByDeadline(String), // removed: nothing produces it
    /* every other variant unchanged */
}

(Every other ErrorSelectingPlan variant is unchanged; see the file.)

milp/model.rs, milp/warm_start.rs, groups.rs, groups/accuracy_positions.rs (crate-private)

/// Options for building the selection MILP; stays.
pub(crate) struct SelectionMilpOptions {
    pub(crate) enforce_accuracy: bool,   // unchanged
    pub(crate) split_into_parts: bool,   // unchanged
    /// Threads HiGHS solves with; default 1 (deterministic tie-breaking).
    pub(crate) highs_threads: i32,       // new (was const HIGHS_THREADS = 4)
}
pub(crate) fn solve_selection_milp(milp: SelectionMilp, build_ms: u64)
    -> Result<SelectionMilpSolution, SelectionMilpError>; // was (milp, deadline, build_ms)
pub(crate) enum SelectionMilpStatus { Optimal, TimeLimit }  // removed
SelectionMilpError::DeadlineReached                         // removed
SelectionMilpError::NoSolution { status }                   // removed
SelectionMilpSolution::relative_gap                         // removed (always 0)
UsageContext::TooManyContexts                               // removed
const WARM_START_MOST_TIME, MOMENT_TO_READ_THE_START_S, HIGHS_THREADS,
      MAX_CONTEXTS, MAX_PRODUCER_CHOICES, MAX_WINDOW_KINDS, MAX_COMPOSITIONS,
      LP_TIME_LIMIT_S;                                      // removed

ASAP_STAGE3_REPORT JSON lines lose relative_gap and the per-part status.

Usage: stage_pipeline --workload tools/dag-viewer/memo-examples/promql/rangeheavy-t8-t5-1h.json --cost-model aws-fargate-us-east-1 --enforce-accuracy-guarantees --out doc.json. The facade runs Pass 1, Pass 2 and Stage 2, then calls select_plan(&physical, &entries, &data, &checks, &cost_inputs, &realizer). That builds the MILP with SelectionMilpOptions { highs_threads: 1, .. }, warm-starts it with up to 2 + MORE_ORDERS query orders, and solves each part with HiGHS to optimality. doc.json's search_report.status is always "optimal", and plan_cost equals the objective.

Major algorithm changes

MILP stopping rule

• Problem: on shared-quantiles-1h-mvp and quantile-single-mvp, the snapshots recorded deadline_reached. The time-limited HiGHS run returned an incumbent without an optimality proof, and the result varied with load.
• Before:
  1. The warm start runs ⅕ of the remaining time, at most 60 s.
  2. Each part gets a time share proportional to its columns.
  3. HiGHS runs with time_limit; at the limit its incumbent is returned as DeadlineReached.
• After:
  1. The warm start tries workload order, its reverse and MORE_ORDERS pseudo-random orders.
  2. HiGHS solves each part with no time limit.
  3. Optimal → the plan; Infeasible → NoValidCandidate; any other status → Solver error.
• Correctness: the existing check that plan_cost equals the objective (relative 10⁻⁶) now always runs in its Optimal branch.
• Complexity: worst case unbounded in wall time; MILP solving is NP-hard. Measured: the focused tests and the 10 corpus snapshots finish (see Verification).
• Limits: planning time is no longer capped by Stage 3. The facade's own budget still bounds the earlier stages until part 2.

Exact costing instead of caps

• Problem: past the caps, a candidate was costed under only the first 256 contexts or producer choices (TooManyContexts, the first usage standing in for the rest). Scans with more than 4 window kinds were charged only the union of all kinds. Past 64 compositions, a set was marked unusable, which dropped it from the search.
• Before / after: groups::costed_usage* and add_unions truncated at the caps; now they enumerate every combination or union. accuracy_positions::combinations returned None past 64; now it returns them all.
• Correctness: TooManyContexts and the union-only charge were the inexact terms behind InexactObjective in these cases; those paths are gone.
• Complexity: contexts and producer choices scale with the product of producer outputs per input. Window unions are 2^k in the scan's window kinds k. Compositions scale with the product of choices per input. No workload in the tests or corpus hit a cap, so no measured change.
• Limits: a workload that used to hit a cap now pays the exponential enumeration instead of approximating. Window unions are enumerated as 2^k subsets although at most k² + k + 1 distinct hulls exist; a scan with 64 or more kinds is an explicit panic rather than a silent zero charge (shift overflow).

Infeasible selections name uncostable candidates

• Problem: since feat(plan-selection): group selection MILP over execution contexts as Stage 3's default search (#1095) #1144, a workload whose source lacks a pricing fact (e.g. sum(rate(m[1m])) with no series count) returned only "the selection MILP is infeasible"; the dag-viewer test test_a_plan_without_a_rate_fact_names_the_missing_fact failed on main.
• After: select::with_uncostable_reasons appends each distinct asap-cost error of the candidates left out as UsageNotCostable to the infeasible rejection's reason (e.g. "pricing source m needs an arrival rate").
• Complexity: linear in the pruned candidates.

Before this PR

Stage 3's plan could depend on wall time and machine load; some costs were approximated past caps without an error.

After this PR

Given the same Stage 2 catalog, Stage 3 selects the same plan with a proven-optimal search status, costed exactly. Full run-to-run determinism also needs part 2 (the facade and Pass 2 deadlines).

Verification

• GitHub CI (test, format-and-lint) passed on e85dddc.

• python3 -m unittest test_server (dag-viewer): 28 passed.

• Also fixes main's build: fix(plan-selection): reject an unpriceable candidate in the reference pricer (#1067) #1164 imported crate::AnalyticalCostError in selector.rs, which refactor(plan-selection): delete the old Stage 3 engine; select_plan runs only the group selection MILP #1187 had moved; now imported from asap_cost::analytical.

• cargo test -p asap-plan-selection --lib: 141 passed. Deleted a_deadline_yields_the_warm_starts_plan (zero time limit).

• cargo test -p asap-planner: lib 11 passed; tests 9 passed, 77 ignored (Re-enable tests ignored for wall-clock planning nondeterminism after Stage 3 design is final #1175 facade-budget markers, part 2).

• cargo test -p asap-devtools --test devtools stage_pipeline: 5 passed, 46 ignored (unchanged from main).

• cargo test -p asap-devtools --release --test corpus_snapshots -- --include-ignored: 10 passed; the sketch snapshot suite (including the two formerly deadline-limited workloads) finishes in 53 s.

• cargo check --workspace --tests: clean. --features lp-diagnosis builds. cargo fmt --all --check: clean.

• Snapshot changes, plans and costs identical, search status only:

  Snapshot	Change
  corpus o11y_bench/q05, q19, q25	proven_optimal false → true
  sketch shared-quantiles-1h-mvp, quantile-single-mvp	search_status deadline_reached → optimal; their deadline_reached_reason manifest entries are removed

• Screenshots: not applicable.

Limitations and follow-ups

• Part 2 of Remove unnecessary deadlines and caps in the group selection MILP #1185: remove the facade's planning budget and round caps (run.rs) and the Pass 2 generation deadline, then re-enable or delete the Planner tests depend on wall-clock search budgets and flake under load #1067/Re-enable tests ignored for wall-clock planning nondeterminism after Stage 3 design is final #1175/test: ignore every test whose result depends on a wall-clock budget or deadline (#1067) #1180 ignored tests.
• No test yet exercises a formerly capped path (more than 4 window kinds, 64 compositions or 256 contexts); add a fixture asserting inexact is empty and objective = plan_cost.
• Efficiency follow-up options: enumerate window unions as (nearest, farthest) kind pairs, O(k²); iterate compositions lazily; measure the uncapped warm start on the 8-user scaling workload.
• ErrorSelectingPlan::EvaluationDeadlineExceeded is now produced only by the reference pricer; revisit in part 2.

Part of #1185.

🤖 Generated with Claude Code