[zzylol]

Experimental. --search memo-milp is opt-in; the legacy search stays the default. This PR is the MVP of the MILP search over the sub-expression memo, merged under a lighter gate (listed under Verification). The full 60-seed three-way test, the slow ignored tests and whole-workload timing are the gate of the follow-up PR.

Part of #879

Why

The legacy and exhaustive searches enumerate whole-workload combinations, which blow up on the evaluation workloads (RangeHeavy quantiles-24h@60s: 359,970 candidates, 617 s, 3.4 GB; RollupHeavy template 10: no plan within 25 min). The memo (#900) represents the workload as groups of equal logical sub-expressions; this PR searches it with a mixed-integer linear program whose answer is checked exactly by Stage 3, so the selected plan is the exhaustive optimum when the reported gap is 0.

What

• --search memo-milp (SearchAlgorithm::MemoMilp): under cost model 1, a lazy loop over a MILP built from the memo. Cost models 2 and 3 run the legacy search and say so (routed_to_legacy: cost_model_not_supported).
• Every search reports incumbent_cost (the selected plan's cost per second). The MILP search adds search.milp: lower_bound, first_lower_bound, gap, rounds, variable_count, constraint_count, solver_time_ms, time_limit_reached.
• --milp-time-limit-s <f64> (Stage1ToStage3Search::milp_time_limit_ms): at the limit the search returns its best evaluated plan with time_limit_reached: true and its gap; it is never called optimal unless the gap is 0. Hitting the limit before any valid plan is an error that says so.
• A workload with no valid plan names the queries that have none on their own (e.g. "no valid plan for query 2 (quantile by (host, job) (0.5, data)) in a deployment that keeps no raw data").
• New dependency: good_lp =1.15.3 with the HiGHS solver (MIT).

How

Terms (joint-planning-search design, "Terms"):

• Coefficient derivation: each choice (a memo candidate at its site thresholds, with a materialization per private slot; scan groups per horizon) gets objective coefficients from Stage 3's per-node cost function, once per choice. Every coefficient is a lower bound on what the choice adds to any plan containing it; terms that do not decompose (held memory × latency, a scan's horizon shared by readers, nodes several groups may share, widened scans) are charged at a lower bound.
• The model: binary choice per needed group, built-once (once) per shared group, consumer reach through direct parts and slots, union rates of schedule classes for built-once and query-time nodes, held memory over each query's query-time latency (linearized), raw data kept on disk per source, latency bounds, requirements (exact children, coupling sites, rollup rules) and pane/top-k exclusivity as constraints, cross-group nodes and slots charged once per exact key.
• Lazy loop: solve → MILP lower bound; evaluate the solution's group choice exactly with Stage 3 (the reference search's evaluation of that group choice: every threshold, identical-expression variant and materialization), which updates the incumbent; cut that group choice (and only it); stop when the lower bound reaches the incumbent cost.
• Prices are scaled per evaluation so HiGHS's tolerances do not swamp small charges; a solve that HiGHS stops within its own gap tolerance counts as optimal with its bound lowered by that tolerance.

Major algorithm changes

• Problem. Example (Instant quantile by (host, job) next to windowed quantiles: no scan sharing, no panes, no ingest-time variant, 100-group default #840, 3 queries every 10 s over 10,000 series: quantile_over_time(0.5, data[1m]), quantile_over_time(0.9, data[5m]), quantile by (host, job) (0.5, data)): the plans differ in which KLL builds are shared, kept at ingestion or built at query time, and how long one raw scan reaches back. The exhaustive search composes and evaluates every combination.
• Before. Exhaustive: compose every combination of per-query candidates, run Stage 2 and Stage 3 on each. Legacy: the same with per-query pruning and components.
• After. Build the memo (8 groups for Instant quantile by (host, job) next to windowed quantiles: no scan sharing, no panes, no ingest-time variant, 100-group default #840), derive coefficients per choice, solve the MILP for a lower bound and a group choice, evaluate that group choice exactly, cut it, repeat until the bound reaches the incumbent.
• Invariant. The model's value of every group choice is at most Stage 3's exact cost of its cheapest plan (checked per group choice by a regression test on Instant quantile by (host, job) next to windowed quantiles: no scan sharing, no panes, no ingest-time variant, 100-group default #840's shape, a sketch rollup and Two windowed quantiles over one metric: shared scan costs more, and nested-window KLL panes are never shared #839's shape, with and without raw data), so stopping when the lower bound reaches the incumbent proves the incumbent optimal over the memo. The memo search equals --exhaustive on the equivalence seeds (feat(plan-selection): memo over logical sub-expressions with a brute-force reference search (--search memo-brute-force) #900).
• Complexity, measured. Instant quantile by (host, job) next to windowed quantiles: no scan sharing, no panes, no ingest-time variant, 100-group default #840 (3 queries): 8 groups, 1,198 variables, 1,603 constraints, 73 rounds, 322 s (exhaustive 4.3 s). quantiles-24h@60s: 4,094 variables, 5,593 constraints, 129 rounds, 735 s, 194 MB (main 617 s, 3.4 GB).
• Limits. Each round still evaluates every variant of a group choice, so whole workloads run out of memory; rounds are many because several lower bounds are loose (shared slots and scans of one source at different horizons are charged at a maximum); a single solve can be slow (Two windowed quantiles over one metric: shared scan costs more, and nested-window KLL panes are never shared #839 raw). Query latency in the model leaves out nodes another group may share, so the latency constraint never cuts a valid plan but is loose. The follow-up replaces the per-round enumeration by solution decoding (one plan per solution, one Stage 3 evaluation per round) and moves validity checks before cost.

Key code interfaces added / modified

crates/plan-selection/src/lib.rs

/// How the Stage 1 to Stage 3 search explores the plans. Stays.
pub enum SearchAlgorithm {
    /// The search over whole-workload candidates (the default). // unchanged
    Legacy,
    /// Every plan of the memo, each evaluated by Stage 3: the reference. // unchanged
    MemoBruteForce,
    /// The memo solved as a MILP whose solutions Stage 3 evaluates exactly. // new
    MemoMilp,
}

/// Why the legacy search ran instead of the requested memo search. New.
pub enum RoutedToLegacy {
    /// The caller asked for the exhaustive search. // new
    ExhaustiveRequested,
    /// Cost models 2 and 3: the MILP prices cost model 1 only. // new
    CostModelNotSupported,
}

/// The search's report. Stays; fields shown are the ones this PR adds.
pub struct SearchReport {
    /// The selected plan's Stage 3 cost per second, every search; None when none was selected. // new
    pub incumbent_cost: Option<f64>,
    /// The MILP search's report; None for every other search. // new
    pub milp: Option<MilpReport>,
    /// The algorithm that ran; omitted in JSON for the legacy search. // new
    pub algorithm: SearchAlgorithm,
    /// Why the legacy search ran instead of the requested one. // new
    pub routed_to_legacy: Option<RoutedToLegacy>,
}

/// The MILP search's bound, gap and effort. New.
pub struct MilpReport {
    /// MILP lower bound at the end on every plan not evaluated. // new
    pub lower_bound: Option<f64>,
    /// The first round's lower bound, before any cut. // new
    pub first_lower_bound: Option<f64>,
    /// (incumbent cost − lower bound) / incumbent cost; 0 = proven optimal. // new
    pub gap: Option<f64>,
    /// Rounds: solve, exact evaluation, cut. // new
    pub rounds: u32,
    /// The last round's model size. // new
    pub variable_count: u64,
    /// The last round's model size. // new
    pub constraint_count: u64,
    /// Time spent in the solver, every round together. // new
    pub solver_time_ms: u64,
    /// Stopped at `milp_time_limit_ms`: optimal only if `gap` is 0. // new
    pub time_limit_reached: bool,
}

pub enum SelectionError {
    /// The MILP search failed: solver error, time limit before any valid
    /// plan, or no valid plan (naming the queries with none). // new
    Solver(String),
    // every other variant unchanged
}

crates/plan-selection/src/pass/stage1_to_stage3.rs

/// How run_stage1_to_stage3_with searches. Stays.
pub struct Stage1ToStage3Search {
    /// Keep every candidate (developer tools). // unchanged
    pub keep_every_candidate: bool,
    /// The exhaustive search. // unchanged
    pub exhaustive: bool,
    /// The default search's algorithm; ignored with `exhaustive`. // was `memo_groups: bool`
    pub search: SearchAlgorithm,
    /// The MILP search's time limit in ms; None for no limit. // new
    pub milp_time_limit_ms: Option<u64>,
}

impl Stage1ToStage3Search {
    /// The default search run with `algorithm`. // new
    pub fn with_algorithm(algorithm: SearchAlgorithm) -> Self;
}

crates/devtools/src/bin/stage_pipeline.rs: --search legacy|memo-brute-force|memo-milp (was legacy|memo-brute-force), --milp-time-limit-s <f64> (new; needs --search memo-milp).

Usage

# The #840 workload: three quantiles every 10 s over 10,000 series sampled
# every 15 s, raw data kept; the experimental MILP search with a 10-minute limit.
stage_pipeline --workload wl840.json \
  --deployment-capabilities '{"raw_data_retained":true,"query_time_results_retained":false}' \
  --search memo-milp --milp-time-limit-s 600 --out wl840.memo-milp.json

The planner lowers the three queries (Stage 0), builds per-query candidates (Pass 1), builds the memo of 8 groups, derives coefficients for a model of 1,198 variables, then runs rounds: HiGHS solves the model, Stage 2 and Stage 3 evaluate the group choice it names, and a cut removes that group choice. The document's stage3_plan_selection.search then reads:

{"algorithm": "memo-milp", "incumbent_cost": 0.1477615599424,
 "milp": {"lower_bound": 0.1477615599424, "first_lower_bound": 0.141229058, "gap": 0.0,
          "rounds": 73, "variable_count": 1198, "constraint_count": 1603,
          "solver_time_ms": 321471, "time_limit_reached": false},
 "memo": {"groups": 8, "candidates": 39, ...}}

with the selected plan's cost equal to --exhaustive's (0.1477616 per second): one raw scan of 200,000 samples (5 minutes) read by both windows, a KLL build per window at query time (40,000 and 200,000 rows into 10,000 states), and a separate 15-second scan with an exact hash aggregate for the third query.

Before this PR

--search offered legacy and memo-brute-force. The brute force evaluates every plan of the memo, so it is no faster than the exhaustive search on large workloads.

After this PR

--search memo-milp selects the exhaustive optimum where it certifies gap 0 (below), reports how far from proven it is otherwise, and stays opt-in.

Verification

Checks that ran (the lighter MVP gate), on head a1f1a62 (the last commit only changes the error path after a time limit and a doc line); runs share one machine with other jobs:

Check	Result
Model ≤ exact for every group choice (lower_bound_check over all group choices)	committed tests (the_model_is_a_lower_bound_*, raw data on and off): #840's shape, a shared group with a rollup, #839's shape, a sketch rollup, a query over two groups of one metric. Ad-hoc runs (not committed): 0 violations on random seed 43, #840 (3 queries), two #840 variants, #839 raw and no raw
#840, 3 queries, raw	certified: gap 0, 0.1477616 = --exhaustive, 73 rounds, 322 s, 123 MB (exhaustive 4.3 s, 134 MB)
RangeHeavy quantiles-24h@60s, raw, Fargate, guarantees enforced	certified: gap 0, 8.73884e-05 = main, 129 rounds, 735 s, 194 MB (main 617 s, 3.4 GB)
#840 whole workload (11 groups)	not certified: time limit (1800 s), gap 0.042, incumbent cost 0.1507878 = exhaustive 0.150788, 168 rounds: follow-up
#839 no raw data	gap 0 against the memo, 129 rounds, 56 s: 115.66951 = --search memo-brute-force; --exhaustive selects 115.66801 (1.3e-5 relative cheaper), a memo-coverage gap, not the MILP's: follow-up
#839 raw data	not certified: time limit (1800 s), gap 0.341, incumbent cost 113.95144 vs exhaustive 113.94994, 64 rounds: follow-up
Three-way (exhaustive = memo brute force = memo MILP, gap 0) on the 20 smallest random seeds, cost model 1	passed (three_way_twenty_smallest_random_workloads_under_cost_model_1, release, 137 s)
cargo test of plan-selection and devtools	passed (plan-selection unit tests incl. lower_bound 4/4 with raw data on and off; stage_pipeline 45, incl. the time-limit and search-flag tests)
fmt, clippy -D warnings on the changed crates	clean

Not run here (follow-up gate): the full 60-seed three-way under cost model 1, the slow ignored equivalence tests, whole paper workloads.

Also: two stage_pipeline tests from #908 shared temporary file names and failed when run in parallel; they now take per-call names.