[zzylol]

Why

The joint search enumerates whole-workload candidates: the product of every query's Pass 1 candidates, every Pass 2 rewrite subset and every materialization. On the paper workloads that is millions of candidates per sharing component, and the whole workloads never finish (#879, #673). The fix (#879) is a memo over logical sub-expressions, solved by a MILP. This PR lands the memo and a brute-force reference search over it. The reference is the second exact oracle the MILP will be checked against.

Part of #879.

What

• The memo (crates/plan-selection/src/pass/subexpr/). The workload's logical DAG is hash-consed into groups, one per distinct sub-expression. Each group gets implementation candidates: Pass 1's per-group output, then the Pass 2 rules as transformations that add candidates:
  • identical expression: group identity;
  • summary capability: a shared build with a size site;
  • window composition: panes, with a chain site for shared and nested chains;
  • rollup: exact or sketch, plus a coarser grouping reading a finer grouping's panes;
  • exact top-k reuse;
  • per-metric scan widening and raw horizon;
  • scan materializations: a scan read through one window is built once.
• The reference search (search::brute_force). It tries every consistent choice of one candidate per needed group and one threshold per coupling site, and lists each composed plan's identical-expression and scan-widening variants as the exhaustive search does. It recomposes guarantees, then has Stage 2 and Stage 3 implement and price each plan.
• Selection. Stage1ToStage3Search::memo_groups selects the reference search, and so does stage_pipeline --search memo-brute-force. The default search is unchanged.
• The report. SearchReport.memo gives the memo's size: groups, shared groups, candidates by origin, coupling sites by kind, build time.
• Per-node pricing. Stage 3's per-node pricing is split out of price_at_scale (price_node, price_each_node, price_private_nodes), with no price change. It is what the MILP's coefficients will come from.

How

• groups::memo_of interns the frontends' roots with the CSE interning. Each Pass 1 target's candidates become its aggregate group's candidates, reading child groups by Rc; fused computations implement several groups.
• The transformations (capability.rs, windows.rs, scans.rs) add candidates and coupling sites, with threshold enumeration per site and requirement checks (windows::requirements_met) that mirror the joint rules.
• search::composed_roots builds each query's own nodes as Pass 1 composes them. search::brute_force enumerates group choices × thresholds, lists the variants and prices them.
• pricing.rs: a candidate's private nodes priced in a consumer context, for the MILP that follows.

Key code interfaces added / modified

crates/plan-selection/src/pass/stage1_to_stage3.rs

/// How the Stage 1 to Stage 3 search runs. The type stays.
pub struct Stage1ToStage3Search {
    pub keep_every_candidate: bool, // unchanged
    pub exhaustive: bool,           // unchanged
    /// Run the brute-force reference over the memo of sub-expressions instead; ignored with `exhaustive`. Default false.
    pub memo_groups: bool,          // new
}

crates/plan-selection/src/lib.rs

pub struct SearchReport {
    // every existing field unchanged
    /// The memo's size when the memo was built; None otherwise.
    pub memo: Option<MemoStats>, // new
}
/// The memo's size, for the selection report.
pub struct MemoStats { // new
    pub groups: usize,
    pub shared_groups: usize,
    pub candidates: usize,
    pub candidates_by_origin: BTreeMap<String, usize>,
    pub coupling_sites_by_kind: BTreeMap<String, usize>,
    pub build_ms: f64,
}

crates/devtools/src/planning_stages.rs

/// The default search with an explicit search setting (e.g. the memo's reference search).
pub fn run_search_over(roots: Vec<QueryRoot>, entries: &[QueryWorkloadEntry], data: &DataWorkload, models: PlanningModels<'_>, search: Stage1ToStage3Search) -> Result<PlanningStagesRun, PlanningStagesError>; // new

crates/logical-asap-optimization/src/asap_aware_cse.rs

pub fn with_pane_chain(...);    // new: an existing chain read points at another chain
pub fn with_resized_build(...); // new: a build resized and its guarantees recomposed

recompose_guarantees and size_for_all_readers are now pub. CLI: stage_pipeline --search legacy|memo-brute-force (new; legacy is the default search, written as "search": "default"; --exhaustive with --search memo-brute-force is an error).

Staged for the MILP PR (crate-private, read only by tests here): subexpr/pricing.rs (PriceCache, frozen_calibration, local prices), price_private_nodes with PrivatePricing, NodeConsumer, HeldMemory, PrivateNodesUsage, LocalUsage, Context, and ImplementationCandidate.materializations. The MILP's coefficients read them.

Usage

# Two quantiles over one window, every minute, raw data kept, 1,000 series sampled every 15 s.
stage_pipeline --promql 'quantile_over_time(0.5, data[5m])' --promql 'quantile_over_time(0.9, data[5m])' \
  --series-count 1000 --sample-interval-ms 15000 --search memo-brute-force --out out.json

stage_pipeline lowers the queries and calls run_search_over with memo_groups: true. run_stage1_to_stage3_from_pass1 then calls search::brute_force: memo builds the groups, group_choices and at_thresholds enumerate plans, and Stage 2 and Stage 3 price each one. out.json has "search": "memo-brute-force" and the selection, with stage3_plan_selection.search.memo = {groups, shared_groups, candidates, candidates_by_origin, coupling_sites_by_kind, build_ms} (4 groups here), and the selected plan's Stage 1 candidate under the id its physical candidates name. The selection, P3-m1 at 5.1116e-05/s, costs what --exhaustive selects (checked: the same plan at the same price), and the viewer's validateStagePipeline accepts the document.

Major algorithm changes

1. The memo over logical sub-expressions.

• Problem: five quantile_over_time(q, data[24h]) queries plus their ratio have about 11.9M whole-workload candidates, though they share one input and differ only in their estimates.
• Before: stage1_logical_asap_optimization composes every product of Pass 1 choices, each with every Pass 2 subset.
• After: memo_of hash-conses the DAG, so the input is one group. Each aggregate group lists its own candidates (exact, KLL, DDSketch, panes, shared builds). The memo is linear in the workload: on whole RangeHeavy (46 queries), 74 groups and 284 candidates; on whole RollupHeavy (62 queries), 158 groups and 584 candidates, built in under 0.3 s.
• Invariant: every plan of the exhaustive search is a combination of the memo's candidates and thresholds, and no combination is cheaper than the exhaustive optimum. A Pass 1 parity lib test checks the first part (every_composed_pass1_candidate_is_a_combination_of_group_candidates); memo_group_search_selects_the_exhaustive_price compares only the selected price with the exhaustive search's, on 60 seeds; it does not check that every exhaustive plan is a memo combination.
• Fused candidates: Pass 1 fuses an aggregate into its single consumer within one query, but the memo merges equal nodes across queries. So a fused candidate now implements an absorbed group only when no other reader needs it: the group is no query's root, and all its parents are in the candidate. Otherwise the group stays its own group for its other readers, and the fused candidate computes its copy privately. Example: Q1 = topk(5, sum by (job) (sum_over_time(m[5m]))) and Q2 = sum by (job) (sum_over_time(m[5m])). Before, the heap absorbed Q2's root, so one plan answered Q2 with Q1's top-k and "Q1 by the heap, Q2 by its own sum" was missing. Tests: a_fused_candidate_does_not_absorb_another_querys_root (every composed plan) and a_fused_top_k_beside_its_inner_sum_selects_the_exhaustive_price (memo == --exhaustive, with and without raw data).
• Sketch rollups of a shared finer build: a rollup's state site (the finer KLL build sized for the finer group and every rolled-up coarse group) is sized by its readers' thresholds (state_size_thresholds, resize_state_at), and the requirement check keys the site by the memo's build, not by the resized copy whose recomposed guarantees change its key. Example (Instant quantile by (host, job) next to windowed quantiles: no scan sharing, no panes, no ingest-time variant, 100-group default #840's workload): quantile_over_time(0.5, data[1m]), quantile by (host, job) (0.5, data) and quantile by (job) (0.5, data) every 10 s over 10,000 series. --exhaustive rolls by (job) up from the by (host, job) KLL build at 0.041256/s; before the fix the memo had no such plan and selected two exact aggregates at 0.047030/s. On the whole Instant quantile by (host, job) next to windowed quantiles: no scan sharing, no panes, no ingest-time variant, 100-group default #840 workload the memo now selects 0.150788/s, as --exhaustive does (was 0.156562/s). Tests: a_sketch_rollup_beside_another_quantile_meets_its_requirements and a_sketch_rollup_beside_other_quantiles_selects_the_exhaustive_price (Q1/Q3/Q4 and Q2/Q3/Q4, with and without raw data).
• Limits: cost model 1 coefficients only (pricing.rs); a coarse group may read a less fine host chain than the joint rule would (not hit by any seed; tracked for the MILP).

2. The brute-force reference.

• It is exponential by design: it enumerates the memo's choices, not whole-workload candidates, and it is an oracle, not a production search.
• Measured on the 60 random seeds: all match --exhaustive, about 12 s in total in release.

Before this PR

The only searches were the default joint search and --exhaustive. Neither finishes on whole paper workloads, and neither reports how large the sharing problem really is.

After this PR

--search memo-brute-force plans small workloads through the memo, and a memo-brute-force selection reports the memo's size. The MILP solver (next PR) reads the same memo and is checked against this reference and --exhaustive.

Verification

• cargo test --release -p asap-integration-tests --test joint_search_equivalence --test sql_group_bound -- --include-ignored (head after merging main): 21 + 7 passed, 0 failed (including the ignored exhaustive runs, memo_group_search_selects_the_exhaustive_price on 60 seeds, and the new top-k and rollup seeds)
• cargo test --release -p asap-plan-selection -p asap-devtools -p asap-logical-asap-optimization -p asap-physical-asap-optimization: all pass, 0 failed (189 plan-selection lib tests, including main's label-set group-bound tests through the per-node pricing; devtools' new memo document and flag tests)
• The Usage document passes the viewer's validateStagePipeline (node 22)
• stage_pipeline on Instant quantile by (host, job) next to windowed quantiles: no scan sharing, no panes, no ingest-time variant, 100-group default #840's workload: --exhaustive P381-m3 0.150788/s; --search memo-brute-force 0.150788/s (was 0.156562/s). Two windowed quantiles over one metric: shared scan costs more, and nested-window KLL panes are never shared #839's: both 0.147762/s
• cargo test --release -p asap-plan-selection -p asap-logical-asap-optimization -p asap-physical-asap-optimization -p asap-planner -p asap-devtools: all pass (0 failed)
• cargo test --release -p asap-integration-tests (every file): all pass (0 failed)
• cargo clippy -D warnings (touched crates, all targets) and cargo fmt --check: clean