[zzylol]

Why

The default joint search composes every combination of each query's candidates within one component. Two kinds of query multiply that space without ever adding a cheaper plan:

• identical queries;
• queries that differ only in a quantile rank.

Examples from the paper workloads:

• RangeHeavy quantiles-24h@60s (raw data retained): five quantile_over_time(q, data[24h]) queries plus their ratio. Main checks 11.86M candidates; the run had not finished after 3 h 20 min on an idle 56-core machine.
• RollupHeavy: about 10× more time per duplicate query. Template 10 with raw data does not finish in 15 min on main.

Also, components were split only by source. Queries over one source that can never share a node were still searched together.

Part of #879 and #673.

What

The default search now applies two exact rules. Both are checked against --exhaustive (joint_search_equivalence).

1. Interchangeable queries take one choice (design method 4). Queries are interchangeable when all of these hold:

   • their workload entries are equal, except for the query text, and none has a latency bound;
   • each reads only its own Pass 1 targets;
   • planned alone, they compose to the same plan for every own choice, up to the quantile rank q;
   • when their ranks differ, no two members have the same rank, and no other query can share a node that carries one of those ranks. Duplicates inside such a class form a class of their own.

   Members of one class then:

   • take the same Pass 1 candidate;
   • must read the same summary builds. Builds over a member's own quantile are excluded.

2. Components split by shared nodes (design method 5). Two queries over one source stay in one component only when a Pass 2 rule can make them read one node, for some choice of their own candidates. The rules checked are: identical expressions (including widened scans), summary capability, shared pane chains, rollup and top-k reuse. With a disk price, raw retention links every query over a source, so the components stay per source.

   The components' selections are combined directly, no longer looked up among the joint candidates. That lookup enumerated every Pass 2 candidate of the whole workload. The combination falls back to searching as one when:

   • it uses exact top-k reuse together with panes; or
   • a component widened a scan, because Stage 1 widens every metric's scans at once.

SearchReport gains interchangeable_queries.

How

• pass/interchangeable.rs (new): Classes::of forms the classes. It compares rank-masked serializations of each query planned alone, then removes members whose rank another query can share (shares_a_rank_node).
  • Pass 1: invalid_prefix skips choices that give two members different candidates.
  • Pass 2: members_read_same_builds filters the Stage 1 candidates.
• Sharing a class's identical nodes before the Pass 2 rules (an earlier revision) is not done: the identical-expression rule runs last, and sharing first reached plans outside the exhaustive search (two duplicates' builds shared, then capability-shared with a third query's, cost more than the best exhaustive plan).
• pass/components.rs: sharing_components runs a union-find over pairs of queries per source, using can_share_a_node. combined_pass2 and widens_a_scan replace the joint-space lookup.

Why the class rule is exact

Swapping two members maps every plan to one of the same price, because nothing else tells them apart. Under cost model 1, a summary build's price is the cheapest of its materializations, and each materialization's price is affine in the number of class readers:

• not materialized: rebuilt for each reader;
• at ingestion, or once per evaluation: built once, at the class's common rate.

A minimum of affine functions is concave. So splitting a class over several builds never costs less than putting it on the best one of them. Classes are formed only under cost model 1 without a measured cost table.

Key code interfaces added / modified

crates/plan-selection/src/lib.rs

/// What the search behind a selection checked and discarded. The type stays.
pub struct SearchReport {
    pub candidates_checked: u64,                     // unchanged
    pub queries: Vec<QuerySearchReport>,             // unchanged
    pub components: Vec<ComponentReport>,            // unchanged (now split by shared nodes, not only sources)
    pub searched_as_one: Option<SearchedAsOne>,      // unchanged
    pub raw_scan_assignments_skipped: u64,           // unchanged
    pub raw_scan_candidates_skipped: u64,            // unchanged
    /// The classes of interchangeable queries, by query index, given one shared choice; empty for the exhaustive search and outside cost model 1.
    pub interchangeable_queries: Vec<Vec<usize>>,    // new
    pub runner_up_cost_per_second: Option<f64>,      // unchanged
}

SearchedAsOne::NotInJointSpace keeps its name. It is now produced by the two combinations above, not by a failed lookup.

Usage

The user plans RangeHeavy's 24 h quantiles component:

# Five quantile_over_time(q, data[24h]) queries and their ratio, every 60 s, raw data kept, AWS Fargate prices.
stage_pipeline --workload quantiles-24h@60s.json \
  --deployment-capabilities '{"raw_data_retained":true,"query_time_results_retained":false}' \
  --cost-model aws-fargate-us-east-1 --out out.json

The facade calls run_stage1_to_stage3_from_pass1:

• sharing_components finds one component, because every query can share the scan;
• Classes::of forms the class [1, 3, 4] (q 0.75, 0.95, 0.99). q 0.5 and q 0.9 stay out, because the ratio can share their exact quantiles;
• the enumeration gives the class one choice, and its members read the same builds.

out.json's stage3_plan_selection.search reports interchangeable_queries: [[1, 3, 4]] and 153,086 candidates checked. The selected plan costs 8.7388e-05 USD/s.

Before this PR

• Every combination of identical or rank-only-different queries was composed.
• Components were per source.
• A component combination was looked up among all of the whole workload's Pass 2 candidates.

After this PR

Measured on an idle 56-core machine, release build. Main is f870130; this PR is rebased on it.

Workload	Main	This PR
RangeHeavy quantiles-24h@60s, raw retained	not finished after 3 h 20 min	92 s, 50 MB, 153,086 checked (measured with the removed pre-sharing; to be re-measured)
RollupHeavy template 10, raw retained	not finished in 15 min	10.8 s, 53 MB, 17,547 checked, cost 76.8385
RollupHeavy template 10, no raw data	48.6 s, 56 MB, 3,258 checked, cost 76.8385	1.1 s, 51 MB, 331 checked, cost 76.8385
RollupHeavy templates 11 and 12	not finished in 15 min	not finished in 15 min (partial)
Whole RollupHeavy, 62 queries, and its 8-user version	not finished	still running after 27 min (partial)
#840 Q1–Q4 workload	0.51 s, 639 checked, cost 0.145781	0.57 s, 639 checked, same cost (no class: different windows)

Before #855, #866 and #869, the same rules planned whole RollupHeavy in 5.7 s (cost 1219.209/s) and its 8-user version in 24 s (1463.051/s). On main, the shared scans (#855), SQL windows as time ranges (#866) and sketch rollup (#869) link most queries over one table into one component again. That is why whole RollupHeavy is still slow.

Not in this PR, as follow-ups:

• memo groups with the cheapest candidate kept per sharing signature;
• per-group pricing;
• pane-subset decomposition by sharing cluster;
• local dominance of pane-chain variants.

A ratio or another interval that shares a member's rank keeps that member out of its class. Matched permutations of two classes (60 s and 300 s copies) are not exploited yet.

Verification

• cargo test --release -p asap-integration-tests --test joint_search_equivalence -- --include-ignored: all 18 pass, including the ignored exhaustive runs under every cost model. New tests:

  • interchangeable_queries_select_the_exhaustive_price;
  • random_workloads_with_interchangeable_queries_select_the_exhaustive_price;
  • queries_over_other_windows_or_with_a_latency_bound_are_in_no_class;
  • identical_sql_queries_are_one_class.

  random_workloads_over_disjoint_sources_combine_their_components now compares prices, not plans: the combination may select another plan of equal price.

• cargo test --release -p asap-plan-selection -p asap-logical-asap-optimization passes. So does -p asap-devtools --test stage_pipeline, with the three workload examples regenerated: same selections and prices, fewer candidates checked.

• cargo fmt --check and cargo clippy -D warnings are clean on the touched crates.

[zzylol]

Superseded by #884 (same change rebased onto main; this PR stopped tracking the branch after a force-push).