[zzylol]

Why

With #797, the 7 queries over data in ASAPQuery's MVP setting take 138 s:

• seed-862 workload-01;
• a deployment that keeps no raw data (--no-raw);
• failure probability 0.3.

145,312 physical candidates are checked, and 145,264 of them are rejected for the same reason: they scan raw data at query time (RawDataNotRetained). Only 48 are valid. Each rejected candidate is still composed, implemented and priced first.

This applies method 2 of docs/design_docs/proposals/joint-planning-search.md ("Reject permanently invalid branches") to Stage 2 assignments and to Stage 1 candidates. The pruning methods mapped to the next PRs (1, 6 and 7, then 3 and 4) come after it. This one removes most of the product on the MVP workloads first.

Builds on #797. Part of #673.

What

In a deployment that keeps no raw data, the default search no longer builds what Stage 3 would reject for scanning raw data at query time.

• Stage 2. A materialization assignment is skipped when a walk from a root reaches a Scan without passing a summary build materialized at ingestion time. Such a scan runs at query time, so that assignment is rejected whatever else it chooses. The remaining candidates keep their positions in their Stage 1 candidate's list (-m<k>); P<n> counts only the Stage 1 candidates enumerated, so ids can differ from the exhaustive search's.
• Stage 1. A candidate is not enumerated, and neither are its identical-expression Pass 2 candidates, when even its most-at-ingestion assignment scans raw data at query time. That assignment puts every build eligible for ingestion at ingestion time.
  • Materializing fewer builds at ingestion only leaves more scans at query time, so no assignment can repair it.
  • Sharing identical nodes cannot repair it either: the shared node is read by every reader of the nodes it replaces, so it is eligible for ingestion only if each of them is.
  • Its other Pass 2 candidates are still enumerated, because window composition can make a build eligible (panes over a window longer than the interval).
• Selection report. SearchReport gains raw_scan_assignments_skipped and raw_scan_candidates_skipped. When no candidate is valid, NoValidCandidate still carries a RawDataNotRetained rejection for what was skipped.
• Unchanged. With raw data retained, or for the exhaustive search or one keeping every candidate, nothing changes.

How

• Stage 2 (physical-asap-optimization).
  • optimize_physical_asap_without_query_time_raw_scans enumerates the same assignments as optimize_physical_asap and drops those reads_raw_at_query_time flags. That check is a walk from the roots that stops at builds assigned MatAtIngestion.
  • every_assignment_scans_raw_at_query_time runs the same walk for the assignment with every eligible build at ingestion.
• Stage 1 (logical-asap-optimization). stage1_logical_asap_optimization_skipping takes a viable predicate. Pass2List::push lists a candidate only when it is viable, and its callers expand identical-expression candidates only from viable ones.
• Wiring (plan-selection). The predicate is every_assignment_scans_raw_at_query_time when the deployment keeps no raw data; otherwise it accepts everything. The components' membership check (Joint planning search: search independent components separately (method 5) #797) uses the same predicate.

Key code interfaces added / modified

crates/physical-asap-optimization/src/lib.rs

/// optimize_physical_asap without the assignments that scan raw data at query time, each with its position in the full list, and how many were left out.
pub fn optimize_physical_asap_without_query_time_raw_scans(from_logical: &str, roots: &[QueryRoot], entries: &[QueryWorkloadEntry], data: &DataWorkload, deployment: &DeploymentCapabilities) -> Result<(Vec<(usize, PhysicalCandidate)>, usize), PhysicalASAPOptimizationError>; // new
/// Whether every assignment of `roots` scans raw data at query time (even with every eligible build at ingestion); false when `roots` cannot be implemented.
pub fn every_assignment_scans_raw_at_query_time(roots: &[QueryRoot], entries: &[QueryWorkloadEntry], data: &DataWorkload, deployment: &DeploymentCapabilities) -> bool; // new

crates/logical-asap-optimization/src/workload_candidates.rs

/// Stage 1 enumeration without the choices `invalid_prefix` rejects, and without the candidates `viable` rejects nor their identical-expression Pass 2 candidates.
pub fn stage1_logical_asap_optimization_skipping<'a, Id: Clone>(per_computation: &'a PerComputationCandidates<Id>, entries: &[QueryWorkloadEntry], invalid_prefix: impl FnMut(&[usize]) -> Option<usize> + 'a, viable: impl Fn(&[QueryRoot]) -> bool + 'a) -> impl Iterator<Item = Stage1Candidate> + 'a; // was: no `viable`
/// The Stage 1 candidates of one Pass 1 choice, the ones `viable` rejects left out.
pub fn stage1_candidates_of_choice<Id: Clone>(per_computation: &PerComputationCandidates<Id>, entries: &[QueryWorkloadEntry], choice: &[usize], viable: &dyn Fn(&[QueryRoot]) -> bool) -> Vec<Stage1Candidate>; // was: no `viable`

crates/plan-selection/src/lib.rs

/// What the search behind a selection checked and discarded. The type stays.
pub struct SearchReport {
    pub candidates_checked: u64,                                     // unchanged
    pub queries: Vec<QuerySearchReport>,                             // unchanged
    pub components: Vec<ComponentReport>,                            // unchanged
    pub searched_as_one: Option<SearchedAsOne>,                      // unchanged
    /// Stage 2 assignments not built because they scan raw data at query time, in a deployment that keeps none.
    pub raw_scan_assignments_skipped: u64,                           // new
    /// Stage 1 candidates not enumerated because every assignment of theirs scans raw data at query time (nor their identical-expression candidates).
    pub raw_scan_candidates_skipped: u64,                            // new
    pub runner_up_cost_per_second: Option<f64>,                      // unchanged
}

Usage

The user plans the first 7 PromQL queries of seed-862 workload-01 for a deployment that keeps no raw data, using the coverage converter with --no-raw --failure-probability 0.3:

// The deployment keeps no raw data: only plans whose scans all run at
// ingestion time are valid.
let capabilities = DeploymentCapabilities { raw_data_retained: false, ..DeploymentCapabilities::default() };
// q01..q07 over `data`: sums, avg of per-series quantiles, topk of sums.
let plan = plan_installable(UserInput::new(&workload, FrontendInput::Promql { histograms: None },
    PlanningModels::builtin().with_deployment_capabilities(&capabilities).with_plan_selection(&settings))).await?;
// Stage 1 enumerates only the Pass 2 candidates whose quantile builds are
// tumbling panes (a whole 100 s window cannot be built at ingestion for a
// 1 s interval); Stage 2 builds only the assignments with every scan at
// ingestion; Stage 3 prices 48 candidates.
let report = plan.selection.unwrap();
assert_eq!(report.cost_per_second, Some(3.983682886503911e-6)); // as with #797
assert_eq!(report.search.candidates_checked, 48);                // #797: 145,312

(The numbers come from running the coverage converter against this commit. The converter isn't part of this PR.)

Before this PR

With #797 and the MVP flags (--no-raw --failure-probability 0.3), seed-862 workload-01, PromQL prefixes:

Queries	Time	Peak memory
5	6.95 s	44 MB
6	42.1 s	106 MB
7	138 s	198 MB

After this PR

Queries	Time	Peak memory	Candidates Stage 3 checked
3	0.41 s	26 MB
4	1.01 s	28 MB
5	1.18 s	28 MB	24
6	1.98 s	31 MB	48
7	2.83 s	39 MB	48
8	3.19 s	56 MB
9	4.21 s	92 MB	50 (components 7 + 2)

The same plans are selected at the same costs (5, 6 and 7 queries compared with #797).

All 30 ASAPQuery workloads, PromQL only (9 queries each), MVP flags, plan_installable end to end:

Set	Time	Peak memory
seed-862 (10 workloads)	3.0–5.8 s	89–93 MB
seed-862-x5 (10)	3.1–6.1 s	89–93 MB
seed-862-long (10)	17.6–45.3 s	88–93 MB

Every workload splits into two components (7 + 2 queries), and Stage 3 checks 19–98 candidates.

On the long set, most of the time is not in the search. A profile shows it in the installable plan's expansion and checks of the pane DAG: executor preflight_depth, logical::restore and PhysicalASAPDAG::validate. With the long windows that DAG grows to about 5,300 nodes. That is follow-up work outside the search.

Verification

• cargo test --workspace --locked: passes.
• joint_search_equivalence.rs. Each check compares the default search with Stage1ToStage3Search::exhaustive() (plan, cost, physical DAG, materialization, front). New tests:
  • without_raw_data_random_workloads_select_the_exhaustive_plan: 60 seeded random workloads over two gauges and a counter, without raw data, each under one cost model in turn. It also asserts that some assignments were actually left out.
  • without_raw_data_planner_layering_examples_select_the_exhaustive_plan: Examples 1 and 2 and Pattern B, without raw data, under every cost model.
• runtime_install.rs (test(planner): assert the undeclared-gauge p99 outcome, not a candidate id #801's tests).
  • The merged-DDSketch-pane test no longer asserts the selected candidate's id (P3-m1 on main, P1-m1 here), which depends on what the search enumerates. Its plan, answer and bound checks are unchanged.
  • The undeclared-gauge test asserts a RawDataNotRetained rejection rather than the id P1.
• Measurements. /usr/bin/time around the coverage converter, with the MVP flags. Selections were compared with Joint planning search: search independent components separately (method 5) #797.
• Not run: clippy, fmt check, GitHub CI.