[zzylol]

Closes #719. Part 2 of 2: planning. Part 1, the IR and executor, is #856 (merged).

Why

W-RollupHeavy (the paper's appendix E) asks the same sketch question under many groupings of one table and window:

• template 12: distinct users under 15 groupings of (region, service, endpoint, status), at 5 and 15 minutes;
• template 13: p99 latency under CUBE(region, service, endpoint).

The planner built one KMV or KLL summary per query, each over its own scan of the raw rows: 30 summaries and 30 scans for template 12. KMV and KLL states merge, so the finest grouping's states could answer every coarser grouping. #856 added the operator for that (a rollup: SummaryMerge with group_by), but no planning rule used it.

What

• Pass 2: rollup rule for sketches. It now also offers a sketch rollup. A coarse per-group sketch build qualifies when:

  • every reader is an estimate whose merged readout keeps a guarantee (so not a heap's top-k, and not UnivMon's entropy);
  • the same algorithm (at any size), update and filter run over the same rows;
  • a finer build's keys include the coarse build's keys.

  The coarse build then becomes a rollup of the finest such build's states, by the coarse keys. This covers KMV, HLL, KLL, DDSketch, Count-Min, Count Sketch and UnivMon, for SQL GROUP BY and PromQL by.

• One choice per finest build. Every coarse grouping of one finest build rolls up together. Template 12's 14 coarser groupings per window are 1 choice, not 2^14 subsets, so the 20-choice cap counts choices.

• Sized for the strictest consumer. The finest build is resized for every reader, its own and each coarse one's, with the same per-parameter rule the summary-capability rule uses for a shared build.

• Guarantee through a rollup. An estimate over a rollup gets the guarantee of the same read of the finest states, when the family's merged error for that read is exact or "as if built over the union":

  • KMV, HLL, DDSketch, Count-Min and Count Sketch: the direct coarse build's state;
  • KLL: the same rank bound.

  The per-value failure probability is unchanged. A whole-result target is checked at the coarse group count, so it is no looser.

• Stage 3 pricing. A rollup costs one merge per input state. It outputs the groups of its keys, using the same estimate as any by (keys), so declared per-label distinct values (feat(plan-selection): price groupings from declared per-label distinct values #849) apply. A rollup to no keys is one row.

How

• rollup_candidates. It finds sketch candidates after the exact ones (sketch_rollup_candidates).
  • The finest build is the finer build with the most keys. Ties between groupings that are not nested go to the sorted key names that come first, so the source doesn't depend on query order.
  • finer_keys compares families with same_family: a sketch's algorithm at any size, exact accumulators by equality as before.
  • RollupCandidate stays one type. For a sketch, finer is the finest SummaryAgg and rollup is the regrouping merge of its states.
• rollup_choices. It groups the candidates: each exact rollup is a choice of its own, and all sketch rollups of one finest build share one choice. rollup_pass2_candidates enumerates subsets of choices. For a chosen sketch choice, sketch_rollups resizes the finest build (sized_for_all) and replaces each coarse build with SummaryMerge { [resized], group_by }. The merge keeps the coarse build's column names, so every reader above it is unchanged.
• rolled_up_guarantee (per_computation_candidates.rs): composes the read of a rollup as the read of its input states, gated on the family-capability table.
• Plan selection. For a rollup, the SummaryMerge arm caps output groups at groups(by(keys)). at_most_one_row treats a rollup to no keys as one row.

Key code interfaces added / modified

crates/logical-asap-optimization/src/asap_aware_cse.rs

/// One coarse build the rollup rule can answer from a finer one. The type stays.
pub struct RollupCandidate {
    /// The coarse build (`SummaryAgg`): an exact accumulator read only through
    /// finalization, or (new) a sketch read only by guaranteed estimates.
    pub coarse: Rc<OperatorNode>, // unchanged field, wider meaning
    /// Exact: the finer build's finalization. Sketch: the finest `SummaryAgg` of the
    /// same algorithm over the same rows.
    pub finer: Rc<OperatorNode>, // unchanged field, wider meaning
    /// Exact: the re-aggregated finalized values. Sketch: the regrouping `SummaryMerge`
    /// of `finer`'s states by the coarse keys (resized in the Pass 2 candidate).
    pub rollup: Rc<OperatorNode>, // unchanged field, wider meaning
}

/// The independent choices among rollup candidates: each exact rollup alone, every
/// sketch rollup of one finest build together. Pass 2 enumerates their subsets.
pub fn rollup_choices(candidates: &[RollupCandidate]) -> Vec<Vec<RollupCandidate>>; // new

/// One Pass 2 candidate per non-empty subset of the choices (was: of the candidates).
pub fn rollup_pass2_candidates(
    roots: &[QueryRoot],
    candidates: &[RollupCandidate],
) -> Result<Vec<RollupPass2Candidate>, PerComputationCandidateError>; // signature unchanged

Usage (the test stage3_selects_a_distinct_count_rollup_when_it_is_cheaper in crates/integration-tests/tests/sketch_rollup_planning.rs):

// The user's workload: two distinct counts of one table and window, every minute,
// relative ε = 0.05 per value at δ = 0.05.
let accuracy = AccuracyTarget::max_error(0.05).failure_probability(0.05)
    .accuracy_metric(AccuracyMetric::Relative).accuracy_scope(AccuracyScope::PerValue);
// `run`: every stage's output, with `http` arriving at 2e6 rows/s.
let run = plan_sql(&[
    "SELECT region, service, COUNT(DISTINCT user_id) AS value FROM http \
     WHERE ts >= now() - INTERVAL '5 minutes' GROUP BY region, service",
    "SELECT service, COUNT(DISTINCT user_id) AS value FROM http \
     WHERE ts >= now() - INTERVAL '5 minutes' GROUP BY service",
], accuracy).await;
// Pass 1 offers HLL, KMV and UnivMon per query. For each pair with the same algorithm,
// Pass 2's rollup rule offers the `service` build as
// SummaryMerge { [the (region, service) build, sized for both], group_by: [1] }.
// Stage 3 prices one scan, one build and one merge against two scans and two builds.
let rolled_up = cheapest(&run, true).unwrap();  // best plan with a rollup
let separate = cheapest(&run, false).unwrap();  // best plan without one
assert!(rolled_up < separate);
// The selected plan builds one summary and reads it through one rollup.
assert_eq!(builds_and_rollups(&logical_roots(&run, &run.selected().unwrap().from_logical)), (1, 1));

Before this PR

W-RollupHeavy templates 12 and 13, each planned alone with stage_pipeline --enforce-accuracy-guarantees. Both runs use the same main (6dd2320b, before #867 and #868), apply the eval's local search shortcuts (ASAP_EVAL_CHEAPEST_OWN_CANDIDATE=1, needed because the default search doesn't finish on these workloads), and declare no label counts:

Template	Summaries built	Scans	Rollups	Cost per second
12 (30 RQEs, distinct users)	30 KMV, k = 3001	30	0	648.19
13 (14 RQEs, p99)	14 KLL, k = 52	14	0	302.42

After this PR

Template	Summaries built	Scans	Rollups	Cost per second
12	2 KMV, k = 3001 (one per window)	2	28	43.39 (15× lower)
13	2 KLL, k = 52 (one per window)	2	12	43.22 (7× lower)

• Planning takes 0.2 s and 0.1 s.
• The 5- and 15-minute windows still build separately: SQL windows are scan filters, not time ranges (SQL trailing windows (WHERE ts >= now() - INTERVAL 'w') stay scan predicates, not TimeRange: no panes, no ingestion-time builds, scans priced without the window #853).
• Running the selected plans with the eval's correctness harness (scaled-down data, 3 evaluations, every group checked):
  • template 12: 161,380 values, 0 over ε, max relative error 0.032;
  • template 13: 37,998 values, 1 over rank ε = 0.05 (0.053), which is within the per-value δ;
  • no missing or extra groups in either.
  • The report's numbers before this change were 0 over (max 0.032) and 3 over (max 0.059).
• Instant quantile by (host, job) next to windowed quantiles: no scan sharing, no panes, no ingest-time variant, 100-group default #840's Q1–Q4 workload (stage_pipeline test scans_projecting_different_labels_are_shared): Stage 3 now selects Kll for Q4 rolled up from Q3, with Q3's KLL built once per evaluation for both queries. It costs 0.1458/s, against 0.1536/s for the previously selected plan. The test's expected selection is updated.

Verification

• Tests written first (they fail on main, where rollup_candidates offers no sketch candidate):
  • asap_aware_cse unit tests:
    • a_coarse_sketch_rolls_up_from_the_finer_build: Instant quantile by (host, job) next to windowed quantiles: no scan sharing, no panes, no ingest-time variant, 100-group default #840 Q3/Q4 shapes; one KLL at the larger k; the coarse query reads group_by: [1]; both guarantees are within the tighter target.
    • coarse_groupings_of_one_finest_build_roll_up_together
    • sketch_rollup_needs_the_same_rows
    • a_global_quantile_rolls_up_to_no_keys
    • the_rollup_source_does_not_depend_on_query_order
  • crates/integration-tests/tests/sketch_rollup_planning.rs:
    • Stage 3 selects the KMV rollup, cheaper than any plan without one.
    • Template 13's shape selects one KLL with two rollups.
    • Instant quantile by (host, job) next to windowed quantiles: no scan sharing, no panes, no ingest-time variant, 100-group default #840's PromQL pair gets a rollup plan that is priced.
  • feat(ir, executor): SummaryMerge regroups states by a subset of their group keys (sketch rollup, part 1) #856's sketch_rollup.rs checks that executor answers equal the separate summaries (KMV, HLL, UnivMon, DDSketch) or are within the bound (KLL).
• Commands:
  • cargo test -p asap-logical-asap-optimization --lib (171 passed, rebased on fix(executor): SQL plans over tables compile and run: table-scan compile check, now(), Int64 ids and COUNT(DISTINCT), exact percentile_cont #864);
  • -p asap-integration-tests --test sketch_rollup_planning --test reuse_rules;
  • -p asap-plan-selection --lib;
  • -p asap-devtools: the committed viewer examples are unchanged; one expectation changes (see After);
  • cargo clippy -D warnings on the touched crates and cargo fmt --check.
• Independent review: nothing blocking. Fixed from it:
  • coarse readers must have a guaranteed merged readout (heap top-k and entropy readers no longer sink a choice);
  • the source is deterministic;
  • errors pass through instead of becoming "no shared size";
  • the finalize arm is gated on merges_groups;
  • labels are now "rollup choices";
  • docs.

Limitations and follow-up

• All or nothing per finest build. One coarse query with a tight ε resizes the finest build for all of them. Plan selection still compares that with no rollup, but not with rolling up only the loose groupings.
• Untested paths.
  • A rollup over tumbling panes (a rollup Pass 2 candidate followed by window composition) has no test. No SQL query reaches it until SQL windows become time ranges (SQL trailing windows (WHERE ts >= now() - INTERVAL 'w') stay scan predicates, not TimeRange: no panes, no ingestion-time builds, scans priced without the window #853).
  • A heap reader as the coarse query has no test either. The rule rejects it through the family table.
• Groups. Without declared label counts, every grouping is still priced at the 100-group default. Declaring them (feat(plan-selection): price groupings from declared per-label distinct values #849) prices the rollup's merge by the finest build's real groups.