Skip to content

refactor(plan-selection): row (K) terms per path of edges, interim until step 6 (#1203 2a) - #1225

Closed
zzylol wants to merge 1 commit into
refactor/1203-2a-5-scan-labelsfrom
refactor/1203-2a-row-k-terms
Closed

zzylol wants to merge 1 commit into
refactor/1203-2a-5-scan-labelsfrom
refactor/1203-2a-row-k-terms

Conversation

@zzylol

@zzylol zzylol commented Nov 5, 2026 •

Copy link
Copy Markdown
Contributor

Stacked on #1222; part of #1203 (decision in the 2026-11-05 comment) and #1204 row (K), tracked in #1207.

Merge unit. This PR merges into main only together with the rest of #1207's merge unit (steps 1a–6, #1207 "Order"): interim items below are deleted by a later PR in the unit.

Why

#1203 decided (2026-11-05) that accuracy weights move from the candidate to how a query reaches it, as in row (K). Stage 3 gives each approximate candidate one weight per query, checked with the candidate's values approximate everywhere the query can read them. Once a state is one group read by several consumers (2a-6 makes a tumbling-window chain and its merge one group, shared by every consumer of the same state and width), a state shared by a ratio's numerator and denominator takes the stricter consumer's weight: it can no longer serve the numerator alone.

The same already happens on today's memo. In q0.9 / q0.5 / q0.1 (the three-quotient of kept_builds_give_the_weights_of_fresh_builds), Pass 2's shared build C1100 (DDSketch, sized for 2 parts) is read by all three operands. Today it is unusable for the query, because the check makes it approximate in all three operands at once. A plan reading it for one or two operands, with the shares row bounding how many values it depends on, is valid.

Interim encoding (decided 2026-11-05). Per-path terms are what 2a-6 needs: once a ratio's numerator and denominator read one shared merge group, a weight per candidate (or per (q, c, i) with values One / Unusable) cannot keep the numerator usable when the denominator is not. #1207 step 6, a later PR of the merge unit, collapses the terms to (q, c, i) with the constant propagation factor λ_{q,c,i} (ErrorPropagation, doc section 1), and deletes AccuracyTerm, AccuracyWeight, TermPath, accuracy_budget_parts, row 8, N[q] and A[q, parts]. This PR does not call a budget share λ: the per-term value is AccuracyWeight, and the dependence variables are d[q, t], not the formulation's $y_{u,c,i}$.

What

  • Row (K) terms. Each query's terms are the paths of edges $(u, c, i)$ from an approximate producer $u$ up to an output candidate of the query, each with its own accuracy weight (AccuracyTerm, AccuracyWeight). One producer reached through the numerator and through the denominator has two terms, with weight One and weight Unusable.
  • Checks per term. The query's output is built with $u$'s values approximate along the term's path and every other input exact, then checked as before (guarantee_check). Pairs are pairs of terms that one plan can depend on together.
  • MILP rows per term. The query depends on a term when every candidate on its path is selected where the path reads it: p[ctx, c] for each candidate on the path, crossing into a shareable candidate's context or an I_r as the realizer does, plus the slot link e of each multi-slot input on it (SelectionMilp::term_path).
    • An unusable term and an excluded pair each get one row saying their indicators are not all 1.
    • The shares row counts the terms depended on.
  • Warm start. It checks the terms its plan depends on in the same way (depends_on). It skips a producer only when every term of the query with that producer is unusable.
  • The per-candidate weight is deleted: MilpInputs::accuracy_weights (a map per query keyed by candidate), candidate-keyed accuracy_excluded_pairs, AccuracyWeight, and accuracy_positions.rs with its per-operator-node grouping and its cycle-aware walk.
  • Plans unchanged: UPDATE_SNAPSHOTS=1 leaves every corpus and sketch-plan snapshot file unchanged. Planning time is 6.89 s → 6.24 s in total; ratio-of-quantiles drops from 2.46 s to 1.60 s.

New abstractions not in #1207 or the doc: decisions (2026-11-05)

Per #1207's interface policy, these names are not defined by #1207, #1203 or section 3 of memo-groups-properties-requirements.md. #1207's steps 1a–6 merge into main as one merge unit (#1207, Order), so an interim abstraction may stay in this PR only when a later PR of the unit deletes it; none reaches main. Rows marked Pending approval are not deleted by any PR in the unit and need the user's decision.

Abstraction What it is for Decision
AccuracyTerm { output_reference, candidates, inputs, weight } + producer() one row (K) term per path of edges (u, c, i) Interim, deleted by #1207 step 6 (terms per (q, c, i) with the constant λ_{q,c,i})
AccuracyWeight { One, Unusable } (now per term) one budget share or unusable Interim, deleted by step 6 (with accuracy_budget_parts, row 8, N[q], A[q,parts])
MilpInputs::accuracy_terms, accuracy_excluded_pairs over term indices, AccuracyTerms, accuracy_terms(), read_together compute terms and excluded pairs Interim, deleted by step 6
TermPath + SelectionMilp::term_path; MILP variables d[q, t] (was y[q, t]) indicators that q depends on a term, over p[ctx, c] and slot links e Interim, deleted by step 6 (and p, e by #1204 PR 1)
accuracy_positions.rs, AccuracyPositions, per-candidate accuracy_weights per-candidate weight Deleted in #1225

How

  1. accuracy_terms::paths_to_approximate_producers: a depth-first walk from each output candidate of each of the query's output references, down producers that have an approximate candidate below them, with no candidate twice on one path (links can close a cycle). Every approximate candidate it meets ends a term.

  2. Checker::check(&[&term]) builds each output reference. A reference no term starts at reads exact stand-ins of its output candidates, as before. Otherwise built(tails) builds the reference from the path tails below each candidate:

    • a candidate that ends every tail is its node with its guarantees composed (with_guarantees);
    • any other candidate is its node rebuilt over its inputs (rebuilt): the tail's own node at the tail's input, and one exact stand-in at every other input.
    • Builds are kept per tails (BuiltBefore). Terms through the same operator nodes are checked once.
  3. Pairs: terms with weight One whose producers are sized for at least 2 parts, and that can be read together (read_together):

    • their paths start at different output references;
    • or they leave each other at two inputs of one consumer;
    • or one path continues the other.

    Two paths through different candidates of one consumer input, or of one output reference, are never in one plan. Each such pair is built with both terms approximate; a failing pair is excluded.

  4. accuracy_rows (row 8, A1 over the terms):

    • an approximate producer of the query computed in an I_r is forbidden, as before;
    • not_all(indicators) for each unusable term, for every term when the target is exact, and for each excluded pair;
    • d (q depends on a term) = AND of a term's indicators, made only when the shares row is needed;
    • the shares row's bound is the number of distinct place lists ((context, slot, input) along the path). Two terms with the same places are never both depended on, since each context holds one candidate per slot.

Representation / rules / enumeration

  • (a) Representation.
    • The memo is unchanged.
    • Stage 3's accuracy input changes: per query, a list of row (K) terms, each a path of edges from an approximate producer to the query's output with its accuracy weight, and pairs of term indices excluded together. No accuracy value is stored per candidate.
  • (b) Rules. Unchanged.
  • (c) Enumeration (the Stage 3 MILP).
    • Row 8 constrains the paths the query depends on, not each run of a producer.
    • A producer may be depended on through one path and not another.
    • The shares row counts every path, so one state read twice counts twice, as row (K)'s composition does.

Notation (formulation → code)

Formulation Code (this PR)
query $q$ the query index; MilpInputs::accuracy_terms[q]
producer $u$ (approximate) AccuracyTerm::producer(), the last of AccuracyTerm::candidates
edge $(u, c, i)$ consecutive AccuracyTerm::candidates[d] (consumer $c$), candidates[d + 1] (its producer), and AccuracyTerm::inputs[d] (input $i$)
term $\lambda_{q,u},\varepsilon_u,a_{q,u}$ of row (K) AccuracyTerm (one per path of edges from $u$ to $q$'s output)
(no formulation term: interim for $\lambda_{q,c,i}$ until step 6) AccuracyTerm::weight: AccuracyWeight (One: one share of the budget; Unusable(reason))
$a_{q,u}$ along one path SelectionMilp::term_path → TermPath::indicators, whose AND is the MILP variable d (q depends on a term)
row (K) accuracy_rows (still #1144 row 8 with N[q], A[q, parts], until #1207 step 6)

Concepts not in the formulation

Key code interfaces added / modified

crates/plan-selection/src/groups.rs (crate-internal)

pub(crate) struct MilpInputs {
    /* ... */
    // pub(crate) accuracy_weights: Vec<BTreeMap<PhysicalCandidateId, AccuracyWeight>>, // removed
    /// Per query: its row (K) terms. // new
    pub(crate) accuracy_terms: Vec<Vec<AccuracyTerm>>,
    /// Per query: pairs of indices into its terms, each with weight One alone,
    /// not depended on together. // was pairs of candidates
    pub(crate) accuracy_excluded_pairs: Vec<Vec<(usize, usize)>>,
}
/// One term of row (K): u's values reaching q's
/// output along one path of edges (u, c, i). // new
pub(crate) struct AccuracyTerm {
    pub(crate) output_reference: usize,
    /// From an output candidate down to u, the last.
    pub(crate) candidates: Vec<PhysicalCandidateId>,
    /// Per edge, top down: the input i at which candidates[d] reads candidates[d + 1].
    pub(crate) inputs: Vec<usize>,
    pub(crate) weight: AccuracyWeight,
}
/// The accuracy weight of one term: one budget share or unusable. // was per candidate
pub(crate) enum AccuracyWeight { One, Unusable(UnusableReason) }

crates/plan-selection/src/groups/accuracy_terms.rs (new; replaces accuracy_positions.rs)

pub(crate) fn accuracy_terms(physical, candidate_data, producer_candidates_per_input,
    output_candidates, entries, data, models) -> AccuracyTerms; // was accuracy_positions
pub(super) fn read_together(one: &AccuracyTerm, other: &AccuracyTerm) -> bool; // new

crates/plan-selection/src/milp/model.rs (crate-internal)

/// Where a term's path is read in the MILP. // new
pub(crate) struct TermPath { pub(crate) indicators: Vec<Variable>,
    pub(crate) places: Vec<(usize, ResultSlot, usize)>, pub(crate) producer_context: usize }
impl SelectionMilp<'_> {
    pub(crate) fn term_path(&self, query: usize, term: &AccuracyTerm) -> Option<TermPath>; // new
}
fn not_all(milp: &mut SelectionMilp<'_>, paths: &[&Vec<Variable>]); // new

crates/plan-selection/src/milp/warm_start.rs: depends_on(milp, plan, query, term) -> bool (new); meets_accuracy counts the terms the plan depends on (was the runs it reaches).

Usage. q0.9 / q0.5 / q0.1 (max_error 0.01), Stage 3's inputs printed from the test (Cn = physical candidate):

# before: one weight per candidate
C1096 (DDSketch, 1 part)   Unusable
C1100 (DDSketch, 2 parts)  Unusable   # Pass 2's shared build, made approximate in all three operands at once
C1103 (DDSketch, 3 parts)  One
excluded pairs: (C1103, C1303), ...    # C1103 in three operands plus C1303
# after: one term per path of edges (consumer c, input i) from u up to the output
[C1095 i0, C730 i0, C3 i0, C1097 i0] → C1096   weight One        # the numerator alone
[C1095 i0, C730 i0, C3 i0, C1098 i0] → C1100   weight One        # numerator
[C1095 i0, C730 i1, C367 i0, C1106 i0] → C1100 weight One        # first denominator
[C1095 i1, C732 i0, C1112 i0] → C1100          weight One        # second denominator
excluded pairs: none   # each pair of 2-part terms meets the target; three are excluded by the shares row (N ≤ parts)

Major algorithm changes

  • Accuracy is checked per path instead of per candidate. A producer reached through several paths is checked along each path alone, then in pairs of paths, instead of along all of them at once.
  • Three or more paths of one producer are bounded by the shares row, as three different producers already were.

Before this PR

w(q, k) per approximate candidate: the check made k approximate wherever q could read it. Row 8 counted runs (context, candidate). The warm start skipped candidates with an unusable weight.

After this PR

An accuracy weight per row (K) term (path of edges): row 8 constrains and counts the paths a plan depends on. The warm start checks the same paths.

Deleted

  • groups/accuracy_positions.rs: accuracy_positions, AccuracyPositions, pairs_that_meet, Walk, BuiltInWalk, the per-operator-node grouping of checks, the cycle handling of builds (a path never repeats a candidate, and off-path inputs are exact stand-ins, so no build crosses a cycle), and its test outputs_built_across_a_cycle_are_not_kept (what it tested is gone).
  • groups.rs: AccuracyWeight (now AccuracyWeight, per term), MilpInputs::accuracy_weights, candidate-keyed accuracy_excluded_pairs.
  • model.rs row 8: the per-run dep (p, a[q, k], y (depends on a run in m)), the (context, slot) places.
  • warm_start.rs: the per-candidate weight lookups in allowed and meets_accuracy.
  • Test changes:
    • ratio_weights_come_from_the_composition reads the weight per term, with the same expectations: at 1 part {1, missed}, at 2 parts {1}, no excluded pair.
    • one_pane_chain_serves_a_ratio_and_its_operands asks that every term with that producer has weight One.
    • kept_builds_give_the_weights_of_fresh_builds compares terms and pairs. On the three-quotient it asserts that pair checks run (a 2-part producer with two weight One terms read together), no longer that a pair is excluded. Today's excluded pairs there all pair C1103 made approximate in three operands at once, which the shares row now excludes.
    • accuracy_rows_follow_the_composition_on_a_ratio uses producers whose every term is unusable, and pairs of 2-part terms.
    • New unit test terms_read_together_where_their_paths_can_meet.

Plan cost, Stage 2 candidates and planning time

UPDATE_SNAPSHOTS=1 left every corpus and sketch-plan snapshot file unchanged. Stage 2 is untouched (counts identical). --run-until stage3 wall time per snapshot input, #1222 vs this PR, release builds on one host:

snapshot Stage 2 candidates, before after seconds, before after
o11y_bench/q02 248 248 0.12 0.12
o11y_bench/q05 498 498 0.24 0.25
o11y_bench/q06 251 251 0.12 0.12
o11y_bench/q11 7 7 0.02 0.02
o11y_bench/q12 2 2 0.01 0.01
o11y_bench/q18 737 737 0.29 0.29
o11y_bench/q19 495 495 0.26 0.27
o11y_bench/q20 4 4 0.01 0.01
o11y_bench/q21 368 368 0.11 0.12
o11y_bench/q22 247 247 0.10 0.11
o11y_bench/q23 247 247 0.10 0.11
o11y_bench/q25 495 495 0.26 0.26
o11y_bench/q26 247 247 0.11 0.11
o11y_bench/q27 4 4 0.01 0.02
sketch/shared-quantiles-1h-mvp 1801 1801 0.89 0.95
sketch/nested-windows-15m-1h-mvp 2281 2281 1.24 1.34
sketch/ratio-of-quantiles-1h-mvp 1802 1802 2.46 1.60
sketch/quantile-single-mvp 201 201 0.08 0.08
sketch/quantile-grouped-mvp 108 108 0.06 0.06
sketch/topk-rate-guarantees-ignored-mvp 87 87 0.05 0.05
sketch/fuzz-issue903-raw-retained 135 135 0.08 0.08
sketch/random-mixed-seed-862-w02-q2-q7-mvp 136 136 0.09 0.09
sketch/random-mixed-seed-862-w01-q3-q8-mvp 226 226 0.11 0.11
sketch/random-mixed-seed-862-long-w01-q3-mvp 72 72 0.09 0.09
total 6.89 6.24

Renames (2026-11-05)

PropagationFactor → AccuracyWeight, field lambda → weight, MILP variables y (q depends on a term) → d (q depends on a term); rebased on #1215–#1222's restack. cargo fmt --all -- --check; cargo clippy --workspace --all-targets -- -D warnings: clean; cargo test -p asap-plan-selection --lib: 142 passed.

Viewer

Viewer tests not updated (#1203 decision 6).

Verification

  • cargo fmt --all -- --check; cargo clippy --workspace --all-targets -- -D warnings: clean (on this head).
  • cargo test -p asap-plan-selection --lib: 142 passed (on this head).
  • Before the restack onto refactor(stage1): summary readouts and inputs as one operator over groups; Memo::add_candidate; one node equality in Pass 1 (#1203 2a-2) #1218's Memo::add_candidate (a change of Stage 1 code only, same candidates):
    • cargo test -p asap-integration-tests --test integration -- memo_enumeration reuse_rules operator_sharing sharing_issue_regressions finer_aggregate_reuse planner_layering_example having_shared_aggregate stage1_logical_asap_optimization joint_search_equivalence sketch_rollup rollup_controlled production_rollup_workloads: 154 passed, 23 ignored (pre-existing).
    • cargo test -p asap-devtools --test devtools -- asap_planner:: asap_planner_acceptance:: memo_pricing:: viewer_contract::: 58 passed, 1 ignored (pre-existing).
    • UPDATE_SNAPSHOTS=1 cargo test --release -p asap-devtools --test corpus_snapshots -- --include-ignored: 9 passed, no file changed.
  • 2a-6 on top of this PR: ratio_weights_come_from_the_composition and kept_builds_give_the_weights_of_fresh_builds, which fail on 2a-6 with per-candidate weights, pass.

Left

@zzylol zzylol added CSE optimizer Workload optimization and plan selection refactor labels Nov 5, 2026
@zzylol
zzylol force-pushed the refactor/1203-2a-5-scan-labels branch from 010cebc to 362d7d9 Compare November 5, 2026 21:14
@zzylol
zzylol force-pushed the refactor/1203-2a-row-k-terms branch from e955c95 to ef65e65 Compare November 5, 2026 21:19
@zzylol zzylol changed the title refactor(plan-selection): row (K) terms per path of edges, not a weight per candidate (#1203 2a, accuracy terms) refactor(plan-selection): row (K) terms per path of edges, interim until step 6 (#1203 2a) Nov 5, 2026
…ht per candidate (#1203)

Stage 3's accuracy weight moves from the approximate candidate to each
path of edges (u, c, i) through which a query reaches it: one producer
read by a ratio's numerator and denominator has a term for each, each
checked by the composition alone, pairs of terms checked together. Row 8
and the warm start constrain and count the paths a plan depends on
(d[q, t]). The per-candidate map (accuracy_weights, accuracy_positions.rs)
is deleted; AccuracyWeight is now per term. Interim until #1207 step 6
replaces the weights with row (K)'s constant lambda_{q,c,i}. Corpus and
sketch-plan snapshots unchanged.
@zzylol

zzylol commented Nov 6, 2026

Copy link
Copy Markdown
Contributor Author

Closed: replaced by #1207's new order (interfaces first, then one PR per stage), starting with PR 0 #1226. The code on this branch is reused in PRs 1–8; the branch is kept.

@zzylol zzylol closed this Nov 6, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CSE optimizer Workload optimization and plan selection refactor

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant