From 991e67ff66c99d7a651cc8f22cc6dd14d98095eb Mon Sep 17 00:00:00 2001 From: zzylol <50204836+zzylol@users.noreply.github.com> Date: Thu, 5 Nov 2026 18:09:17 +0000 Subject: [PATCH 01/44] docs(concepts): memo groups, properties, requirements and the property decision tree Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01K1M6br9PL4SDeVS1XpezKJ --- docs/design_docs/concepts/README.md | 1 + .../memo-groups-properties-requirements.md | 307 ++++++++++++++++++ 2 files changed, 308 insertions(+) create mode 100644 docs/design_docs/concepts/memo-groups-properties-requirements.md diff --git a/docs/design_docs/concepts/README.md b/docs/design_docs/concepts/README.md index 8510e12e9..c204e363a 100644 --- a/docs/design_docs/concepts/README.md +++ b/docs/design_docs/concepts/README.md @@ -7,3 +7,4 @@ These pages introduce the vocabulary and representations used throughout the rep - [Mapping concepts](asap-aware-mapping.md) - [Pre-ASAP IR](pre-asap-ir.md) - [Post-ASAP IR](post-asap-ir.md) +- [Memo groups, properties and requirements](memo-groups-properties-requirements.md) diff --git a/docs/design_docs/concepts/memo-groups-properties-requirements.md b/docs/design_docs/concepts/memo-groups-properties-requirements.md new file mode 100644 index 000000000..74d5b56a8 --- /dev/null +++ b/docs/design_docs/concepts/memo-groups-properties-requirements.md @@ -0,0 +1,307 @@ +# Memo groups, properties and requirements + +This page defines the memo's main data structures: groups, candidates, properties and requirements. It also lists every property and requirement, and gives the decision tree Stage 1 and Stage 2 walk to enumerate candidates. The symbols follow the planner formulation: +- a candidate is $v = (e_v, p_v)$; +- its logical properties are $p^{\log}$ and its physical properties are $p^{\mathrm{phys}}$; +- a consumer input's requirements are $r^{\log}_{c,i}$ and $r^{\mathrm{phys}}_{c,i}$; +- whether a producer satisfies them is $\alpha^{\log}$ and $\alpha^{\mathrm{phys}}$. + +## Example + +Query A: `quantile_over_time(0.95, latency[1m])` with `max_error 0.01`. Query B: `quantile_over_time(0.99, latency[1m]) / quantile_over_time(0.50, latency[1m])` with the same target. + +``` +G1 Scan latency[1m] + producers: v1 = Scan (QueryTime, None) +G2 KLL state over G1 one group, whatever the size + producers: v2 = SummaryAgg[KLL](G1) θ = {k = 200} ε = 0.008 δ = 0.01 + v3 = SummaryAgg[KLL](G1) θ = {k = 345} ε = 0.005 δ = 0.01 + (each also expanded by Stage 2 into its (τ, ℓ, σ) alternatives) +G3 p95 of G2 producers: SummaryEstimate[q=0.95](G2) +G4 p99 of G2 producers: SummaryEstimate[q=0.99](G2) +G5 p50 of G2 producers: SummaryEstimate[q=0.50](G2) +G6 G4 / G5 producers: Divide(G4, G5) input requirement: error_cap on each input +Query roots: A → G3, B → G6 +``` + +The ε values are illustrative. + +- G2 is one group: the KLL state over `latency[1m]`. Its sizes are candidates of G2, not separate groups. +- G3, G4 and G5 are consumers of G2. They are not stored in G2; each is a group with its own producers whose expression reads G2. +- A meets 0.01 with either size. B adds its two operands' errors (0.016 with `k = 200`, 0.010 with `k = 345`) and needs `k = 345` for both, so a plan where all three read v3 builds one KLL state. + +## 1. Properties and requirements + +### Properties + +A property is a value a candidate has. Every property is one dimension of a product: a struct with one field per dimension, never a flat enum that lists the combinations. Invalid combinations are rejected by the struct's checked constructor. + +| Dimension | Values | Logical or physical | Fixed by | +|---|---|---|---| +| Which sketch to use (representation) | `Exact` (exact accumulator kinds: Sum, Count, Min, Max, Increase, Rate, IRate), or `Summary { family }` with family in KLL, DDSketch, CMS, CountSketch, CMS with heap, CountSketch with heap, UnivMon, HLL, KMV, the Hydra variants | logical | the operator $\operatorname{op}_v$ | +| Sketch parameters $\theta$ | per family, from a predefined list: KLL `k`; DDSketch `alpha`; CMS / CountSketch `width × depth`; CMS / CountSketch with heap `width × depth × heap_size`; UnivMon `heap_size × rows × cols × layers`; HLL `precision`; KMV `k` | logical | enumerated | +| Error bound $(\varepsilon_v, \delta_v)$ | precomputed per $\theta$, in the family's accuracy metric (below); exact is $(0, 0)$ | logical | derived from $\theta$, not chosen | +| Window configuration $\omega$ | none; or tumbling windows of width $w$ and count $n$, with $w$ from the gcd closure of the readers' ranges and steps. Later: sliding windows by merge or by subtraction | logical | enumerated | +| Timing $\tau$ | `IngestionTime`, `QueryTime` | physical | enumerated | +| Lifetime $\ell$ | `None`, `OneEvaluation(Medium)`, `AcrossEvaluations(Medium)`, with `Medium` in `Memory`, `Disk` | physical | enumerated | +| Schedule $\sigma$ | only when $\ell \ne$ None. `Periodic { interval, phase }` for `OneEvaluation`; `TumblingWindowGrid { phase }` for `AcrossEvaluations` | physical | enumerated from the readers' schedules | + +A candidate is materialized when $\ell \ne$ None. This is derived, not stored. + +Invalid combinations, rejected by `new`: + +| Combination | Why | +|---|---| +| (IngestionTime, OneEvaluation) | there is no evaluation at ingestion time | +| $\ell$ = None with a schedule | where an unmaterialized result runs is its reader's execution context, not a property | +| $\ell \ne$ None without a schedule | a materialized result must say on which times or grid it is delivered | +| OneEvaluation with a grid, AcrossEvaluations with a periodic schedule | each lifetime has one kind of schedule | +| `Exact` with a non-zero error bound | exact is $(0, 0)$ | + +Valid physical states: (QueryTime, None), (IngestionTime, None), (QueryTime, OneEvaluation(m)), (QueryTime, AcrossEvaluations(m)), (IngestionTime, AcrossEvaluations(m)), for each medium $m$. + +### Requirements + +A requirement is a constraint that a consumer, a query or the deployment puts on producers. One requirement can constrain several property dimensions, and one dimension can be constrained by several requirements. + +| Requirement | Who states it | Values | Constrains | Checked where | +|---|---|---|---|---| +| Accuracy | each query, at its root | metric: absolute (in the value's units, or times the stream's L1 or L2 norm), relative, rank; bound $\varepsilon_q$; failure probability $\delta_q$; scope: per value, whole result, or groups above a share of the records | representation and $\theta$ of every approximate candidate the query reads, through their $(\varepsilon_v, \delta_v)$ | Stage 3, row (K): the composed error at the root. Not per edge, because a per-input bound would fix each input's share of the budget | +| Validity cap `error_cap` | a non-linear consumer (ratio, product), per input | $\varepsilon$ cap | $\theta$ of the producer | per edge, $\alpha^{\log}$: row (K)'s linear bound holds only below the cap | +| Summary family | a consumer that reads summary state, per input | a family, e.g. KLL for a quantile readout | representation | per edge, $\alpha^{\log}$ | +| Window | a consumer that reads windows, per input | the reader's range and step | window configuration: the width must divide both | per edge, $\alpha^{\log}$ | +| Capability of ingestion-time compute | the deployment and the data workload | the source has an ingestion pipeline; the executor supports the family; query-time results may be kept across evaluations | timing; representation; (QueryTime, AcrossEvaluations) | candidate generation: such candidates are not offered | +| Reader timing and evaluation | a consumer, per input | when it reads (ingestion or query time) and from which evaluation (same or later) | timing and lifetime of the producer | per edge, $\alpha^{\mathrm{phys}}$ | +| Reader schedule | a periodic consumer, per input | its interval and phase $(I_c, \varphi_c)$ | schedule of the producer | per edge, $\alpha^{\mathrm{phys}}$ | +| Latency, resource budgets | the workload | per-query latency bound; memory and CPU budgets | the whole plan | Stage 3, rows (L) and (B) | + +Accuracy metrics that may be added later, when a primitive needs them: top-k recall or precision, and a false-positive rate for membership summaries. + +How requirements map to property dimensions: + +``` +Accuracy (query) ──► representation, θ (through ε_v, δ_v; row (K)) +error_cap (edge) ──► θ +Summary family (edge) ──► representation +Window (edge) ──► window configuration +Ingestion capability (deploy)──► timing, representation, lifetime +Reader timing/evaluation ──► timing, lifetime +Reader schedule ──► schedule +Latency, budgets (workload) ──► every dimension, through cost +``` + +## 2. Decision tree of properties + +Each memo group's candidates are the leaves of a tree. Every level decides one property dimension, and each edge out of a node is one value of that dimension. A path from the root to a leaf is one candidate. + +```mermaid +flowchart TD + G["Group G
(what is computed)"] --> R1["Exact"] + G --> R2["Summary: KLL"] + G --> R3["Summary: DDSketch"] + G --> R4["…other families
(prune: executor support,
consumer's family requirement)"] + + R2 --> T1["θ: k = 200
ε = 0.008"] + R2 --> T2["θ: k = 345
ε = 0.005"] + R2 --> T3["θ: k = …
(prune: ε above every
reading query's target)"] + + T2 --> W1["ω: no windows"] + T2 --> W2["ω: tumbling, width w
(prune: w must divide
some reader's range and step)"] + + W2 --> P1["τ = QueryTime"] + W2 --> P2["τ = IngestionTime
(prune: no ingestion pipeline)"] + + P1 --> L1["ℓ = None"] + P1 --> L2["ℓ = OneEvaluation(m)"] + P1 --> L3["ℓ = AcrossEvaluations(m)
(prune: deployment keeps
no query-time results)"] + P2 --> L4["ℓ = None"] + P2 --> L5["ℓ = AcrossEvaluations(m)"] + P2 -. invalid .-> L6["ℓ = OneEvaluation"] + + L2 --> S1["σ: Periodic (I, φ)
from the gcd closure
of the readers' schedules"] + L3 --> S2["σ: grid phase φ mod w"] + L5 --> S3["σ: grid phase 0"] +``` + +The levels, top to bottom: + +| Level | Dimension | Stage | Values offered | Pruned when | +|---|---|---|---|---| +| 1 | representation (operator) | Stage 1, Pass 1 and Pass 2 rules | each way to compute the group | the executor cannot build the family; no consumer's family requirement accepts it | +| 2 | $\theta$ | Stage 1, Pass 1 | the family's predefined list | $\varepsilon_v$ alone exceeds every reading query's target | +| 3 | window configuration $\omega$ | Stage 1, Pass 2 | tumbling widths from the gcd closure of readers' ranges and steps | no reader's range and step is a multiple of the width | +| 4 | timing $\tau$ | Stage 2 | QueryTime; IngestionTime | the source has no ingestion pipeline | +| 5 | lifetime $\ell$ | Stage 2 | None, OneEvaluation(m), AcrossEvaluations(m) | invalid with $\tau$; the deployment keeps no query-time results across evaluations | +| 6 | schedule $\sigma$ | Stage 2 | the gcd closure $\Sigma$ of the readers' schedules | only offered when $\ell \ne$ None | + +What the tree does not decide: +- **Which producer each consumer input reads.** That is the edge choice $y_{u,c,i}$ in Stage 3. Two consumers of one group may read different leaves, e.g. A reads `k = 200` and B reads `k = 345`. +- **Where an unmaterialized candidate runs.** A candidate with $\ell$ = None runs inside each reader's execution context; Stage 3 chooses it. +- **How much a materialized candidate retains.** The amount $h_v$ depends on the readers' longest range; Stage 3 chooses it. + +### Order of decisions + +**The order does not affect correctness.** The set of leaves is the product of the dimensions' value sets, minus the invalid combinations and the pruned values. Every prune rule depends only on the value it removes and on values already decided above it, so any order that decides a pruned dimension after the dimensions its prune rule reads yields the same set of leaves. + +**The order affects the efficiency of enumeration.** The number of nodes visited is $\sum_{d} \prod_{d' \le d} |\text{values kept at } d'|$, so a dimension that prunes much should come early, and a dimension whose values multiply without pruning should come late. Four considerations set the order above: + +1. **Logical before physical.** Logical dimensions (representation, $\theta$, $\omega$) change the result's values or error; physical dimensions ($\tau$, $\ell$, $\sigma$) change only when and how long it is computed. Deciding logical dimensions first lets Stage 1 hash-cons groups and check $\alpha^{\log}$ on edges once, before Stage 2 multiplies each logical candidate by its physical alternatives. +2. **Strong prunes early.** The family is pruned by the executor and by consumers' family requirements, and $\theta$ by the accuracy target, before the window and physical levels multiply them. +3. **Dependent dimensions adjacent.** $\ell$ is constrained by $\tau$ (no OneEvaluation at ingestion), and $\sigma$ exists only when $\ell \ne$ None and its kind follows $\ell$. Deciding them in the order $\tau \to \ell \to \sigma$ means each invalid combination is pruned at the first level where it can be detected. +4. **Deployment-wide prunes as early as they can go.** If the deployment has no ingestion pipeline, the IngestionTime subtree is pruned once per candidate. Moving $\tau$ above $\omega$ would prune it once per group instead; this is an alternative order with the same leaves. + +Alternative orders worth comparing: +- **$\tau$ first.** In a deployment where most sources have no ingestion pipeline, deciding timing at the group level prunes half the tree before $\theta$ and $\omega$ multiply it. +- **$\theta$ last among logical dimensions.** $\theta$ has the most values. Deciding $\omega$ before $\theta$ visits fewer nodes when many widths are pruned by readers' ranges. +- **Requirement-driven (top-down) vs producer-driven (bottom-up).** Top-down enumeration starts from each consumer's requirement and offers only the values it accepts; bottom-up offers every value and lets $\alpha$ filter edges. Top-down visits fewer nodes but must union the requirements of all consumers of a group, so that sharing between consumers is not lost. + +## 3. Modeling a memo group by its producers + +### Producers, not consumers + +A memo group stores its **producers**: the candidates that compute it. It does not store its consumers. A consumer of $G$ is a candidate of another group whose expression reads $G$; the producer–consumer edges $(u, c, i)$ connect them. + +| | Group stores its producers (chosen) | Group stores each consumer's own producer | +|---|---|---| +| Group identity | what is computed: operator and input groups, without properties | what is computed **for one reader** | +| Two readers of the same state | one group, two edges | two copies, joined only by an extra "shared build" candidate | +| A rule adds a new way to compute | one new producer in the group; every reader sees it | the rule must copy every reader that could use it | +| Where requirements live | on the consumer's input, $r_{c,i}$ | on the producer, so a producer is fixed to one reader | +| Accuracy | checked at each query root over the chosen edges | fixed per producer as a share of one query's budget | + +Groups found by hash-consing are therefore shared automatically. Two queries that read the same state read one group, and Stage 3 decides through edges whether they read the same producer. + +Consumers of a group are derived, not stored: `consumers(G)` is the set of candidates whose expression has $G$ as an input. + +### Memo group data structure + +```rust +/// The memo: every group, and each query's root over groups. +pub struct Memo { + pub groups: Vec, + pub query_roots: Vec>, +} + +/// G: one logical result, or one state family, over specified input data. +/// Groups are hash-consed on `identity`, so an identical sub-expression of two +/// queries is one group. +pub struct MemoGroup { + pub id: MemoGroupId, + /// What is computed, without any property: the operator kind, its input + /// groups, and the arguments that change the result's meaning (family, + /// reduction, grouping, filter, query window). Sketch parameters, error + /// bounds and window layout are not part of it. + pub identity: GroupIdentity, + /// The producers of this group: every way to compute it. + pub candidates: Vec, +} +``` + +### Property definition + +```rust +/// p_v^log: one field per dimension. `new` rejects `Exact` with a non-zero +/// error bound. +pub struct LogicalProperties { + representation: Representation, // which sketch, and θ + error_bound: ErrorBound, // (ε_v, δ_v), precomputed per θ + window_layout: Option, // ω_v +} +pub enum Representation { + Exact, + Summary { family: SummaryFamily, parameters: SummaryParameters }, +} +pub struct ErrorBound { metric: AccuracyMetric, epsilon: f64, delta: f64 } +pub enum WindowLayout { Tumbling { width_ms: u64, count: u32 } } // sliding variants later + +/// p_v^phys: one field per dimension. `new` rejects the invalid combinations. +pub struct PhysicalProperties { + timing: ExecutionTiming, // τ + lifetime: Lifetime, // ℓ + schedule: Option, // σ, only when ℓ ≠ None +} +pub enum ExecutionTiming { IngestionTime, QueryTime } +pub enum Lifetime { None, OneEvaluation(Medium), AcrossEvaluations(Medium) } +pub enum Medium { Memory, Disk } +pub enum Schedule { + Periodic { interval_ms: u64, phase_ms: u64 }, + TumblingWindowGrid { phase_ms: u64 }, +} +``` + +### Property requirement definition + +```rust +/// r^log_{c,i}: one optional constraint per dimension; `None` accepts any value. +pub struct LogicalRequirement { + pub family: Option, + pub window: Option, // the reader's range and step + pub error_cap: Option, // validity of row (K)'s linear bound +} + +/// r^phys_{c,i}: one field per dimension. `new` rejects (IngestionTime, Later). +pub struct PhysicalRequirement { + reader_timing: ExecutionTiming, + reads_from: Evaluation, // Same or Later + reader_schedule: Option, // (I_c, φ_c) of a periodic reader +} + +/// α^log and α^phys: one check per dimension, joined by "and". +pub fn satisfies(producer: &LogicalProperties, requirement: &LogicalRequirement) -> bool; +pub fn satisfies_physical(producer: &PhysicalProperties, requirement: &PhysicalRequirement) -> bool; + +/// The accuracy target stays at the query root; Stage 3 checks it (row (K)). +pub struct QueryRequirement { pub accuracy: AccuracyTarget } +``` + +### Candidate data structure definition + +```rust +/// v = (e_v, p_v^log): one producer of a memo group, from Stage 1. +pub struct CandidateForMemoGroup { + pub expression: CandidateExpression, // e_v + pub logical_properties: LogicalProperties, // p_v^log + /// r^log_{c,i}: one per input group of `expression`. + pub input_requirements: Vec, +} + +/// e_v = op_v(G_{v,1}, …, G_{v,n}): one operator with its arguments, whose +/// children are groups. No private nodes. +pub struct CandidateExpression { pub operator: Operator } + +/// One physical alternative of a logical candidate, from Stage 2. +pub struct PhysicalCandidate { + pub logical: (MemoGroupId, usize), // which logical candidate + pub properties: PhysicalProperties, // p_v^phys + pub input_requirements: Vec, // r^phys_{c,i} +} + +/// (u, c, i): producer u may serve input i of consumer c, when α^log and α^phys hold. +pub struct Edge { pub producer: CandidateId, pub consumer: CandidateId, pub input: usize } +``` + +### Extensibility + +**Add a value to a property dimension**, e.g. a new summary family or `Medium::Disk`: +1. Add the enum variant. +2. Every exhaustive `match` on the dimension stops compiling until it handles the new value. This is why matches on properties name every value and never use a `_` arm. +3. For a family: add its predefined $\theta$ list, its error-bound function, its cost, and the readouts it answers. +4. Add any new invalid combinations to `new`. + +**Add a property dimension**, e.g. a compression codec or a partitioning: +1. Decide whether it is logical (it changes the result's values or error) or physical (it changes only when, where or how long the result is computed). Add a field to `LogicalProperties` or `PhysicalProperties`. +2. If consumers can constrain it, add an optional field to the matching requirement struct and one check to `satisfies` or `satisfies_physical`. +3. Add its invalid combinations with the other dimensions to `new`. +4. Add a level to the decision tree, with its prune rule, and place it by the considerations in "Order of decisions". +5. Add its cost terms. + +**Add a requirement:** +1. If it constrains one edge (a consumer input), add an optional field to `LogicalRequirement` or `PhysicalRequirement`, set it where the consumer is created, and check it in `satisfies`. +2. If it constrains a whole query or the whole plan (accuracy, latency, budgets), add it to the query's or workload's requirements and a row to Stage 3. Do not split it into per-edge requirements: that fixes each input's share in advance. +3. If it comes from the deployment, prune the values it rules out during candidate generation. + +**Add a rule:** +1. A rule matches over groups and their candidates' expressions, never over a reader's private operators. +2. It adds candidates $(e, p^{\log})$ to an existing group, or adds a new group with its candidates. It never copies a reader: every existing consumer of the group sees the new candidate through edges. +3. A candidate whose result has the same meaning as an existing group is added to that group, so hash-consing keeps sharing. +4. Each new candidate states its input requirements. From 965a556347f3bc1b7b80af639c191b3d4097c3b1 Mon Sep 17 00:00:00 2001 From: zzylol <50204836+zzylol@users.noreply.github.com> Date: Thu, 5 Nov 2026 19:01:51 +0000 Subject: [PATCH 02/44] docs(concepts): memo group as expression, input requirements and candidates over existing IR Co-Authored-By: Claude Opus 5.5 --- .../memo-groups-properties-requirements.md | 228 +++++++++++------- 1 file changed, 136 insertions(+), 92 deletions(-) diff --git a/docs/design_docs/concepts/memo-groups-properties-requirements.md b/docs/design_docs/concepts/memo-groups-properties-requirements.md index 74d5b56a8..37623e34c 100644 --- a/docs/design_docs/concepts/memo-groups-properties-requirements.md +++ b/docs/design_docs/concepts/memo-groups-properties-requirements.md @@ -1,7 +1,7 @@ # Memo groups, properties and requirements This page defines the memo's main data structures: groups, candidates, properties and requirements. It also lists every property and requirement, and gives the decision tree Stage 1 and Stage 2 walk to enumerate candidates. The symbols follow the planner formulation: -- a candidate is $v = (e_v, p_v)$; +- a group has an expression $e_G$ over its producer groups, and a candidate $v$ of it has properties $p_v$; - its logical properties are $p^{\log}$ and its physical properties are $p^{\mathrm{phys}}$; - a consumer input's requirements are $r^{\log}_{c,i}$ and $r^{\mathrm{phys}}_{c,i}$; - whether a producer satisfies them is $\alpha^{\log}$ and $\alpha^{\mathrm{phys}}$. @@ -30,7 +30,7 @@ The ε values are illustrative. - G3, G4 and G5 are consumers of G2. They are not stored in G2; each is a group with its own producers whose expression reads G2. - A meets 0.01 with either size. B adds its two operands' errors (0.016 with `k = 200`, 0.010 with `k = 345`) and needs `k = 345` for both, so a plan where all three read v3 builds one KLL state. -## 1. Properties and requirements +## 1. Properties and requirements over properties ### Properties @@ -38,9 +38,9 @@ A property is a value a candidate has. Every property is one dimension of a prod | Dimension | Values | Logical or physical | Fixed by | |---|---|---|---| -| Which sketch to use (representation) | `Exact` (exact accumulator kinds: Sum, Count, Min, Max, Increase, Rate, IRate), or `Summary { family }` with family in KLL, DDSketch, CMS, CountSketch, CMS with heap, CountSketch with heap, UnivMon, HLL, KMV, the Hydra variants | logical | the operator $\operatorname{op}_v$ | -| Sketch parameters $\theta$ | per family, from a predefined list: KLL `k`; DDSketch `alpha`; CMS / CountSketch `width × depth`; CMS / CountSketch with heap `width × depth × heap_size`; UnivMon `heap_size × rows × cols × layers`; HLL `precision`; KMV `k` | logical | enumerated | -| Error bound $(\varepsilon_v, \delta_v)$ | precomputed per $\theta$, in the family's accuracy metric (below); exact is $(0, 0)$ | logical | derived from $\theta$, not chosen | +| Which family to use (operator) | `Exact` (exact accumulator kinds: Sum, Count, Min, Max, Increase, Rate, IRate), or `Summary { family }` with family in KLL, DDSketch, CMS, CountSketch, CMS with heap, CountSketch with heap, UnivMon, HLL, KMV, the Hydra variants | logical | enumerated | +| Parameter configuration $\theta$ | per family, from a predefined list: KLL `k`; DDSketch `alpha`; CMS / CountSketch `width × depth`; CMS / CountSketch with heap `width × depth × heap_size`; UnivMon `heap_size × rows × cols × layers`; HLL `precision`; KMV `k` | logical | enumerated | +| Error bound $(\varepsilon_v, \delta_v)$ | computed on demand from $\theta$ and the statistic, in the family's accuracy metric (below); exact is $(0, 0)$ | logical | derived from $\theta$, not stored or chosen | | Window configuration $\omega$ | none; or tumbling windows of width $w$ and count $n$, with $w$ from the gcd closure of the readers' ranges and steps. Later: sliding windows by merge or by subtraction | logical | enumerated | | Timing $\tau$ | `IngestionTime`, `QueryTime` | physical | enumerated | | Lifetime $\ell$ | `None`, `OneEvaluation(Medium)`, `AcrossEvaluations(Medium)`, with `Medium` in `Memory`, `Disk` | physical | enumerated | @@ -56,21 +56,20 @@ Invalid combinations, rejected by `new`: | $\ell$ = None with a schedule | where an unmaterialized result runs is its reader's execution context, not a property | | $\ell \ne$ None without a schedule | a materialized result must say on which times or grid it is delivered | | OneEvaluation with a grid, AcrossEvaluations with a periodic schedule | each lifetime has one kind of schedule | -| `Exact` with a non-zero error bound | exact is $(0, 0)$ | Valid physical states: (QueryTime, None), (IngestionTime, None), (QueryTime, OneEvaluation(m)), (QueryTime, AcrossEvaluations(m)), (IngestionTime, AcrossEvaluations(m)), for each medium $m$. -### Requirements +### Requirements over properties A requirement is a constraint that a consumer, a query or the deployment puts on producers. One requirement can constrain several property dimensions, and one dimension can be constrained by several requirements. | Requirement | Who states it | Values | Constrains | Checked where | |---|---|---|---|---| -| Accuracy | each query, at its root | metric: absolute (in the value's units, or times the stream's L1 or L2 norm), relative, rank; bound $\varepsilon_q$; failure probability $\delta_q$; scope: per value, whole result, or groups above a share of the records | representation and $\theta$ of every approximate candidate the query reads, through their $(\varepsilon_v, \delta_v)$ | Stage 3, row (K): the composed error at the root. Not per edge, because a per-input bound would fix each input's share of the budget | +| Accuracy | each query, at its root | metric: absolute (in the value's units, or times the stream's L1 or L2 norm), relative, rank; bound $\varepsilon_q$; failure probability $\delta_q$; scope: per value, whole result, or groups above a share of the records | operator family and $\theta$ of every approximate candidate the query reads, through their $(\varepsilon_v, \delta_v)$ | Stage 3, row (K): the composed error at the root. Not per edge, because a per-input bound would fix each input's share of the budget | | Validity cap `error_cap` | a non-linear consumer (ratio, product), per input | $\varepsilon$ cap | $\theta$ of the producer | per edge, $\alpha^{\log}$: row (K)'s linear bound holds only below the cap | -| Summary family | a consumer that reads summary state, per input | a family, e.g. KLL for a quantile readout | representation | per edge, $\alpha^{\log}$ | +| Summary family | a consumer that reads summary state, per input | a family, e.g. KLL for a quantile readout | operator family | per edge, $\alpha^{\log}$ | | Window | a consumer that reads windows, per input | the reader's range and step | window configuration: the width must divide both | per edge, $\alpha^{\log}$ | -| Capability of ingestion-time compute | the deployment and the data workload | the source has an ingestion pipeline; the executor supports the family; query-time results may be kept across evaluations | timing; representation; (QueryTime, AcrossEvaluations) | candidate generation: such candidates are not offered | +| Capability of ingestion-time compute | the deployment and the data workload | the source has an ingestion pipeline; the executor supports the family; query-time results may be kept across evaluations | timing; operator family; (QueryTime, AcrossEvaluations) | candidate generation: such candidates are not offered | | Reader timing and evaluation | a consumer, per input | when it reads (ingestion or query time) and from which evaluation (same or later) | timing and lifetime of the producer | per edge, $\alpha^{\mathrm{phys}}$ | | Reader schedule | a periodic consumer, per input | its interval and phase $(I_c, \varphi_c)$ | schedule of the producer | per edge, $\alpha^{\mathrm{phys}}$ | | Latency, resource budgets | the workload | per-query latency bound; memory and CPU budgets | the whole plan | Stage 3, rows (L) and (B) | @@ -80,11 +79,11 @@ Accuracy metrics that may be added later, when a primitive needs them: top-k rec How requirements map to property dimensions: ``` -Accuracy (query) ──► representation, θ (through ε_v, δ_v; row (K)) +Accuracy (query) ──► operator family, θ (through ε_v, δ_v; row (K)) error_cap (edge) ──► θ -Summary family (edge) ──► representation +Summary family (edge) ──► operator family Window (edge) ──► window configuration -Ingestion capability (deploy)──► timing, representation, lifetime +Ingestion capability (deploy)──► timing, operator family, lifetime Reader timing/evaluation ──► timing, lifetime Reader schedule ──► schedule Latency, budgets (workload) ──► every dimension, through cost @@ -127,7 +126,7 @@ The levels, top to bottom: | Level | Dimension | Stage | Values offered | Pruned when | |---|---|---|---|---| -| 1 | representation (operator) | Stage 1, Pass 1 and Pass 2 rules | each way to compute the group | the executor cannot build the family; no consumer's family requirement accepts it | +| 1 | family (operator) | Stage 1, Pass 1 and Pass 2 rules | each way to compute the group | the executor cannot build the family; no consumer's family requirement accepts it | | 2 | $\theta$ | Stage 1, Pass 1 | the family's predefined list | $\varepsilon_v$ alone exceeds every reading query's target | | 3 | window configuration $\omega$ | Stage 1, Pass 2 | tumbling widths from the gcd closure of readers' ranges and steps | no reader's range and step is a multiple of the width | | 4 | timing $\tau$ | Stage 2 | QueryTime; IngestionTime | the source has no ingestion pipeline | @@ -145,7 +144,7 @@ What the tree does not decide: **The order affects the efficiency of enumeration.** The number of nodes visited is $\sum_{d} \prod_{d' \le d} |\text{values kept at } d'|$, so a dimension that prunes much should come early, and a dimension whose values multiply without pruning should come late. Four considerations set the order above: -1. **Logical before physical.** Logical dimensions (representation, $\theta$, $\omega$) change the result's values or error; physical dimensions ($\tau$, $\ell$, $\sigma$) change only when and how long it is computed. Deciding logical dimensions first lets Stage 1 hash-cons groups and check $\alpha^{\log}$ on edges once, before Stage 2 multiplies each logical candidate by its physical alternatives. +1. **Logical before physical.** Logical dimensions (operator family, $\theta$, $\omega$) change the result's values or error; physical dimensions ($\tau$, $\ell$, $\sigma$) change only when and how long it is computed. Deciding logical dimensions first lets Stage 1 hash-cons groups and check $\alpha^{\log}$ on edges once, before Stage 2 multiplies each logical candidate by its physical alternatives. 2. **Strong prunes early.** The family is pruned by the executor and by consumers' family requirements, and $\theta$ by the accuracy target, before the window and physical levels multiply them. 3. **Dependent dimensions adjacent.** $\ell$ is constrained by $\tau$ (no OneEvaluation at ingestion), and $\sigma$ exists only when $\ell \ne$ None and its kind follows $\ell$. Deciding them in the order $\tau \to \ell \to \sigma$ means each invalid combination is pruned at the first level where it can be detected. 4. **Deployment-wide prunes as early as they can go.** If the deployment has no ingestion pipeline, the IngestionTime subtree is pruned once per candidate. Moving $\tau$ above $\omega$ would prune it once per group instead; this is an alternative order with the same leaves. @@ -155,26 +154,31 @@ Alternative orders worth comparing: - **$\theta$ last among logical dimensions.** $\theta$ has the most values. Deciding $\omega$ before $\theta$ visits fewer nodes when many widths are pruned by readers' ranges. - **Requirement-driven (top-down) vs producer-driven (bottom-up).** Top-down enumeration starts from each consumer's requirement and offers only the values it accepts; bottom-up offers every value and lets $\alpha$ filter edges. Top-down visits fewer nodes but must union the requirements of all consumers of a group, so that sharing between consumers is not lost. -## 3. Modeling a memo group by its producers +## 3. Modeling a memo group -### Producers, not consumers +A memo group stores its **producers**: all candidate ways to compute its result, before Stage 3 selects any. A consumer of $G$ is a candidate of another group whose expression reads $G$; the producer–consumer edges $(u, c, i)$ connect them. Here, `consumers(G)` is derived from those input references rather than stored in the group. -A memo group stores its **producers**: the candidates that compute it. It does not store its consumers. A consumer of $G$ is a candidate of another group whose expression reads $G$; the producer–consumer edges $(u, c, i)$ connect them. +> Side note: [CMU-DB's optd-original](https://github.com/cmu-db/optd-original) stores the list of children as input producers to a memo group; while storing consumers vs storing producers in the memo group structure to form the DAG are equivalent in terms of correctness and efficiency. -| | Group stores its producers (chosen) | Group stores each consumer's own producer | -|---|---|---| -| Group identity | what is computed: operator and input groups, without properties | what is computed **for one reader** | -| Two readers of the same state | one group, two edges | two copies, joined only by an extra "shared build" candidate | -| A rule adds a new way to compute | one new producer in the group; every reader sees it | the rule must copy every reader that could use it | -| Where requirements live | on the consumer's input, $r_{c,i}$ | on the producer, so a producer is fixed to one reader | -| Accuracy | checked at each query root over the chosen edges | fixed per producer as a share of one query's budget | +Candidate enumeration efficiency depends on: + +- **Property and requirement fields:** which dimensions are represented, their legal values, which values are derived, and which constraints permit pruning or sharing (Section 1). +- **Decision-tree order:** when each dimension is decided and its requirements are checked, so invalid combinations can be pruned before later dimensions multiply them (Section 2). + +Groups found by hash-consing expose the same producer alternatives to all their readers. Stage 3 decides whether readers use the same producer candidate or different ones. **Reusing a computation alternative in the memo is distinct from reusing its computed result at runtime. Selecting the same candidate for multiple readers does not by itself mean computing it only once.** -Groups found by hash-consing are therefore shared automatically. Two queries that read the same state read one group, and Stage 3 decides through edges whether they read the same producer. +- **Materialized candidate** ($\ell \ne$ None): its result is retained. Readers connected to the same materialized producer can reuse that result instead of computing it separately, subject to its lifetime and schedule. +- **Unmaterialized candidate** ($\ell =$ None): no independent shared result is retained. The computation runs as part of each reader's execution, so multiple readers selecting the same candidate may each execute it separately. -Consumers of a group are derived, not stored: `consumers(G)` is the set of candidates whose expression has $G$ as an input. +For example, if p50 and p99 select the same KLL candidate, a shared materialized producer builds and retains one KLL state for both readouts within the applicable lifetime and schedule. With an unmaterialized candidate, the p50 execution can build its own KLL state and the p99 execution another, even though both builds use the same candidate configuration. ### Memo group data structure +A memo group holds three things: +- **Query semantics:** its expression, and the producer groups the expression reads. +- **Eligibility:** for each input, the requirement a producer must meet. +- **Candidates:** the ways to compute it. + ```rust /// The memo: every group, and each query's root over groups. pub struct Memo { @@ -182,39 +186,66 @@ pub struct Memo { pub query_roots: Vec>, } -/// G: one logical result, or one state family, over specified input data. -/// Groups are hash-consed on `identity`, so an identical sub-expression of two -/// queries is one group. pub struct MemoGroup { - pub id: MemoGroupId, - /// What is computed, without any property: the operator kind, its input - /// groups, and the arguments that change the result's meaning (family, - /// reduction, grouping, filter, query window). Sketch parameters, error - /// bounds and window layout are not part of it. - pub identity: GroupIdentity, - /// The producers of this group: every way to compute it. - pub candidates: Vec, + // Query semantics. + /// The logical operator, from the existing IR. Its children are the + /// producer groups G_i it reads, in order (`expression.children()`), which + /// fix the input data and schema. + pub expression: Operator, + // Eligibility. + /// r_i: one requirement per producer group, by input index i. + pub input_requirements: Vec, + // Candidates. + /// The ways to compute this group; x_v in the MILP. + pub candidates: Vec, +} +``` + +The expression is the existing IR's [`Operator`](../../../crates/types/src/ir/operator/node.rs), with `C = MemoGroupId`. It wraps [`NonASAPOp`](../../../crates/types/src/ir/operator/non_asap.rs) or [`ASAPOp`](../../../crates/types/src/ir/operator/asap.rs), and each variant carries its own arguments and its inputs. For example, `Scan` carries the source, predicates and schema, and `Filter` carries a predicate and its child. No new identity type is needed. + +Hash-consing finds or inserts a group by its expression: the operator, its arguments and its input group ids. For example, the KLL state over one source and query window is one group for `k = 200` and `k = 345`; these are two candidates of that group. A different source, filter or query window gives a different group. Properties are not part of the expression: the family, sketch parameters, window layout, timing, lifetime and schedule belong to candidates. A requested query window belongs to the expression; the internal window layout used to compute it does not. + +`MemoGroupId` is the handle of an interned group. Looking up an existing expression returns that group's id instead of adding a second group. + +### Candidate data structure definition + +```rust +/// v: one way to compute its group. The group's expression says what is +/// computed; the candidate's properties say how. +pub struct Candidate { + /// One value per property dimension. + pub properties: Properties, +} +``` + +Stage 1 decides the logical dimensions of each candidate, and Stage 2 expands each one into its physical alternatives: + +```rust +pub struct Properties { + pub logical: LogicalProperties, // Stage 1 + pub physical: PhysicalProperties, // Stage 2 } + +/// (u, c, i): producer u may serve input i of consumer c when u's properties +/// satisfy c's group's input_requirements[i]. +pub struct Edge { pub producer: CandidateId, pub consumer: CandidateId, pub input: usize } ``` ### Property definition +A property is a struct with one field per dimension (P1, P2, …), each its own enum: + ```rust -/// p_v^log: one field per dimension. `new` rejects `Exact` with a non-zero -/// error bound. +/// p_v^log. pub struct LogicalProperties { - representation: Representation, // which sketch, and θ - error_bound: ErrorBound, // (ε_v, δ_v), precomputed per θ - window_layout: Option, // ω_v -} -pub enum Representation { - Exact, - Summary { family: SummaryFamily, parameters: SummaryParameters }, + family: Family, // which sketch, or exact + parameter_configuration: Option, // θ; None for exact + window_layout: Option, // ω_v } -pub struct ErrorBound { metric: AccuracyMetric, epsilon: f64, delta: f64 } +pub enum Family { Exact(ExactKind), Sketch(SketchAlgorithm) } pub enum WindowLayout { Tumbling { width_ms: u64, count: u32 } } // sliding variants later -/// p_v^phys: one field per dimension. `new` rejects the invalid combinations. +/// p_v^phys. `new` rejects the invalid combinations. pub struct PhysicalProperties { timing: ExecutionTiming, // τ lifetime: Lifetime, // ℓ @@ -227,59 +258,72 @@ pub enum Schedule { Periodic { interval_ms: u64, phase_ms: u64 }, TumblingWindowGrid { phase_ms: u64 }, } + +// Reuse the existing family-specific conversions in asap_accuracy. +// Parameter configuration → error bound (including metric and failure probability). +// `family` describes the exact value or sketch, including its parameters; +// `query` identifies the statistic read from it. +pub(crate) fn local_guarantee( + family: &FieldDataType, + query: &SketchStatistic, +) -> Option; + +// Error target → parameter configuration for the chosen family and intent. +pub(crate) fn size_params( + kind: SketchAlgorithm, + intent: &AggIntent, + eps: f64, + delta: f64, +) -> SketchParams; ``` +`new` checks that `parameter_configuration` belongs to `family`, as [`SketchKind::new`](../../../crates/types/src/ir/schema/sketch.rs) does today, and reuses the existing `SketchParams` type. + +`error_bound` is derived when needed, not stored in `LogicalProperties`. Reuse [`estimators::local_guarantee`](../../../crates/accuracy/src/estimators/mod.rs) for parameters → guarantee and [`estimators::size_params`](../../../crates/accuracy/src/estimators/mod.rs) for target → parameters. These existing crate-private functions dispatch to the family's accuracy and sizing implementations; the declarations above document those interfaces, not new implementations. [`sizing::default_size_params`](../../../crates/accuracy/src/sizing.rs) already delegates to `size_params`. If another crate needs direct access, expose the existing implementation through the accuracy crate rather than duplicate its formulas. + +Deriving a guarantee also needs the statistic being read: one family's parameters can give different guarantees for different statistics. Exact values yield an exact guarantee; an unsupported statistic or missing error model yields `None`. Target → parameters is a sizing operation with discrete values and family limits, not an exact mathematical inverse; validate the derived guarantee against the target rather than assume the target is achievable. + ### Property requirement definition +A property requirement is also a struct with one field per dimension (R1, R2, …), each its own enum. `None` accepts any value. + ```rust -/// r^log_{c,i}: one optional constraint per dimension; `None` accepts any value. -pub struct LogicalRequirement { +/// r_i: what input i's producer must satisfy. +pub struct PropertyRequirement { + pub accuracy: Option, pub family: Option, - pub window: Option, // the reader's range and step - pub error_cap: Option, // validity of row (K)'s linear bound + pub window: Option, // the reader's range and step + pub reader_timing: Option, + pub reads_from: Option, // Same or Later + pub reader_schedule: Option, // (I_c, φ_c) of a periodic reader } -/// r^phys_{c,i}: one field per dimension. `new` rejects (IngestionTime, Later). -pub struct PhysicalRequirement { - reader_timing: ExecutionTiming, - reads_from: Evaluation, // Same or Later - reader_schedule: Option, // (I_c, φ_c) of a periodic reader -} - -/// α^log and α^phys: one check per dimension, joined by "and". -pub fn satisfies(producer: &LogicalProperties, requirement: &LogicalRequirement) -> bool; -pub fn satisfies_physical(producer: &PhysicalProperties, requirement: &PhysicalRequirement) -> bool; - -/// The accuracy target stays at the query root; Stage 3 checks it (row (K)). -pub struct QueryRequirement { pub accuracy: AccuracyTarget } +/// The set of legal properties for an input: legal_properties(r) = f(r). +pub fn satisfies(producer: &Properties, requirement: &PropertyRequirement) -> bool; ``` -### Candidate data structure definition +The set of legal properties is a function of the requirement, $\text{legal}(r) = f(r)$. One requirement can be written at different levels. For example, the same accuracy requirement can be stated two ways: -```rust -/// v = (e_v, p_v^log): one producer of a memo group, from Stage 1. -pub struct CandidateForMemoGroup { - pub expression: CandidateExpression, // e_v - pub logical_properties: LogicalProperties, // p_v^log - /// r^log_{c,i}: one per input group of `expression`. - pub input_requirements: Vec, -} - -/// e_v = op_v(G_{v,1}, …, G_{v,n}): one operator with its arguments, whose -/// children are groups. No private nodes. -pub struct CandidateExpression { pub operator: Operator } +| Requirement | Mapped to properties | Producer candidates enumerated | +|---|---|---| +| Accuracy = 95% rank accuracy | every family with rank error, and every configuration of it that meets 95% rank accuracy | (KLL, `k = 100`), (KLL, `k = 200`), … | +| Accuracy = KLL's `k ≥ 100` | KLL with `k ≥ 100` | (KLL, `k = 100`), (KLL, `k = 200`), … | -/// One physical alternative of a logical candidate, from Stage 2. -pub struct PhysicalCandidate { - pub logical: (MemoGroupId, usize), // which logical candidate - pub properties: PhysicalProperties, // p_v^phys - pub input_requirements: Vec, // r^phys_{c,i} +```rust +pub enum AccuracyRequirement { + /// Stated on the result: a metric, bound and failure probability. The + /// legal properties are derived through `size_params` / `local_guarantee`. + Target(AccuracyTarget), + /// Stated on the properties directly, e.g. KLL with k ≥ 100. + Parameters { family: SketchAlgorithm, at_least: SketchParams }, } - -/// (u, c, i): producer u may serve input i of consumer c, when α^log and α^phys hold. -pub struct Edge { pub producer: CandidateId, pub consumer: CandidateId, pub input: usize } ``` +What goes into a property requirement affects only the efficiency of enumeration, not correctness: +- **Some requirements already hold by the query semantics.** The expression and its producer groups already fix the input data and its fields, so no requirement needs to restate them. +- **It depends on how rules generate the candidate space.** A rule that only generates legal candidates needs no requirement for that dimension; a rule that generates every value needs the requirement to filter them. +- **A requirement prunes; it does not decide the plan.** A tighter requirement removes candidates earlier. Each query's composed error is still checked at its root in Stage 3 (row (K)), so a requirement that is too loose costs enumeration time, not a wrong plan. + ### Extensibility **Add a value to a property dimension**, e.g. a new summary family or `Medium::Disk`: @@ -289,19 +333,19 @@ pub struct Edge { pub producer: CandidateId, pub consumer: CandidateId, pub inpu 4. Add any new invalid combinations to `new`. **Add a property dimension**, e.g. a compression codec or a partitioning: -1. Decide whether it is logical (it changes the result's values or error) or physical (it changes only when, where or how long the result is computed). Add a field to `LogicalProperties` or `PhysicalProperties`. -2. If consumers can constrain it, add an optional field to the matching requirement struct and one check to `satisfies` or `satisfies_physical`. +1. Decide whether it is logical (it changes the result's values or error) or physical (it changes only when, where or how long the result is computed). Add a field (a new enum) to `LogicalProperties` or `PhysicalProperties`. +2. If consumers can constrain it, add an optional field to `PropertyRequirement` and one check to `satisfies`. 3. Add its invalid combinations with the other dimensions to `new`. 4. Add a level to the decision tree, with its prune rule, and place it by the considerations in "Order of decisions". 5. Add its cost terms. **Add a requirement:** -1. If it constrains one edge (a consumer input), add an optional field to `LogicalRequirement` or `PhysicalRequirement`, set it where the consumer is created, and check it in `satisfies`. +1. If it constrains one edge (a consumer input), add an optional field to `PropertyRequirement`, set it where the consumer's group is created, and check it in `satisfies`. 2. If it constrains a whole query or the whole plan (accuracy, latency, budgets), add it to the query's or workload's requirements and a row to Stage 3. Do not split it into per-edge requirements: that fixes each input's share in advance. 3. If it comes from the deployment, prune the values it rules out during candidate generation. **Add a rule:** 1. A rule matches over groups and their candidates' expressions, never over a reader's private operators. -2. It adds candidates $(e, p^{\log})$ to an existing group, or adds a new group with its candidates. It never copies a reader: every existing consumer of the group sees the new candidate through edges. +2. It adds candidates to an existing group, or adds a new group with its expression, input requirements and candidates. It never copies a reader: every existing consumer of the group sees the new candidate through edges. 3. A candidate whose result has the same meaning as an existing group is added to that group, so hash-consing keeps sharing. -4. Each new candidate states its input requirements. +4. Each new group states its input requirements. From a039b0523342234ebc5ab0e3b72aebf50b247c2a Mon Sep 17 00:00:00 2001 From: zzylol <50204836+zzylol@users.noreply.github.com> Date: Thu, 5 Nov 2026 19:08:22 +0000 Subject: [PATCH 03/44] docs(concepts): align memo group, candidate, property and requirement definitions with the code and #1200 Co-Authored-By: Claude Opus 5.5 --- .../memo-groups-properties-requirements.md | 275 +++++++++++------- 1 file changed, 165 insertions(+), 110 deletions(-) diff --git a/docs/design_docs/concepts/memo-groups-properties-requirements.md b/docs/design_docs/concepts/memo-groups-properties-requirements.md index 37623e34c..6fb8cd85d 100644 --- a/docs/design_docs/concepts/memo-groups-properties-requirements.md +++ b/docs/design_docs/concepts/memo-groups-properties-requirements.md @@ -1,7 +1,7 @@ # Memo groups, properties and requirements This page defines the memo's main data structures: groups, candidates, properties and requirements. It also lists every property and requirement, and gives the decision tree Stage 1 and Stage 2 walk to enumerate candidates. The symbols follow the planner formulation: -- a group has an expression $e_G$ over its producer groups, and a candidate $v$ of it has properties $p_v$; +- a candidate is $v = (e_v, p_v)$; - its logical properties are $p^{\log}$ and its physical properties are $p^{\mathrm{phys}}$; - a consumer input's requirements are $r^{\log}_{c,i}$ and $r^{\mathrm{phys}}_{c,i}$; - whether a producer satisfies them is $\alpha^{\log}$ and $\alpha^{\mathrm{phys}}$. @@ -36,19 +36,19 @@ The ε values are illustrative. A property is a value a candidate has. Every property is one dimension of a product: a struct with one field per dimension, never a flat enum that lists the combinations. Invalid combinations are rejected by the struct's checked constructor. -| Dimension | Values | Logical or physical | Fixed by | -|---|---|---|---| -| Which family to use (operator) | `Exact` (exact accumulator kinds: Sum, Count, Min, Max, Increase, Rate, IRate), or `Summary { family }` with family in KLL, DDSketch, CMS, CountSketch, CMS with heap, CountSketch with heap, UnivMon, HLL, KMV, the Hydra variants | logical | enumerated | -| Parameter configuration $\theta$ | per family, from a predefined list: KLL `k`; DDSketch `alpha`; CMS / CountSketch `width × depth`; CMS / CountSketch with heap `width × depth × heap_size`; UnivMon `heap_size × rows × cols × layers`; HLL `precision`; KMV `k` | logical | enumerated | -| Error bound $(\varepsilon_v, \delta_v)$ | computed on demand from $\theta$ and the statistic, in the family's accuracy metric (below); exact is $(0, 0)$ | logical | derived from $\theta$, not stored or chosen | -| Window configuration $\omega$ | none; or tumbling windows of width $w$ and count $n$, with $w$ from the gcd closure of the readers' ranges and steps. Later: sliding windows by merge or by subtraction | logical | enumerated | -| Timing $\tau$ | `IngestionTime`, `QueryTime` | physical | enumerated | -| Lifetime $\ell$ | `None`, `OneEvaluation(Medium)`, `AcrossEvaluations(Medium)`, with `Medium` in `Memory`, `Disk` | physical | enumerated | -| Schedule $\sigma$ | only when $\ell \ne$ None. `Periodic { interval, phase }` for `OneEvaluation`; `TumblingWindowGrid { phase }` for `AcrossEvaluations` | physical | enumerated from the readers' schedules | +| Dimension | Values | Logical or physical | Fixed by | Code | +|---|---|---|---|---| +| Which family to use (operator) | `Exact` (exact accumulator kinds: Sum, Count, Min, Max, Increase, Rate, IRate), or `Summary { family }` with family in KLL, DDSketch, CMS, CountSketch, CMS with heap, CountSketch with heap, UnivMon, HLL, KMV, the Hydra variants | logical | enumerated | `FieldDataType::{ExactAggregate, Sketch}` in the candidate's operator ([`schema/mod.rs`](../../../crates/types/src/ir/schema/mod.rs)); `SketchAlgorithm` | +| Parameter configuration $\theta$ | per family, from a predefined list: KLL `k`; DDSketch `alpha`; CMS / CountSketch `width × depth`; CMS / CountSketch with heap `width × depth × heap_size`; UnivMon `heap_size × rows × cols × layers`; HLL `precision`; KMV `k` | logical | enumerated | `SketchParams` inside `SketchKind` ([`sketch.rs`](../../../crates/types/src/ir/schema/sketch.rs)), sized by `size_params` per `accuracy_budget_parts` | +| Error bound $(\varepsilon_v, \delta_v)$ | computed on demand from $\theta$ and the statistic, in the family's accuracy metric (below); exact is $(0, 0)$ | logical | derived from $\theta$, not stored or chosen | `local_guarantee` → `AccuracyGuarantee` ([`estimators`](../../../crates/accuracy/src/estimators/mod.rs)) | +| Window configuration $\omega$ | none; or tumbling windows of width $w$ and count $n$, with $w$ from the gcd closure of the readers' ranges and steps. Later: sliding windows by merge or by subtraction | logical | enumerated | `SummaryTumblingWindowChain` arguments, added by Pass 2 ([`window_candidates.rs`](../../../crates/logical-asap-optimization/src/pass2/memo/window_candidates.rs)) | +| Timing $\tau$ | `IngestionTime`, `QueryTime` | physical | enumerated | `ExecutionTiming` ([`execution_data_state.rs`](../../../crates/types/src/ir/properties/execution_data_state.rs)); `PhysicalCandidate::timing` | +| Lifetime $\ell$ | `None`, `OneEvaluation(Medium)`, `AcrossEvaluations(Medium)`, with `Medium` in `Memory`, `Disk` | physical | enumerated | `Materialization` on main; `Lifetime`, `Medium` in #1214 ([`physical/`](../../../crates/types/src/physical)) | +| Schedule $\sigma$ | only when $\ell \ne$ None. `Periodic { interval, phase }` for `OneEvaluation`; `TumblingWindowGrid { phase }` for `AcrossEvaluations` | physical | enumerated from the readers' schedules | not a property: schedule groups `z[m, G]` in the MILP ([`cost_terms.rs`](../../../crates/plan-selection/src/milp/cost_terms.rs)); a property in #1200 PR 2 | A candidate is materialized when $\ell \ne$ None. This is derived, not stored. -Invalid combinations, rejected by `new`: +Invalid combinations, rejected by `PhysicalProperties::new` (the list in #1200; #1214 implements the first, the others come with the schedule in #1200 PR 2): | Combination | Why | |---|---| @@ -56,6 +56,8 @@ Invalid combinations, rejected by `new`: | $\ell$ = None with a schedule | where an unmaterialized result runs is its reader's execution context, not a property | | $\ell \ne$ None without a schedule | a materialized result must say on which times or grid it is delivered | | OneEvaluation with a grid, AcrossEvaluations with a periodic schedule | each lifetime has one kind of schedule | +| a phase at or above the grid width or interval, or interval 0 | the phase is taken modulo the width or interval | +| IngestionTime with a grid phase ≠ 0 | ingestion-time grids are anchored at the epoch today: the plan carries no origin | Valid physical states: (QueryTime, None), (IngestionTime, None), (QueryTime, OneEvaluation(m)), (QueryTime, AcrossEvaluations(m)), (IngestionTime, AcrossEvaluations(m)), for each medium $m$. @@ -124,14 +126,14 @@ flowchart TD The levels, top to bottom: -| Level | Dimension | Stage | Values offered | Pruned when | -|---|---|---|---|---| -| 1 | family (operator) | Stage 1, Pass 1 and Pass 2 rules | each way to compute the group | the executor cannot build the family; no consumer's family requirement accepts it | -| 2 | $\theta$ | Stage 1, Pass 1 | the family's predefined list | $\varepsilon_v$ alone exceeds every reading query's target | -| 3 | window configuration $\omega$ | Stage 1, Pass 2 | tumbling widths from the gcd closure of readers' ranges and steps | no reader's range and step is a multiple of the width | -| 4 | timing $\tau$ | Stage 2 | QueryTime; IngestionTime | the source has no ingestion pipeline | -| 5 | lifetime $\ell$ | Stage 2 | None, OneEvaluation(m), AcrossEvaluations(m) | invalid with $\tau$; the deployment keeps no query-time results across evaluations | -| 6 | schedule $\sigma$ | Stage 2 | the gcd closure $\Sigma$ of the readers' schedules | only offered when $\ell \ne$ None | +| Level | Dimension | Stage | Values offered | Pruned when | Code | +|---|---|---|---|---|---| +| 1 | family (operator) | Stage 1, Pass 1 and Pass 2 rules | each way to compute the group | the executor cannot build the family; no consumer's family requirement accepts it | Pass 1 ([`per_computation_candidates.rs`](../../../crates/logical-asap-optimization/src/pass1/per_computation_candidates.rs)); Pass 2 rules ([`rules/`](../../../crates/logical-asap-optimization/src/pass2/memo/rules)) | +| 2 | $\theta$ | Stage 1, Pass 1 | the family's predefined list | $\varepsilon_v$ alone exceeds every reading query's target | Pass 1 sizing per `accuracy_budget_parts` ([`memo_population.rs`](../../../crates/logical-asap-optimization/src/pass1/memo_population.rs)); a predefined list in #1203 PR 2 | +| 3 | window configuration $\omega$ | Stage 1, Pass 2 | tumbling widths from the gcd closure of readers' ranges and steps | no reader's range and step is a multiple of the width | Pass 2 ([`window_candidates.rs`](../../../crates/logical-asap-optimization/src/pass2/memo/window_candidates.rs), [`state_groups.rs`](../../../crates/logical-asap-optimization/src/pass2/memo/state_groups.rs)) | +| 4 | timing $\tau$ | Stage 2 | QueryTime; IngestionTime | the source has no ingestion pipeline | Stage 2 `enumerate_candidate_configurations` ([`configurations.rs`](../../../crates/physical-asap-optimization/src/catalog/configurations.rs)) | +| 5 | lifetime $\ell$ | Stage 2 | None, OneEvaluation(m), AcrossEvaluations(m) | invalid with $\tau$; the deployment keeps no query-time results across evaluations | same function, with `DeploymentCapabilities::query_time_results_retained` | +| 6 | schedule $\sigma$ | Stage 2 | the gcd closure $\Sigma$ of the readers' schedules | only offered when $\ell \ne$ None | Stage 3 MILP today; Stage 2 in #1200 PR 2 | What the tree does not decide: - **Which producer each consumer input reads.** That is the edge choice $y_{u,c,i}$ in Stage 3. Two consumers of one group may read different leaves, e.g. A reads `k = 200` and B reads `k = 345`. @@ -179,129 +181,190 @@ A memo group holds three things: - **Eligibility:** for each input, the requirement a producer must meet. - **Candidates:** the ways to compute it. +In the code, these three parts are stored in two places: the group itself ([`MemoGroup`](../../../crates/types/src/stages/memo.rs)) holds the query semantics, and the memo ([`Memo`](../../../crates/types/src/stages/candidates.rs)) holds each group's candidates by group index. Eligibility is not stored yet. + ```rust -/// The memo: every group, and each query's root over groups. +// crates/types/src/stages/memo.rs +pub struct MemoGroup { + // Query semantics. + /// Its logical node as the frontends see it; its children are other + /// groups' nodes. + pub logical_node: Rc, + /// Its child groups (the producer groups G_i), in the node's child order. + pub child_groups: Vec, + // Derived, kept for traversal. + pub parents: Vec, + pub queries: Vec, +} + +// crates/types/src/stages/candidates.rs pub struct Memo { pub groups: Vec, pub query_roots: Vec>, -} - -pub struct MemoGroup { - // Query semantics. - /// The logical operator, from the existing IR. Its children are the - /// producer groups G_i it reads, in order (`expression.children()`), which - /// fix the input data and schema. - pub expression: Operator, - // Eligibility. - /// r_i: one requirement per producer group, by input index i. - pub input_requirements: Vec, - // Candidates. - /// The ways to compute this group; x_v in the MILP. - pub candidates: Vec, + // Candidates: one list per group, by group index; x_v in the MILP. + pub candidates: Vec>, } ``` -The expression is the existing IR's [`Operator`](../../../crates/types/src/ir/operator/node.rs), with `C = MemoGroupId`. It wraps [`NonASAPOp`](../../../crates/types/src/ir/operator/non_asap.rs) or [`ASAPOp`](../../../crates/types/src/ir/operator/asap.rs), and each variant carries its own arguments and its inputs. For example, `Scan` carries the source, predicates and schema, and `Filter` carries a predicate and its child. No new identity type is needed. +| Part | Code | Target | +|---|---|---| +| Expression | `MemoGroup::logical_node`, an [`OperatorNode`](../../../crates/types/src/ir/operator/node.rs) whose [`Operator`](../../../crates/types/src/ir/operator/node.rs) is a `NonASAPOp` or `ASAPOp` variant with its own arguments | unchanged: the existing IR, no new identity type | +| Producer groups $G_i$ | `MemoGroup::child_groups` | unchanged | +| Eligibility $r_{G,i}$ | not stored: each candidate's private node chain fixes its producer | one logical requirement (#1203) and one physical requirement (#1200) per consumer input; see "Property requirement definition" | +| Candidates | `Memo::candidates[g]` | unchanged place; each candidate restructured (#1203) | +| Consumers | `MemoGroup::parents` (derived) | unchanged: derived from the producer groups | -Hash-consing finds or inserts a group by its expression: the operator, its arguments and its input group ids. For example, the KLL state over one source and query window is one group for `k = 200` and `k = 345`; these are two candidates of that group. A different source, filter or query window gives a different group. Properties are not part of the expression: the family, sketch parameters, window layout, timing, lifetime and schedule belong to candidates. A requested query window belongs to the expression; the internal window layout used to compute it does not. +Hash-consing finds or inserts a group by its logical node: the operator, its arguments and its child groups. Stage 0 does this today (`hash_consed` in [`stage0.rs`](../../../crates/logical-asap-optimization/src/stage0.rs)); #1203 makes Pass 1 look up existing groups the same way. A different source, filter or query window gives a different group. A requested query window belongs to the logical node; the internal window layout used to compute it does not. -`MemoGroupId` is the handle of an interned group. Looking up an existing expression returns that group's id instead of adding a second group. +The group's logical node is the frontend's operator, e.g. `quantile_over_time(0.95, latency[1m])`. It names no family and no sketch parameters: those are chosen by the candidates. A candidate's own operator (below) is an `ASAPOp` such as `SummaryAgg`, whose `FieldDataType::Sketch(SketchKind, …)` carries the family and $\theta$. Today the KLL state is not its own group: it is a private node inside each reader's candidate. #1203 makes every intermediate result its own group, so the KLL state over `latency[1m]` becomes a group, and `k = 200` and `k = 345` become two candidates of it. Its identity drops the sketch parameters, $\delta$ and the tumbling-window layout (#1203, decision 1). ### Candidate data structure definition +A candidate is one way to compute its group: an operator over producer groups, plus its properties. + ```rust -/// v: one way to compute its group. The group's expression says what is -/// computed; the candidate's properties say how. -pub struct Candidate { - /// One value per property dimension. - pub properties: Properties, +// crates/types/src/stages/candidates.rs (code today) +pub struct CandidateForMemoGroup { + pub fused_groups: Vec, + /// The implementation's operator, and its private nodes down to + /// `input_groups`. Family and θ are inside it (FieldDataType). + pub root_node: Rc, + pub input_groups: Vec, + pub origin: OriginOfCandidateForMemoGroup, + pub accuracy_budget_parts: usize, } ``` -Stage 1 decides the logical dimensions of each candidate, and Stage 2 expands each one into its physical alternatives: - ```rust -pub struct Properties { - pub logical: LogicalProperties, // Stage 1 - pub physical: PhysicalProperties, // Stage 2 +// Target (#1203): e_v and p_v^log as separate fields, one operator, no private nodes. +pub struct CandidateForMemoGroup { + pub expression: CandidateExpression, // e_v + pub logical_properties: LogicalProperties, // p_v^log + pub input_requirements: Vec, // r^log_{c,i}, one per input group + pub origin: OriginOfCandidateForMemoGroup, + pub fused_groups: Vec, } +pub struct CandidateExpression { pub operator: Operator } +``` -/// (u, c, i): producer u may serve input i of consumer c when u's properties -/// satisfy c's group's input_requirements[i]. -pub struct Edge { pub producer: CandidateId, pub consumer: CandidateId, pub input: usize } +Stage 2 expands each operator node of a logical candidate into physical candidates, one per valid physical property value: + +```rust +// crates/types/src/stages/physical/memo_candidates.rs, after #1200 PR 1 (#1214) +pub struct PhysicalCandidate { + pub operator_node: OperatorNodeOfCandidateForMemoGroupId, + pub properties: PhysicalProperties, // p_v^phys; was timing + materialization + /// Per input, the candidates that may produce it (filtered by α^phys). + pub producer_candidates_per_input: Vec, + pub may_produce_query_output: bool, + pub may_keep_across_evaluations_per_sole_reader_query: Vec, +} ``` +| Formulation | Code today | Target | +|---|---|---| +| $e_v$ | `root_node` + `input_groups` (may hold private nodes) | `expression: Operator` (#1203) | +| $p^{\log}_v$ | inside `root_node`'s `FieldDataType`; sized per `accuracy_budget_parts` | `logical_properties` (#1203) | +| $p^{\mathrm{phys}}_v$ | `PhysicalCandidate::{timing, materialization}` on main | `PhysicalCandidate::properties` (#1200 PR 1, #1214) | +| edges $(u, c, i)$ | `producer_candidates_per_input` (sets, no edge ids) | `LogicalEdge`, `PhysicalEdge` (#1201) | + ### Property definition -A property is a struct with one field per dimension (P1, P2, …), each its own enum: +Each property is a struct with one field per dimension, each its own enum. + +**Logical properties** (#1203 target; today inside `root_node`): ```rust -/// p_v^log. pub struct LogicalProperties { - family: Family, // which sketch, or exact - parameter_configuration: Option, // θ; None for exact - window_layout: Option, // ω_v + representation: Representation, // family and θ + window_layout: Option, // ω_v +} +/// Reuses the code's checked types: `SketchKind::new` already rejects +/// parameters of another algorithm. +pub enum Representation { + Exact(ExactKind), // crates/types/src/ir/schema/sketch.rs + Summary(SketchKind), // SketchKind { algorithm, params } } -pub enum Family { Exact(ExactKind), Sketch(SketchAlgorithm) } -pub enum WindowLayout { Tumbling { width_ms: u64, count: u32 } } // sliding variants later +pub enum WindowLayout { Tumbling { width_ms: u64, count: u32 } } // sliding variants: #1206 +``` + +Today these values are the `FieldDataType` of the candidate's summary operator: `ExactAggregate(ExactKind, ExactParams)` or `Sketch(SketchKind, GroupingStrategy)` ([`schema/mod.rs`](../../../crates/types/src/ir/schema/mod.rs)). The tumbling-window layout is the `SummaryTumblingWindowChain` operator's arguments. + +`error_bound` is derived when needed, not stored in `LogicalProperties`. Reuse [`estimators::local_guarantee`](../../../crates/accuracy/src/estimators/mod.rs) for parameters → guarantee and [`estimators::size_params`](../../../crates/accuracy/src/estimators/mod.rs) for target → parameters. These existing crate-private functions dispatch to the family's accuracy and sizing implementations. [`sizing::default_size_params`](../../../crates/accuracy/src/sizing.rs) already delegates to `size_params`. If another crate needs direct access, expose the existing implementation through the accuracy crate rather than duplicate its formulas. (#1203 currently proposes storing `error_bound`; this page derives it instead.) + +```rust +// crates/accuracy/src/estimators/mod.rs (code today) +pub(crate) fn local_guarantee(family: &FieldDataType, query: &SketchStatistic) -> Option; +pub(crate) fn size_params(kind: SketchAlgorithm, intent: &AggIntent, eps: f64, delta: f64) -> SketchParams; +``` + +Deriving a guarantee also needs the statistic being read: one family's parameters can give different guarantees for different statistics. Exact values yield an exact guarantee; an unsupported statistic or missing error model yields `None`. Target → parameters is a sizing operation with discrete values and family limits, not an exact mathematical inverse; validate the derived guarantee against the target rather than assume the target is achievable. -/// p_v^phys. `new` rejects the invalid combinations. +**Physical properties** (#1200). PR 1 (#1214) adds timing and lifetime; PR 2 adds the schedule. + +```rust +// crates/types/src/ir/properties/execution_data_state.rs (code today) +pub enum ExecutionTiming { IngestionTime, QueryTime } + +// crates/types/src/physical/properties.rs (#1214) +pub enum Medium { Memory, Disk } +pub enum Lifetime { None, OneEvaluation(Medium), AcrossEvaluations(Medium) } pub struct PhysicalProperties { timing: ExecutionTiming, // τ lifetime: Lifetime, // ℓ - schedule: Option, // σ, only when ℓ ≠ None + // schedule: Option, // σ, #1200 PR 2 } -pub enum ExecutionTiming { IngestionTime, QueryTime } -pub enum Lifetime { None, OneEvaluation(Medium), AcrossEvaluations(Medium) } -pub enum Medium { Memory, Disk } -pub enum Schedule { - Periodic { interval_ms: u64, phase_ms: u64 }, - TumblingWindowGrid { phase_ms: u64 }, +impl PhysicalProperties { + pub fn new(timing: ExecutionTiming, lifetime: Lifetime) -> Result; + pub fn timing(self) -> ExecutionTiming; + pub fn lifetime(self) -> Lifetime; + pub fn is_materialized(self) -> bool; // ℓ ≠ None } -// Reuse the existing family-specific conversions in asap_accuracy. -// Parameter configuration → error bound (including metric and failure probability). -// `family` describes the exact value or sketch, including its parameters; -// `query` identifies the statistic read from it. -pub(crate) fn local_guarantee( - family: &FieldDataType, - query: &SketchStatistic, -) -> Option; - -// Error target → parameter configuration for the chosen family and intent. -pub(crate) fn size_params( - kind: SketchAlgorithm, - intent: &AggIntent, - eps: f64, - delta: f64, -) -> SketchParams; +// #1200 PR 2 +pub enum Schedule { + Periodic { interval_ms: u64, phase_ms: u64 }, // OneEvaluation + TumblingWindowGrid { phase_ms: u64 }, // AcrossEvaluations +} ``` -`new` checks that `parameter_configuration` belongs to `family`, as [`SketchKind::new`](../../../crates/types/src/ir/schema/sketch.rs) does today, and reuses the existing `SketchParams` type. - -`error_bound` is derived when needed, not stored in `LogicalProperties`. Reuse [`estimators::local_guarantee`](../../../crates/accuracy/src/estimators/mod.rs) for parameters → guarantee and [`estimators::size_params`](../../../crates/accuracy/src/estimators/mod.rs) for target → parameters. These existing crate-private functions dispatch to the family's accuracy and sizing implementations; the declarations above document those interfaces, not new implementations. [`sizing::default_size_params`](../../../crates/accuracy/src/sizing.rs) already delegates to `size_params`. If another crate needs direct access, expose the existing implementation through the accuracy crate rather than duplicate its formulas. - -Deriving a guarantee also needs the statistic being read: one family's parameters can give different guarantees for different statistics. Exact values yield an exact guarantee; an unsupported statistic or missing error model yields `None`. Target → parameters is a sizing operation with discrete values and family limits, not an exact mathematical inverse; validate the derived guarantee against the target rather than assume the target is achievable. +On main, timing and lifetime are still folded into one enum, [`Materialization`](../../../crates/types/src/physical/materialization.rs) (`NotMaterialized`, `MatAtQuery`, `MatAtQueryKeptAcrossEvaluations`, `MatAtIngestion`), next to `PhysicalCandidate::timing`. #1214 deletes it. The schedule is chosen in the MILP today (schedule groups `z[m, G]`, [`cost_terms.rs`](../../../crates/plan-selection/src/milp/cost_terms.rs)); #1200 PR 2 makes it a property. ### Property requirement definition -A property requirement is also a struct with one field per dimension (R1, R2, …), each its own enum. `None` accepts any value. +A requirement is stated per consumer input $(c, i)$, in two parts that follow the two property parts. Each is a struct with one field per dimension. ```rust -/// r_i: what input i's producer must satisfy. -pub struct PropertyRequirement { - pub accuracy: Option, +/// r^log_{c,i} (#1203): one optional constraint per dimension; None accepts any value. +pub struct LogicalRequirement { pub family: Option, - pub window: Option, // the reader's range and step - pub reader_timing: Option, - pub reads_from: Option, // Same or Later - pub reader_schedule: Option, // (I_c, φ_c) of a periodic reader + pub window: Option, // the reader's range and step + pub error_cap: Option, // validity of row (K)'s linear bound } - -/// The set of legal properties for an input: legal_properties(r) = f(r). -pub fn satisfies(producer: &Properties, requirement: &PropertyRequirement) -> bool; +/// α^log +pub fn satisfies(producer: &LogicalProperties, requirement: &LogicalRequirement) -> bool; + +/// r^phys_{c,i} (#1200): when the consumer reads the input, and on which times. +/// `new` rejects (IngestionTime, Later): ingestion has no evaluations. +pub struct PhysicalRequirement { + reader_timing: ExecutionTiming, + reads_from: Evaluation, // which evaluation's output the reader reads + reader_schedule: Option, // (I_c, φ_c); None for a non-periodic reader +} +pub enum Evaluation { Same, Later } +/// α^phys +pub fn satisfies_physical(producer: PhysicalProperties, requirement: PhysicalRequirement) -> bool; ``` +| Requirement | Code today | Target | +|---|---|---| +| $r^{\log}_{c,i}$ | not represented | `LogicalRequirement` (#1203) | +| $\alpha^{\log}$ | not computed | `satisfies` (#1203) | +| $r^{\mathrm{phys}}_{c,i}$ | the consumer's timing | `PhysicalRequirement` (#1200 PR 2) | +| $\alpha^{\mathrm{phys}}$ | [`compatible_timing`](../../../crates/physical-asap-optimization/src/catalog/configurations.rs) on main; `PhysicalProperties::readable_at(reader_timing)` in #1214 | `satisfies_physical` (#1200 PR 2) | +| accuracy | [`AccuracyTarget`](../../../crates/types/src/ir/properties/accuracy.rs) at the query root, split into equal shares by `accuracy_budget_parts` | `AccuracyTarget` at the query root, checked by row (K) (#1204) | +| deployment capability | [`DeploymentCapabilities`](../../../crates/logical-asap-optimization/src/deployment_capabilities.rs): `supported_summaries`, `raw_data_retained`, `query_time_results_retained` | unchanged; prunes during generation | + The set of legal properties is a function of the requirement, $\text{legal}(r) = f(r)$. One requirement can be written at different levels. For example, the same accuracy requirement can be stated two ways: | Requirement | Mapped to properties | Producer candidates enumerated | @@ -309,15 +372,7 @@ The set of legal properties is a function of the requirement, $\text{legal}(r) = | Accuracy = 95% rank accuracy | every family with rank error, and every configuration of it that meets 95% rank accuracy | (KLL, `k = 100`), (KLL, `k = 200`), … | | Accuracy = KLL's `k ≥ 100` | KLL with `k ≥ 100` | (KLL, `k = 100`), (KLL, `k = 200`), … | -```rust -pub enum AccuracyRequirement { - /// Stated on the result: a metric, bound and failure probability. The - /// legal properties are derived through `size_params` / `local_guarantee`. - Target(AccuracyTarget), - /// Stated on the properties directly, e.g. KLL with k ≥ 100. - Parameters { family: SketchAlgorithm, at_least: SketchParams }, -} -``` +In the code, the first form is the query's `AccuracyTarget`; Pass 1 maps it to parameters with `size_params`. The second form is what a sized candidate's `SketchParams` fixes today. #1203 keeps the first form at the query root and adds only `error_cap` per edge. What goes into a property requirement affects only the efficiency of enumeration, not correctness: - **Some requirements already hold by the query semantics.** The expression and its producer groups already fix the input data and its fields, so no requirement needs to restate them. @@ -334,18 +389,18 @@ What goes into a property requirement affects only the efficiency of enumeration **Add a property dimension**, e.g. a compression codec or a partitioning: 1. Decide whether it is logical (it changes the result's values or error) or physical (it changes only when, where or how long the result is computed). Add a field (a new enum) to `LogicalProperties` or `PhysicalProperties`. -2. If consumers can constrain it, add an optional field to `PropertyRequirement` and one check to `satisfies`. +2. If consumers can constrain it, add an optional field to `LogicalRequirement` or `PhysicalRequirement` and one check to `satisfies` or `satisfies_physical`. 3. Add its invalid combinations with the other dimensions to `new`. 4. Add a level to the decision tree, with its prune rule, and place it by the considerations in "Order of decisions". 5. Add its cost terms. **Add a requirement:** -1. If it constrains one edge (a consumer input), add an optional field to `PropertyRequirement`, set it where the consumer's group is created, and check it in `satisfies`. +1. If it constrains one edge (a consumer input), add a field to `LogicalRequirement` or `PhysicalRequirement`, set it where the consumer is created, and check it in `satisfies` or `satisfies_physical`. 2. If it constrains a whole query or the whole plan (accuracy, latency, budgets), add it to the query's or workload's requirements and a row to Stage 3. Do not split it into per-edge requirements: that fixes each input's share in advance. 3. If it comes from the deployment, prune the values it rules out during candidate generation. **Add a rule:** 1. A rule matches over groups and their candidates' expressions, never over a reader's private operators. -2. It adds candidates to an existing group, or adds a new group with its expression, input requirements and candidates. It never copies a reader: every existing consumer of the group sees the new candidate through edges. +2. It adds candidates to an existing group, or adds a new group with its candidates. It never copies a reader: every existing consumer of the group sees the new candidate through edges. 3. A candidate whose result has the same meaning as an existing group is added to that group, so hash-consing keeps sharing. -4. Each new group states its input requirements. +4. Each new candidate states its input requirements. From a2e24d50485f70ee48b39becc0f53cda02186ee4 Mon Sep 17 00:00:00 2001 From: zzylol <50204836+zzylol@users.noreply.github.com> Date: Thu, 5 Nov 2026 19:11:14 +0000 Subject: [PATCH 04/44] docs(concepts): data structures and interfaces per stage, from the #1207 formulation Co-Authored-By: Claude Opus 5.5 --- .../memo-groups-properties-requirements.md | 229 ++++++++++++++++++ 1 file changed, 229 insertions(+) diff --git a/docs/design_docs/concepts/memo-groups-properties-requirements.md b/docs/design_docs/concepts/memo-groups-properties-requirements.md index 6fb8cd85d..120fc501c 100644 --- a/docs/design_docs/concepts/memo-groups-properties-requirements.md +++ b/docs/design_docs/concepts/memo-groups-properties-requirements.md @@ -404,3 +404,232 @@ What goes into a property requirement affects only the efficiency of enumeration 2. It adds candidates to an existing group, or adds a new group with its candidates. It never copies a reader: every existing consumer of the group sees the new candidate through edges. 3. A candidate whose result has the same meaning as an existing group is added to that group, so hash-consing keeps sharing. 4. Each new candidate states its input requirements. + +## 4. Data structures and interfaces per stage + +This section gives the target interface of each stage after the formulation refactor (#1207). Each stage's boundary type carries the notation that stage defines, and no stage reads a notation a later stage defines: logical stages carry no physical properties. + +| Stage | Input | Output (boundary type) | Notation it defines | Sub-issue | +|---|---|---|---|---| +| Stage 0: frontends, hash-consing | queries | `DagOfLogicalMemoGroups` | $G$, query roots | — | +| Stage 1, Pass 1 | `DagOfLogicalMemoGroups` | `Memo` | $v = (e_v, p^{\log}_v)$, $r^{\log}_{c,i}$, $\theta$ list | #1203 | +| Stage 1, Pass 2 | `Memo` | `LogicalASAPMemoCandidates` | rule-added candidates; logical edges with $\alpha^{\log}$ | #1203, #1201 | +| Stage 2 | `LogicalASAPMemoCandidates` | `PhysicalASAPMemoCandidates` | $p^{\mathrm{phys}}_v = (\tau, \ell, \sigma)$, $r^{\mathrm{phys}}_{c,i}$, physical edges $\mathcal E$ with $\alpha^{\mathrm{phys}}$ | #1200, #1201 | +| Stage 3 | `PhysicalASAPMemoCandidates` | `DagOfSelectedCandidates` | contexts $\kappa$; cost $w_v$ (`asap-cost`); $x$, $y$, $z$, $t$, $h$; rows (A)–(K), (R), (L), (B) | #1204 | +| Realizer, export | `DagOfSelectedCandidates` | `PhysicalASAPDAG` | — | #1200 (export fields) | + +### Stage 0: frontends and hash-consing + +No change. One group per distinct logical sub-expression; the queries' roots point at groups. + +```rust +// crates/types/src/stages/memo.rs +pub struct MemoGroupId(pub u32); + +/// G: one logical result or one state family over specified input data. +pub struct MemoGroup { + pub logical_node: Rc, // expression; no physical fields (#1200) + pub child_groups: Vec, // producer groups G_i + pub parents: Vec, // derived: consumers' groups + pub queries: Vec, // derived: queries that reach it +} + +pub struct DagOfLogicalMemoGroups { + pub groups: Vec, + pub query_roots: Vec>, +} +``` + +### Stage 1, Pass 1: candidates of each group + +Pass 1 adds every way to compute each group as candidates. Every intermediate result is its own group (no private nodes), and a summary family's sizes are candidates of one state group. + +```rust +// crates/types/src/stages/candidates.rs +pub struct CandidateForMemoGroupId { pub group: MemoGroupId, pub index: usize } + +pub struct Memo { + pub groups: Vec, + pub query_roots: Vec>, + pub candidates: Vec>, // by group +} +impl Memo { + /// Hash-consing: the group with this expression, added if new. + pub fn find_or_add_group(&mut self, expression: Operator) -> MemoGroupId; + pub fn add_candidate(&mut self, group: MemoGroupId, candidate: CandidateForMemoGroup) -> CandidateForMemoGroupId; + pub fn candidate(&self, id: CandidateForMemoGroupId) -> &CandidateForMemoGroup; +} + +/// v = (e_v, p_v^log) +pub struct CandidateForMemoGroup { + pub expression: CandidateExpression, // e_v + pub logical_properties: LogicalProperties, // p_v^log + pub input_requirements: Vec, // r^log_{c,i}, one per input group + pub fused_groups: Vec, +} +/// e_v = op_v(G_{v,1}, …, G_{v,n}): one operator, its children are groups. +pub struct CandidateExpression { pub operator: Operator } + +pub struct LogicalProperties { + representation: Representation, // family and θ, from the predefined list + window_layout: Option, // ω_v +} +impl LogicalProperties { + pub fn new(representation: Representation, window_layout: Option) -> Result; + /// (ε_v, δ_v) for `statistic`, derived through `local_guarantee`. + pub fn error_bound(&self, statistic: &SketchStatistic) -> Option; +} +pub enum Representation { Exact(ExactKind), Summary(SketchKind) } +pub enum WindowLayout { Tumbling { width_ms: u64, count: u32 } } + +pub struct LogicalRequirement { + pub family: Option, + pub window: Option, // reader's range and step: a width must divide both + pub error_cap: Option, // validity of row (K)'s linear bound +} +/// α^log_{u,c,i} +pub fn satisfies(producer: &LogicalProperties, requirement: &LogicalRequirement) -> bool; +``` + +Deleted: `root_node` with private nodes, `input_groups` (the expression's children), `accuracy_budget_parts`, the `SharedBuild` and `SummaryResult` origins. + +### Stage 1, Pass 2: rules and logical edges + +Rules add candidates to existing groups, or add groups. A rule never copies a reader. After the last rule, Pass 2 builds the logical edges once. + +```rust +// crates/logical-asap-optimization/src/pass2 +/// A rule: reads the memo, adds candidates or groups. +fn apply(memo: &mut Memo) -> Result<(), Pass2RuleError>; + +// crates/types/src/stages/logical_memo_candidates.rs +/// (u, c, i) over logical candidates: u is a candidate of G_{c,i}. +pub struct LogicalEdge { + pub producer: CandidateForMemoGroupId, + pub consumer: CandidateForMemoGroupId, + pub input: usize, + pub satisfies_logical_requirement: bool, // α^log +} +pub struct LogicalASAPMemoCandidates { + pub memo: Memo, + pub edges: Vec, // grouped by (consumer, input) +} +``` + +### Stage 2: physical candidates and edges + +Stage 2 expands each logical candidate into one physical candidate per valid $(\tau, \ell, \sigma)$, gives each input its physical requirement, and turns each logical edge into the physical edges between the two candidates' physical alternatives. + +```rust +// crates/types/src/physical/properties.rs +pub enum ExecutionTiming { IngestionTime, QueryTime } +pub enum Medium { Memory, Disk } +pub enum Lifetime { None, OneEvaluation(Medium), AcrossEvaluations(Medium) } +pub enum Schedule { + Periodic { interval_ms: u64, phase_ms: u64 }, // OneEvaluation + TumblingWindowGrid { phase_ms: u64 }, // AcrossEvaluations +} +/// p_v^phys = (τ, ℓ, σ); `new` rejects the invalid combinations (Section 1). +pub struct PhysicalProperties { timing: ExecutionTiming, lifetime: Lifetime, schedule: Option } +impl PhysicalProperties { + pub fn new(timing: ExecutionTiming, lifetime: Lifetime, schedule: Option) -> Result; + pub fn timing(&self) -> ExecutionTiming; + pub fn lifetime(&self) -> Lifetime; + pub fn schedule(&self) -> Option<&Schedule>; + pub fn is_materialized(&self) -> bool; // ℓ ≠ None +} + +/// r^phys_{c,i}; `new` rejects (IngestionTime, Later). +pub struct PhysicalRequirement { + reader_timing: ExecutionTiming, + reads_from: Evaluation, + reader_schedule: Option, // (I_c, φ_c) +} +pub enum Evaluation { Same, Later } +/// α^phys_{u,c,i} +pub fn satisfies_physical(producer: &PhysicalProperties, requirement: &PhysicalRequirement) -> bool; + +// crates/types/src/stages/physical +pub struct PhysicalCandidateId(pub usize); +pub struct PhysicalEdgeId(pub usize); + +/// One physical alternative of a logical candidate. +pub struct PhysicalCandidate { + pub logical: CandidateForMemoGroupId, // e_v and p_v^log + pub properties: PhysicalProperties, // p_v^phys + pub input_requirements: Vec, // r^phys_{c,i} +} +/// (u, c, i) over physical candidates. +pub struct PhysicalEdge { + pub producer: PhysicalCandidateId, + pub consumer: PhysicalCandidateId, + pub input: usize, + pub satisfies_logical_requirement: bool, // α^log + pub satisfies_physical_requirement: bool, // α^phys +} +pub struct PhysicalASAPMemoCandidates { + pub logical: LogicalASAPMemoCandidates, + pub candidates: Vec, // C + pub query_roots: Vec>, + pub pruned_candidates: Vec>, + // edges stored per (consumer, input); see `edges()` +} +impl PhysicalASAPMemoCandidates { + pub fn edges(&self) -> impl Iterator; // E + pub fn admissible_edges_into(&self, consumer: PhysicalCandidateId, input: usize) -> &[PhysicalEdgeId]; // P_{c,i} + pub fn admissible_edges_out_of(&self, producer: PhysicalCandidateId) -> &[PhysicalEdgeId]; +} +``` + +Deleted: `Materialization`, `PhysicalCandidate::{timing, materialization, operator_node}`, `producer_candidates_per_input` as the interface, `compatible_timing`, `ReaderConstraint::{IncompatibleOutputs, AllowedOutputs}` (now $\alpha^{\mathrm{phys}}$ through $\sigma$), `may_keep_across_evaluations_per_sole_reader_query`. + +### Stage 3: the MILP + +Stage 3 reads the physical candidates and edges and selects the DAG. Variables and rows are named after the formulation. + +```rust +// crates/plan-selection/src/milp/model.rs +/// κ: where an ℓ = None candidate runs. Only those candidates are indexed by it. +/// `ContextId` indexes the workload's contexts; the id types inside are illustrative. +pub struct ContextId(pub usize); +pub enum Context { + QueryEvaluation(QueryId), // κ_q + IngestionPipeline(RefreshWindowId), // I_r + Materialized(PhysicalCandidateId), // κ_m: inside a materialized m's computation +} + +struct SelectionVariables { + x_materialized: Vec, // x_v, ℓ ≠ None + x_in_context: HashMap<(ContextId, PhysicalCandidateId), Variable>, // x_{κ,v}, ℓ = None + y: HashMap<(Option, PhysicalEdgeId), Variable>, // y_{u,c,i} on admissible edges + z: Vec, // z_{q,v} + t: Vec, // t_v, cyclic components only + h: Vec, // h_v, ℓ ≠ None + latency: LatencyVariables, // (L) +} +fn input_satisfaction_rows(..); // (A) Σ_u y_{u,c,i} = x_c +fn producer_selection_rows(..); // (B) y_{u,c,i} ≤ x_u +fn query_coverage_rows(..); // (C), (D) +fn acyclicity_rows(..); // (E) +fn utilization_rows(..); // (F) +fn reader_constraint_rows(..); // (J) +fn accuracy_rows(..); // (K) Σ λ_{q,u} ε_u a_{q,u} ≤ ε_q, Σ δ_u a_{q,u} ≤ δ_q +fn retention_rows(..); // (R) h_v ≥ need_{c,i} · y_{v,c,i} +fn latency_rows(..); // (L) +fn resource_budget_rows(..); // (B) budgets +// Objective: min Σ w_v x_v + Σ ρ_κ w_v x_{κ,v} + Σ π_v h_v + +// crates/types/src/stages/physical/graph.rs (unchanged) +pub struct SelectedEdge { pub producer: SelectedNodeId, pub consumer: SelectedNodeId, pub input: usize } +pub struct DagOfSelectedCandidates { + pub candidate_of_node: Vec, + pub edges: Vec, // decoded from y = 1 + pub node_of_output_reference: Vec, +} +``` + +(G) $y \le \alpha^{\log}$ and (H) $y \le \alpha^{\mathrm{phys}}$ are not rows: $y$ exists only on admissible edges. Deleted: `ResultSlot`, `slot_of`, the `e` binaries, schedule groups `z[m, G]`, `N[q]`, `A[q, parts]`, and `p[ctx, c]` for materialized candidates. + +### Realizer and export + +The realizer builds the plan from `DagOfSelectedCandidates`; each node's physical properties come from its `PhysicalCandidate`. The export writes timing and lifetime (and later the schedule) per node of `PhysicalASAPDAG`, changed together with the backend. From cd8117761ce54b119ad0b22f60a893d3d7ac4189 Mon Sep 17 00:00:00 2001 From: zzylol <50204836+zzylol@users.noreply.github.com> Date: Thu, 5 Nov 2026 19:20:20 +0000 Subject: [PATCH 05/44] docs(concepts): keep the per-stage interfaces as section 3, drop the earlier modeling section Co-Authored-By: Claude Opus 5.5 --- .../memo-groups-properties-requirements.md | 250 +----------------- 1 file changed, 1 insertion(+), 249 deletions(-) diff --git a/docs/design_docs/concepts/memo-groups-properties-requirements.md b/docs/design_docs/concepts/memo-groups-properties-requirements.md index 120fc501c..7769b3b10 100644 --- a/docs/design_docs/concepts/memo-groups-properties-requirements.md +++ b/docs/design_docs/concepts/memo-groups-properties-requirements.md @@ -156,256 +156,8 @@ Alternative orders worth comparing: - **$\theta$ last among logical dimensions.** $\theta$ has the most values. Deciding $\omega$ before $\theta$ visits fewer nodes when many widths are pruned by readers' ranges. - **Requirement-driven (top-down) vs producer-driven (bottom-up).** Top-down enumeration starts from each consumer's requirement and offers only the values it accepts; bottom-up offers every value and lets $\alpha$ filter edges. Top-down visits fewer nodes but must union the requirements of all consumers of a group, so that sharing between consumers is not lost. -## 3. Modeling a memo group -A memo group stores its **producers**: all candidate ways to compute its result, before Stage 3 selects any. A consumer of $G$ is a candidate of another group whose expression reads $G$; the producer–consumer edges $(u, c, i)$ connect them. Here, `consumers(G)` is derived from those input references rather than stored in the group. - -> Side note: [CMU-DB's optd-original](https://github.com/cmu-db/optd-original) stores the list of children as input producers to a memo group; while storing consumers vs storing producers in the memo group structure to form the DAG are equivalent in terms of correctness and efficiency. - -Candidate enumeration efficiency depends on: - -- **Property and requirement fields:** which dimensions are represented, their legal values, which values are derived, and which constraints permit pruning or sharing (Section 1). -- **Decision-tree order:** when each dimension is decided and its requirements are checked, so invalid combinations can be pruned before later dimensions multiply them (Section 2). - -Groups found by hash-consing expose the same producer alternatives to all their readers. Stage 3 decides whether readers use the same producer candidate or different ones. **Reusing a computation alternative in the memo is distinct from reusing its computed result at runtime. Selecting the same candidate for multiple readers does not by itself mean computing it only once.** - -- **Materialized candidate** ($\ell \ne$ None): its result is retained. Readers connected to the same materialized producer can reuse that result instead of computing it separately, subject to its lifetime and schedule. -- **Unmaterialized candidate** ($\ell =$ None): no independent shared result is retained. The computation runs as part of each reader's execution, so multiple readers selecting the same candidate may each execute it separately. - -For example, if p50 and p99 select the same KLL candidate, a shared materialized producer builds and retains one KLL state for both readouts within the applicable lifetime and schedule. With an unmaterialized candidate, the p50 execution can build its own KLL state and the p99 execution another, even though both builds use the same candidate configuration. - -### Memo group data structure - -A memo group holds three things: -- **Query semantics:** its expression, and the producer groups the expression reads. -- **Eligibility:** for each input, the requirement a producer must meet. -- **Candidates:** the ways to compute it. - -In the code, these three parts are stored in two places: the group itself ([`MemoGroup`](../../../crates/types/src/stages/memo.rs)) holds the query semantics, and the memo ([`Memo`](../../../crates/types/src/stages/candidates.rs)) holds each group's candidates by group index. Eligibility is not stored yet. - -```rust -// crates/types/src/stages/memo.rs -pub struct MemoGroup { - // Query semantics. - /// Its logical node as the frontends see it; its children are other - /// groups' nodes. - pub logical_node: Rc, - /// Its child groups (the producer groups G_i), in the node's child order. - pub child_groups: Vec, - // Derived, kept for traversal. - pub parents: Vec, - pub queries: Vec, -} - -// crates/types/src/stages/candidates.rs -pub struct Memo { - pub groups: Vec, - pub query_roots: Vec>, - // Candidates: one list per group, by group index; x_v in the MILP. - pub candidates: Vec>, -} -``` - -| Part | Code | Target | -|---|---|---| -| Expression | `MemoGroup::logical_node`, an [`OperatorNode`](../../../crates/types/src/ir/operator/node.rs) whose [`Operator`](../../../crates/types/src/ir/operator/node.rs) is a `NonASAPOp` or `ASAPOp` variant with its own arguments | unchanged: the existing IR, no new identity type | -| Producer groups $G_i$ | `MemoGroup::child_groups` | unchanged | -| Eligibility $r_{G,i}$ | not stored: each candidate's private node chain fixes its producer | one logical requirement (#1203) and one physical requirement (#1200) per consumer input; see "Property requirement definition" | -| Candidates | `Memo::candidates[g]` | unchanged place; each candidate restructured (#1203) | -| Consumers | `MemoGroup::parents` (derived) | unchanged: derived from the producer groups | - -Hash-consing finds or inserts a group by its logical node: the operator, its arguments and its child groups. Stage 0 does this today (`hash_consed` in [`stage0.rs`](../../../crates/logical-asap-optimization/src/stage0.rs)); #1203 makes Pass 1 look up existing groups the same way. A different source, filter or query window gives a different group. A requested query window belongs to the logical node; the internal window layout used to compute it does not. - -The group's logical node is the frontend's operator, e.g. `quantile_over_time(0.95, latency[1m])`. It names no family and no sketch parameters: those are chosen by the candidates. A candidate's own operator (below) is an `ASAPOp` such as `SummaryAgg`, whose `FieldDataType::Sketch(SketchKind, …)` carries the family and $\theta$. Today the KLL state is not its own group: it is a private node inside each reader's candidate. #1203 makes every intermediate result its own group, so the KLL state over `latency[1m]` becomes a group, and `k = 200` and `k = 345` become two candidates of it. Its identity drops the sketch parameters, $\delta$ and the tumbling-window layout (#1203, decision 1). - -### Candidate data structure definition - -A candidate is one way to compute its group: an operator over producer groups, plus its properties. - -```rust -// crates/types/src/stages/candidates.rs (code today) -pub struct CandidateForMemoGroup { - pub fused_groups: Vec, - /// The implementation's operator, and its private nodes down to - /// `input_groups`. Family and θ are inside it (FieldDataType). - pub root_node: Rc, - pub input_groups: Vec, - pub origin: OriginOfCandidateForMemoGroup, - pub accuracy_budget_parts: usize, -} -``` - -```rust -// Target (#1203): e_v and p_v^log as separate fields, one operator, no private nodes. -pub struct CandidateForMemoGroup { - pub expression: CandidateExpression, // e_v - pub logical_properties: LogicalProperties, // p_v^log - pub input_requirements: Vec, // r^log_{c,i}, one per input group - pub origin: OriginOfCandidateForMemoGroup, - pub fused_groups: Vec, -} -pub struct CandidateExpression { pub operator: Operator } -``` - -Stage 2 expands each operator node of a logical candidate into physical candidates, one per valid physical property value: - -```rust -// crates/types/src/stages/physical/memo_candidates.rs, after #1200 PR 1 (#1214) -pub struct PhysicalCandidate { - pub operator_node: OperatorNodeOfCandidateForMemoGroupId, - pub properties: PhysicalProperties, // p_v^phys; was timing + materialization - /// Per input, the candidates that may produce it (filtered by α^phys). - pub producer_candidates_per_input: Vec, - pub may_produce_query_output: bool, - pub may_keep_across_evaluations_per_sole_reader_query: Vec, -} -``` - -| Formulation | Code today | Target | -|---|---|---| -| $e_v$ | `root_node` + `input_groups` (may hold private nodes) | `expression: Operator` (#1203) | -| $p^{\log}_v$ | inside `root_node`'s `FieldDataType`; sized per `accuracy_budget_parts` | `logical_properties` (#1203) | -| $p^{\mathrm{phys}}_v$ | `PhysicalCandidate::{timing, materialization}` on main | `PhysicalCandidate::properties` (#1200 PR 1, #1214) | -| edges $(u, c, i)$ | `producer_candidates_per_input` (sets, no edge ids) | `LogicalEdge`, `PhysicalEdge` (#1201) | - -### Property definition - -Each property is a struct with one field per dimension, each its own enum. - -**Logical properties** (#1203 target; today inside `root_node`): - -```rust -pub struct LogicalProperties { - representation: Representation, // family and θ - window_layout: Option, // ω_v -} -/// Reuses the code's checked types: `SketchKind::new` already rejects -/// parameters of another algorithm. -pub enum Representation { - Exact(ExactKind), // crates/types/src/ir/schema/sketch.rs - Summary(SketchKind), // SketchKind { algorithm, params } -} -pub enum WindowLayout { Tumbling { width_ms: u64, count: u32 } } // sliding variants: #1206 -``` - -Today these values are the `FieldDataType` of the candidate's summary operator: `ExactAggregate(ExactKind, ExactParams)` or `Sketch(SketchKind, GroupingStrategy)` ([`schema/mod.rs`](../../../crates/types/src/ir/schema/mod.rs)). The tumbling-window layout is the `SummaryTumblingWindowChain` operator's arguments. - -`error_bound` is derived when needed, not stored in `LogicalProperties`. Reuse [`estimators::local_guarantee`](../../../crates/accuracy/src/estimators/mod.rs) for parameters → guarantee and [`estimators::size_params`](../../../crates/accuracy/src/estimators/mod.rs) for target → parameters. These existing crate-private functions dispatch to the family's accuracy and sizing implementations. [`sizing::default_size_params`](../../../crates/accuracy/src/sizing.rs) already delegates to `size_params`. If another crate needs direct access, expose the existing implementation through the accuracy crate rather than duplicate its formulas. (#1203 currently proposes storing `error_bound`; this page derives it instead.) - -```rust -// crates/accuracy/src/estimators/mod.rs (code today) -pub(crate) fn local_guarantee(family: &FieldDataType, query: &SketchStatistic) -> Option; -pub(crate) fn size_params(kind: SketchAlgorithm, intent: &AggIntent, eps: f64, delta: f64) -> SketchParams; -``` - -Deriving a guarantee also needs the statistic being read: one family's parameters can give different guarantees for different statistics. Exact values yield an exact guarantee; an unsupported statistic or missing error model yields `None`. Target → parameters is a sizing operation with discrete values and family limits, not an exact mathematical inverse; validate the derived guarantee against the target rather than assume the target is achievable. - -**Physical properties** (#1200). PR 1 (#1214) adds timing and lifetime; PR 2 adds the schedule. - -```rust -// crates/types/src/ir/properties/execution_data_state.rs (code today) -pub enum ExecutionTiming { IngestionTime, QueryTime } - -// crates/types/src/physical/properties.rs (#1214) -pub enum Medium { Memory, Disk } -pub enum Lifetime { None, OneEvaluation(Medium), AcrossEvaluations(Medium) } -pub struct PhysicalProperties { - timing: ExecutionTiming, // τ - lifetime: Lifetime, // ℓ - // schedule: Option, // σ, #1200 PR 2 -} -impl PhysicalProperties { - pub fn new(timing: ExecutionTiming, lifetime: Lifetime) -> Result; - pub fn timing(self) -> ExecutionTiming; - pub fn lifetime(self) -> Lifetime; - pub fn is_materialized(self) -> bool; // ℓ ≠ None -} - -// #1200 PR 2 -pub enum Schedule { - Periodic { interval_ms: u64, phase_ms: u64 }, // OneEvaluation - TumblingWindowGrid { phase_ms: u64 }, // AcrossEvaluations -} -``` - -On main, timing and lifetime are still folded into one enum, [`Materialization`](../../../crates/types/src/physical/materialization.rs) (`NotMaterialized`, `MatAtQuery`, `MatAtQueryKeptAcrossEvaluations`, `MatAtIngestion`), next to `PhysicalCandidate::timing`. #1214 deletes it. The schedule is chosen in the MILP today (schedule groups `z[m, G]`, [`cost_terms.rs`](../../../crates/plan-selection/src/milp/cost_terms.rs)); #1200 PR 2 makes it a property. - -### Property requirement definition - -A requirement is stated per consumer input $(c, i)$, in two parts that follow the two property parts. Each is a struct with one field per dimension. - -```rust -/// r^log_{c,i} (#1203): one optional constraint per dimension; None accepts any value. -pub struct LogicalRequirement { - pub family: Option, - pub window: Option, // the reader's range and step - pub error_cap: Option, // validity of row (K)'s linear bound -} -/// α^log -pub fn satisfies(producer: &LogicalProperties, requirement: &LogicalRequirement) -> bool; - -/// r^phys_{c,i} (#1200): when the consumer reads the input, and on which times. -/// `new` rejects (IngestionTime, Later): ingestion has no evaluations. -pub struct PhysicalRequirement { - reader_timing: ExecutionTiming, - reads_from: Evaluation, // which evaluation's output the reader reads - reader_schedule: Option, // (I_c, φ_c); None for a non-periodic reader -} -pub enum Evaluation { Same, Later } -/// α^phys -pub fn satisfies_physical(producer: PhysicalProperties, requirement: PhysicalRequirement) -> bool; -``` - -| Requirement | Code today | Target | -|---|---|---| -| $r^{\log}_{c,i}$ | not represented | `LogicalRequirement` (#1203) | -| $\alpha^{\log}$ | not computed | `satisfies` (#1203) | -| $r^{\mathrm{phys}}_{c,i}$ | the consumer's timing | `PhysicalRequirement` (#1200 PR 2) | -| $\alpha^{\mathrm{phys}}$ | [`compatible_timing`](../../../crates/physical-asap-optimization/src/catalog/configurations.rs) on main; `PhysicalProperties::readable_at(reader_timing)` in #1214 | `satisfies_physical` (#1200 PR 2) | -| accuracy | [`AccuracyTarget`](../../../crates/types/src/ir/properties/accuracy.rs) at the query root, split into equal shares by `accuracy_budget_parts` | `AccuracyTarget` at the query root, checked by row (K) (#1204) | -| deployment capability | [`DeploymentCapabilities`](../../../crates/logical-asap-optimization/src/deployment_capabilities.rs): `supported_summaries`, `raw_data_retained`, `query_time_results_retained` | unchanged; prunes during generation | - -The set of legal properties is a function of the requirement, $\text{legal}(r) = f(r)$. One requirement can be written at different levels. For example, the same accuracy requirement can be stated two ways: - -| Requirement | Mapped to properties | Producer candidates enumerated | -|---|---|---| -| Accuracy = 95% rank accuracy | every family with rank error, and every configuration of it that meets 95% rank accuracy | (KLL, `k = 100`), (KLL, `k = 200`), … | -| Accuracy = KLL's `k ≥ 100` | KLL with `k ≥ 100` | (KLL, `k = 100`), (KLL, `k = 200`), … | - -In the code, the first form is the query's `AccuracyTarget`; Pass 1 maps it to parameters with `size_params`. The second form is what a sized candidate's `SketchParams` fixes today. #1203 keeps the first form at the query root and adds only `error_cap` per edge. - -What goes into a property requirement affects only the efficiency of enumeration, not correctness: -- **Some requirements already hold by the query semantics.** The expression and its producer groups already fix the input data and its fields, so no requirement needs to restate them. -- **It depends on how rules generate the candidate space.** A rule that only generates legal candidates needs no requirement for that dimension; a rule that generates every value needs the requirement to filter them. -- **A requirement prunes; it does not decide the plan.** A tighter requirement removes candidates earlier. Each query's composed error is still checked at its root in Stage 3 (row (K)), so a requirement that is too loose costs enumeration time, not a wrong plan. - -### Extensibility - -**Add a value to a property dimension**, e.g. a new summary family or `Medium::Disk`: -1. Add the enum variant. -2. Every exhaustive `match` on the dimension stops compiling until it handles the new value. This is why matches on properties name every value and never use a `_` arm. -3. For a family: add its predefined $\theta$ list, its error-bound function, its cost, and the readouts it answers. -4. Add any new invalid combinations to `new`. - -**Add a property dimension**, e.g. a compression codec or a partitioning: -1. Decide whether it is logical (it changes the result's values or error) or physical (it changes only when, where or how long the result is computed). Add a field (a new enum) to `LogicalProperties` or `PhysicalProperties`. -2. If consumers can constrain it, add an optional field to `LogicalRequirement` or `PhysicalRequirement` and one check to `satisfies` or `satisfies_physical`. -3. Add its invalid combinations with the other dimensions to `new`. -4. Add a level to the decision tree, with its prune rule, and place it by the considerations in "Order of decisions". -5. Add its cost terms. - -**Add a requirement:** -1. If it constrains one edge (a consumer input), add a field to `LogicalRequirement` or `PhysicalRequirement`, set it where the consumer is created, and check it in `satisfies` or `satisfies_physical`. -2. If it constrains a whole query or the whole plan (accuracy, latency, budgets), add it to the query's or workload's requirements and a row to Stage 3. Do not split it into per-edge requirements: that fixes each input's share in advance. -3. If it comes from the deployment, prune the values it rules out during candidate generation. - -**Add a rule:** -1. A rule matches over groups and their candidates' expressions, never over a reader's private operators. -2. It adds candidates to an existing group, or adds a new group with its candidates. It never copies a reader: every existing consumer of the group sees the new candidate through edges. -3. A candidate whose result has the same meaning as an existing group is added to that group, so hash-consing keeps sharing. -4. Each new candidate states its input requirements. - -## 4. Data structures and interfaces per stage +## 3. Data structures and interfaces per stage This section gives the target interface of each stage after the formulation refactor (#1207). Each stage's boundary type carries the notation that stage defines, and no stage reads a notation a later stage defines: logical stages carry no physical properties. From 238e6bb15a09688312a74d617b31e15db89a3feb Mon Sep 17 00:00:00 2001 From: zzylol <50204836+zzylol@users.noreply.github.com> Date: Thu, 5 Nov 2026 20:14:57 +0000 Subject: [PATCH 06/44] docs(concepts): sections 1-2 in the formulation's terms; section 4 rules per pass and stage Co-Authored-By: Claude Opus 5.5 --- .../memo-groups-properties-requirements.md | 263 ++++++++++++------ 1 file changed, 176 insertions(+), 87 deletions(-) diff --git a/docs/design_docs/concepts/memo-groups-properties-requirements.md b/docs/design_docs/concepts/memo-groups-properties-requirements.md index 7769b3b10..602dd1d22 100644 --- a/docs/design_docs/concepts/memo-groups-properties-requirements.md +++ b/docs/design_docs/concepts/memo-groups-properties-requirements.md @@ -1,110 +1,138 @@ # Memo groups, properties and requirements -This page defines the memo's main data structures: groups, candidates, properties and requirements. It also lists every property and requirement, and gives the decision tree Stage 1 and Stage 2 walk to enumerate candidates. The symbols follow the planner formulation: -- a candidate is $v = (e_v, p_v)$; -- its logical properties are $p^{\log}$ and its physical properties are $p^{\mathrm{phys}}$; -- a consumer input's requirements are $r^{\log}_{c,i}$ and $r^{\mathrm{phys}}_{c,i}$; -- whether a producer satisfies them is $\alpha^{\log}$ and $\alpha^{\mathrm{phys}}$. +This page defines the memo's data structures in the terms of the optimization problem formulation: workloads, memo groups, candidates, properties and property requirements. It also gives the decision tree Stage 1 and Stage 2 walk to enumerate candidates, and each stage's interfaces (section 3). Where tracking issue #1207 extends the formulation, the extension is marked **(#1207)**. + +## Notation + +| Notation | Definition | +|---|---| +| $\mathcal W = \{(q, \mathcal T_q, A_q) \mid q \in \mathcal Q\}$ | workload over an evaluation horizon $H$: each query expression $q$, its evaluation schedule $\mathcal T_q \subseteq [0, H)$, and its accuracy target $A_q$ | +| $\mathcal Q$ | set of queries, each with one final output | +| $G$ | memo group: a logical result or an intermediate state family over specified input data, independent of parameter configurations | +| $G_q$ | memo group of query $q$'s final result | +| $v = (e_v, p_v)$ | candidate of a memo group: an expression with its properties | +| $e_v = \operatorname{op}_v(G_{v,1}, \dots, G_{v,n_v})$ | expression: operator $\operatorname{op}_v$ with its operator arguments (e.g. the summary family), over input memo groups $G_{v,i}$ | +| $p_v = (p^{\log}_v, p^{\mathrm{phys}}_v)$ | properties: logical and physical | +| $p^{\log}_v = \boldsymbol\theta_v$ | summary algorithm parameters, e.g. sketch size; **(#1207)** also the window layout $\omega_v$ | +| $p^{\mathrm{phys}}_v = (\tau_v, \ell_v)$ | execution timing and lifetime; **(#1207)** also the schedule $\sigma_v$ when $\ell_v \ne$ None | +| $(\varepsilon_v, \delta_v)$ | error bound and failure probability of $\boldsymbol\theta_v$ for the statistic $\operatorname{op}_v$ reads; derived, not a property | +| $c \in G_c$, $u \in G_u$ | consumer candidate and its group; producer candidate and its group | +| $\mathcal I(c)$ | input indices of consumer $c$ | +| $G_{c,i}$ | memo group required by input $i$ of consumer $c$ | +| $r_{c,i} = (r^{\log}_{c,i}, r^{\mathrm{phys}}_{c,i})$ | property requirement of input $i$ of consumer $c$ | +| $p(u) \succeq r_{c,i}$ | producer $u$'s properties satisfy the requirement: $p^{\log}(u) \succeq r^{\log}_{c,i} \land p^{\mathrm{phys}}(u) \succeq r^{\mathrm{phys}}_{c,i}$ | +| $\alpha^{\log}_{u,c,i}$, $\alpha^{\mathrm{phys}}_{u,c,i}$ | fixed indicators of $p^{\log}_u \succeq r^{\log}_{c,i}$ and $p^{\mathrm{phys}}_u \succeq r^{\mathrm{phys}}_{c,i}$ | +| $\mathcal C$ | physical candidates, each with a fixed parameter configuration, timing and lifetime | +| $P_{c,i}$ | physical candidates in the required input group, $G_{c,i} \cap \mathcal C$, before checking requirements | +| $\mathcal E$ | potential dependencies $\{(u, c, i) \mid c \in \mathcal C,\ i \in \mathcal I(c),\ u \in P_{c,i}\}$ | +| $x_v$, $y_{u,c,i}$, $z_{q,v}$, $t_v$ | MILP variables: candidate selected; producer $u$ supplies input $i$ of $c$; $v \in G_q$ supplies $q$'s result; topological rank | +| (A)–(I) | MILP rows: input satisfaction, producer selection and sharing, query coverage, output selection, acyclicity, utilization, logical requirements ($y \le \alpha^{\log}$), physical requirements ($y \le \alpha^{\mathrm{phys}}$), domains | +| $w_v$ | fixed cost: $C_{\text{read}}(v) + C_{\text{compute}}(v)$ plus $C^{\text{one}}_{\text{retain}}(v)$ or $C^{\text{across}}_{\text{retain}}(v)$ by $\ell_v$ | +| $\Pi$, $V_\Pi$ | selected physical ASAP DAG and its node set; $\operatorname{Cost}(\Pi) = \sum_{v \in \mathcal C} w_v x_v$ | +| **(#1207)** $\kappa$ | execution context of a candidate with $\ell_v$ = None: a query evaluation $\kappa_q$, an ingestion pipeline $I_r$, or a materialized candidate's computation $\kappa_m$ | +| **(#1207)** (K), (R), (L), budgets | accuracy at each query root, $\sum_u \lambda_{q,u}\varepsilon_u a_{q,u} \le \varepsilon_q$ and $\sum_u \delta_u a_{q,u} \le \delta_q$; retention $h_v$; latency; resource budgets | ## Example -Query A: `quantile_over_time(0.95, latency[1m])` with `max_error 0.01`. Query B: `quantile_over_time(0.99, latency[1m]) / quantile_over_time(0.50, latency[1m])` with the same target. +Workload: query A, `quantile_over_time(0.95, latency[1m])`, and query B, `quantile_over_time(0.99, latency[1m]) / quantile_over_time(0.50, latency[1m])`. Both are evaluated every 10 s ($\mathcal T_A = \mathcal T_B = 10\,\text{s}\cdot\mathbb Z$) with accuracy target $A_q$: rank error 0.01. ``` G1 Scan latency[1m] - producers: v1 = Scan (QueryTime, None) -G2 KLL state over G1 one group, whatever the size - producers: v2 = SummaryAgg[KLL](G1) θ = {k = 200} ε = 0.008 δ = 0.01 - v3 = SummaryAgg[KLL](G1) θ = {k = 345} ε = 0.005 δ = 0.01 - (each also expanded by Stage 2 into its (τ, ℓ, σ) alternatives) -G3 p95 of G2 producers: SummaryEstimate[q=0.95](G2) -G4 p99 of G2 producers: SummaryEstimate[q=0.99](G2) -G5 p50 of G2 producers: SummaryEstimate[q=0.50](G2) -G6 G4 / G5 producers: Divide(G4, G5) input requirement: error_cap on each input -Query roots: A → G3, B → G6 + candidate: v1 = Scan p^phys = (QueryTime, None) +G2 KLL state over G1 one group, whatever θ + candidates: v2 = SummaryAgg[KLL](G1) θ = {k = 200} ε = 0.008 δ = 0.01 + v3 = SummaryAgg[KLL](G1) θ = {k = 345} ε = 0.005 δ = 0.01 + (Stage 2 expands each into its (τ, ℓ) alternatives) +G3 = G_A SummaryEstimate[q=0.95](G2) +G4 SummaryEstimate[q=0.99](G2) +G5 SummaryEstimate[q=0.50](G2) +G6 = G_B Divide(G4, G5) r^log on each input: error cap (#1207) ``` The ε values are illustrative. -- G2 is one group: the KLL state over `latency[1m]`. Its sizes are candidates of G2, not separate groups. -- G3, G4 and G5 are consumers of G2. They are not stored in G2; each is a group with its own producers whose expression reads G2. -- A meets 0.01 with either size. B adds its two operands' errors (0.016 with `k = 200`, 0.010 with `k = 345`) and needs `k = 345` for both, so a plan where all three read v3 builds one KLL state. +- G2 is one memo group: the KLL state over `latency[1m]`. Its parameter configurations are candidates of G2, not separate groups. The family KLL is an argument of $\operatorname{op}_v$. +- G3, G4 and G5 are consumers of G2: their candidates' expressions read G2 as $G_{c,1}$. G2 does not store them. +- A meets 0.01 with either θ. B adds its two operands' errors (0.016 with `k = 200`, 0.010 with `k = 345`) and needs `k = 345` for both, so a plan where G3's, G4's and G5's candidates all read v3 builds one KLL state. -## 1. Properties and requirements over properties +## 1. Properties and property requirements ### Properties -A property is a value a candidate has. Every property is one dimension of a product: a struct with one field per dimension, never a flat enum that lists the combinations. Invalid combinations are rejected by the struct's checked constructor. +A property is a value a candidate has. Properties are products of dimensions: one field per dimension, never a flat enum that lists the combinations. Invalid combinations are rejected by the checked constructor. -| Dimension | Values | Logical or physical | Fixed by | Code | -|---|---|---|---|---| -| Which family to use (operator) | `Exact` (exact accumulator kinds: Sum, Count, Min, Max, Increase, Rate, IRate), or `Summary { family }` with family in KLL, DDSketch, CMS, CountSketch, CMS with heap, CountSketch with heap, UnivMon, HLL, KMV, the Hydra variants | logical | enumerated | `FieldDataType::{ExactAggregate, Sketch}` in the candidate's operator ([`schema/mod.rs`](../../../crates/types/src/ir/schema/mod.rs)); `SketchAlgorithm` | -| Parameter configuration $\theta$ | per family, from a predefined list: KLL `k`; DDSketch `alpha`; CMS / CountSketch `width × depth`; CMS / CountSketch with heap `width × depth × heap_size`; UnivMon `heap_size × rows × cols × layers`; HLL `precision`; KMV `k` | logical | enumerated | `SketchParams` inside `SketchKind` ([`sketch.rs`](../../../crates/types/src/ir/schema/sketch.rs)), sized by `size_params` per `accuracy_budget_parts` | -| Error bound $(\varepsilon_v, \delta_v)$ | computed on demand from $\theta$ and the statistic, in the family's accuracy metric (below); exact is $(0, 0)$ | logical | derived from $\theta$, not stored or chosen | `local_guarantee` → `AccuracyGuarantee` ([`estimators`](../../../crates/accuracy/src/estimators/mod.rs)) | -| Window configuration $\omega$ | none; or tumbling windows of width $w$ and count $n$, with $w$ from the gcd closure of the readers' ranges and steps. Later: sliding windows by merge or by subtraction | logical | enumerated | `SummaryTumblingWindowChain` arguments, added by Pass 2 ([`window_candidates.rs`](../../../crates/logical-asap-optimization/src/pass2/memo/window_candidates.rs)) | -| Timing $\tau$ | `IngestionTime`, `QueryTime` | physical | enumerated | `ExecutionTiming` ([`execution_data_state.rs`](../../../crates/types/src/ir/properties/execution_data_state.rs)); `PhysicalCandidate::timing` | -| Lifetime $\ell$ | `None`, `OneEvaluation(Medium)`, `AcrossEvaluations(Medium)`, with `Medium` in `Memory`, `Disk` | physical | enumerated | `Materialization` on main; `Lifetime`, `Medium` in #1214 ([`physical/`](../../../crates/types/src/physical)) | -| Schedule $\sigma$ | only when $\ell \ne$ None. `Periodic { interval, phase }` for `OneEvaluation`; `TumblingWindowGrid { phase }` for `AcrossEvaluations` | physical | enumerated from the readers' schedules | not a property: schedule groups `z[m, G]` in the MILP ([`cost_terms.rs`](../../../crates/plan-selection/src/milp/cost_terms.rs)); a property in #1200 PR 2 | +What is computed is the expression $e_v$, not a property: the operator, its arguments (including the summary family), and its input groups. A memo group's candidates differ in their properties, and, when rules add other ways to compute the same result, in their expressions. -A candidate is materialized when $\ell \ne$ None. This is derived, not stored. +| Dimension | Values | Part of | Code | +|---|---|---|---| +| Parameter configuration $\boldsymbol\theta_v$ | per family, from a predefined list: KLL `k`; DDSketch `alpha`; CMS / CountSketch `width × depth`; CMS / CountSketch with heap `width × depth × heap_size`; UnivMon `heap_size × rows × cols × layers`; HLL `precision`; KMV `k`. None for an exact operator | $p^{\log}_v$ | `SketchParams` inside `SketchKind` ([`sketch.rs`](../../../crates/types/src/ir/schema/sketch.rs)), sized by `size_params` per `accuracy_budget_parts` today | +| **(#1207)** Window layout $\omega_v$ | none; or tumbling windows of width $w$ and count $n$, with $w$ from the gcd closure of the readers' ranges and steps. Later: sliding windows by merge or by subtraction | $p^{\log}_v$ | `SummaryTumblingWindowChain` arguments, added by Pass 2 ([`window_candidates.rs`](../../../crates/logical-asap-optimization/src/pass2/memo/window_candidates.rs)) | +| Execution timing $\tau_v$ | IngestionTime, QueryTime | $p^{\mathrm{phys}}_v$ | `ExecutionTiming` ([`execution_data_state.rs`](../../../crates/types/src/ir/properties/execution_data_state.rs)) | +| Lifetime $\ell_v$ | None, OneEvaluation, AcrossEvaluations; **(#1207)** a materialized lifetime carries its medium, Memory or Disk | $p^{\mathrm{phys}}_v$ | `Materialization` on main; `Lifetime`, `Medium` in #1214 ([`physical/`](../../../crates/types/src/physical)) | +| **(#1207)** Schedule $\sigma_v$ | only when $\ell_v \ne$ None: Periodic (interval, phase) for OneEvaluation; tumbling-window grid phase for AcrossEvaluations | $p^{\mathrm{phys}}_v$ | schedule groups `z[m, G]` in the MILP today ([`cost_terms.rs`](../../../crates/plan-selection/src/milp/cost_terms.rs)); a property in #1200 PR 2 | -Invalid combinations, rejected by `PhysicalProperties::new` (the list in #1200; #1214 implements the first, the others come with the schedule in #1200 PR 2): +Lifetime None means the output is discarded after its reader's execution completes. OneEvaluation retains the result until all readers within one evaluation finish. AcrossEvaluations retains it throughout the evaluation period, so later evaluations reuse it. A candidate is materialized when $\ell_v \ne$ None; this is derived, not stored. + +The error bound $(\varepsilon_v, \delta_v)$ is not a dimension: it is a function of $\boldsymbol\theta_v$ and the statistic $\operatorname{op}_v$ reads, derived when needed (`local_guarantee` in [`estimators`](../../../crates/accuracy/src/estimators/mod.rs)). An exact candidate has $(0, 0)$. + +Invalid combinations of $p^{\mathrm{phys}}$, rejected by `PhysicalProperties::new` (#1200; #1214 implements the first, the others come with $\sigma$ in #1200 PR 2): | Combination | Why | |---|---| | (IngestionTime, OneEvaluation) | there is no evaluation at ingestion time | -| $\ell$ = None with a schedule | where an unmaterialized result runs is its reader's execution context, not a property | +| $\ell$ = None with a schedule | where an unmaterialized result runs is its reader's execution context $\kappa$, not a property | | $\ell \ne$ None without a schedule | a materialized result must say on which times or grid it is delivered | | OneEvaluation with a grid, AcrossEvaluations with a periodic schedule | each lifetime has one kind of schedule | | a phase at or above the grid width or interval, or interval 0 | the phase is taken modulo the width or interval | | IngestionTime with a grid phase ≠ 0 | ingestion-time grids are anchored at the epoch today: the plan carries no origin | -Valid physical states: (QueryTime, None), (IngestionTime, None), (QueryTime, OneEvaluation(m)), (QueryTime, AcrossEvaluations(m)), (IngestionTime, AcrossEvaluations(m)), for each medium $m$. +Valid $(\tau, \ell)$: (QueryTime, None), (IngestionTime, None), (QueryTime, OneEvaluation), (QueryTime, AcrossEvaluations), (IngestionTime, AcrossEvaluations), each materialized one for each medium. + +### Property requirements -### Requirements over properties +Each input $i$ of consumer $c$ names a producer group $G_{c,i}$ and a property requirement $r_{c,i} = (r^{\log}_{c,i}, r^{\mathrm{phys}}_{c,i})$. Logical satisfaction checks configuration requirements; physical satisfaction checks whether the producer's timing and lifetime jointly support the required access. For a monotone size parameter, $\succeq$ reduces to numerical dominance, e.g. $k_u \ge k_{\text{required}}$ for KLL. -A requirement is a constraint that a consumer, a query or the deployment puts on producers. One requirement can constrain several property dimensions, and one dimension can be constrained by several requirements. +The workload's accuracy target $A_q$ is not an input requirement **(#1207)**: a per-input bound would fix each input's share of the query's budget. Row (K) bounds each query's composed error at its root instead. What stays in $r^{\log}$ is what depends only on the consumer. -| Requirement | Who states it | Values | Constrains | Checked where | +| Requirement | Stated by | Values | Constrains | Checked by | |---|---|---|---|---| -| Accuracy | each query, at its root | metric: absolute (in the value's units, or times the stream's L1 or L2 norm), relative, rank; bound $\varepsilon_q$; failure probability $\delta_q$; scope: per value, whole result, or groups above a share of the records | operator family and $\theta$ of every approximate candidate the query reads, through their $(\varepsilon_v, \delta_v)$ | Stage 3, row (K): the composed error at the root. Not per edge, because a per-input bound would fix each input's share of the budget | -| Validity cap `error_cap` | a non-linear consumer (ratio, product), per input | $\varepsilon$ cap | $\theta$ of the producer | per edge, $\alpha^{\log}$: row (K)'s linear bound holds only below the cap | -| Summary family | a consumer that reads summary state, per input | a family, e.g. KLL for a quantile readout | operator family | per edge, $\alpha^{\log}$ | -| Window | a consumer that reads windows, per input | the reader's range and step | window configuration: the width must divide both | per edge, $\alpha^{\log}$ | -| Capability of ingestion-time compute | the deployment and the data workload | the source has an ingestion pipeline; the executor supports the family; query-time results may be kept across evaluations | timing; operator family; (QueryTime, AcrossEvaluations) | candidate generation: such candidates are not offered | -| Reader timing and evaluation | a consumer, per input | when it reads (ingestion or query time) and from which evaluation (same or later) | timing and lifetime of the producer | per edge, $\alpha^{\mathrm{phys}}$ | -| Reader schedule | a periodic consumer, per input | its interval and phase $(I_c, \varphi_c)$ | schedule of the producer | per edge, $\alpha^{\mathrm{phys}}$ | -| Latency, resource budgets | the workload | per-query latency bound; memory and CPU budgets | the whole plan | Stage 3, rows (L) and (B) | +| Accuracy target $A_q$ | the workload, per query | metric: absolute (in the value's units, or times the stream's L1 or L2 norm), relative, rank; bound $\varepsilon_q$; failure probability $\delta_q$; scope: per value, whole result, or groups above a share of the records | $\boldsymbol\theta_u$ of every approximate candidate $q$ depends on, through $(\varepsilon_u, \delta_u)$ | row (K) **(#1207)** | +| Error cap, in $r^{\log}_{c,i}$ | a non-linear consumer (ratio, product), per input | a cap on $\varepsilon_u$; for a monotone size parameter, a minimum size $k_u \ge k_{\text{required}}$ | $\boldsymbol\theta_u$ | $\alpha^{\log}$, row (G): row (K)'s linear bound holds only below the cap | +| Window, in $r^{\log}_{c,i}$ **(#1207)** | a consumer that reads windows, per input | the reader's range and step | $\omega_u$: the width must divide both | $\alpha^{\log}$, row (G) | +| Summary family, in $r^{\log}_{c,i}$ | a consumer that reads summary state | a family | usually already fixed by $G_{c,i}$, whose $\operatorname{op}$ names the family | $\alpha^{\log}$, row (G) | +| Reader timing and evaluation, in $r^{\mathrm{phys}}_{c,i}$ | the consumer, per input | when it reads (ingestion or query time) and from which evaluation (same or later) | $\tau_u$, $\ell_u$ | $\alpha^{\mathrm{phys}}$, row (H) | +| Reader schedule, in $r^{\mathrm{phys}}_{c,i}$ **(#1207)** | a periodic consumer, from $\mathcal T_q$ | its interval and phase $(I_c, \varphi_c)$ | $\sigma_u$ | $\alpha^{\mathrm{phys}}$, row (H) | +| Deployment capability | the deployment and data | the source has an ingestion pipeline; the executor supports the family; query-time results may be kept across evaluations | $\tau$, $\operatorname{op}$, (QueryTime, AcrossEvaluations) | candidate generation: such candidates are not in $\mathcal C$ | +| Latency, resource budgets **(#1207)** | the workload | per-query latency bound; memory and CPU budgets | the whole plan | rows (L) and budgets | Accuracy metrics that may be added later, when a primitive needs them: top-k recall or precision, and a false-positive rate for membership summaries. -How requirements map to property dimensions: +How requirements map to properties: ``` -Accuracy (query) ──► operator family, θ (through ε_v, δ_v; row (K)) -error_cap (edge) ──► θ -Summary family (edge) ──► operator family -Window (edge) ──► window configuration -Ingestion capability (deploy)──► timing, operator family, lifetime -Reader timing/evaluation ──► timing, lifetime -Reader schedule ──► schedule -Latency, budgets (workload) ──► every dimension, through cost +A_q (workload) ──► θ_u of every approximate input (through ε_u, δ_u; row (K)) +error cap r^log ──► θ_u +window r^log ──► ω_u +reader timing r^phys ──► τ_u, ℓ_u +reader schedule r^phys ──► σ_u +deployment ──► τ, op, ℓ (not generated) +latency, budgets ──► every dimension, through w_v ``` ## 2. Decision tree of properties -Each memo group's candidates are the leaves of a tree. Every level decides one property dimension, and each edge out of a node is one value of that dimension. A path from the root to a leaf is one candidate. +Each memo group's candidates are the leaves of a tree. The first level chooses the expression $e_v$; every later level decides one property dimension. A path from the root to a leaf is one physical candidate in $\mathcal C$. ```mermaid flowchart TD - G["Group G
(what is computed)"] --> R1["Exact"] - G --> R2["Summary: KLL"] - G --> R3["Summary: DDSketch"] - G --> R4["…other families
(prune: executor support,
consumer's family requirement)"] + G["Memo group G
(logical result)"] --> R1["e_v: exact operator"] + G --> R2["e_v: SummaryAgg[KLL]"] + G --> R3["e_v: SummaryAgg[DDSketch]"] + G --> R4["e_v: …other operators
(prune: executor support)"] R2 --> T1["θ: k = 200
ε = 0.008"] R2 --> T2["θ: k = 345
ε = 0.005"] - R2 --> T3["θ: k = …
(prune: ε above every
reading query's target)"] + R2 --> T3["θ: k = …
(prune: ε above every
reading query's ε_q)"] T2 --> W1["ω: no windows"] T2 --> W2["ω: tumbling, width w
(prune: w must divide
some reader's range and step)"] @@ -113,49 +141,48 @@ flowchart TD W2 --> P2["τ = IngestionTime
(prune: no ingestion pipeline)"] P1 --> L1["ℓ = None"] - P1 --> L2["ℓ = OneEvaluation(m)"] - P1 --> L3["ℓ = AcrossEvaluations(m)
(prune: deployment keeps
no query-time results)"] + P1 --> L2["ℓ = OneEvaluation"] + P1 --> L3["ℓ = AcrossEvaluations
(prune: deployment keeps
no query-time results)"] P2 --> L4["ℓ = None"] - P2 --> L5["ℓ = AcrossEvaluations(m)"] + P2 --> L5["ℓ = AcrossEvaluations"] P2 -. invalid .-> L6["ℓ = OneEvaluation"] - L2 --> S1["σ: Periodic (I, φ)
from the gcd closure
of the readers' schedules"] + L2 --> S1["σ: Periodic (I, φ)
from the gcd closure
of the readers' 𝒯_q"] L3 --> S2["σ: grid phase φ mod w"] L5 --> S3["σ: grid phase 0"] ``` The levels, top to bottom: -| Level | Dimension | Stage | Values offered | Pruned when | Code | +| Level | Decides | Stage | Values offered | Pruned when | Code | |---|---|---|---|---|---| -| 1 | family (operator) | Stage 1, Pass 1 and Pass 2 rules | each way to compute the group | the executor cannot build the family; no consumer's family requirement accepts it | Pass 1 ([`per_computation_candidates.rs`](../../../crates/logical-asap-optimization/src/pass1/per_computation_candidates.rs)); Pass 2 rules ([`rules/`](../../../crates/logical-asap-optimization/src/pass2/memo/rules)) | -| 2 | $\theta$ | Stage 1, Pass 1 | the family's predefined list | $\varepsilon_v$ alone exceeds every reading query's target | Pass 1 sizing per `accuracy_budget_parts` ([`memo_population.rs`](../../../crates/logical-asap-optimization/src/pass1/memo_population.rs)); a predefined list in #1203 PR 2 | -| 3 | window configuration $\omega$ | Stage 1, Pass 2 | tumbling widths from the gcd closure of readers' ranges and steps | no reader's range and step is a multiple of the width | Pass 2 ([`window_candidates.rs`](../../../crates/logical-asap-optimization/src/pass2/memo/window_candidates.rs), [`state_groups.rs`](../../../crates/logical-asap-optimization/src/pass2/memo/state_groups.rs)) | -| 4 | timing $\tau$ | Stage 2 | QueryTime; IngestionTime | the source has no ingestion pipeline | Stage 2 `enumerate_candidate_configurations` ([`configurations.rs`](../../../crates/physical-asap-optimization/src/catalog/configurations.rs)) | -| 5 | lifetime $\ell$ | Stage 2 | None, OneEvaluation(m), AcrossEvaluations(m) | invalid with $\tau$; the deployment keeps no query-time results across evaluations | same function, with `DeploymentCapabilities::query_time_results_retained` | -| 6 | schedule $\sigma$ | Stage 2 | the gcd closure $\Sigma$ of the readers' schedules | only offered when $\ell \ne$ None | Stage 3 MILP today; Stage 2 in #1200 PR 2 | - -What the tree does not decide: -- **Which producer each consumer input reads.** That is the edge choice $y_{u,c,i}$ in Stage 3. Two consumers of one group may read different leaves, e.g. A reads `k = 200` and B reads `k = 345`. -- **Where an unmaterialized candidate runs.** A candidate with $\ell$ = None runs inside each reader's execution context; Stage 3 chooses it. -- **How much a materialized candidate retains.** The amount $h_v$ depends on the readers' longest range; Stage 3 chooses it. +| 1 | expression $e_v$ ($\operatorname{op}_v$ with its family) | Stage 1, Pass 1 and Pass 2 rules | each way to compute the group | the executor cannot build the family | Pass 1 ([`per_computation_candidates.rs`](../../../crates/logical-asap-optimization/src/pass1/per_computation_candidates.rs)); Pass 2 rules ([`rules/`](../../../crates/logical-asap-optimization/src/pass2/memo/rules)) | +| 2 | $\boldsymbol\theta_v$ | Stage 1, Pass 1 | the family's predefined list | $\varepsilon_v$ alone exceeds every reading query's $\varepsilon_q$ | Pass 1 sizing per `accuracy_budget_parts` ([`memo_population.rs`](../../../crates/logical-asap-optimization/src/pass1/memo_population.rs)); a predefined list in #1203 PR 2 | +| 3 | $\omega_v$ **(#1207)** | Stage 1, Pass 2 | tumbling widths from the gcd closure of readers' ranges and steps | no reader's range and step is a multiple of the width | Pass 2 ([`window_candidates.rs`](../../../crates/logical-asap-optimization/src/pass2/memo/window_candidates.rs), [`state_groups.rs`](../../../crates/logical-asap-optimization/src/pass2/memo/state_groups.rs)) | +| 4 | $\tau_v$ | Stage 2 | QueryTime; IngestionTime | the source has no ingestion pipeline | `enumerate_candidate_configurations` ([`configurations.rs`](../../../crates/physical-asap-optimization/src/catalog/configurations.rs)) | +| 5 | $\ell_v$ | Stage 2 | None, OneEvaluation, AcrossEvaluations | invalid with $\tau_v$; the deployment keeps no query-time results across evaluations | same function, with `DeploymentCapabilities::query_time_results_retained` | +| 6 | $\sigma_v$ **(#1207)** | Stage 2 | the gcd closure $\Sigma$ of the readers' schedules $\mathcal T_q$ | only offered when $\ell_v \ne$ None | Stage 3 MILP today; Stage 2 in #1200 PR 2 | + +What the tree does not decide, because it is a choice of the MILP over $\mathcal C$ and $\mathcal E$: +- **Which producer each consumer input reads:** $y_{u,c,i}$. Two consumers of one group may read different leaves, e.g. one reads `k = 200` and another `k = 345`. +- **Where an unmaterialized candidate runs (#1207):** a candidate with $\ell_v$ = None runs inside each reading context $\kappa$. +- **How much a materialized candidate retains (#1207):** $h_v$, from the selected readers' longest range, row (R). ### Order of decisions -**The order does not affect correctness.** The set of leaves is the product of the dimensions' value sets, minus the invalid combinations and the pruned values. Every prune rule depends only on the value it removes and on values already decided above it, so any order that decides a pruned dimension after the dimensions its prune rule reads yields the same set of leaves. +**The order does not affect correctness.** The leaves are the product of the dimensions' value sets, minus the invalid combinations and the pruned values. Every prune rule depends only on the value it removes and on values decided above it, so any order that decides a pruned dimension after the dimensions its prune rule reads yields the same $\mathcal C$. **The order affects the efficiency of enumeration.** The number of nodes visited is $\sum_{d} \prod_{d' \le d} |\text{values kept at } d'|$, so a dimension that prunes much should come early, and a dimension whose values multiply without pruning should come late. Four considerations set the order above: -1. **Logical before physical.** Logical dimensions (operator family, $\theta$, $\omega$) change the result's values or error; physical dimensions ($\tau$, $\ell$, $\sigma$) change only when and how long it is computed. Deciding logical dimensions first lets Stage 1 hash-cons groups and check $\alpha^{\log}$ on edges once, before Stage 2 multiplies each logical candidate by its physical alternatives. -2. **Strong prunes early.** The family is pruned by the executor and by consumers' family requirements, and $\theta$ by the accuracy target, before the window and physical levels multiply them. -3. **Dependent dimensions adjacent.** $\ell$ is constrained by $\tau$ (no OneEvaluation at ingestion), and $\sigma$ exists only when $\ell \ne$ None and its kind follows $\ell$. Deciding them in the order $\tau \to \ell \to \sigma$ means each invalid combination is pruned at the first level where it can be detected. -4. **Deployment-wide prunes as early as they can go.** If the deployment has no ingestion pipeline, the IngestionTime subtree is pruned once per candidate. Moving $\tau$ above $\omega$ would prune it once per group instead; this is an alternative order with the same leaves. +1. **Logical before physical.** $e_v$, $\boldsymbol\theta_v$ and $\omega_v$ change the result's values or error; $\tau_v$, $\ell_v$ and $\sigma_v$ change only when and how long it is computed and kept. Deciding $p^{\log}$ first lets Stage 1 hash-cons groups and compute $\alpha^{\log}$ once, before Stage 2 multiplies each logical candidate by its $p^{\mathrm{phys}}$ alternatives. +2. **Strong prunes early.** The operator is pruned by the executor, and $\boldsymbol\theta_v$ by the accuracy targets, before the window and physical levels multiply them. +3. **Dependent dimensions adjacent.** $\ell_v$ is constrained by $\tau_v$, and $\sigma_v$ exists only when $\ell_v \ne$ None and its kind follows $\ell_v$. Deciding $\tau \to \ell \to \sigma$ prunes each invalid combination at the first level where it can be detected. +4. **Deployment-wide prunes as early as they can go.** Without an ingestion pipeline, the IngestionTime subtree is pruned once per candidate. Moving $\tau$ above $\omega$ would prune it once per group instead; this is an alternative order with the same leaves. Alternative orders worth comparing: -- **$\tau$ first.** In a deployment where most sources have no ingestion pipeline, deciding timing at the group level prunes half the tree before $\theta$ and $\omega$ multiply it. -- **$\theta$ last among logical dimensions.** $\theta$ has the most values. Deciding $\omega$ before $\theta$ visits fewer nodes when many widths are pruned by readers' ranges. -- **Requirement-driven (top-down) vs producer-driven (bottom-up).** Top-down enumeration starts from each consumer's requirement and offers only the values it accepts; bottom-up offers every value and lets $\alpha$ filter edges. Top-down visits fewer nodes but must union the requirements of all consumers of a group, so that sharing between consumers is not lost. - +- **$\tau$ first.** Where most sources have no ingestion pipeline, deciding timing at the group level prunes half the tree before $\boldsymbol\theta$ and $\omega$ multiply it. +- **$\boldsymbol\theta$ last among logical dimensions.** $\boldsymbol\theta$ has the most values. Deciding $\omega$ before $\boldsymbol\theta$ visits fewer nodes when many widths are pruned by readers' ranges. +- **Requirement-driven (top-down) vs producer-driven (bottom-up).** Top-down enumeration starts from each consumer's $r_{c,i}$ and offers only the values it accepts; bottom-up offers every value and lets $\alpha$ filter $\mathcal E$. Top-down visits fewer nodes but must union the requirements of all consumers of a group, so that sharing between consumers is not lost. ## 3. Data structures and interfaces per stage @@ -167,7 +194,7 @@ This section gives the target interface of each stage after the formulation refa | Stage 1, Pass 1 | `DagOfLogicalMemoGroups` | `Memo` | $v = (e_v, p^{\log}_v)$, $r^{\log}_{c,i}$, $\theta$ list | #1203 | | Stage 1, Pass 2 | `Memo` | `LogicalASAPMemoCandidates` | rule-added candidates; logical edges with $\alpha^{\log}$ | #1203, #1201 | | Stage 2 | `LogicalASAPMemoCandidates` | `PhysicalASAPMemoCandidates` | $p^{\mathrm{phys}}_v = (\tau, \ell, \sigma)$, $r^{\mathrm{phys}}_{c,i}$, physical edges $\mathcal E$ with $\alpha^{\mathrm{phys}}$ | #1200, #1201 | -| Stage 3 | `PhysicalASAPMemoCandidates` | `DagOfSelectedCandidates` | contexts $\kappa$; cost $w_v$ (`asap-cost`); $x$, $y$, $z$, $t$, $h$; rows (A)–(K), (R), (L), (B) | #1204 | +| Stage 3 | `PhysicalASAPMemoCandidates` | `DagOfSelectedCandidates` | contexts $\kappa$; cost $w_v$ (`asap-cost`); $x$, $y$, $z$, $t$, $h$; rows (A)–(K), (R), (L), budgets | #1204 | | Realizer, export | `DagOfSelectedCandidates` | `PhysicalASAPDAG` | — | #1200 (export fields) | ### Stage 0: frontends and hash-consing @@ -368,7 +395,7 @@ fn reader_constraint_rows(..); // (J) fn accuracy_rows(..); // (K) Σ λ_{q,u} ε_u a_{q,u} ≤ ε_q, Σ δ_u a_{q,u} ≤ δ_q fn retention_rows(..); // (R) h_v ≥ need_{c,i} · y_{v,c,i} fn latency_rows(..); // (L) -fn resource_budget_rows(..); // (B) budgets +fn resource_budget_rows(..); // budgets (#1207) // Objective: min Σ w_v x_v + Σ ρ_κ w_v x_{κ,v} + Σ π_v h_v // crates/types/src/stages/physical/graph.rs (unchanged) @@ -385,3 +412,65 @@ pub struct DagOfSelectedCandidates { ### Realizer and export The realizer builds the plan from `DagOfSelectedCandidates`; each node's physical properties come from its `PhysicalCandidate`. The export writes timing and lifetime (and later the schedule) per node of `PhysicalASAPDAG`, changed together with the backend. + +## 4. Rules: how each pass and stage adds candidates + +A rule reads the memo and adds to it only through the interfaces of section 3: +- `Memo::find_or_add_group(expression)` for a group, so an expression that already exists returns its group (hash-consing); +- `Memo::add_candidate(group, CandidateForMemoGroup { expression, logical_properties, input_requirements, fused_groups })` for a candidate $v = (e_v, p^{\log}_v)$ with its $r^{\log}_{v,i}$. + +A rule never copies a reader and never edits an existing candidate: every consumer of a group sees a new candidate of that group through $\mathcal E$. Rules match over groups and their candidates' expressions, never over a reader's private operators. The order of the rules below is the order the code runs them; it affects which groups exist when a later rule matches, not which plans are valid. + +### Stage 0: hash-consing + +| Rule | Code | Matches | Adds | +|---|---|---|---| +| Hash-consing | `hash_consed` ([`stage0.rs`](../../../crates/logical-asap-optimization/src/stage0.rs)) | every operator node of every query root | one `MemoGroup` per distinct expression; `query_roots` point at $G_q$. No candidates. | + +### Stage 1, Pass 1: ways to compute each group + +Pass 1 adds, for each group a query computes, every way to compute it. Each intermediate result of a way to compute is its own group (`find_or_add_group`), so `Project ← SummaryEstimate ← SummaryAgg` is three groups, each with a one-operator candidate. + +| Way to compute | Code | Adds | +|---|---|---| +| Exact | [`per_computation_candidates.rs`](../../../crates/logical-asap-optimization/src/pass1/per_computation_candidates.rs), [`memo_population.rs`](../../../crates/logical-asap-optimization/src/pass1/memo_population.rs) | a candidate with the exact operator; $\boldsymbol\theta_v$ none, $(\varepsilon_v, \delta_v) = (0, 0)$ | +| Exact accumulator | same | a state group `ExactAggregate(kind)` over the input group, one candidate; a `FinalizeExactAccumulator` candidate in the result's group reading it | +| Summary, per family the executor supports | same | a state group `SummaryAgg[family]` over the input group, with **one candidate per $\boldsymbol\theta$ in the family's predefined list**, dropping a $\boldsymbol\theta$ whose $\varepsilon_v$ alone exceeds every reading query's $\varepsilon_q$; a `SummaryEstimate[statistic]` candidate in the result's group reading the state group, with $r^{\log}$ = error cap where the consumer is non-linear | +| Fused build (#1203 2c) | same | a build that computes several targets at once: a candidate of the outer target's state group, with `fused_groups` naming the others | + +Example: for A and B, Pass 1 adds G2 = `SummaryAgg[KLL](G1)` with candidates `k = 200` and `k = 345`, and G3, G4, G5 = `SummaryEstimate[q](G2)`. A and B's three estimates read one state group because their expressions over G1 hash-cons to one group. + +Today, Pass 1 instead sizes one $\boldsymbol\theta$ per `accuracy_budget_parts` and keeps `SummaryEstimate ← SummaryAgg` as private nodes inside one candidate. #1203 2a makes the intermediate results groups; #1203 PR 2 replaces sizing by the predefined list. + +### Stage 1, Pass 2: rules over groups + +| # | Rule | Code | Matches | Adds | Status | +|---|---|---|---|---|---| +| 1 | Local window composition (tumbling windows) | `add_local_window_composition` ([`window_candidates.rs`](../../../crates/logical-asap-optimization/src/pass2/memo/window_candidates.rs)) | a summary state group read by window readers whose ranges and steps share a gcd width $w$ | a candidate of the **same state group** with $\omega_v$ = tumbling $(w, n)$, its $r^{\log}$ window requirement on each reader input | today it copies readers per layout (`TumblingWindows` origin); #1160 makes it a candidate of the state group | +| 2 | Scan widening | `add_scan_label_candidates` ([`scan_labels.rs`](../../../crates/logical-asap-optimization/src/pass2/memo/rules/scan_labels.rs)) | several scan groups of one metric with different label sets | a group for the widened scan (`find_or_add_group`), and in each original scan group a candidate `Project(widened)` | #1203 2a-5 (#1222) | +| 3 | Summary inputs | `factor_summary_inputs` ([`summary_inputs.rs`](../../../crates/logical-asap-optimization/src/pass2/memo/rules/summary_inputs.rs)) | a value projection or filter below a summary build | that projection or filter as its own group; the build's candidate reads it | #1203 2a-2 (#1218); after 2a, Pass 1 already makes it a group | +| 4 | Top-k reuse | `add_topk_candidates` ([`topk.rs`](../../../crates/logical-asap-optimization/src/pass2/memo/rules/topk.rs)) | top-k groups over one ranking with different `k` | a candidate of each top-k group reading the largest-`k` group, with an exact-input requirement | #1203 2a-3 (#1220) | +| 5 | Finer aggregate | `add_finer_aggregate_candidates` ([`finer_aggregate.rs`](../../../crates/logical-asap-optimization/src/pass2/memo/rules/finer_aggregate.rs)) | a SQL aggregate whose grouping is coarser than another aggregate group's | a candidate of the coarse group: filter, then merge the finer group's rows with each measure's rollup | #1203 2a-3 (#1220) | +| 6 | Rollup | `add_rollup_candidates` ([`rollup.rs`](../../../crates/logical-asap-optimization/src/pass2/memo/rules/rollup.rs)) | a coarse aggregate or summary group and a finer group whose state rolls up | a candidate of the coarse group reading the finer group (exact or sketch merge) | #1203 2a-4 (#1221) | +| 7 | Window rollup | `add_window_rollup_candidates` (same) | a longer-range group whose windows a shorter-range group's state covers | a candidate of the longer-range group merging the shorter one's windows | #1203 2a-4 (#1221) | +| 8 | Summary results | `factor_summary_results` ([`summary_results.rs`](../../../crates/logical-asap-optimization/src/pass2/memo/rules/summary_results.rs)) | a summary readout inside a composite candidate | the readout as its own group | #1203 2a-2 (#1218) makes the readout a group in Pass 1; the rule is then deleted | +| 9 | Shared configurations | `add_state_configuration_candidates` ([`shared_configuration_readers.rs`](../../../crates/logical-asap-optimization/src/pass2/memo/rules/shared_configuration_readers.rs)) | readers that could share a summary at a size or window horizon that covers them | reader copies at the shared size (`SharedBuild` origin) | deleted in #1203 2b: the state group already holds every $\boldsymbol\theta$, and $y$ chooses per reader | +| 10 | Shared summary state groups | `add_shared_summary_state_groups` ([`state_groups.rs`](../../../crates/logical-asap-optimization/src/pass2/memo/state_groups.rs)) | summary builds directly over input groups | per-size concrete state groups with reader copies | deleted in #1203 2b and #1160: one state group per state family | +| — | Logical edges (#1201) | after the last rule | every candidate $c$ and input $i$ | one `LogicalEdge { producer, consumer, input, satisfies_logical_requirement }` per $u \in G_{c,i}$, with $\alpha^{\log}_{u,c,i}$ = `satisfies(p^log_u, r^log_{c,i})`; output `LogicalASAPMemoCandidates { memo, edges }` | #1201 | + +After #1203 2b, every Pass 2 rule has the same shape: match groups, then `find_or_add_group` for any new intermediate result and `add_candidate` for the new way to compute an existing group. A rule adds no candidate a group already has (same expression and $p^{\log}$). + +### Stage 2: physical expansion and edges + +Stage 2 adds no logical candidates. It expands each logical candidate into physical candidates, then builds $\mathcal E$. + +| Step | Code | Reads | Adds | +|---|---|---|---| +| Physical expansion | `enumerate_candidate_configurations` ([`configurations.rs`](../../../crates/physical-asap-optimization/src/catalog/configurations.rs)) | each `CandidateForMemoGroup`, deployment capabilities, readers' $\mathcal T_q$ | one `PhysicalCandidate { logical, properties, input_requirements }` per valid $p^{\mathrm{phys}} = (\tau, \ell, \sigma)$ that the deployment allows (section 2, levels 4–6), with one `PhysicalRequirement` per input: the reader's timing, the evaluation it reads from, and its schedule | +| Physical edges (#1201) | after expansion | each `LogicalEdge` | one `PhysicalEdge { producer, consumer, input, satisfies_logical_requirement, satisfies_physical_requirement }` per pair of the two logical candidates' physical candidates, with $\alpha^{\mathrm{phys}}$ = `satisfies_physical(p^phys_u, r^phys_{c,i})`; the admissible ones ($\alpha^{\log} = \alpha^{\mathrm{phys}} = 1$) through `admissible_edges_into` / `admissible_edges_out_of` | + +Example: v3 (`k = 345`) becomes, among others, (QueryTime, None), (QueryTime, OneEvaluation, Periodic(10 s, 0)) and (IngestionTime, AcrossEvaluations, grid phase 0). G3's estimate at query time has an admissible edge to each of them: same timing, or a query-time reader of an ingestion-time result kept across evaluations. + +### Stage 3: no rules + +Stage 3 adds no candidates and no edges. It reads $\mathcal C$ and the admissible part of $\mathcal E$, creates the MILP variables of section 3 over them, and decodes $\{x = 1\}$ and $\{y = 1\}$ into `DagOfSelectedCandidates`. From 35934a8ba02e275868ba999edc0e4f9bcb060782 Mon Sep 17 00:00:00 2001 From: zzylol <50204836+zzylol@users.noreply.github.com> Date: Thu, 5 Nov 2026 20:29:23 +0000 Subject: [PATCH 07/44] docs(concepts): error propagation function per consumer; validity and pruning caps Co-Authored-By: Claude Opus 5.5 --- .../memo-groups-properties-requirements.md | 51 ++++++++++++++++--- 1 file changed, 45 insertions(+), 6 deletions(-) diff --git a/docs/design_docs/concepts/memo-groups-properties-requirements.md b/docs/design_docs/concepts/memo-groups-properties-requirements.md index 602dd1d22..d29e6a1d4 100644 --- a/docs/design_docs/concepts/memo-groups-properties-requirements.md +++ b/docs/design_docs/concepts/memo-groups-properties-requirements.md @@ -30,6 +30,9 @@ This page defines the memo's data structures in the terms of the optimization pr | $w_v$ | fixed cost: $C_{\text{read}}(v) + C_{\text{compute}}(v)$ plus $C^{\text{one}}_{\text{retain}}(v)$ or $C^{\text{across}}_{\text{retain}}(v)$ by $\ell_v$ | | $\Pi$, $V_\Pi$ | selected physical ASAP DAG and its node set; $\operatorname{Cost}(\Pi) = \sum_{v \in \mathcal C} w_v x_v$ | | **(#1207)** $\kappa$ | execution context of a candidate with $\ell_v$ = None: a query evaluation $\kappa_q$, an ingestion pipeline $I_r$, or a materialized candidate's computation $\kappa_m$ | +| $g_c$ | error propagation function of consumer $c$: $\varepsilon_c \le g_c(\varepsilon_{c,1}, \dots, \varepsilon_{c,n})$ | +| $\lambda_{q,c,i}$ | slope of the composed propagation functions from input $i$ of $c$ to query $q$'s result | +| $\text{cap}_{c,i}$ | error cap of input $i$ of $c$: the smaller of $g_c$'s validity cap and the pruning cap | | **(#1207)** (K), (R), (L), budgets | accuracy at each query root, $\sum_u \lambda_{q,u}\varepsilon_u a_{q,u} \le \varepsilon_q$ and $\sum_u \delta_u a_{q,u} \le \delta_q$; retention $h_v$; latency; resource budgets | ## Example @@ -46,7 +49,7 @@ G2 KLL state over G1 one group, whatever θ G3 = G_A SummaryEstimate[q=0.95](G2) G4 SummaryEstimate[q=0.99](G2) G5 SummaryEstimate[q=0.50](G2) -G6 = G_B Divide(G4, G5) r^log on each input: error cap (#1207) +G6 = G_B Divide(G4, G5) r^log on each input: cap = min(validity cap of /, ε_B / λ) ``` The ε values are illustrative. @@ -94,10 +97,39 @@ Each input $i$ of consumer $c$ names a producer group $G_{c,i}$ and a property r The workload's accuracy target $A_q$ is not an input requirement **(#1207)**: a per-input bound would fix each input's share of the query's budget. Row (K) bounds each query's composed error at its root instead. What stays in $r^{\log}$ is what depends only on the consumer. +#### Error propagation and the error cap + +Each consumer $c$ has an error propagation function $g_c$ that bounds its output error by its input errors: + +$$\varepsilon_c \le g_c(\varepsilon_{c,1}, \dots, \varepsilon_{c,n}).$$ + +Composing them from each query's root down gives the query's error, and the accuracy target $A_q$ is $g_q(\dots) \le \varepsilon_q$ at the root. + +| Consumer | $g_c$ | Linear | +|---|---|---| +| single readout, `Project`, filter | $\varepsilon_{c,1}$ | yes | +| `sum by` over $n$ series (absolute error) | $n\,\varepsilon_{c,1}$ | yes | +| $X + Y$, $X - Y$ (absolute error) | $\varepsilon_X + \varepsilon_Y$ | yes | +| $X / Y$ (relative error) | $\dfrac{\varepsilon_X + \varepsilon_Y}{1 - \varepsilon_Y}$ | no | + +Row (K) is the composed $g$ made linear: $\lambda_{q,c,i}$ is its slope from input $i$ of $c$ to $q$'s result, so the weights sit on $(q, c, i)$. Failure probabilities add by a union bound and need no $g$. + +From $g_c$ come two caps on an input's error, both checked per edge as part of $r^{\log}_{c,i}$: + +1. **Validity cap: correctness, non-linear consumers only.** A non-linear $g_c$ is bounded by its linear form only in a region, e.g. $\varepsilon_Y \le$ cap for a ratio, where $\frac{\varepsilon_X+\varepsilon_Y}{1-\varepsilon_Y} \le \frac{1}{1-\text{cap}}(\varepsilon_X+\varepsilon_Y)$, so $\lambda = \frac{1}{1-\text{cap}}$ (1.11 for cap 0.1). Without the cap, row (K) could accept a plan whose true error is unbounded. A linear $g_c$ is exact everywhere and has no validity cap. +2. **Pruning cap: efficiency only, every consumer, linear ones included.** An input whose error alone breaks every target it could serve is in no feasible plan: $\lambda_{q,c,i}\,\varepsilon_u > \varepsilon_q$ for every query $q$ that may reach $c$. Removing such edges removes no feasible plan, since row (K) rejects them anyway; it keeps them out of $\mathcal E$ and the MILP. It is the per-edge form of Pass 1's "drop a $\boldsymbol\theta$ whose $\varepsilon$ alone exceeds every reading query's target", and prunes more because it uses each reader's own $\lambda$. + +$$\text{cap}_{c,i} = \min\Bigl(\text{validity cap of } g_c,\ \max_{q \text{ that may reach } c} \frac{\varepsilon_q}{\lambda_{q,c,i}}\Bigr), \qquad \alpha^{\log}_{u,c,i} = 0 \text{ when } \varepsilon_u > \text{cap}_{c,i}.$$ + +For a monotone size parameter, $\varepsilon_u \le \text{cap}_{c,i}$ is a minimum size, e.g. $k_u \ge k_{\text{required}}$ for KLL. + +**What stays at the root.** A per-input cap cannot replace row (K): with several inputs, $\varepsilon_X + \varepsilon_Y \le \varepsilon_q$ couples them, and one cap per input would fix each input's share in advance (the rejected budget shares). The caps only prune; row (K) decides. + + | Requirement | Stated by | Values | Constrains | Checked by | |---|---|---|---|---| | Accuracy target $A_q$ | the workload, per query | metric: absolute (in the value's units, or times the stream's L1 or L2 norm), relative, rank; bound $\varepsilon_q$; failure probability $\delta_q$; scope: per value, whole result, or groups above a share of the records | $\boldsymbol\theta_u$ of every approximate candidate $q$ depends on, through $(\varepsilon_u, \delta_u)$ | row (K) **(#1207)** | -| Error cap, in $r^{\log}_{c,i}$ | a non-linear consumer (ratio, product), per input | a cap on $\varepsilon_u$; for a monotone size parameter, a minimum size $k_u \ge k_{\text{required}}$ | $\boldsymbol\theta_u$ | $\alpha^{\log}$, row (G): row (K)'s linear bound holds only below the cap | +| Error cap $\text{cap}_{c,i}$, in $r^{\log}_{c,i}$ | every consumer, per input, from $g_c$ and the targets of the queries that may reach it | the smaller of the validity cap (non-linear $g_c$ only) and the pruning cap; for a monotone size parameter, a minimum size $k_u \ge k_{\text{required}}$ | $\boldsymbol\theta_u$ | $\alpha^{\log}$, row (G): validity cap for correctness of (K), pruning cap for efficiency | | Window, in $r^{\log}_{c,i}$ **(#1207)** | a consumer that reads windows, per input | the reader's range and step | $\omega_u$: the width must divide both | $\alpha^{\log}$, row (G) | | Summary family, in $r^{\log}_{c,i}$ | a consumer that reads summary state | a family | usually already fixed by $G_{c,i}$, whose $\operatorname{op}$ names the family | $\alpha^{\log}$, row (G) | | Reader timing and evaluation, in $r^{\mathrm{phys}}_{c,i}$ | the consumer, per input | when it reads (ingestion or query time) and from which evaluation (same or later) | $\tau_u$, $\ell_u$ | $\alpha^{\mathrm{phys}}$, row (H) | @@ -111,7 +143,7 @@ How requirements map to properties: ``` A_q (workload) ──► θ_u of every approximate input (through ε_u, δ_u; row (K)) -error cap r^log ──► θ_u +error cap r^log ──► θ_u (validity cap from g_c; pruning cap from ε_q / λ_{q,c,i}) window r^log ──► ω_u reader timing r^phys ──► τ_u, ℓ_u reader schedule r^phys ──► σ_u @@ -264,12 +296,19 @@ pub enum WindowLayout { Tumbling { width_ms: u64, count: u32 } } pub struct LogicalRequirement { pub family: Option, pub window: Option, // reader's range and step: a width must divide both - pub error_cap: Option, // validity of row (K)'s linear bound + pub error_cap: Option, // cap_{c,i}: min(validity cap of g_c, pruning cap) } /// α^log_{u,c,i} pub fn satisfies(producer: &LogicalProperties, requirement: &LogicalRequirement) -> bool; + +/// g_c linearized: λ per input and the validity cap (None when g_c is linear). +/// New abstraction, not yet in #1207: pending approval. +pub struct ErrorPropagation { pub slope_per_input: Vec, pub validity_cap: Option } +pub fn error_propagation(consumer: &Operator) -> ErrorPropagation; ``` +The validity cap is known when the consumer is created. The pruning cap needs $\lambda_{q,c,i}$ and the queries that may reach $c$, so it is filled in when Pass 2 builds the logical edges. + Deleted: `root_node` with private nodes, `input_groups` (the expression's children), `accuracy_budget_parts`, the `SharedBuild` and `SummaryResult` origins. ### Stage 1, Pass 2: rules and logical edges @@ -435,7 +474,7 @@ Pass 1 adds, for each group a query computes, every way to compute it. Each inte |---|---|---| | Exact | [`per_computation_candidates.rs`](../../../crates/logical-asap-optimization/src/pass1/per_computation_candidates.rs), [`memo_population.rs`](../../../crates/logical-asap-optimization/src/pass1/memo_population.rs) | a candidate with the exact operator; $\boldsymbol\theta_v$ none, $(\varepsilon_v, \delta_v) = (0, 0)$ | | Exact accumulator | same | a state group `ExactAggregate(kind)` over the input group, one candidate; a `FinalizeExactAccumulator` candidate in the result's group reading it | -| Summary, per family the executor supports | same | a state group `SummaryAgg[family]` over the input group, with **one candidate per $\boldsymbol\theta$ in the family's predefined list**, dropping a $\boldsymbol\theta$ whose $\varepsilon_v$ alone exceeds every reading query's $\varepsilon_q$; a `SummaryEstimate[statistic]` candidate in the result's group reading the state group, with $r^{\log}$ = error cap where the consumer is non-linear | +| Summary, per family the executor supports | same | a state group `SummaryAgg[family]` over the input group, with **one candidate per $\boldsymbol\theta$ in the family's predefined list**, dropping a $\boldsymbol\theta$ whose $\varepsilon_v$ alone exceeds every reading query's $\varepsilon_q$; a `SummaryEstimate[statistic]` candidate in the result's group reading the state group. Each consumer's `error_cap` starts as its $g_c$'s validity cap (none for a linear $g_c$) | | Fused build (#1203 2c) | same | a build that computes several targets at once: a candidate of the outer target's state group, with `fused_groups` naming the others | Example: for A and B, Pass 1 adds G2 = `SummaryAgg[KLL](G1)` with candidates `k = 200` and `k = 345`, and G3, G4, G5 = `SummaryEstimate[q](G2)`. A and B's three estimates read one state group because their expressions over G1 hash-cons to one group. @@ -456,7 +495,7 @@ Today, Pass 1 instead sizes one $\boldsymbol\theta$ per `accuracy_budget_parts` | 8 | Summary results | `factor_summary_results` ([`summary_results.rs`](../../../crates/logical-asap-optimization/src/pass2/memo/rules/summary_results.rs)) | a summary readout inside a composite candidate | the readout as its own group | #1203 2a-2 (#1218) makes the readout a group in Pass 1; the rule is then deleted | | 9 | Shared configurations | `add_state_configuration_candidates` ([`shared_configuration_readers.rs`](../../../crates/logical-asap-optimization/src/pass2/memo/rules/shared_configuration_readers.rs)) | readers that could share a summary at a size or window horizon that covers them | reader copies at the shared size (`SharedBuild` origin) | deleted in #1203 2b: the state group already holds every $\boldsymbol\theta$, and $y$ chooses per reader | | 10 | Shared summary state groups | `add_shared_summary_state_groups` ([`state_groups.rs`](../../../crates/logical-asap-optimization/src/pass2/memo/state_groups.rs)) | summary builds directly over input groups | per-size concrete state groups with reader copies | deleted in #1203 2b and #1160: one state group per state family | -| — | Logical edges (#1201) | after the last rule | every candidate $c$ and input $i$ | one `LogicalEdge { producer, consumer, input, satisfies_logical_requirement }` per $u \in G_{c,i}$, with $\alpha^{\log}_{u,c,i}$ = `satisfies(p^log_u, r^log_{c,i})`; output `LogicalASAPMemoCandidates { memo, edges }` | #1201 | +| — | Logical edges (#1201) | after the last rule | every candidate $c$ and input $i$ | one `LogicalEdge { producer, consumer, input, satisfies_logical_requirement }` per $u \in G_{c,i}$, after tightening each `error_cap` with the pruning cap $\max_q \varepsilon_q / \lambda_{q,c,i}$; $\alpha^{\log}_{u,c,i}$ = `satisfies(p^log_u, r^log_{c,i})`; output `LogicalASAPMemoCandidates { memo, edges }` | #1201 | After #1203 2b, every Pass 2 rule has the same shape: match groups, then `find_or_add_group` for any new intermediate result and `add_candidate` for the new way to compute an existing group. A rule adds no candidate a group already has (same expression and $p^{\log}$). From 9c5fca3b5bb7b8f3f1899ec1283ab547ac28018b Mon Sep 17 00:00:00 2001 From: zzylol <50204836+zzylol@users.noreply.github.com> Date: Thu, 5 Nov 2026 20:31:18 +0000 Subject: [PATCH 08/44] docs(concepts): ErrorPropagation approved Co-Authored-By: Claude Opus 5.5 --- .../design_docs/concepts/memo-groups-properties-requirements.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/design_docs/concepts/memo-groups-properties-requirements.md b/docs/design_docs/concepts/memo-groups-properties-requirements.md index d29e6a1d4..04d7f1e8a 100644 --- a/docs/design_docs/concepts/memo-groups-properties-requirements.md +++ b/docs/design_docs/concepts/memo-groups-properties-requirements.md @@ -302,7 +302,7 @@ pub struct LogicalRequirement { pub fn satisfies(producer: &LogicalProperties, requirement: &LogicalRequirement) -> bool; /// g_c linearized: λ per input and the validity cap (None when g_c is linear). -/// New abstraction, not yet in #1207: pending approval. +/// Approved 2026-11-05. pub struct ErrorPropagation { pub slope_per_input: Vec, pub validity_cap: Option } pub fn error_propagation(consumer: &Operator) -> ErrorPropagation; ``` From e2235db4113595c586765de20fabda0ba9316f0b Mon Sep 17 00:00:00 2001 From: zzylol <50204836+zzylol@users.noreply.github.com> Date: Thu, 5 Nov 2026 20:33:54 +0000 Subject: [PATCH 09/44] =?UTF-8?q?docs(concepts):=20=CE=BB=20is=20a=20propa?= =?UTF-8?q?gation=20factor=20read=20from=20the=20query=20expression?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Claude Opus 5.5 --- .../memo-groups-properties-requirements.md | 15 ++++++++++----- 1 file changed, 10 insertions(+), 5 deletions(-) diff --git a/docs/design_docs/concepts/memo-groups-properties-requirements.md b/docs/design_docs/concepts/memo-groups-properties-requirements.md index 04d7f1e8a..9d4b5d34e 100644 --- a/docs/design_docs/concepts/memo-groups-properties-requirements.md +++ b/docs/design_docs/concepts/memo-groups-properties-requirements.md @@ -31,7 +31,7 @@ This page defines the memo's data structures in the terms of the optimization pr | $\Pi$, $V_\Pi$ | selected physical ASAP DAG and its node set; $\operatorname{Cost}(\Pi) = \sum_{v \in \mathcal C} w_v x_v$ | | **(#1207)** $\kappa$ | execution context of a candidate with $\ell_v$ = None: a query evaluation $\kappa_q$, an ingestion pipeline $I_r$, or a materialized candidate's computation $\kappa_m$ | | $g_c$ | error propagation function of consumer $c$: $\varepsilon_c \le g_c(\varepsilon_{c,1}, \dots, \varepsilon_{c,n})$ | -| $\lambda_{q,c,i}$ | slope of the composed propagation functions from input $i$ of $c$ to query $q$'s result | +| $\lambda_{q,c,i}$ | propagation factor of input $i$ of $c$ to query $q$'s result, read off $q$'s expression | | $\text{cap}_{c,i}$ | error cap of input $i$ of $c$: the smaller of $g_c$'s validity cap and the pruning cap | | **(#1207)** (K), (R), (L), budgets | accuracy at each query root, $\sum_u \lambda_{q,u}\varepsilon_u a_{q,u} \le \varepsilon_q$ and $\sum_u \delta_u a_{q,u} \le \delta_q$; retention $h_v$; latency; resource budgets | @@ -112,11 +112,15 @@ Composing them from each query's root down gives the query's error, and the accu | $X + Y$, $X - Y$ (absolute error) | $\varepsilon_X + \varepsilon_Y$ | yes | | $X / Y$ (relative error) | $\dfrac{\varepsilon_X + \varepsilon_Y}{1 - \varepsilon_Y}$ | no | -Row (K) is the composed $g$ made linear: $\lambda_{q,c,i}$ is its slope from input $i$ of $c$ to $q$'s result, so the weights sit on $(q, c, i)$. Failure probabilities add by a union bound and need no $g$. +Each $g_c$ is a sum of its inputs' errors, each times a constant **propagation factor** $\lambda_{c,i}$: $g_c = \sum_i \lambda_{c,i}\,\varepsilon_{c,i}$ (for a non-linear $g_c$, within its validity cap, below). The factors come from the query expression itself: each operator and its arguments, e.g. `sum by` over $n$ series has factor $n$. No measurement or optimization is involved. A query's factor for input $i$ of $c$ multiplies the factors along the path in $q$'s expression from that input to $q$'s root, and adds over paths when the input reaches the root more than once: + +$$\lambda_{q,c,i} = \sum_{\text{paths } c = c_0 \to c_1 \to \dots \to c_m = \text{root of } q} \lambda_{c_0,i}\,\lambda_{c_1,i_1} \cdots \lambda_{c_m,i_m}.$$ + +Row (K) is $\sum \lambda_{q,c,i}\,\varepsilon_u\,y \le \varepsilon_q$ with these factors, so the weights sit on $(q, c, i)$. Two queries reading one group through different expressions have different factors. Failure probabilities add by a union bound and need no factor. From $g_c$ come two caps on an input's error, both checked per edge as part of $r^{\log}_{c,i}$: -1. **Validity cap: correctness, non-linear consumers only.** A non-linear $g_c$ is bounded by its linear form only in a region, e.g. $\varepsilon_Y \le$ cap for a ratio, where $\frac{\varepsilon_X+\varepsilon_Y}{1-\varepsilon_Y} \le \frac{1}{1-\text{cap}}(\varepsilon_X+\varepsilon_Y)$, so $\lambda = \frac{1}{1-\text{cap}}$ (1.11 for cap 0.1). Without the cap, row (K) could accept a plan whose true error is unbounded. A linear $g_c$ is exact everywhere and has no validity cap. +1. **Validity cap: correctness, non-linear consumers only.** A non-linear $g_c$ is bounded by its linear form only in a region, e.g. $\varepsilon_Y \le$ cap for a ratio, where $\frac{\varepsilon_X+\varepsilon_Y}{1-\varepsilon_Y} \le \frac{1}{1-\text{cap}}(\varepsilon_X+\varepsilon_Y)$, so both factors are $\frac{1}{1-\text{cap}}$ (1.11 for cap 0.1). Without the cap, row (K) could accept a plan whose true error is unbounded. A linear $g_c$ is exact everywhere and has no validity cap. 2. **Pruning cap: efficiency only, every consumer, linear ones included.** An input whose error alone breaks every target it could serve is in no feasible plan: $\lambda_{q,c,i}\,\varepsilon_u > \varepsilon_q$ for every query $q$ that may reach $c$. Removing such edges removes no feasible plan, since row (K) rejects them anyway; it keeps them out of $\mathcal E$ and the MILP. It is the per-edge form of Pass 1's "drop a $\boldsymbol\theta$ whose $\varepsilon$ alone exceeds every reading query's target", and prunes more because it uses each reader's own $\lambda$. $$\text{cap}_{c,i} = \min\Bigl(\text{validity cap of } g_c,\ \max_{q \text{ that may reach } c} \frac{\varepsilon_q}{\lambda_{q,c,i}}\Bigr), \qquad \alpha^{\log}_{u,c,i} = 0 \text{ when } \varepsilon_u > \text{cap}_{c,i}.$$ @@ -301,9 +305,10 @@ pub struct LogicalRequirement { /// α^log_{u,c,i} pub fn satisfies(producer: &LogicalProperties, requirement: &LogicalRequirement) -> bool; -/// g_c linearized: λ per input and the validity cap (None when g_c is linear). +/// g_c as propagation factors λ_{c,i}, read from the consumer's operator and +/// arguments, and the validity cap (None when g_c is linear). /// Approved 2026-11-05. -pub struct ErrorPropagation { pub slope_per_input: Vec, pub validity_cap: Option } +pub struct ErrorPropagation { pub factor_per_input: Vec, pub validity_cap: Option } pub fn error_propagation(consumer: &Operator) -> ErrorPropagation; ``` From 21d44dac7e191373cfd2ed53c7963efdff13f370 Mon Sep 17 00:00:00 2001 From: zzylol <50204836+zzylol@users.noreply.github.com> Date: Thu, 5 Nov 2026 20:40:09 +0000 Subject: [PATCH 10/44] docs(concepts): keep the candidate's origin, listed as not in the formulation --- .../concepts/memo-groups-properties-requirements.md | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/docs/design_docs/concepts/memo-groups-properties-requirements.md b/docs/design_docs/concepts/memo-groups-properties-requirements.md index 9d4b5d34e..5843e188c 100644 --- a/docs/design_docs/concepts/memo-groups-properties-requirements.md +++ b/docs/design_docs/concepts/memo-groups-properties-requirements.md @@ -281,6 +281,9 @@ pub struct CandidateForMemoGroup { pub logical_properties: LogicalProperties, // p_v^log pub input_requirements: Vec, // r^log_{c,i}, one per input group pub fused_groups: Vec, + /// Which rule added it. Not in the formulation: kept for `stage_pipeline --text` + /// and the viewer (approved 2026-11-05). + pub origin: OriginOfCandidateForMemoGroup, } /// e_v = op_v(G_{v,1}, …, G_{v,n}): one operator, its children are groups. pub struct CandidateExpression { pub operator: Operator } @@ -316,6 +319,8 @@ The validity cap is known when the consumer is created. The pruning cap needs $\ Deleted: `root_node` with private nodes, `input_groups` (the expression's children), `accuracy_budget_parts`, the `SharedBuild` and `SummaryResult` origins. +Concepts not in the formulation: `origin: OriginOfCandidateForMemoGroup` (Pass1, SummaryInput, TumblingWindows, Rollup, TopKReuse, WidenedScans) records which rule added a candidate; nothing in planning reads it. + ### Stage 1, Pass 2: rules and logical edges Rules add candidates to existing groups, or add groups. A rule never copies a reader. After the last rule, Pass 2 builds the logical edges once. From 6f487bd0d56b9b9f34c2c209c7c09217b966a952 Mon Sep 17 00:00:00 2001 From: zzylol <50204836+zzylol@users.noreply.github.com> Date: Thu, 5 Nov 2026 20:55:52 +0000 Subject: [PATCH 11/44] docs(concepts): consumer and query/context instead of reader; PhysicalRequirement.consumer_{timing,schedule} Co-Authored-By: Claude Opus 5.5 --- .../memo-groups-properties-requirements.md | 66 ++++++++++--------- 1 file changed, 35 insertions(+), 31 deletions(-) diff --git a/docs/design_docs/concepts/memo-groups-properties-requirements.md b/docs/design_docs/concepts/memo-groups-properties-requirements.md index 5843e188c..fff471115 100644 --- a/docs/design_docs/concepts/memo-groups-properties-requirements.md +++ b/docs/design_docs/concepts/memo-groups-properties-requirements.md @@ -16,7 +16,7 @@ This page defines the memo's data structures in the terms of the optimization pr | $p^{\log}_v = \boldsymbol\theta_v$ | summary algorithm parameters, e.g. sketch size; **(#1207)** also the window layout $\omega_v$ | | $p^{\mathrm{phys}}_v = (\tau_v, \ell_v)$ | execution timing and lifetime; **(#1207)** also the schedule $\sigma_v$ when $\ell_v \ne$ None | | $(\varepsilon_v, \delta_v)$ | error bound and failure probability of $\boldsymbol\theta_v$ for the statistic $\operatorname{op}_v$ reads; derived, not a property | -| $c \in G_c$, $u \in G_u$ | consumer candidate and its group; producer candidate and its group | +| $c \in G_c$, $u \in G_u$ | consumer candidate and its group; producer candidate and its group. This page says *consumer* for $c$ and *query* $q$ (or its execution context $\kappa$) for who needs a result on a schedule; it does not use "reader" | | $\mathcal I(c)$ | input indices of consumer $c$ | | $G_{c,i}$ | memo group required by input $i$ of consumer $c$ | | $r_{c,i} = (r^{\log}_{c,i}, r^{\mathrm{phys}}_{c,i})$ | property requirement of input $i$ of consumer $c$ | @@ -69,12 +69,12 @@ What is computed is the expression $e_v$, not a property: the operator, its argu | Dimension | Values | Part of | Code | |---|---|---|---| | Parameter configuration $\boldsymbol\theta_v$ | per family, from a predefined list: KLL `k`; DDSketch `alpha`; CMS / CountSketch `width × depth`; CMS / CountSketch with heap `width × depth × heap_size`; UnivMon `heap_size × rows × cols × layers`; HLL `precision`; KMV `k`. None for an exact operator | $p^{\log}_v$ | `SketchParams` inside `SketchKind` ([`sketch.rs`](../../../crates/types/src/ir/schema/sketch.rs)), sized by `size_params` per `accuracy_budget_parts` today | -| **(#1207)** Window layout $\omega_v$ | none; or tumbling windows of width $w$ and count $n$, with $w$ from the gcd closure of the readers' ranges and steps. Later: sliding windows by merge or by subtraction | $p^{\log}_v$ | `SummaryTumblingWindowChain` arguments, added by Pass 2 ([`window_candidates.rs`](../../../crates/logical-asap-optimization/src/pass2/memo/window_candidates.rs)) | +| **(#1207)** Window layout $\omega_v$ | none; or tumbling windows of width $w$ and count $n$, with $w$ from the gcd closure of the ranges and steps of the queries that reach the group. Later: sliding windows by merge or by subtraction | $p^{\log}_v$ | `SummaryTumblingWindowChain` arguments, added by Pass 2 ([`window_candidates.rs`](../../../crates/logical-asap-optimization/src/pass2/memo/window_candidates.rs)) | | Execution timing $\tau_v$ | IngestionTime, QueryTime | $p^{\mathrm{phys}}_v$ | `ExecutionTiming` ([`execution_data_state.rs`](../../../crates/types/src/ir/properties/execution_data_state.rs)) | | Lifetime $\ell_v$ | None, OneEvaluation, AcrossEvaluations; **(#1207)** a materialized lifetime carries its medium, Memory or Disk | $p^{\mathrm{phys}}_v$ | `Materialization` on main; `Lifetime`, `Medium` in #1214 ([`physical/`](../../../crates/types/src/physical)) | | **(#1207)** Schedule $\sigma_v$ | only when $\ell_v \ne$ None: Periodic (interval, phase) for OneEvaluation; tumbling-window grid phase for AcrossEvaluations | $p^{\mathrm{phys}}_v$ | schedule groups `z[m, G]` in the MILP today ([`cost_terms.rs`](../../../crates/plan-selection/src/milp/cost_terms.rs)); a property in #1200 PR 2 | -Lifetime None means the output is discarded after its reader's execution completes. OneEvaluation retains the result until all readers within one evaluation finish. AcrossEvaluations retains it throughout the evaluation period, so later evaluations reuse it. A candidate is materialized when $\ell_v \ne$ None; this is derived, not stored. +Lifetime None means the output is discarded after its consumer's execution completes. OneEvaluation retains the result until all its consumers within one evaluation finish. AcrossEvaluations retains it throughout the evaluation period, so later evaluations reuse it. A candidate is materialized when $\ell_v \ne$ None; this is derived, not stored. The error bound $(\varepsilon_v, \delta_v)$ is not a dimension: it is a function of $\boldsymbol\theta_v$ and the statistic $\operatorname{op}_v$ reads, derived when needed (`local_guarantee` in [`estimators`](../../../crates/accuracy/src/estimators/mod.rs)). An exact candidate has $(0, 0)$. @@ -83,7 +83,7 @@ Invalid combinations of $p^{\mathrm{phys}}$, rejected by `PhysicalProperties::ne | Combination | Why | |---|---| | (IngestionTime, OneEvaluation) | there is no evaluation at ingestion time | -| $\ell$ = None with a schedule | where an unmaterialized result runs is its reader's execution context $\kappa$, not a property | +| $\ell$ = None with a schedule | where an unmaterialized result runs is its consumer's execution context $\kappa$, not a property | | $\ell \ne$ None without a schedule | a materialized result must say on which times or grid it is delivered | | OneEvaluation with a grid, AcrossEvaluations with a periodic schedule | each lifetime has one kind of schedule | | a phase at or above the grid width or interval, or interval 0 | the phase is taken modulo the width or interval | @@ -121,7 +121,7 @@ Row (K) is $\sum \lambda_{q,c,i}\,\varepsilon_u\,y \le \varepsilon_q$ with these From $g_c$ come two caps on an input's error, both checked per edge as part of $r^{\log}_{c,i}$: 1. **Validity cap: correctness, non-linear consumers only.** A non-linear $g_c$ is bounded by its linear form only in a region, e.g. $\varepsilon_Y \le$ cap for a ratio, where $\frac{\varepsilon_X+\varepsilon_Y}{1-\varepsilon_Y} \le \frac{1}{1-\text{cap}}(\varepsilon_X+\varepsilon_Y)$, so both factors are $\frac{1}{1-\text{cap}}$ (1.11 for cap 0.1). Without the cap, row (K) could accept a plan whose true error is unbounded. A linear $g_c$ is exact everywhere and has no validity cap. -2. **Pruning cap: efficiency only, every consumer, linear ones included.** An input whose error alone breaks every target it could serve is in no feasible plan: $\lambda_{q,c,i}\,\varepsilon_u > \varepsilon_q$ for every query $q$ that may reach $c$. Removing such edges removes no feasible plan, since row (K) rejects them anyway; it keeps them out of $\mathcal E$ and the MILP. It is the per-edge form of Pass 1's "drop a $\boldsymbol\theta$ whose $\varepsilon$ alone exceeds every reading query's target", and prunes more because it uses each reader's own $\lambda$. +2. **Pruning cap: efficiency only, every consumer, linear ones included.** An input whose error alone breaks every target it could serve is in no feasible plan: $\lambda_{q,c,i}\,\varepsilon_u > \varepsilon_q$ for every query $q$ that may reach $c$. Removing such edges removes no feasible plan, since row (K) rejects them anyway; it keeps them out of $\mathcal E$ and the MILP. It is the per-edge form of Pass 1's "drop a $\boldsymbol\theta$ whose $\varepsilon$ alone exceeds every reading query's target", and prunes more because it uses each consumer's own $\lambda$. $$\text{cap}_{c,i} = \min\Bigl(\text{validity cap of } g_c,\ \max_{q \text{ that may reach } c} \frac{\varepsilon_q}{\lambda_{q,c,i}}\Bigr), \qquad \alpha^{\log}_{u,c,i} = 0 \text{ when } \varepsilon_u > \text{cap}_{c,i}.$$ @@ -134,10 +134,10 @@ For a monotone size parameter, $\varepsilon_u \le \text{cap}_{c,i}$ is a minimum |---|---|---|---|---| | Accuracy target $A_q$ | the workload, per query | metric: absolute (in the value's units, or times the stream's L1 or L2 norm), relative, rank; bound $\varepsilon_q$; failure probability $\delta_q$; scope: per value, whole result, or groups above a share of the records | $\boldsymbol\theta_u$ of every approximate candidate $q$ depends on, through $(\varepsilon_u, \delta_u)$ | row (K) **(#1207)** | | Error cap $\text{cap}_{c,i}$, in $r^{\log}_{c,i}$ | every consumer, per input, from $g_c$ and the targets of the queries that may reach it | the smaller of the validity cap (non-linear $g_c$ only) and the pruning cap; for a monotone size parameter, a minimum size $k_u \ge k_{\text{required}}$ | $\boldsymbol\theta_u$ | $\alpha^{\log}$, row (G): validity cap for correctness of (K), pruning cap for efficiency | -| Window, in $r^{\log}_{c,i}$ **(#1207)** | a consumer that reads windows, per input | the reader's range and step | $\omega_u$: the width must divide both | $\alpha^{\log}$, row (G) | +| Window, in $r^{\log}_{c,i}$ **(#1207)** | a consumer that reads windows, per input | the range and step of the queries that reach the consumer | $\omega_u$: the width must divide both | $\alpha^{\log}$, row (G) | | Summary family, in $r^{\log}_{c,i}$ | a consumer that reads summary state | a family | usually already fixed by $G_{c,i}$, whose $\operatorname{op}$ names the family | $\alpha^{\log}$, row (G) | -| Reader timing and evaluation, in $r^{\mathrm{phys}}_{c,i}$ | the consumer, per input | when it reads (ingestion or query time) and from which evaluation (same or later) | $\tau_u$, $\ell_u$ | $\alpha^{\mathrm{phys}}$, row (H) | -| Reader schedule, in $r^{\mathrm{phys}}_{c,i}$ **(#1207)** | a periodic consumer, from $\mathcal T_q$ | its interval and phase $(I_c, \varphi_c)$ | $\sigma_u$ | $\alpha^{\mathrm{phys}}$, row (H) | +| Consumer timing and evaluation, in $r^{\mathrm{phys}}_{c,i}$ | the consumer, per input | when it reads (ingestion or query time) and from which evaluation (same or later) | $\tau_u$, $\ell_u$ | $\alpha^{\mathrm{phys}}$, row (H) | +| Consumer schedule, in $r^{\mathrm{phys}}_{c,i}$ **(#1207)** | the consumer's execution context $\kappa$, from the $\mathcal T_q$ of its query | the context's interval and phase $(I_\kappa, \varphi_\kappa)$; the producer must deliver on them, $T_\kappa \subseteq T_{\sigma_u}$ | $\sigma_u$ | $\alpha^{\mathrm{phys}}$, row (H) | | Deployment capability | the deployment and data | the source has an ingestion pipeline; the executor supports the family; query-time results may be kept across evaluations | $\tau$, $\operatorname{op}$, (QueryTime, AcrossEvaluations) | candidate generation: such candidates are not in $\mathcal C$ | | Latency, resource budgets **(#1207)** | the workload | per-query latency bound; memory and CPU budgets | the whole plan | rows (L) and budgets | @@ -149,8 +149,8 @@ How requirements map to properties: A_q (workload) ──► θ_u of every approximate input (through ε_u, δ_u; row (K)) error cap r^log ──► θ_u (validity cap from g_c; pruning cap from ε_q / λ_{q,c,i}) window r^log ──► ω_u -reader timing r^phys ──► τ_u, ℓ_u -reader schedule r^phys ──► σ_u +consumer timing r^phys ──► τ_u, ℓ_u +consumer schedule r^phys ──► σ_u deployment ──► τ, op, ℓ (not generated) latency, budgets ──► every dimension, through w_v ``` @@ -168,10 +168,10 @@ flowchart TD R2 --> T1["θ: k = 200
ε = 0.008"] R2 --> T2["θ: k = 345
ε = 0.005"] - R2 --> T3["θ: k = …
(prune: ε above every
reading query's ε_q)"] + R2 --> T3["θ: k = …
(prune: ε above the ε_q of
every query reaching G)"] T2 --> W1["ω: no windows"] - T2 --> W2["ω: tumbling, width w
(prune: w must divide
some reader's range and step)"] + T2 --> W2["ω: tumbling, width w
(prune: w must divide the range and
step of some query reaching G)"] W2 --> P1["τ = QueryTime"] W2 --> P2["τ = IngestionTime
(prune: no ingestion pipeline)"] @@ -183,7 +183,7 @@ flowchart TD P2 --> L5["ℓ = AcrossEvaluations"] P2 -. invalid .-> L6["ℓ = OneEvaluation"] - L2 --> S1["σ: Periodic (I, φ)
from the gcd closure
of the readers' 𝒯_q"] + L2 --> S1["σ: Periodic (I, φ)
from the gcd closure
of the queries' 𝒯_q"] L3 --> S2["σ: grid phase φ mod w"] L5 --> S3["σ: grid phase 0"] ``` @@ -193,16 +193,16 @@ The levels, top to bottom: | Level | Decides | Stage | Values offered | Pruned when | Code | |---|---|---|---|---|---| | 1 | expression $e_v$ ($\operatorname{op}_v$ with its family) | Stage 1, Pass 1 and Pass 2 rules | each way to compute the group | the executor cannot build the family | Pass 1 ([`per_computation_candidates.rs`](../../../crates/logical-asap-optimization/src/pass1/per_computation_candidates.rs)); Pass 2 rules ([`rules/`](../../../crates/logical-asap-optimization/src/pass2/memo/rules)) | -| 2 | $\boldsymbol\theta_v$ | Stage 1, Pass 1 | the family's predefined list | $\varepsilon_v$ alone exceeds every reading query's $\varepsilon_q$ | Pass 1 sizing per `accuracy_budget_parts` ([`memo_population.rs`](../../../crates/logical-asap-optimization/src/pass1/memo_population.rs)); a predefined list in #1203 PR 2 | -| 3 | $\omega_v$ **(#1207)** | Stage 1, Pass 2 | tumbling widths from the gcd closure of readers' ranges and steps | no reader's range and step is a multiple of the width | Pass 2 ([`window_candidates.rs`](../../../crates/logical-asap-optimization/src/pass2/memo/window_candidates.rs), [`state_groups.rs`](../../../crates/logical-asap-optimization/src/pass2/memo/state_groups.rs)) | +| 2 | $\boldsymbol\theta_v$ | Stage 1, Pass 1 | the family's predefined list | $\varepsilon_v$ alone exceeds the $\varepsilon_q$ of every query that reaches the group | Pass 1 sizing per `accuracy_budget_parts` ([`memo_population.rs`](../../../crates/logical-asap-optimization/src/pass1/memo_population.rs)); a predefined list in #1203 PR 2 | +| 3 | $\omega_v$ **(#1207)** | Stage 1, Pass 2 | tumbling widths from the gcd closure of the ranges and steps of the queries reaching the group | no such query's range and step is a multiple of the width | Pass 2 ([`window_candidates.rs`](../../../crates/logical-asap-optimization/src/pass2/memo/window_candidates.rs), [`state_groups.rs`](../../../crates/logical-asap-optimization/src/pass2/memo/state_groups.rs)) | | 4 | $\tau_v$ | Stage 2 | QueryTime; IngestionTime | the source has no ingestion pipeline | `enumerate_candidate_configurations` ([`configurations.rs`](../../../crates/physical-asap-optimization/src/catalog/configurations.rs)) | | 5 | $\ell_v$ | Stage 2 | None, OneEvaluation, AcrossEvaluations | invalid with $\tau_v$; the deployment keeps no query-time results across evaluations | same function, with `DeploymentCapabilities::query_time_results_retained` | -| 6 | $\sigma_v$ **(#1207)** | Stage 2 | the gcd closure $\Sigma$ of the readers' schedules $\mathcal T_q$ | only offered when $\ell_v \ne$ None | Stage 3 MILP today; Stage 2 in #1200 PR 2 | +| 6 | $\sigma_v$ **(#1207)** | Stage 2 | the gcd closure $\Sigma$ of the schedules $\mathcal T_q$ of the queries reaching the group | only offered when $\ell_v \ne$ None | Stage 3 MILP today; Stage 2 in #1200 PR 2 | What the tree does not decide, because it is a choice of the MILP over $\mathcal C$ and $\mathcal E$: - **Which producer each consumer input reads:** $y_{u,c,i}$. Two consumers of one group may read different leaves, e.g. one reads `k = 200` and another `k = 345`. - **Where an unmaterialized candidate runs (#1207):** a candidate with $\ell_v$ = None runs inside each reading context $\kappa$. -- **How much a materialized candidate retains (#1207):** $h_v$, from the selected readers' longest range, row (R). +- **How much a materialized candidate retains (#1207):** $h_v$, from the longest range among the queries its selected consumers serve, row (R). ### Order of decisions @@ -217,7 +217,7 @@ What the tree does not decide, because it is a choice of the MILP over $\mathcal Alternative orders worth comparing: - **$\tau$ first.** Where most sources have no ingestion pipeline, deciding timing at the group level prunes half the tree before $\boldsymbol\theta$ and $\omega$ multiply it. -- **$\boldsymbol\theta$ last among logical dimensions.** $\boldsymbol\theta$ has the most values. Deciding $\omega$ before $\boldsymbol\theta$ visits fewer nodes when many widths are pruned by readers' ranges. +- **$\boldsymbol\theta$ last among logical dimensions.** $\boldsymbol\theta$ has the most values. Deciding $\omega$ before $\boldsymbol\theta$ visits fewer nodes when many widths are pruned by the queries' ranges. - **Requirement-driven (top-down) vs producer-driven (bottom-up).** Top-down enumeration starts from each consumer's $r_{c,i}$ and offers only the values it accepts; bottom-up offers every value and lets $\alpha$ filter $\mathcal E$. Top-down visits fewer nodes but must union the requirements of all consumers of a group, so that sharing between consumers is not lost. ## 3. Data structures and interfaces per stage @@ -302,7 +302,7 @@ pub enum WindowLayout { Tumbling { width_ms: u64, count: u32 } } pub struct LogicalRequirement { pub family: Option, - pub window: Option, // reader's range and step: a width must divide both + pub window: Option, // range and step of the queries reaching the consumer: a width must divide both pub error_cap: Option, // cap_{c,i}: min(validity cap of g_c, pruning cap) } /// α^log_{u,c,i} @@ -323,7 +323,7 @@ Concepts not in the formulation: `origin: OriginOfCandidateForMemoGroup` (Pass1, ### Stage 1, Pass 2: rules and logical edges -Rules add candidates to existing groups, or add groups. A rule never copies a reader. After the last rule, Pass 2 builds the logical edges once. +Rules add candidates to existing groups, or add groups. A rule never copies a consumer. After the last rule, Pass 2 builds the logical edges once. ```rust // crates/logical-asap-optimization/src/pass2 @@ -369,10 +369,14 @@ impl PhysicalProperties { /// r^phys_{c,i}; `new` rejects (IngestionTime, Later). pub struct PhysicalRequirement { - reader_timing: ExecutionTiming, + consumer_timing: ExecutionTiming, reads_from: Evaluation, - reader_schedule: Option, // (I_c, φ_c) + /// (I_κ, φ_κ) of the consumer's execution context, from its query's T_q; + /// None for a non-periodic query. + consumer_schedule: Option, } +/// A periodic evaluation schedule φ + I·ℤ. +pub struct EvaluationSchedule { interval_ms: u64, phase_ms: u64 } pub enum Evaluation { Same, Later } /// α^phys_{u,c,i} pub fn satisfies_physical(producer: &PhysicalProperties, requirement: &PhysicalRequirement) -> bool; @@ -409,7 +413,7 @@ impl PhysicalASAPMemoCandidates { } ``` -Deleted: `Materialization`, `PhysicalCandidate::{timing, materialization, operator_node}`, `producer_candidates_per_input` as the interface, `compatible_timing`, `ReaderConstraint::{IncompatibleOutputs, AllowedOutputs}` (now $\alpha^{\mathrm{phys}}$ through $\sigma$), `may_keep_across_evaluations_per_sole_reader_query`. +Deleted: `Materialization`, `PhysicalCandidate::{timing, materialization, operator_node}`, `producer_candidates_per_input` as the interface, `compatible_timing`, the reader constraints `ReaderConstraint::{IncompatibleOutputs, AllowedOutputs}` (now $\alpha^{\mathrm{phys}}$ through $\sigma$), `may_keep_across_evaluations_per_sole_reader_query`. ### Stage 3: the MILP @@ -440,7 +444,7 @@ fn producer_selection_rows(..); // (B) y_{u,c,i} ≤ x_u fn query_coverage_rows(..); // (C), (D) fn acyclicity_rows(..); // (E) fn utilization_rows(..); // (F) -fn reader_constraint_rows(..); // (J) +fn consumer_constraint_rows(..); // (J): MinimumQueryReaders, HighestSharedQueryMaterialization over reach fn accuracy_rows(..); // (K) Σ λ_{q,u} ε_u a_{q,u} ≤ ε_q, Σ δ_u a_{q,u} ≤ δ_q fn retention_rows(..); // (R) h_v ≥ need_{c,i} · y_{v,c,i} fn latency_rows(..); // (L) @@ -468,7 +472,7 @@ A rule reads the memo and adds to it only through the interfaces of section 3: - `Memo::find_or_add_group(expression)` for a group, so an expression that already exists returns its group (hash-consing); - `Memo::add_candidate(group, CandidateForMemoGroup { expression, logical_properties, input_requirements, fused_groups })` for a candidate $v = (e_v, p^{\log}_v)$ with its $r^{\log}_{v,i}$. -A rule never copies a reader and never edits an existing candidate: every consumer of a group sees a new candidate of that group through $\mathcal E$. Rules match over groups and their candidates' expressions, never over a reader's private operators. The order of the rules below is the order the code runs them; it affects which groups exist when a later rule matches, not which plans are valid. +A rule never copies a consumer and never edits an existing candidate: every consumer of a group sees a new candidate of that group through $\mathcal E$. Rules match over groups and their candidates' expressions, never over a consumer's private operators. The order of the rules below is the order the code runs them; it affects which groups exist when a later rule matches, not which plans are valid. ### Stage 0: hash-consing @@ -484,7 +488,7 @@ Pass 1 adds, for each group a query computes, every way to compute it. Each inte |---|---|---| | Exact | [`per_computation_candidates.rs`](../../../crates/logical-asap-optimization/src/pass1/per_computation_candidates.rs), [`memo_population.rs`](../../../crates/logical-asap-optimization/src/pass1/memo_population.rs) | a candidate with the exact operator; $\boldsymbol\theta_v$ none, $(\varepsilon_v, \delta_v) = (0, 0)$ | | Exact accumulator | same | a state group `ExactAggregate(kind)` over the input group, one candidate; a `FinalizeExactAccumulator` candidate in the result's group reading it | -| Summary, per family the executor supports | same | a state group `SummaryAgg[family]` over the input group, with **one candidate per $\boldsymbol\theta$ in the family's predefined list**, dropping a $\boldsymbol\theta$ whose $\varepsilon_v$ alone exceeds every reading query's $\varepsilon_q$; a `SummaryEstimate[statistic]` candidate in the result's group reading the state group. Each consumer's `error_cap` starts as its $g_c$'s validity cap (none for a linear $g_c$) | +| Summary, per family the executor supports | same | a state group `SummaryAgg[family]` over the input group, with **one candidate per $\boldsymbol\theta$ in the family's predefined list**, dropping a $\boldsymbol\theta$ whose $\varepsilon_v$ alone exceeds the $\varepsilon_q$ of every query that reaches the group; a `SummaryEstimate[statistic]` candidate in the result's group reading the state group. Each consumer's `error_cap` starts as its $g_c$'s validity cap (none for a linear $g_c$) | | Fused build (#1203 2c) | same | a build that computes several targets at once: a candidate of the outer target's state group, with `fused_groups` naming the others | Example: for A and B, Pass 1 adds G2 = `SummaryAgg[KLL](G1)` with candidates `k = 200` and `k = 345`, and G3, G4, G5 = `SummaryEstimate[q](G2)`. A and B's three estimates read one state group because their expressions over G1 hash-cons to one group. @@ -495,7 +499,7 @@ Today, Pass 1 instead sizes one $\boldsymbol\theta$ per `accuracy_budget_parts` | # | Rule | Code | Matches | Adds | Status | |---|---|---|---|---|---| -| 1 | Local window composition (tumbling windows) | `add_local_window_composition` ([`window_candidates.rs`](../../../crates/logical-asap-optimization/src/pass2/memo/window_candidates.rs)) | a summary state group read by window readers whose ranges and steps share a gcd width $w$ | a candidate of the **same state group** with $\omega_v$ = tumbling $(w, n)$, its $r^{\log}$ window requirement on each reader input | today it copies readers per layout (`TumblingWindows` origin); #1160 makes it a candidate of the state group | +| 1 | Local window composition (tumbling windows) | `add_local_window_composition` ([`window_candidates.rs`](../../../crates/logical-asap-optimization/src/pass2/memo/window_candidates.rs)) | a summary state group whose window consumers serve queries whose ranges and steps share a gcd width $w$ | a candidate of the **same state group** with $\omega_v$ = tumbling $(w, n)$, its $r^{\log}$ window requirement on each consumer input | today it copies consumers per layout (`TumblingWindows` origin); #1160 makes it a candidate of the state group | | 2 | Scan widening | `add_scan_label_candidates` ([`scan_labels.rs`](../../../crates/logical-asap-optimization/src/pass2/memo/rules/scan_labels.rs)) | several scan groups of one metric with different label sets | a group for the widened scan (`find_or_add_group`), and in each original scan group a candidate `Project(widened)` | #1203 2a-5 (#1222) | | 3 | Summary inputs | `factor_summary_inputs` ([`summary_inputs.rs`](../../../crates/logical-asap-optimization/src/pass2/memo/rules/summary_inputs.rs)) | a value projection or filter below a summary build | that projection or filter as its own group; the build's candidate reads it | #1203 2a-2 (#1218); after 2a, Pass 1 already makes it a group | | 4 | Top-k reuse | `add_topk_candidates` ([`topk.rs`](../../../crates/logical-asap-optimization/src/pass2/memo/rules/topk.rs)) | top-k groups over one ranking with different `k` | a candidate of each top-k group reading the largest-`k` group, with an exact-input requirement | #1203 2a-3 (#1220) | @@ -503,8 +507,8 @@ Today, Pass 1 instead sizes one $\boldsymbol\theta$ per `accuracy_budget_parts` | 6 | Rollup | `add_rollup_candidates` ([`rollup.rs`](../../../crates/logical-asap-optimization/src/pass2/memo/rules/rollup.rs)) | a coarse aggregate or summary group and a finer group whose state rolls up | a candidate of the coarse group reading the finer group (exact or sketch merge) | #1203 2a-4 (#1221) | | 7 | Window rollup | `add_window_rollup_candidates` (same) | a longer-range group whose windows a shorter-range group's state covers | a candidate of the longer-range group merging the shorter one's windows | #1203 2a-4 (#1221) | | 8 | Summary results | `factor_summary_results` ([`summary_results.rs`](../../../crates/logical-asap-optimization/src/pass2/memo/rules/summary_results.rs)) | a summary readout inside a composite candidate | the readout as its own group | #1203 2a-2 (#1218) makes the readout a group in Pass 1; the rule is then deleted | -| 9 | Shared configurations | `add_state_configuration_candidates` ([`shared_configuration_readers.rs`](../../../crates/logical-asap-optimization/src/pass2/memo/rules/shared_configuration_readers.rs)) | readers that could share a summary at a size or window horizon that covers them | reader copies at the shared size (`SharedBuild` origin) | deleted in #1203 2b: the state group already holds every $\boldsymbol\theta$, and $y$ chooses per reader | -| 10 | Shared summary state groups | `add_shared_summary_state_groups` ([`state_groups.rs`](../../../crates/logical-asap-optimization/src/pass2/memo/state_groups.rs)) | summary builds directly over input groups | per-size concrete state groups with reader copies | deleted in #1203 2b and #1160: one state group per state family | +| 9 | Shared configurations | `add_state_configuration_candidates` ([`shared_configuration_readers.rs`](../../../crates/logical-asap-optimization/src/pass2/memo/rules/shared_configuration_readers.rs)) | consumers that could share a summary at a size or window horizon that covers them | consumer copies at the shared size (`SharedBuild` origin) | deleted in #1203 2b: the state group already holds every $\boldsymbol\theta$, and $y$ chooses per consumer input | +| 10 | Shared summary state groups | `add_shared_summary_state_groups` ([`state_groups.rs`](../../../crates/logical-asap-optimization/src/pass2/memo/state_groups.rs)) | summary builds directly over input groups | per-size concrete state groups with consumer copies | deleted in #1203 2b and #1160: one state group per state family | | — | Logical edges (#1201) | after the last rule | every candidate $c$ and input $i$ | one `LogicalEdge { producer, consumer, input, satisfies_logical_requirement }` per $u \in G_{c,i}$, after tightening each `error_cap` with the pruning cap $\max_q \varepsilon_q / \lambda_{q,c,i}$; $\alpha^{\log}_{u,c,i}$ = `satisfies(p^log_u, r^log_{c,i})`; output `LogicalASAPMemoCandidates { memo, edges }` | #1201 | After #1203 2b, every Pass 2 rule has the same shape: match groups, then `find_or_add_group` for any new intermediate result and `add_candidate` for the new way to compute an existing group. A rule adds no candidate a group already has (same expression and $p^{\log}$). @@ -515,10 +519,10 @@ Stage 2 adds no logical candidates. It expands each logical candidate into physi | Step | Code | Reads | Adds | |---|---|---|---| -| Physical expansion | `enumerate_candidate_configurations` ([`configurations.rs`](../../../crates/physical-asap-optimization/src/catalog/configurations.rs)) | each `CandidateForMemoGroup`, deployment capabilities, readers' $\mathcal T_q$ | one `PhysicalCandidate { logical, properties, input_requirements }` per valid $p^{\mathrm{phys}} = (\tau, \ell, \sigma)$ that the deployment allows (section 2, levels 4–6), with one `PhysicalRequirement` per input: the reader's timing, the evaluation it reads from, and its schedule | +| Physical expansion | `enumerate_candidate_configurations` ([`configurations.rs`](../../../crates/physical-asap-optimization/src/catalog/configurations.rs)) | each `CandidateForMemoGroup`, deployment capabilities, the queries' $\mathcal T_q$ | one `PhysicalCandidate { logical, properties, input_requirements }` per valid $p^{\mathrm{phys}} = (\tau, \ell, \sigma)$ that the deployment allows (section 2, levels 4–6), with one `PhysicalRequirement` per input: the consumer's timing, the evaluation it reads from, and its context's schedule | | Physical edges (#1201) | after expansion | each `LogicalEdge` | one `PhysicalEdge { producer, consumer, input, satisfies_logical_requirement, satisfies_physical_requirement }` per pair of the two logical candidates' physical candidates, with $\alpha^{\mathrm{phys}}$ = `satisfies_physical(p^phys_u, r^phys_{c,i})`; the admissible ones ($\alpha^{\log} = \alpha^{\mathrm{phys}} = 1$) through `admissible_edges_into` / `admissible_edges_out_of` | -Example: v3 (`k = 345`) becomes, among others, (QueryTime, None), (QueryTime, OneEvaluation, Periodic(10 s, 0)) and (IngestionTime, AcrossEvaluations, grid phase 0). G3's estimate at query time has an admissible edge to each of them: same timing, or a query-time reader of an ingestion-time result kept across evaluations. +Example: v3 (`k = 345`) becomes, among others, (QueryTime, None), (QueryTime, OneEvaluation, Periodic(10 s, 0)) and (IngestionTime, AcrossEvaluations, grid phase 0). G3's estimate at query time has an admissible edge to each of them: same timing, or a query-time consumer of an ingestion-time result kept across evaluations. ### Stage 3: no rules From 47fea024efbb71d5637390a8c0907e64c40cfe1d Mon Sep 17 00:00:00 2001 From: zzylol <50204836+zzylol@users.noreply.github.com> Date: Thu, 5 Nov 2026 21:27:57 +0000 Subject: [PATCH 12/44] docs(concepts): section 5, printing every logical DAG a memo group represents Co-Authored-By: Claude Opus 5.5 --- .../memo-groups-properties-requirements.md | 125 ++++++++++++++++++ 1 file changed, 125 insertions(+) diff --git a/docs/design_docs/concepts/memo-groups-properties-requirements.md b/docs/design_docs/concepts/memo-groups-properties-requirements.md index fff471115..8855040e4 100644 --- a/docs/design_docs/concepts/memo-groups-properties-requirements.md +++ b/docs/design_docs/concepts/memo-groups-properties-requirements.md @@ -527,3 +527,128 @@ Example: v3 (`k = 345`) becomes, among others, (QueryTime, None), (QueryTime, On ### Stage 3: no rules Stage 3 adds no candidates and no edges. It reads $\mathcal C$ and the admissible part of $\mathcal E$, creates the MILP variables of section 3 over them, and decodes $\{x = 1\}$ and $\{y = 1\}$ into `DagOfSelectedCandidates`. + +## 5. Printing every logical DAG a memo group represents + +A memo group stores its candidates and, through their expressions, the groups they read. It represents a set of logical DAGs without listing them. This section shows how to list them, using only the section 3 interfaces. Planning never does this: Stage 3's MILP chooses one DAG without listing the others. The listing is for inspection and for checking the MILP on small workloads. + +### What one logical DAG is + +A logical DAG rooted at group $G$ is a set of logical candidates and edges $(u, c, i)$ such that: +1. exactly one candidate of $G$ is in it (the root); +2. every candidate $c$ in it has, for each input $i$, exactly one edge $(u, c, i)$, with $u$ a candidate of $G_{c,i}$ and $\alpha^{\log}_{u,c,i} = 1$ (rows (A) and (G)); +3. every candidate in it is the root or the producer of some edge (row (F)); +4. the edges form no cycle (row (E)). + +A candidate is one node however many edges read it: two consumers that choose the same producer share it, which is what $x_u \in \{0, 1\}$ says. Two consumers of one group may choose different candidates of it. The listing is therefore a choice of one producer per consumer input $(c, i)$, not one candidate per group. + +### Pseudocode + +```rust +/// One logical DAG: its nodes and, for each node's inputs, the chosen producer. +struct LogicalDag { + root: CandidateForMemoGroupId, + /// For each node c in the DAG, one producer per input i, in input order. + producers: BTreeMap>, +} + +/// Every logical DAG rooted at `root_group`, each exactly once. +fn logical_dags(memo: &Memo, root_group: MemoGroupId) -> Vec { + let mut all = Vec::new(); + for root in candidates_of(memo, root_group) { + let dag = LogicalDag { root, producers: BTreeMap::new() }; + expand(memo, dag, VecDeque::from([root]), &mut all); + } + all +} + +/// Choose producers for the inputs of every node in `pending`, in FIFO order. +/// Each DAG is reached by exactly one sequence of choices, so none repeats. +fn expand(memo: &Memo, dag: LogicalDag, mut pending: VecDeque, all: &mut Vec) { + let Some(consumer) = pending.pop_front() else { + all.push(dag); // every input of every node has a producer + return; + }; + let candidate = memo.candidate(consumer); + let inputs: Vec = candidate.expression.operator.children().copied().collect(); + // Every way to choose one producer per input of `consumer`: the product over i + // of the admissible producers in G_{c,i}. + for choice in cartesian_product(inputs.iter().enumerate().map(|(i, &group)| { + candidates_of(memo, group) + .filter(|&u| satisfies(&memo.candidate(u).logical_properties, // α^log = 1 + &candidate.input_requirements[i])) + .collect::>() + })) { + // Row (E): a producer that already reaches `consumer` would close a cycle. + if choice.iter().any(|&u| reaches(&dag, u, consumer)) { + continue; + } + let mut next = dag.clone(); + next.producers.insert(consumer, choice.clone()); + let mut next_pending = pending.clone(); + for &u in &choice { + // A producer already in the DAG is shared, not expanded again. + if !next.producers.contains_key(&u) && !next_pending.contains(&u) && u != next.root { + next_pending.push_back(u); + } + } + expand(memo, next, next_pending, all); + } +} + +fn candidates_of(memo: &Memo, group: MemoGroupId) -> impl Iterator + '_ { + (0..memo.candidates[group.0 as usize].len()).map(move |index| CandidateForMemoGroupId { group, index }) +} + +/// Whether `from` reaches `to` through the producers chosen so far. +fn reaches(dag: &LogicalDag, from: CandidateForMemoGroupId, to: CandidateForMemoGroupId) -> bool; + +/// Print one DAG as an indented tree; a node read a second time is printed as a +/// reference to its first occurrence, so sharing is visible. +fn print(memo: &Memo, dag: &LogicalDag) { + fn node(memo: &Memo, dag: &LogicalDag, c: CandidateForMemoGroupId, depth: usize, seen: &mut HashSet) { + let candidate = memo.candidate(c); + if !seen.insert(c) { + println!("{}↑ G{}[{}] (shared)", " ".repeat(depth), c.group.0, c.index); + return; + } + println!("{}G{}[{}] {} {:?}", " ".repeat(depth), c.group.0, c.index, + candidate.expression.operator.name(), candidate.logical_properties); + for &u in dag.producers.get(&c).into_iter().flatten() { + node(memo, dag, u, depth + 1, seen); + } + } + node(memo, dag, dag.root, 0, &mut HashSet::new()); +} +``` + +For a workload, list from every query root at once: start `pending` with one root candidate per $G_q$ (rows (C) and (D)), taking the product over queries of their root candidates. A candidate two queries reach is one node, as above. + +### Example + +For the example's query A (root G3) and query B (root G6), with no error caps applied: + +``` +A: 2 DAGs B: 4 DAGs (G4 and G5 choose independently) +G3[0] SummaryEstimate q=0.95 G6[0] Divide + G2[0] SummaryAgg KLL θ={k=200} G4[0] SummaryEstimate q=0.99 + G1[0] Scan G2[0] SummaryAgg KLL θ={k=200} +G3[0] SummaryEstimate q=0.95 G1[0] Scan + G2[1] SummaryAgg KLL θ={k=345} G5[0] SummaryEstimate q=0.50 + G1[0] Scan ↑ G2[0] (shared) + … and (k=200, k=345), (k=345, k=200), (k=345, k=345) +``` + +Listed from both roots at once there are $2 \times 4 = 8$ DAGs. In the one where all three estimates choose `k = 345`, `G2[1]` is a single node read by three consumers, so its cost is counted once. Row (K) later accepts only the DAGs that meet each query's target: for B, only (k=345, k=345). Error caps (section 1) remove edges before the listing, so a capped producer never appears. + +### Size + +Without sharing, the number of DAGs rooted at $G$ satisfies + +$$N(G) = \sum_{v \in G} \prod_{i} \sum_{u \in G_{v,i},\ \alpha^{\log}_{u,v,i} = 1} N(u), \qquad N(u) = \prod_i \sum_{w} N(w) \text{ for } u\text{'s own inputs},$$ + +which grows as a product over the DAG's depth. Sharing only identifies DAGs that choose the same candidate twice; it does not reduce the count much. This is why only small workloads are listed, and why Stage 3 chooses with the MILP instead. + +### Physical DAGs + +The same procedure lists physical DAGs from Stage 2's output. Use `PhysicalCandidate`s instead of logical candidates, and `admissible_edges_into(c, i)` (which already applies $\alpha^{\log}$ and $\alpha^{\mathrm{phys}}$) instead of `candidates_of` with `satisfies`. A physical candidate with $\ell = $ None that two consumers in different contexts $\kappa$ choose runs once per context; in the listing it is still one node, and the contexts are a Stage 3 choice (#1207). From b890866e35fae49b327d67e6fdad511d921a15f7 Mon Sep 17 00:00:00 2001 From: zzylol <50204836+zzylol@users.noreply.github.com> Date: Thu, 5 Nov 2026 21:28:12 +0000 Subject: [PATCH 13/44] docs(concepts): section 5 size bound; reaches covers self-reads Co-Authored-By: Claude Opus 5.5 --- .../concepts/memo-groups-properties-requirements.md | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/docs/design_docs/concepts/memo-groups-properties-requirements.md b/docs/design_docs/concepts/memo-groups-properties-requirements.md index 8855040e4..fe138037a 100644 --- a/docs/design_docs/concepts/memo-groups-properties-requirements.md +++ b/docs/design_docs/concepts/memo-groups-properties-requirements.md @@ -600,7 +600,8 @@ fn candidates_of(memo: &Memo, group: MemoGroupId) -> impl Iterator bool; /// Print one DAG as an indented tree; a node read a second time is printed as a @@ -643,11 +644,11 @@ Listed from both roots at once there are $2 \times 4 = 8$ DAGs. In the one where ### Size -Without sharing, the number of DAGs rooted at $G$ satisfies +Counting each consumer input's choice independently gives an upper bound on the number of DAGs rooted at $G$: -$$N(G) = \sum_{v \in G} \prod_{i} \sum_{u \in G_{v,i},\ \alpha^{\log}_{u,v,i} = 1} N(u), \qquad N(u) = \prod_i \sum_{w} N(w) \text{ for } u\text{'s own inputs},$$ +$$N(G) = \sum_{v \in G} N(v), \qquad N(v) = \prod_{i} \sum_{u \in P_{v,i},\ \alpha^{\log}_{u,v,i} = 1} N(u),$$ -which grows as a product over the DAG's depth. Sharing only identifies DAGs that choose the same candidate twice; it does not reduce the count much. This is why only small workloads are listed, and why Stage 3 chooses with the MILP instead. +with $N(v) = 1$ for a candidate with no inputs. It is an upper bound because a shared producer is expanded once in a DAG, where the formula expands it once per consumer, and because cyclic choices are dropped. $N$ multiplies along the depth of the memo, which is why only small workloads are listed, and why Stage 3 chooses with the MILP instead. ### Physical DAGs From 21efef04ec68891ec4e920e57635f535d4eb0b03 Mon Sep 17 00:00:00 2001 From: zzylol <50204836+zzylol@users.noreply.github.com> Date: Thu, 5 Nov 2026 22:12:52 +0000 Subject: [PATCH 14/44] docs(concepts): logical properties and requirement mirror the physical interfaces; satisfies_logical Co-Authored-By: Claude Opus 5.5 --- .../memo-groups-properties-requirements.md | 63 ++++++++++++++----- 1 file changed, 47 insertions(+), 16 deletions(-) diff --git a/docs/design_docs/concepts/memo-groups-properties-requirements.md b/docs/design_docs/concepts/memo-groups-properties-requirements.md index fe138037a..33c5df6f2 100644 --- a/docs/design_docs/concepts/memo-groups-properties-requirements.md +++ b/docs/design_docs/concepts/memo-groups-properties-requirements.md @@ -288,34 +288,57 @@ pub struct CandidateForMemoGroup { /// e_v = op_v(G_{v,1}, …, G_{v,n}): one operator, its children are groups. pub struct CandidateExpression { pub operator: Operator } +/// p_v^log = (θ_v, ω_v); `new` rejects invalid combinations (parameters of +/// another family, a window layout on an exact operator that cannot be windowed). pub struct LogicalProperties { representation: Representation, // family and θ, from the predefined list window_layout: Option, // ω_v } impl LogicalProperties { pub fn new(representation: Representation, window_layout: Option) -> Result; - /// (ε_v, δ_v) for `statistic`, derived through `local_guarantee`. - pub fn error_bound(&self, statistic: &SketchStatistic) -> Option; + pub fn representation(&self) -> &Representation; + pub fn window_layout(&self) -> Option<&WindowLayout>; } pub enum Representation { Exact(ExactKind), Summary(SketchKind) } pub enum WindowLayout { Tumbling { width_ms: u64, count: u32 } } +/// r^log_{c,i}: one optional constraint per dimension; None accepts any value. +/// `new` rejects invalid combinations (a cap that is not positive and finite, +/// a cap without the statistic it applies to, a window with zero range or step). pub struct LogicalRequirement { - pub family: Option, - pub window: Option, // range and step of the queries reaching the consumer: a width must divide both - pub error_cap: Option, // cap_{c,i}: min(validity cap of g_c, pruning cap) + family: Option, + /// Range and step of the queries reaching the consumer: a width must divide both. + window: Option, + /// The statistic the consumer reads from this input; the producer's ε is + /// derived for it. + statistic: Option, + /// cap_{c,i} = min(validity cap of g_c, pruning cap max_q ε_q / λ_{q,c,i}). + error_cap: Option, +} +impl LogicalRequirement { + pub fn new(family: Option, window: Option, + statistic: Option, error_cap: Option) -> Result; + pub fn family(&self) -> Option; + pub fn window(&self) -> Option<&WindowRequirement>; + pub fn statistic(&self) -> Option<&SketchStatistic>; + pub fn error_cap(&self) -> Option; + /// g_c of a consumer: the propagation factor λ_{c,i} per input, read from the + /// operator and its arguments, and the validity cap (None when g_c is linear). + /// Sets each input's first `error_cap`; Stage 3 composes the factors into λ_{q,c,i}. + pub fn error_propagation(consumer: &Operator) -> ErrorPropagation; } -/// α^log_{u,c,i} -pub fn satisfies(producer: &LogicalProperties, requirement: &LogicalRequirement) -> bool; - -/// g_c as propagation factors λ_{c,i}, read from the consumer's operator and -/// arguments, and the validity cap (None when g_c is linear). /// Approved 2026-11-05. pub struct ErrorPropagation { pub factor_per_input: Vec, pub validity_cap: Option } -pub fn error_propagation(consumer: &Operator) -> ErrorPropagation; + +/// α^log_{u,c,i}: one check per dimension, joined by "and". The error check +/// derives the producer's (ε_u, δ_u) for `requirement.statistic()` through +/// `estimators::local_guarantee` and compares ε_u with `error_cap`. +pub fn satisfies_logical(producer: &LogicalProperties, requirement: &LogicalRequirement) -> bool; ``` -The validity cap is known when the consumer is created. The pruning cap needs $\lambda_{q,c,i}$ and the queries that may reach $c$, so it is filled in when Pass 2 builds the logical edges. +The logical and physical sides have the same shape: a properties struct and a requirement struct, each with private fields, a checked `new`, one accessor per dimension, and one `satisfies_*` function. Invalid values of both logical structs are reported through one error type, `InvalidLogicalProperties`, as `InvalidPhysicalProperties` does for the physical side. + +The error bound is not a method of `LogicalProperties`: it depends on the statistic the consumer reads, so `satisfies_logical` derives it from the requirement's statistic. The validity cap is known when the consumer is created (`LogicalRequirement::error_propagation`). The pruning cap needs $\lambda_{q,c,i}$ and the queries that may reach $c$, so Pass 2 builds each input's final requirement with `LogicalRequirement::new` when it builds the logical edges. Deleted: `root_node` with private nodes, `input_groups` (the expression's children), `accuracy_budget_parts`, the `SharedBuild` and `SummaryResult` origins. @@ -367,7 +390,8 @@ impl PhysicalProperties { pub fn is_materialized(&self) -> bool; // ℓ ≠ None } -/// r^phys_{c,i}; `new` rejects (IngestionTime, Later). +/// r^phys_{c,i}; `new` rejects (IngestionTime, Later), reported through +/// `InvalidPhysicalProperties` (one error type per side). pub struct PhysicalRequirement { consumer_timing: ExecutionTiming, reads_from: Evaluation, @@ -375,6 +399,13 @@ pub struct PhysicalRequirement { /// None for a non-periodic query. consumer_schedule: Option, } +impl PhysicalRequirement { + pub fn new(consumer_timing: ExecutionTiming, reads_from: Evaluation, + consumer_schedule: Option) -> Result; + pub fn consumer_timing(&self) -> ExecutionTiming; + pub fn reads_from(&self) -> Evaluation; + pub fn consumer_schedule(&self) -> Option<&EvaluationSchedule>; +} /// A periodic evaluation schedule φ + I·ℤ. pub struct EvaluationSchedule { interval_ms: u64, phase_ms: u64 } pub enum Evaluation { Same, Later } @@ -509,7 +540,7 @@ Today, Pass 1 instead sizes one $\boldsymbol\theta$ per `accuracy_budget_parts` | 8 | Summary results | `factor_summary_results` ([`summary_results.rs`](../../../crates/logical-asap-optimization/src/pass2/memo/rules/summary_results.rs)) | a summary readout inside a composite candidate | the readout as its own group | #1203 2a-2 (#1218) makes the readout a group in Pass 1; the rule is then deleted | | 9 | Shared configurations | `add_state_configuration_candidates` ([`shared_configuration_readers.rs`](../../../crates/logical-asap-optimization/src/pass2/memo/rules/shared_configuration_readers.rs)) | consumers that could share a summary at a size or window horizon that covers them | consumer copies at the shared size (`SharedBuild` origin) | deleted in #1203 2b: the state group already holds every $\boldsymbol\theta$, and $y$ chooses per consumer input | | 10 | Shared summary state groups | `add_shared_summary_state_groups` ([`state_groups.rs`](../../../crates/logical-asap-optimization/src/pass2/memo/state_groups.rs)) | summary builds directly over input groups | per-size concrete state groups with consumer copies | deleted in #1203 2b and #1160: one state group per state family | -| — | Logical edges (#1201) | after the last rule | every candidate $c$ and input $i$ | one `LogicalEdge { producer, consumer, input, satisfies_logical_requirement }` per $u \in G_{c,i}$, after tightening each `error_cap` with the pruning cap $\max_q \varepsilon_q / \lambda_{q,c,i}$; $\alpha^{\log}_{u,c,i}$ = `satisfies(p^log_u, r^log_{c,i})`; output `LogicalASAPMemoCandidates { memo, edges }` | #1201 | +| — | Logical edges (#1201) | after the last rule | every candidate $c$ and input $i$ | one `LogicalEdge { producer, consumer, input, satisfies_logical_requirement }` per $u \in G_{c,i}$, after tightening each `error_cap` with the pruning cap $\max_q \varepsilon_q / \lambda_{q,c,i}$; $\alpha^{\log}_{u,c,i}$ = `satisfies_logical(p^log_u, r^log_{c,i})`; output `LogicalASAPMemoCandidates { memo, edges }` | #1201 | After #1203 2b, every Pass 2 rule has the same shape: match groups, then `find_or_add_group` for any new intermediate result and `add_candidate` for the new way to compute an existing group. A rule adds no candidate a group already has (same expression and $p^{\log}$). @@ -575,7 +606,7 @@ fn expand(memo: &Memo, dag: LogicalDag, mut pending: VecDeque>() })) { @@ -652,4 +683,4 @@ with $N(v) = 1$ for a candidate with no inputs. It is an upper bound because a s ### Physical DAGs -The same procedure lists physical DAGs from Stage 2's output. Use `PhysicalCandidate`s instead of logical candidates, and `admissible_edges_into(c, i)` (which already applies $\alpha^{\log}$ and $\alpha^{\mathrm{phys}}$) instead of `candidates_of` with `satisfies`. A physical candidate with $\ell = $ None that two consumers in different contexts $\kappa$ choose runs once per context; in the listing it is still one node, and the contexts are a Stage 3 choice (#1207). +The same procedure lists physical DAGs from Stage 2's output. Use `PhysicalCandidate`s instead of logical candidates, and `admissible_edges_into(c, i)` (which already applies $\alpha^{\log}$ and $\alpha^{\mathrm{phys}}$) instead of `candidates_of` with `satisfies_logical`. A physical candidate with $\ell = $ None that two consumers in different contexts $\kappa$ choose runs once per context; in the listing it is still one node, and the contexts are a Stage 3 choice (#1207). From 2004cdac76afff006ecead217e875643a6e07633 Mon Sep 17 00:00:00 2001 From: zzylol <50204836+zzylol@users.noreply.github.com> Date: Thu, 5 Nov 2026 22:24:42 +0000 Subject: [PATCH 15/44] docs(concepts): drop fields other things fix; LogicalProperties {parameters, window_layout}, LogicalRequirement {parameter_floor}, r^phys = f(p^phys_c); at-a-glance tables Co-Authored-By: Claude Opus 5.5 --- .../memo-groups-properties-requirements.md | 174 ++++++++++-------- 1 file changed, 99 insertions(+), 75 deletions(-) diff --git a/docs/design_docs/concepts/memo-groups-properties-requirements.md b/docs/design_docs/concepts/memo-groups-properties-requirements.md index 33c5df6f2..3031eea50 100644 --- a/docs/design_docs/concepts/memo-groups-properties-requirements.md +++ b/docs/design_docs/concepts/memo-groups-properties-requirements.md @@ -19,7 +19,7 @@ This page defines the memo's data structures in the terms of the optimization pr | $c \in G_c$, $u \in G_u$ | consumer candidate and its group; producer candidate and its group. This page says *consumer* for $c$ and *query* $q$ (or its execution context $\kappa$) for who needs a result on a schedule; it does not use "reader" | | $\mathcal I(c)$ | input indices of consumer $c$ | | $G_{c,i}$ | memo group required by input $i$ of consumer $c$ | -| $r_{c,i} = (r^{\log}_{c,i}, r^{\mathrm{phys}}_{c,i})$ | property requirement of input $i$ of consumer $c$ | +| $r_{c,i} = (r^{\log}_{c,i}, r^{\mathrm{phys}}_{c,i})$ | property requirement of input $i$ of consumer $c$; $r^{\mathrm{phys}}_{c,i}$ is a function of $p^{\mathrm{phys}}_c$ | | $p(u) \succeq r_{c,i}$ | producer $u$'s properties satisfy the requirement: $p^{\log}(u) \succeq r^{\log}_{c,i} \land p^{\mathrm{phys}}(u) \succeq r^{\mathrm{phys}}_{c,i}$ | | $\alpha^{\log}_{u,c,i}$, $\alpha^{\mathrm{phys}}_{u,c,i}$ | fixed indicators of $p^{\log}_u \succeq r^{\log}_{c,i}$ and $p^{\mathrm{phys}}_u \succeq r^{\mathrm{phys}}_{c,i}$ | | $\mathcal C$ | physical candidates, each with a fixed parameter configuration, timing and lifetime | @@ -32,7 +32,7 @@ This page defines the memo's data structures in the terms of the optimization pr | **(#1207)** $\kappa$ | execution context of a candidate with $\ell_v$ = None: a query evaluation $\kappa_q$, an ingestion pipeline $I_r$, or a materialized candidate's computation $\kappa_m$ | | $g_c$ | error propagation function of consumer $c$: $\varepsilon_c \le g_c(\varepsilon_{c,1}, \dots, \varepsilon_{c,n})$ | | $\lambda_{q,c,i}$ | propagation factor of input $i$ of $c$ to query $q$'s result, read off $q$'s expression | -| $\text{cap}_{c,i}$ | error cap of input $i$ of $c$: the smaller of $g_c$'s validity cap and the pruning cap | +| $\text{cap}_{c,i}$, $\boldsymbol\theta^{\min}_{c,i}$ | error cap of input $i$ of $c$ (the smaller of $g_c$'s validity cap and the pruning cap), and the parameter floor $r^{\log}_{c,i}$ stores: the smallest listed $\boldsymbol\theta$ whose $\varepsilon$ is at most the cap | | **(#1207)** (K), (R), (L), budgets | accuracy at each query root, $\sum_u \lambda_{q,u}\varepsilon_u a_{q,u} \le \varepsilon_q$ and $\sum_u \delta_u a_{q,u} \le \delta_q$; retention $h_v$; latency; resource budgets | ## Example @@ -69,7 +69,7 @@ What is computed is the expression $e_v$, not a property: the operator, its argu | Dimension | Values | Part of | Code | |---|---|---|---| | Parameter configuration $\boldsymbol\theta_v$ | per family, from a predefined list: KLL `k`; DDSketch `alpha`; CMS / CountSketch `width × depth`; CMS / CountSketch with heap `width × depth × heap_size`; UnivMon `heap_size × rows × cols × layers`; HLL `precision`; KMV `k`. None for an exact operator | $p^{\log}_v$ | `SketchParams` inside `SketchKind` ([`sketch.rs`](../../../crates/types/src/ir/schema/sketch.rs)), sized by `size_params` per `accuracy_budget_parts` today | -| **(#1207)** Window layout $\omega_v$ | none; or tumbling windows of width $w$ and count $n$, with $w$ from the gcd closure of the ranges and steps of the queries that reach the group. Later: sliding windows by merge or by subtraction | $p^{\log}_v$ | `SummaryTumblingWindowChain` arguments, added by Pass 2 ([`window_candidates.rs`](../../../crates/logical-asap-optimization/src/pass2/memo/window_candidates.rs)) | +| **(#1207)** Window layout $\omega_v$ | none; or tumbling windows of width $w$, from the gcd closure of the ranges and steps of the queries that reach the group. The window count is not a property: it is the group's query range divided by $w$, and how many windows to retain is $h_v$ (Stage 3). Later: sliding windows by merge or by subtraction | $p^{\log}_v$ | `SummaryTumblingWindowChain` arguments, added by Pass 2 ([`window_candidates.rs`](../../../crates/logical-asap-optimization/src/pass2/memo/window_candidates.rs)) | | Execution timing $\tau_v$ | IngestionTime, QueryTime | $p^{\mathrm{phys}}_v$ | `ExecutionTiming` ([`execution_data_state.rs`](../../../crates/types/src/ir/properties/execution_data_state.rs)) | | Lifetime $\ell_v$ | None, OneEvaluation, AcrossEvaluations; **(#1207)** a materialized lifetime carries its medium, Memory or Disk | $p^{\mathrm{phys}}_v$ | `Materialization` on main; `Lifetime`, `Medium` in #1214 ([`physical/`](../../../crates/types/src/physical)) | | **(#1207)** Schedule $\sigma_v$ | only when $\ell_v \ne$ None: Periodic (interval, phase) for OneEvaluation; tumbling-window grid phase for AcrossEvaluations | $p^{\mathrm{phys}}_v$ | schedule groups `z[m, G]` in the MILP today ([`cost_terms.rs`](../../../crates/plan-selection/src/milp/cost_terms.rs)); a property in #1200 PR 2 | @@ -93,7 +93,7 @@ Valid $(\tau, \ell)$: (QueryTime, None), (IngestionTime, None), (QueryTime, OneE ### Property requirements -Each input $i$ of consumer $c$ names a producer group $G_{c,i}$ and a property requirement $r_{c,i} = (r^{\log}_{c,i}, r^{\mathrm{phys}}_{c,i})$. Logical satisfaction checks configuration requirements; physical satisfaction checks whether the producer's timing and lifetime jointly support the required access. For a monotone size parameter, $\succeq$ reduces to numerical dominance, e.g. $k_u \ge k_{\text{required}}$ for KLL. +Each input $i$ of consumer $c$ names a producer group $G_{c,i}$ and a property requirement $r_{c,i} = (r^{\log}_{c,i}, r^{\mathrm{phys}}_{c,i})$. A requirement holds only what neither $G_{c,i}$'s identity nor the consumer's own properties already fix (decided 2026-11-05): the family and the query window are in $G_{c,i}$'s identity, and $r^{\mathrm{phys}}_{c,i}$ is a function of the consumer's own $p^{\mathrm{phys}}_c$, not a stored value. Logical satisfaction checks configuration requirements; physical satisfaction checks whether the producer's timing and lifetime jointly support the required access. For a monotone size parameter, $\succeq$ reduces to numerical dominance, e.g. $k_u \ge k_{\text{required}}$ for KLL. The workload's accuracy target $A_q$ is not an input requirement **(#1207)**: a per-input bound would fix each input's share of the query's budget. Row (K) bounds each query's composed error at its root instead. What stays in $r^{\log}$ is what depends only on the consumer. @@ -133,11 +133,9 @@ For a monotone size parameter, $\varepsilon_u \le \text{cap}_{c,i}$ is a minimum | Requirement | Stated by | Values | Constrains | Checked by | |---|---|---|---|---| | Accuracy target $A_q$ | the workload, per query | metric: absolute (in the value's units, or times the stream's L1 or L2 norm), relative, rank; bound $\varepsilon_q$; failure probability $\delta_q$; scope: per value, whole result, or groups above a share of the records | $\boldsymbol\theta_u$ of every approximate candidate $q$ depends on, through $(\varepsilon_u, \delta_u)$ | row (K) **(#1207)** | -| Error cap $\text{cap}_{c,i}$, in $r^{\log}_{c,i}$ | every consumer, per input, from $g_c$ and the targets of the queries that may reach it | the smaller of the validity cap (non-linear $g_c$ only) and the pruning cap; for a monotone size parameter, a minimum size $k_u \ge k_{\text{required}}$ | $\boldsymbol\theta_u$ | $\alpha^{\log}$, row (G): validity cap for correctness of (K), pruning cap for efficiency | -| Window, in $r^{\log}_{c,i}$ **(#1207)** | a consumer that reads windows, per input | the range and step of the queries that reach the consumer | $\omega_u$: the width must divide both | $\alpha^{\log}$, row (G) | -| Summary family, in $r^{\log}_{c,i}$ | a consumer that reads summary state | a family | usually already fixed by $G_{c,i}$, whose $\operatorname{op}$ names the family | $\alpha^{\log}$, row (G) | -| Consumer timing and evaluation, in $r^{\mathrm{phys}}_{c,i}$ | the consumer, per input | when it reads (ingestion or query time) and from which evaluation (same or later) | $\tau_u$, $\ell_u$ | $\alpha^{\mathrm{phys}}$, row (H) | -| Consumer schedule, in $r^{\mathrm{phys}}_{c,i}$ **(#1207)** | the consumer's execution context $\kappa$, from the $\mathcal T_q$ of its query | the context's interval and phase $(I_\kappa, \varphi_\kappa)$; the producer must deliver on them, $T_\kappa \subseteq T_{\sigma_u}$ | $\sigma_u$ | $\alpha^{\mathrm{phys}}$, row (H) | +| Parameter floor $\boldsymbol\theta^{\min}_{c,i}$, the only field of $r^{\log}_{c,i}$ | every consumer, per input, from $g_c$ and the targets of the queries that may reach it | the smallest $\boldsymbol\theta$ in the family's predefined list whose $\varepsilon$, for the statistic $c$ reads, is at most $\text{cap}_{c,i}$ = min(validity cap, pruning cap); e.g. $k_u \ge k_{\text{required}}$ | $\boldsymbol\theta_u$ | $\alpha^{\log}$, row (G): $\boldsymbol\theta_u \succeq \boldsymbol\theta^{\min}_{c,i}$ | +| Consumer timing, $r^{\mathrm{phys}}_{c,i}$ from $\tau_c$ | the consumer's own $p^{\mathrm{phys}}_c$ | when $c$ runs: ingestion or query time | $\tau_u$, $\ell_u$ | $\alpha^{\mathrm{phys}}$, row (H) | +| Consumer schedule, $r^{\mathrm{phys}}_{c,i}$ from $\sigma_c$ or $\kappa$ **(#1207)** | a materialized consumer's own $\sigma_c$; for $\ell_c$ = None, the times $T_\kappa$ of each context $\kappa$ it runs in (from its query's $\mathcal T_q$) | interval and phase | $\sigma_u$: the producer delivers on those times | $\alpha^{\mathrm{phys}}$, row (H): in Stage 2 for a materialized consumer; per context in Stage 3 for $\ell_c$ = None | | Deployment capability | the deployment and data | the source has an ingestion pipeline; the executor supports the family; query-time results may be kept across evaluations | $\tau$, $\operatorname{op}$, (QueryTime, AcrossEvaluations) | candidate generation: such candidates are not in $\mathcal C$ | | Latency, resource budgets **(#1207)** | the workload | per-query latency bound; memory and CPU budgets | the whole plan | rows (L) and budgets | @@ -146,15 +144,55 @@ Accuracy metrics that may be added later, when a primitive needs them: top-k rec How requirements map to properties: ``` -A_q (workload) ──► θ_u of every approximate input (through ε_u, δ_u; row (K)) -error cap r^log ──► θ_u (validity cap from g_c; pruning cap from ε_q / λ_{q,c,i}) -window r^log ──► ω_u -consumer timing r^phys ──► τ_u, ℓ_u -consumer schedule r^phys ──► σ_u -deployment ──► τ, op, ℓ (not generated) -latency, budgets ──► every dimension, through w_v +A_q (workload) ──► θ_u of every approximate input (through ε_u, δ_u; row (K)) +parameter floor θ^min r^log ──► θ_u (from cap = min(validity cap of g_c, pruning cap ε_q / λ_{q,c,i})) +τ_c r^phys = f(p^phys_c) ──► τ_u, ℓ_u +σ_c or T_κ r^phys = f(p^phys_c) ──► σ_u +G_{c,i}'s identity (family, query window) ──► op_u, ω_u (by construction: no requirement) +deployment ──► τ, op, ℓ (not generated) +latency, budgets ──► every dimension, through w_v ``` +### Properties and requirements at a glance (decided 2026-11-05) + +A field is kept only when it is a real choice: anything the expression, $G_{c,i}$'s identity or the consumer's own properties already fix is not repeated. + +| Struct | Field | Values | Symbol | `new` rejects | +|---|---|---|---|---| +| `LogicalProperties` ($p^{\log}_v$) | `parameters` | `Option`: θ from the family's predefined list; None for an exact operator | $\boldsymbol\theta_v$ | parameters of a family other than the operator's; parameters on an exact operator | +| | `window_layout` | `None`, or `Tumbling { width_ms }` (later: sliding by merge or subtraction) | $\omega_v$ | a window on an operator that cannot be windowed; width 0 | +| `LogicalRequirement` ($r^{\log}_{c,i}$) | `parameter_floor` | `Option`: the smallest θ in the list whose $\varepsilon$ ≤ $\text{cap}_{c,i}$ | $\boldsymbol\theta^{\min}_{c,i}$ | a floor not in the family's predefined list | +| `PhysicalProperties` ($p^{\mathrm{phys}}_v$) | `timing` | IngestionTime, QueryTime | $\tau_v$ | (IngestionTime, OneEvaluation) | +| | `lifetime` | None, OneEvaluation(Medium), AcrossEvaluations(Medium); Medium ∈ {Memory, Disk} | $\ell_v$ | | +| | `schedule` | None; Periodic { interval, phase }; TumblingWindowGrid { phase } | $\sigma_v$ | the schedule combinations listed above | +| ($r^{\mathrm{phys}}_{c,i}$) | none | a function of the consumer's $p^{\mathrm{phys}}_c$: its $\tau_c$, and its $\sigma_c$ or each context's $T_\kappa$ | | | + +Not fields, because something else already fixes them: the summary family ($\operatorname{op}_v$'s argument, part of the group identity); the window count (query range ÷ width; retention is $h_v$); a family or window requirement on an input ($G_{c,i}$'s identity, and the schedule check); the statistic a consumer reads (its own operator, used once to compute $\boldsymbol\theta^{\min}$); which evaluation produced the output (reuse across evaluations is the producer's $\ell$, a cost choice). + +**α^log** = `satisfies_logical(p^log_u, r^log_{c,i})`: $\boldsymbol\theta_u \succeq \boldsymbol\theta^{\min}_{c,i}$, compared per family on the parameters that set $\varepsilon$ (KLL `k` ≥; CMS / CountSketch `width` ≥; DDSketch `alpha` ≤; HLL `precision` ≥; KMV `k` ≥; UnivMon and the heap variants on each sizing parameter). No floor, or an exact producer, always satisfies. + +**α^phys** = `satisfies_physical(p^phys_u, p^phys_c)`, timing and lifetime: + +| Consumer $\tau_c$ ↓ / producer $(\tau_u, \ell_u)$ → | (QT, None) | (QT, OneEval) | (QT, AcrossEval) | (IT, None) | (IT, AcrossEval) | +|---|---|---|---|---|---| +| QueryTime | ✅ | ✅ | ✅ | ❌ an unmaterialized ingestion-time output is gone by query time | ✅ | +| IngestionTime | ❌ | ❌ | ❌ | ✅ | ✅ | + +and schedule, for a producer with $\ell_u \ne$ None, against the consumer's times (its own $\sigma_c$ in Stage 2 when $\ell_c \ne$ None; each context's $T_\kappa$ in Stage 3 when $\ell_c$ = None): + +| Producer $\sigma_u$ | Delivers on consumer times $(I, \varphi)$ when | +|---|---| +| none ($\ell_u$ = None) | always: it runs inside the consumer's context, chosen by Stage 3 ($x_{\kappa,v}$) | +| Periodic $(I_\sigma, \varphi_\sigma)$ | $I_\sigma \mid I$ and $\varphi \equiv \varphi_\sigma \pmod{I_\sigma}$; a non-periodic consumer is never served | +| TumblingWindowGrid $\varphi_g$, width $w$ | $w \mid I$ and $\varphi \equiv \varphi_g \pmod w$; a non-periodic consumer is never served | + +| Struct | Set by | When | +|---|---|---| +| `LogicalProperties` | Pass 1 (one candidate per θ in the list); Pass 2 rules (window layouts) | when the candidate is created | +| `LogicalRequirement` | the consumer: floor from the validity cap of `LogicalRequirement::error_propagation(consumer)` | when the consumer is created; Pass 2 tightens it with the pruning cap when it builds the logical edges | +| `PhysicalProperties` | Stage 2 physical expansion (every valid $(\tau, \ell, \sigma)$) | Stage 2 | +| $r^{\mathrm{phys}}_{c,i}$ | derived from $p^{\mathrm{phys}}_c$ | checked in Stage 2 (timing, lifetime; schedule of a materialized consumer) and Stage 3 (schedule per context) | + ## 2. Decision tree of properties Each memo group's candidates are the leaves of a tree. The first level chooses the expression $e_v$; every later level decides one property dimension. A path from the root to a leaf is one physical candidate in $\mathcal C$. @@ -288,57 +326,51 @@ pub struct CandidateForMemoGroup { /// e_v = op_v(G_{v,1}, …, G_{v,n}): one operator, its children are groups. pub struct CandidateExpression { pub operator: Operator } -/// p_v^log = (θ_v, ω_v); `new` rejects invalid combinations (parameters of -/// another family, a window layout on an exact operator that cannot be windowed). +/// p_v^log = (θ_v, ω_v). The family is the operator's argument, not a property. +/// `new` rejects parameters of another family (or on an exact operator) and a +/// window on an operator that cannot be windowed. pub struct LogicalProperties { - representation: Representation, // family and θ, from the predefined list - window_layout: Option, // ω_v + parameters: Option, // θ_v, from the family's predefined list; None for exact + window_layout: Option, // ω_v } impl LogicalProperties { - pub fn new(representation: Representation, window_layout: Option) -> Result; - pub fn representation(&self) -> &Representation; + pub fn new(expression: &CandidateExpression, parameters: Option, + window_layout: Option) -> Result; + pub fn parameters(&self) -> Option<&SketchParams>; pub fn window_layout(&self) -> Option<&WindowLayout>; } -pub enum Representation { Exact(ExactKind), Summary(SketchKind) } -pub enum WindowLayout { Tumbling { width_ms: u64, count: u32 } } +/// The window count is the group's query range ÷ width; retention is h_v (Stage 3). +pub enum WindowLayout { Tumbling { width_ms: u64 } } + +/// The family's predefined θ list (#1207 step 6), each with its error bound +/// derived through `estimators::local_guarantee` for a given statistic. +pub fn parameter_list(family: SketchAlgorithm) -> &'static [SketchParams]; -/// r^log_{c,i}: one optional constraint per dimension; None accepts any value. -/// `new` rejects invalid combinations (a cap that is not positive and finite, -/// a cap without the statistic it applies to, a window with zero range or step). +/// r^log_{c,i}: only what G_{c,i}'s identity does not fix. `new` rejects a floor +/// that is not in the family's predefined list. pub struct LogicalRequirement { - family: Option, - /// Range and step of the queries reaching the consumer: a width must divide both. - window: Option, - /// The statistic the consumer reads from this input; the producer's ε is - /// derived for it. - statistic: Option, - /// cap_{c,i} = min(validity cap of g_c, pruning cap max_q ε_q / λ_{q,c,i}). - error_cap: Option, + /// θ^min_{c,i}: the smallest θ in the list whose ε, for the statistic the + /// consumer reads, is at most cap_{c,i} = min(validity cap of g_c, pruning cap). + parameter_floor: Option, } impl LogicalRequirement { - pub fn new(family: Option, window: Option, - statistic: Option, error_cap: Option) -> Result; - pub fn family(&self) -> Option; - pub fn window(&self) -> Option<&WindowRequirement>; - pub fn statistic(&self) -> Option<&SketchStatistic>; - pub fn error_cap(&self) -> Option; - /// g_c of a consumer: the propagation factor λ_{c,i} per input, read from the - /// operator and its arguments, and the validity cap (None when g_c is linear). - /// Sets each input's first `error_cap`; Stage 3 composes the factors into λ_{q,c,i}. + pub fn new(parameter_floor: Option) -> Result; + pub fn parameter_floor(&self) -> Option<&SketchParams>; + /// g_c: λ_{c,i} per input, read from the operator and its arguments, and the + /// validity cap (None when g_c is linear). Sets each input's first floor; + /// Stage 3 composes the factors into λ_{q,c,i}. pub fn error_propagation(consumer: &Operator) -> ErrorPropagation; } /// Approved 2026-11-05. pub struct ErrorPropagation { pub factor_per_input: Vec, pub validity_cap: Option } -/// α^log_{u,c,i}: one check per dimension, joined by "and". The error check -/// derives the producer's (ε_u, δ_u) for `requirement.statistic()` through -/// `estimators::local_guarantee` and compares ε_u with `error_cap`. +/// α^log_{u,c,i}: θ_u ⪰ θ^min_{c,i}, compared per family on the parameters that set ε. pub fn satisfies_logical(producer: &LogicalProperties, requirement: &LogicalRequirement) -> bool; ``` -The logical and physical sides have the same shape: a properties struct and a requirement struct, each with private fields, a checked `new`, one accessor per dimension, and one `satisfies_*` function. Invalid values of both logical structs are reported through one error type, `InvalidLogicalProperties`, as `InvalidPhysicalProperties` does for the physical side. +The logical and physical properties have the same shape: private fields, a checked `new`, one accessor per dimension, one error type per side (`InvalidLogicalProperties`, `InvalidPhysicalProperties`), and one `satisfies_*` function. Only the logical side has a stored requirement; the physical requirement is a function of the consumer's own physical properties. -The error bound is not a method of `LogicalProperties`: it depends on the statistic the consumer reads, so `satisfies_logical` derives it from the requirement's statistic. The validity cap is known when the consumer is created (`LogicalRequirement::error_propagation`). The pruning cap needs $\lambda_{q,c,i}$ and the queries that may reach $c$, so Pass 2 builds each input's final requirement with `LogicalRequirement::new` when it builds the logical edges. +The error bound is not stored: it depends on the statistic the consumer reads, which is the consumer's own operator argument. It is used once, when the consumer's floor is computed: the smallest θ in `parameter_list(family)` whose `local_guarantee` for that statistic has ε ≤ cap. The validity cap is known when the consumer is created (`LogicalRequirement::error_propagation`). The pruning cap needs $\lambda_{q,c,i}$ and the queries that may reach $c$, so Pass 2 builds each input's final requirement with `LogicalRequirement::new` when it builds the logical edges. Deleted: `root_node` with private nodes, `input_groups` (the expression's children), `accuracy_budget_parts`, the `SharedBuild` and `SummaryResult` origins. @@ -390,27 +422,18 @@ impl PhysicalProperties { pub fn is_materialized(&self) -> bool; // ℓ ≠ None } -/// r^phys_{c,i}; `new` rejects (IngestionTime, Later), reported through -/// `InvalidPhysicalProperties` (one error type per side). -pub struct PhysicalRequirement { - consumer_timing: ExecutionTiming, - reads_from: Evaluation, - /// (I_κ, φ_κ) of the consumer's execution context, from its query's T_q; - /// None for a non-periodic query. - consumer_schedule: Option, -} -impl PhysicalRequirement { - pub fn new(consumer_timing: ExecutionTiming, reads_from: Evaluation, - consumer_schedule: Option) -> Result; - pub fn consumer_timing(&self) -> ExecutionTiming; - pub fn reads_from(&self) -> Evaluation; - pub fn consumer_schedule(&self) -> Option<&EvaluationSchedule>; -} -/// A periodic evaluation schedule φ + I·ℤ. +/// A periodic schedule φ + I·ℤ: a materialized consumer's own times, or a +/// context's times T_κ (Stage 3). pub struct EvaluationSchedule { interval_ms: u64, phase_ms: u64 } -pub enum Evaluation { Same, Later } -/// α^phys_{u,c,i} -pub fn satisfies_physical(producer: &PhysicalProperties, requirement: &PhysicalRequirement) -> bool; +impl PhysicalProperties { + /// Whether this producer delivers on `times` (the schedule table in section 1); + /// always true when ℓ = None. + pub fn delivers_on(&self, times: &EvaluationSchedule) -> bool; +} +/// α^phys_{u,c,i} = r^phys_{c,i} as a function of the consumer's own p^phys_c: +/// timing and lifetime, and, for a materialized consumer, `delivers_on` its σ_c. +/// For a consumer with ℓ = None, Stage 3 checks `delivers_on(T_κ)` per context. +pub fn satisfies_physical(producer: &PhysicalProperties, consumer: &PhysicalProperties) -> bool; // crates/types/src/stages/physical pub struct PhysicalCandidateId(pub usize); @@ -420,7 +443,6 @@ pub struct PhysicalEdgeId(pub usize); pub struct PhysicalCandidate { pub logical: CandidateForMemoGroupId, // e_v and p_v^log pub properties: PhysicalProperties, // p_v^phys - pub input_requirements: Vec, // r^phys_{c,i} } /// (u, c, i) over physical candidates. pub struct PhysicalEdge { @@ -460,6 +482,8 @@ pub enum Context { IngestionPipeline(RefreshWindowId), // I_r Materialized(PhysicalCandidateId), // κ_m: inside a materialized m's computation } +/// T_κ: when the context runs; y_{κ,u,c,i} exists only if `p^phys_u.delivers_on(T_κ)`. +fn context_times(context: &Context) -> Option; struct SelectionVariables { x_materialized: Vec, // x_v, ℓ ≠ None @@ -519,7 +543,7 @@ Pass 1 adds, for each group a query computes, every way to compute it. Each inte |---|---|---| | Exact | [`per_computation_candidates.rs`](../../../crates/logical-asap-optimization/src/pass1/per_computation_candidates.rs), [`memo_population.rs`](../../../crates/logical-asap-optimization/src/pass1/memo_population.rs) | a candidate with the exact operator; $\boldsymbol\theta_v$ none, $(\varepsilon_v, \delta_v) = (0, 0)$ | | Exact accumulator | same | a state group `ExactAggregate(kind)` over the input group, one candidate; a `FinalizeExactAccumulator` candidate in the result's group reading it | -| Summary, per family the executor supports | same | a state group `SummaryAgg[family]` over the input group, with **one candidate per $\boldsymbol\theta$ in the family's predefined list**, dropping a $\boldsymbol\theta$ whose $\varepsilon_v$ alone exceeds the $\varepsilon_q$ of every query that reaches the group; a `SummaryEstimate[statistic]` candidate in the result's group reading the state group. Each consumer's `error_cap` starts as its $g_c$'s validity cap (none for a linear $g_c$) | +| Summary, per family the executor supports | same | a state group `SummaryAgg[family]` over the input group, with **one candidate per $\boldsymbol\theta$ in the family's predefined list**, dropping a $\boldsymbol\theta$ whose $\varepsilon_v$ alone exceeds the $\varepsilon_q$ of every query that reaches the group; a `SummaryEstimate[statistic]` candidate in the result's group reading the state group. Each consumer's `parameter_floor` starts from its $g_c$'s validity cap (none for a linear $g_c$) | | Fused build (#1203 2c) | same | a build that computes several targets at once: a candidate of the outer target's state group, with `fused_groups` naming the others | Example: for A and B, Pass 1 adds G2 = `SummaryAgg[KLL](G1)` with candidates `k = 200` and `k = 345`, and G3, G4, G5 = `SummaryEstimate[q](G2)`. A and B's three estimates read one state group because their expressions over G1 hash-cons to one group. @@ -530,7 +554,7 @@ Today, Pass 1 instead sizes one $\boldsymbol\theta$ per `accuracy_budget_parts` | # | Rule | Code | Matches | Adds | Status | |---|---|---|---|---|---| -| 1 | Local window composition (tumbling windows) | `add_local_window_composition` ([`window_candidates.rs`](../../../crates/logical-asap-optimization/src/pass2/memo/window_candidates.rs)) | a summary state group whose window consumers serve queries whose ranges and steps share a gcd width $w$ | a candidate of the **same state group** with $\omega_v$ = tumbling $(w, n)$, its $r^{\log}$ window requirement on each consumer input | today it copies consumers per layout (`TumblingWindows` origin); #1160 makes it a candidate of the state group | +| 1 | Local window composition (tumbling windows) | `add_local_window_composition` ([`window_candidates.rs`](../../../crates/logical-asap-optimization/src/pass2/memo/window_candidates.rs)) | a summary state group whose window consumers serve queries whose ranges and steps share a gcd width $w$ | a candidate of the **same state group** with $\omega_v$ = tumbling width $w$; only widths dividing the group's query range are offered, and the step and phase are checked by the schedule rule ($\alpha^{\mathrm{phys}}$) | today it copies consumers per layout (`TumblingWindows` origin); #1160 makes it a candidate of the state group | | 2 | Scan widening | `add_scan_label_candidates` ([`scan_labels.rs`](../../../crates/logical-asap-optimization/src/pass2/memo/rules/scan_labels.rs)) | several scan groups of one metric with different label sets | a group for the widened scan (`find_or_add_group`), and in each original scan group a candidate `Project(widened)` | #1203 2a-5 (#1222) | | 3 | Summary inputs | `factor_summary_inputs` ([`summary_inputs.rs`](../../../crates/logical-asap-optimization/src/pass2/memo/rules/summary_inputs.rs)) | a value projection or filter below a summary build | that projection or filter as its own group; the build's candidate reads it | #1203 2a-2 (#1218); after 2a, Pass 1 already makes it a group | | 4 | Top-k reuse | `add_topk_candidates` ([`topk.rs`](../../../crates/logical-asap-optimization/src/pass2/memo/rules/topk.rs)) | top-k groups over one ranking with different `k` | a candidate of each top-k group reading the largest-`k` group, with an exact-input requirement | #1203 2a-3 (#1220) | @@ -540,7 +564,7 @@ Today, Pass 1 instead sizes one $\boldsymbol\theta$ per `accuracy_budget_parts` | 8 | Summary results | `factor_summary_results` ([`summary_results.rs`](../../../crates/logical-asap-optimization/src/pass2/memo/rules/summary_results.rs)) | a summary readout inside a composite candidate | the readout as its own group | #1203 2a-2 (#1218) makes the readout a group in Pass 1; the rule is then deleted | | 9 | Shared configurations | `add_state_configuration_candidates` ([`shared_configuration_readers.rs`](../../../crates/logical-asap-optimization/src/pass2/memo/rules/shared_configuration_readers.rs)) | consumers that could share a summary at a size or window horizon that covers them | consumer copies at the shared size (`SharedBuild` origin) | deleted in #1203 2b: the state group already holds every $\boldsymbol\theta$, and $y$ chooses per consumer input | | 10 | Shared summary state groups | `add_shared_summary_state_groups` ([`state_groups.rs`](../../../crates/logical-asap-optimization/src/pass2/memo/state_groups.rs)) | summary builds directly over input groups | per-size concrete state groups with consumer copies | deleted in #1203 2b and #1160: one state group per state family | -| — | Logical edges (#1201) | after the last rule | every candidate $c$ and input $i$ | one `LogicalEdge { producer, consumer, input, satisfies_logical_requirement }` per $u \in G_{c,i}$, after tightening each `error_cap` with the pruning cap $\max_q \varepsilon_q / \lambda_{q,c,i}$; $\alpha^{\log}_{u,c,i}$ = `satisfies_logical(p^log_u, r^log_{c,i})`; output `LogicalASAPMemoCandidates { memo, edges }` | #1201 | +| — | Logical edges (#1201) | after the last rule | every candidate $c$ and input $i$ | one `LogicalEdge { producer, consumer, input, satisfies_logical_requirement }` per $u \in G_{c,i}$, after tightening each `parameter_floor` with the pruning cap $\max_q \varepsilon_q / \lambda_{q,c,i}$; $\alpha^{\log}_{u,c,i}$ = `satisfies_logical(p^log_u, r^log_{c,i})`; output `LogicalASAPMemoCandidates { memo, edges }` | #1201 | After #1203 2b, every Pass 2 rule has the same shape: match groups, then `find_or_add_group` for any new intermediate result and `add_candidate` for the new way to compute an existing group. A rule adds no candidate a group already has (same expression and $p^{\log}$). @@ -550,8 +574,8 @@ Stage 2 adds no logical candidates. It expands each logical candidate into physi | Step | Code | Reads | Adds | |---|---|---|---| -| Physical expansion | `enumerate_candidate_configurations` ([`configurations.rs`](../../../crates/physical-asap-optimization/src/catalog/configurations.rs)) | each `CandidateForMemoGroup`, deployment capabilities, the queries' $\mathcal T_q$ | one `PhysicalCandidate { logical, properties, input_requirements }` per valid $p^{\mathrm{phys}} = (\tau, \ell, \sigma)$ that the deployment allows (section 2, levels 4–6), with one `PhysicalRequirement` per input: the consumer's timing, the evaluation it reads from, and its context's schedule | -| Physical edges (#1201) | after expansion | each `LogicalEdge` | one `PhysicalEdge { producer, consumer, input, satisfies_logical_requirement, satisfies_physical_requirement }` per pair of the two logical candidates' physical candidates, with $\alpha^{\mathrm{phys}}$ = `satisfies_physical(p^phys_u, r^phys_{c,i})`; the admissible ones ($\alpha^{\log} = \alpha^{\mathrm{phys}} = 1$) through `admissible_edges_into` / `admissible_edges_out_of` | +| Physical expansion | `enumerate_candidate_configurations` ([`configurations.rs`](../../../crates/physical-asap-optimization/src/catalog/configurations.rs)) | each `CandidateForMemoGroup`, deployment capabilities, the queries' $\mathcal T_q$ | one `PhysicalCandidate { logical, properties, input_requirements }` per valid $p^{\mathrm{phys}} = (\tau, \ell, \sigma)$ that the deployment allows (section 2, levels 4–6), | +| Physical edges (#1201) | after expansion | each `LogicalEdge` | one `PhysicalEdge { producer, consumer, input, satisfies_logical_requirement, satisfies_physical_requirement }` per pair of the two logical candidates' physical candidates, with $\alpha^{\mathrm{phys}}$ = `satisfies_physical(p^phys_u, p^phys_c)`; the admissible ones ($\alpha^{\log} = \alpha^{\mathrm{phys}} = 1$) through `admissible_edges_into` / `admissible_edges_out_of` | Example: v3 (`k = 345`) becomes, among others, (QueryTime, None), (QueryTime, OneEvaluation, Periodic(10 s, 0)) and (IngestionTime, AcrossEvaluations, grid phase 0). G3's estimate at query time has an admissible edge to each of them: same timing, or a query-time consumer of an ingestion-time result kept across evaluations. From 17f34e3e3b3911fa6b494b3bde5a1b276dc653d6 Mon Sep 17 00:00:00 2001 From: zzylol <50204836+zzylol@users.noreply.github.com> Date: Thu, 5 Nov 2026 22:28:58 +0000 Subject: [PATCH 16/44] docs(concepts): parameter_list approved Co-Authored-By: Claude Opus 5.5 --- .../design_docs/concepts/memo-groups-properties-requirements.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/design_docs/concepts/memo-groups-properties-requirements.md b/docs/design_docs/concepts/memo-groups-properties-requirements.md index 3031eea50..faed1f308 100644 --- a/docs/design_docs/concepts/memo-groups-properties-requirements.md +++ b/docs/design_docs/concepts/memo-groups-properties-requirements.md @@ -342,7 +342,7 @@ impl LogicalProperties { /// The window count is the group's query range ÷ width; retention is h_v (Stage 3). pub enum WindowLayout { Tumbling { width_ms: u64 } } -/// The family's predefined θ list (#1207 step 6), each with its error bound +/// Approved 2026-11-05. The family's predefined θ list, each with its error bound /// derived through `estimators::local_guarantee` for a given statistic. pub fn parameter_list(family: SketchAlgorithm) -> &'static [SketchParams]; From 40c3e43c14be7b7bbe0eb6a40856ed3a976574d8 Mon Sep 17 00:00:00 2001 From: zzylol <50204836+zzylol@users.noreply.github.com> Date: Thu, 5 Nov 2026 22:29:57 +0000 Subject: [PATCH 17/44] docs(concepts): context_times -> Context::evaluation_schedule Co-Authored-By: Claude Opus 5.5 --- .../concepts/memo-groups-properties-requirements.md | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/docs/design_docs/concepts/memo-groups-properties-requirements.md b/docs/design_docs/concepts/memo-groups-properties-requirements.md index faed1f308..952ee15fc 100644 --- a/docs/design_docs/concepts/memo-groups-properties-requirements.md +++ b/docs/design_docs/concepts/memo-groups-properties-requirements.md @@ -482,8 +482,11 @@ pub enum Context { IngestionPipeline(RefreshWindowId), // I_r Materialized(PhysicalCandidateId), // κ_m: inside a materialized m's computation } -/// T_κ: when the context runs; y_{κ,u,c,i} exists only if `p^phys_u.delivers_on(T_κ)`. -fn context_times(context: &Context) -> Option; +/// T_κ: when the context runs; y_{κ,u,c,i} exists only if `p^phys_u.delivers_on(κ.evaluation_schedule())`. +impl Context { + /// The context's evaluation schedule T_κ (None for a non-periodic query). + pub fn evaluation_schedule(&self) -> Option; +} struct SelectionVariables { x_materialized: Vec, // x_v, ℓ ≠ None From cb59c4ac351f13da7afadfd070cae3fac7f8df68 Mon Sep 17 00:00:00 2001 From: zzylol <50204836+zzylol@users.noreply.github.com> Date: Fri, 6 Nov 2026 02:50:43 +0000 Subject: [PATCH 18/44] docs(concepts): constraints, not rows (constraint (K), *_constraints functions) Co-Authored-By: Claude Opus 5.5 --- .../memo-groups-properties-requirements.md | 64 +++++++++---------- 1 file changed, 32 insertions(+), 32 deletions(-) diff --git a/docs/design_docs/concepts/memo-groups-properties-requirements.md b/docs/design_docs/concepts/memo-groups-properties-requirements.md index 952ee15fc..e002301cc 100644 --- a/docs/design_docs/concepts/memo-groups-properties-requirements.md +++ b/docs/design_docs/concepts/memo-groups-properties-requirements.md @@ -26,7 +26,7 @@ This page defines the memo's data structures in the terms of the optimization pr | $P_{c,i}$ | physical candidates in the required input group, $G_{c,i} \cap \mathcal C$, before checking requirements | | $\mathcal E$ | potential dependencies $\{(u, c, i) \mid c \in \mathcal C,\ i \in \mathcal I(c),\ u \in P_{c,i}\}$ | | $x_v$, $y_{u,c,i}$, $z_{q,v}$, $t_v$ | MILP variables: candidate selected; producer $u$ supplies input $i$ of $c$; $v \in G_q$ supplies $q$'s result; topological rank | -| (A)–(I) | MILP rows: input satisfaction, producer selection and sharing, query coverage, output selection, acyclicity, utilization, logical requirements ($y \le \alpha^{\log}$), physical requirements ($y \le \alpha^{\mathrm{phys}}$), domains | +| (A)–(I) | MILP constraints: input satisfaction, producer selection and sharing, query coverage, output selection, acyclicity, utilization, logical requirements ($y \le \alpha^{\log}$), physical requirements ($y \le \alpha^{\mathrm{phys}}$), domains | | $w_v$ | fixed cost: $C_{\text{read}}(v) + C_{\text{compute}}(v)$ plus $C^{\text{one}}_{\text{retain}}(v)$ or $C^{\text{across}}_{\text{retain}}(v)$ by $\ell_v$ | | $\Pi$, $V_\Pi$ | selected physical ASAP DAG and its node set; $\operatorname{Cost}(\Pi) = \sum_{v \in \mathcal C} w_v x_v$ | | **(#1207)** $\kappa$ | execution context of a candidate with $\ell_v$ = None: a query evaluation $\kappa_q$, an ingestion pipeline $I_r$, or a materialized candidate's computation $\kappa_m$ | @@ -95,7 +95,7 @@ Valid $(\tau, \ell)$: (QueryTime, None), (IngestionTime, None), (QueryTime, OneE Each input $i$ of consumer $c$ names a producer group $G_{c,i}$ and a property requirement $r_{c,i} = (r^{\log}_{c,i}, r^{\mathrm{phys}}_{c,i})$. A requirement holds only what neither $G_{c,i}$'s identity nor the consumer's own properties already fix (decided 2026-11-05): the family and the query window are in $G_{c,i}$'s identity, and $r^{\mathrm{phys}}_{c,i}$ is a function of the consumer's own $p^{\mathrm{phys}}_c$, not a stored value. Logical satisfaction checks configuration requirements; physical satisfaction checks whether the producer's timing and lifetime jointly support the required access. For a monotone size parameter, $\succeq$ reduces to numerical dominance, e.g. $k_u \ge k_{\text{required}}$ for KLL. -The workload's accuracy target $A_q$ is not an input requirement **(#1207)**: a per-input bound would fix each input's share of the query's budget. Row (K) bounds each query's composed error at its root instead. What stays in $r^{\log}$ is what depends only on the consumer. +The workload's accuracy target $A_q$ is not an input requirement **(#1207)**: a per-input bound would fix each input's share of the query's budget. Constraint (K) bounds each query's composed error at its root instead. What stays in $r^{\log}$ is what depends only on the consumer. #### Error propagation and the error cap @@ -116,35 +116,35 @@ Each $g_c$ is a sum of its inputs' errors, each times a constant **propagation f $$\lambda_{q,c,i} = \sum_{\text{paths } c = c_0 \to c_1 \to \dots \to c_m = \text{root of } q} \lambda_{c_0,i}\,\lambda_{c_1,i_1} \cdots \lambda_{c_m,i_m}.$$ -Row (K) is $\sum \lambda_{q,c,i}\,\varepsilon_u\,y \le \varepsilon_q$ with these factors, so the weights sit on $(q, c, i)$. Two queries reading one group through different expressions have different factors. Failure probabilities add by a union bound and need no factor. +Constraint (K) is $\sum \lambda_{q,c,i}\,\varepsilon_u\,y \le \varepsilon_q$ with these factors, so the weights sit on $(q, c, i)$. Two queries reading one group through different expressions have different factors. Failure probabilities add by a union bound and need no factor. From $g_c$ come two caps on an input's error, both checked per edge as part of $r^{\log}_{c,i}$: -1. **Validity cap: correctness, non-linear consumers only.** A non-linear $g_c$ is bounded by its linear form only in a region, e.g. $\varepsilon_Y \le$ cap for a ratio, where $\frac{\varepsilon_X+\varepsilon_Y}{1-\varepsilon_Y} \le \frac{1}{1-\text{cap}}(\varepsilon_X+\varepsilon_Y)$, so both factors are $\frac{1}{1-\text{cap}}$ (1.11 for cap 0.1). Without the cap, row (K) could accept a plan whose true error is unbounded. A linear $g_c$ is exact everywhere and has no validity cap. -2. **Pruning cap: efficiency only, every consumer, linear ones included.** An input whose error alone breaks every target it could serve is in no feasible plan: $\lambda_{q,c,i}\,\varepsilon_u > \varepsilon_q$ for every query $q$ that may reach $c$. Removing such edges removes no feasible plan, since row (K) rejects them anyway; it keeps them out of $\mathcal E$ and the MILP. It is the per-edge form of Pass 1's "drop a $\boldsymbol\theta$ whose $\varepsilon$ alone exceeds every reading query's target", and prunes more because it uses each consumer's own $\lambda$. +1. **Validity cap: correctness, non-linear consumers only.** A non-linear $g_c$ is bounded by its linear form only in a region, e.g. $\varepsilon_Y \le$ cap for a ratio, where $\frac{\varepsilon_X+\varepsilon_Y}{1-\varepsilon_Y} \le \frac{1}{1-\text{cap}}(\varepsilon_X+\varepsilon_Y)$, so both factors are $\frac{1}{1-\text{cap}}$ (1.11 for cap 0.1). Without the cap, constraint (K) could accept a plan whose true error is unbounded. A linear $g_c$ is exact everywhere and has no validity cap. +2. **Pruning cap: efficiency only, every consumer, linear ones included.** An input whose error alone breaks every target it could serve is in no feasible plan: $\lambda_{q,c,i}\,\varepsilon_u > \varepsilon_q$ for every query $q$ that may reach $c$. Removing such edges removes no feasible plan, since constraint (K) rejects them anyway; it keeps them out of $\mathcal E$ and the MILP. It is the per-edge form of Pass 1's "drop a $\boldsymbol\theta$ whose $\varepsilon$ alone exceeds every reading query's target", and prunes more because it uses each consumer's own $\lambda$. $$\text{cap}_{c,i} = \min\Bigl(\text{validity cap of } g_c,\ \max_{q \text{ that may reach } c} \frac{\varepsilon_q}{\lambda_{q,c,i}}\Bigr), \qquad \alpha^{\log}_{u,c,i} = 0 \text{ when } \varepsilon_u > \text{cap}_{c,i}.$$ For a monotone size parameter, $\varepsilon_u \le \text{cap}_{c,i}$ is a minimum size, e.g. $k_u \ge k_{\text{required}}$ for KLL. -**What stays at the root.** A per-input cap cannot replace row (K): with several inputs, $\varepsilon_X + \varepsilon_Y \le \varepsilon_q$ couples them, and one cap per input would fix each input's share in advance (the rejected budget shares). The caps only prune; row (K) decides. +**What stays at the root.** A per-input cap cannot replace constraint (K): with several inputs, $\varepsilon_X + \varepsilon_Y \le \varepsilon_q$ couples them, and one cap per input would fix each input's share in advance (the rejected budget shares). The caps only prune; constraint (K) decides. | Requirement | Stated by | Values | Constrains | Checked by | |---|---|---|---|---| -| Accuracy target $A_q$ | the workload, per query | metric: absolute (in the value's units, or times the stream's L1 or L2 norm), relative, rank; bound $\varepsilon_q$; failure probability $\delta_q$; scope: per value, whole result, or groups above a share of the records | $\boldsymbol\theta_u$ of every approximate candidate $q$ depends on, through $(\varepsilon_u, \delta_u)$ | row (K) **(#1207)** | -| Parameter floor $\boldsymbol\theta^{\min}_{c,i}$, the only field of $r^{\log}_{c,i}$ | every consumer, per input, from $g_c$ and the targets of the queries that may reach it | the smallest $\boldsymbol\theta$ in the family's predefined list whose $\varepsilon$, for the statistic $c$ reads, is at most $\text{cap}_{c,i}$ = min(validity cap, pruning cap); e.g. $k_u \ge k_{\text{required}}$ | $\boldsymbol\theta_u$ | $\alpha^{\log}$, row (G): $\boldsymbol\theta_u \succeq \boldsymbol\theta^{\min}_{c,i}$ | -| Consumer timing, $r^{\mathrm{phys}}_{c,i}$ from $\tau_c$ | the consumer's own $p^{\mathrm{phys}}_c$ | when $c$ runs: ingestion or query time | $\tau_u$, $\ell_u$ | $\alpha^{\mathrm{phys}}$, row (H) | -| Consumer schedule, $r^{\mathrm{phys}}_{c,i}$ from $\sigma_c$ or $\kappa$ **(#1207)** | a materialized consumer's own $\sigma_c$; for $\ell_c$ = None, the times $T_\kappa$ of each context $\kappa$ it runs in (from its query's $\mathcal T_q$) | interval and phase | $\sigma_u$: the producer delivers on those times | $\alpha^{\mathrm{phys}}$, row (H): in Stage 2 for a materialized consumer; per context in Stage 3 for $\ell_c$ = None | +| Accuracy target $A_q$ | the workload, per query | metric: absolute (in the value's units, or times the stream's L1 or L2 norm), relative, rank; bound $\varepsilon_q$; failure probability $\delta_q$; scope: per value, whole result, or groups above a share of the records | $\boldsymbol\theta_u$ of every approximate candidate $q$ depends on, through $(\varepsilon_u, \delta_u)$ | constraint (K) **(#1207)** | +| Parameter floor $\boldsymbol\theta^{\min}_{c,i}$, the only field of $r^{\log}_{c,i}$ | every consumer, per input, from $g_c$ and the targets of the queries that may reach it | the smallest $\boldsymbol\theta$ in the family's predefined list whose $\varepsilon$, for the statistic $c$ reads, is at most $\text{cap}_{c,i}$ = min(validity cap, pruning cap); e.g. $k_u \ge k_{\text{required}}$ | $\boldsymbol\theta_u$ | $\alpha^{\log}$, constraint (G): $\boldsymbol\theta_u \succeq \boldsymbol\theta^{\min}_{c,i}$ | +| Consumer timing, $r^{\mathrm{phys}}_{c,i}$ from $\tau_c$ | the consumer's own $p^{\mathrm{phys}}_c$ | when $c$ runs: ingestion or query time | $\tau_u$, $\ell_u$ | $\alpha^{\mathrm{phys}}$, constraint (H) | +| Consumer schedule, $r^{\mathrm{phys}}_{c,i}$ from $\sigma_c$ or $\kappa$ **(#1207)** | a materialized consumer's own $\sigma_c$; for $\ell_c$ = None, the times $T_\kappa$ of each context $\kappa$ it runs in (from its query's $\mathcal T_q$) | interval and phase | $\sigma_u$: the producer delivers on those times | $\alpha^{\mathrm{phys}}$, constraint (H): in Stage 2 for a materialized consumer; per context in Stage 3 for $\ell_c$ = None | | Deployment capability | the deployment and data | the source has an ingestion pipeline; the executor supports the family; query-time results may be kept across evaluations | $\tau$, $\operatorname{op}$, (QueryTime, AcrossEvaluations) | candidate generation: such candidates are not in $\mathcal C$ | -| Latency, resource budgets **(#1207)** | the workload | per-query latency bound; memory and CPU budgets | the whole plan | rows (L) and budgets | +| Latency, resource budgets **(#1207)** | the workload | per-query latency bound; memory and CPU budgets | the whole plan | constraints (L) and budgets | Accuracy metrics that may be added later, when a primitive needs them: top-k recall or precision, and a false-positive rate for membership summaries. How requirements map to properties: ``` -A_q (workload) ──► θ_u of every approximate input (through ε_u, δ_u; row (K)) +A_q (workload) ──► θ_u of every approximate input (through ε_u, δ_u; constraint (K)) parameter floor θ^min r^log ──► θ_u (from cap = min(validity cap of g_c, pruning cap ε_q / λ_{q,c,i})) τ_c r^phys = f(p^phys_c) ──► τ_u, ℓ_u σ_c or T_κ r^phys = f(p^phys_c) ──► σ_u @@ -240,7 +240,7 @@ The levels, top to bottom: What the tree does not decide, because it is a choice of the MILP over $\mathcal C$ and $\mathcal E$: - **Which producer each consumer input reads:** $y_{u,c,i}$. Two consumers of one group may read different leaves, e.g. one reads `k = 200` and another `k = 345`. - **Where an unmaterialized candidate runs (#1207):** a candidate with $\ell_v$ = None runs inside each reading context $\kappa$. -- **How much a materialized candidate retains (#1207):** $h_v$, from the longest range among the queries its selected consumers serve, row (R). +- **How much a materialized candidate retains (#1207):** $h_v$, from the longest range among the queries its selected consumers serve, constraint (R). ### Order of decisions @@ -268,7 +268,7 @@ This section gives the target interface of each stage after the formulation refa | Stage 1, Pass 1 | `DagOfLogicalMemoGroups` | `Memo` | $v = (e_v, p^{\log}_v)$, $r^{\log}_{c,i}$, $\theta$ list | #1203 | | Stage 1, Pass 2 | `Memo` | `LogicalASAPMemoCandidates` | rule-added candidates; logical edges with $\alpha^{\log}$ | #1203, #1201 | | Stage 2 | `LogicalASAPMemoCandidates` | `PhysicalASAPMemoCandidates` | $p^{\mathrm{phys}}_v = (\tau, \ell, \sigma)$, $r^{\mathrm{phys}}_{c,i}$, physical edges $\mathcal E$ with $\alpha^{\mathrm{phys}}$ | #1200, #1201 | -| Stage 3 | `PhysicalASAPMemoCandidates` | `DagOfSelectedCandidates` | contexts $\kappa$; cost $w_v$ (`asap-cost`); $x$, $y$, $z$, $t$, $h$; rows (A)–(K), (R), (L), budgets | #1204 | +| Stage 3 | `PhysicalASAPMemoCandidates` | `DagOfSelectedCandidates` | contexts $\kappa$; cost $w_v$ (`asap-cost`); $x$, $y$, $z$, $t$, $h$; constraints (A)–(K), (R), (L), budgets | #1204 | | Realizer, export | `DagOfSelectedCandidates` | `PhysicalASAPDAG` | — | #1200 (export fields) | ### Stage 0: frontends and hash-consing @@ -470,7 +470,7 @@ Deleted: `Materialization`, `PhysicalCandidate::{timing, materialization, operat ### Stage 3: the MILP -Stage 3 reads the physical candidates and edges and selects the DAG. Variables and rows are named after the formulation. +Stage 3 reads the physical candidates and edges and selects the DAG. Variables and constraints are named after the formulation. ```rust // crates/plan-selection/src/milp/model.rs @@ -497,16 +497,16 @@ struct SelectionVariables { h: Vec, // h_v, ℓ ≠ None latency: LatencyVariables, // (L) } -fn input_satisfaction_rows(..); // (A) Σ_u y_{u,c,i} = x_c -fn producer_selection_rows(..); // (B) y_{u,c,i} ≤ x_u -fn query_coverage_rows(..); // (C), (D) -fn acyclicity_rows(..); // (E) -fn utilization_rows(..); // (F) -fn consumer_constraint_rows(..); // (J): MinimumQueryReaders, HighestSharedQueryMaterialization over reach -fn accuracy_rows(..); // (K) Σ λ_{q,u} ε_u a_{q,u} ≤ ε_q, Σ δ_u a_{q,u} ≤ δ_q -fn retention_rows(..); // (R) h_v ≥ need_{c,i} · y_{v,c,i} -fn latency_rows(..); // (L) -fn resource_budget_rows(..); // budgets (#1207) +fn input_satisfaction_constraints(..); // (A) Σ_u y_{u,c,i} = x_c +fn producer_selection_constraints(..); // (B) y_{u,c,i} ≤ x_u +fn query_coverage_constraints(..); // (C), (D) +fn acyclicity_constraints(..); // (E) +fn utilization_constraints(..); // (F) +fn consumer_constraint_constraints(..); // (J): MinimumQueryReaders, HighestSharedQueryMaterialization over reach +fn accuracy_constraints(..); // (K) Σ λ_{q,u} ε_u a_{q,u} ≤ ε_q, Σ δ_u a_{q,u} ≤ δ_q +fn retention_constraints(..); // (R) h_v ≥ need_{c,i} · y_{v,c,i} +fn latency_constraints(..); // (L) +fn resource_budget_constraints(..); // budgets (#1207) // Objective: min Σ w_v x_v + Σ ρ_κ w_v x_{κ,v} + Σ π_v h_v // crates/types/src/stages/physical/graph.rs (unchanged) @@ -518,7 +518,7 @@ pub struct DagOfSelectedCandidates { } ``` -(G) $y \le \alpha^{\log}$ and (H) $y \le \alpha^{\mathrm{phys}}$ are not rows: $y$ exists only on admissible edges. Deleted: `ResultSlot`, `slot_of`, the `e` binaries, schedule groups `z[m, G]`, `N[q]`, `A[q, parts]`, and `p[ctx, c]` for materialized candidates. +(G) $y \le \alpha^{\log}$ and (H) $y \le \alpha^{\mathrm{phys}}$ are not constraints in the model: $y$ exists only on admissible edges. Deleted: `ResultSlot`, `slot_of`, the `e` binaries, schedule groups `z[m, G]`, `N[q]`, `A[q, parts]`, and `p[ctx, c]` for materialized candidates. ### Realizer and export @@ -594,9 +594,9 @@ A memo group stores its candidates and, through their expressions, the groups th A logical DAG rooted at group $G$ is a set of logical candidates and edges $(u, c, i)$ such that: 1. exactly one candidate of $G$ is in it (the root); -2. every candidate $c$ in it has, for each input $i$, exactly one edge $(u, c, i)$, with $u$ a candidate of $G_{c,i}$ and $\alpha^{\log}_{u,c,i} = 1$ (rows (A) and (G)); -3. every candidate in it is the root or the producer of some edge (row (F)); -4. the edges form no cycle (row (E)). +2. every candidate $c$ in it has, for each input $i$, exactly one edge $(u, c, i)$, with $u$ a candidate of $G_{c,i}$ and $\alpha^{\log}_{u,c,i} = 1$ (constraints (A) and (G)); +3. every candidate in it is the root or the producer of some edge (constraint (F)); +4. the edges form no cycle (constraint (E)). A candidate is one node however many edges read it: two consumers that choose the same producer share it, which is what $x_u \in \{0, 1\}$ says. Two consumers of one group may choose different candidates of it. The listing is therefore a choice of one producer per consumer input $(c, i)$, not one candidate per group. @@ -637,7 +637,7 @@ fn expand(memo: &Memo, dag: LogicalDag, mut pending: VecDeque>() })) { - // Row (E): a producer that already reaches `consumer` would close a cycle. + // Constraint (E): a producer that already reaches `consumer` would close a cycle. if choice.iter().any(|&u| reaches(&dag, u, consumer)) { continue; } @@ -681,7 +681,7 @@ fn print(memo: &Memo, dag: &LogicalDag) { } ``` -For a workload, list from every query root at once: start `pending` with one root candidate per $G_q$ (rows (C) and (D)), taking the product over queries of their root candidates. A candidate two queries reach is one node, as above. +For a workload, list from every query root at once: start `pending` with one root candidate per $G_q$ (constraints (C) and (D)), taking the product over queries of their root candidates. A candidate two queries reach is one node, as above. ### Example @@ -698,7 +698,7 @@ G3[0] SummaryEstimate q=0.95 G1[0] Scan … and (k=200, k=345), (k=345, k=200), (k=345, k=345) ``` -Listed from both roots at once there are $2 \times 4 = 8$ DAGs. In the one where all three estimates choose `k = 345`, `G2[1]` is a single node read by three consumers, so its cost is counted once. Row (K) later accepts only the DAGs that meet each query's target: for B, only (k=345, k=345). Error caps (section 1) remove edges before the listing, so a capped producer never appears. +Listed from both roots at once there are $2 \times 4 = 8$ DAGs. In the one where all three estimates choose `k = 345`, `G2[1]` is a single node read by three consumers, so its cost is counted once. Constraint (K) later accepts only the DAGs that meet each query's target: for B, only (k=345, k=345). Error caps (section 1) remove edges before the listing, so a capped producer never appears. ### Size From cbc26543e605e54db3ec49d898d066aaa258c0ab Mon Sep 17 00:00:00 2001 From: Zeying Zhu <50204836+zzylol@users.noreply.github.com> Date: Thu, 5 Nov 2026 22:04:01 -0500 Subject: [PATCH 19/44] Update memo-groups-properties-requirements.md --- .../memo-groups-properties-requirements.md | 960 ++++++++++-------- 1 file changed, 518 insertions(+), 442 deletions(-) diff --git a/docs/design_docs/concepts/memo-groups-properties-requirements.md b/docs/design_docs/concepts/memo-groups-properties-requirements.md index e002301cc..ca3c51934 100644 --- a/docs/design_docs/concepts/memo-groups-properties-requirements.md +++ b/docs/design_docs/concepts/memo-groups-properties-requirements.md @@ -1,290 +1,179 @@ # Memo groups, properties and requirements -This page defines the memo's data structures in the terms of the optimization problem formulation: workloads, memo groups, candidates, properties and property requirements. It also gives the decision tree Stage 1 and Stage 2 walk to enumerate candidates, and each stage's interfaces (section 3). Where tracking issue #1207 extends the formulation, the extension is marked **(#1207)**. +This document explains how the optimizer represents alternatives, checks whether they can be composed, and selects an executable plan. It follows the supplied **Optimization Problem Formulation** for the baseline model and labels additional design work **Extension (#1207)**. -## Notation +The Rust blocks are interface sketches, not standalone compilable implementations. Repository paths and issue references are retained from the supplied draft; their current implementation status has not been independently checked. -| Notation | Definition | -|---|---| -| $\mathcal W = \{(q, \mathcal T_q, A_q) \mid q \in \mathcal Q\}$ | workload over an evaluation horizon $H$: each query expression $q$, its evaluation schedule $\mathcal T_q \subseteq [0, H)$, and its accuracy target $A_q$ | -| $\mathcal Q$ | set of queries, each with one final output | -| $G$ | memo group: a logical result or an intermediate state family over specified input data, independent of parameter configurations | -| $G_q$ | memo group of query $q$'s final result | -| $v = (e_v, p_v)$ | candidate of a memo group: an expression with its properties | -| $e_v = \operatorname{op}_v(G_{v,1}, \dots, G_{v,n_v})$ | expression: operator $\operatorname{op}_v$ with its operator arguments (e.g. the summary family), over input memo groups $G_{v,i}$ | -| $p_v = (p^{\log}_v, p^{\mathrm{phys}}_v)$ | properties: logical and physical | -| $p^{\log}_v = \boldsymbol\theta_v$ | summary algorithm parameters, e.g. sketch size; **(#1207)** also the window layout $\omega_v$ | -| $p^{\mathrm{phys}}_v = (\tau_v, \ell_v)$ | execution timing and lifetime; **(#1207)** also the schedule $\sigma_v$ when $\ell_v \ne$ None | -| $(\varepsilon_v, \delta_v)$ | error bound and failure probability of $\boldsymbol\theta_v$ for the statistic $\operatorname{op}_v$ reads; derived, not a property | -| $c \in G_c$, $u \in G_u$ | consumer candidate and its group; producer candidate and its group. This page says *consumer* for $c$ and *query* $q$ (or its execution context $\kappa$) for who needs a result on a schedule; it does not use "reader" | -| $\mathcal I(c)$ | input indices of consumer $c$ | -| $G_{c,i}$ | memo group required by input $i$ of consumer $c$ | -| $r_{c,i} = (r^{\log}_{c,i}, r^{\mathrm{phys}}_{c,i})$ | property requirement of input $i$ of consumer $c$; $r^{\mathrm{phys}}_{c,i}$ is a function of $p^{\mathrm{phys}}_c$ | -| $p(u) \succeq r_{c,i}$ | producer $u$'s properties satisfy the requirement: $p^{\log}(u) \succeq r^{\log}_{c,i} \land p^{\mathrm{phys}}(u) \succeq r^{\mathrm{phys}}_{c,i}$ | -| $\alpha^{\log}_{u,c,i}$, $\alpha^{\mathrm{phys}}_{u,c,i}$ | fixed indicators of $p^{\log}_u \succeq r^{\log}_{c,i}$ and $p^{\mathrm{phys}}_u \succeq r^{\mathrm{phys}}_{c,i}$ | -| $\mathcal C$ | physical candidates, each with a fixed parameter configuration, timing and lifetime | -| $P_{c,i}$ | physical candidates in the required input group, $G_{c,i} \cap \mathcal C$, before checking requirements | -| $\mathcal E$ | potential dependencies $\{(u, c, i) \mid c \in \mathcal C,\ i \in \mathcal I(c),\ u \in P_{c,i}\}$ | -| $x_v$, $y_{u,c,i}$, $z_{q,v}$, $t_v$ | MILP variables: candidate selected; producer $u$ supplies input $i$ of $c$; $v \in G_q$ supplies $q$'s result; topological rank | -| (A)–(I) | MILP constraints: input satisfaction, producer selection and sharing, query coverage, output selection, acyclicity, utilization, logical requirements ($y \le \alpha^{\log}$), physical requirements ($y \le \alpha^{\mathrm{phys}}$), domains | -| $w_v$ | fixed cost: $C_{\text{read}}(v) + C_{\text{compute}}(v)$ plus $C^{\text{one}}_{\text{retain}}(v)$ or $C^{\text{across}}_{\text{retain}}(v)$ by $\ell_v$ | -| $\Pi$, $V_\Pi$ | selected physical ASAP DAG and its node set; $\operatorname{Cost}(\Pi) = \sum_{v \in \mathcal C} w_v x_v$ | -| **(#1207)** $\kappa$ | execution context of a candidate with $\ell_v$ = None: a query evaluation $\kappa_q$, an ingestion pipeline $I_r$, or a materialized candidate's computation $\kappa_m$ | -| $g_c$ | error propagation function of consumer $c$: $\varepsilon_c \le g_c(\varepsilon_{c,1}, \dots, \varepsilon_{c,n})$ | -| $\lambda_{q,c,i}$ | propagation factor of input $i$ of $c$ to query $q$'s result, read off $q$'s expression | -| $\text{cap}_{c,i}$, $\boldsymbol\theta^{\min}_{c,i}$ | error cap of input $i$ of $c$ (the smaller of $g_c$'s validity cap and the pruning cap), and the parameter floor $r^{\log}_{c,i}$ stores: the smallest listed $\boldsymbol\theta$ whose $\varepsilon$ is at most the cap | -| **(#1207)** (K), (R), (L), budgets | accuracy at each query root, $\sum_u \lambda_{q,u}\varepsilon_u a_{q,u} \le \varepsilon_q$ and $\sum_u \delta_u a_{q,u} \le \delta_q$; retention $h_v$; latency; resource budgets | - -## Example - -Workload: query A, `quantile_over_time(0.95, latency[1m])`, and query B, `quantile_over_time(0.99, latency[1m]) / quantile_over_time(0.50, latency[1m])`. Both are evaluated every 10 s ($\mathcal T_A = \mathcal T_B = 10\,\text{s}\cdot\mathbb Z$) with accuracy target $A_q$: rank error 0.01. +## 1. Model and running example -``` -G1 Scan latency[1m] - candidate: v1 = Scan p^phys = (QueryTime, None) -G2 KLL state over G1 one group, whatever θ - candidates: v2 = SummaryAgg[KLL](G1) θ = {k = 200} ε = 0.008 δ = 0.01 - v3 = SummaryAgg[KLL](G1) θ = {k = 345} ε = 0.005 δ = 0.01 - (Stage 2 expands each into its (τ, ℓ) alternatives) -G3 = G_A SummaryEstimate[q=0.95](G2) -G4 SummaryEstimate[q=0.99](G2) -G5 SummaryEstimate[q=0.50](G2) -G6 = G_B Divide(G4, G5) r^log on each input: cap = min(validity cap of /, ε_B / λ) -``` +A **workload** contains query expressions, their evaluation schedules, and their accuracy targets. A **memo group** represents one logical result or one intermediate state family over specified input data. A **candidate** is one way to compute that result: one operator over input groups, together with its properties. -The ε values are illustrative. +Input groups leave producer choices open. Selecting a candidate does not select its entire input plan. Each consumer input chooses a producer separately, and several inputs may choose the same producer. -- G2 is one memo group: the KLL state over `latency[1m]`. Its parameter configurations are candidates of G2, not separate groups. The family KLL is an argument of $\operatorname{op}_v$. -- G3, G4 and G5 are consumers of G2: their candidates' expressions read G2 as $G_{c,1}$. G2 does not store them. -- A meets 0.01 with either θ. B adds its two operands' errors (0.016 with `k = 200`, 0.010 with `k = 345`) and needs `k = 345` for both, so a plan where G3's, G4's and G5's candidates all read v3 builds one KLL state. +The expression includes the operator's arguments, including the summary family. In this design, KLL and DDSketch states belong to separate state groups. A final-result group can offer readout candidates that use either family. This resolves the old decision tree's contradiction: it must not put KLL and DDSketch builds in one state group. -## 1. Properties and property requirements +### Running example -### Properties +Query A requests the p95 latency over one minute. Query B requests p99 divided by p50 over the same input. Both run every ten seconds. -A property is a value a candidate has. Properties are products of dimensions: one field per dimension, never a flat enum that lists the combinations. Invalid combinations are rejected by the checked constructor. +| Group | Result | Candidate expressions | +|---|---|---| +| G1 | Raw latency input | `Scan latency[1m]` | +| G2 | KLL state over G1 | `SummaryAgg[KLL](G1)` with `k = 200` or `k = 345` | +| G3 | Query A output | `SummaryEstimate[q=0.95](G2)` | +| G4 | p99 latency | `SummaryEstimate[q=0.99](G2)` | +| G5 | p50 latency | `SummaryEstimate[q=0.50](G2)` | +| G6 | Query B output | `Divide(G4, G5)` | -What is computed is the expression $e_v$, not a property: the operator, its arguments (including the summary family), and its input groups. A memo group's candidates differ in their properties, and, when rules add other ways to compute the same result, in their expressions. +G3, G4, and G5 choose a G2 producer independently. If all choose the same retained physical producer, they can reuse one computed state when their schedules and windows are compatible. An unmaterialized candidate may instead execute separately in different contexts; Section 5 explains that distinction. -| Dimension | Values | Part of | Code | -|---|---|---|---| -| Parameter configuration $\boldsymbol\theta_v$ | per family, from a predefined list: KLL `k`; DDSketch `alpha`; CMS / CountSketch `width × depth`; CMS / CountSketch with heap `width × depth × heap_size`; UnivMon `heap_size × rows × cols × layers`; HLL `precision`; KMV `k`. None for an exact operator | $p^{\log}_v$ | `SketchParams` inside `SketchKind` ([`sketch.rs`](../../../crates/types/src/ir/schema/sketch.rs)), sized by `size_params` per `accuracy_budget_parts` today | -| **(#1207)** Window layout $\omega_v$ | none; or tumbling windows of width $w$, from the gcd closure of the ranges and steps of the queries that reach the group. The window count is not a property: it is the group's query range divided by $w$, and how many windows to retain is $h_v$ (Stage 3). Later: sliding windows by merge or by subtraction | $p^{\log}_v$ | `SummaryTumblingWindowChain` arguments, added by Pass 2 ([`window_candidates.rs`](../../../crates/logical-asap-optimization/src/pass2/memo/window_candidates.rs)) | -| Execution timing $\tau_v$ | IngestionTime, QueryTime | $p^{\mathrm{phys}}_v$ | `ExecutionTiming` ([`execution_data_state.rs`](../../../crates/types/src/ir/properties/execution_data_state.rs)) | -| Lifetime $\ell_v$ | None, OneEvaluation, AcrossEvaluations; **(#1207)** a materialized lifetime carries its medium, Memory or Disk | $p^{\mathrm{phys}}_v$ | `Materialization` on main; `Lifetime`, `Medium` in #1214 ([`physical/`](../../../crates/types/src/physical)) | -| **(#1207)** Schedule $\sigma_v$ | only when $\ell_v \ne$ None: Periodic (interval, phase) for OneEvaluation; tumbling-window grid phase for AcrossEvaluations | $p^{\mathrm{phys}}_v$ | schedule groups `z[m, G]` in the MILP today ([`cost_terms.rs`](../../../crates/plan-selection/src/milp/cost_terms.rs)); a property in #1200 PR 2 | +The old example assigned illustrative KLL rank errors of 0.008 and 0.005 and added them to obtain Query B's ratio error. That inference is invalid: rank error does not directly bound numerical quantile error or relative ratio error. The example here demonstrates representation and sharing only. Accepting Query B requires an applicable error model, including assumptions about its denominator. -Lifetime None means the output is discarded after its consumer's execution completes. OneEvaluation retains the result until all its consumers within one evaluation finish. AcrossEvaluations retains it throughout the evaluation period, so later evaluations reuse it. A candidate is materialized when $\ell_v \ne$ None; this is derived, not stored. +### Notation used by the baseline model -The error bound $(\varepsilon_v, \delta_v)$ is not a dimension: it is a function of $\boldsymbol\theta_v$ and the statistic $\operatorname{op}_v$ reads, derived when needed (`local_guarantee` in [`estimators`](../../../crates/accuracy/src/estimators/mod.rs)). An exact candidate has $(0, 0)$. +Symbols are collected here and in the sections that use equations. The prose and code use descriptive names. -Invalid combinations of $p^{\mathrm{phys}}$, rejected by `PhysicalProperties::new` (#1200; #1214 implements the first, the others come with $\sigma$ in #1200 PR 2): - -| Combination | Why | +| Symbol | Meaning | |---|---| -| (IngestionTime, OneEvaluation) | there is no evaluation at ingestion time | -| $\ell$ = None with a schedule | where an unmaterialized result runs is its consumer's execution context $\kappa$, not a property | -| $\ell \ne$ None without a schedule | a materialized result must say on which times or grid it is delivered | -| OneEvaluation with a grid, AcrossEvaluations with a periodic schedule | each lifetime has one kind of schedule | -| a phase at or above the grid width or interval, or interval 0 | the phase is taken modulo the width or interval | -| IngestionTime with a grid phase ≠ 0 | ingestion-time grids are anchored at the epoch today: the plan carries no origin | - -Valid $(\tau, \ell)$: (QueryTime, None), (IngestionTime, None), (QueryTime, OneEvaluation), (QueryTime, AcrossEvaluations), (IngestionTime, AcrossEvaluations), each materialized one for each medium. +| $\mathcal W$, $\mathcal Q$, $H$ | Workload, query set, and common evaluation horizon | +| $\mathcal T_q$, $A_q$ | Query evaluation schedule and accuracy target | +| $G$, $G_q$ | Memo group and query-output group | +| $v$, $u$, $c$, $i$ | Candidate, producer candidate, consumer candidate, and consumer input index | +| $e_v$, $p_v$ | Candidate expression and properties | +| $\boldsymbol\theta_v$, $\tau_v$, $\ell_v$ | Parameter configuration, execution timing, and result lifetime | +| $\mathcal I(c)$, $G_{c,i}$ | Consumer input indices and required group for one input | +| $r_{c,i}$ | Logical and physical requirements of one input | +| $\mathcal C$ | Physical candidate set | +| $\alpha^{\mathrm{log}}_{u,c,i}$, $\alpha^{\mathrm{phys}}_{u,c,i}$ | Fixed logical and physical compatibility indicators | -### Property requirements +The candidate and property definitions are: -Each input $i$ of consumer $c$ names a producer group $G_{c,i}$ and a property requirement $r_{c,i} = (r^{\log}_{c,i}, r^{\mathrm{phys}}_{c,i})$. A requirement holds only what neither $G_{c,i}$'s identity nor the consumer's own properties already fix (decided 2026-11-05): the family and the query window are in $G_{c,i}$'s identity, and $r^{\mathrm{phys}}_{c,i}$ is a function of the consumer's own $p^{\mathrm{phys}}_c$, not a stored value. Logical satisfaction checks configuration requirements; physical satisfaction checks whether the producer's timing and lifetime jointly support the required access. For a monotone size parameter, $\succeq$ reduces to numerical dominance, e.g. $k_u \ge k_{\text{required}}$ for KLL. +$$ +v=(e_v,p_v), \qquad e_v=\operatorname{op}_v(G_{v,1},\ldots,G_{v,n_v}). +$$ -The workload's accuracy target $A_q$ is not an input requirement **(#1207)**: a per-input bound would fix each input's share of the query's budget. Constraint (K) bounds each query's composed error at its root instead. What stays in $r^{\log}$ is what depends only on the consumer. +$$ +p_v=(p_v^{\mathrm{log}},p_v^{\mathrm{phys}}),\qquad +p_v^{\mathrm{log}}=\boldsymbol\theta_v,\qquad +p_v^{\mathrm{phys}}=(\tau_v,\ell_v). +$$ -#### Error propagation and the error cap +The supplied LaTeX uses the producer-set symbol in two different ways: the prose defines a filtered set, while its notation table defines an unfiltered set. This document follows the table and constraints (G)–(H), and gives the filtered set a separate name: -Each consumer $c$ has an error propagation function $g_c$ that bounds its output error by its input errors: +$$ +P_{c,i}=G_{c,i}\cap\mathcal C, +\qquad +P^{\mathrm{adm}}_{c,i} +=\{u\in P_{c,i}:\alpha^{\mathrm{log}}_{u,c,i}=\alpha^{\mathrm{phys}}_{u,c,i}=1\}. +$$ -$$\varepsilon_c \le g_c(\varepsilon_{c,1}, \dots, \varepsilon_{c,n}).$$ +$$ +\mathcal E=\{(u,c,i):c\in\mathcal C,\ i\in\mathcal I(c),\ u\in P_{c,i}\}. +$$ -Composing them from each query's root down gives the query's error, and the accuracy target $A_q$ is $g_q(\dots) \le \varepsilon_q$ at the root. - -| Consumer | $g_c$ | Linear | -|---|---|---| -| single readout, `Project`, filter | $\varepsilon_{c,1}$ | yes | -| `sum by` over $n$ series (absolute error) | $n\,\varepsilon_{c,1}$ | yes | -| $X + Y$, $X - Y$ (absolute error) | $\varepsilon_X + \varepsilon_Y$ | yes | -| $X / Y$ (relative error) | $\dfrac{\varepsilon_X + \varepsilon_Y}{1 - \varepsilon_Y}$ | no | +Here, group membership denotes the physical candidates belonging to that group. Property satisfaction means that both checks succeed; it is not a universal numerical ordering. -Each $g_c$ is a sum of its inputs' errors, each times a constant **propagation factor** $\lambda_{c,i}$: $g_c = \sum_i \lambda_{c,i}\,\varepsilon_{c,i}$ (for a non-linear $g_c$, within its validity cap, below). The factors come from the query expression itself: each operator and its arguments, e.g. `sum by` over $n$ series has factor $n$. No measurement or optimization is involved. A query's factor for input $i$ of $c$ multiplies the factors along the path in $q$'s expression from that input to $q$'s root, and adds over paths when the input reaches the root more than once: +## 2. Properties and input requirements -$$\lambda_{q,c,i} = \sum_{\text{paths } c = c_0 \to c_1 \to \dots \to c_m = \text{root of } q} \lambda_{c_0,i}\,\lambda_{c_1,i_1} \cdots \lambda_{c_m,i_m}.$$ +A property describes what a candidate delivers. An input requirement describes what a consumer can accept. Workload requirements such as final accuracy and latency apply to the composed plan. -Constraint (K) is $\sum \lambda_{q,c,i}\,\varepsilon_u\,y \le \varepsilon_q$ with these factors, so the weights sit on $(q, c, i)$. Two queries reading one group through different expressions have different factors. Failure probabilities add by a union bound and need no factor. +### Property dimensions -From $g_c$ come two caps on an input's error, both checked per edge as part of $r^{\log}_{c,i}$: +This is the single reference table for property dimensions. Stage descriptions below explain when these choices are made without repeating their definitions. -1. **Validity cap: correctness, non-linear consumers only.** A non-linear $g_c$ is bounded by its linear form only in a region, e.g. $\varepsilon_Y \le$ cap for a ratio, where $\frac{\varepsilon_X+\varepsilon_Y}{1-\varepsilon_Y} \le \frac{1}{1-\text{cap}}(\varepsilon_X+\varepsilon_Y)$, so both factors are $\frac{1}{1-\text{cap}}$ (1.11 for cap 0.1). Without the cap, constraint (K) could accept a plan whose true error is unbounded. A linear $g_c$ is exact everywhere and has no validity cap. -2. **Pruning cap: efficiency only, every consumer, linear ones included.** An input whose error alone breaks every target it could serve is in no feasible plan: $\lambda_{q,c,i}\,\varepsilon_u > \varepsilon_q$ for every query $q$ that may reach $c$. Removing such edges removes no feasible plan, since constraint (K) rejects them anyway; it keeps them out of $\mathcal E$ and the MILP. It is the per-edge form of Pass 1's "drop a $\boldsymbol\theta$ whose $\varepsilon$ alone exceeds every reading query's target", and prunes more because it uses each consumer's own $\lambda$. - -$$\text{cap}_{c,i} = \min\Bigl(\text{validity cap of } g_c,\ \max_{q \text{ that may reach } c} \frac{\varepsilon_q}{\lambda_{q,c,i}}\Bigr), \qquad \alpha^{\log}_{u,c,i} = 0 \text{ when } \varepsilon_u > \text{cap}_{c,i}.$$ - -For a monotone size parameter, $\varepsilon_u \le \text{cap}_{c,i}$ is a minimum size, e.g. $k_u \ge k_{\text{required}}$ for KLL. - -**What stays at the root.** A per-input cap cannot replace constraint (K): with several inputs, $\varepsilon_X + \varepsilon_Y \le \varepsilon_q$ couples them, and one cap per input would fix each input's share in advance (the rejected budget shares). The caps only prune; constraint (K) decides. - - -| Requirement | Stated by | Values | Constrains | Checked by | -|---|---|---|---|---| -| Accuracy target $A_q$ | the workload, per query | metric: absolute (in the value's units, or times the stream's L1 or L2 norm), relative, rank; bound $\varepsilon_q$; failure probability $\delta_q$; scope: per value, whole result, or groups above a share of the records | $\boldsymbol\theta_u$ of every approximate candidate $q$ depends on, through $(\varepsilon_u, \delta_u)$ | constraint (K) **(#1207)** | -| Parameter floor $\boldsymbol\theta^{\min}_{c,i}$, the only field of $r^{\log}_{c,i}$ | every consumer, per input, from $g_c$ and the targets of the queries that may reach it | the smallest $\boldsymbol\theta$ in the family's predefined list whose $\varepsilon$, for the statistic $c$ reads, is at most $\text{cap}_{c,i}$ = min(validity cap, pruning cap); e.g. $k_u \ge k_{\text{required}}$ | $\boldsymbol\theta_u$ | $\alpha^{\log}$, constraint (G): $\boldsymbol\theta_u \succeq \boldsymbol\theta^{\min}_{c,i}$ | -| Consumer timing, $r^{\mathrm{phys}}_{c,i}$ from $\tau_c$ | the consumer's own $p^{\mathrm{phys}}_c$ | when $c$ runs: ingestion or query time | $\tau_u$, $\ell_u$ | $\alpha^{\mathrm{phys}}$, constraint (H) | -| Consumer schedule, $r^{\mathrm{phys}}_{c,i}$ from $\sigma_c$ or $\kappa$ **(#1207)** | a materialized consumer's own $\sigma_c$; for $\ell_c$ = None, the times $T_\kappa$ of each context $\kappa$ it runs in (from its query's $\mathcal T_q$) | interval and phase | $\sigma_u$: the producer delivers on those times | $\alpha^{\mathrm{phys}}$, constraint (H): in Stage 2 for a materialized consumer; per context in Stage 3 for $\ell_c$ = None | -| Deployment capability | the deployment and data | the source has an ingestion pipeline; the executor supports the family; query-time results may be kept across evaluations | $\tau$, $\operatorname{op}$, (QueryTime, AcrossEvaluations) | candidate generation: such candidates are not in $\mathcal C$ | -| Latency, resource budgets **(#1207)** | the workload | per-query latency bound; memory and CPU budgets | the whole plan | constraints (L) and budgets | +| Dimension | Values and meaning | Code reference or target | +|---|---|---| +| Parameters | A predefined configuration list per family: KLL and KMV `k`; DDSketch `alpha`; HLL `precision`; CMS and CountSketch `width` and `depth`; heap variants also `heap_size`; UnivMon its sizing parameters. Exact operators have no sketch parameters. | `SketchParams` in `crates/types/src/ir/schema/sketch.rs`; predefined lists are the #1203 target | +| Window layout — Extension | No layout, or tumbling windows with a positive width. Candidate widths come from supported divisors derived from workload ranges and steps. Sliding merge/subtraction layouts remain future work. Window count is derived from range and width; retention amount is selected later. | `WindowLayout`; `pass2/memo/window_candidates.rs`; #1160 | +| Timing | `IngestionTime` or `QueryTime` | `ExecutionTiming` | +| Lifetime | `None`: discard after the consuming execution; `OneEvaluation`: retain until consumers within that evaluation finish; `AcrossEvaluations`: retain across evaluations in the specified period. A non-`None` lifetime means materialized. | `Lifetime`; #1200 / #1214 | +| Medium — Extension | `Memory` or `Disk`, carried by a materialized lifetime | `Medium`; #1214 | +| Schedule — Extension | For `OneEvaluation`, a periodic interval and phase; for windowed `AcrossEvaluations`, a tumbling-window grid phase. Unmaterialized candidates have no independent schedule. | `Schedule`; #1200 PR 2 | -Accuracy metrics that may be added later, when a primitive needs them: top-k recall or precision, and a false-positive rate for membership summaries. +Error guarantees are derived from parameters and the statistic being read. They include an error metric, a bound, and a failure probability; exact computation introduces no approximation error. The same state may support different readouts, so its guarantee must be interpreted for the relevant statistic. -How requirements map to properties: +The extended property tuples add window layout and schedule: -``` -A_q (workload) ──► θ_u of every approximate input (through ε_u, δ_u; constraint (K)) -parameter floor θ^min r^log ──► θ_u (from cap = min(validity cap of g_c, pruning cap ε_q / λ_{q,c,i})) -τ_c r^phys = f(p^phys_c) ──► τ_u, ℓ_u -σ_c or T_κ r^phys = f(p^phys_c) ──► σ_u -G_{c,i}'s identity (family, query window) ──► op_u, ω_u (by construction: no requirement) -deployment ──► τ, op, ℓ (not generated) -latency, budgets ──► every dimension, through w_v -``` +$$ +p_v^{\mathrm{log}}=(\boldsymbol\theta_v,\omega_v),\qquad +p_v^{\mathrm{phys}}=(\tau_v,\ell_v,\sigma_v). +$$ -### Properties and requirements at a glance (decided 2026-11-05) +### Logical requirements -A field is kept only when it is a real choice: anything the expression, $G_{c,i}$'s identity or the consumer's own properties already fix is not repeated. +The proposed `LogicalRequirement` stores a **parameter floor**: the weakest configuration in a family's ordered list that satisfies the applicable input error cap. Section 4 defines the cap and distinguishes validity from pruning. It does not allocate a fixed share of the query's error budget to each input. -| Struct | Field | Values | Symbol | `new` rejects | -|---|---|---|---|---| -| `LogicalProperties` ($p^{\log}_v$) | `parameters` | `Option`: θ from the family's predefined list; None for an exact operator | $\boldsymbol\theta_v$ | parameters of a family other than the operator's; parameters on an exact operator | -| | `window_layout` | `None`, or `Tumbling { width_ms }` (later: sliding by merge or subtraction) | $\omega_v$ | a window on an operator that cannot be windowed; width 0 | -| `LogicalRequirement` ($r^{\log}_{c,i}$) | `parameter_floor` | `Option`: the smallest θ in the list whose $\varepsilon$ ≤ $\text{cap}_{c,i}$ | $\boldsymbol\theta^{\min}_{c,i}$ | a floor not in the family's predefined list | -| `PhysicalProperties` ($p^{\mathrm{phys}}_v$) | `timing` | IngestionTime, QueryTime | $\tau_v$ | (IngestionTime, OneEvaluation) | -| | `lifetime` | None, OneEvaluation(Medium), AcrossEvaluations(Medium); Medium ∈ {Memory, Disk} | $\ell_v$ | | -| | `schedule` | None; Periodic { interval, phase }; TumblingWindowGrid { phase } | $\sigma_v$ | the schedule combinations listed above | -| ($r^{\mathrm{phys}}_{c,i}$) | none | a function of the consumer's $p^{\mathrm{phys}}_c$: its $\tau_c$, and its $\sigma_c$ or each context's $T_\kappa$ | | | +For KLL, a larger `k` satisfies a smaller minimum `k`. For DDSketch, a smaller `alpha` can be stronger. Multi-parameter families require the family's comparison rule, including parameters needed for failure guarantees. A single floor is valid only if the accepted configurations form an upward-closed set with a representable least element. Otherwise, compatibility must use an explicit admissible set or predicate; it must not invent a unique minimum. -Not fields, because something else already fixes them: the summary family ($\operatorname{op}_v$'s argument, part of the group identity); the window count (query range ÷ width; retention is $h_v$); a family or window requirement on an input ($G_{c,i}$'s identity, and the schedule check); the statistic a consumer reads (its own operator, used once to compute $\boldsymbol\theta^{\min}$); which evaluation produced the output (reuse across evaluations is the producer's $\ell$, a cost choice). +The required input group's identity already constrains the state family and data/window semantics. An exact producer satisfies an approximation floor only if it is also a valid producer of that group and input type. -**α^log** = `satisfies_logical(p^log_u, r^log_{c,i})`: $\boldsymbol\theta_u \succeq \boldsymbol\theta^{\min}_{c,i}$, compared per family on the parameters that set $\varepsilon$ (KLL `k` ≥; CMS / CountSketch `width` ≥; DDSketch `alpha` ≤; HLL `precision` ≥; KMV `k` ≥; UnivMon and the heap variants on each sizing parameter). No floor, or an exact producer, always satisfies. +### Physical requirements -**α^phys** = `satisfies_physical(p^phys_u, p^phys_c)`, timing and lifetime: +Physical requirements are derived from the consumer's timing and schedule, rather than stored redundantly. For an unmaterialized consumer, the schedule comes from its execution context during selection. Stage 2 checks the parts known from candidate properties; Stage 3 checks any remaining context-dependent delivery condition. -| Consumer $\tau_c$ ↓ / producer $(\tau_u, \ell_u)$ → | (QT, None) | (QT, OneEval) | (QT, AcrossEval) | (IT, None) | (IT, AcrossEval) | +| Consumer timing | QT, None | QT, OneEvaluation | QT, AcrossEvaluations | IT, None | IT, AcrossEvaluations | |---|---|---|---|---|---| -| QueryTime | ✅ | ✅ | ✅ | ❌ an unmaterialized ingestion-time output is gone by query time | ✅ | -| IngestionTime | ❌ | ❌ | ❌ | ✅ | ✅ | +| QueryTime | Allowed | Allowed | Allowed | Rejected | Allowed | +| IngestionTime | Rejected | Rejected | Rejected | Allowed | Allowed | -and schedule, for a producer with $\ell_u \ne$ None, against the consumer's times (its own $\sigma_c$ in Stage 2 when $\ell_c \ne$ None; each context's $T_\kappa$ in Stage 3 when $\ell_c$ = None): +QT means QueryTime; IT means IngestionTime. Schedule, data coverage, and retention checks still apply to allowed combinations. -| Producer $\sigma_u$ | Delivers on consumer times $(I, \varphi)$ when | -|---|---| -| none ($\ell_u$ = None) | always: it runs inside the consumer's context, chosen by Stage 3 ($x_{\kappa,v}$) | -| Periodic $(I_\sigma, \varphi_\sigma)$ | $I_\sigma \mid I$ and $\varphi \equiv \varphi_\sigma \pmod{I_\sigma}$; a non-periodic consumer is never served | -| TumblingWindowGrid $\varphi_g$, width $w$ | $w \mid I$ and $\varphi \equiv \varphi_g \pmod w$; a non-periodic consumer is never served | - -| Struct | Set by | When | -|---|---|---| -| `LogicalProperties` | Pass 1 (one candidate per θ in the list); Pass 2 rules (window layouts) | when the candidate is created | -| `LogicalRequirement` | the consumer: floor from the validity cap of `LogicalRequirement::error_propagation(consumer)` | when the consumer is created; Pass 2 tightens it with the pruning cap when it builds the logical edges | -| `PhysicalProperties` | Stage 2 physical expansion (every valid $(\tau, \ell, \sigma)$) | Stage 2 | -| $r^{\mathrm{phys}}_{c,i}$ | derived from $p^{\mathrm{phys}}_c$ | checked in Stage 2 (timing, lifetime; schedule of a materialized consumer) and Stage 3 (schedule per context) | - -## 2. Decision tree of properties - -Each memo group's candidates are the leaves of a tree. The first level chooses the expression $e_v$; every later level decides one property dimension. A path from the root to a leaf is one physical candidate in $\mathcal C$. - -```mermaid -flowchart TD - G["Memo group G
(logical result)"] --> R1["e_v: exact operator"] - G --> R2["e_v: SummaryAgg[KLL]"] - G --> R3["e_v: SummaryAgg[DDSketch]"] - G --> R4["e_v: …other operators
(prune: executor support)"] - - R2 --> T1["θ: k = 200
ε = 0.008"] - R2 --> T2["θ: k = 345
ε = 0.005"] - R2 --> T3["θ: k = …
(prune: ε above the ε_q of
every query reaching G)"] - - T2 --> W1["ω: no windows"] - T2 --> W2["ω: tumbling, width w
(prune: w must divide the range and
step of some query reaching G)"] - - W2 --> P1["τ = QueryTime"] - W2 --> P2["τ = IngestionTime
(prune: no ingestion pipeline)"] - - P1 --> L1["ℓ = None"] - P1 --> L2["ℓ = OneEvaluation"] - P1 --> L3["ℓ = AcrossEvaluations
(prune: deployment keeps
no query-time results)"] - P2 --> L4["ℓ = None"] - P2 --> L5["ℓ = AcrossEvaluations"] - P2 -. invalid .-> L6["ℓ = OneEvaluation"] - - L2 --> S1["σ: Periodic (I, φ)
from the gcd closure
of the queries' 𝒯_q"] - L3 --> S2["σ: grid phase φ mod w"] - L5 --> S3["σ: grid phase 0"] -``` +**Periodic versus grid.** A periodic schedule with interval ten seconds and phase two produces at 2, 12, 22, … seconds. A five-second tumbling grid with phase zero partitions the input into windows [0, 5), [5, 10), … . The first describes evaluation times; the second describes window boundaries. Grid alignment alone does not ensure sufficient retained history. -The levels, top to bottom: +For the proposed periodic consumer path, let the consumer interval and phase be $I$ and $\varphi$. Let the producer interval and phase be $I_s$ and $\varphi_s$, and its tumbling width and grid phase be $w$ and $\varphi_g$. -| Level | Decides | Stage | Values offered | Pruned when | Code | -|---|---|---|---|---|---| -| 1 | expression $e_v$ ($\operatorname{op}_v$ with its family) | Stage 1, Pass 1 and Pass 2 rules | each way to compute the group | the executor cannot build the family | Pass 1 ([`per_computation_candidates.rs`](../../../crates/logical-asap-optimization/src/pass1/per_computation_candidates.rs)); Pass 2 rules ([`rules/`](../../../crates/logical-asap-optimization/src/pass2/memo/rules)) | -| 2 | $\boldsymbol\theta_v$ | Stage 1, Pass 1 | the family's predefined list | $\varepsilon_v$ alone exceeds the $\varepsilon_q$ of every query that reaches the group | Pass 1 sizing per `accuracy_budget_parts` ([`memo_population.rs`](../../../crates/logical-asap-optimization/src/pass1/memo_population.rs)); a predefined list in #1203 PR 2 | -| 3 | $\omega_v$ **(#1207)** | Stage 1, Pass 2 | tumbling widths from the gcd closure of the ranges and steps of the queries reaching the group | no such query's range and step is a multiple of the width | Pass 2 ([`window_candidates.rs`](../../../crates/logical-asap-optimization/src/pass2/memo/window_candidates.rs), [`state_groups.rs`](../../../crates/logical-asap-optimization/src/pass2/memo/state_groups.rs)) | -| 4 | $\tau_v$ | Stage 2 | QueryTime; IngestionTime | the source has no ingestion pipeline | `enumerate_candidate_configurations` ([`configurations.rs`](../../../crates/physical-asap-optimization/src/catalog/configurations.rs)) | -| 5 | $\ell_v$ | Stage 2 | None, OneEvaluation, AcrossEvaluations | invalid with $\tau_v$; the deployment keeps no query-time results across evaluations | same function, with `DeploymentCapabilities::query_time_results_retained` | -| 6 | $\sigma_v$ **(#1207)** | Stage 2 | the gcd closure $\Sigma$ of the schedules $\mathcal T_q$ of the queries reaching the group | only offered when $\ell_v \ne$ None | Stage 3 MILP today; Stage 2 in #1200 PR 2 | +| Producer schedule | Required alignment | +|---|---| +| Periodic | $I_s\mid I$ and $\varphi\equiv\varphi_s\pmod{I_s}$ | +| Tumbling grid | $w\mid I$ and $\varphi\equiv\varphi_g\pmod w$ | +| None | Runs in the consuming context; no independent delivery schedule | -What the tree does not decide, because it is a choice of the MILP over $\mathcal C$ and $\mathcal E$: -- **Which producer each consumer input reads:** $y_{u,c,i}$. Two consumers of one group may read different leaves, e.g. one reads `k = 200` and another `k = 345`. -- **Where an unmaterialized candidate runs (#1207):** a candidate with $\ell_v$ = None runs inside each reading context $\kappa$. -- **How much a materialized candidate retains (#1207):** $h_v$, from the longest range among the queries its selected consumers serve, constraint (R). +The supplied formulation also permits a single, non-periodic evaluation. The interval/phase interface below covers only periodic consumers. Rejecting non-periodic consumers is a current scope restriction of that proposed interface, not a mathematical impossibility; supporting them needs explicit evaluation-time membership checks. -### Order of decisions +### Invalid combinations -**The order does not affect correctness.** The leaves are the product of the dimensions' value sets, minus the invalid combinations and the pruned values. Every prune rule depends only on the value it removes and on values decided above it, so any order that decides a pruned dimension after the dimensions its prune rule reads yields the same $\mathcal C$. +Checked constructors reject these combinations under the proposed schedule model: -**The order affects the efficiency of enumeration.** The number of nodes visited is $\sum_{d} \prod_{d' \le d} |\text{values kept at } d'|$, so a dimension that prunes much should come early, and a dimension whose values multiply without pruning should come late. Four considerations set the order above: +| Combination | Reason | +|---|---| +| IngestionTime with OneEvaluation | No query evaluation lifetime at ingestion time | +| None with an independent schedule | Execution follows the consuming context | +| Materialized lifetime without its required schedule | Delivery times or window grid would be unspecified | +| OneEvaluation with a grid, or windowed AcrossEvaluations with a periodic schedule | Schedule kind does not match this model's lifetime representation | +| Zero interval/width, or phase outside the canonical interval/width | Invalid schedule representation | +| Ingestion-time grid with nonzero phase | Unsupported by the draft's epoch-anchored ingestion representation | -1. **Logical before physical.** $e_v$, $\boldsymbol\theta_v$ and $\omega_v$ change the result's values or error; $\tau_v$, $\ell_v$ and $\sigma_v$ change only when and how long it is computed and kept. Deciding $p^{\log}$ first lets Stage 1 hash-cons groups and compute $\alpha^{\log}$ once, before Stage 2 multiplies each logical candidate by its $p^{\mathrm{phys}}$ alternatives. -2. **Strong prunes early.** The operator is pruned by the executor, and $\boldsymbol\theta_v$ by the accuracy targets, before the window and physical levels multiply them. -3. **Dependent dimensions adjacent.** $\ell_v$ is constrained by $\tau_v$, and $\sigma_v$ exists only when $\ell_v \ne$ None and its kind follows $\ell_v$. Deciding $\tau \to \ell \to \sigma$ prunes each invalid combination at the first level where it can be detected. -4. **Deployment-wide prunes as early as they can go.** Without an ingestion pipeline, the IngestionTime subtree is pruned once per candidate. Moving $\tau$ above $\omega$ would prune it once per group instead; this is an alternative order with the same leaves. +A tumbling grid also needs a known positive width. The `PhysicalProperties` constructor alone cannot validate a width held in logical properties; physical candidate construction must validate the pair. Non-windowed AcrossEvaluations schedules need an explicit representation before that alternative can be offered. -Alternative orders worth comparing: -- **$\tau$ first.** Where most sources have no ingestion pipeline, deciding timing at the group level prunes half the tree before $\boldsymbol\theta$ and $\omega$ multiply it. -- **$\boldsymbol\theta$ last among logical dimensions.** $\boldsymbol\theta$ has the most values. Deciding $\omega$ before $\boldsymbol\theta$ visits fewer nodes when many widths are pruned by the queries' ranges. -- **Requirement-driven (top-down) vs producer-driven (bottom-up).** Top-down enumeration starts from each consumer's $r_{c,i}$ and offers only the values it accepts; bottom-up offers every value and lets $\alpha$ filter $\mathcal E$. Top-down visits fewer nodes but must union the requirements of all consumers of a group, so that sharing between consumers is not lost. +## 3. Pipeline: interfaces and rules -## 3. Data structures and interfaces per stage +| Stage | Input | Output | Responsibility | +|---|---|---|---| +| Stage 0 | Query IR | `DagOfLogicalMemoGroups` | Hash-cons logical expressions | +| Stage 1, Pass 1 | Group DAG | `Memo` | Add initial logical candidates and intermediate groups | +| Stage 1, Pass 2 | Memo | `LogicalASAPMemoCandidates` | Add reuse alternatives and compute logical edges | +| Stage 2 | Logical memo candidates | `PhysicalASAPMemoCandidates` | Expand physical alternatives and compute compatibility | +| Stage 3 | Physical memo candidates | `DagOfSelectedCandidates` | Select implementations, dependencies, and executions | +| Realizer/export | Selected DAG | `PhysicalASAPDAG` | Build and export the executable graph | -This section gives the target interface of each stage after the formulation refactor (#1207). Each stage's boundary type carries the notation that stage defines, and no stage reads a notation a later stage defines: logical stages carry no physical properties. +Enumeration proceeds through expression, parameters, window layout, timing, lifetime, and schedule. Executor and deployment restrictions prune unsupported alternatives as soon as their prerequisites are known. Accuracy pruning is allowed only under the conditions in Section 4. Reordering dimensions preserves the candidate set only when the same alternatives and sound pruning predicates are retained. -| Stage | Input | Output (boundary type) | Notation it defines | Sub-issue | -|---|---|---|---|---| -| Stage 0: frontends, hash-consing | queries | `DagOfLogicalMemoGroups` | $G$, query roots | — | -| Stage 1, Pass 1 | `DagOfLogicalMemoGroups` | `Memo` | $v = (e_v, p^{\log}_v)$, $r^{\log}_{c,i}$, $\theta$ list | #1203 | -| Stage 1, Pass 2 | `Memo` | `LogicalASAPMemoCandidates` | rule-added candidates; logical edges with $\alpha^{\log}$ | #1203, #1201 | -| Stage 2 | `LogicalASAPMemoCandidates` | `PhysicalASAPMemoCandidates` | $p^{\mathrm{phys}}_v = (\tau, \ell, \sigma)$, $r^{\mathrm{phys}}_{c,i}$, physical edges $\mathcal E$ with $\alpha^{\mathrm{phys}}$ | #1200, #1201 | -| Stage 3 | `PhysicalASAPMemoCandidates` | `DagOfSelectedCandidates` | contexts $\kappa$; cost $w_v$ (`asap-cost`); $x$, $y$, $z$, $t$, $h$; constraints (A)–(K), (R), (L), budgets | #1204 | -| Realizer, export | `DagOfSelectedCandidates` | `PhysicalASAPDAG` | — | #1200 (export fields) | +Rules add groups through `find_or_add_group` and candidates through `add_candidate`. They match group expressions, avoid duplicating an existing expression/property pair, and do not copy consumers to make a new producer visible. After logical generation finishes, edge construction records the possible producer choices. -### Stage 0: frontends and hash-consing +### Stage 0: groups -No change. One group per distinct logical sub-expression; the queries' roots point at groups. +`hash_consed` in `crates/logical-asap-optimization/src/stage0.rs` identifies shared logical expressions and query roots. The following preserves the existing group representation described in the draft. Candidate ownership is exposed through the group in the proposed Stage 1 interface. ```rust -// crates/types/src/stages/memo.rs pub struct MemoGroupId(pub u32); -/// G: one logical result or one state family over specified input data. pub struct MemoGroup { - pub logical_node: Rc, // expression; no physical fields (#1200) - pub child_groups: Vec, // producer groups G_i - pub parents: Vec, // derived: consumers' groups - pub queries: Vec, // derived: queries that reach it + pub logical_node: Rc, + pub child_groups: Vec, + pub parents: Vec, + pub queries: Vec, + // Empty after Stage 0; populated by Stage 1. + pub candidates: Vec, } pub struct DagOfLogicalMemoGroups { @@ -293,317 +182,492 @@ pub struct DagOfLogicalMemoGroups { } ``` -### Stage 1, Pass 1: candidates of each group +`OperatorNode` recursively contains chosen child nodes and therefore describes a fixed plan fragment. `Operator` contains input group IDs and leaves their implementations open. A candidate needs its own expression because a group may offer several ways to compute the same result; it cannot reuse the group's representative `logical_node` as every candidate's expression. -Pass 1 adds every way to compute each group as candidates. Every intermediate result is its own group (no private nodes), and a summary family's sizes are candidates of one state group. +Replacing the group's representative node and child list with a single group-based expression, or removing the one-field `CandidateExpression` wrapper, remains an optional simplification from the review. Neither is required here. + +### Stage 1, Pass 1: initial candidates + +Implementation locations: `crates/logical-asap-optimization/src/pass1/per_computation_candidates.rs` and `memo_population.rs`. The #1203 target gives every intermediate result its own group; candidates contain one operator rather than private operator chains. + +| Alternative | Added representation | +|---|---| +| Exact | Exact candidate in the result group | +| Exact accumulator | Accumulator-state group plus a finalize candidate in the result group | +| Summary | One state group per supported family; one build candidate per listed parameter configuration; separate readout candidates | +| Fused build (#1203 2c) | Candidate with `fused_groups` identifying additional results; selection must enforce their availability consistently | ```rust -// crates/types/src/stages/candidates.rs -pub struct CandidateForMemoGroupId { pub group: MemoGroupId, pub index: usize } +pub struct CandidateForMemoGroupId { + pub group: MemoGroupId, + pub index: usize, +} pub struct Memo { pub groups: Vec, pub query_roots: Vec>, - pub candidates: Vec>, // by group } + impl Memo { - /// Hash-consing: the group with this expression, added if new. - pub fn find_or_add_group(&mut self, expression: Operator) -> MemoGroupId; - pub fn add_candidate(&mut self, group: MemoGroupId, candidate: CandidateForMemoGroup) -> CandidateForMemoGroupId; + pub fn find_or_add_group( + &mut self, + expression: Operator, + ) -> MemoGroupId; + + pub fn add_candidate( + &mut self, + group: MemoGroupId, + candidate: CandidateForMemoGroup, + ) -> CandidateForMemoGroupId; + pub fn candidate(&self, id: CandidateForMemoGroupId) -> &CandidateForMemoGroup; } -/// v = (e_v, p_v^log) +pub struct CandidateExpression { + pub operator: Operator, +} + pub struct CandidateForMemoGroup { - pub expression: CandidateExpression, // e_v - pub logical_properties: LogicalProperties, // p_v^log - pub input_requirements: Vec, // r^log_{c,i}, one per input group + pub expression: CandidateExpression, + pub logical_properties: LogicalProperties, + pub input_requirements: Vec, pub fused_groups: Vec, - /// Which rule added it. Not in the formulation: kept for `stage_pipeline --text` - /// and the viewer (approved 2026-11-05). + // Diagnostic provenance; not a planning decision. pub origin: OriginOfCandidateForMemoGroup, } -/// e_v = op_v(G_{v,1}, …, G_{v,n}): one operator, its children are groups. -pub struct CandidateExpression { pub operator: Operator } -/// p_v^log = (θ_v, ω_v). The family is the operator's argument, not a property. -/// `new` rejects parameters of another family (or on an exact operator) and a -/// window on an operator that cannot be windowed. pub struct LogicalProperties { - parameters: Option, // θ_v, from the family's predefined list; None for exact - window_layout: Option, // ω_v + parameters: Option, + window_layout: Option, +} + +pub enum WindowLayout { + Tumbling { width_ms: u64 }, } + impl LogicalProperties { - pub fn new(expression: &CandidateExpression, parameters: Option, - window_layout: Option) -> Result; + pub fn new( + expression: &CandidateExpression, + parameters: Option, + window_layout: Option, + ) -> Result; + pub fn parameters(&self) -> Option<&SketchParams>; pub fn window_layout(&self) -> Option<&WindowLayout>; } -/// The window count is the group's query range ÷ width; retention is h_v (Stage 3). -pub enum WindowLayout { Tumbling { width_ms: u64 } } -/// Approved 2026-11-05. The family's predefined θ list, each with its error bound -/// derived through `estimators::local_guarantee` for a given statistic. pub fn parameter_list(family: SketchAlgorithm) -> &'static [SketchParams]; -/// r^log_{c,i}: only what G_{c,i}'s identity does not fix. `new` rejects a floor -/// that is not in the family's predefined list. pub struct LogicalRequirement { - /// θ^min_{c,i}: the smallest θ in the list whose ε, for the statistic the - /// consumer reads, is at most cap_{c,i} = min(validity cap of g_c, pruning cap). parameter_floor: Option, } + impl LogicalRequirement { - pub fn new(parameter_floor: Option) -> Result; + pub fn new( + parameter_floor: Option, + ) -> Result; + pub fn parameter_floor(&self) -> Option<&SketchParams>; - /// g_c: λ_{c,i} per input, read from the operator and its arguments, and the - /// validity cap (None when g_c is linear). Sets each input's first floor; - /// Stage 3 composes the factors into λ_{q,c,i}. - pub fn error_propagation(consumer: &Operator) -> ErrorPropagation; } -/// Approved 2026-11-05. -pub struct ErrorPropagation { pub factor_per_input: Vec, pub validity_cap: Option } -/// α^log_{u,c,i}: θ_u ⪰ θ^min_{c,i}, compared per family on the parameters that set ε. -pub fn satisfies_logical(producer: &LogicalProperties, requirement: &LogicalRequirement) -> bool; +pub fn satisfies_logical( + producer: &LogicalProperties, + requirement: &LogicalRequirement, +) -> bool; ``` -The logical and physical properties have the same shape: private fields, a checked `new`, one accessor per dimension, one error type per side (`InvalidLogicalProperties`, `InvalidPhysicalProperties`), and one `satisfies_*` function. Only the logical side has a stored requirement; the physical requirement is a function of the consumer's own physical properties. +The floor representation has the applicability condition stated in Section 2. Creating requirements also needs the input family's configuration domain; a constructor without that context can validate only the information encoded in its argument. Error-model interfaces are grouped with accuracy in Section 4. -The error bound is not stored: it depends on the statistic the consumer reads, which is the consumer's own operator argument. It is used once, when the consumer's floor is computed: the smallest θ in `parameter_list(family)` whose `local_guarantee` for that statistic has ε ≤ cap. The validity cap is known when the consumer is created (`LogicalRequirement::error_propagation`). The pruning cap needs $\lambda_{q,c,i}$ and the queries that may reach $c$, so Pass 2 builds each input's final requirement with `LogicalRequirement::new` when it builds the logical edges. +Migration targets: remove private `root_node` chains, duplicate `input_groups`, and `accuracy_budget_parts`. The proposed predefined parameter lists replace sizing one candidate per budget partition. -Deleted: `root_node` with private nodes, `input_groups` (the expression's children), `accuracy_budget_parts`, the `SharedBuild` and `SummaryResult` origins. +### Stage 1, Pass 2: reuse rules and logical edges -Concepts not in the formulation: `origin: OriginOfCandidateForMemoGroup` (Pass1, SummaryInput, TumblingWindows, Rollup, TopKReuse, WidenedScans) records which rule added a candidate; nothing in planning reads it. +The following preserves the supplied rule inventory and tracking references. Paths are relative to `crates/logical-asap-optimization/src/pass2/memo/`. -### Stage 1, Pass 2: rules and logical edges - -Rules add candidates to existing groups, or add groups. A rule never copies a consumer. After the last rule, Pass 2 builds the logical edges once. +| Rule / function | File | Result or migration target | +|---|---|---| +| Local windows: `add_local_window_composition` | `window_candidates.rs` | Add layouts to the state group; #1160 removes copied window consumers | +| Scan widening: `add_scan_label_candidates` | `rules/scan_labels.rs` | Add a widened scan group and projections in the original groups; #1222 | +| Summary inputs: `factor_summary_inputs` | `rules/summary_inputs.rs` | Expose projections/filters as input groups; #1218 moves this into initial population | +| Top-k reuse: `add_topk_candidates` | `rules/topk.rs` | Read a compatible larger top-k result with an exact-input requirement; #1220 | +| Finer aggregate: `add_finer_aggregate_candidates` | `rules/finer_aggregate.rs` | Filter and roll up finer groups, using separate intermediate groups; #1220 | +| Rollup: `add_rollup_candidates` | `rules/rollup.rs` | Add compatible finer-state merges; #1221 | +| Window rollup: `add_window_rollup_candidates` | `rules/rollup.rs` | Merge shorter windows into longer-range results; #1221 | +| Summary results: `factor_summary_results` | `rules/summary_results.rs` | Removed once Pass 1 exposes readouts; #1218 | +| Shared configurations: `add_state_configuration_candidates` | `rules/shared_configuration_readers.rs` | Remove copied consumers once producer variants are selectable; #1203 2b | +| Shared state groups: `add_shared_summary_state_groups` | `state_groups.rs` | Remove per-size groups; #1203 2b / #1160 | + +After the rules, build logical edges and apply any proven-safe accuracy pruning. `input_index_of_consumer` replaces the ambiguous `input` field name throughout this document's edge interfaces. ```rust -// crates/logical-asap-optimization/src/pass2 -/// A rule: reads the memo, adds candidates or groups. fn apply(memo: &mut Memo) -> Result<(), Pass2RuleError>; -// crates/types/src/stages/logical_memo_candidates.rs -/// (u, c, i) over logical candidates: u is a candidate of G_{c,i}. pub struct LogicalEdge { pub producer: CandidateForMemoGroupId, pub consumer: CandidateForMemoGroupId, - pub input: usize, - pub satisfies_logical_requirement: bool, // α^log + pub input_index_of_consumer: usize, + pub satisfies_logical_requirement: bool, } + pub struct LogicalASAPMemoCandidates { pub memo: Memo, - pub edges: Vec, // grouped by (consumer, input) + pub edges: Vec, } ``` -### Stage 2: physical candidates and edges +### Stage 2: physical alternatives -Stage 2 expands each logical candidate into one physical candidate per valid $(\tau, \ell, \sigma)$, gives each input its physical requirement, and turns each logical edge into the physical edges between the two candidates' physical alternatives. +`enumerate_candidate_configurations` in `crates/physical-asap-optimization/src/catalog/configurations.rs` expands each logical candidate using the dimensions and checks in Section 2. It creates no new logical candidates. Each logical edge is expanded between the relevant physical alternatives. ```rust // crates/types/src/physical/properties.rs -pub enum ExecutionTiming { IngestionTime, QueryTime } -pub enum Medium { Memory, Disk } -pub enum Lifetime { None, OneEvaluation(Medium), AcrossEvaluations(Medium) } +pub enum ExecutionTiming { + IngestionTime, + QueryTime, +} + +pub enum Medium { + Memory, + Disk, +} + +pub enum Lifetime { + None, + OneEvaluation(Medium), + AcrossEvaluations(Medium), +} pub enum Schedule { - Periodic { interval_ms: u64, phase_ms: u64 }, // OneEvaluation - TumblingWindowGrid { phase_ms: u64 }, // AcrossEvaluations + Periodic { interval_ms: u64, phase_ms: u64 }, + TumblingWindowGrid { phase_ms: u64 }, +} +pub struct PhysicalProperties { + timing: ExecutionTiming, + lifetime: Lifetime, + schedule: Option, } -/// p_v^phys = (τ, ℓ, σ); `new` rejects the invalid combinations (Section 1). -pub struct PhysicalProperties { timing: ExecutionTiming, lifetime: Lifetime, schedule: Option } impl PhysicalProperties { - pub fn new(timing: ExecutionTiming, lifetime: Lifetime, schedule: Option) -> Result; + pub fn new( + timing: ExecutionTiming, + lifetime: Lifetime, + schedule: Option, + ) -> Result; pub fn timing(&self) -> ExecutionTiming; pub fn lifetime(&self) -> Lifetime; pub fn schedule(&self) -> Option<&Schedule>; - pub fn is_materialized(&self) -> bool; // ℓ ≠ None + pub fn is_materialized(&self) -> bool; } -/// A periodic schedule φ + I·ℤ: a materialized consumer's own times, or a -/// context's times T_κ (Stage 3). -pub struct EvaluationSchedule { interval_ms: u64, phase_ms: u64 } +pub struct EvaluationSchedule { + interval_ms: u64, + phase_ms: u64, +} impl PhysicalProperties { - /// Whether this producer delivers on `times` (the schedule table in section 1); - /// always true when ℓ = None. pub fn delivers_on(&self, times: &EvaluationSchedule) -> bool; } -/// α^phys_{u,c,i} = r^phys_{c,i} as a function of the consumer's own p^phys_c: -/// timing and lifetime, and, for a materialized consumer, `delivers_on` its σ_c. -/// For a consumer with ℓ = None, Stage 3 checks `delivers_on(T_κ)` per context. -pub fn satisfies_physical(producer: &PhysicalProperties, consumer: &PhysicalProperties) -> bool; - -// crates/types/src/stages/physical +pub fn satisfies_physical( + producer: &PhysicalProperties, + consumer: &PhysicalProperties, +) -> bool; pub struct PhysicalCandidateId(pub usize); pub struct PhysicalEdgeId(pub usize); -/// One physical alternative of a logical candidate. pub struct PhysicalCandidate { - pub logical: CandidateForMemoGroupId, // e_v and p_v^log - pub properties: PhysicalProperties, // p_v^phys + pub logical: CandidateForMemoGroupId, + pub properties: PhysicalProperties, } -/// (u, c, i) over physical candidates. pub struct PhysicalEdge { pub producer: PhysicalCandidateId, pub consumer: PhysicalCandidateId, - pub input: usize, - pub satisfies_logical_requirement: bool, // α^log - pub satisfies_physical_requirement: bool, // α^phys + pub input_index_of_consumer: usize, + pub satisfies_logical_requirement: bool, + pub satisfies_physical_requirement: bool, } pub struct PhysicalASAPMemoCandidates { pub logical: LogicalASAPMemoCandidates, - pub candidates: Vec, // C + pub candidates: Vec, pub query_roots: Vec>, pub pruned_candidates: Vec>, - // edges stored per (consumer, input); see `edges()` } impl PhysicalASAPMemoCandidates { - pub fn edges(&self) -> impl Iterator; // E - pub fn admissible_edges_into(&self, consumer: PhysicalCandidateId, input: usize) -> &[PhysicalEdgeId]; // P_{c,i} - pub fn admissible_edges_out_of(&self, producer: PhysicalCandidateId) -> &[PhysicalEdgeId]; + pub fn edges(&self) -> impl Iterator; + pub fn admissible_edges_into( + &self, + consumer: PhysicalCandidateId, + input_index_of_consumer: usize, + ) -> &[PhysicalEdgeId]; + pub fn admissible_edges_out_of( + &self, + producer: PhysicalCandidateId, + ) -> &[PhysicalEdgeId]; +} + +// Identifies a rejected alternative by content: it has no allocated candidate ID. +pub struct PhysicalCandidateKey { + pub logical: CandidateForMemoGroupId, + pub properties: PhysicalProperties, } ``` -Deleted: `Materialization`, `PhysicalCandidate::{timing, materialization, operator_node}`, `producer_candidates_per_input` as the interface, `compatible_timing`, the reader constraints `ReaderConstraint::{IncompatibleOutputs, AllowedOutputs}` (now $\alpha^{\mathrm{phys}}$ through $\sigma$), `may_keep_across_evaluations_per_sole_reader_query`. +`PhysicalCandidateId` indexes an emitted candidate. `PhysicalCandidateKey` describes an alternative rejected before it received an ID, allowing diagnostics to explain pruning. The admissible-edge accessor returns edge IDs, not the unfiltered producer set from Section 1. + +The `delivers_on` sketch needs access to a tumbling producer's logical window width. Supply that context in its implementation or move the check to the full physical-candidate interface; the phase-only schedule is insufficient by itself. + +Migration targets include replacing `Materialization` and duplicated physical fields with checked properties, and replacing `ReaderConstraint::{IncompatibleOutputs, AllowedOutputs}` with compatibility and schedule checks. Policy constraints involving the number of selected consuming queries are a separate concern; see Section 5. + +### Stage 3: baseline selection and realization -### Stage 3: the MILP +The baseline uses the variables and constraints in the supplied LaTeX. Stage 3 adds no candidates or potential dependencies. -Stage 3 reads the physical candidates and edges and selects the DAG. Variables and constraints are named after the formulation. +| Variable | Meaning | +|---|---| +| $x_v$ | Candidate selected | +| $y_{u,c,i}$ | Producer supplies the specified consumer input | +| $z_{q,v}$ | Candidate supplies the query output | +| $t_v$ | Topological rank | + +| Constraint | Meaning | Encoding | +|---|---|---| +| (A) | One producer per selected consumer input | $\sum_{u\in P_{c,i}}y_{u,c,i}=x_c$ | +| (B) | Chosen producers are selected | $y_{u,c,i}\le x_u$ | +| (C) | One output per query | $\sum_{v\in G_q}z_{q,v}=1$ | +| (D) | Output candidates are selected | $z_{q,v}\le x_v$ | +| (E) | Acyclic dependencies | $t_u+1\le t_c+\lvert\mathcal C\rvert(1-y_{u,c,i})$ | +| (F) | Selected candidates are used | $x_v\le\sum_{(v,c,i)\in\mathcal E}y_{v,c,i}+\sum_{q:v\in G_q}z_{q,v}$ | +| (G) | Logical compatibility | $y_{u,c,i}\le\alpha^{\mathrm{log}}_{u,c,i}$ | +| (H) | Physical compatibility | $y_{u,c,i}\le\alpha^{\mathrm{phys}}_{u,c,i}$ | +| (I) | Domains | $x_v,y_{u,c,i},z_{q,v}\in\{0,1\}$; $0\le t_v\le\lvert\mathcal C\rvert-1$ | + +Constraints are applied for every relevant candidate, input, query, or potential edge. An implementation may allocate edge variables only on admissible edges, making (G) and (H) implicit without changing the baseline feasible set. + +The baseline objective charges each selected candidate a fixed cost over the common horizon: + +$$ +\min\sum_{v\in\mathcal C}w_vx_v. +$$ + +The cost includes input reading, the candidate's own computation, and lifetime-dependent retention. It excludes producers' computation, which is charged at those producers. This objective assumes execution frequencies and retention costs are fixed before optimization and independent of the selected consumers. When that assumption does not hold, use the execution extension in Section 5; do not silently reuse the same cost coefficient. ```rust -// crates/plan-selection/src/milp/model.rs -/// κ: where an ℓ = None candidate runs. Only those candidates are indexed by it. -/// `ContextId` indexes the workload's contexts; the id types inside are illustrative. -pub struct ContextId(pub usize); -pub enum Context { - QueryEvaluation(QueryId), // κ_q - IngestionPipeline(RefreshWindowId), // I_r - Materialized(PhysicalCandidateId), // κ_m: inside a materialized m's computation +struct SelectionVariables { + x: Vec, + y: HashMap, + z: HashMap<(QueryId, PhysicalCandidateId), Variable>, + t: Vec, } -/// T_κ: when the context runs; y_{κ,u,c,i} exists only if `p^phys_u.delivers_on(κ.evaluation_schedule())`. -impl Context { - /// The context's evaluation schedule T_κ (None for a non-periodic query). - pub fn evaluation_schedule(&self) -> Option; + +// Schematic constraint-building entry points; signatures omitted. +// input_satisfaction_constraints: (A) +// producer_selection_constraints: (B) +// query_coverage_constraints: (C), (D) +// acyclicity_constraints: (E) +// utilization_constraints: (F) +// Compatibility is enforced when allocating y on admissible edges. + +pub struct SelectedEdge { + pub producer: SelectedNodeId, + pub consumer: SelectedNodeId, + pub input_index_of_consumer: usize, } -struct SelectionVariables { - x_materialized: Vec, // x_v, ℓ ≠ None - x_in_context: HashMap<(ContextId, PhysicalCandidateId), Variable>, // x_{κ,v}, ℓ = None - y: HashMap<(Option, PhysicalEdgeId), Variable>, // y_{u,c,i} on admissible edges - z: Vec, // z_{q,v} - t: Vec, // t_v, cyclic components only - h: Vec, // h_v, ℓ ≠ None - latency: LatencyVariables, // (L) -} -fn input_satisfaction_constraints(..); // (A) Σ_u y_{u,c,i} = x_c -fn producer_selection_constraints(..); // (B) y_{u,c,i} ≤ x_u -fn query_coverage_constraints(..); // (C), (D) -fn acyclicity_constraints(..); // (E) -fn utilization_constraints(..); // (F) -fn consumer_constraint_constraints(..); // (J): MinimumQueryReaders, HighestSharedQueryMaterialization over reach -fn accuracy_constraints(..); // (K) Σ λ_{q,u} ε_u a_{q,u} ≤ ε_q, Σ δ_u a_{q,u} ≤ δ_q -fn retention_constraints(..); // (R) h_v ≥ need_{c,i} · y_{v,c,i} -fn latency_constraints(..); // (L) -fn resource_budget_constraints(..); // budgets (#1207) -// Objective: min Σ w_v x_v + Σ ρ_κ w_v x_{κ,v} + Σ π_v h_v - -// crates/types/src/stages/physical/graph.rs (unchanged) -pub struct SelectedEdge { pub producer: SelectedNodeId, pub consumer: SelectedNodeId, pub input: usize } pub struct DagOfSelectedCandidates { pub candidate_of_node: Vec, - pub edges: Vec, // decoded from y = 1 + pub edges: Vec, pub node_of_output_reference: Vec, } ``` -(G) $y \le \alpha^{\log}$ and (H) $y \le \alpha^{\mathrm{phys}}$ are not constraints in the model: $y$ exists only on admissible edges. Deleted: `ResultSlot`, `slot_of`, the `e` binaries, schedule groups `z[m, G]`, `N[q]`, `A[q, parts]`, and `p[ctx, c]` for materialized candidates. +The realizer uses the selected dependencies to build the executable graph. Export carries each node's timing, lifetime, and supported schedule fields. Under the context extension, multiple selected execution nodes may refer to the same physical candidate; realization must preserve their contexts rather than merge them solely by candidate ID. -### Realizer and export +## 4. Accuracy and safe pruning — Extension (#1207) -The realizer builds the plan from `DagOfSelectedCandidates`; each node's physical properties come from its `PhysicalCandidate`. The export writes timing and lifetime (and later the schedule) per node of `PhysicalASAPDAG`, changed together with the backend. +The supplied LaTeX checks logical requirements per edge. The proposed extension additionally checks composed accuracy at each query root. This section describes the error model needed for that check; it is not an additional property dimension. -## 4. Rules: how each pass and stage adds candidates +### Local propagation -A rule reads the memo and adds to it only through the interfaces of section 3: -- `Memo::find_or_add_group(expression)` for a group, so an expression that already exists returns its group (hash-consing); -- `Memo::add_candidate(group, CandidateForMemoGroup { expression, logical_properties, input_requirements, fused_groups })` for a candidate $v = (e_v, p^{\log}_v)$ with its $r^{\log}_{v,i}$. +For a consumer, an error function bounds output error from input errors: -A rule never copies a consumer and never edits an existing candidate: every consumer of a group sees a new candidate of that group through $\mathcal E$. Rules match over groups and their candidates' expressions, never over a consumer's private operators. The order of the rules below is the order the code runs them; it affects which groups exist when a later rule matches, not which plans are valid. +$$ +\varepsilon_c\le g_c(\varepsilon_{c,1},\ldots,\varepsilon_{c,n}). +$$ -### Stage 0: hash-consing +Use **beta** for local input factors and **lambda** only for factors composed to a query root: -| Rule | Code | Matches | Adds | -|---|---|---|---| -| Hash-consing | `hash_consed` ([`stage0.rs`](../../../crates/logical-asap-optimization/src/stage0.rs)) | every operator node of every query root | one `MemoGroup` per distinct expression; `query_roots` point at $G_q$. No candidates. | +$$ +g_c(\varepsilon_{c,1},\ldots,\varepsilon_{c,n}) +\le\sum_i\beta_{c,i}\varepsilon_{c,i}. +$$ -### Stage 1, Pass 1: ways to compute each group +| Operation and metric | Applicable bound or condition | +|---|---| +| Exact pass-through | Preserves the input guarantee only when it preserves the metric's semantics | +| Sum/difference, absolute error | Sum of the operands' absolute error bounds | +| Sum over a known number of values | Sum of their bounds; a common per-value bound can be multiplied by that count | +| Ratio, relative error | Requires nonzero true denominator and denominator relative error below one; bound given below | -Pass 1 adds, for each group a query computes, every way to compute it. Each intermediate result of a way to compute is its own group (`find_or_add_group`), so `Project ← SummaryEstimate ← SummaryAgg` is three groups, each with a one-operator candidate. +A filter does not automatically preserve every accuracy guarantee. Group counts and metric conversions may need metadata or assumptions beyond the expression syntax. Propagation factors must be justified by the relevant operator/error contract. -| Way to compute | Code | Adds | -|---|---|---| -| Exact | [`per_computation_candidates.rs`](../../../crates/logical-asap-optimization/src/pass1/per_computation_candidates.rs), [`memo_population.rs`](../../../crates/logical-asap-optimization/src/pass1/memo_population.rs) | a candidate with the exact operator; $\boldsymbol\theta_v$ none, $(\varepsilon_v, \delta_v) = (0, 0)$ | -| Exact accumulator | same | a state group `ExactAggregate(kind)` over the input group, one candidate; a `FinalizeExactAccumulator` candidate in the result's group reading it | -| Summary, per family the executor supports | same | a state group `SummaryAgg[family]` over the input group, with **one candidate per $\boldsymbol\theta$ in the family's predefined list**, dropping a $\boldsymbol\theta$ whose $\varepsilon_v$ alone exceeds the $\varepsilon_q$ of every query that reaches the group; a `SummaryEstimate[statistic]` candidate in the result's group reading the state group. Each consumer's `parameter_floor` starts from its $g_c$'s validity cap (none for a linear $g_c$) | -| Fused build (#1203 2c) | same | a build that computes several targets at once: a candidate of the outer target's state group, with `fused_groups` naming the others | +For a ratio, let the operand relative-error bounds be $\varepsilon_X$ and $\varepsilon_Y$. If the numerator's relative error is defined and the denominator is nonzero, then: -Example: for A and B, Pass 1 adds G2 = `SummaryAgg[KLL](G1)` with candidates `k = 200` and `k = 345`, and G3, G4, G5 = `SummaryEstimate[q](G2)`. A and B's three estimates read one state group because their expressions over G1 hash-cons to one group. +$$ +\varepsilon_{X/Y}\le +\frac{\varepsilon_X+\varepsilon_Y}{1-\varepsilon_Y}. +$$ -Today, Pass 1 instead sizes one $\boldsymbol\theta$ per `accuracy_budget_parts` and keeps `SummaryEstimate ← SummaryAgg` as private nodes inside one candidate. #1203 2a makes the intermediate results groups; #1203 PR 2 replaces sizing by the predefined list. +Choosing a fixed validity cap $b<1$ for the denominator gives a linear upper bound within that region: -### Stage 1, Pass 2: rules over groups +$$ +\varepsilon_Y\le b +\quad\Longrightarrow\quad +\varepsilon_{X/Y}\le\frac{1}{1-b}(\varepsilon_X+\varepsilon_Y). +$$ -| # | Rule | Code | Matches | Adds | Status | -|---|---|---|---|---|---| -| 1 | Local window composition (tumbling windows) | `add_local_window_composition` ([`window_candidates.rs`](../../../crates/logical-asap-optimization/src/pass2/memo/window_candidates.rs)) | a summary state group whose window consumers serve queries whose ranges and steps share a gcd width $w$ | a candidate of the **same state group** with $\omega_v$ = tumbling width $w$; only widths dividing the group's query range are offered, and the step and phase are checked by the schedule rule ($\alpha^{\mathrm{phys}}$) | today it copies consumers per layout (`TumblingWindows` origin); #1160 makes it a candidate of the state group | -| 2 | Scan widening | `add_scan_label_candidates` ([`scan_labels.rs`](../../../crates/logical-asap-optimization/src/pass2/memo/rules/scan_labels.rs)) | several scan groups of one metric with different label sets | a group for the widened scan (`find_or_add_group`), and in each original scan group a candidate `Project(widened)` | #1203 2a-5 (#1222) | -| 3 | Summary inputs | `factor_summary_inputs` ([`summary_inputs.rs`](../../../crates/logical-asap-optimization/src/pass2/memo/rules/summary_inputs.rs)) | a value projection or filter below a summary build | that projection or filter as its own group; the build's candidate reads it | #1203 2a-2 (#1218); after 2a, Pass 1 already makes it a group | -| 4 | Top-k reuse | `add_topk_candidates` ([`topk.rs`](../../../crates/logical-asap-optimization/src/pass2/memo/rules/topk.rs)) | top-k groups over one ranking with different `k` | a candidate of each top-k group reading the largest-`k` group, with an exact-input requirement | #1203 2a-3 (#1220) | -| 5 | Finer aggregate | `add_finer_aggregate_candidates` ([`finer_aggregate.rs`](../../../crates/logical-asap-optimization/src/pass2/memo/rules/finer_aggregate.rs)) | a SQL aggregate whose grouping is coarser than another aggregate group's | a candidate of the coarse group: filter, then merge the finer group's rows with each measure's rollup | #1203 2a-3 (#1220) | -| 6 | Rollup | `add_rollup_candidates` ([`rollup.rs`](../../../crates/logical-asap-optimization/src/pass2/memo/rules/rollup.rs)) | a coarse aggregate or summary group and a finer group whose state rolls up | a candidate of the coarse group reading the finer group (exact or sketch merge) | #1203 2a-4 (#1221) | -| 7 | Window rollup | `add_window_rollup_candidates` (same) | a longer-range group whose windows a shorter-range group's state covers | a candidate of the longer-range group merging the shorter one's windows | #1203 2a-4 (#1221) | -| 8 | Summary results | `factor_summary_results` ([`summary_results.rs`](../../../crates/logical-asap-optimization/src/pass2/memo/rules/summary_results.rs)) | a summary readout inside a composite candidate | the readout as its own group | #1203 2a-2 (#1218) makes the readout a group in Pass 1; the rule is then deleted | -| 9 | Shared configurations | `add_state_configuration_candidates` ([`shared_configuration_readers.rs`](../../../crates/logical-asap-optimization/src/pass2/memo/rules/shared_configuration_readers.rs)) | consumers that could share a summary at a size or window horizon that covers them | consumer copies at the shared size (`SharedBuild` origin) | deleted in #1203 2b: the state group already holds every $\boldsymbol\theta$, and $y$ chooses per consumer input | -| 10 | Shared summary state groups | `add_shared_summary_state_groups` ([`state_groups.rs`](../../../crates/logical-asap-optimization/src/pass2/memo/state_groups.rs)) | summary builds directly over input groups | per-size concrete state groups with consumer copies | deleted in #1203 2b and #1160: one state group per state family | -| — | Logical edges (#1201) | after the last rule | every candidate $c$ and input $i$ | one `LogicalEdge { producer, consumer, input, satisfies_logical_requirement }` per $u \in G_{c,i}$, after tightening each `parameter_floor` with the pruning cap $\max_q \varepsilon_q / \lambda_{q,c,i}$; $\alpha^{\log}_{u,c,i}$ = `satisfies_logical(p^log_u, r^log_{c,i})`; output `LogicalASAPMemoCandidates { memo, edges }` | #1201 | +Both local factors are then `1 / (1 - b)`. This is a sufficient bound, not an exact equality or a claim that all feasible nonlinear plans remain represented. -After #1203 2b, every Pass 2 rule has the same shape: match groups, then `find_or_add_group` for any new intermediate result and `add_candidate` for the new way to compute an existing group. A rule adds no candidate a group already has (same expression and $p^{\log}$). +The draft's approved interface is retained here as an interface proposal: -### Stage 2: physical expansion and edges +```rust +pub struct ErrorPropagation { + pub factor_per_input: Vec, + pub validity_cap: Option, +} -Stage 2 adds no logical candidates. It expands each logical candidate into physical candidates, then builds $\mathcal E$. +impl LogicalRequirement { + pub fn error_propagation( + consumer: &Operator, + ) -> ErrorPropagation; +} +``` -| Step | Code | Reads | Adds | -|---|---|---|---| -| Physical expansion | `enumerate_candidate_configurations` ([`configurations.rs`](../../../crates/physical-asap-optimization/src/catalog/configurations.rs)) | each `CandidateForMemoGroup`, deployment capabilities, the queries' $\mathcal T_q$ | one `PhysicalCandidate { logical, properties, input_requirements }` per valid $p^{\mathrm{phys}} = (\tau, \ell, \sigma)$ that the deployment allows (section 2, levels 4–6), | -| Physical edges (#1201) | after expansion | each `LogicalEdge` | one `PhysicalEdge { producer, consumer, input, satisfies_logical_requirement, satisfies_physical_requirement }` per pair of the two logical candidates' physical candidates, with $\alpha^{\mathrm{phys}}$ = `satisfies_physical(p^phys_u, p^phys_c)`; the admissible ones ($\alpha^{\log} = \alpha^{\mathrm{phys}} = 1$) through `admissible_edges_into` / `admissible_edges_out_of` | +`factor_per_input` contains the local beta factors. A scalar `validity_cap` can represent a conservative common cap. Operator-specific caps, such as a bound only on a ratio's denominator, need either a documented input mapping or a per-input representation before implementation. Unsupported error models must be reported as unsupported rather than supplied with guessed factors. + +### Query composition and root checks + +An **error source** is one local approximation event with a stated metric and guarantee. Let $s$ index such sources in a selected, acyclic plan. Let $\varepsilon_s$ and $\delta_s$ be its bound and failure probability. Define $\lambda_{q,s}$ as the sum, over paths from that source to the query output, of products of local beta factors. Define $a_{q,s}$ as a binary indicator that query q's selected dependency graph uses that source. + +The intended root conditions are: + +$$ +\sum_s\lambda_{q,s}\varepsilon_s a_{q,s}\le\varepsilon_q, +\qquad +\sum_s\delta_s a_{q,s}\le\delta_q. +$$ + +Here, $\varepsilon_q$ and $\delta_q$ are the query target's error and failure limits. Repeated paths can amplify error and therefore contribute to lambda. A shared failure event is counted once in the union bound; distinct readout events need distinct source identities unless a joint guarantee is available. + +These equations describe the required bound. They are a linear MILP encoding only when the factors and source identities are fixed for the indexed alternatives and the reachability indicators are correctly linked to selected edges and outputs. Candidate-dependent paths, metric changes, or execution-dependent random events require a more detailed encoding. The draft does not supply that encoding, so the equations must not be described as a completed constraint implementation. + +This replaces the old unexplained `a_{q,u}` and the mixture of candidate-, edge-, and query-indexed lambda symbols. Local propagation uses beta; query propagation uses lambda; neither should be inferred from bare edge selection alone. + +### Validity caps and pruning caps + +A **validity cap** ensures that a chosen local bound applies. A **pruning cap** removes an alternative only when its nonnegative contribution alone exceeds every query target it could serve. Neither reserves an error-budget share for one input. + +For an input whose error can be expressed in the same metric, let $Q_{c,i}$ be the queries that could use it. Let $\lambda_{q,c,i}$ denote that input's query-level factor, and let $b_{c,i}$ denote its validity cap, or infinity when no cap is required. With fixed positive factors, a safe cap is: + +$$ +\operatorname{cap}_{c,i} +=\min\left(b_{c,i},\ \max_{q\in Q_{c,i}} +\frac{\varepsilon_q}{\lambda_{q,c,i}}\right). +$$ + +A zero factor gives no pruning restriction for that query. If a factor depends on the eventual plan, use a proven lower bound over all feasible downstream alternatives for this pruning test, or defer pruning. Using an arbitrary path's larger factor could discard a feasible plan. The maximum over queries protects alternatives that can serve at least one query. + +Compute the parameter floor from this cap only under the ordering condition in Section 2. Validity information is available when the consumer's error model is established; query-sensitive pruning follows logical generation and reachability analysis. Root checks still couple all contributing errors and determine feasibility. + +## 5. Execution contexts and other selection extensions — #1207 + +These extensions go beyond constraints (A)–(I) and the fixed-cost assumption in the supplied LaTeX. They are separated here so that the baseline remains understandable without an unfinished context MILP. + +### Execution identity -Example: v3 (`k = 345`) becomes, among others, (QueryTime, None), (QueryTime, OneEvaluation, Periodic(10 s, 0)) and (IngestionTime, AcrossEvaluations, grid phase 0). G3's estimate at query time has an admissible edge to each of them: same timing, or a query-time consumer of an ingestion-time result kept across evaluations. +A materialized candidate has its own delivery schedule. An unmaterialized candidate runs inside an execution that needs it. For example, a finalize candidate used by queries A and B may execute in both query contexts even though Stage 2 enumerates it only once. -### Stage 3: no rules +A **context** identifies a class of executions with a common owner and schedule across the accounting horizon. It is not necessarily one timestamp. -Stage 3 adds no candidates and no edges. It reads $\mathcal C$ and the admissible part of $\mathcal E$, creates the MILP variables of section 3 over them, and decodes $\{x = 1\}$ and $\{y = 1\}$ into `DagOfSelectedCandidates`. +```rust +pub struct ContextId(pub usize); + +pub enum Context { + QueryEvaluation(QueryId), + IngestionPipeline(RefreshWindowId), + Materialized(PhysicalCandidateId), +} + +impl Context { + // Periodic schedule when supported; None is not an implicit compatibility pass. + pub fn evaluation_schedule(&self) -> Option; +} +``` + +An unmaterialized chain inherits its consuming context. Encountering a materialized producer switches to that producer's own computation context when traversing its inputs. The context identifies where the operation runs; schedule compatibility decides whether a materialized input can supply it. + +| Extended variable | Meaning | +|---|---| +| $x_v^{\mathrm{mat}}$ | Materialized candidate selected | +| $x_{\kappa,v}$ | Unmaterialized candidate executes in context kappa | +| $y_{\kappa,u,c,i}$ | Input edge selected for a consumer execution in context kappa | +| $h_v$ | Selected retention amount for a materialized producer | + +The unindexed edge variable belongs to the baseline only. In the extension, every edge is indexed by its consumer's context, including the dedicated computation context of a materialized consumer. This avoids silently switching between two meanings of `y`. + +Activation, input satisfaction, utilization, output selection, and acyclicity must be defined for these execution instances. One binary per context/candidate assumes at most one reusable execution of that candidate within that context. If two uses require independent computations, the context or execution identity must distinguish them. + +### Cost and retention -## 5. Printing every logical DAG a memo group represents +To avoid confusing per-execution costs with the baseline's horizon-wide cost, use separate coefficient names: -A memo group stores its candidates and, through their expressions, the groups they read. It represents a set of logical DAGs without listing them. This section shows how to list them, using only the section 3 interfaces. Planning never does this: Stage 3's MILP chooses one DAG without listing the others. The listing is for inspection and for checking the MILP on small workloads. +| Symbol | Definition | +|---|---| +| $W_v^{\mathrm{mat}}$ | Fixed computation and access cost over the horizon for a materialized candidate's own schedule, excluding variable retention | +| $c_{\kappa,v}$ | Cost of one unmaterialized execution in the specified context | +| $\rho_\kappa$ | Fixed number or expected number of context executions over the common horizon | +| $h_v$ | Retention amount in a declared unit, such as retained windows | +| $\pi_v$ | Horizon-wide retention cost per unit of retained amount | +| $\operatorname{need}_{\kappa,v,c,i}$ | Retention amount producer v needs to serve this consumer input in this context | + +A proposed linear objective is: + +$$ +\min\left( +\sum_{v\text{ materialized}}W_v^{\mathrm{mat}}x_v^{\mathrm{mat}} ++\sum_{\kappa,v\text{ unmaterialized}}\rho_\kappa c_{\kappa,v}x_{\kappa,v} ++\sum_{v\text{ materialized}}\pi_v h_v +\right). +$$ + +A selected input edge imposes the corresponding retention lower bound: + +$$ +h_v\ge\operatorname{need}_{\kappa,v,c,i}\,y_{\kappa,v,c,i}. +$$ + +The retention domain, upper bounds, activation linking, and all coefficient units must be specified by the implementation. If required retention depends on selected downstream queries, that dependence needs explicit linking constraints; it is not a fixed coefficient merely because it has a name. + +### Extension status and legacy policy names + +| Earlier label | Meaning and status | +|---|---| +| (J) | Placeholder for consumer/reachability policies; absent from the supplied LaTeX and not defined by this document | +| `MinimumQueryReaders` | Legacy policy name suggesting a minimum number of distinct consuming queries. Its threshold, counting scope, and activation rule are unspecified in the supplied material; no constraint is asserted here. | +| `HighestSharedQueryMaterialization` | Legacy policy name with no supplied mathematical definition; retained as a migration question, not a defined constraint | +| (K) | Proposed root-accuracy check; Section 4 states the semantic bound and encoding conditions | +| (R) | Proposed retention constraints; lower-bound pattern above | +| (L) | Proposed per-query latency bounds; requires an execution/critical-path model | +| Resource budgets | Proposed memory and CPU limits; requires explicit accounting units and aggregation rules | + +These labels must not be presented as an implemented sequence of constraints (A)–(K). The review text contains no definitions that would justify filling the gaps by assumption. -### What one logical DAG is +## Appendix A. Enumerating DAGs for debugging -A logical DAG rooted at group $G$ is a set of logical candidates and edges $(u, c, i)$ such that: -1. exactly one candidate of $G$ is in it (the root); -2. every candidate $c$ in it has, for each input $i$, exactly one edge $(u, c, i)$, with $u$ a candidate of $G_{c,i}$ and $\alpha^{\log}_{u,c,i} = 1$ (constraints (A) and (G)); -3. every candidate in it is the root or the producer of some edge (constraint (F)); -4. the edges form no cycle (constraint (E)). +This is an inspection tool for small memos, not a planning stage. The MILP selects a plan without listing all alternatives. -A candidate is one node however many edges read it: two consumers that choose the same producer share it, which is what $x_u \in \{0, 1\}$ says. Two consumers of one group may choose different candidates of it. The listing is therefore a choice of one producer per consumer input $(c, i)$, not one candidate per group. +A logical DAG has one selected root candidate, one compatible producer per input of every reached candidate, no unused nodes, and no cycles. Producer choices are per consumer input, not per group. Two consumers can choose different candidates of the same group; choosing the same candidate shares one logical node. -### Pseudocode +The following Rust-style pseudocode preserves the draft's enumeration procedure. Helpers such as `cartesian_product` are schematic; `reaches` follows the stored consumer-to-producer dependency direction and is reflexive. This direction makes the cycle test precise. ```rust /// One logical DAG: its nodes and, for each node's inputs, the chosen producer. +#[derive(Clone)] struct LogicalDag { root: CandidateForMemoGroupId, /// For each node c in the DAG, one producer per input i, in input order. @@ -622,9 +686,14 @@ fn logical_dags(memo: &Memo, root_group: MemoGroupId) -> Vec { /// Choose producers for the inputs of every node in `pending`, in FIFO order. /// Each DAG is reached by exactly one sequence of choices, so none repeats. -fn expand(memo: &Memo, dag: LogicalDag, mut pending: VecDeque, all: &mut Vec) { +fn expand( + memo: &Memo, + dag: LogicalDag, + mut pending: VecDeque, + all: &mut Vec, +) { let Some(consumer) = pending.pop_front() else { - all.push(dag); // every input of every node has a producer + all.push(dag); // Every reached input has a producer. return; }; let candidate = memo.candidate(consumer); @@ -633,11 +702,15 @@ fn expand(memo: &Memo, dag: LogicalDag, mut pending: VecDeque>() })) { - // Constraint (E): a producer that already reaches `consumer` would close a cycle. + // In the stored consumer-to-producer direction, this would close a cycle. if choice.iter().any(|&u| reaches(&dag, u, consumer)) { continue; } @@ -654,8 +727,12 @@ fn expand(memo: &Memo, dag: LogicalDag, mut pending: VecDeque impl Iterator + '_ { - (0..memo.candidates[group.0 as usize].len()).map(move |index| CandidateForMemoGroupId { group, index }) +fn candidates_of( + memo: &Memo, + group: MemoGroupId, +) -> impl Iterator + '_ { + (0..memo.groups[group.0 as usize].candidates.len()) + .map(move |index| CandidateForMemoGroupId { group, index }) } /// Whether `from` reaches `to` through the producers chosen so far @@ -665,7 +742,13 @@ fn reaches(dag: &LogicalDag, from: CandidateForMemoGroupId, to: CandidateForMemo /// Print one DAG as an indented tree; a node read a second time is printed as a /// reference to its first occurrence, so sharing is visible. fn print(memo: &Memo, dag: &LogicalDag) { - fn node(memo: &Memo, dag: &LogicalDag, c: CandidateForMemoGroupId, depth: usize, seen: &mut HashSet) { + fn node( + memo: &Memo, + dag: &LogicalDag, + c: CandidateForMemoGroupId, + depth: usize, + seen: &mut HashSet, + ) { let candidate = memo.candidate(c); if !seen.insert(c) { println!("{}↑ G{}[{}] (shared)", " ".repeat(depth), c.group.0, c.index); @@ -681,33 +764,26 @@ fn print(memo: &Memo, dag: &LogicalDag) { } ``` -For a workload, list from every query root at once: start `pending` with one root candidate per $G_q$ (constraints (C) and (D)), taking the product over queries of their root candidates. A candidate two queries reach is one node, as above. - -### Example - -For the example's query A (root G3) and query B (root G6), with no error caps applied: +For a workload, choose one root candidate per query and initialize the pending queue with the distinct chosen roots. Deduplicate that queue before expansion so a shared root is processed once. -``` -A: 2 DAGs B: 4 DAGs (G4 and G5 choose independently) -G3[0] SummaryEstimate q=0.95 G6[0] Divide - G2[0] SummaryAgg KLL θ={k=200} G4[0] SummaryEstimate q=0.99 - G1[0] Scan G2[0] SummaryAgg KLL θ={k=200} -G3[0] SummaryEstimate q=0.95 G1[0] Scan - G2[1] SummaryAgg KLL θ={k=345} G5[0] SummaryEstimate q=0.50 - G1[0] Scan ↑ G2[0] (shared) - … and (k=200, k=345), (k=345, k=200), (k=345, k=345) -``` +Before accuracy filtering, the running example has two alternatives for A and four for B: G4 and G5 choose independently between the two G2 builds. Joint enumeration therefore has eight choices. These are structural choices only; no claim is made that they meet B's ratio-accuracy requirement. -Listed from both roots at once there are $2 \times 4 = 8$ DAGs. In the one where all three estimates choose `k = 345`, `G2[1]` is a single node read by three consumers, so its cost is counted once. Constraint (K) later accepts only the DAGs that meet each query's target: for B, only (k=345, k=345). Error caps (section 1) remove edges before the listing, so a capped producer never appears. - -### Size +| B's p99 producer | B's p50 producer | State sharing within B | +|---|---|---| +| `k = 200` | `k = 200` | One logical state candidate | +| `k = 200` | `k = 345` | Two state candidates | +| `k = 345` | `k = 200` | Two state candidates | +| `k = 345` | `k = 345` | One logical state candidate | -Counting each consumer input's choice independently gives an upper bound on the number of DAGs rooted at $G$: +The printer marks repeated candidates as references rather than expanding them twice. Physical sharing and cost additionally depend on lifetime and execution context. -$$N(G) = \sum_{v \in G} N(v), \qquad N(v) = \prod_{i} \sum_{u \in P_{v,i},\ \alpha^{\log}_{u,v,i} = 1} N(u),$$ +For an acyclic candidate-dependency space, a tree-expansion count bounds the number of DAGs: -with $N(v) = 1$ for a candidate with no inputs. It is an upper bound because a shared producer is expanded once in a DAG, where the formula expands it once per consumer, and because cyclic choices are dropped. $N$ multiplies along the depth of the memo, which is why only small workloads are listed, and why Stage 3 chooses with the MILP instead. +$$ +N(G)=\sum_{v\in G}N(v),\qquad +N(v)=\prod_i\sum_{u\in P^{\mathrm{log,adm}}_{v,i}}N(u). +$$ -### Physical DAGs +Here, $P^{\mathrm{log,adm}}_{v,i}$ is the set of logically compatible producers for that input, and a leaf has count one. The bound overcounts shared subgraphs. For a cyclic memo, this recurrence is not well-founded; use cycle-aware enumeration or an explicitly bounded unfolding instead. -The same procedure lists physical DAGs from Stage 2's output. Use `PhysicalCandidate`s instead of logical candidates, and `admissible_edges_into(c, i)` (which already applies $\alpha^{\log}$ and $\alpha^{\mathrm{phys}}$) instead of `candidates_of` with `satisfies_logical`. A physical candidate with $\ell = $ None that two consumers in different contexts $\kappa$ choose runs once per context; in the listing it is still one node, and the contexts are a Stage 3 choice (#1207). +For physical inspection, replace logical candidates and compatibility checks with physical candidates and admissible edges. Under the execution-context extension, candidate-level listing alone is not a list of executable plans: instantiate contexts and apply schedule, retention, root-accuracy, and other enabled constraints before comparing results with Stage 3. From 210de2b9158237002bf86352c31f745435cd93da Mon Sep 17 00:00:00 2001 From: zzylol <50204836+zzylol@users.noreply.github.com> Date: Fri, 6 Nov 2026 03:34:35 +0000 Subject: [PATCH 20/44] docs(concepts): fix review issues: one Memo type, decision tree restored, grid width from the window layout, per-input validity caps, metric rule, (K) encoding, (J) definitions, (L)/budgets out of scope Co-Authored-By: Claude Opus 5.5 --- .../memo-groups-properties-requirements.md | 186 +++++++++++++----- 1 file changed, 138 insertions(+), 48 deletions(-) diff --git a/docs/design_docs/concepts/memo-groups-properties-requirements.md b/docs/design_docs/concepts/memo-groups-properties-requirements.md index ca3c51934..d1fda2719 100644 --- a/docs/design_docs/concepts/memo-groups-properties-requirements.md +++ b/docs/design_docs/concepts/memo-groups-properties-requirements.md @@ -1,8 +1,8 @@ # Memo groups, properties and requirements -This document explains how the optimizer represents alternatives, checks whether they can be composed, and selects an executable plan. It follows the supplied **Optimization Problem Formulation** for the baseline model and labels additional design work **Extension (#1207)**. +This document explains how the optimizer represents alternatives, checks whether they can be composed, and selects an executable plan. It follows the optimization problem formulation (paper, "Optimization Problem Formulation") for the baseline model and labels additional design work **Extension (#1207)**. -The Rust blocks are interface sketches, not standalone compilable implementations. Repository paths and issue references are retained from the supplied draft; their current implementation status has not been independently checked. +The Rust blocks are interface sketches, not compilable code. Paths name today's code; issue numbers name the work that changes it. ## 1. Model and running example @@ -10,7 +10,7 @@ A **workload** contains query expressions, their evaluation schedules, and their Input groups leave producer choices open. Selecting a candidate does not select its entire input plan. Each consumer input chooses a producer separately, and several inputs may choose the same producer. -The expression includes the operator's arguments, including the summary family. In this design, KLL and DDSketch states belong to separate state groups. A final-result group can offer readout candidates that use either family. This resolves the old decision tree's contradiction: it must not put KLL and DDSketch builds in one state group. +The expression includes the operator's arguments, including the summary family. In this design, KLL and DDSketch states belong to separate state groups. A final-result group can offer readout candidates that use either family. So the decision tree's first level (Section 2) chooses the expression, family included; a state group never mixes families. ### Running example @@ -27,7 +27,7 @@ Query A requests the p95 latency over one minute. Query B requests p99 divided b G3, G4, and G5 choose a G2 producer independently. If all choose the same retained physical producer, they can reuse one computed state when their schedules and windows are compatible. An unmaterialized candidate may instead execute separately in different contexts; Section 5 explains that distinction. -The old example assigned illustrative KLL rank errors of 0.008 and 0.005 and added them to obtain Query B's ratio error. That inference is invalid: rank error does not directly bound numerical quantile error or relative ratio error. The example here demonstrates representation and sharing only. Accepting Query B requires an applicable error model, including assumptions about its denominator. +The example shows representation and sharing. Accuracy is checked by Section 4, whose metric rule matters here: KLL bounds **rank** error, and rank error does not bound a quantile's value error or the relative error of a ratio. Query B's relative ratio target therefore needs operands with a relative value-error guarantee, e.g. a DDSketch state group over G1 with `alpha` candidates, unless a declared distribution assumption converts rank error to value error. Query A, with a rank target, can use the KLL candidates directly. ### Notation used by the baseline model @@ -58,7 +58,7 @@ p_v^{\mathrm{log}}=\boldsymbol\theta_v,\qquad p_v^{\mathrm{phys}}=(\tau_v,\ell_v). $$ -The supplied LaTeX uses the producer-set symbol in two different ways: the prose defines a filtered set, while its notation table defines an unfiltered set. This document follows the table and constraints (G)–(H), and gives the filtered set a separate name: +The formulation's text defines the producer set after the requirement check, and its notation table before it. This document follows the table and constraints (G)–(H), and names the filtered set separately: $$ P_{c,i}=G_{c,i}\cap\mathcal C, @@ -128,7 +128,7 @@ For the proposed periodic consumer path, let the consumer interval and phase be | Tumbling grid | $w\mid I$ and $\varphi\equiv\varphi_g\pmod w$ | | None | Runs in the consuming context; no independent delivery schedule | -The supplied formulation also permits a single, non-periodic evaluation. The interval/phase interface below covers only periodic consumers. Rejecting non-periodic consumers is a current scope restriction of that proposed interface, not a mathematical impossibility; supporting them needs explicit evaluation-time membership checks. +The formulation also permits a single, non-periodic evaluation. The interval/phase interface below covers only periodic consumers. Rejecting non-periodic consumers is a current scope restriction of that proposed interface, not a mathematical impossibility; supporting them needs explicit evaluation-time membership checks. ### Invalid combinations @@ -141,28 +141,83 @@ Checked constructors reject these combinations under the proposed schedule model | Materialized lifetime without its required schedule | Delivery times or window grid would be unspecified | | OneEvaluation with a grid, or windowed AcrossEvaluations with a periodic schedule | Schedule kind does not match this model's lifetime representation | | Zero interval/width, or phase outside the canonical interval/width | Invalid schedule representation | -| Ingestion-time grid with nonzero phase | Unsupported by the draft's epoch-anchored ingestion representation | +| Ingestion-time grid with nonzero phase | Ingestion-time grids are anchored at the epoch: the plan carries no origin | -A tumbling grid also needs a known positive width. The `PhysicalProperties` constructor alone cannot validate a width held in logical properties; physical candidate construction must validate the pair. Non-windowed AcrossEvaluations schedules need an explicit representation before that alternative can be offered. +A tumbling grid's width is the candidate's logical window layout $\omega_v$; it is not repeated in the schedule. Building a `PhysicalCandidate` checks the pair: a grid schedule needs a tumbling layout, and its phase must be below the width. Non-windowed AcrossEvaluations schedules need their own representation before that alternative can be offered. + +### Decision tree and order of decisions + +A memo group's candidates are the leaves of a tree. The first level chooses the expression $e_v$ (operator and family); each later level decides one property dimension. A path from the root to a leaf is one physical candidate. + +```mermaid +flowchart TD + G["Memo group G
(logical result)"] --> R1["e_v: exact operator"] + G --> R2["e_v: readout of KLL state"] + G --> R3["e_v: readout of DDSketch state"] + G --> R4["e_v: other operators
(prune: executor support)"] + + R2 --> T1["θ: k = 200"] + R2 --> T2["θ: k = 345"] + R2 --> T3["θ: k = …
(prune: pruning cap)"] + + T2 --> W1["ω: no windows"] + T2 --> W2["ω: tumbling, width w
(prune: w must divide
the group's query range)"] + + W2 --> P1["τ = QueryTime"] + W2 --> P2["τ = IngestionTime
(prune: no ingestion pipeline)"] + + P1 --> L1["ℓ = None"] + P1 --> L2["ℓ = OneEvaluation"] + P1 --> L3["ℓ = AcrossEvaluations
(prune: deployment keeps
no query-time results)"] + P2 --> L4["ℓ = None"] + P2 --> L5["ℓ = AcrossEvaluations"] + P2 -. invalid .-> L6["ℓ = OneEvaluation"] + + L2 --> S1["σ: Periodic (I, φ)
from the gcd closure
of the queries' schedules"] + L3 --> S2["σ: grid phase φ mod w"] + L5 --> S3["σ: grid phase 0"] +``` + +| Level | Decides | Stage | Values offered | Pruned when | +|---|---|---|---|---| +| 1 | expression $e_v$ | Pass 1 and Pass 2 rules | each way to compute the group | the executor cannot build the family | +| 2 | $\boldsymbol\theta_v$ | Pass 1 | the family's predefined list | the pruning cap (Section 4) | +| 3 | $\omega_v$ — Extension | Pass 2 | tumbling widths from the gcd closure of the queries' ranges and steps | the width does not divide the group's query range | +| 4 | $\tau_v$ | Stage 2 | QueryTime, IngestionTime | the source has no ingestion pipeline | +| 5 | $\ell_v$ | Stage 2 | None, OneEvaluation, AcrossEvaluations | invalid with $\tau_v$; the deployment keeps no query-time results | +| 6 | $\sigma_v$ — Extension | Stage 2 | the gcd closure of the schedules of the queries reaching the group | only when $\ell_v \ne$ None | + +The tree does not decide which producer each consumer input reads ($y$), where an unmaterialized candidate runs (its context, Section 5), or how much a materialized candidate retains ($h_v$). Those are Stage 3 choices. + +**The order does not change the candidate set.** The leaves are the product of the dimensions' values minus invalid combinations and pruned values. Each prune rule reads only the value it removes and values decided above it, so any order that keeps each prune rule below the dimensions it reads yields the same leaves. + +**The order changes how many nodes enumeration visits:** $\sum_d \prod_{d' \le d} \lvert \text{values kept at } d' \rvert$. Dimensions that prune much go early; dimensions that multiply without pruning go late. The order above follows four rules: + +1. **Logical before physical.** Stage 1 hash-conses groups and computes $\alpha^{\log}$ once, before Stage 2 multiplies each logical candidate by its physical alternatives. +2. **Strong prunes early.** Executor support and the pruning cap act before windows and physical properties multiply the tree. +3. **Dependent dimensions adjacent.** $\tau \to \ell \to \sigma$ rejects each invalid combination at the first level that can detect it. +4. **Deployment-wide prunes as early as possible.** Without an ingestion pipeline the IngestionTime subtree is pruned once per candidate; moving $\tau$ above $\omega$ would prune it once per group, with the same leaves. + +Alternative orders worth measuring: $\tau$ first, when most sources lack an ingestion pipeline; $\omega$ before $\boldsymbol\theta$, when queries' ranges prune many widths; and requirement-driven (top-down) enumeration, which visits fewer nodes but must take the union of all consumers' requirements so that sharing is not lost. ## 3. Pipeline: interfaces and rules | Stage | Input | Output | Responsibility | |---|---|---|---| -| Stage 0 | Query IR | `DagOfLogicalMemoGroups` | Hash-cons logical expressions | +| Stage 0 | Query IR | `Memo` (groups without candidates) | Hash-cons logical expressions | | Stage 1, Pass 1 | Group DAG | `Memo` | Add initial logical candidates and intermediate groups | | Stage 1, Pass 2 | Memo | `LogicalASAPMemoCandidates` | Add reuse alternatives and compute logical edges | | Stage 2 | Logical memo candidates | `PhysicalASAPMemoCandidates` | Expand physical alternatives and compute compatibility | | Stage 3 | Physical memo candidates | `DagOfSelectedCandidates` | Select implementations, dependencies, and executions | | Realizer/export | Selected DAG | `PhysicalASAPDAG` | Build and export the executable graph | -Enumeration proceeds through expression, parameters, window layout, timing, lifetime, and schedule. Executor and deployment restrictions prune unsupported alternatives as soon as their prerequisites are known. Accuracy pruning is allowed only under the conditions in Section 4. Reordering dimensions preserves the candidate set only when the same alternatives and sound pruning predicates are retained. +Enumeration follows the decision tree in Section 2. Accuracy pruning uses only the caps of Section 4. Rules add groups through `find_or_add_group` and candidates through `add_candidate`. They match group expressions, avoid duplicating an existing expression/property pair, and do not copy consumers to make a new producer visible. After logical generation finishes, edge construction records the possible producer choices. ### Stage 0: groups -`hash_consed` in `crates/logical-asap-optimization/src/stage0.rs` identifies shared logical expressions and query roots. The following preserves the existing group representation described in the draft. Candidate ownership is exposed through the group in the proposed Stage 1 interface. +`hash_consed` in `crates/logical-asap-optimization/src/stage0.rs` identifies shared logical expressions and query roots. Stage 0 and Stage 1 share one type, `Memo`: Stage 0 returns it with every group's candidate list empty, and Stage 1 fills the lists. ```rust pub struct MemoGroupId(pub u32); @@ -176,7 +231,7 @@ pub struct MemoGroup { pub candidates: Vec, } -pub struct DagOfLogicalMemoGroups { +pub struct Memo { pub groups: Vec, pub query_roots: Vec>, } @@ -184,7 +239,7 @@ pub struct DagOfLogicalMemoGroups { `OperatorNode` recursively contains chosen child nodes and therefore describes a fixed plan fragment. `Operator` contains input group IDs and leaves their implementations open. A candidate needs its own expression because a group may offer several ways to compute the same result; it cannot reuse the group's representative `logical_node` as every candidate's expression. -Replacing the group's representative node and child list with a single group-based expression, or removing the one-field `CandidateExpression` wrapper, remains an optional simplification from the review. Neither is required here. +Two simplifications are possible but not adopted: storing the group as one `Operator` instead of its node and child list, and removing the one-field `CandidateExpression` wrapper. ### Stage 1, Pass 1: initial candidates @@ -203,11 +258,6 @@ pub struct CandidateForMemoGroupId { pub index: usize, } -pub struct Memo { - pub groups: Vec, - pub query_roots: Vec>, -} - impl Memo { pub fn find_or_add_group( &mut self, @@ -282,7 +332,7 @@ Migration targets: remove private `root_node` chains, duplicate `input_groups`, ### Stage 1, Pass 2: reuse rules and logical edges -The following preserves the supplied rule inventory and tracking references. Paths are relative to `crates/logical-asap-optimization/src/pass2/memo/`. +Paths are relative to `crates/logical-asap-optimization/src/pass2/memo/`. | Rule / function | File | Result or migration target | |---|---|---| @@ -297,7 +347,7 @@ The following preserves the supplied rule inventory and tracking references. Pat | Shared configurations: `add_state_configuration_candidates` | `rules/shared_configuration_readers.rs` | Remove copied consumers once producer variants are selectable; #1203 2b | | Shared state groups: `add_shared_summary_state_groups` | `state_groups.rs` | Remove per-size groups; #1203 2b / #1160 | -After the rules, build logical edges and apply any proven-safe accuracy pruning. `input_index_of_consumer` replaces the ambiguous `input` field name throughout this document's edge interfaces. +After the rules, Pass 2 builds the logical edges and applies the pruning caps of Section 4. Edges name the consumer input `input_index_of_consumer`. ```rust fn apply(memo: &mut Memo) -> Result<(), Pass2RuleError>; @@ -361,12 +411,17 @@ pub struct EvaluationSchedule { interval_ms: u64, phase_ms: u64, } -impl PhysicalProperties { - pub fn delivers_on(&self, times: &EvaluationSchedule) -> bool; +impl PhysicalCandidate { + /// Whether this producer delivers on `times` (Section 2's alignment table). + /// Reads the grid width from the candidate's logical window layout. + pub fn delivers_on(&self, memo: &Memo, times: &EvaluationSchedule) -> bool; } +/// α^phys: timing and lifetime (Section 2's table), and, for a materialized +/// consumer, that the producer delivers on the consumer's own schedule. pub fn satisfies_physical( - producer: &PhysicalProperties, - consumer: &PhysicalProperties, + memo: &Memo, + producer: &PhysicalCandidate, + consumer: &PhysicalCandidate, ) -> bool; pub struct PhysicalCandidateId(pub usize); pub struct PhysicalEdgeId(pub usize); @@ -408,15 +463,13 @@ pub struct PhysicalCandidateKey { } ``` -`PhysicalCandidateId` indexes an emitted candidate. `PhysicalCandidateKey` describes an alternative rejected before it received an ID, allowing diagnostics to explain pruning. The admissible-edge accessor returns edge IDs, not the unfiltered producer set from Section 1. - -The `delivers_on` sketch needs access to a tumbling producer's logical window width. Supply that context in its implementation or move the check to the full physical-candidate interface; the phase-only schedule is insufficient by itself. +`PhysicalCandidateId` indexes an emitted candidate. `PhysicalCandidateKey` describes an alternative rejected before it received an ID, allowing diagnostics to explain pruning. The admissible-edge accessor returns edge IDs, not the unfiltered producer set from Section 1. `delivers_on` and `satisfies_physical` take candidates rather than bare properties because a tumbling grid's width lives in the logical window layout. Migration targets include replacing `Materialization` and duplicated physical fields with checked properties, and replacing `ReaderConstraint::{IncompatibleOutputs, AllowedOutputs}` with compatibility and schedule checks. Policy constraints involving the number of selected consuming queries are a separate concern; see Section 5. ### Stage 3: baseline selection and realization -The baseline uses the variables and constraints in the supplied LaTeX. Stage 3 adds no candidates or potential dependencies. +The baseline uses the formulation's variables and constraints. Stage 3 adds no candidates or potential dependencies. | Variable | Meaning | |---|---| @@ -480,7 +533,7 @@ The realizer uses the selected dependencies to build the executable graph. Expor ## 4. Accuracy and safe pruning — Extension (#1207) -The supplied LaTeX checks logical requirements per edge. The proposed extension additionally checks composed accuracy at each query root. This section describes the error model needed for that check; it is not an additional property dimension. +The formulation checks logical requirements per edge. This extension also checks composed accuracy at each query root. This section gives the error model and its MILP encoding; error is not a property dimension. ### Local propagation @@ -504,7 +557,9 @@ $$ | Sum over a known number of values | Sum of their bounds; a common per-value bound can be multiplied by that count | | Ratio, relative error | Requires nonzero true denominator and denominator relative error below one; bound given below | -A filter does not automatically preserve every accuracy guarantee. Group counts and metric conversions may need metadata or assumptions beyond the expression syntax. Propagation factors must be justified by the relevant operator/error contract. +A filter does not automatically preserve every accuracy guarantee. Group counts may need metadata beyond the expression syntax. Each factor must follow from the operator's error contract. + +**Metric rule.** Every local guarantee states its metric: absolute (in value units, or a multiple of a stream norm), relative, or rank. A factor $\beta_{c,i}$ is defined for one input metric and one output metric. An input whose metric the consumer's error contract cannot convert gives no edge ($\alpha^{\log} = 0$); the conversion is never assumed. For example, a relative-error ratio cannot read a KLL quantile (rank error) unless a declared distribution assumption supplies the conversion. For a ratio, let the operand relative-error bounds be $\varepsilon_X$ and $\varepsilon_Y$. If the numerator's relative error is defined and the denominator is nonzero, then: @@ -523,12 +578,15 @@ $$ Both local factors are then `1 / (1 - b)`. This is a sufficient bound, not an exact equality or a claim that all feasible nonlinear plans remain represented. -The draft's approved interface is retained here as an interface proposal: +Interface (approved 2026-11-05): ```rust pub struct ErrorPropagation { + /// β_{c,i} per input, for that input's metric. pub factor_per_input: Vec, - pub validity_cap: Option, + /// b_{c,i} per input: the cap under which the linear bound holds; None + /// when the bound holds everywhere (e.g. a ratio caps only its denominator). + pub validity_cap_per_input: Vec>, } impl LogicalRequirement { @@ -538,7 +596,7 @@ impl LogicalRequirement { } ``` -`factor_per_input` contains the local beta factors. A scalar `validity_cap` can represent a conservative common cap. Operator-specific caps, such as a bound only on a ratio's denominator, need either a documented input mapping or a per-input representation before implementation. Unsupported error models must be reported as unsupported rather than supplied with guessed factors. +`factor_per_input` holds the local β factors and `validity_cap_per_input` the per-input caps, aligned by input index. An operator without an error contract is reported as unsupported; no factor is guessed. ### Query composition and root checks @@ -554,9 +612,33 @@ $$ Here, $\varepsilon_q$ and $\delta_q$ are the query target's error and failure limits. Repeated paths can amplify error and therefore contribute to lambda. A shared failure event is counted once in the union bound; distinct readout events need distinct source identities unless a joint guarantee is available. -These equations describe the required bound. They are a linear MILP encoding only when the factors and source identities are fixed for the indexed alternatives and the reachability indicators are correctly linked to selected edges and outputs. Candidate-dependent paths, metric changes, or execution-dependent random events require a more detailed encoding. The draft does not supply that encoding, so the equations must not be described as a completed constraint implementation. +**MILP encoding.** Path factors $\lambda_{q,s}$ depend on which edges are selected, so the MILP does not use them directly. It carries an error bound per selected candidate instead, which needs no path enumeration and is the same for every query reading that candidate: + +$$ +E_c \ge \varepsilon^{\mathrm{loc}}_c + \sum_{i} \beta_{c,i} F_{c,i}, +\qquad +F_{c,i} \ge E_u - M_{c,i}\,(1 - y_{u,c,i}) \quad \forall (u,c,i)\in\mathcal E, +$$ -This replaces the old unexplained `a_{q,u}` and the mixture of candidate-, edge-, and query-indexed lambda symbols. Local propagation uses beta; query propagation uses lambda; neither should be inferred from bare edge selection alone. +$$ +D_c \ge \delta^{\mathrm{loc}}_c + \sum_{i} H_{c,i}, +\qquad +H_{c,i} \ge D_u - (1 - y_{u,c,i}) \quad \forall (u,c,i)\in\mathcal E, +$$ + +$$ +E_v \le \varepsilon_q + M_q\,(1 - z_{q,v}), +\qquad +D_v \le \delta_q + (1 - z_{q,v}) +\qquad \forall q,\ v\in G_q. \tag{K} +$$ + +- $E_c, F_{c,i}, D_c, H_{c,i} \ge 0$ are continuous. $\varepsilon^{\mathrm{loc}}_c$ and $\delta^{\mathrm{loc}}_c$ are the candidate's own guarantee (zero for an exact operator); $\beta_{c,i}$ and the metric rule come from `error_propagation`. +- $M_{c,i}$ is any upper bound on $E_u$ over the producers of input $i$ (their largest feasible bound); $M_q$ likewise for the root. Exactly one $y_{u,c,i} = 1$ per selected input (constraint (A)), so $F_{c,i}$ equals the chosen producer's error bound. +- A producer reached by two paths contributes on both, which is the worst-case sum. The failure probabilities add by a union bound; a failure event reached twice is counted twice, which is conservative. +- Under the context extension (Section 5), the variables are indexed by context, $E_{\kappa,c}$, and the edges by $y_{\kappa,u,c,i}$. + +$\lambda_{q,c,i}$ is still used, but only for the pruning caps below, where a fixed lower bound suffices. ### Validity caps and pruning caps @@ -576,7 +658,7 @@ Compute the parameter floor from this cap only under the ordering condition in S ## 5. Execution contexts and other selection extensions — #1207 -These extensions go beyond constraints (A)–(I) and the fixed-cost assumption in the supplied LaTeX. They are separated here so that the baseline remains understandable without an unfinished context MILP. +These extensions go beyond constraints (A)–(I) and the formulation's fixed-cost assumption. They are separated here so that the baseline reads on its own. ### Execution identity @@ -643,19 +725,27 @@ $$ The retention domain, upper bounds, activation linking, and all coefficient units must be specified by the implementation. If required retention depends on selected downstream queries, that dependence needs explicit linking constraints; it is not a fixed coefficient merely because it has a name. -### Extension status and legacy policy names +### Consumer-count policies (J) -| Earlier label | Meaning and status | -|---|---| -| (J) | Placeholder for consumer/reachability policies; absent from the supplied LaTeX and not defined by this document | -| `MinimumQueryReaders` | Legacy policy name suggesting a minimum number of distinct consuming queries. Its threshold, counting scope, and activation rule are unspecified in the supplied material; no constraint is asserted here. | -| `HighestSharedQueryMaterialization` | Legacy policy name with no supplied mathematical definition; retained as a migration question, not a defined constraint | -| (K) | Proposed root-accuracy check; Section 4 states the semantic bound and encoding conditions | -| (R) | Proposed retention constraints; lower-bound pattern above | -| (L) | Proposed per-query latency bounds; requires an execution/critical-path model | -| Resource budgets | Proposed memory and CPU limits; requires explicit accounting units and aggregation rules | +Two policies from today's code (`ReaderConstraint` in `crates/types/src/stages/physical/graph.rs`) restrict which queries may reach a node. They need reach indicators $a_{q,v} \in \{0,1\}$: query $q$'s selected plan contains $v$. + +$$ +a_{q,v} \ge z_{q,v},\qquad +a_{q,u} \ge a_{q,c} + y_{u,c,i} - 1,\qquad +a_{q,u} \le z_{q,u} + \sum_{(u,c,i)\in\mathcal E} w_{q,u,c,i},\qquad +w_{q,u,c,i} \le a_{q,c},\ w_{q,u,c,i} \le y_{u,c,i}. +$$ + +| Policy | Meaning | Constraint (J) | +|---|---|---| +| `MinimumQueryReaders { candidate, minimum }` | a selected candidate is worth keeping only when at least `minimum` queries reach it | $\sum_q a_{q,v} \ge \text{minimum}\cdot x_v$ | +| `HighestSharedQueryMaterialization { candidate }` | a non-summary query materialization is placed where sharing stops: it is a query output, or one of its selected consumers is reached by at most one query | $\sum_q z_{q,v} + \sum_{(v,c,i)} s_{v,c,i} \ge x_v$, with $s_{v,c,i} \le y_{v,c,i}$ and $\sum_q a_{q,c} \le 1 + \lvert\mathcal Q\rvert\,(1 - s_{v,c,i})$ | + +`ReaderConstraint::{IncompatibleOutputs, AllowedOutputs}` are not kept: the schedule check of Section 2 replaces them. + +### Out of this design -These labels must not be presented as an implemented sequence of constraints (A)–(K). The review text contains no definitions that would justify filling the gaps by assumption. +Latency bounds (L) and resource budgets need an execution and critical-path model, and accounting units, that this document does not define. They are left to a later design; no interface for them is part of this one. ## Appendix A. Enumerating DAGs for debugging @@ -663,7 +753,7 @@ This is an inspection tool for small memos, not a planning stage. The MILP selec A logical DAG has one selected root candidate, one compatible producer per input of every reached candidate, no unused nodes, and no cycles. Producer choices are per consumer input, not per group. Two consumers can choose different candidates of the same group; choosing the same candidate shares one logical node. -The following Rust-style pseudocode preserves the draft's enumeration procedure. Helpers such as `cartesian_product` are schematic; `reaches` follows the stored consumer-to-producer dependency direction and is reflexive. This direction makes the cycle test precise. +Rust-style pseudocode follows. Helpers such as `cartesian_product` are schematic; `reaches` follows the stored consumer-to-producer dependency direction and is reflexive. This direction makes the cycle test precise. ```rust /// One logical DAG: its nodes and, for each node's inputs, the chosen producer. From eccf69081097ff20bf75448504cf75c0235a1955 Mon Sep 17 00:00:00 2001 From: zzylol <50204836+zzylol@users.noreply.github.com> Date: Fri, 6 Nov 2026 03:45:14 +0000 Subject: [PATCH 21/44] docs(concepts): delete both consumer-count policies; no constraint (J) Co-Authored-By: Claude Opus 5.5 --- .../memo-groups-properties-requirements.md | 20 +++---------------- 1 file changed, 3 insertions(+), 17 deletions(-) diff --git a/docs/design_docs/concepts/memo-groups-properties-requirements.md b/docs/design_docs/concepts/memo-groups-properties-requirements.md index d1fda2719..b674f226d 100644 --- a/docs/design_docs/concepts/memo-groups-properties-requirements.md +++ b/docs/design_docs/concepts/memo-groups-properties-requirements.md @@ -465,7 +465,7 @@ pub struct PhysicalCandidateKey { `PhysicalCandidateId` indexes an emitted candidate. `PhysicalCandidateKey` describes an alternative rejected before it received an ID, allowing diagnostics to explain pruning. The admissible-edge accessor returns edge IDs, not the unfiltered producer set from Section 1. `delivers_on` and `satisfies_physical` take candidates rather than bare properties because a tumbling grid's width lives in the logical window layout. -Migration targets include replacing `Materialization` and duplicated physical fields with checked properties, and replacing `ReaderConstraint::{IncompatibleOutputs, AllowedOutputs}` with compatibility and schedule checks. Policy constraints involving the number of selected consuming queries are a separate concern; see Section 5. +Migration targets include replacing `Materialization` and duplicated physical fields with checked properties, and replacing `ReaderConstraint::{IncompatibleOutputs, AllowedOutputs}` with compatibility and schedule checks. `ReaderConstraint::{MinimumQueryReaders, HighestSharedQueryMaterialization}` are deleted too; see Section 5. ### Stage 3: baseline selection and realization @@ -725,23 +725,9 @@ $$ The retention domain, upper bounds, activation linking, and all coefficient units must be specified by the implementation. If required retention depends on selected downstream queries, that dependence needs explicit linking constraints; it is not a fixed coefficient merely because it has a name. -### Consumer-count policies (J) +### No consumer-count policies -Two policies from today's code (`ReaderConstraint` in `crates/types/src/stages/physical/graph.rs`) restrict which queries may reach a node. They need reach indicators $a_{q,v} \in \{0,1\}$: query $q$'s selected plan contains $v$. - -$$ -a_{q,v} \ge z_{q,v},\qquad -a_{q,u} \ge a_{q,c} + y_{u,c,i} - 1,\qquad -a_{q,u} \le z_{q,u} + \sum_{(u,c,i)\in\mathcal E} w_{q,u,c,i},\qquad -w_{q,u,c,i} \le a_{q,c},\ w_{q,u,c,i} \le y_{u,c,i}. -$$ - -| Policy | Meaning | Constraint (J) | -|---|---|---| -| `MinimumQueryReaders { candidate, minimum }` | a selected candidate is worth keeping only when at least `minimum` queries reach it | $\sum_q a_{q,v} \ge \text{minimum}\cdot x_v$ | -| `HighestSharedQueryMaterialization { candidate }` | a non-summary query materialization is placed where sharing stops: it is a query output, or one of its selected consumers is reached by at most one query | $\sum_q z_{q,v} + \sum_{(v,c,i)} s_{v,c,i} \ge x_v$, with $s_{v,c,i} \le y_{v,c,i}$ and $\sum_q a_{q,c} \le 1 + \lvert\mathcal Q\rvert\,(1 - s_{v,c,i})$ | - -`ReaderConstraint::{IncompatibleOutputs, AllowedOutputs}` are not kept: the schedule check of Section 2 replaces them. +Today's `ReaderConstraint` (`crates/types/src/stages/physical/graph.rs`) is deleted with all four variants (decided 2026-11-06). `IncompatibleOutputs` and `AllowedOutputs` are replaced by the schedule check of Section 2. `MinimumQueryReaders` (keep a candidate only if enough queries reach it) and `HighestSharedQueryMaterialization` (materialize where sharing stops) are left to the cost model: materializing for one query, or below the point where sharing stops, costs more and is not chosen. There is no constraint (J). ### Out of this design From afee671a1bc29f769d58545c43419f400a759792 Mon Sep 17 00:00:00 2001 From: zzylol <50204836+zzylol@users.noreply.github.com> Date: Fri, 6 Nov 2026 03:47:11 +0000 Subject: [PATCH 22/44] docs(concepts): (K) encoding approved; worst-case counting affects bounds, not cost Co-Authored-By: Claude Opus 5.5 --- .../design_docs/concepts/memo-groups-properties-requirements.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/design_docs/concepts/memo-groups-properties-requirements.md b/docs/design_docs/concepts/memo-groups-properties-requirements.md index b674f226d..647b188ca 100644 --- a/docs/design_docs/concepts/memo-groups-properties-requirements.md +++ b/docs/design_docs/concepts/memo-groups-properties-requirements.md @@ -635,7 +635,7 @@ $$ - $E_c, F_{c,i}, D_c, H_{c,i} \ge 0$ are continuous. $\varepsilon^{\mathrm{loc}}_c$ and $\delta^{\mathrm{loc}}_c$ are the candidate's own guarantee (zero for an exact operator); $\beta_{c,i}$ and the metric rule come from `error_propagation`. - $M_{c,i}$ is any upper bound on $E_u$ over the producers of input $i$ (their largest feasible bound); $M_q$ likewise for the root. Exactly one $y_{u,c,i} = 1$ per selected input (constraint (A)), so $F_{c,i}$ equals the chosen producer's error bound. -- A producer reached by two paths contributes on both, which is the worst-case sum. The failure probabilities add by a union bound; a failure event reached twice is counted twice, which is conservative. +- A producer reached by two paths contributes on both, which is the worst-case sum. The failure probabilities add by a union bound; a failure event reached twice is counted twice, which is conservative. This affects only the error bounds, not cost: the objective counts each selected candidate once through $x_v$. Being conservative can only make the MILP pass over a cheaper plan that would in fact meet the target; it never accepts one that does not. Counting a shared failure event once needs failure-event identities, left for later. - Under the context extension (Section 5), the variables are indexed by context, $E_{\kappa,c}$, and the edges by $y_{\kappa,u,c,i}$. $\lambda_{q,c,i}$ is still used, but only for the pruning caps below, where a fixed lower bound suffices. From 816ab3696694015513650022730c5d5b2a9c7ce4 Mon Sep 17 00:00:00 2001 From: zzylol <50204836+zzylol@users.noreply.github.com> Date: Fri, 6 Nov 2026 03:54:06 +0000 Subject: [PATCH 23/44] docs(concepts): running example follows the metric rule (B reads a DDSketch state) Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01K1M6bq8xVaGKyinf4ohK6q --- .../memo-groups-properties-requirements.md | 21 ++++++++++--------- 1 file changed, 11 insertions(+), 10 deletions(-) diff --git a/docs/design_docs/concepts/memo-groups-properties-requirements.md b/docs/design_docs/concepts/memo-groups-properties-requirements.md index 647b188ca..003c6e75d 100644 --- a/docs/design_docs/concepts/memo-groups-properties-requirements.md +++ b/docs/design_docs/concepts/memo-groups-properties-requirements.md @@ -14,20 +14,21 @@ The expression includes the operator's arguments, including the summary family. ### Running example -Query A requests the p95 latency over one minute. Query B requests p99 divided by p50 over the same input. Both run every ten seconds. +Query A requests the p95 latency over one minute with a rank-error target. Query B requests p99 divided by p50 over the same input with a relative-error target. Both run every ten seconds. | Group | Result | Candidate expressions | |---|---|---| | G1 | Raw latency input | `Scan latency[1m]` | | G2 | KLL state over G1 | `SummaryAgg[KLL](G1)` with `k = 200` or `k = 345` | +| G2′ | DDSketch state over G1 | `SummaryAgg[DDSketch](G1)` with `alpha = 0.01` or `alpha = 0.02` | | G3 | Query A output | `SummaryEstimate[q=0.95](G2)` | -| G4 | p99 latency | `SummaryEstimate[q=0.99](G2)` | -| G5 | p50 latency | `SummaryEstimate[q=0.50](G2)` | +| G4 | p99 latency | `SummaryEstimate[q=0.99](G2′)` | +| G5 | p50 latency | `SummaryEstimate[q=0.50](G2′)` | | G6 | Query B output | `Divide(G4, G5)` | -G3, G4, and G5 choose a G2 producer independently. If all choose the same retained physical producer, they can reuse one computed state when their schedules and windows are compatible. An unmaterialized candidate may instead execute separately in different contexts; Section 5 explains that distinction. +KLL and DDSketch states are separate groups because the family is part of the expression. The metric rule of Section 4 decides which consumer may read which: KLL bounds **rank** error, which does not bound a quantile's value error or a ratio's relative error, so B's operands read the DDSketch state (relative value error), and A reads the KLL state. -The example shows representation and sharing. Accuracy is checked by Section 4, whose metric rule matters here: KLL bounds **rank** error, and rank error does not bound a quantile's value error or the relative error of a ratio. Query B's relative ratio target therefore needs operands with a relative value-error guarantee, e.g. a DDSketch state group over G1 with `alpha` candidates, unless a declared distribution assumption converts rank error to value error. Query A, with a rank target, can use the KLL candidates directly. +G4 and G5 choose a G2′ producer independently. If both choose the same retained physical producer, they reuse one computed state when their schedules and windows are compatible. An unmaterialized candidate may instead execute separately in different contexts; Section 5 explains that distinction. ### Notation used by the baseline model @@ -842,14 +843,14 @@ fn print(memo: &Memo, dag: &LogicalDag) { For a workload, choose one root candidate per query and initialize the pending queue with the distinct chosen roots. Deduplicate that queue before expansion so a shared root is processed once. -Before accuracy filtering, the running example has two alternatives for A and four for B: G4 and G5 choose independently between the two G2 builds. Joint enumeration therefore has eight choices. These are structural choices only; no claim is made that they meet B's ratio-accuracy requirement. +Before accuracy filtering, the running example has two alternatives for A (the two G2 builds) and four for B: G4 and G5 choose independently between the two G2′ builds. Joint enumeration therefore has eight choices. These are structural choices only; constraint (K) decides which meet each target. | B's p99 producer | B's p50 producer | State sharing within B | |---|---|---| -| `k = 200` | `k = 200` | One logical state candidate | -| `k = 200` | `k = 345` | Two state candidates | -| `k = 345` | `k = 200` | Two state candidates | -| `k = 345` | `k = 345` | One logical state candidate | +| `alpha = 0.01` | `alpha = 0.01` | One logical state candidate | +| `alpha = 0.01` | `alpha = 0.02` | Two state candidates | +| `alpha = 0.02` | `alpha = 0.01` | Two state candidates | +| `alpha = 0.02` | `alpha = 0.02` | One logical state candidate | The printer marks repeated candidates as references rather than expanding them twice. Physical sharing and cost additionally depend on lifetime and execution context. From f3687a0552c19aabb97c77247583305c33b99b33 Mon Sep 17 00:00:00 2001 From: zzylol <50204836+zzylol@users.noreply.github.com> Date: Fri, 6 Nov 2026 04:02:18 +0000 Subject: [PATCH 24/44] docs(concepts): retention need = time horizon of the served queries; latency ignored for now Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01K1M6bq8xVaGKyinf4ohK6q --- .../concepts/memo-groups-properties-requirements.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/design_docs/concepts/memo-groups-properties-requirements.md b/docs/design_docs/concepts/memo-groups-properties-requirements.md index 003c6e75d..e3f2d7699 100644 --- a/docs/design_docs/concepts/memo-groups-properties-requirements.md +++ b/docs/design_docs/concepts/memo-groups-properties-requirements.md @@ -706,7 +706,7 @@ To avoid confusing per-execution costs with the baseline's horizon-wide cost, us | $\rho_\kappa$ | Fixed number or expected number of context executions over the common horizon | | $h_v$ | Retention amount in a declared unit, such as retained windows | | $\pi_v$ | Horizon-wide retention cost per unit of retained amount | -| $\operatorname{need}_{\kappa,v,c,i}$ | Retention amount producer v needs to serve this consumer input in this context | +| $\operatorname{need}_{\kappa,v,c,i}$ | Retention amount producer v needs to serve this consumer input in this context: the time horizon the queries served by that context read, e.g. a 5m range over 10 s tumbling windows needs 30 windows. Query latency is ignored for now (decided 2026-11-06) | A proposed linear objective is: @@ -724,7 +724,7 @@ $$ h_v\ge\operatorname{need}_{\kappa,v,c,i}\,y_{\kappa,v,c,i}. $$ -The retention domain, upper bounds, activation linking, and all coefficient units must be specified by the implementation. If required retention depends on selected downstream queries, that dependence needs explicit linking constraints; it is not a fixed coefficient merely because it has a name. +The need is fixed per edge from the time horizon of the queries it serves, so constraint (R) is linear. It ignores query latency: a result is assumed read as soon as its evaluation starts, so a slow consumer that reads later does not extend the horizon. Modeling that needs the latency design, which is out of scope (Section 5). The retention unit (windows, or seconds of data) and an upper bound on $h_v$ are fixed per candidate when Stage 2 builds it. ### No consumer-count policies From 3f327a6ba8707558017c9ccbc25e67a1a578c991 Mon Sep 17 00:00:00 2001 From: zzylol <50204836+zzylol@users.noreply.github.com> Date: Fri, 6 Nov 2026 14:33:22 +0000 Subject: [PATCH 25/44] docs(concepts): CandidateForMemoGroupId.index -> index_in_group Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01K1M6bq8xVaGKyinf4ohK6q --- .../concepts/memo-groups-properties-requirements.md | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/docs/design_docs/concepts/memo-groups-properties-requirements.md b/docs/design_docs/concepts/memo-groups-properties-requirements.md index e3f2d7699..d902b8b5e 100644 --- a/docs/design_docs/concepts/memo-groups-properties-requirements.md +++ b/docs/design_docs/concepts/memo-groups-properties-requirements.md @@ -256,7 +256,8 @@ Implementation locations: `crates/logical-asap-optimization/src/pass1/per_comput ```rust pub struct CandidateForMemoGroupId { pub group: MemoGroupId, - pub index: usize, + /// Position in the group's `candidates` list. + pub index_in_group: usize, } impl Memo { @@ -809,7 +810,7 @@ fn candidates_of( group: MemoGroupId, ) -> impl Iterator + '_ { (0..memo.groups[group.0 as usize].candidates.len()) - .map(move |index| CandidateForMemoGroupId { group, index }) + .map(move |index_in_group| CandidateForMemoGroupId { group, index_in_group }) } /// Whether `from` reaches `to` through the producers chosen so far @@ -828,10 +829,10 @@ fn print(memo: &Memo, dag: &LogicalDag) { ) { let candidate = memo.candidate(c); if !seen.insert(c) { - println!("{}↑ G{}[{}] (shared)", " ".repeat(depth), c.group.0, c.index); + println!("{}↑ G{}[{}] (shared)", " ".repeat(depth), c.group.0, c.index_in_group); return; } - println!("{}G{}[{}] {} {:?}", " ".repeat(depth), c.group.0, c.index, + println!("{}G{}[{}] {} {:?}", " ".repeat(depth), c.group.0, c.index_in_group, candidate.expression.operator.name(), candidate.logical_properties); for &u in dag.producers.get(&c).into_iter().flatten() { node(memo, dag, u, depth + 1, seen); From 155a462499bd795fb86e3705edc39775ee9d215b Mon Sep 17 00:00:00 2001 From: zzylol <50204836+zzylol@users.noreply.github.com> Date: Fri, 6 Nov 2026 15:07:35 +0000 Subject: [PATCH 26/44] =?UTF-8?q?docs(concepts):=20define=20schedule=20?= =?UTF-8?q?=CF=83;=20a=20grid=20window=20is=20readable=20once=20it=20ends?= =?UTF-8?q?=20(no=20refresh=20interval)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01K1M6bq8xVaGKyinf4ohK6q --- .../concepts/memo-groups-properties-requirements.md | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-) diff --git a/docs/design_docs/concepts/memo-groups-properties-requirements.md b/docs/design_docs/concepts/memo-groups-properties-requirements.md index d902b8b5e..6523ddf8f 100644 --- a/docs/design_docs/concepts/memo-groups-properties-requirements.md +++ b/docs/design_docs/concepts/memo-groups-properties-requirements.md @@ -89,7 +89,7 @@ This is the single reference table for property dimensions. Stage descriptions b | Timing | `IngestionTime` or `QueryTime` | `ExecutionTiming` | | Lifetime | `None`: discard after the consuming execution; `OneEvaluation`: retain until consumers within that evaluation finish; `AcrossEvaluations`: retain across evaluations in the specified period. A non-`None` lifetime means materialized. | `Lifetime`; #1200 / #1214 | | Medium — Extension | `Memory` or `Disk`, carried by a materialized lifetime | `Medium`; #1214 | -| Schedule — Extension | For `OneEvaluation`, a periodic interval and phase; for windowed `AcrossEvaluations`, a tumbling-window grid phase. Unmaterialized candidates have no independent schedule. | `Schedule`; #1200 PR 2 | +| Schedule — Extension | When a materialized result can be read. For `OneEvaluation`: the periodic times it is computed (interval and phase). For windowed `AcrossEvaluations`: the phase of its tumbling-window grid; the width is the window layout's, and each window is readable as soon as it ends. Unmaterialized candidates have no independent schedule. | `Schedule`; #1200 PR 2 | Error guarantees are derived from parameters and the statistic being read. They include an error metric, a bound, and a failure probability; exact computation introduces no approximation error. The same state may support different readouts, so its guarantee must be interpreted for the relevant statistic. @@ -119,7 +119,11 @@ Physical requirements are derived from the consumer's timing and schedule, rathe QT means QueryTime; IT means IngestionTime. Schedule, data coverage, and retention checks still apply to allowed combinations. -**Periodic versus grid.** A periodic schedule with interval ten seconds and phase two produces at 2, 12, 22, … seconds. A five-second tumbling grid with phase zero partitions the input into windows [0, 5), [5, 10), … . The first describes evaluation times; the second describes window boundaries. Grid alignment alone does not ensure sufficient retained history. +**Periodic versus grid.** The schedule σ says when a materialized result can be read. +- **Periodic** (`OneEvaluation`): the result is computed at φ + I·ℤ and held until that evaluation's consumers finish. With interval 10 s and phase 2 s it exists at 2, 12, 22, … s. +- **Tumbling-window grid** (windowed `AcrossEvaluations`): the input is cut into windows [φ + k·w, φ + (k+1)·w), with w from the window layout ω. Each window is built once, when it ends, and is readable from then on (decided 2026-11-06; there is no separate refresh interval). A consumer can read a range that ends on a window boundary. With w = 10 s and phase 0, a `[1m]` query evaluated every 30 s at phase 0 merges the newest 6 windows; one evaluated at phase 15 s cannot, because its range ends inside a window. + +The grid stores only its phase: the width is ω's, so nothing is stored twice. Building each window when it ends costs the same number of window builds as building several at once later. Grid alignment alone does not ensure sufficient retained history; that is $h_v$ (Section 5). For the proposed periodic consumer path, let the consumer interval and phase be $I$ and $\varphi$. Let the producer interval and phase be $I_s$ and $\varphi_s$, and its tumbling width and grid phase be $w$ and $\varphi_g$. @@ -390,6 +394,8 @@ pub enum Lifetime { } pub enum Schedule { Periodic { interval_ms: u64, phase_ms: u64 }, + /// Phase of the window grid; the width is the window layout's (ω), and + /// each window is readable once it ends. TumblingWindowGrid { phase_ms: u64 }, } pub struct PhysicalProperties { From 0208f45fa09fc360c8ceac986c0fdd1311ba3616 Mon Sep 17 00:00:00 2001 From: zzylol <50204836+zzylol@users.noreply.github.com> Date: Fri, 6 Nov 2026 15:11:37 +0000 Subject: [PATCH 27/44] =?UTF-8?q?docs(concepts):=20drop=20the=20physical?= =?UTF-8?q?=20schedule=20=CF=83;=20p^phys=20=3D=20(=CF=84,=20=E2=84=93),?= =?UTF-8?q?=20grids=20anchored=20at=20phase=200?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A OneEvaluation result belongs to one evaluation context and runs on that context's schedule, chosen by Stage 3 through x_{κ,v}. Every tumbling grid is anchored at the epoch; a phase is absorbed by the width, whose candidates come from the gcd closure of ranges, steps and phases. A consumer in κ reads a windowed chain iff w | I_κ and w | φ_κ (delivers_on). The only invalid combination left is (IngestionTime, OneEvaluation). Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01K1M6bq8xVaGKyinf4ohK6q --- .../memo-groups-properties-requirements.md | 105 ++++++++---------- 1 file changed, 45 insertions(+), 60 deletions(-) diff --git a/docs/design_docs/concepts/memo-groups-properties-requirements.md b/docs/design_docs/concepts/memo-groups-properties-requirements.md index 6523ddf8f..4c737e0f7 100644 --- a/docs/design_docs/concepts/memo-groups-properties-requirements.md +++ b/docs/design_docs/concepts/memo-groups-properties-requirements.md @@ -85,19 +85,18 @@ This is the single reference table for property dimensions. Stage descriptions b | Dimension | Values and meaning | Code reference or target | |---|---|---| | Parameters | A predefined configuration list per family: KLL and KMV `k`; DDSketch `alpha`; HLL `precision`; CMS and CountSketch `width` and `depth`; heap variants also `heap_size`; UnivMon its sizing parameters. Exact operators have no sketch parameters. | `SketchParams` in `crates/types/src/ir/schema/sketch.rs`; predefined lists are the #1203 target | -| Window layout — Extension | No layout, or tumbling windows with a positive width. Candidate widths come from supported divisors derived from workload ranges and steps. Sliding merge/subtraction layouts remain future work. Window count is derived from range and width; retention amount is selected later. | `WindowLayout`; `pass2/memo/window_candidates.rs`; #1160 | +| Window layout — Extension | No layout, or tumbling windows with a positive width. Every grid is anchored at phase 0 (the epoch), so the width alone fixes it. Candidate widths come from the gcd closure of the queries' ranges, steps and phases. Sliding merge/subtraction layouts remain future work. Window count is derived from range and width; retention amount is selected later. | `WindowLayout`; `pass2/memo/window_candidates.rs`; #1160 | | Timing | `IngestionTime` or `QueryTime` | `ExecutionTiming` | | Lifetime | `None`: discard after the consuming execution; `OneEvaluation`: retain until consumers within that evaluation finish; `AcrossEvaluations`: retain across evaluations in the specified period. A non-`None` lifetime means materialized. | `Lifetime`; #1200 / #1214 | | Medium — Extension | `Memory` or `Disk`, carried by a materialized lifetime | `Medium`; #1214 | -| Schedule — Extension | When a materialized result can be read. For `OneEvaluation`: the periodic times it is computed (interval and phase). For windowed `AcrossEvaluations`: the phase of its tumbling-window grid; the width is the window layout's, and each window is readable as soon as it ends. Unmaterialized candidates have no independent schedule. | `Schedule`; #1200 PR 2 | Error guarantees are derived from parameters and the statistic being read. They include an error metric, a bound, and a failure probability; exact computation introduces no approximation error. The same state may support different readouts, so its guarantee must be interpreted for the relevant statistic. -The extended property tuples add window layout and schedule: +The extended logical properties add the window layout; the physical properties stay the formulation's: $$ p_v^{\mathrm{log}}=(\boldsymbol\theta_v,\omega_v),\qquad -p_v^{\mathrm{phys}}=(\tau_v,\ell_v,\sigma_v). +p_v^{\mathrm{phys}}=(\tau_v,\ell_v). $$ ### Logical requirements @@ -110,45 +109,41 @@ The required input group's identity already constrains the state family and data ### Physical requirements -Physical requirements are derived from the consumer's timing and schedule, rather than stored redundantly. For an unmaterialized consumer, the schedule comes from its execution context during selection. Stage 2 checks the parts known from candidate properties; Stage 3 checks any remaining context-dependent delivery condition. +Physical requirements are derived from the consumer's timing and execution context, rather than stored redundantly. Stage 2 checks timing against lifetime (the table below). Stage 3 checks alignment per context (below), with the schedule of the context the consumer runs in. | Consumer timing | QT, None | QT, OneEvaluation | QT, AcrossEvaluations | IT, None | IT, AcrossEvaluations | |---|---|---|---|---|---| | QueryTime | Allowed | Allowed | Allowed | Rejected | Allowed | | IngestionTime | Rejected | Rejected | Rejected | Allowed | Allowed | -QT means QueryTime; IT means IngestionTime. Schedule, data coverage, and retention checks still apply to allowed combinations. +QT means QueryTime; IT means IngestionTime. Alignment, data coverage, and retention checks still apply to allowed combinations. -**Periodic versus grid.** The schedule σ says when a materialized result can be read. -- **Periodic** (`OneEvaluation`): the result is computed at φ + I·ℤ and held until that evaluation's consumers finish. With interval 10 s and phase 2 s it exists at 2, 12, 22, … s. -- **Tumbling-window grid** (windowed `AcrossEvaluations`): the input is cut into windows [φ + k·w, φ + (k+1)·w), with w from the window layout ω. Each window is built once, when it ends, and is readable from then on (decided 2026-11-06; there is no separate refresh interval). A consumer can read a range that ends on a window boundary. With w = 10 s and phase 0, a `[1m]` query evaluated every 30 s at phase 0 merges the newest 6 windows; one evaluated at phase 15 s cannot, because its range ends inside a window. +**Alignment.** There is no physical schedule property. When a result can be read follows from its lifetime and window layout: +- **None** and **OneEvaluation**: the result is computed inside one evaluation context κ (Section 5) and read there. A OneEvaluation result belongs to one context: queries with the same schedule are evaluated together, and that evaluation's consumers share the result. When it runs is the context's schedule, and Stage 3 chooses it through $x_{\kappa,v}$, just as for ℓ = None. Sharing across differently scheduled queries through a gcd schedule is not offered: it is rarely beneficial and wastes runs. +- **Windowed AcrossEvaluations**: the input is cut into tumbling windows [k·w, (k+1)·w), with w from the window layout ω and the grid anchored at phase 0 (the epoch). Each window is built when it ends and can be read from then on. A consumer can read a range that ends on a window boundary. With w = 10 s, a `[1m]` query evaluated every 30 s at phase 0 merges the newest 6 windows; one evaluated at phase 15 s cannot, because its range ends inside a window; with that query in the workload, the gcd closure also offers w = 5 s, which both can read. -The grid stores only its phase: the width is ω's, so nothing is stored twice. Building each window when it ends costs the same number of window builds as building several at once later. Grid alignment alone does not ensure sufficient retained history; that is $h_v$ (Section 5). +A grid phase is not needed: a phase is absorbed by the width, because width candidates come from the gcd closure of the queries' ranges, steps and phases. For example, phases 0 and 5 s with w = 10 s give w = 5 s. Alignment alone does not ensure sufficient retained history; that is $h_v$ (Section 5). -For the proposed periodic consumer path, let the consumer interval and phase be $I$ and $\varphi$. Let the producer interval and phase be $I_s$ and $\varphi_s$, and its tumbling width and grid phase be $w$ and $\varphi_g$. +Let a context κ have interval $I_\kappa$ and phase $\varphi_\kappa$, and let a windowed producer have tumbling width $w$. A consumer running in κ can read the producer when: -| Producer schedule | Required alignment | +| Producer lifetime | Required alignment | |---|---| -| Periodic | $I_s\mid I$ and $\varphi\equiv\varphi_s\pmod{I_s}$ | -| Tumbling grid | $w\mid I$ and $\varphi\equiv\varphi_g\pmod w$ | -| None | Runs in the consuming context; no independent delivery schedule | +| None | Always: runs in κ | +| OneEvaluation | The result belongs to κ | +| AcrossEvaluations, tumbling width $w$ | $w\mid I_\kappa$ and $w\mid\varphi_\kappa$ | +| AcrossEvaluations, no window layout | Not offered yet: it needs its own delivery rule | The formulation also permits a single, non-periodic evaluation. The interval/phase interface below covers only periodic consumers. Rejecting non-periodic consumers is a current scope restriction of that proposed interface, not a mathematical impossibility; supporting them needs explicit evaluation-time membership checks. ### Invalid combinations -Checked constructors reject these combinations under the proposed schedule model: +The checked constructor rejects one combination: | Combination | Reason | |---|---| | IngestionTime with OneEvaluation | No query evaluation lifetime at ingestion time | -| None with an independent schedule | Execution follows the consuming context | -| Materialized lifetime without its required schedule | Delivery times or window grid would be unspecified | -| OneEvaluation with a grid, or windowed AcrossEvaluations with a periodic schedule | Schedule kind does not match this model's lifetime representation | -| Zero interval/width, or phase outside the canonical interval/width | Invalid schedule representation | -| Ingestion-time grid with nonzero phase | Ingestion-time grids are anchored at the epoch: the plan carries no origin | -A tumbling grid's width is the candidate's logical window layout $\omega_v$; it is not repeated in the schedule. Building a `PhysicalCandidate` checks the pair: a grid schedule needs a tumbling layout, and its phase must be below the width. Non-windowed AcrossEvaluations schedules need their own representation before that alternative can be offered. +A tumbling grid's width is the candidate's logical window layout $\omega_v$ and its phase is 0, so no physical field repeats or extends it. ### Decision tree and order of decisions @@ -166,7 +161,7 @@ flowchart TD R2 --> T3["θ: k = …
(prune: pruning cap)"] T2 --> W1["ω: no windows"] - T2 --> W2["ω: tumbling, width w
(prune: w must divide
the group's query range)"] + T2 --> W2["ω: tumbling, width w
from the gcd closure of ranges,
steps and phases (prune: w must
divide the group's query range)"] W2 --> P1["τ = QueryTime"] W2 --> P2["τ = IngestionTime
(prune: no ingestion pipeline)"] @@ -177,22 +172,17 @@ flowchart TD P2 --> L4["ℓ = None"] P2 --> L5["ℓ = AcrossEvaluations"] P2 -. invalid .-> L6["ℓ = OneEvaluation"] - - L2 --> S1["σ: Periodic (I, φ)
from the gcd closure
of the queries' schedules"] - L3 --> S2["σ: grid phase φ mod w"] - L5 --> S3["σ: grid phase 0"] ``` | Level | Decides | Stage | Values offered | Pruned when | |---|---|---|---|---| | 1 | expression $e_v$ | Pass 1 and Pass 2 rules | each way to compute the group | the executor cannot build the family | | 2 | $\boldsymbol\theta_v$ | Pass 1 | the family's predefined list | the pruning cap (Section 4) | -| 3 | $\omega_v$ — Extension | Pass 2 | tumbling widths from the gcd closure of the queries' ranges and steps | the width does not divide the group's query range | +| 3 | $\omega_v$ — Extension | Pass 2 | tumbling widths from the gcd closure of the queries' ranges, steps and phases | the width does not divide the group's query range | | 4 | $\tau_v$ | Stage 2 | QueryTime, IngestionTime | the source has no ingestion pipeline | | 5 | $\ell_v$ | Stage 2 | None, OneEvaluation, AcrossEvaluations | invalid with $\tau_v$; the deployment keeps no query-time results | -| 6 | $\sigma_v$ — Extension | Stage 2 | the gcd closure of the schedules of the queries reaching the group | only when $\ell_v \ne$ None | -The tree does not decide which producer each consumer input reads ($y$), where an unmaterialized candidate runs (its context, Section 5), or how much a materialized candidate retains ($h_v$). Those are Stage 3 choices. +The tree does not decide which producer each consumer input reads ($y$), in which context a candidate with ℓ = None or OneEvaluation runs (Section 5), or how much an AcrossEvaluations candidate retains ($h_v$). Those are Stage 3 choices. **The order does not change the candidate set.** The leaves are the product of the dimensions' values minus invalid combinations and pruned values. Each prune rule reads only the value it removes and values decided above it, so any order that keeps each prune rule below the dimensions it reads yields the same leaves. @@ -200,7 +190,7 @@ The tree does not decide which producer each consumer input reads ($y$), where a 1. **Logical before physical.** Stage 1 hash-conses groups and computes $\alpha^{\log}$ once, before Stage 2 multiplies each logical candidate by its physical alternatives. 2. **Strong prunes early.** Executor support and the pruning cap act before windows and physical properties multiply the tree. -3. **Dependent dimensions adjacent.** $\tau \to \ell \to \sigma$ rejects each invalid combination at the first level that can detect it. +3. **Dependent dimensions adjacent.** $\tau \to \ell$ rejects the invalid combination at the first level that can detect it. 4. **Deployment-wide prunes as early as possible.** Without an ingestion pipeline the IngestionTime subtree is pruned once per candidate; moving $\tau$ above $\omega$ would prune it once per group, with the same leaves. Alternative orders worth measuring: $\tau$ first, when most sources lack an ingestion pipeline; $\omega$ before $\boldsymbol\theta$, when queries' ranges prune many widths; and requirement-driven (top-down) enumeration, which visits fewer nodes but must take the union of all consumers' requirements so that sharing is not lost. @@ -392,40 +382,35 @@ pub enum Lifetime { OneEvaluation(Medium), AcrossEvaluations(Medium), } -pub enum Schedule { - Periodic { interval_ms: u64, phase_ms: u64 }, - /// Phase of the window grid; the width is the window layout's (ω), and - /// each window is readable once it ends. - TumblingWindowGrid { phase_ms: u64 }, -} pub struct PhysicalProperties { timing: ExecutionTiming, lifetime: Lifetime, - schedule: Option, } impl PhysicalProperties { + /// Rejects only (IngestionTime, OneEvaluation). pub fn new( timing: ExecutionTiming, lifetime: Lifetime, - schedule: Option, ) -> Result; pub fn timing(&self) -> ExecutionTiming; pub fn lifetime(&self) -> Lifetime; - pub fn schedule(&self) -> Option<&Schedule>; pub fn is_materialized(&self) -> bool; } +/// When a context runs: φ + I·ℤ. pub struct EvaluationSchedule { interval_ms: u64, phase_ms: u64, } impl PhysicalCandidate { - /// Whether this producer delivers on `times` (Section 2's alignment table). - /// Reads the grid width from the candidate's logical window layout. + /// Whether a consumer running on `times` can read this producer + /// (Section 2's alignment table): w | I and w | φ for a windowed + /// AcrossEvaluations producer, with w read from its window layout; + /// true for ℓ = None and OneEvaluation, which run in that context. pub fn delivers_on(&self, memo: &Memo, times: &EvaluationSchedule) -> bool; } -/// α^phys: timing and lifetime (Section 2's table), and, for a materialized -/// consumer, that the producer delivers on the consumer's own schedule. +/// α^phys: timing and lifetime only (Section 2's table). Alignment is +/// checked per context in Stage 3 with `delivers_on`. pub fn satisfies_physical( memo: &Memo, producer: &PhysicalCandidate, @@ -471,9 +456,9 @@ pub struct PhysicalCandidateKey { } ``` -`PhysicalCandidateId` indexes an emitted candidate. `PhysicalCandidateKey` describes an alternative rejected before it received an ID, allowing diagnostics to explain pruning. The admissible-edge accessor returns edge IDs, not the unfiltered producer set from Section 1. `delivers_on` and `satisfies_physical` take candidates rather than bare properties because a tumbling grid's width lives in the logical window layout. +`PhysicalCandidateId` indexes an emitted candidate. `PhysicalCandidateKey` describes an alternative rejected before it received an ID, allowing diagnostics to explain pruning. The admissible-edge accessor returns edge IDs, not the unfiltered producer set from Section 1. `delivers_on` takes a candidate rather than bare properties because a tumbling grid's width lives in the logical window layout. -Migration targets include replacing `Materialization` and duplicated physical fields with checked properties, and replacing `ReaderConstraint::{IncompatibleOutputs, AllowedOutputs}` with compatibility and schedule checks. `ReaderConstraint::{MinimumQueryReaders, HighestSharedQueryMaterialization}` are deleted too; see Section 5. +Migration targets include replacing `Materialization` and duplicated physical fields with checked properties, and replacing `ReaderConstraint::{IncompatibleOutputs, AllowedOutputs}` with compatibility and alignment checks. `ReaderConstraint::{MinimumQueryReaders, HighestSharedQueryMaterialization}` are deleted too; see Section 5. ### Stage 3: baseline selection and realization @@ -537,7 +522,7 @@ pub struct DagOfSelectedCandidates { } ``` -The realizer uses the selected dependencies to build the executable graph. Export carries each node's timing, lifetime, and supported schedule fields. Under the context extension, multiple selected execution nodes may refer to the same physical candidate; realization must preserve their contexts rather than merge them solely by candidate ID. +The realizer uses the selected dependencies to build the executable graph. Export carries each node's timing and lifetime. Under the context extension, multiple selected execution nodes may refer to the same physical candidate; realization must preserve their contexts rather than merge them solely by candidate ID. ## 4. Accuracy and safe pruning — Extension (#1207) @@ -670,7 +655,7 @@ These extensions go beyond constraints (A)–(I) and the formulation's fixed-cos ### Execution identity -A materialized candidate has its own delivery schedule. An unmaterialized candidate runs inside an execution that needs it. For example, a finalize candidate used by queries A and B may execute in both query contexts even though Stage 2 enumerates it only once. +An AcrossEvaluations candidate has its own computation: each window is built when it ends. A candidate with ℓ = None runs inside an execution that needs it. For example, a finalize candidate used by queries A and B may execute in both query contexts even though Stage 2 enumerates it only once. A OneEvaluation candidate also runs inside one evaluation context κ: queries with the same schedule are evaluated together, and that evaluation's consumers share the result. A **context** identifies a class of executions with a common owner and schedule across the accounting horizon. It is not necessarily one timestamp. @@ -689,16 +674,16 @@ impl Context { } ``` -An unmaterialized chain inherits its consuming context. Encountering a materialized producer switches to that producer's own computation context when traversing its inputs. The context identifies where the operation runs; schedule compatibility decides whether a materialized input can supply it. +A chain of ℓ = None and OneEvaluation candidates inherits its consuming context. Encountering an AcrossEvaluations producer switches to that producer's own computation context when traversing its inputs. The context identifies where the operation runs; `delivers_on` with the context's schedule decides whether a windowed producer can supply it. | Extended variable | Meaning | |---|---| -| $x_v^{\mathrm{mat}}$ | Materialized candidate selected | -| $x_{\kappa,v}$ | Unmaterialized candidate executes in context kappa | +| $x_v^{\mathrm{mat}}$ | AcrossEvaluations candidate selected | +| $x_{\kappa,v}$ | Candidate with ℓ = None or OneEvaluation executes in context kappa | | $y_{\kappa,u,c,i}$ | Input edge selected for a consumer execution in context kappa | -| $h_v$ | Selected retention amount for a materialized producer | +| $h_v$ | Selected retention amount for an AcrossEvaluations producer | -The unindexed edge variable belongs to the baseline only. In the extension, every edge is indexed by its consumer's context, including the dedicated computation context of a materialized consumer. This avoids silently switching between two meanings of `y`. +The unindexed edge variable belongs to the baseline only. In the extension, every edge is indexed by its consumer's context, including the dedicated computation context of an AcrossEvaluations consumer. This avoids silently switching between two meanings of `y`. Activation, input satisfaction, utilization, output selection, and acyclicity must be defined for these execution instances. One binary per context/candidate assumes at most one reusable execution of that candidate within that context. If two uses require independent computations, the context or execution identity must distinguish them. @@ -708,8 +693,8 @@ To avoid confusing per-execution costs with the baseline's horizon-wide cost, us | Symbol | Definition | |---|---| -| $W_v^{\mathrm{mat}}$ | Fixed computation and access cost over the horizon for a materialized candidate's own schedule, excluding variable retention | -| $c_{\kappa,v}$ | Cost of one unmaterialized execution in the specified context | +| $W_v^{\mathrm{mat}}$ | Fixed computation and access cost over the horizon for an AcrossEvaluations candidate, one build per window, excluding variable retention | +| $c_{\kappa,v}$ | Cost of one execution in the specified context of a candidate with ℓ = None or OneEvaluation, including a OneEvaluation result's storage within that evaluation | | $\rho_\kappa$ | Fixed number or expected number of context executions over the common horizon | | $h_v$ | Retention amount in a declared unit, such as retained windows | | $\pi_v$ | Horizon-wide retention cost per unit of retained amount | @@ -719,9 +704,9 @@ A proposed linear objective is: $$ \min\left( -\sum_{v\text{ materialized}}W_v^{\mathrm{mat}}x_v^{\mathrm{mat}} -+\sum_{\kappa,v\text{ unmaterialized}}\rho_\kappa c_{\kappa,v}x_{\kappa,v} -+\sum_{v\text{ materialized}}\pi_v h_v +\sum_{v\text{ across evaluations}}W_v^{\mathrm{mat}}x_v^{\mathrm{mat}} ++\sum_{\kappa,\ v\text{ None or OneEvaluation}}\rho_\kappa c_{\kappa,v}x_{\kappa,v} ++\sum_{v\text{ across evaluations}}\pi_v h_v \right). $$ @@ -735,7 +720,7 @@ The need is fixed per edge from the time horizon of the queries it serves, so co ### No consumer-count policies -Today's `ReaderConstraint` (`crates/types/src/stages/physical/graph.rs`) is deleted with all four variants (decided 2026-11-06). `IncompatibleOutputs` and `AllowedOutputs` are replaced by the schedule check of Section 2. `MinimumQueryReaders` (keep a candidate only if enough queries reach it) and `HighestSharedQueryMaterialization` (materialize where sharing stops) are left to the cost model: materializing for one query, or below the point where sharing stops, costs more and is not chosen. There is no constraint (J). +Today's `ReaderConstraint` (`crates/types/src/stages/physical/graph.rs`) is deleted with all four variants (decided 2026-11-06). `IncompatibleOutputs` and `AllowedOutputs` are replaced by the alignment check of Section 2. `MinimumQueryReaders` (keep a candidate only if enough queries reach it) and `HighestSharedQueryMaterialization` (materialize where sharing stops) are left to the cost model: materializing for one query, or below the point where sharing stops, costs more and is not chosen. There is no constraint (J). ### Out of this design @@ -870,4 +855,4 @@ $$ Here, $P^{\mathrm{log,adm}}_{v,i}$ is the set of logically compatible producers for that input, and a leaf has count one. The bound overcounts shared subgraphs. For a cyclic memo, this recurrence is not well-founded; use cycle-aware enumeration or an explicitly bounded unfolding instead. -For physical inspection, replace logical candidates and compatibility checks with physical candidates and admissible edges. Under the execution-context extension, candidate-level listing alone is not a list of executable plans: instantiate contexts and apply schedule, retention, root-accuracy, and other enabled constraints before comparing results with Stage 3. +For physical inspection, replace logical candidates and compatibility checks with physical candidates and admissible edges. Under the execution-context extension, candidate-level listing alone is not a list of executable plans: instantiate contexts and apply alignment, retention, root-accuracy, and other enabled constraints before comparing results with Stage 3. From 407a533f44057e54bf1e44d7eb5120025cefd3d8 Mon Sep 17 00:00:00 2001 From: zzylol <50204836+zzylol@users.noreply.github.com> Date: Fri, 6 Nov 2026 15:29:36 +0000 Subject: [PATCH 28/44] docs(concepts): theta listed, floors free; heap 1-2-5 steps; one evaluation context per schedule MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A sketch build's θ must be in its family's list; a requirement's floor is any value of the parameter type (the weakest θ with ε ≤ cap, or a direct need such as heap_size ≥ k). Heap variants take heap sizes from the 1-2-5 series 10 … 10 000; KMV and UnivMon distinct-count lists stop at ε = 0.01. satisfies_physical takes producer and consumer only. Context::Evaluation is keyed by the evaluation schedule. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01K1M6bq8xVaGKyinf4ohK6q --- .../memo-groups-properties-requirements.md | 27 +++++++++---------- 1 file changed, 13 insertions(+), 14 deletions(-) diff --git a/docs/design_docs/concepts/memo-groups-properties-requirements.md b/docs/design_docs/concepts/memo-groups-properties-requirements.md index 4c737e0f7..f1efd530d 100644 --- a/docs/design_docs/concepts/memo-groups-properties-requirements.md +++ b/docs/design_docs/concepts/memo-groups-properties-requirements.md @@ -84,7 +84,7 @@ This is the single reference table for property dimensions. Stage descriptions b | Dimension | Values and meaning | Code reference or target | |---|---|---| -| Parameters | A predefined configuration list per family: KLL and KMV `k`; DDSketch `alpha`; HLL `precision`; CMS and CountSketch `width` and `depth`; heap variants also `heap_size`; UnivMon its sizing parameters. Exact operators have no sketch parameters. | `SketchParams` in `crates/types/src/ir/schema/sketch.rs`; predefined lists are the #1203 target | +| Parameters | A predefined configuration list per family: KLL and KMV `k`; DDSketch `alpha`; HLL `precision`; CMS and CountSketch `width` and `depth`; heap variants also `heap_size`; UnivMon its sizing parameters. The lists are today's sizing over ε = 0.1 down to 0.001; KMV's and UnivMon's distinct-count entries stop at ε = 0.01. A heap variant combines each width × depth with heap sizes from the 1-2-5 series 10, 20, 50, …, 10 000: a top-k consumer's floor is heap_size ≥ k, and the smallest listed heap that meets it is the next step up, at most 2.5× too large. A sketch build's parameters must be in its family's list; exact operators have no sketch parameters. | `SketchParams` in `crates/types/src/ir/schema/sketch.rs`; predefined lists are the #1203 target | | Window layout — Extension | No layout, or tumbling windows with a positive width. Every grid is anchored at phase 0 (the epoch), so the width alone fixes it. Candidate widths come from the gcd closure of the queries' ranges, steps and phases. Sliding merge/subtraction layouts remain future work. Window count is derived from range and width; retention amount is selected later. | `WindowLayout`; `pass2/memo/window_candidates.rs`; #1160 | | Timing | `IngestionTime` or `QueryTime` | `ExecutionTiming` | | Lifetime | `None`: discard after the consuming execution; `OneEvaluation`: retain until consumers within that evaluation finish; `AcrossEvaluations`: retain across evaluations in the specified period. A non-`None` lifetime means materialized. | `Lifetime`; #1200 / #1214 | @@ -101,7 +101,7 @@ $$ ### Logical requirements -The proposed `LogicalRequirement` stores a **parameter floor**: the weakest configuration in a family's ordered list that satisfies the applicable input error cap. Section 4 defines the cap and distinguishes validity from pruning. It does not allocate a fixed share of the query's error budget to each input. +The proposed `LogicalRequirement` stores a **parameter floor** $\boldsymbol\theta^{\min}$: the weakest parameters whose ε is at most the applicable input error cap, or a direct need such as heap_size ≥ k for a top-k. The floor is any value of the family's parameter type and need not be in the family's list; a listed producer satisfies it when $\boldsymbol\theta_u \succeq \boldsymbol\theta^{\min}$. Section 4 defines the cap and distinguishes validity from pruning. It does not allocate a fixed share of the query's error budget to each input. For KLL, a larger `k` satisfies a smaller minimum `k`. For DDSketch, a smaller `alpha` can be stronger. Multi-parameter families require the family's comparison rule, including parameters needed for failure guarantees. A single floor is valid only if the accepted configurations form an upward-closed set with a representable least element. Otherwise, compatibility must use an explicit admissible set or predicate; it must not invent a unique minimum. @@ -292,6 +292,9 @@ pub enum WindowLayout { } impl LogicalProperties { + /// Rejects parameters missing on a sketch build, present on an exact + /// operator, of another family or not in the family's list, and a window + /// of width 0 or on an operator that cannot be windowed. pub fn new( expression: &CandidateExpression, parameters: Option, @@ -309,9 +312,8 @@ pub struct LogicalRequirement { } impl LogicalRequirement { - pub fn new( - parameter_floor: Option, - ) -> Result; + /// Any floor of the family's parameter type, listed or not. + pub fn new(parameter_floor: Option) -> Self; pub fn parameter_floor(&self) -> Option<&SketchParams>; } @@ -411,11 +413,7 @@ impl PhysicalCandidate { } /// α^phys: timing and lifetime only (Section 2's table). Alignment is /// checked per context in Stage 3 with `delivers_on`. -pub fn satisfies_physical( - memo: &Memo, - producer: &PhysicalCandidate, - consumer: &PhysicalCandidate, -) -> bool; +pub fn satisfies_physical(producer: &PhysicalCandidate, consumer: &PhysicalCandidate) -> bool; pub struct PhysicalCandidateId(pub usize); pub struct PhysicalEdgeId(pub usize); @@ -655,21 +653,22 @@ These extensions go beyond constraints (A)–(I) and the formulation's fixed-cos ### Execution identity -An AcrossEvaluations candidate has its own computation: each window is built when it ends. A candidate with ℓ = None runs inside an execution that needs it. For example, a finalize candidate used by queries A and B may execute in both query contexts even though Stage 2 enumerates it only once. A OneEvaluation candidate also runs inside one evaluation context κ: queries with the same schedule are evaluated together, and that evaluation's consumers share the result. +An AcrossEvaluations candidate has its own computation: each window is built when it ends. A candidate with ℓ = None runs inside an execution that needs it. For example, a finalize candidate used by queries A (every 10 s) and C (every 15 s) may execute in both evaluation contexts even though Stage 2 enumerates it only once. A OneEvaluation candidate also runs inside one evaluation context κ, and that evaluation's consumers share the result. -A **context** identifies a class of executions with a common owner and schedule across the accounting horizon. It is not necessarily one timestamp. +A **context** identifies a class of executions with a common owner and schedule across the accounting horizon. It is not necessarily one timestamp. There is one evaluation context per evaluation schedule (interval, phase): all queries with that schedule are evaluated together and share its OneEvaluation results. Queries A and B of the running example, both every 10 s at phase 0, form one context. A non-periodic (one-off) query has no evaluation context yet: it is out of scope, as in Section 2. ```rust pub struct ContextId(pub usize); pub enum Context { - QueryEvaluation(QueryId), + Evaluation(EvaluationSchedule), IngestionPipeline(RefreshWindowId), Materialized(PhysicalCandidateId), } impl Context { - // Periodic schedule when supported; None is not an implicit compatibility pass. + // An evaluation context's schedule; None for the other contexts, which + // is not an implicit compatibility pass. pub fn evaluation_schedule(&self) -> Option; } ``` From 66275132d1a9d0b935841140ae7039bd5800f34a Mon Sep 17 00:00:00 2001 From: zzylol <50204836+zzylol@users.noreply.github.com> Date: Fri, 6 Nov 2026 15:37:44 +0000 Subject: [PATCH 29/44] docs(concepts): rename the interfaces as agreed with the reviewer LogicalPropertiesError, PhysicalPropertiesError, LogicalRequirements (parameter_floor_for_required_accuracy), producer_satisfies_logical_requirements, producer_satisfies_physical_requirements, StorageMedium, ExecutionContext and ExecutionContextId; PhysicalCandidate / PhysicalCandidateKey fields logical_candidate_id and physical_properties; edge fields producer_candidate_id and consumer_candidate_id; PhysicalASAPMemoCandidates fields logical_candidates and physical_candidates. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01K1M6bq8xVaGKyinf4ohK6q --- .../memo-groups-properties-requirements.md | 65 ++++++++++--------- 1 file changed, 34 insertions(+), 31 deletions(-) diff --git a/docs/design_docs/concepts/memo-groups-properties-requirements.md b/docs/design_docs/concepts/memo-groups-properties-requirements.md index f1efd530d..af6c6c38c 100644 --- a/docs/design_docs/concepts/memo-groups-properties-requirements.md +++ b/docs/design_docs/concepts/memo-groups-properties-requirements.md @@ -88,7 +88,7 @@ This is the single reference table for property dimensions. Stage descriptions b | Window layout — Extension | No layout, or tumbling windows with a positive width. Every grid is anchored at phase 0 (the epoch), so the width alone fixes it. Candidate widths come from the gcd closure of the queries' ranges, steps and phases. Sliding merge/subtraction layouts remain future work. Window count is derived from range and width; retention amount is selected later. | `WindowLayout`; `pass2/memo/window_candidates.rs`; #1160 | | Timing | `IngestionTime` or `QueryTime` | `ExecutionTiming` | | Lifetime | `None`: discard after the consuming execution; `OneEvaluation`: retain until consumers within that evaluation finish; `AcrossEvaluations`: retain across evaluations in the specified period. A non-`None` lifetime means materialized. | `Lifetime`; #1200 / #1214 | -| Medium — Extension | `Memory` or `Disk`, carried by a materialized lifetime | `Medium`; #1214 | +| Storage medium — Extension | `Memory` or `Disk`, carried by a materialized lifetime | `StorageMedium`; #1214 | Error guarantees are derived from parameters and the statistic being read. They include an error metric, a bound, and a failure probability; exact computation introduces no approximation error. The same state may support different readouts, so its guarantee must be interpreted for the relevant statistic. @@ -101,7 +101,7 @@ $$ ### Logical requirements -The proposed `LogicalRequirement` stores a **parameter floor** $\boldsymbol\theta^{\min}$: the weakest parameters whose ε is at most the applicable input error cap, or a direct need such as heap_size ≥ k for a top-k. The floor is any value of the family's parameter type and need not be in the family's list; a listed producer satisfies it when $\boldsymbol\theta_u \succeq \boldsymbol\theta^{\min}$. Section 4 defines the cap and distinguishes validity from pruning. It does not allocate a fixed share of the query's error budget to each input. +The proposed `LogicalRequirements` stores a **parameter floor** $\boldsymbol\theta^{\min}$: the weakest parameters whose ε is at most the applicable input error cap, or a direct need such as heap_size ≥ k for a top-k. The floor is any value of the family's parameter type and need not be in the family's list; a listed producer satisfies it when $\boldsymbol\theta_u \succeq \boldsymbol\theta^{\min}$. Section 4 defines the cap and distinguishes validity from pruning. It does not allocate a fixed share of the query's error budget to each input. For KLL, a larger `k` satisfies a smaller minimum `k`. For DDSketch, a smaller `alpha` can be stronger. Multi-parameter families require the family's comparison rule, including parameters needed for failure guarantees. A single floor is valid only if the accepted configurations form an upward-closed set with a representable least element. Otherwise, compatibility must use an explicit admissible set or predicate; it must not invent a unique minimum. @@ -276,7 +276,7 @@ pub struct CandidateExpression { pub struct CandidateForMemoGroup { pub expression: CandidateExpression, pub logical_properties: LogicalProperties, - pub input_requirements: Vec, + pub input_requirements: Vec, pub fused_groups: Vec, // Diagnostic provenance; not a planning decision. pub origin: OriginOfCandidateForMemoGroup, @@ -299,7 +299,7 @@ impl LogicalProperties { expression: &CandidateExpression, parameters: Option, window_layout: Option, - ) -> Result; + ) -> Result; pub fn parameters(&self) -> Option<&SketchParams>; pub fn window_layout(&self) -> Option<&WindowLayout>; @@ -307,20 +307,20 @@ impl LogicalProperties { pub fn parameter_list(family: SketchAlgorithm) -> &'static [SketchParams]; -pub struct LogicalRequirement { - parameter_floor: Option, +pub struct LogicalRequirements { + parameter_floor_for_required_accuracy: Option, } -impl LogicalRequirement { +impl LogicalRequirements { /// Any floor of the family's parameter type, listed or not. - pub fn new(parameter_floor: Option) -> Self; + pub fn new(parameter_floor_for_required_accuracy: Option) -> Self; - pub fn parameter_floor(&self) -> Option<&SketchParams>; + pub fn parameter_floor_for_required_accuracy(&self) -> Option<&SketchParams>; } -pub fn satisfies_logical( +pub fn producer_satisfies_logical_requirements( producer: &LogicalProperties, - requirement: &LogicalRequirement, + requirement: &LogicalRequirements, ) -> bool; ``` @@ -351,8 +351,8 @@ After the rules, Pass 2 builds the logical edges and applies the pruning caps of fn apply(memo: &mut Memo) -> Result<(), Pass2RuleError>; pub struct LogicalEdge { - pub producer: CandidateForMemoGroupId, - pub consumer: CandidateForMemoGroupId, + pub producer_candidate_id: CandidateForMemoGroupId, + pub consumer_candidate_id: CandidateForMemoGroupId, pub input_index_of_consumer: usize, pub satisfies_logical_requirement: bool, } @@ -374,15 +374,15 @@ pub enum ExecutionTiming { QueryTime, } -pub enum Medium { +pub enum StorageMedium { Memory, Disk, } pub enum Lifetime { None, - OneEvaluation(Medium), - AcrossEvaluations(Medium), + OneEvaluation(StorageMedium), + AcrossEvaluations(StorageMedium), } pub struct PhysicalProperties { timing: ExecutionTiming, @@ -393,7 +393,7 @@ impl PhysicalProperties { pub fn new( timing: ExecutionTiming, lifetime: Lifetime, - ) -> Result; + ) -> Result; pub fn timing(&self) -> ExecutionTiming; pub fn lifetime(&self) -> Lifetime; pub fn is_materialized(&self) -> bool; @@ -413,24 +413,27 @@ impl PhysicalCandidate { } /// α^phys: timing and lifetime only (Section 2's table). Alignment is /// checked per context in Stage 3 with `delivers_on`. -pub fn satisfies_physical(producer: &PhysicalCandidate, consumer: &PhysicalCandidate) -> bool; +pub fn producer_satisfies_physical_requirements( + producer: &PhysicalCandidate, + consumer: &PhysicalCandidate, +) -> bool; pub struct PhysicalCandidateId(pub usize); pub struct PhysicalEdgeId(pub usize); pub struct PhysicalCandidate { - pub logical: CandidateForMemoGroupId, - pub properties: PhysicalProperties, + pub logical_candidate_id: CandidateForMemoGroupId, + pub physical_properties: PhysicalProperties, } pub struct PhysicalEdge { - pub producer: PhysicalCandidateId, - pub consumer: PhysicalCandidateId, + pub producer_candidate_id: PhysicalCandidateId, + pub consumer_candidate_id: PhysicalCandidateId, pub input_index_of_consumer: usize, pub satisfies_logical_requirement: bool, pub satisfies_physical_requirement: bool, } pub struct PhysicalASAPMemoCandidates { - pub logical: LogicalASAPMemoCandidates, - pub candidates: Vec, + pub logical_candidates: LogicalASAPMemoCandidates, + pub physical_candidates: Vec, pub query_roots: Vec>, pub pruned_candidates: Vec>, } @@ -449,8 +452,8 @@ impl PhysicalASAPMemoCandidates { // Identifies a rejected alternative by content: it has no allocated candidate ID. pub struct PhysicalCandidateKey { - pub logical: CandidateForMemoGroupId, - pub properties: PhysicalProperties, + pub logical_candidate_id: CandidateForMemoGroupId, + pub physical_properties: PhysicalProperties, } ``` @@ -580,7 +583,7 @@ pub struct ErrorPropagation { pub validity_cap_per_input: Vec>, } -impl LogicalRequirement { +impl LogicalRequirements { pub fn error_propagation( consumer: &Operator, ) -> ErrorPropagation; @@ -658,15 +661,15 @@ An AcrossEvaluations candidate has its own computation: each window is built whe A **context** identifies a class of executions with a common owner and schedule across the accounting horizon. It is not necessarily one timestamp. There is one evaluation context per evaluation schedule (interval, phase): all queries with that schedule are evaluated together and share its OneEvaluation results. Queries A and B of the running example, both every 10 s at phase 0, form one context. A non-periodic (one-off) query has no evaluation context yet: it is out of scope, as in Section 2. ```rust -pub struct ContextId(pub usize); +pub struct ExecutionContextId(pub usize); -pub enum Context { +pub enum ExecutionContext { Evaluation(EvaluationSchedule), IngestionPipeline(RefreshWindowId), Materialized(PhysicalCandidateId), } -impl Context { +impl ExecutionContext { // An evaluation context's schedule; None for the other contexts, which // is not an implicit compatibility pass. pub fn evaluation_schedule(&self) -> Option; @@ -771,7 +774,7 @@ fn expand( for choice in cartesian_product(inputs.iter().enumerate().map(|(i, &group)| { candidates_of(memo, group) .filter(|&u| { - satisfies_logical( + producer_satisfies_logical_requirements( &memo.candidate(u).logical_properties, &candidate.input_requirements[i], ) From 79ac5fd4bcb900cd370cdbf00f23e7cd8a0b6665 Mon Sep 17 00:00:00 2001 From: zzylol <50204836+zzylol@users.noreply.github.com> Date: Fri, 6 Nov 2026 15:38:29 +0000 Subject: [PATCH 30/44] docs(concepts): note the empty parameter lists of unsized families (#1228) Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01K1M6bq8xVaGKyinf4ohK6q --- .../design_docs/concepts/memo-groups-properties-requirements.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/design_docs/concepts/memo-groups-properties-requirements.md b/docs/design_docs/concepts/memo-groups-properties-requirements.md index af6c6c38c..d647f4bb1 100644 --- a/docs/design_docs/concepts/memo-groups-properties-requirements.md +++ b/docs/design_docs/concepts/memo-groups-properties-requirements.md @@ -84,7 +84,7 @@ This is the single reference table for property dimensions. Stage descriptions b | Dimension | Values and meaning | Code reference or target | |---|---|---| -| Parameters | A predefined configuration list per family: KLL and KMV `k`; DDSketch `alpha`; HLL `precision`; CMS and CountSketch `width` and `depth`; heap variants also `heap_size`; UnivMon its sizing parameters. The lists are today's sizing over ε = 0.1 down to 0.001; KMV's and UnivMon's distinct-count entries stop at ε = 0.01. A heap variant combines each width × depth with heap sizes from the 1-2-5 series 10, 20, 50, …, 10 000: a top-k consumer's floor is heap_size ≥ k, and the smallest listed heap that meets it is the next step up, at most 2.5× too large. A sketch build's parameters must be in its family's list; exact operators have no sketch parameters. | `SketchParams` in `crates/types/src/ir/schema/sketch.rs`; predefined lists are the #1203 target | +| Parameters | A predefined configuration list per family: KLL and KMV `k`; DDSketch `alpha`; HLL `precision`; CMS and CountSketch `width` and `depth`; heap variants also `heap_size`; UnivMon its sizing parameters. The lists are today's sizing over ε = 0.1 down to 0.001; KMV's and UnivMon's distinct-count entries stop at ε = 0.01. A heap variant combines each width × depth with heap sizes from the 1-2-5 series 10, 20, 50, …, 10 000: a top-k consumer's floor is heap_size ≥ k, and the smallest listed heap that meets it is the next step up, at most 2.5× too large. A sketch build's parameters must be in its family's list; exact operators have no sketch parameters. Hydra and the other families Pass 1 does not size have empty `parameter_list`s, so no build of theirs is a valid candidate until their error models exist (#1228). | `SketchParams` in `crates/types/src/ir/schema/sketch.rs`; predefined lists are the #1203 target | | Window layout — Extension | No layout, or tumbling windows with a positive width. Every grid is anchored at phase 0 (the epoch), so the width alone fixes it. Candidate widths come from the gcd closure of the queries' ranges, steps and phases. Sliding merge/subtraction layouts remain future work. Window count is derived from range and width; retention amount is selected later. | `WindowLayout`; `pass2/memo/window_candidates.rs`; #1160 | | Timing | `IngestionTime` or `QueryTime` | `ExecutionTiming` | | Lifetime | `None`: discard after the consuming execution; `OneEvaluation`: retain until consumers within that evaluation finish; `AcrossEvaluations`: retain across evaluations in the specified period. A non-`None` lifetime means materialized. | `Lifetime`; #1200 / #1214 | From f9c8aa46c9d423914c12794f966c0c56fdb4c5fa Mon Sep 17 00:00:00 2001 From: zzylol <50204836+zzylol@users.noreply.github.com> Date: Fri, 6 Nov 2026 15:48:10 +0000 Subject: [PATCH 31/44] docs(concepts): execution contexts reuse the workload's recurrence; one-off queries supported MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit An evaluation context is Evaluation { recurrence, range_evaluation }, deduplicated by equality. delivers_on takes the context: w divides the evaluation grid of FixedIntervalAt (interval, step, phase), every Scheduled time, or a OneTime query's time; never when the times are not known. ρ_κ reuses execution_demand's evaluation counting. EvaluationSchedule is deleted. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01K1M6bq8xVaGKyinf4ohK6q --- .../memo-groups-properties-requirements.md | 48 +++++++++---------- 1 file changed, 22 insertions(+), 26 deletions(-) diff --git a/docs/design_docs/concepts/memo-groups-properties-requirements.md b/docs/design_docs/concepts/memo-groups-properties-requirements.md index d647f4bb1..a30f8348c 100644 --- a/docs/design_docs/concepts/memo-groups-properties-requirements.md +++ b/docs/design_docs/concepts/memo-groups-properties-requirements.md @@ -119,21 +119,24 @@ Physical requirements are derived from the consumer's timing and execution conte QT means QueryTime; IT means IngestionTime. Alignment, data coverage, and retention checks still apply to allowed combinations. **Alignment.** There is no physical schedule property. When a result can be read follows from its lifetime and window layout: -- **None** and **OneEvaluation**: the result is computed inside one evaluation context κ (Section 5) and read there. A OneEvaluation result belongs to one context: queries with the same schedule are evaluated together, and that evaluation's consumers share the result. When it runs is the context's schedule, and Stage 3 chooses it through $x_{\kappa,v}$, just as for ℓ = None. Sharing across differently scheduled queries through a gcd schedule is not offered: it is rarely beneficial and wastes runs. +- **None** and **OneEvaluation**: the result is computed inside one evaluation context κ (Section 5) and read there. A OneEvaluation result belongs to one context: queries with equal recurrence and range evaluation are evaluated together, and that evaluation's consumers share the result. When it runs is the context's schedule, and Stage 3 chooses it through $x_{\kappa,v}$, just as for ℓ = None. Sharing across differently scheduled queries through a gcd schedule is not offered: it is rarely beneficial and wastes runs. - **Windowed AcrossEvaluations**: the input is cut into tumbling windows [k·w, (k+1)·w), with w from the window layout ω and the grid anchored at phase 0 (the epoch). Each window is built when it ends and can be read from then on. A consumer can read a range that ends on a window boundary. With w = 10 s, a `[1m]` query evaluated every 30 s at phase 0 merges the newest 6 windows; one evaluated at phase 15 s cannot, because its range ends inside a window; with that query in the workload, the gcd closure also offers w = 5 s, which both can read. A grid phase is not needed: a phase is absorbed by the width, because width candidates come from the gcd closure of the queries' ranges, steps and phases. For example, phases 0 and 5 s with w = 10 s give w = 5 s. Alignment alone does not ensure sufficient retained history; that is $h_v$ (Section 5). -Let a context κ have interval $I_\kappa$ and phase $\varphi_\kappa$, and let a windowed producer have tumbling width $w$. A consumer running in κ can read the producer when: +An evaluation context κ is described by the workload's own recurrence and range evaluation (Section 5). Let a windowed producer have tumbling width $w$. A consumer running in κ can read the producer when $w$ divides every evaluation time of κ: -| Producer lifetime | Required alignment | +| Producer lifetime, or κ's recurrence for a windowed AcrossEvaluations producer | Required alignment | |---|---| | None | Always: runs in κ | | OneEvaluation | The result belongs to κ | -| AcrossEvaluations, tumbling width $w$ | $w\mid I_\kappa$ and $w\mid\varphi_\kappa$ | +| `FixedIntervalAt { interval, evaluation_phase }`, range step $s$ | $w \mid \gcd(\text{interval}, s, \text{phase})$, the evaluation grid; $s$ counts only for a range evaluation of more than one step | +| `Scheduled(times)` | $w$ divides every time (and $s$) | +| `OneTime { execute_at: Some(t) }` | $w \mid t$ (and $s$) | +| `FixedInterval` without a phase, `OneTime { execute_at: None }`, `EstimatedRate`, `Unknown` | Never: the evaluation times are not known | | AcrossEvaluations, no window layout | Not offered yet: it needs its own delivery rule | -The formulation also permits a single, non-periodic evaluation. The interval/phase interface below covers only periodic consumers. Rejecting non-periodic consumers is a current scope restriction of that proposed interface, not a mathematical impossibility; supporting them needs explicit evaluation-time membership checks. +The evaluation grid is `QueryRecurrence::evaluation_grid_ms(range_evaluation)` in `crates/types/src/workload/mod.rs`, which the workload entries also use. One-off queries are supported: each is a context of its own recurrence, run `invocations` times. ### Invalid combinations @@ -186,7 +189,7 @@ The tree does not decide which producer each consumer input reads ($y$), in whic **The order does not change the candidate set.** The leaves are the product of the dimensions' values minus invalid combinations and pruned values. Each prune rule reads only the value it removes and values decided above it, so any order that keeps each prune rule below the dimensions it reads yields the same leaves. -**The order changes how many nodes enumeration visits:** $\sum_d \prod_{d' \le d} \lvert \text{values kept at } d' \rvert$. Dimensions that prune much go early; dimensions that multiply without pruning go late. The order above follows four rules: +**The order changes how many nodes enumeration visits:** $\sum_d \prod_{d' \le d} \lvert \text{values left at } d' \rvert$. Dimensions that prune much go early; dimensions that multiply without pruning go late. The order above follows four rules: 1. **Logical before physical.** Stage 1 hash-conses groups and computes $\alpha^{\log}$ once, before Stage 2 multiplies each logical candidate by its physical alternatives. 2. **Strong prunes early.** Executor support and the pruning cap act before windows and physical properties multiply the tree. @@ -399,17 +402,13 @@ impl PhysicalProperties { pub fn is_materialized(&self) -> bool; } -/// When a context runs: φ + I·ℤ. -pub struct EvaluationSchedule { - interval_ms: u64, - phase_ms: u64, -} impl PhysicalCandidate { - /// Whether a consumer running on `times` can read this producer - /// (Section 2's alignment table): w | I and w | φ for a windowed - /// AcrossEvaluations producer, with w read from its window layout; - /// true for ℓ = None and OneEvaluation, which run in that context. - pub fn delivers_on(&self, memo: &Memo, times: &EvaluationSchedule) -> bool; + /// Whether a consumer running in `context` can read this producer + /// (Section 2's alignment table): for a windowed AcrossEvaluations + /// producer, w (read from its window layout) divides every evaluation + /// time of the context; true for ℓ = None, and for OneEvaluation in the + /// same context. + pub fn delivers_on(&self, memo: &Memo, context: &ExecutionContext) -> bool; } /// α^phys: timing and lifetime only (Section 2's table). Alignment is /// checked per context in Stage 3 with `delivers_on`. @@ -658,25 +657,22 @@ These extensions go beyond constraints (A)–(I) and the formulation's fixed-cos An AcrossEvaluations candidate has its own computation: each window is built when it ends. A candidate with ℓ = None runs inside an execution that needs it. For example, a finalize candidate used by queries A (every 10 s) and C (every 15 s) may execute in both evaluation contexts even though Stage 2 enumerates it only once. A OneEvaluation candidate also runs inside one evaluation context κ, and that evaluation's consumers share the result. -A **context** identifies a class of executions with a common owner and schedule across the accounting horizon. It is not necessarily one timestamp. There is one evaluation context per evaluation schedule (interval, phase): all queries with that schedule are evaluated together and share its OneEvaluation results. Queries A and B of the running example, both every 10 s at phase 0, form one context. A non-periodic (one-off) query has no evaluation context yet: it is out of scope, as in Section 2. +A **context** identifies a class of executions with a common owner and schedule across the accounting horizon. It is not necessarily one timestamp. An evaluation context reuses the workload's own description of when queries run, their `QueryRecurrence` and `RangeEvaluation`: queries with equal ones are evaluated together and share its OneEvaluation results. Queries A and B of the running example, both every 10 s at phase 0, form one context; a one-off query forms the context of its `OneTime` recurrence. `QueryRecurrence` holds an `f64` (`EstimatedRate`) and cannot be hashed, so contexts are deduplicated by equality and `ExecutionContextId` indexes the list of distinct contexts. ```rust pub struct ExecutionContextId(pub usize); pub enum ExecutionContext { - Evaluation(EvaluationSchedule), + Evaluation { + recurrence: QueryRecurrence, + range_evaluation: Option, + }, IngestionPipeline(RefreshWindowId), Materialized(PhysicalCandidateId), } - -impl ExecutionContext { - // An evaluation context's schedule; None for the other contexts, which - // is not an implicit compatibility pass. - pub fn evaluation_schedule(&self) -> Option; -} ``` -A chain of ℓ = None and OneEvaluation candidates inherits its consuming context. Encountering an AcrossEvaluations producer switches to that producer's own computation context when traversing its inputs. The context identifies where the operation runs; `delivers_on` with the context's schedule decides whether a windowed producer can supply it. +A chain of ℓ = None and OneEvaluation candidates inherits its consuming context. Encountering an AcrossEvaluations producer switches to that producer's own computation context when traversing its inputs. The context identifies where the operation runs; `delivers_on` with the context decides whether a windowed producer can supply it. | Extended variable | Meaning | |---|---| @@ -697,7 +693,7 @@ To avoid confusing per-execution costs with the baseline's horizon-wide cost, us |---|---| | $W_v^{\mathrm{mat}}$ | Fixed computation and access cost over the horizon for an AcrossEvaluations candidate, one build per window, excluding variable retention | | $c_{\kappa,v}$ | Cost of one execution in the specified context of a candidate with ℓ = None or OneEvaluation, including a OneEvaluation result's storage within that evaluation | -| $\rho_\kappa$ | Fixed number or expected number of context executions over the common horizon | +| $\rho_\kappa$ | Fixed number or expected number of context executions over the common horizon, from the existing evaluation counting in `crates/cost/src/execution_demand.rs` (a one-off query's is its `invocations`) | | $h_v$ | Retention amount in a declared unit, such as retained windows | | $\pi_v$ | Horizon-wide retention cost per unit of retained amount | | $\operatorname{need}_{\kappa,v,c,i}$ | Retention amount producer v needs to serve this consumer input in this context: the time horizon the queries served by that context read, e.g. a 5m range over 10 s tumbling windows needs 30 windows. Query latency is ignored for now (decided 2026-11-06) | From 2d136e6637bb2293c32770de431eb3333dc69e5c Mon Sep 17 00:00:00 2001 From: zzylol <50204836+zzylol@users.noreply.github.com> Date: Fri, 6 Nov 2026 18:53:41 +0000 Subject: [PATCH 32/44] docs(concepts): Memo methods return Result; CandidateError find_or_add_group returns the node rules' SchemaDerivationError; add_candidate checks the candidate's expression by the same rules and requires the group's output schema and kind (CandidateError). Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01K1M6bq8xVaGKyinf4ohK6q --- .../memo-groups-properties-requirements.md | 21 +++++++++++++++++-- 1 file changed, 19 insertions(+), 2 deletions(-) diff --git a/docs/design_docs/concepts/memo-groups-properties-requirements.md b/docs/design_docs/concepts/memo-groups-properties-requirements.md index a30f8348c..d6ece4103 100644 --- a/docs/design_docs/concepts/memo-groups-properties-requirements.md +++ b/docs/design_docs/concepts/memo-groups-properties-requirements.md @@ -258,20 +258,37 @@ pub struct CandidateForMemoGroupId { } impl Memo { + /// The group of `expression` (operator, arguments and input groups), + /// added if new. Fails when the expression breaks the node rules + /// (schema derivation, input validation, merge check). pub fn find_or_add_group( &mut self, expression: Operator, - ) -> MemoGroupId; + ) -> Result; + /// Adds `candidate` to `group`, or returns the id of the candidate there + /// with the same expression and logical properties. The candidate's + /// expression is checked by the same node rules as a group's, and its + /// output schema and kind must equal the group's; a rejected candidate + /// leaves the memo unchanged. pub fn add_candidate( &mut self, group: MemoGroupId, candidate: CandidateForMemoGroup, - ) -> CandidateForMemoGroupId; + ) -> Result; pub fn candidate(&self, id: CandidateForMemoGroupId) -> &CandidateForMemoGroup; } +pub enum CandidateError { + /// The candidate's expression fails the node rules (schema derivation, + /// input validation, merge check). + Schema(SchemaDerivationError), + /// Its output schema or output kind differs from the group's: it does + /// not compute the group's result. + OutputDiffersFromGroup { group: MemoGroupId }, +} + pub struct CandidateExpression { pub operator: Operator, } From a77c8ad647ded5c5babb82198b67c72f50015816 Mon Sep 17 00:00:00 2001 From: zzylol <50204836+zzylol@users.noreply.github.com> Date: Fri, 6 Nov 2026 19:17:19 +0000 Subject: [PATCH 33/44] docs(concepts): a memo group is its expression; private Memo groups MemoGroup holds expression, schema and result kind (no logical node or child list); Memo's groups are private behind groups(), added only through find_or_add_group (hash-consing on the expression), and Stage 0 adds each query with add_query. Appendix A reads memo.groups(). Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01K1M6bq8xVaGKyinf4ohK6q --- .../memo-groups-properties-requirements.md | 30 ++++++++++++++----- 1 file changed, 23 insertions(+), 7 deletions(-) diff --git a/docs/design_docs/concepts/memo-groups-properties-requirements.md b/docs/design_docs/concepts/memo-groups-properties-requirements.md index d6ece4103..f68a03a07 100644 --- a/docs/design_docs/concepts/memo-groups-properties-requirements.md +++ b/docs/design_docs/concepts/memo-groups-properties-requirements.md @@ -215,14 +215,18 @@ Rules add groups through `find_or_add_group` and candidates through `add_candida ### Stage 0: groups -`hash_consed` in `crates/logical-asap-optimization/src/stage0.rs` identifies shared logical expressions and query roots. Stage 0 and Stage 1 share one type, `Memo`: Stage 0 returns it with every group's candidate list empty, and Stage 1 fills the lists. +`memo_of_queries` in `crates/stage0-memo` builds the groups bottom-up from the queries' nodes: each node's operator, over its children's groups, goes through `Memo::find_or_add_group`, so identical sub-expressions are one group; each query's root is then added with `Memo::add_query`. Stage 0 and Stage 1 share one type, `Memo`: Stage 0 returns it with every group's candidate list empty, and Stage 1 fills the lists. ```rust pub struct MemoGroupId(pub u32); pub struct MemoGroup { - pub logical_node: Rc, - pub child_groups: Vec, + /// What it computes; its children are the input groups. + pub expression: Operator, + /// Derived once, when the group is added. + pub schema: Schema, + /// Derived once, when the group is added. + pub result_kind: OperatorResultKind, pub parents: Vec, pub queries: Vec, // Empty after Stage 0; populated by Stage 1. @@ -230,14 +234,26 @@ pub struct MemoGroup { } pub struct Memo { - pub groups: Vec, + // Private: groups are added only through `find_or_add_group`, so the + // hash-consing index covers every group. + groups: Vec, pub query_roots: Vec>, + // Private: the groups `find_or_add_group` may return, by expression hash. + groups_by_expression_hash: HashMap>, +} + +impl Memo { + pub fn groups(&self) -> &[MemoGroup]; + /// Adds a query with this root; its groups and those below list it. + pub fn add_query(&mut self, root: QueryRoot) -> usize; } ``` -`OperatorNode` recursively contains chosen child nodes and therefore describes a fixed plan fragment. `Operator` contains input group IDs and leaves their implementations open. A candidate needs its own expression because a group may offer several ways to compute the same result; it cannot reuse the group's representative `logical_node` as every candidate's expression. +A group is its expression: one operator, with its arguments, over input groups. Hash-consing compares and hashes the expression directly, so two expressions are one group when their operators and arguments are equal and their input groups are the same. θ is not an operator argument, so builds that differ only in θ share one group. An expression whose copies may return different rows (an unordered `LIMIT`) always gets a new group. + +The group's schema and result kind come from the node rules (schema derivation, input validation, merge check) applied to its expression over its input groups. A candidate's expression is checked by the same rules and must produce the group's schema and result kind. `OperatorNode` recursively contains chosen child nodes and therefore describes a fixed plan fragment; `Operator` contains input group IDs and leaves their implementations open. A candidate needs its own expression because a group may offer several ways to compute the same result. -Two simplifications are possible but not adopted: storing the group as one `Operator` instead of its node and child list, and removing the one-field `CandidateExpression` wrapper. +Removing the one-field `CandidateExpression` wrapper is possible but not adopted. ### Stage 1, Pass 1: initial candidates @@ -815,7 +831,7 @@ fn candidates_of( memo: &Memo, group: MemoGroupId, ) -> impl Iterator + '_ { - (0..memo.groups[group.0 as usize].candidates.len()) + (0..memo.groups()[group.0 as usize].candidates.len()) .map(move |index_in_group| CandidateForMemoGroupId { group, index_in_group }) } From cca5eeeabd8a0ea69dd9504e86b12d1cb52d1b96 Mon Sep 17 00:00:00 2001 From: zzylol <50204836+zzylol@users.noreply.github.com> Date: Fri, 6 Nov 2026 19:40:01 +0000 Subject: [PATCH 34/44] docs(concepts): GroupProperties derived from input groups; SummaryDefinition over group ids A memo group is its expression and its GroupProperties (schema, result kind, summary coverage, shareable_when_equal), derived once from its input groups' expressions and properties by the node rules, never from node trees. A group's summary coverage has a SummaryDefinition: the residual links over group ids. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01K1M6bq8xVaGKyinf4ohK6q --- .../memo-groups-properties-requirements.md | 41 ++++++++++++++++--- 1 file changed, 35 insertions(+), 6 deletions(-) diff --git a/docs/design_docs/concepts/memo-groups-properties-requirements.md b/docs/design_docs/concepts/memo-groups-properties-requirements.md index f68a03a07..4fea1d307 100644 --- a/docs/design_docs/concepts/memo-groups-properties-requirements.md +++ b/docs/design_docs/concepts/memo-groups-properties-requirements.md @@ -221,18 +221,38 @@ Rules add groups through `find_or_add_group` and candidates through `add_candida pub struct MemoGroupId(pub u32); pub struct MemoGroup { - /// What it computes; its children are the input groups. + /// What it computes: one operator with its arguments. Each `MemoGroupId` + /// in it is an input group, never this group's own id. pub expression: Operator, - /// Derived once, when the group is added. - pub schema: Schema, - /// Derived once, when the group is added. - pub result_kind: OperatorResultKind, + /// Derived once when the group is created, from its input groups' + /// `expression` and `group_properties` (optd/TiDB-style); never changes. + pub group_properties: GroupProperties, pub parents: Vec, pub queries: Vec, // Empty after Stage 0; populated by Stage 1. pub candidates: Vec, } +pub struct GroupProperties { + pub schema: Schema, + pub result_kind: OperatorResultKind, + /// What a summary-state group covers; `None` for any other group. + pub summary_coverage: Option>, + /// Whether two equal expressions return the same rows for the same + /// input rows, so they may be one group (not an unordered `LIMIT`). + pub shareable_when_equal: bool, +} + +/// The `SummaryAgg` with its lifted selection removed, written over group +/// ids: the residual links from `reads` up to the `SummaryAgg`. +pub struct SummaryDefinition { + /// The group below the residual links; `None` when the bottom link is + /// the `Scan`, rebuilt without the lifted predicates. + pub reads: Option, + /// Bottom-up, each link without its input; the last is the `SummaryAgg`. + pub links: Vec>, +} + pub struct Memo { // Private: groups are added only through `find_or_add_group`, so the // hash-consing index covers every group. @@ -249,9 +269,18 @@ impl Memo { } ``` +`SummaryCoverage` is the coverage type of today's IR with its definition as a parameter: a group's coverage has a `SummaryDefinition`, an operator node's (executor, export) its residual node tree. + A group is its expression: one operator, with its arguments, over input groups. Hash-consing compares and hashes the expression directly, so two expressions are one group when their operators and arguments are equal and their input groups are the same. θ is not an operator argument, so builds that differ only in θ share one group. An expression whose copies may return different rows (an unordered `LIMIT`) always gets a new group. -The group's schema and result kind come from the node rules (schema derivation, input validation, merge check) applied to its expression over its input groups. A candidate's expression is checked by the same rules and must produce the group's schema and result kind. `OperatorNode` recursively contains chosen child nodes and therefore describes a fixed plan fragment; `Operator` contains input group IDs and leaves their implementations open. A candidate needs its own expression because a group may offer several ways to compute the same result. +Everything that was computed on operator-node trees is computed on groups. `GroupProperties` is derived once, when the group is created, by the same rules that check operator nodes (schema derivation, input validation, merge check), reading only the input groups' `expression` and `group_properties` from the memo, never a node tree: + +- `schema` and `result_kind` from the operator and its input groups' schemas and result kinds; a scalar subquery reads the referenced group's schema. +- `summary_coverage`, for a summary state: walking down through the input groups' expressions (Filter, Project, TimeRange, TimeShift, down to the Scan), lifting their predicates into the selection; the rest is the `SummaryDefinition`. A merge's coverage comes from its inputs' coverage; a merge is valid only over inputs with the same definition and disjoint selections. +- The checks that read an input's operator (a merge of the newest windows reads a window chain; a selected build) read the input group's `expression`. +- `shareable_when_equal`: a `LIMIT` is shareable over a `Sort` with the same partitions (the input group's `expression`); a `Dedup` on every column (the input group's schema). + +A candidate's expression goes through the same derivation and must produce the group's schema and result kind. The executor and the export keep their node-tree summary coverage (`definition_id`) until Stage 3's export builds node trees from the memo. `OperatorNode` recursively contains chosen child nodes and therefore describes a fixed plan fragment; `Operator` contains input group IDs and leaves their implementations open. A candidate needs its own expression because a group may offer several ways to compute the same result. Removing the one-field `CandidateExpression` wrapper is possible but not adopted. From cf095348d08077dd087ed5686122fd8b8a774a9e Mon Sep 17 00:00:00 2001 From: zzylol <50204836+zzylol@users.noreply.github.com> Date: Fri, 6 Nov 2026 21:09:42 +0000 Subject: [PATCH 35/44] docs(concepts): SummaryStateDefinition, OperatorInput and MemoGroupAsInput; coverage only in the memo Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01K1M6bq8xVaGKyinf4ohK6q --- .../memo-groups-properties-requirements.md | 73 +++++++++++++++---- 1 file changed, 60 insertions(+), 13 deletions(-) diff --git a/docs/design_docs/concepts/memo-groups-properties-requirements.md b/docs/design_docs/concepts/memo-groups-properties-requirements.md index 4fea1d307..ebdabda09 100644 --- a/docs/design_docs/concepts/memo-groups-properties-requirements.md +++ b/docs/design_docs/concepts/memo-groups-properties-requirements.md @@ -237,20 +237,38 @@ pub struct GroupProperties { pub schema: Schema, pub result_kind: OperatorResultKind, /// What a summary-state group covers; `None` for any other group. - pub summary_coverage: Option>, + pub summary_coverage: Option, /// Whether two equal expressions return the same rows for the same /// input rows, so they may be one group (not an unordered `LIMIT`). pub shareable_when_equal: bool, } -/// The `SummaryAgg` with its lifted selection removed, written over group -/// ids: the residual links from `reads` up to the `SummaryAgg`. -pub struct SummaryDefinition { - /// The group below the residual links; `None` when the bottom link is - /// the `Scan`, rebuilt without the lifted predicates. - pub reads: Option, - /// Bottom-up, each link without its input; the last is the `SummaryAgg`. - pub links: Vec>, +/// What a summary state computes, with every lifted selection removed. +pub struct SummaryStateDefinition { + /// The group the operators below read: the first input under the + /// SummaryAgg that is not Filter / range TimeRange / TimeShift without @ + /// / direct-column Project. + pub input_group: MemoGroupId, + /// From the SummaryAgg down, each operator not lifted into the + /// selection, with only its residual arguments; `()` marks its single + /// input (the next one, or input_group). Two states may merge only if + /// input_group and these operators are equal. + pub operators_kept_in_definition: Vec>, +} + +pub struct SummaryCoverage { + pub(crate) definition: SummaryStateDefinition, + /// Union of boxes over the output rows of the operators' input. + pub(crate) selection: Vec, +} + +impl SummaryCoverage { + pub(crate) fn derive_from_inputs( + operator: &Operator>, + ) -> Result; + /// A hash of the `SummaryStateDefinition`; equal definitions in one + /// memo get equal ids. + pub(crate) fn definition_id(&self) -> String; } pub struct Memo { @@ -269,18 +287,47 @@ impl Memo { } ``` -`SummaryCoverage` is the coverage type of today's IR with its definition as a parameter: a group's coverage has a `SummaryDefinition`, an operator node's (executor, export) its residual node tree. +The rules read an operator's inputs through one trait, so the same rules derive a front-end node's schema and a group's `GroupProperties`: + +```rust +pub(crate) trait OperatorInput: Sized { + fn schema(&self) -> &Schema; + fn result_kind(&self) -> OperatorResultKind; + /// The input's own operator, its children as inputs again. + fn operator(&self) -> Operator; + fn summary_coverage(&self) -> Option<&SummaryCoverage>; +} +// Front-end node trees; summary_coverage is None. +impl OperatorInput for Rc { … } +/// A memo group read as an input: its group_properties and expression. +pub(crate) struct MemoGroupAsInput<'m> { memo: &'m Memo, group_id: MemoGroupId } +impl OperatorInput for MemoGroupAsInput<'_> { … } + +// The existing rules, generic over the input; existing names kept. +impl Operator { + pub(crate) fn output_schema(&self) -> Result; + pub(crate) fn output_kind(&self) -> OperatorResultKind; + pub(crate) fn validate_inputs(&self) -> Result<(), SchemaDerivationError>; + pub(crate) fn is_shareable_when_equal(&self) -> bool; +} +fn derive_group_properties( + memo: &Memo, + expression: &Operator, +) -> Result; +``` + +Summary coverage exists only in the memo: a node tree has none, and `SummaryCoverage` has no type parameter. The executor reads only `definition_id` from the plan. A group is its expression: one operator, with its arguments, over input groups. Hash-consing compares and hashes the expression directly, so two expressions are one group when their operators and arguments are equal and their input groups are the same. θ is not an operator argument, so builds that differ only in θ share one group. An expression whose copies may return different rows (an unordered `LIMIT`) always gets a new group. -Everything that was computed on operator-node trees is computed on groups. `GroupProperties` is derived once, when the group is created, by the same rules that check operator nodes (schema derivation, input validation, merge check), reading only the input groups' `expression` and `group_properties` from the memo, never a node tree: +Everything that was computed on operator-node trees is computed on groups. `find_or_add_group` derives `GroupProperties` once, when the group is created, with `derive_group_properties`: the rules that check operator nodes (schema derivation, input validation, merge check), applied to the expression over `MemoGroupAsInput`s, which read only the input groups' `expression` and `group_properties`, never a node tree: - `schema` and `result_kind` from the operator and its input groups' schemas and result kinds; a scalar subquery reads the referenced group's schema. -- `summary_coverage`, for a summary state: walking down through the input groups' expressions (Filter, Project, TimeRange, TimeShift, down to the Scan), lifting their predicates into the selection; the rest is the `SummaryDefinition`. A merge's coverage comes from its inputs' coverage; a merge is valid only over inputs with the same definition and disjoint selections. +- `summary_coverage`, for a summary state: walking down from the SummaryAgg through the input groups' expressions while each is a Filter, a range TimeRange, a TimeShift without `@` or a direct-column Project, lifting their predicates into the selection. The group below them is the definition's `input_group`; a Scan is never lifted through, so its predicates stay in that group. The operators not lifted, each with its residual arguments, are `operators_kept_in_definition`. A tumbling-window chain covers what one build over its whole window does. A merge's coverage comes from its inputs' stored coverage; a merge is valid only over inputs with equal definitions and disjoint selections. - The checks that read an input's operator (a merge of the newest windows reads a window chain; a selected build) read the input group's `expression`. - `shareable_when_equal`: a `LIMIT` is shareable over a `Sort` with the same partitions (the input group's `expression`); a `Dedup` on every column (the input group's schema). -A candidate's expression goes through the same derivation and must produce the group's schema and result kind. The executor and the export keep their node-tree summary coverage (`definition_id`) until Stage 3's export builds node trees from the memo. `OperatorNode` recursively contains chosen child nodes and therefore describes a fixed plan fragment; `Operator` contains input group IDs and leaves their implementations open. A candidate needs its own expression because a group may offer several ways to compute the same result. +`add_candidate` derives a candidate's properties with the same `derive_group_properties` and requires its schema and result kind to equal the group's. `OperatorNode` recursively contains chosen child nodes and therefore describes a fixed plan fragment; `Operator` contains input group IDs and leaves their implementations open. A candidate needs its own expression because a group may offer several ways to compute the same result. Removing the one-field `CandidateExpression` wrapper is possible but not adopted. From a38adb82ede7687652212bab33e1c5c2ae2de800 Mon Sep 17 00:00:00 2001 From: zzylol <50204836+zzylol@users.noreply.github.com> Date: Fri, 6 Nov 2026 21:17:35 +0000 Subject: [PATCH 36/44] docs(concepts): unlifted_operators; point to the derivation examples Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01K1M6bq8xVaGKyinf4ohK6q --- .../memo-groups-properties-requirements.md | 14 ++++++++------ 1 file changed, 8 insertions(+), 6 deletions(-) diff --git a/docs/design_docs/concepts/memo-groups-properties-requirements.md b/docs/design_docs/concepts/memo-groups-properties-requirements.md index ebdabda09..4424f2a2c 100644 --- a/docs/design_docs/concepts/memo-groups-properties-requirements.md +++ b/docs/design_docs/concepts/memo-groups-properties-requirements.md @@ -249,11 +249,11 @@ pub struct SummaryStateDefinition { /// SummaryAgg that is not Filter / range TimeRange / TimeShift without @ /// / direct-column Project. pub input_group: MemoGroupId, - /// From the SummaryAgg down, each operator not lifted into the - /// selection, with only its residual arguments; `()` marks its single - /// input (the next one, or input_group). Two states may merge only if - /// input_group and these operators are equal. - pub operators_kept_in_definition: Vec>, + /// The operators not lifted into the selection, listed top down from + /// the SummaryAgg, each with only its residual arguments; `()` marks its + /// single input (the next one, or input_group). Two states may merge only + /// if input_group and these operators are equal. + pub unlifted_operators: Vec>, } pub struct SummaryCoverage { @@ -287,6 +287,8 @@ impl Memo { } ``` +How the definition and selection are derived, with examples: `docs/design_docs/proposals/asap-primitive-schema.md` §4.2.2. + The rules read an operator's inputs through one trait, so the same rules derive a front-end node's schema and a group's `GroupProperties`: ```rust @@ -323,7 +325,7 @@ A group is its expression: one operator, with its arguments, over input groups. Everything that was computed on operator-node trees is computed on groups. `find_or_add_group` derives `GroupProperties` once, when the group is created, with `derive_group_properties`: the rules that check operator nodes (schema derivation, input validation, merge check), applied to the expression over `MemoGroupAsInput`s, which read only the input groups' `expression` and `group_properties`, never a node tree: - `schema` and `result_kind` from the operator and its input groups' schemas and result kinds; a scalar subquery reads the referenced group's schema. -- `summary_coverage`, for a summary state: walking down from the SummaryAgg through the input groups' expressions while each is a Filter, a range TimeRange, a TimeShift without `@` or a direct-column Project, lifting their predicates into the selection. The group below them is the definition's `input_group`; a Scan is never lifted through, so its predicates stay in that group. The operators not lifted, each with its residual arguments, are `operators_kept_in_definition`. A tumbling-window chain covers what one build over its whole window does. A merge's coverage comes from its inputs' stored coverage; a merge is valid only over inputs with equal definitions and disjoint selections. +- `summary_coverage`, for a summary state: walking down from the SummaryAgg through the input groups' expressions while each is a Filter, a range TimeRange, a TimeShift without `@` or a direct-column Project, lifting their predicates into the selection. The group below them is the definition's `input_group`; a Scan is never lifted through, so its predicates stay in that group. The operators not lifted, each with its residual arguments, are `unlifted_operators`. A tumbling-window chain covers what one build over its whole window does. A merge's coverage comes from its inputs' stored coverage; a merge is valid only over inputs with equal definitions and disjoint selections. - The checks that read an input's operator (a merge of the newest windows reads a window chain; a selected build) read the input group's `expression`. - `shareable_when_equal`: a `LIMIT` is shareable over a `Sort` with the same partitions (the input group's `expression`); a `Dedup` on every column (the input group's schema). From c23fe4e3653092788f42f0bef7b2d4266cd8595c Mon Sep 17 00:00:00 2001 From: zzylol <50204836+zzylol@users.noreply.github.com> Date: Fri, 6 Nov 2026 22:46:04 +0000 Subject: [PATCH 37/44] docs(concepts): unlifted_operators name the groups their subqueries read Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01K1M6bq8xVaGKyinf4ohK6q --- .../concepts/memo-groups-properties-requirements.md | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/docs/design_docs/concepts/memo-groups-properties-requirements.md b/docs/design_docs/concepts/memo-groups-properties-requirements.md index 4424f2a2c..4bf19d370 100644 --- a/docs/design_docs/concepts/memo-groups-properties-requirements.md +++ b/docs/design_docs/concepts/memo-groups-properties-requirements.md @@ -250,10 +250,12 @@ pub struct SummaryStateDefinition { /// / direct-column Project. pub input_group: MemoGroupId, /// The operators not lifted into the selection, listed top down from - /// the SummaryAgg, each with only its residual arguments; `()` marks its - /// single input (the next one, or input_group). Two states may merge only - /// if input_group and these operators are equal. - pub unlifted_operators: Vec>, + /// the SummaryAgg, each with only its residual arguments. `None` marks + /// an operator's main input (the next unlifted operator, or input_group); + /// `Some(g)` is another input group it reads, such as the group a scalar + /// subquery in a Filter reads. Two states may merge only if input_group + /// and these operators are equal. + pub unlifted_operators: Vec>>, } pub struct SummaryCoverage { From 4fef7a7858598274600d395aaf0461e375d6436a Mon Sep 17 00:00:00 2001 From: zzylol <50204836+zzylol@users.noreply.github.com> Date: Fri, 6 Nov 2026 22:58:47 +0000 Subject: [PATCH 38/44] docs(concepts): column lineage replaces table qualifiers, with memo-group examples Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01K1M6bq8xVaGKyinf4ohK6q --- .../memo-groups-properties-requirements.md | 75 +++++++++++++++++++ 1 file changed, 75 insertions(+) diff --git a/docs/design_docs/concepts/memo-groups-properties-requirements.md b/docs/design_docs/concepts/memo-groups-properties-requirements.md index 4bf19d370..f6d0eb9a5 100644 --- a/docs/design_docs/concepts/memo-groups-properties-requirements.md +++ b/docs/design_docs/concepts/memo-groups-properties-requirements.md @@ -335,6 +335,81 @@ Everything that was computed on operator-node trees is computed on groups. `find Removing the one-field `CandidateExpression` wrapper is possible but not adopted. +#### Column lineage + +Every output column of a group has a **lineage**: the source column its values are copied from, or none when the column is computed. It is a group property, derived once from the input groups' lineage (as optd's column-reference property is), and it replaces the table qualifier that schema fields carried. A schema field keeps only its name, type and nullability; a front end resolves a qualified name such as `f.src` with its own binding scope while it lowers the query, and the IR never needs the qualifier again. + +```rust +pub struct GroupProperties { + pub schema: Schema, // fields: name, type, nullability; no table qualifier + pub result_kind: OperatorResultKind, + /// One entry per output column, in schema order. + pub column_lineage: Vec, + pub summary_coverage: Option, + pub shareable_when_equal: bool, +} + +/// Where an output column's values come from. +pub enum ColumnLineage { + /// Copied unchanged from a column of a source (a Scan's table or metric). + SourceColumn { source: String, column: String }, + /// Computed by an operator: a measure, an expression, a summary state or readout. + Computed, +} +``` + +**Derivation, per operator** (from the input groups' `column_lineage`): + +| Operator | Output column lineage | +|---|---| +| `Scan` | each column: `SourceColumn { source, column }` | +| `Filter`, `TimeRange`, `TimeShift`, `Sort`, `Limit`, `Dedup` | the input's, unchanged | +| `Project` | a column item, renamed or not: that input column's lineage; an expression item: `Computed` | +| `Aggregate`, `SummaryAgg`, readouts | group-by keys: the input's lineage of that key; measures, states and values: `Computed` | +| `Join` | the left input's columns, then the right input's | +| `SetOp`, `Concat` | per column: the inputs' lineage if they all agree, else `Computed` | +| `BinaryOp` (vector match) | label columns: the matched side's lineage; the value: `Computed` | + +**What reads it:** +- **Summary coverage.** A selection constrains columns identified by lineage (`ColumnIdentity` is a `SourceColumn`), not by (qualifier, name). A predicate on a `Computed` column is not lifted; it stays in `unlifted_operators`. A merge's group keys must have equal lineage. +- **Candidate checks.** `add_candidate` compares a candidate's schema with its group's exactly: names, types, nullability, order. There is no qualifier to compare, so a readout renamed to the group's column name matches. + +**Example 1: a SQL aggregate and its summary candidate.** + +```sql +SELECT f.src, approx_distinct(f.dst) FROM flows f WHERE f.proto = 'tcp' GROUP BY f.src +``` + +| Group | Expression | Output columns | `column_lineage` | +|---|---|---|---| +| G1 | `Scan flows` | `src, dst, proto, ts` | `flows.src, flows.dst, flows.proto, flows.ts` | +| G2 | `Filter(proto = 'tcp')(G1)` | `src, dst, proto, ts` | same as G1 | +| G3 | `Aggregate(by src; approx_distinct(dst))(G2)` | `src, approx_distinct(dst)` | `flows.src, Computed` | +| G4 | `SummaryAgg[HLL](by src; dst)(G2)` | `src, state` | `flows.src, Computed` | +| G5 | `SummaryEstimate[cardinality](G4)` | `src, cardinality` | `flows.src, Computed` | +| — | candidate of G3: `Project(src, cardinality AS approx_distinct(dst))(G5)` | `src, approx_distinct(dst)` | `flows.src, Computed` | + +The candidate's schema equals G3's, so `add_candidate` accepts it. Before lineage, G3's key carried the qualifier `flows` and the renaming `Project` could not keep it on one column while renaming another, so this candidate was rejected. + +**Example 2: renames keep lineage.** + +```sql +SELECT source, count(*) FROM (SELECT f.src AS source FROM flows f) GROUP BY source +``` + +The inner `Project` renames `src` to `source`; its lineage stays `flows.src`. Two summary states, one grouped by `source` here and one grouped by `src` in another query, have keys of equal lineage, so a merge of the two may pass the group-key check. Compared by (qualifier, name), `source` and `flows.src` looked like different columns. + +**Example 3: the same name from two sources.** + +```sql +SELECT s.region, count(*) FROM sales s JOIN users u ON s.user = u.id GROUP BY s.region +``` + +After the join, `region` may exist on both sides. The front end resolves `s.region` to the left input's column while lowering; in the IR the two columns have lineage `sales.region` and `users.region`, so a selection on `sales.region` is never confused with one on `users.region`. + +**Example 4: PromQL label matchers.** `quantile_over_time(0.9, latency{job="api"}[1m])` and the same with `job="web"`. Each Scan's `job` column has lineage `latency.job`. The lifted matchers become selections `latency.job ∈ {api}` and `latency.job ∈ {web}`: disjoint selections on the same column, so the two KLL states may merge. + + ### Stage 1, Pass 1: initial candidates Implementation locations: `crates/logical-asap-optimization/src/pass1/per_computation_candidates.rs` and `memo_population.rs`. The #1203 target gives every intermediate result its own group; candidates contain one operator rather than private operator chains. From 48e410da3622d7dde1899330e2dc996446613da3 Mon Sep 17 00:00:00 2001 From: zzylol <50204836+zzylol@users.noreply.github.com> Date: Fri, 6 Nov 2026 23:13:23 +0000 Subject: [PATCH 39/44] docs(concepts): input_group is optional when Scan predicates lift; define fused groups Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01K1M6bq8xVaGKyinf4ohK6q --- .../memo-groups-properties-requirements.md | 21 +++++++++++++++---- 1 file changed, 17 insertions(+), 4 deletions(-) diff --git a/docs/design_docs/concepts/memo-groups-properties-requirements.md b/docs/design_docs/concepts/memo-groups-properties-requirements.md index f6d0eb9a5..531ba465b 100644 --- a/docs/design_docs/concepts/memo-groups-properties-requirements.md +++ b/docs/design_docs/concepts/memo-groups-properties-requirements.md @@ -247,8 +247,10 @@ pub struct GroupProperties { pub struct SummaryStateDefinition { /// The group the operators below read: the first input under the /// SummaryAgg that is not Filter / range TimeRange / TimeShift without @ - /// / direct-column Project. - pub input_group: MemoGroupId, + /// / direct-column Project. `None` only when a Scan's predicates are + /// lifted; the Scan, with its residual predicates, is then the last + /// unlifted operator. + pub input_group: Option, /// The operators not lifted into the selection, listed top down from /// the SummaryAgg, each with only its residual arguments. `None` marks /// an operator's main input (the next unlifted operator, or input_group); @@ -327,7 +329,7 @@ A group is its expression: one operator, with its arguments, over input groups. Everything that was computed on operator-node trees is computed on groups. `find_or_add_group` derives `GroupProperties` once, when the group is created, with `derive_group_properties`: the rules that check operator nodes (schema derivation, input validation, merge check), applied to the expression over `MemoGroupAsInput`s, which read only the input groups' `expression` and `group_properties`, never a node tree: - `schema` and `result_kind` from the operator and its input groups' schemas and result kinds; a scalar subquery reads the referenced group's schema. -- `summary_coverage`, for a summary state: walking down from the SummaryAgg through the input groups' expressions while each is a Filter, a range TimeRange, a TimeShift without `@` or a direct-column Project, lifting their predicates into the selection. The group below them is the definition's `input_group`; a Scan is never lifted through, so its predicates stay in that group. The operators not lifted, each with its residual arguments, are `unlifted_operators`. A tumbling-window chain covers what one build over its whole window does. A merge's coverage comes from its inputs' stored coverage; a merge is valid only over inputs with equal definitions and disjoint selections. +- `summary_coverage`, for a summary state: walking down from the SummaryAgg through the input groups' expressions while each is a Filter, a range TimeRange, a TimeShift without `@` or a direct-column Project, lifting their predicates into the selection. The group below them is the definition's `input_group`. A Scan's predicates (PromQL label matchers) are lifted too; the Scan with its residual predicates is then the last unlifted operator and `input_group` is `None`. The operators not lifted, each with its residual arguments, are `unlifted_operators`. A tumbling-window chain covers what one build over its whole window does. A merge's coverage comes from its inputs' stored coverage; a merge is valid only over inputs with equal definitions and disjoint selections. - The checks that read an input's operator (a merge of the newest windows reads a window chain; a selected build) read the input group's `expression`. - `shareable_when_equal`: a `LIMIT` is shareable over a `Sort` with the same partitions (the input group's `expression`); a `Dedup` on every column (the input group's schema). @@ -419,7 +421,7 @@ Implementation locations: `crates/logical-asap-optimization/src/pass1/per_comput | Exact | Exact candidate in the result group | | Exact accumulator | Accumulator-state group plus a finalize candidate in the result group | | Summary | One state group per supported family; one build candidate per listed parameter configuration; separate readout candidates | -| Fused build (#1203 2c) | Candidate with `fused_groups` identifying additional results; selection must enforce their availability consistently | +| Fused build (#1203 2c) | Several operators of one sub-DAG pattern, in one appearance of the sub-expression, fused into one candidate operator; `fused_groups` lists the groups of the operators it replaces | ```rust pub struct CandidateForMemoGroupId { @@ -468,6 +470,10 @@ pub struct CandidateForMemoGroup { pub expression: CandidateExpression, pub logical_properties: LogicalProperties, pub input_requirements: Vec, + /// The groups of the operators this candidate fuses: a sub-DAG pattern + /// of several operators, in one appearance of the sub-expression, + /// computed by this one operator. Empty for a candidate that fuses + /// nothing. pub fused_groups: Vec, // Diagnostic provenance; not a planning decision. pub origin: OriginOfCandidateForMemoGroup, @@ -517,6 +523,13 @@ pub fn producer_satisfies_logical_requirements( The floor representation has the applicability condition stated in Section 2. Creating requirements also needs the input family's configuration domain; a constructor without that context can validate only the information encoded in its argument. Error-model interfaces are grouped with accuracy in Section 4. +**Fused groups.** A fused candidate computes, as one operator, a sub-DAG pattern of several operators in one appearance of the sub-expression; `fused_groups` lists the groups of the operators it replaces. Reusing a common sub-expression across appearances is Pass 2's reuse, not fusion. Fusion rules: +- top-k (a sort and limit over an aggregate, as one heap summary); +- sum over sum, min over min, max over max (distributive aggregates, as one aggregate); +- possibly Hydra (a summary over every sub-population at once). + +TODO: revisit each fused candidate's inputs and outputs to check that the fusing rules are correct. + Migration targets: remove private `root_node` chains, duplicate `input_groups`, and `accuracy_budget_parts`. The proposed predefined parameter lists replace sizing one candidate per budget partition. ### Stage 1, Pass 2: reuse rules and logical edges From d62b58c14ebf9eade4190fed51a389350f6ece1b Mon Sep 17 00:00:00 2001 From: zzylol <50204836+zzylol@users.noreply.github.com> Date: Sat, 7 Nov 2026 03:44:14 +0000 Subject: [PATCH 40/44] docs(concepts): SourceColumn is one struct shared by lineage and summary coverage Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01K1M6bq8xVaGKyinf4ohK6q --- .../memo-groups-properties-requirements.md | 20 +++++++++++++++---- 1 file changed, 16 insertions(+), 4 deletions(-) diff --git a/docs/design_docs/concepts/memo-groups-properties-requirements.md b/docs/design_docs/concepts/memo-groups-properties-requirements.md index 531ba465b..ac587bcef 100644 --- a/docs/design_docs/concepts/memo-groups-properties-requirements.md +++ b/docs/design_docs/concepts/memo-groups-properties-requirements.md @@ -351,20 +351,32 @@ pub struct GroupProperties { pub shareable_when_equal: bool, } +/// A column of a source (a Scan's table or metric), by name. +pub struct SourceColumn { + pub source: String, + pub column: String, +} + /// Where an output column's values come from. pub enum ColumnLineage { - /// Copied unchanged from a column of a source (a Scan's table or metric). - SourceColumn { source: String, column: String }, + /// Copied unchanged from a column of a source. + SourceColumn(SourceColumn), /// Computed by an operator: a measure, an expression, a summary state or readout. Computed, } + +/// Summary coverage keys its selections by the same type. +pub struct SelectionBox { + pub columns: BTreeMap, + pub relative_time: Option<(Bound, Bound)>, +} ``` **Derivation, per operator** (from the input groups' `column_lineage`): | Operator | Output column lineage | |---|---| -| `Scan` | each column: `SourceColumn { source, column }` | +| `Scan` | each column: `SourceColumn(SourceColumn { source, column })` | | `Filter`, `TimeRange`, `TimeShift`, `Sort`, `Limit`, `Dedup` | the input's, unchanged | | `Project` | a column item, renamed or not: that input column's lineage; an expression item: `Computed` | | `Aggregate`, `SummaryAgg`, readouts | group-by keys: the input's lineage of that key; measures, states and values: `Computed` | @@ -373,7 +385,7 @@ pub enum ColumnLineage { | `BinaryOp` (vector match) | label columns: the matched side's lineage; the value: `Computed` | **What reads it:** -- **Summary coverage.** A selection constrains columns identified by lineage (`ColumnIdentity` is a `SourceColumn`), not by (qualifier, name). A predicate on a `Computed` column is not lifted; it stays in `unlifted_operators`. A merge's group keys must have equal lineage. +- **Summary coverage.** A selection constrains columns keyed by their `SourceColumn` lineage, not by (qualifier, name). A predicate on a `Computed` column is not lifted; it stays in `unlifted_operators`. A merge's group keys must have equal lineage. - **Candidate checks.** `add_candidate` compares a candidate's schema with its group's exactly: names, types, nullability, order. There is no qualifier to compare, so a readout renamed to the group's column name matches. **Example 1: a SQL aggregate and its summary candidate.** From 39967773ed6912f728d3e50540dacf15d7c612f2 Mon Sep 17 00:00:00 2001 From: zzylol <50204836+zzylol@users.noreply.github.com> Date: Sat, 7 Nov 2026 16:46:57 +0000 Subject: [PATCH 41/44] docs(concepts): chain groups, general rollup rule, dead-candidate elimination, TimeBound coverage Logical properties are parameters only; tumbling windows are range-free chain groups read through TimeRange/TimeShift and merged by SummaryMerge. Pass 2 order: rollup from the finest common group, top-k reuse, reach and floors (set_input_requirements), dead-candidate elimination with root-cause traces, chain groups, logical edges. SelectionBox keeps only columns, with TimeBound intervals on the timestamp column; aggregate_coverage covers exact Aggregate groups too. evaluation_grid_ms is None without a phase. error_propagation takes a metric (b = 0.1, constant contracts for key-only Filters and abs/clamp/negation); floor_for_cap lives in asap-accuracy. Chain retention need is max(offset + range). Worked examples throughout. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01K1M6bq8xVaGKyinf4ohK6q --- .../memo-groups-properties-requirements.md | 426 ++++++++++++++---- 1 file changed, 333 insertions(+), 93 deletions(-) diff --git a/docs/design_docs/concepts/memo-groups-properties-requirements.md b/docs/design_docs/concepts/memo-groups-properties-requirements.md index ac587bcef..330b55cf4 100644 --- a/docs/design_docs/concepts/memo-groups-properties-requirements.md +++ b/docs/design_docs/concepts/memo-groups-properties-requirements.md @@ -85,19 +85,15 @@ This is the single reference table for property dimensions. Stage descriptions b | Dimension | Values and meaning | Code reference or target | |---|---|---| | Parameters | A predefined configuration list per family: KLL and KMV `k`; DDSketch `alpha`; HLL `precision`; CMS and CountSketch `width` and `depth`; heap variants also `heap_size`; UnivMon its sizing parameters. The lists are today's sizing over ε = 0.1 down to 0.001; KMV's and UnivMon's distinct-count entries stop at ε = 0.01. A heap variant combines each width × depth with heap sizes from the 1-2-5 series 10, 20, 50, …, 10 000: a top-k consumer's floor is heap_size ≥ k, and the smallest listed heap that meets it is the next step up, at most 2.5× too large. A sketch build's parameters must be in its family's list; exact operators have no sketch parameters. Hydra and the other families Pass 1 does not size have empty `parameter_list`s, so no build of theirs is a valid candidate until their error models exist (#1228). | `SketchParams` in `crates/types/src/ir/schema/sketch.rs`; predefined lists are the #1203 target | -| Window layout — Extension | No layout, or tumbling windows with a positive width. Every grid is anchored at phase 0 (the epoch), so the width alone fixes it. Candidate widths come from the gcd closure of the queries' ranges, steps and phases. Sliding merge/subtraction layouts remain future work. Window count is derived from range and width; retention amount is selected later. | `WindowLayout`; `pass2/memo/window_candidates.rs`; #1160 | | Timing | `IngestionTime` or `QueryTime` | `ExecutionTiming` | -| Lifetime | `None`: discard after the consuming execution; `OneEvaluation`: retain until consumers within that evaluation finish; `AcrossEvaluations`: retain across evaluations in the specified period. A non-`None` lifetime means materialized. | `Lifetime`; #1200 / #1214 | -| Storage medium — Extension | `Memory` or `Disk`, carried by a materialized lifetime | `StorageMedium`; #1214 | +| Lifetime | `None`: discard after the consuming execution; `OneEvaluation`: retain until consumers within that evaluation finish; `AcrossEvaluations`: retain across evaluations in the specified period. A non-`None` lifetime means materialized. | `Lifetime`; #1200 | +| Storage medium — Extension | `Memory` or `Disk`, carried by a materialized lifetime | `StorageMedium` | Error guarantees are derived from parameters and the statistic being read. They include an error metric, a bound, and a failure probability; exact computation introduces no approximation error. The same state may support different readouts, so its guarantee must be interpreted for the relevant statistic. -The extended logical properties add the window layout; the physical properties stay the formulation's: +The properties are the formulation's, $p_v^{\mathrm{log}}=\boldsymbol\theta_v$ and $p_v^{\mathrm{phys}}=(\tau_v,\ell_v)$; in code, `LogicalProperties { parameters }` and `PhysicalProperties { timing, lifetime }`. -$$ -p_v^{\mathrm{log}}=(\boldsymbol\theta_v,\omega_v),\qquad -p_v^{\mathrm{phys}}=(\tau_v,\ell_v). -$$ +**Tumbling windows are expressions, not properties.** A tumbling-window chain is its own memo group, `SummaryTumblingWindowChain[w](X)`, with the window width $w$ as an operator argument (Section 3, "Chain groups"). Every grid is anchored at phase 0 (the epoch), so the width alone fixes it. A query over a range reads the chain through `TimeRange` and `TimeShift` groups and merges the selected windows with `SummaryMerge`. No property carries the width, and two queries with different ranges or offsets share one chain group. Sliding merge/subtraction layouts remain future work; how many windows a chain retains is $h_v$ (Section 5). ### Logical requirements @@ -118,25 +114,35 @@ Physical requirements are derived from the consumer's timing and execution conte QT means QueryTime; IT means IngestionTime. Alignment, data coverage, and retention checks still apply to allowed combinations. -**Alignment.** There is no physical schedule property. When a result can be read follows from its lifetime and window layout: +**Alignment.** There is no physical schedule property. When a result can be read follows from its lifetime and, for a chain, its window width: - **None** and **OneEvaluation**: the result is computed inside one evaluation context κ (Section 5) and read there. A OneEvaluation result belongs to one context: queries with equal recurrence and range evaluation are evaluated together, and that evaluation's consumers share the result. When it runs is the context's schedule, and Stage 3 chooses it through $x_{\kappa,v}$, just as for ℓ = None. Sharing across differently scheduled queries through a gcd schedule is not offered: it is rarely beneficial and wastes runs. -- **Windowed AcrossEvaluations**: the input is cut into tumbling windows [k·w, (k+1)·w), with w from the window layout ω and the grid anchored at phase 0 (the epoch). Each window is built when it ends and can be read from then on. A consumer can read a range that ends on a window boundary. With w = 10 s, a `[1m]` query evaluated every 30 s at phase 0 merges the newest 6 windows; one evaluated at phase 15 s cannot, because its range ends inside a window; with that query in the workload, the gcd closure also offers w = 5 s, which both can read. +- **AcrossEvaluations** is offered only to chain candidates, `SummaryTumblingWindowChain[w](X)`: the input is cut into tumbling windows [k·w, (k+1)·w), with the grid anchored at phase 0 (the epoch). Each window is built when it ends and can be read from then on. A `TimeRange` over the chain selects the windows that lie fully inside [t − range, t), so a consumer can read a range whose ends lie on window boundaries. With w = 10 s, a `[1m]` query evaluated every 30 s at phase 0 merges 6 windows; one evaluated at phase 15 s cannot, because its range ends inside a window; with that query in the workload, the gcd closure also offers w = 5 s, which both can read. Every other candidate, including the `TimeRange` and `TimeShift` groups over a chain, is computed within one evaluation. + +A grid phase is not needed: a phase is absorbed by the width, because width candidates come from the gcd closure of the queries' grids, ranges and offsets (Section 3). For example, phases 0 and 5 s with w = 10 s give w = 5 s. Alignment alone does not ensure sufficient retained history; that is $h_v$ (Section 5). + +**Why a chain needs a declared phase.** Take `sum_over_time(m[5m])` evaluated every 60 s, and a 60 s chain with windows [12:00:00, 12:01:00), [12:01:00, 12:02:00), … + +| Case | Evaluation at | Data needed | Windows | +|---|---|---|---| +| Phase 0 declared | 12:05:00 | [12:00:00, 12:05:00) | exactly the 5 windows [12:00, 12:01) … [12:04, 12:05): the merge is correct | +| No phase declared; the engine actually evaluates at second 17 | 12:05:17 | [12:00:17, 12:05:17) | [12:00, 12:01) holds only 43 s of the range and [12:05, 12:06) only 17 s: merging only full windows misses [12:00:17, 12:01:00) and [12:05:00, 12:05:17); adding the partial windows counts [12:00:00, 12:00:17) and [12:05:17, 12:06:00) too. Both answers are wrong | +| Phase 15 s declared | 12:05:15 | [12:00:15, 12:05:15) | the widths divide gcd(60 s, 5 m, 15 s) = 15 s; with w = 15 s the range is exactly 20 windows | -A grid phase is not needed: a phase is absorbed by the width, because width candidates come from the gcd closure of the queries' ranges, steps and phases. For example, phases 0 and 5 s with w = 10 s give w = 5 s. Alignment alone does not ensure sufficient retained history; that is $h_v$ (Section 5). +Prometheus staggers rule-group evaluations by a hash of the group, so a fixed interval without a declared phase does not mean phase 0. One rule decides both sides: `QueryRecurrence::evaluation_grid_ms` returns `None` for a `FixedInterval` without a phase. Pass 2 then offers that query no chain width, and `delivers_on` lets it read no chain. Workloads declare the phase with `FixedIntervalAt { interval, evaluation_phase }` to use chains. -An evaluation context κ is described by the workload's own recurrence and range evaluation (Section 5). Let a windowed producer have tumbling width $w$. A consumer running in κ can read the producer when $w$ divides every evaluation time of κ: +An evaluation context κ is described by the workload's own recurrence and range evaluation (Section 5). Let a chain producer have tumbling width $w$. A consumer running in κ can read the producer when $w$ divides every evaluation time of κ: -| Producer lifetime, or κ's recurrence for a windowed AcrossEvaluations producer | Required alignment | +| Producer lifetime, or κ's recurrence for a chain producer with ℓ = AcrossEvaluations | Required alignment | |---|---| | None | Always: runs in κ | | OneEvaluation | The result belongs to κ | | `FixedIntervalAt { interval, evaluation_phase }`, range step $s$ | $w \mid \gcd(\text{interval}, s, \text{phase})$, the evaluation grid; $s$ counts only for a range evaluation of more than one step | | `Scheduled(times)` | $w$ divides every time (and $s$) | | `OneTime { execute_at: Some(t) }` | $w \mid t$ (and $s$) | -| `FixedInterval` without a phase, `OneTime { execute_at: None }`, `EstimatedRate`, `Unknown` | Never: the evaluation times are not known | -| AcrossEvaluations, no window layout | Not offered yet: it needs its own delivery rule | +| `FixedInterval` without a phase, `OneTime { execute_at: None }`, `EstimatedRate`, `Unknown` | Never: the evaluation times are not known, and `evaluation_grid_ms` is `None` | +| AcrossEvaluations on a candidate that is not a chain | Not offered: only chain candidates are materialized across evaluations | -The evaluation grid is `QueryRecurrence::evaluation_grid_ms(range_evaluation)` in `crates/types/src/workload/mod.rs`, which the workload entries also use. One-off queries are supported: each is a context of its own recurrence, run `invocations` times. +The evaluation grid is `QueryRecurrence::evaluation_grid_ms(range_evaluation)` in `crates/types/src/workload/mod.rs`, which the workload entries, Pass 2's chain widths and `delivers_on` all use. One-off queries are supported: each is a context of its own recurrence, run `invocations` times. ### Invalid combinations @@ -146,7 +152,7 @@ The checked constructor rejects one combination: |---|---| | IngestionTime with OneEvaluation | No query evaluation lifetime at ingestion time | -A tumbling grid's width is the candidate's logical window layout $\omega_v$ and its phase is 0, so no physical field repeats or extends it. +A tumbling grid's width is an argument of the chain operator, part of the expression, and its phase is 0, so no property repeats or extends it. ### Decision tree and order of decisions @@ -157,21 +163,19 @@ flowchart TD G["Memo group G
(logical result)"] --> R1["e_v: exact operator"] G --> R2["e_v: readout of KLL state"] G --> R3["e_v: readout of DDSketch state"] + G --> R5["e_v: SummaryMerge over TimeRange
of a chain of width w
(state groups; widths from the
gcd closure of grids, ranges, offsets)"] G --> R4["e_v: other operators
(prune: executor support)"] R2 --> T1["θ: k = 200"] R2 --> T2["θ: k = 345"] R2 --> T3["θ: k = …
(prune: pruning cap)"] - T2 --> W1["ω: no windows"] - T2 --> W2["ω: tumbling, width w
from the gcd closure of ranges,
steps and phases (prune: w must
divide the group's query range)"] - - W2 --> P1["τ = QueryTime"] - W2 --> P2["τ = IngestionTime
(prune: no ingestion pipeline)"] + T2 --> P1["τ = QueryTime"] + T2 --> P2["τ = IngestionTime
(prune: no ingestion pipeline)"] P1 --> L1["ℓ = None"] P1 --> L2["ℓ = OneEvaluation"] - P1 --> L3["ℓ = AcrossEvaluations
(prune: deployment keeps
no query-time results)"] + P1 --> L3["ℓ = AcrossEvaluations
(prune: not a chain; deployment
materializes no query-time results)"] P2 --> L4["ℓ = None"] P2 --> L5["ℓ = AcrossEvaluations"] P2 -. invalid .-> L6["ℓ = OneEvaluation"] @@ -179,11 +183,10 @@ flowchart TD | Level | Decides | Stage | Values offered | Pruned when | |---|---|---|---|---| -| 1 | expression $e_v$ | Pass 1 and Pass 2 rules | each way to compute the group | the executor cannot build the family | -| 2 | $\boldsymbol\theta_v$ | Pass 1 | the family's predefined list | the pruning cap (Section 4) | -| 3 | $\omega_v$ — Extension | Pass 2 | tumbling widths from the gcd closure of the queries' ranges, steps and phases | the width does not divide the group's query range | -| 4 | $\tau_v$ | Stage 2 | QueryTime, IngestionTime | the source has no ingestion pipeline | -| 5 | $\ell_v$ | Stage 2 | None, OneEvaluation, AcrossEvaluations | invalid with $\tau_v$; the deployment keeps no query-time results | +| 1 | expression $e_v$, including a merge over a chain of width $w$ | Pass 1 and Pass 2 rules | each way to compute the group; chain widths from the gcd closure of the reaching queries' grids, ranges and offsets | the executor cannot build the family; no reaching query has a known grid | +| 2 | $\boldsymbol\theta_v$ | Pass 1 | the family's predefined list | the pruning cap (Section 4); dead-candidate elimination (Section 3) | +| 3 | $\tau_v$ | Stage 2 | QueryTime, IngestionTime | the source has no ingestion pipeline | +| 4 | $\ell_v$ | Stage 2 | None, OneEvaluation, AcrossEvaluations | invalid with $\tau_v$; AcrossEvaluations on a candidate that is not a chain; the deployment materializes no query-time results | The tree does not decide which producer each consumer input reads ($y$), in which context a candidate with ℓ = None or OneEvaluation runs (Section 5), or how much an AcrossEvaluations candidate retains ($h_v$). Those are Stage 3 choices. @@ -192,11 +195,11 @@ The tree does not decide which producer each consumer input reads ($y$), in whic **The order changes how many nodes enumeration visits:** $\sum_d \prod_{d' \le d} \lvert \text{values left at } d' \rvert$. Dimensions that prune much go early; dimensions that multiply without pruning go late. The order above follows four rules: 1. **Logical before physical.** Stage 1 hash-conses groups and computes $\alpha^{\log}$ once, before Stage 2 multiplies each logical candidate by its physical alternatives. -2. **Strong prunes early.** Executor support and the pruning cap act before windows and physical properties multiply the tree. +2. **Strong prunes early.** Executor support and the pruning cap act before physical properties multiply the tree. 3. **Dependent dimensions adjacent.** $\tau \to \ell$ rejects the invalid combination at the first level that can detect it. -4. **Deployment-wide prunes as early as possible.** Without an ingestion pipeline the IngestionTime subtree is pruned once per candidate; moving $\tau$ above $\omega$ would prune it once per group, with the same leaves. +4. **Deployment-wide prunes as early as possible.** Without an ingestion pipeline the IngestionTime subtree is pruned once per candidate; moving $\tau$ above $\boldsymbol\theta$ would prune it once per group, with the same leaves. -Alternative orders worth measuring: $\tau$ first, when most sources lack an ingestion pipeline; $\omega$ before $\boldsymbol\theta$, when queries' ranges prune many widths; and requirement-driven (top-down) enumeration, which visits fewer nodes but must take the union of all consumers' requirements so that sharing is not lost. +Alternative orders worth measuring: $\tau$ first, when most sources lack an ingestion pipeline; and requirement-driven (top-down) enumeration, which visits fewer nodes but must take the union of all consumers' requirements so that sharing is not lost. ## 3. Pipeline: interfaces and rules @@ -204,7 +207,7 @@ Alternative orders worth measuring: $\tau$ first, when most sources lack an inge |---|---|---|---| | Stage 0 | Query IR | `Memo` (groups without candidates) | Hash-cons logical expressions | | Stage 1, Pass 1 | Group DAG | `Memo` | Add initial logical candidates and intermediate groups | -| Stage 1, Pass 2 | Memo | `LogicalASAPMemoCandidates` | Add reuse alternatives and compute logical edges | +| Stage 1, Pass 2 | Memo | `LogicalASAPMemoCandidates` | Add rollup, top-k and chain alternatives, set parameter floors, eliminate dead candidates, and compute logical edges | | Stage 2 | Logical memo candidates | `PhysicalASAPMemoCandidates` | Expand physical alternatives and compute compatibility | | Stage 3 | Physical memo candidates | `DagOfSelectedCandidates` | Select implementations, dependencies, and executions | | Realizer/export | Selected DAG | `PhysicalASAPDAG` | Build and export the executable graph | @@ -236,23 +239,26 @@ pub struct MemoGroup { pub struct GroupProperties { pub schema: Schema, pub result_kind: OperatorResultKind, - /// What a summary-state group covers; `None` for any other group. - pub summary_coverage: Option, + /// What an aggregate group covers: a summary state, a chain, a merge + /// over a chain's windows, or an exact `Aggregate` result; `None` for + /// any other group. + pub aggregate_coverage: Option, /// Whether two equal expressions return the same rows for the same /// input rows, so they may be one group (not an unordered `LIMIT`). pub shareable_when_equal: bool, } -/// What a summary state computes, with every lifted selection removed. +/// What a summary state or exact aggregate computes, with every lifted +/// selection removed. pub struct SummaryStateDefinition { /// The group the operators below read: the first input under the - /// SummaryAgg that is not Filter / range TimeRange / TimeShift without @ + /// SummaryAgg (or Aggregate) that is not Filter / TimeRange / TimeShift /// / direct-column Project. `None` only when a Scan's predicates are /// lifted; the Scan, with its residual predicates, is then the last /// unlifted operator. pub input_group: Option, /// The operators not lifted into the selection, listed top down from - /// the SummaryAgg, each with only its residual arguments. `None` marks + /// the SummaryAgg (or Aggregate), each with only its residual arguments. `None` marks /// an operator's main input (the next unlifted operator, or input_group); /// `Some(g)` is another input group it reads, such as the group a scalar /// subquery in a Filter reads. Two states may merge only if input_group @@ -301,9 +307,9 @@ pub(crate) trait OperatorInput: Sized { fn result_kind(&self) -> OperatorResultKind; /// The input's own operator, its children as inputs again. fn operator(&self) -> Operator; - fn summary_coverage(&self) -> Option<&SummaryCoverage>; + fn aggregate_coverage(&self) -> Option<&SummaryCoverage>; } -// Front-end node trees; summary_coverage is None. +// Front-end node trees; aggregate_coverage is None. impl OperatorInput for Rc { … } /// A memo group read as an input: its group_properties and expression. pub(crate) struct MemoGroupAsInput<'m> { memo: &'m Memo, group_id: MemoGroupId } @@ -322,15 +328,15 @@ fn derive_group_properties( ) -> Result; ``` -Summary coverage exists only in the memo: a node tree has none, and `SummaryCoverage` has no type parameter. The executor reads only `definition_id` from the plan. +Aggregate coverage exists only in the memo: a node tree has none, and `SummaryCoverage` has no type parameter. The executor reads only `definition_id` from the plan. A group is its expression: one operator, with its arguments, over input groups. Hash-consing compares and hashes the expression directly, so two expressions are one group when their operators and arguments are equal and their input groups are the same. θ is not an operator argument, so builds that differ only in θ share one group. An expression whose copies may return different rows (an unordered `LIMIT`) always gets a new group. Everything that was computed on operator-node trees is computed on groups. `find_or_add_group` derives `GroupProperties` once, when the group is created, with `derive_group_properties`: the rules that check operator nodes (schema derivation, input validation, merge check), applied to the expression over `MemoGroupAsInput`s, which read only the input groups' `expression` and `group_properties`, never a node tree: - `schema` and `result_kind` from the operator and its input groups' schemas and result kinds; a scalar subquery reads the referenced group's schema. -- `summary_coverage`, for a summary state: walking down from the SummaryAgg through the input groups' expressions while each is a Filter, a range TimeRange, a TimeShift without `@` or a direct-column Project, lifting their predicates into the selection. The group below them is the definition's `input_group`. A Scan's predicates (PromQL label matchers) are lifted too; the Scan with its residual predicates is then the last unlifted operator and `input_group` is `None`. The operators not lifted, each with its residual arguments, are `unlifted_operators`. A tumbling-window chain covers what one build over its whole window does. A merge's coverage comes from its inputs' stored coverage; a merge is valid only over inputs with equal definitions and disjoint selections. -- The checks that read an input's operator (a merge of the newest windows reads a window chain; a selected build) read the input group's `expression`. +- `aggregate_coverage`, for a summary state or an exact `Aggregate` result: walking down from the SummaryAgg or Aggregate through the input groups' expressions while each is a Filter, a TimeRange, a TimeShift (with or without `@`) or a direct-column Project, lifting their predicates into the selection; a TimeRange or TimeShift becomes an interval on the timestamp column (below). The group below them is the definition's `input_group`. A Scan's predicates (PromQL label matchers, SQL filters, including absolute timestamp filters) are lifted too; the Scan with its residual predicates is then the last unlifted operator and `input_group` is `None`. The operators not lifted, each with its residual arguments, are `unlifted_operators`. A chain `SummaryTumblingWindowChain[w](X)` has the definition of a build over X and leaves the timestamp column unconstrained; a `TimeRange` or `TimeShift` over a chain constrains it as over raw rows. A merge's coverage comes from its inputs' stored coverage; a merge is valid only over inputs with equal definitions and disjoint selections, or over one time-indexed state. So the coverage of `SummaryMerge(TimeRange(TimeShift(Chain[w](X))))` equals that of `SummaryAgg(TimeRange(TimeShift(X)))`, and `add_candidate` accepts the merge in the original state group. +- The checks that read an input's operator (a merge over a time-indexed state reads a `TimeRange` over a chain; a selected build) read the input group's `expression`. - `shareable_when_equal`: a `LIMIT` is shareable over a `Sort` with the same partitions (the input group's `expression`); a `Dedup` on every column (the input group's schema). `add_candidate` derives a candidate's properties with the same `derive_group_properties` and requires its schema and result kind to equal the group's. `OperatorNode` recursively contains chosen child nodes and therefore describes a fixed plan fragment; `Operator` contains input group IDs and leaves their implementations open. A candidate needs its own expression because a group may offer several ways to compute the same result. @@ -347,7 +353,7 @@ pub struct GroupProperties { pub result_kind: OperatorResultKind, /// One entry per output column, in schema order. pub column_lineage: Vec, - pub summary_coverage: Option, + pub aggregate_coverage: Option, pub shareable_when_equal: bool, } @@ -365,13 +371,32 @@ pub enum ColumnLineage { Computed, } -/// Summary coverage keys its selections by the same type. +/// Aggregate coverage keys its selections by the same type. The timestamp +/// column is one of the columns; there is no separate time field. pub struct SelectionBox { pub columns: BTreeMap, - pub relative_time: Option<(Bound, Bound)>, +} + +/// An endpoint of an interval on a timestamp column. +pub enum TimeBound { + /// An absolute time, in ms since the epoch. + At(i64), + /// An offset, in ms, from the evaluation time t. + FromEvaluationTime(i64), } ``` +A repeatedly evaluated query covers different absolute times at every evaluation, so its group's coverage is written relative to the evaluation time t; an `@ T` modifier or a SQL timestamp filter is absolute. Both are intervals on the timestamp column, compared and lifted like any other column constraint. For a range evaluation over [t − D, t] (D is the query range's duration): + +| Query fragment | Constraint on `m.ts` | +|---|---| +| `m[5m]` | [t − 5m, t): both ends `FromEvaluationTime` | +| `m[5m] offset 1h` | [t − 1h − 5m, t − 1h) | +| `m[5m] @ end()` | [t − 5m, t): the same as no `@` in an instant query | +| `m[5m] @ start()` | [t − D − 5m, t − D): an offset of D | +| `m[5m] @ 1700000000` | [T − 5m, T) with T = 1700000000 s: both ends `At` | +| SQL `WHERE ts >= '2026-11-01'` | [2026-11-01, ∞): `At` | + **Derivation, per operator** (from the input groups' `column_lineage`): | Operator | Output column lineage | @@ -433,7 +458,9 @@ Implementation locations: `crates/logical-asap-optimization/src/pass1/per_comput | Exact | Exact candidate in the result group | | Exact accumulator | Accumulator-state group plus a finalize candidate in the result group | | Summary | One state group per supported family; one build candidate per listed parameter configuration; separate readout candidates | -| Fused build (#1203 2c) | Several operators of one sub-DAG pattern, in one appearance of the sub-expression, fused into one candidate operator; `fused_groups` lists the groups of the operators it replaces | + +Pass 1 candidates carry no parameter floors, not even a top-k's heap_size ≥ k: Pass 2 sets every floor once all candidates exist. +| Fused build | Several operators of one sub-DAG pattern, in one appearance of the sub-expression, fused into one candidate operator; `fused_groups` lists the groups of the operators it replaces | ```rust pub struct CandidateForMemoGroupId { @@ -463,6 +490,17 @@ impl Memo { ) -> Result; pub fn candidate(&self, id: CandidateForMemoGroupId) -> &CandidateForMemoGroup; + + /// Replaces candidate `id`'s input requirements r^log_{c,i}, one per + /// input group. The only way to change a candidate's r^log: Pass 2 sets + /// each parameter floor from the input's caps and direct needs after the + /// rules have run. Panics when the vector does not have one entry per + /// input group. + pub fn set_input_requirements( + &mut self, + id: CandidateForMemoGroupId, + input_requirements: Vec, + ); } pub enum CandidateError { @@ -481,6 +519,8 @@ pub struct CandidateExpression { pub struct CandidateForMemoGroup { pub expression: CandidateExpression, pub logical_properties: LogicalProperties, + /// r^log_{c,i}, one per input group; changed only through + /// `Memo::set_input_requirements`. pub input_requirements: Vec, /// The groups of the operators this candidate fuses: a sub-DAG pattern /// of several operators, in one appearance of the sub-expression, @@ -493,25 +533,18 @@ pub struct CandidateForMemoGroup { pub struct LogicalProperties { parameters: Option, - window_layout: Option, -} - -pub enum WindowLayout { - Tumbling { width_ms: u64 }, } impl LogicalProperties { - /// Rejects parameters missing on a sketch build, present on an exact - /// operator, of another family or not in the family's list, and a window - /// of width 0 or on an operator that cannot be windowed. + /// Rejects parameters missing on a sketch build (a chain included), + /// present on an exact operator, of another family or not in the + /// family's list. pub fn new( expression: &CandidateExpression, parameters: Option, - window_layout: Option, ) -> Result; pub fn parameters(&self) -> Option<&SketchParams>; - pub fn window_layout(&self) -> Option<&WindowLayout>; } pub fn parameter_list(family: SketchAlgorithm) -> &'static [SketchParams]; @@ -546,22 +579,222 @@ Migration targets: remove private `root_node` chains, duplicate `input_groups`, ### Stage 1, Pass 2: reuse rules and logical edges -Paths are relative to `crates/logical-asap-optimization/src/pass2/memo/`. +Pass 2 (`crates/stage1-pass2-reuse-candidates`) runs these steps in order: -| Rule / function | File | Result or migration target | +| Step | What it does | Why it is here | |---|---|---| -| Local windows: `add_local_window_composition` | `window_candidates.rs` | Add layouts to the state group; #1160 removes copied window consumers | -| Scan widening: `add_scan_label_candidates` | `rules/scan_labels.rs` | Add a widened scan group and projections in the original groups; #1222 | -| Summary inputs: `factor_summary_inputs` | `rules/summary_inputs.rs` | Expose projections/filters as input groups; #1218 moves this into initial population | -| Top-k reuse: `add_topk_candidates` | `rules/topk.rs` | Read a compatible larger top-k result with an exact-input requirement; #1220 | -| Finer aggregate: `add_finer_aggregate_candidates` | `rules/finer_aggregate.rs` | Filter and roll up finer groups, using separate intermediate groups; #1220 | -| Rollup: `add_rollup_candidates` | `rules/rollup.rs` | Add compatible finer-state merges; #1221 | -| Window rollup: `add_window_rollup_candidates` | `rules/rollup.rs` | Merge shorter windows into longer-range results; #1221 | -| Summary results: `factor_summary_results` | `rules/summary_results.rs` | Removed once Pass 1 exposes readouts; #1218 | -| Shared configurations: `add_state_configuration_candidates` | `rules/shared_configuration_readers.rs` | Remove copied consumers once producer variants are selectable; #1203 2b | -| Shared state groups: `add_shared_summary_state_groups` | `state_groups.rs` | Remove per-size groups; #1203 2b / #1160 | - -After the rules, Pass 2 builds the logical edges and applies the pruning caps of Section 4. Edges name the consumer input `input_index_of_consumer`. +| 1. Rollup from the finest common group | Adds candidates that compute a coarser group from a finer one, and synthesizes one finest common group per cluster of equal definitions | Its candidates are new consumers that later steps must see | +| 2. Top-k reuse | Reads a compatible larger top-k result with an exact-input requirement | Same reason | +| 3. Reach, caps and floors | Computes which queries reach each candidate along candidate edges, the validity and pruning caps (Section 4), and each input's parameter floor, written with `Memo::set_input_requirements` | Needs every consumer added by steps 1 and 2 | +| 4. Dead-candidate elimination | Removes candidates no query can use, transitively, and records the root cause of each | Needs the floors; keeps step 5 from building windows for dead candidates | +| 5. Chain groups | Adds tumbling-window chain groups and the merges that read them, with widths from the queries that reach each state group | Needs the reach of step 3, including rollup consumers, and only the survivors of step 4 | +| 6. Logical edges | Records every (producer, consumer, input) and whether it satisfies the input's logical requirement | Runs on the final memo | + +Rules match group expressions, add groups with `find_or_add_group` and candidates with `add_candidate`, and never copy consumers to make a new producer visible. + +#### Rollup from the finest common group + +One rule computes a coarse group $G_c$ from a finer group $G_f$. It covers coarser SQL aggregates, coarser summary states, filtered sketch rollups, rollups of finalized values, and scan widening. $G_c$ can be computed from $G_f$ when all of these hold: + +1. **Equal definition.** Both have an `aggregate_coverage` and equal `definition`s: the same computation over the same input rows, apart from the selection. +2. **Rows are a subset.** $G_c$'s selection is inside $G_f$'s. Where it is narrower, the narrowing constrains only columns that are group-by keys of $G_f$ (matched by column lineage), so a compensating `Filter` over $G_f$'s output can select those rows. A `Filter` over state rows keeps them states. +3. **Keys are coarser.** $G_c$'s keys are a subset of $G_f$'s keys, by lineage. +4. **A rollup exists.** States roll up with `SummaryMerge(group_by)`. Finalized values roll up as follows; anything else rolls up only from states: + + | Finer measure | Coarser measure | + |---|---| + | sum | sum | + | count | sum | + | min / max | min / max | + | avg, quantile, top-k, distinct count | only from states | +5. **Equal time constraints.** The constraints on the timestamp column are equal. A longer range is read from a shorter one's windows through a shared chain (below), not by this rule. + +The candidate added to $G_c$ is `Rollup(by keys of Gc)(Filter(compensation, Gf))`, without the `Filter` when the selections are equal. The compensation must use only key columns because each output row of $G_f$ has already combined all raw rows of its key: a predicate on any other column, or on a computed value, cannot separate them. + +**Finest common group.** For every cluster of at least two groups with equal definitions and equal time constraints, the rule synthesizes one group $G^*$ that every member can roll up from: +- its selection is, per column, the union of the members' constraints; +- its keys are the union of the members' keys, plus every column on which some member's selection is narrower than $G^*$'s, so each member's compensation uses only $G^*$'s keys. + +$G^*$ is added with `find_or_add_group`, so it is an existing group when one matches, and each member gets a rollup candidate from it. Only this finest group is synthesized; intermediate groupings are not enumerated. $G^*$ is no query's output: Stage 3 selects it only when it is cheaper, and it is eliminated in step 4 when nothing reads it. + +A Scan has no aggregation, so for scans every column is a key and the rollup step is empty: the finest common Scan reads the union of the members' label or column selections, and each member reads it through a compensating `Filter`/`Project`. + +**Example 1: filtered sketch rollup.** F = `quantile by (job, host)(…latency[5m])`, C = `quantile by (host)(…latency{job="api"}[5m])`. + +| Group | Expression | Coverage (definition, selection) | Keys | Candidates | +|---|---|---|---|---| +| G1 | `Scan latency` | | | | +| G2 | `TimeRange(5m, G1)` | | | | +| G3 | `SummaryAgg[KLL, by job, host](G2)` | D, `ts` ∈ [t − 5m, t) | job, host | builds | +| G4 | `Scan latency{job="api"}` | | | | +| G5 | `TimeRange(5m, G4)` | | | | +| G6 | `SummaryAgg[KLL, by host](G5)` | D, `ts` ∈ [t − 5m, t), `job` ∈ {api} | host | builds; **`SummaryMerge(by host)(G7)`** | +| G7 | `Filter(job = "api", G3)` | | | its own expression; output is states | + +The definitions are equal, the narrowing `job ∈ {api}` is on a key of G3, {host} ⊆ {job, host}, and KLL states merge. G3 is already the finest group of the cluster {G3, G6}, so $G^*$ = G3. A query `quantile by (host)(…latency{status="500"}[5m])` gets no such candidate: `status` is not a key of G3, and every state already mixes status 200 and 500 samples. + +**Example 2: rollup of a finer finalized sum.** F = `sum by (job, instance)(rate(x[1m]))`, C = `sum by (job)(rate(x[1m]))`. + +| Group | Expression | Keys | Candidates | +|---|---|---|---| +| G3 | `rate(x[1m])` per series | | | +| G4 | `Aggregate(by job, instance; sum)(G3)` | job, instance | exact; finalize of an exact accumulator state | +| G5 | `Aggregate(by job; sum)(G3)` | job | exact; finalize of an exact accumulator state; **`Aggregate(by job; sum)(G4)`** | + +Both are exact `Aggregate` groups with equal definitions and selections, so they have `aggregate_coverage` too. Stage 3 chooses by cost between reading F's finalized result (cheap when it is materialized) and merging accumulator states. + +**Example 3: finest common group that no query asks for.** Q1 = `quantile by (host)(latency[5m])`, Q2 = `quantile by (job)(latency{host="xxx"}[5m])`. + +| Group | Expression | Coverage (definition, selection) | Keys | Candidates | +|---|---|---|---|---| +| G1 | `SummaryAgg[KLL, by host](TimeRange 5m, Scan latency)` | D, all | host | builds; **`SummaryMerge(by host)(G*)`** | +| G2 | `SummaryAgg[KLL, by job](TimeRange 5m, Scan latency{host="xxx"})` | D, `host` ∈ {xxx} | job | builds; **`SummaryMerge(by job)(G3)`** | +| **G\*** | `SummaryAgg[KLL, by job, host](TimeRange 5m, Scan latency)` | D, all | job, host | builds | +| G3 | `Filter(host = "xxx", G*)` | | | its own expression | + +The hull of {all, host ∈ {xxx}} is all; the keys are {host} ∪ {job} ∪ {host}, where the last `host` is the column on which G2 is narrower. Stage 3 chooses between two KLL builds and one build of G\* with job × host states. + +#### Reach, caps and floors + +A query reaches a candidate when a path of candidate edges leads from the query's output group to it; this is the reach $a_{q,v}$, computed top down after steps 1 and 2 so that rollup consumers count. Section 4 defines the caps from it. Each input's floor is the strongest of the floor for its cap (`floor_for_cap` in `asap-accuracy`) and any direct need such as heap_size ≥ k under a top-k. `Memo::set_input_requirements` writes it; nothing else changes a candidate's r^log. + +#### Dead-candidate elimination + +A candidate that is not a query output is **dead** when it never had a consumer, or when every edge to a consumer fails: the producer is below the input's floor, the metric rule forbids the edge (Section 4), or the consumer is dead. Elimination repeats until nothing changes. Dead candidates get no logical edges and Stage 2 does not expand them; each is recorded with its root causes: + +```rust +pub struct LogicalASAPMemoCandidates { + pub memo: Memo, + pub edges: Vec, + pub pruned: Vec, +} + +/// A candidate eliminated by Pass 2. +pub struct PrunedLogicalCandidate { + pub candidate: CandidateForMemoGroupId, + /// Every cause: one per failing edge it is cut off by, so mixed causes + /// (a floor on one edge, the metric rule on another) are all listed. + pub causes: Vec, +} + +/// One path from the candidate, through consumers that are themselves +/// eliminated, to the edge that fails. One shortest path per failing edge. +pub struct PruneCause { + /// The eliminated consumers on the path, nearest first. + pub through_pruned_consumers: Vec, + pub root: RootPruneReason, +} + +pub enum RootPruneReason { + /// The producer's parameters are weaker than the consumer input's floor. + BelowAccuracyFloorRequirement { + consumer: CandidateForMemoGroupId, + input: usize, + floor: SketchParams, + }, + /// The consumer's error contract carries none of the producer's metrics. + MetricRule { + consumer: CandidateForMemoGroupId, + input: usize, + producer_metric: AccuracyMetric, + consumer_carries: Vec, + }, + /// The candidate never had a consumer and is not a query output. A + /// candidate whose consumers were eliminated is traced to their causes. + NoConsumer, +} +``` + +**Example: ratio of quantiles.** `quantile_over_time(0.99, m[5m]) / quantile_over_time(0.5, m[5m])` with relative error 0.05. + +| Group | Expression | Candidates | +|---|---|---| +| G1 | `Scan m` | | +| G2 | `TimeRange(5m, G1)` | | +| G3 | `SummaryAgg[KLL](G2)` | #0 … #6: k = 132 … 2863 | +| G3′ | `SummaryAgg[DDSketch](G2)` | #0 … #6: alpha = 0.1, 0.05, 0.02, 0.01, 0.005, 0.002, 0.001 | +| G4 | p99 | #0 exact; #1 `SummaryEstimate[q=0.99](G3)`; #2 `SummaryEstimate[q=0.99](G3′)` | +| G5 | p50 | #0 exact; #1 `SummaryEstimate[q=0.5](G3)`; #2 `SummaryEstimate[q=0.5](G3′)` | +| G6 | `Divide(G4, G5)` | #0, the query output | + +The Divide carries only relative error. Its inputs' cap is 0.05 / 1.111 = 0.045 (Section 4), so the DDSketch readouts' floor is alpha ≤ 0.02. Pass 2 records: + +``` +G4#1 SummaryEstimate[q=0.99](G3) + G4#1 → G6#0 input 0: MetricRule { producer_metric: Rank, consumer_carries: [Relative] } +G5#1 SummaryEstimate[q=0.5](G3) + G5#1 → G6#0 input 1: MetricRule { producer_metric: Rank, consumer_carries: [Relative] } +G3#0 SummaryAgg[KLL] k=132 (and likewise G3#1 … G3#6) + G3#0 → G4#1 → G6#0 input 0: MetricRule { producer_metric: Rank, consumer_carries: [Relative] } + G3#0 → G5#1 → G6#0 input 1: MetricRule { producer_metric: Rank, consumer_carries: [Relative] } +G3′#0 SummaryAgg[DDSketch] alpha=0.1 (and likewise G3′#1, alpha=0.05) + G3′#0 → G4#2 input 0: BelowAccuracyFloorRequirement { floor: alpha 0.02 } + G3′#0 → G5#2 input 0: BelowAccuracyFloorRequirement { floor: alpha 0.02 } +``` + +The KLL builds' own edges satisfy their consumers; they are dead because their only consumers are, and the trace leads through those consumers to the metric rule. The exact candidates and the DDSketch builds with alpha ≤ 0.02 survive. + +#### Chain groups + +A state group whose expression is `SummaryAgg(TimeRange(W, TimeShift*(X)))` can be computed from a chain of tumbling windows over X: + +```rust +// The chain: one row per window, (window end, keys, state): a State with a +// time index. No range and no window count: how much it retains is h_v. +SummaryTumblingWindowChain { child: X, family, input, reduction, grouping, filter, + tumbling_window_width_ms: u64 } +// Merges every input state, or regroups them; no window-selection field. +SummaryMerge { children, group_by } +``` + +`TimeRange` and `TimeShift` accept a time-indexed state: over a chain, `TimeRange(W)` at evaluation time t selects the windows lying fully inside [t − W, t), and `TimeShift(o)` moves that interval to [t − o − W, t − o). This needs w to divide W and o, and t to lie on the grid; the widths below and `delivers_on` (Section 2) ensure both. The `TimeRange` and `TimeShift` groups over a chain have only their own expression as a candidate and are computed within one evaluation; only the chain is materialized across evaluations. + +**The rule, for each width w of a state group S:** +1. Walk down from S: `SummaryAgg` → `TimeRange(W)` → zero or more `TimeShift`s → X. +2. `find_or_add_group(SummaryTumblingWindowChain[w](X))` → $G_w$. The shifts are stripped, so queries with different ranges and offsets over the same X get the same $G_w$. +3. Re-wrap the stripped shifts and then the `TimeRange` around $G_w$, in their original order, each with `find_or_add_group`. +4. `add_candidate(S, SummaryMerge(…))` over the outermost re-wrapped group. Its coverage equals S's (Section 3, Stage 0), so the candidate is accepted. + +There is one chain per width, shared by every state group over the same X and family. + +**Widths.** Each query q that reaches S (step 3) and repeats with a known evaluation grid $g_q$ (`evaluation_grid_ms` is `Some`) gives the width $\gcd(g_q, W, o_1, \ldots)$, with the offsets $o_j$ on its path; S's widths are the gcd closure of these. Queries without a known grid give none, so a group read only by such queries gets no chain. `@` modifiers: +- `@ end()` reads [t − W, t) relative to the end of the query range, as no `@` does: it is stripped and re-wrapped like a shift and adds no offset. +- `@ start()` is an offset of the query range's duration D and enters the gcd as one. +- `@ T` (absolute) contributes no widths: S gets a merge only over chains that other queries' widths created, and only for widths that divide T. Whether reusing the chain is cheaper than computing the result once and materializing it is Stage 3's choice; its retention need is in Section 5. + +**Example: three queries share one chain.** A = `sum_over_time(m[5m])`, B = `sum_over_time(m[5m] offset 1h)`, C = `sum_over_time(m[1h])`, each `FixedIntervalAt { 60 s, phase 0 }`. The widths are gcd(60 s, 5m) = gcd(60 s, 5m, 1h) = gcd(60 s, 1h) = 60 s. + +| Group | Expression | Candidates | +|---|---|---| +| G1 | `Scan m` | | +| G3a | `SummaryAgg[Sum](TimeRange(5m, G1))` | build; **`SummaryMerge(Ga)`** | +| G3b | `SummaryAgg[Sum](TimeRange(5m, TimeShift(1h, G1)))` | build; **`SummaryMerge(Gb)`** | +| G3c | `SummaryAgg[Sum](TimeRange(1h, G1))` | build; **`SummaryMerge(Gc)`** | +| **Gw** | `SummaryTumblingWindowChain[Sum, w = 60 s](G1)` | one build per θ | +| Ga | `TimeRange(5m, Gw)` | its own expression | +| Gs | `TimeShift(1h, Gw)` | its own expression | +| Gb | `TimeRange(5m, Gs)` | its own expression | +| Gc | `TimeRange(1h, Gw)` | its own expression | + +At t = 12:00, Ga selects the 5 windows [11:55, 11:56) … [11:59, 12:00), Gb the 5 windows [10:55, 10:56) … [10:59, 11:00), and Gc 60 windows. Each sample updates one chain instead of three, and the chain retains max(5m, 1h + 5m, 1h) = 65m, 65 windows (Section 5). + +**Example: the rule step by step, for B.** Before the rule: + +| Group | Expression | +|---|---| +| G1 | `Scan m` | +| G2 | `TimeShift(1h, G1)` | +| G3 | `TimeRange(5m, G2)` | +| G4 | `SummaryAgg[Sum](G3)`: B's state group, one build candidate | + +1. Walk down from G4: `SummaryAgg` reads G3; G3 is `TimeRange(W = 5m)` over G2; G2 is `TimeShift(1h)` over G1; G1 is not a shift. So W = 5m, the shifts are [`TimeShift(1h)`], X = G1. +2. `find_or_add_group(SummaryTumblingWindowChain[Sum, w = 60 s](G1))` → Gw. A and C get the same Gw. +3. `find_or_add_group(TimeShift(1h, Gw))` → Gs; `find_or_add_group(TimeRange(5m, Gs))` → Gb. +4. `add_candidate(G4, SummaryMerge(Gb))`. + +After the rule, G4 has two candidates, the build and `SummaryMerge(Gb)`; G2 and G3 stay. With widths {30 s, 60 s}, steps 2–4 run once per width, giving two chains and two merge candidates in G4; Stage 3 chooses. + +#### Logical edges ```rust fn apply(memo: &mut Memo) -> Result<(), Pass2RuleError>; @@ -572,16 +805,13 @@ pub struct LogicalEdge { pub input_index_of_consumer: usize, pub satisfies_logical_requirement: bool, } - -pub struct LogicalASAPMemoCandidates { - pub memo: Memo, - pub edges: Vec, -} ``` +An edge satisfies the logical requirement when the producer meets the input's floor and the consumer's error contract carries the producer's metric (Section 4). + ### Stage 2: physical alternatives -`enumerate_candidate_configurations` in `crates/physical-asap-optimization/src/catalog/configurations.rs` expands each logical candidate using the dimensions and checks in Section 2. It creates no new logical candidates. Each logical edge is expanded between the relevant physical alternatives. +`enumerate_candidate_configurations` in `crates/physical-asap-optimization/src/catalog/configurations.rs` expands each logical candidate that survived Pass 2 using the dimensions and checks in Section 2; it offers ℓ = AcrossEvaluations only to chain candidates, and records every other AcrossEvaluations alternative as pruned with `PruneReason::AcrossEvaluationsOnlyForChains`. It creates no new logical candidates. Each logical edge is expanded between the relevant physical alternatives. ```rust // crates/types/src/physical/properties.rs @@ -617,9 +847,9 @@ impl PhysicalProperties { impl PhysicalCandidate { /// Whether a consumer running in `context` can read this producer - /// (Section 2's alignment table): for a windowed AcrossEvaluations - /// producer, w (read from its window layout) divides every evaluation - /// time of the context; true for ℓ = None, and for OneEvaluation in the + /// (Section 2's alignment table): for a chain producer with + /// ℓ = AcrossEvaluations, w (the chain operator's width) divides every + /// evaluation time of the context (`evaluation_grid_ms`); true for ℓ = None, and for OneEvaluation in the /// same context. pub fn delivers_on(&self, memo: &Memo, context: &ExecutionContext) -> bool; } @@ -669,7 +899,7 @@ pub struct PhysicalCandidateKey { } ``` -`PhysicalCandidateId` indexes an emitted candidate. `PhysicalCandidateKey` describes an alternative rejected before it received an ID, allowing diagnostics to explain pruning. The admissible-edge accessor returns edge IDs, not the unfiltered producer set from Section 1. `delivers_on` takes a candidate rather than bare properties because a tumbling grid's width lives in the logical window layout. +`PhysicalCandidateId` indexes an emitted candidate. `PhysicalCandidateKey` describes an alternative rejected before it received an ID, allowing diagnostics to explain pruning. The admissible-edge accessor returns edge IDs, not the unfiltered producer set from Section 1. `delivers_on` takes a candidate and the memo rather than bare properties because a tumbling grid's width is an argument of the chain operator in the candidate's expression. Migration targets include replacing `Materialization` and duplicated physical fields with checked properties, and replacing `ReaderConstraint::{IncompatibleOutputs, AllowedOutputs}` with compatibility and alignment checks. `ReaderConstraint::{MinimumQueryReaders, HighestSharedQueryMaterialization}` are deleted too; see Section 5. @@ -758,12 +988,18 @@ $$ | Operation and metric | Applicable bound or condition | |---|---| -| Exact pass-through | Preserves the input guarantee only when it preserves the metric's semantics | +| Exact pass-through (column `Project`, `Sort`, `TimeRange`, `TimeShift`, `Concat`, readouts, finalize, `SummaryMerge`) | Preserves the input guarantee only when it preserves the metric's semantics; β = 1 in any metric | +| `Filter` whose predicate reads only columns copied from a source (lineage `SourceColumn`: labels and keys, which are exact) | β = 1 in any metric: it selects whole rows and never compares an approximate value | +| abs, clamp, clamp_min, clamp_max, negation | Absolute error, β = 1 (1-Lipschitz) | | Sum/difference, absolute error | Sum of the operands' absolute error bounds | | Sum over a known number of values | Sum of their bounds; a common per-value bound can be multiplied by that count | +| min / max aggregate | Absolute or relative error, β = 1 | +| avg aggregate, absolute error | β = 1 | | Ratio, relative error | Requires nonzero true denominator and denominator relative error below one; bound given below | +| Product, relative error | (1 + b) for the left operand and 1 for the right, whose relative error is capped at b: $(1+x)(1+y)-1\le(1+b)x+y$ when $y\le b$ | +| sum / avg, relative error | Holds when the values are known to be nonnegative; not offered until that sign information is an input of the error model | -A filter does not automatically preserve every accuracy guarantee. Group counts may need metadata beyond the expression syntax. Each factor must follow from the operator's error contract. +A constant contract is unconditional: it needs no row count or sign. A filter does not automatically preserve every accuracy guarantee; a `Filter` that compares an approximate value (an alert threshold such as `quantile_over_time(…) > 0.5`) has no contract. Group counts may need metadata beyond the expression syntax. Each factor must follow from the operator's error contract. **Metric rule.** Every local guarantee states its metric: absolute (in value units, or a multiple of a stream norm), relative, or rank. A factor $\beta_{c,i}$ is defined for one input metric and one output metric. An input whose metric the consumer's error contract cannot convert gives no edge ($\alpha^{\log} = 0$); the conversion is never assumed. For example, a relative-error ratio cannot read a KLL quantile (rank error) unless a declared distribution assumption supplies the conversion. @@ -774,7 +1010,7 @@ $$ \frac{\varepsilon_X+\varepsilon_Y}{1-\varepsilon_Y}. $$ -Choosing a fixed validity cap $b<1$ for the denominator gives a linear upper bound within that region: +Choosing a fixed validity cap $b<1$ for the denominator gives a linear upper bound within that region. The design fixes $b = 0.1$, for a ratio's denominator and a product's right operand alike: 0.1 is the weakest ε in every family's list, so the cap itself never excludes a listed configuration and only raises β by about 11%: $$ \varepsilon_Y\le b @@ -782,9 +1018,9 @@ $$ \varepsilon_{X/Y}\le\frac{1}{1-b}(\varepsilon_X+\varepsilon_Y). $$ -Both local factors are then `1 / (1 - b)`. This is a sufficient bound, not an exact equality or a claim that all feasible nonlinear plans remain represented. +Both local factors are then `1 / (1 - b)` = 1.111. This is a sufficient bound, not an exact equality or a claim that all feasible nonlinear plans remain represented. -Interface (approved 2026-11-05): +Interface: ```rust pub struct ErrorPropagation { @@ -796,13 +1032,17 @@ pub struct ErrorPropagation { } impl LogicalRequirements { + /// g_c's linear bound for errors in `metric`, each input's and the + /// output's. `None` when the consumer has no error contract for `metric`: + /// an input whose error is in `metric` then gives no edge (metric rule). pub fn error_propagation( consumer: &Operator, - ) -> ErrorPropagation; + metric: AccuracyMetric, + ) -> Option; } ``` -`factor_per_input` holds the local β factors and `validity_cap_per_input` the per-input caps, aligned by input index. An operator without an error contract is reported as unsupported; no factor is guessed. +`factor_per_input` holds the local β factors and `validity_cap_per_input` the per-input caps, aligned by input index. An operator without an error contract for a metric returns `None`; no factor is guessed. Every contract keeps the metric: the input's and the output's are the same. For the ratio query of Section 3, `error_propagation(Divide, Relative)` is `Some { factor_per_input: [1.111, 1.111], validity_cap_per_input: [None, Some(0.1)] }`, and the absolute and rank calls return `None`. The key-only `Filter` contract reads the input group's column lineage. ### Query composition and root checks @@ -860,7 +1100,7 @@ $$ A zero factor gives no pruning restriction for that query. If a factor depends on the eventual plan, use a proven lower bound over all feasible downstream alternatives for this pruning test, or defer pruning. Using an arbitrary path's larger factor could discard a feasible plan. The maximum over queries protects alternatives that can serve at least one query. -Compute the parameter floor from this cap only under the ordering condition in Section 2. Validity information is available when the consumer's error model is established; query-sensitive pruning follows logical generation and reachability analysis. Root checks still couple all contributing errors and determine feasibility. +Compute the parameter floor from this cap only under the ordering condition in Section 2. `floor_for_cap` in `asap-accuracy` maps a cap to the weakest listed configuration whose ε meets it, for a family and statistic; Pass 2 takes the strongest of it and any direct need such as heap_size ≥ k. For example, with relative error 0.05 at a ratio, each operand's cap is 0.05 / 1.111 = 0.045 and the denominator's is min(0.1, 0.045) = 0.045, so DDSketch's floor is alpha ≤ 0.02. Validity information is available when the consumer's error model is established; query-sensitive pruning follows logical generation and reachability analysis. Root checks still couple all contributing errors and determine feasibility. ## 5. Execution contexts and other selection extensions — #1207 @@ -868,7 +1108,7 @@ These extensions go beyond constraints (A)–(I) and the formulation's fixed-cos ### Execution identity -An AcrossEvaluations candidate has its own computation: each window is built when it ends. A candidate with ℓ = None runs inside an execution that needs it. For example, a finalize candidate used by queries A (every 10 s) and C (every 15 s) may execute in both evaluation contexts even though Stage 2 enumerates it only once. A OneEvaluation candidate also runs inside one evaluation context κ, and that evaluation's consumers share the result. +An AcrossEvaluations candidate, always a chain, has its own computation: each window is built when it ends. A candidate with ℓ = None runs inside an execution that needs it. For example, a finalize candidate used by queries A (every 10 s) and C (every 15 s) may execute in both evaluation contexts even though Stage 2 enumerates it only once. A OneEvaluation candidate also runs inside one evaluation context κ, and that evaluation's consumers share the result. A **context** identifies a class of executions with a common owner and schedule across the accounting horizon. It is not necessarily one timestamp. An evaluation context reuses the workload's own description of when queries run, their `QueryRecurrence` and `RangeEvaluation`: queries with equal ones are evaluated together and share its OneEvaluation results. Queries A and B of the running example, both every 10 s at phase 0, form one context; a one-off query forms the context of its `OneTime` recurrence. `QueryRecurrence` holds an `f64` (`EstimatedRate`) and cannot be hashed, so contexts are deduplicated by equality and `ExecutionContextId` indexes the list of distinct contexts. @@ -885,7 +1125,7 @@ pub enum ExecutionContext { } ``` -A chain of ℓ = None and OneEvaluation candidates inherits its consuming context. Encountering an AcrossEvaluations producer switches to that producer's own computation context when traversing its inputs. The context identifies where the operation runs; `delivers_on` with the context decides whether a windowed producer can supply it. +A chain of ℓ = None and OneEvaluation candidates inherits its consuming context. Encountering an AcrossEvaluations producer switches to that producer's own computation context when traversing its inputs. The context identifies where the operation runs; `delivers_on` with the context decides whether a chain producer can supply it. | Extended variable | Meaning | |---|---| @@ -904,12 +1144,12 @@ To avoid confusing per-execution costs with the baseline's horizon-wide cost, us | Symbol | Definition | |---|---| -| $W_v^{\mathrm{mat}}$ | Fixed computation and access cost over the horizon for an AcrossEvaluations candidate, one build per window, excluding variable retention | +| $W_v^{\mathrm{mat}}$ | Fixed computation and access cost over the horizon for an AcrossEvaluations candidate (a chain), one build per window, excluding variable retention | | $c_{\kappa,v}$ | Cost of one execution in the specified context of a candidate with ℓ = None or OneEvaluation, including a OneEvaluation result's storage within that evaluation | | $\rho_\kappa$ | Fixed number or expected number of context executions over the common horizon, from the existing evaluation counting in `crates/cost/src/execution_demand.rs` (a one-off query's is its `invocations`) | | $h_v$ | Retention amount in a declared unit, such as retained windows | | $\pi_v$ | Horizon-wide retention cost per unit of retained amount | -| $\operatorname{need}_{\kappa,v,c,i}$ | Retention amount producer v needs to serve this consumer input in this context: the time horizon the queries served by that context read, e.g. a 5m range over 10 s tumbling windows needs 30 windows. Query latency is ignored for now (decided 2026-11-06) | +| $\operatorname{need}_{\kappa,v,c,i}$ | Retention amount producer v needs to serve this consumer input in this context: the time horizon the queries served by that context read, e.g. a 5m range over 10 s tumbling windows needs 30 windows. Query latency is ignored for now | A proposed linear objective is: @@ -927,7 +1167,7 @@ $$ h_v\ge\operatorname{need}_{\kappa,v,c,i}\,y_{\kappa,v,c,i}. $$ -The need is fixed per edge from the time horizon of the queries it serves, so constraint (R) is linear. It ignores query latency: a result is assumed read as soon as its evaluation starts, so a slow consumer that reads later does not extend the horizon. Modeling that needs the latency design, which is out of scope (Section 5). The retention unit (windows, or seconds of data) and an upper bound on $h_v$ are fixed per candidate when Stage 2 builds it. +The need is fixed per edge from the time horizon of the queries it serves, so constraint (R) is linear. For a chain it is the reading path's offset plus range: in the example of Section 3, the edge to A's path needs 5m, B's 1h + 5m and C's 1h, so $h_v$ for the shared 60 s chain is at least 65m, 65 windows. `@ end()` needs the range; `@ start()` needs the query range's duration plus the range. An `@ T` path needs the end of the horizon − T + range; when the horizon is unbounded the need is too, and that merge candidate is not offered. It ignores query latency: a result is assumed read as soon as its evaluation starts, so a slow consumer that reads later does not extend the horizon. Modeling that needs the latency design, which is out of scope (Section 5). The retention unit (windows, or seconds of data) and an upper bound on $h_v$ are fixed per candidate when Stage 2 builds it. ### No consumer-count policies From bde2ff6f4c077efe58b0100d58ff025ac1e14081 Mon Sep 17 00:00:00 2001 From: zzylol <50204836+zzylol@users.noreply.github.com> Date: Sat, 7 Nov 2026 17:00:47 +0000 Subject: [PATCH 42/44] docs(concepts): no executor-support check, ingestion-time counter and filtered builds, Stage 2 interface Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01K1M6bq8xVaGKyinf4ohK6q --- .../memo-groups-properties-requirements.md | 87 +++++++++++++++---- 1 file changed, 71 insertions(+), 16 deletions(-) diff --git a/docs/design_docs/concepts/memo-groups-properties-requirements.md b/docs/design_docs/concepts/memo-groups-properties-requirements.md index 330b55cf4..324b084f8 100644 --- a/docs/design_docs/concepts/memo-groups-properties-requirements.md +++ b/docs/design_docs/concepts/memo-groups-properties-requirements.md @@ -164,7 +164,7 @@ flowchart TD G --> R2["e_v: readout of KLL state"] G --> R3["e_v: readout of DDSketch state"] G --> R5["e_v: SummaryMerge over TimeRange
of a chain of width w
(state groups; widths from the
gcd closure of grids, ranges, offsets)"] - G --> R4["e_v: other operators
(prune: executor support)"] + G --> R4["e_v: other operators"] R2 --> T1["θ: k = 200"] R2 --> T2["θ: k = 345"] @@ -175,7 +175,7 @@ flowchart TD P1 --> L1["ℓ = None"] P1 --> L2["ℓ = OneEvaluation"] - P1 --> L3["ℓ = AcrossEvaluations
(prune: not a chain; deployment
materializes no query-time results)"] + P1 --> L3["ℓ = AcrossEvaluations
(offered only to chains; prune: deployment
retains no query-time results)"] P2 --> L4["ℓ = None"] P2 --> L5["ℓ = AcrossEvaluations"] P2 -. invalid .-> L6["ℓ = OneEvaluation"] @@ -183,10 +183,12 @@ flowchart TD | Level | Decides | Stage | Values offered | Pruned when | |---|---|---|---|---| -| 1 | expression $e_v$, including a merge over a chain of width $w$ | Pass 1 and Pass 2 rules | each way to compute the group; chain widths from the gcd closure of the reaching queries' grids, ranges and offsets | the executor cannot build the family; no reaching query has a known grid | +| 1 | expression $e_v$, including a merge over a chain of width $w$ | Pass 1 and Pass 2 rules | each way to compute the group; chain widths from the gcd closure of the reaching queries' grids, ranges and offsets | no reaching query has a known grid (no chain is offered) | | 2 | $\boldsymbol\theta_v$ | Pass 1 | the family's predefined list | the pruning cap (Section 4); dead-candidate elimination (Section 3) | -| 3 | $\tau_v$ | Stage 2 | QueryTime, IngestionTime | the source has no ingestion pipeline | -| 4 | $\ell_v$ | Stage 2 | None, OneEvaluation, AcrossEvaluations | invalid with $\tau_v$; AcrossEvaluations on a candidate that is not a chain; the deployment materializes no query-time results | +| 3 | $\tau_v$ | Stage 2 | QueryTime, IngestionTime; IngestionTime for a counter state only when the source's out-of-order bound is known (Stage 2) | the source has no ingestion pipeline; QueryTime over raw rows the deployment does not retain | +| 4 | $\ell_v$ | Stage 2 | None, OneEvaluation, AcrossEvaluations; AcrossEvaluations only for chain candidates, and Disk only where the deployment offers it | invalid with $\tau_v$; the deployment retains no query-time results | + +The planner does not check whether an executor implements an operator or family: every candidate the rules produce is offered, and an executor that lacks one is extended. A value outside a dimension's declared value set (AcrossEvaluations on a candidate that is not a chain, Disk where the deployment has no disk storage) is not offered, so it is not recorded as pruned. The tree does not decide which producer each consumer input reads ($y$), in which context a candidate with ℓ = None or OneEvaluation runs (Section 5), or how much an AcrossEvaluations candidate retains ($h_v$). Those are Stage 3 choices. @@ -195,7 +197,7 @@ The tree does not decide which producer each consumer input reads ($y$), in whic **The order changes how many nodes enumeration visits:** $\sum_d \prod_{d' \le d} \lvert \text{values left at } d' \rvert$. Dimensions that prune much go early; dimensions that multiply without pruning go late. The order above follows four rules: 1. **Logical before physical.** Stage 1 hash-conses groups and computes $\alpha^{\log}$ once, before Stage 2 multiplies each logical candidate by its physical alternatives. -2. **Strong prunes early.** Executor support and the pruning cap act before physical properties multiply the tree. +2. **Strong prunes early.** The pruning cap and dead-candidate elimination act before physical properties multiply the tree. 3. **Dependent dimensions adjacent.** $\tau \to \ell$ rejects the invalid combination at the first level that can detect it. 4. **Deployment-wide prunes as early as possible.** Without an ingestion pipeline the IngestionTime subtree is pruned once per candidate; moving $\tau$ above $\boldsymbol\theta$ would prune it once per group, with the same leaves. @@ -811,7 +813,54 @@ An edge satisfies the logical requirement when the producer meets the input's fl ### Stage 2: physical alternatives -`enumerate_candidate_configurations` in `crates/physical-asap-optimization/src/catalog/configurations.rs` expands each logical candidate that survived Pass 2 using the dimensions and checks in Section 2; it offers ℓ = AcrossEvaluations only to chain candidates, and records every other AcrossEvaluations alternative as pruned with `PruneReason::AcrossEvaluationsOnlyForChains`. It creates no new logical candidates. Each logical edge is expanded between the relevant physical alternatives. +`enumerate_candidate_configurations` in `crates/physical-asap-optimization/src/catalog/configurations.rs` expands each logical candidate that survived Pass 2 using the dimensions and checks in Section 2. It offers ℓ = AcrossEvaluations only to chain candidates; the other candidates never get that value, and nothing is recorded for them. It creates no new logical candidates. Each logical edge is expanded between the relevant physical alternatives. + +Stage 2 reads the logical candidates, the data workload (per-source arrival, below) and the deployment's capabilities: + +```rust +// crates/physical-asap-optimization/src/catalog/mod.rs +pub fn generate_physical_asap_memo_candidates( + logical: &LogicalASAPMemoCandidates, + data: &DataWorkload, + capabilities: &DeploymentCapabilities, +) -> Result; +``` + +#### Data arrival + +```rust +// crates/types/src/workload/mod.rs +pub enum DataArrival { + /// Every row is already stored; no build runs at ingestion time. + AtRest, + /// Rows keep arriving while the queries run. + Ingesting { rows_per_second: f64, max_out_of_order_ms: Option }, + /// Older rows are stored and new rows keep arriving. + Mixed { rows_per_second: f64, max_out_of_order_ms: Option }, + /// Not declared: no build runs at ingestion time. + Unknown, +} +``` + +`max_out_of_order_ms` bounds how late a row may arrive relative to the newest row of its series. + +**Counter states at ingestion time.** A counter state (`rate`, `increase`, `irate`) needs each series' samples in timestamp order to detect resets. It may still run at ingestion time: the ingestion data plane keeps a per-series reorder buffer that holds `max_out_of_order_ms` of rows and releases them in order. The buffer's memory, about rows per second × out-of-order bound × bytes per row, is part of that build's cost. For `rate(c_total[5m])` with 100,000 samples/s, a 2 s bound and 16 B per row, the buffer is about 100,000 × 2 × 16 B ≈ 3.2 MB. When `max_out_of_order_ms` is `None`, a PromQL source is taken to arrive in order (bound 0, no buffer); a SQL source gets no ingestion-time counter build, because SQL rows have no ordering guarantee. + +**Filtered builds at ingestion time.** A filtered build needs no extra rule. α^phys already requires an ingestion-time build's input to be produced at ingestion time; the build reads each input row with all its columns and evaluates the filter before updating the state. For SQL `sum(x) FILTER (WHERE region = 'eu')` over `Join(events, hosts)`, the build runs at ingestion time only when the join does; it then reads each joined row, checks `region`, and adds `x` for the rows that pass. + +#### Prune reasons + +Stage 2 records every alternative it rejects, with one of these reasons; Stage 3 adds its own. + +| Reason | Rejects | +|---|---| +| `NoIngestionPipeline` | IngestionTime when the source has no ingestion pipeline | +| `RawDataNotRetained` | QueryTime over raw rows the deployment does not retain | +| `QueryTimeResultsNotRetained` | QueryTime with ℓ = OneEvaluation or AcrossEvaluations when the deployment retains no query-time results | +| `InputHasNoBuildableProducer` | a candidate for which some input has no surviving producer that satisfies α^phys | +| `ReachesNoQueryOutput` | a candidate from which no admissible path leads to a query root | + +The planner does not check executor support, so no reason names it. ```rust // crates/types/src/physical/properties.rs @@ -876,10 +925,22 @@ pub struct PhysicalEdge { pub struct PhysicalASAPMemoCandidates { pub logical_candidates: LogicalASAPMemoCandidates, pub physical_candidates: Vec, - pub query_roots: Vec>, - pub pruned_candidates: Vec>, + pub pruned_candidates: Vec>, +} +/// A rejected alternative and why; the candidate names its group +/// through its logical candidate, so no separate group field. +pub struct PrunedCandidate { + pub candidate: C, + pub reason: PruneReason, } impl PhysicalASAPMemoCandidates { + /// Query roots are read from `logical_candidates`, not copied. + pub fn new( + logical_candidates: LogicalASAPMemoCandidates, + physical_candidates: Vec, + edges: Vec, + pruned_candidates: Vec>, + ) -> Self; pub fn edges(&self) -> impl Iterator; pub fn admissible_edges_into( &self, @@ -891,15 +952,9 @@ impl PhysicalASAPMemoCandidates { producer: PhysicalCandidateId, ) -> &[PhysicalEdgeId]; } - -// Identifies a rejected alternative by content: it has no allocated candidate ID. -pub struct PhysicalCandidateKey { - pub logical_candidate_id: CandidateForMemoGroupId, - pub physical_properties: PhysicalProperties, -} ``` -`PhysicalCandidateId` indexes an emitted candidate. `PhysicalCandidateKey` describes an alternative rejected before it received an ID, allowing diagnostics to explain pruning. The admissible-edge accessor returns edge IDs, not the unfiltered producer set from Section 1. `delivers_on` takes a candidate and the memo rather than bare properties because a tumbling grid's width is an argument of the chain operator in the candidate's expression. +`PhysicalCandidateId` indexes an emitted candidate. A pruned alternative is stored as its `PhysicalCandidate` value, which has no ID, so diagnostics can explain each prune. The admissible-edge accessor returns edge IDs, not the unfiltered producer set from Section 1. `delivers_on` takes a candidate and the memo rather than bare properties because a tumbling grid's width is an argument of the chain operator in the candidate's expression. Migration targets include replacing `Materialization` and duplicated physical fields with checked properties, and replacing `ReaderConstraint::{IncompatibleOutputs, AllowedOutputs}` with compatibility and alignment checks. `ReaderConstraint::{MinimumQueryReaders, HighestSharedQueryMaterialization}` are deleted too; see Section 5. From 206d84b907963ee290593153801b5fab129fe505 Mon Sep 17 00:00:00 2001 From: zzylol <50204836+zzylol@users.noreply.github.com> Date: Sat, 7 Nov 2026 17:01:09 +0000 Subject: [PATCH 43/44] docs(concepts): Stage 2 entry returns no Result; current crate path Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01K1M6bq8xVaGKyinf4ohK6q --- .../concepts/memo-groups-properties-requirements.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/design_docs/concepts/memo-groups-properties-requirements.md b/docs/design_docs/concepts/memo-groups-properties-requirements.md index 324b084f8..e7d51c814 100644 --- a/docs/design_docs/concepts/memo-groups-properties-requirements.md +++ b/docs/design_docs/concepts/memo-groups-properties-requirements.md @@ -813,17 +813,17 @@ An edge satisfies the logical requirement when the producer meets the input's fl ### Stage 2: physical alternatives -`enumerate_candidate_configurations` in `crates/physical-asap-optimization/src/catalog/configurations.rs` expands each logical candidate that survived Pass 2 using the dimensions and checks in Section 2. It offers ℓ = AcrossEvaluations only to chain candidates; the other candidates never get that value, and nothing is recorded for them. It creates no new logical candidates. Each logical edge is expanded between the relevant physical alternatives. +`enumerate_candidate_configurations` in `crates/stage2-physical-optimization/src/catalog/configurations.rs` expands each logical candidate that survived Pass 2 using the dimensions and checks in Section 2. It offers ℓ = AcrossEvaluations only to chain candidates; the other candidates never get that value, and nothing is recorded for them. It creates no new logical candidates. Each logical edge is expanded between the relevant physical alternatives. Stage 2 reads the logical candidates, the data workload (per-source arrival, below) and the deployment's capabilities: ```rust -// crates/physical-asap-optimization/src/catalog/mod.rs +// crates/stage2-physical-optimization/src/catalog/mod.rs pub fn generate_physical_asap_memo_candidates( logical: &LogicalASAPMemoCandidates, data: &DataWorkload, capabilities: &DeploymentCapabilities, -) -> Result; +) -> PhysicalASAPMemoCandidates; // Stage 2 cannot fail ``` #### Data arrival From 1f0292a11b667465228fe615941e34a12f459fcf Mon Sep 17 00:00:00 2001 From: zzylol <50204836+zzylol@users.noreply.github.com> Date: Sat, 7 Nov 2026 20:50:14 +0000 Subject: [PATCH 44/44] docs(concepts): IntervalConstraintLimit replaces TimeBound, with examples Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01K1M6bq8xVaGKyinf4ohK6q --- .../memo-groups-properties-requirements.md | 44 ++++++++++++------- 1 file changed, 29 insertions(+), 15 deletions(-) diff --git a/docs/design_docs/concepts/memo-groups-properties-requirements.md b/docs/design_docs/concepts/memo-groups-properties-requirements.md index e7d51c814..1992180f9 100644 --- a/docs/design_docs/concepts/memo-groups-properties-requirements.md +++ b/docs/design_docs/concepts/memo-groups-properties-requirements.md @@ -379,25 +379,39 @@ pub struct SelectionBox { pub columns: BTreeMap, } -/// An endpoint of an interval on a timestamp column. -pub enum TimeBound { - /// An absolute time, in ms since the epoch. - At(i64), - /// An offset, in ms, from the evaluation time t. - FromEvaluationTime(i64), +/// A constraint on one column of a selection. +pub enum Constraint { + In(Vec), + NotIn(Vec), + /// Each end is `Included`, `Excluded` or `Unbounded`. + Interval { + lower: Bound, + upper: Bound, + }, +} + +/// An end of an interval. Two limits compare only within one variant. +pub enum IntervalConstraintLimit { + /// A constant: `Float64(10.0)` for `price > 10`; `Timestamp(ms since the + /// epoch)` for `@ T` or a SQL timestamp literal. + FixedValue(ScalarValue), + /// The evaluation time t plus this many ms (negative: before t); only on + /// the timestamp column. + EvaluationTimePlusMs(i64), } ``` -A repeatedly evaluated query covers different absolute times at every evaluation, so its group's coverage is written relative to the evaluation time t; an `@ T` modifier or a SQL timestamp filter is absolute. Both are intervals on the timestamp column, compared and lifted like any other column constraint. For a range evaluation over [t − D, t] (D is the query range's duration): +A repeatedly evaluated query covers different absolute times at every evaluation, so its group's coverage is written relative to the evaluation time t (`EvaluationTimePlusMs`); an `@ T` modifier or a SQL timestamp filter is absolute (`FixedValue(Timestamp)`). Both are intervals on the timestamp column, compared and lifted like any other column constraint; a value column such as `price` uses `FixedValue` ends. For a range evaluation over [t − D, t] (D is the query range's duration): -| Query fragment | Constraint on `m.ts` | -|---|---| -| `m[5m]` | [t − 5m, t): both ends `FromEvaluationTime` | -| `m[5m] offset 1h` | [t − 1h − 5m, t − 1h) | -| `m[5m] @ end()` | [t − 5m, t): the same as no `@` in an instant query | -| `m[5m] @ start()` | [t − D − 5m, t − D): an offset of D | -| `m[5m] @ 1700000000` | [T − 5m, T) with T = 1700000000 s: both ends `At` | -| SQL `WHERE ts >= '2026-11-01'` | [2026-11-01, ∞): `At` | +| Query fragment | `lower` | `upper` | +|---|---|---| +| `m[5m]` | `Included(EvaluationTimePlusMs(-300_000))` | `Excluded(EvaluationTimePlusMs(0))` | +| `m[5m] offset 1h` | `Included(EvaluationTimePlusMs(-3_900_000))` | `Excluded(EvaluationTimePlusMs(-3_600_000))` | +| `m[5m] @ end()` | as `m[5m]`: the same as no `@` in an instant query | | +| `m[5m] @ start()` | `Included(EvaluationTimePlusMs(-D - 300_000))` | `Excluded(EvaluationTimePlusMs(-D))`: an offset of D | +| `m[5m] @ 1700000000` | `Included(FixedValue(Timestamp(1_699_999_700_000)))` | `Excluded(FixedValue(Timestamp(1_700_000_000_000)))` | +| SQL `WHERE ts >= '2026-11-01'` | `Included(FixedValue(Timestamp(..)))` | `Unbounded` | +| SQL `WHERE price > 10` (column `price`) | `Excluded(FixedValue(Float64(10.0)))` | `Unbounded` | **Derivation, per operator** (from the input groups' `column_lineage`):