From 70db0cccbd0bf270a54bc40f1fb5c5dd8c976e37 Mon Sep 17 00:00:00 2001 From: zzylol <50204836+zzylol@users.noreply.github.com> Date: Tue, 20 Oct 2026 15:56:02 +0000 Subject: [PATCH 01/53] docs: design joint planning search with exact pruning --- docs/design_docs/proposals/README.md | 1 + .../proposals/joint-planning-search.md | 139 ++++++++++++++++++ 2 files changed, 140 insertions(+) create mode 100644 docs/design_docs/proposals/joint-planning-search.md diff --git a/docs/design_docs/proposals/README.md b/docs/design_docs/proposals/README.md index 6c4ed49f8..1c10845d3 100644 --- a/docs/design_docs/proposals/README.md +++ b/docs/design_docs/proposals/README.md @@ -11,4 +11,5 @@ extensions. A design document is not a promise of downstream runtime support. - [Operator sharing](operator-sharing.md) - [Decoupling operators from scalar expressions](decoupling_op_and_expr.md) - [ASAPPlanner layering](planner-layering.md) +- [Joint planning search](joint-planning-search.md) - [Schema and physical data for ASAP primitives](asap-primitive-schema.md) diff --git a/docs/design_docs/proposals/joint-planning-search.md b/docs/design_docs/proposals/joint-planning-search.md new file mode 100644 index 000000000..f0e886188 --- /dev/null +++ b/docs/design_docs/proposals/joint-planning-search.md @@ -0,0 +1,139 @@ +# Joint Planning Search: Pruning the Whole-Workload Candidate Space + +This document designs how ASAPPlanner selects one plan for a whole workload without building every whole-workload candidate. It builds on the planning stages in [planner-layering.md](planner-layering.md) and changes how Stage 1 to Stage 3 search, not what they decide. + +## Contents + +- [Problem](#problem) +- [Goals and non-goals](#goals-and-non-goals) +- [Where the candidates come from](#where-the-candidates-come-from) +- [Design](#design) + - [1. Independent components](#1-independent-components) + - [2. Per-query validity before composition](#2-per-query-validity-before-composition) + - [3. Lower-bound dominance](#3-lower-bound-dominance) + - [4. Local materialization choice](#4-local-materialization-choice) + - [5. Sharing that never costs more](#5-sharing-that-never-costs-more) + - [6. Streaming selection and a search budget](#6-streaming-selection-and-a-search-budget) +- [Cost model 3](#cost-model-3) +- [What the selection report shows](#what-the-selection-report-shows) +- [Validation](#validation) +- [Relation to other search approaches](#relation-to-other-search-approaches) + +## Problem + +Planning a workload jointly lets queries share computation: one summary answers several queries, identical sub-expressions run once, panes are reused across evaluations. Today the planner finds the best joint plan by enumerating every whole-workload candidate and pricing each one, and it keeps all of them in memory. + +That does not scale past a handful of queries. Planning a growing prefix of one 10-query workload of ASAPQuery's random mixed workloads: + +| Queries | Time | Peak memory | +| ---: | ---: | ---: | +| 3 | 0.4 s | 0.14 GB | +| 4 | 5 s | 1.5 GB | +| 5 | 129 s | 24.7 GB | +| 6 | stopped at ~228 s | ~100 GB | + +Each added query multiplies the work by roughly 20. None of the 30 workloads of that set plan jointly, and neither does the Kubernetes recording-rule set. Planned one query at a time, every query plans in milliseconds, but sharing across queries is lost. + +## Goals and non-goals + +**Goals** + +- Joint planning of realistic workloads (tens of queries) in seconds and under a few GB. +- **The same answer as exhaustive search.** Every pruning step below is exact under an additive cost: it only discards candidates that cannot be selected. The selected plan's cost equals the exhaustive search's. +- When a search must stop early, it says so. Truncation is never silent. + +**Non-goals** + +- Changing what the stages decide: candidate generation rules, sharing rules, materializations, accuracy and cost checks stay as they are. +- A solver-based formulation (MILP). That remains an option; see the last section. +- Re-planning or incremental planning when the workload changes. + +## Where the candidates come from + +The number of whole-workload candidates is the product of three factors: + +| Stage | Factor | Grows as | +| --- | --- | --- | +| Stage 1, Pass 1 | One per-computation candidate per target, combined across targets | product of the targets' candidate counts | +| Stage 1, Pass 2 | Sharing alternatives of each composed candidate: identical expressions, summary capability, window composition, rollup, top-k reuse | subsets of the sharing opportunities | +| Stage 2 | One materialization per summary build: at ingestion, at query time, at query time kept across evaluations | three per build | + +Most of these candidates can never win. Either they lose to another candidate of the same query regardless of sharing, or their choices in one part of the workload do not affect any other part. + +## Design + +Six steps, applied in this order. Steps 1 to 5 are exact. Step 6 bounds the work and reports when it does. + +### 1. Independent components + +Two queries interact only through sharing. A Pass 2 rule can link them only if they read the same source and have a sharing opportunity between them: an identical sub-expression, a summary able to answer both, a rollup, or a top-k reuse. + +- Build a graph with one vertex per query and an edge for each possible sharing opportunity. +- Plan each connected component on its own. +- Because cost is additive across queries that share nothing, the workload's best plan is the union of each component's best plan. + +A workload over many metrics, such as a set of recording rules, splits into many small components. ASAPQuery's mixed workloads read two metrics, so they split into at most two components per workload. + +### 2. Per-query validity before composition + +Some checks depend on one query's candidate only: + +- whether the deployment supports the candidate's summaries; +- whether the executor can run it; +- whether its accuracy guarantee, or the deployment's evidence, meets the query's target. + +These run on each query's candidates before they are combined. Today they run in Stage 3, after composition, so invalid candidates still multiply the product. + +### 3. Lower-bound dominance + +For each candidate *c* of query *q*: + +- **alone(c):** the cost of *c* planned by itself, with its best materialization. +- **shareable(c):** an upper bound on what sharing can save *c*. This is the cost of the parts of *c* that another query in the component could share. + +No joint plan containing *c* costs *q* less than **alone(c) − shareable(c)**. So *c* is discarded when + +> alone(c) − shareable(c) > min over the other candidates c′ of q of alone(c′). + +This also decides which sharing candidates are kept: a candidate whose sharing could still make it win stays, and every other candidate goes. A candidate with nothing to share has shareable(c) = 0, so only the query's cheapest candidates survive. + +### 4. Local materialization choice + +A summary build that no other query reads affects only its own query's cost. Its materialization is chosen locally: the cheapest of the three. Only builds read by several queries are chosen jointly. The Stage 2 factor shrinks from three per build to three per shared build. + +### 5. Sharing that never costs more + +Running an identical sub-expression once is never more expensive than running it twice. So the identical-expression rule is always applied, and its subsets are not enumerated. + +Two kinds of sharing can cost more, and keep both alternatives: + +- **Summary capability.** A shared summary is sized for its strictest reader, which can make it larger than either reader's own summary. +- **Rollup.** Answering a coarse query from a finer summary can cost more than a summary of its own. + +### 6. Streaming selection and a search budget + +- **Streaming.** Candidates are priced as they are generated, and only the best so far is kept. For cost model 3, the Pareto front is kept instead. The full candidate lists are kept only when a developer tool asks for them. +- **Budget.** A budget on candidates priced, or on time, guards against a component that is still too large after steps 1 to 5. +- **When the budget runs out:** selection returns the best plan found so far and marks the result as truncated, with how much of the space was searched. The default budget is large, so workloads that fit are never truncated. + +## Cost model 3 + +Cost model 3 (the deployment's own cluster) is not additive: memory, disk and cores are hard limits, and the objective chooses among the candidates that fit. Steps 1 to 6 apply with these changes: + +- **Step 1:** each component keeps its Pareto front over memory, disk and latency, not a single best plan. The workload's plans are the combinations of component fronts that fit the limits. +- **Step 3:** a candidate is dominated only if another candidate of the same query is at least as good on every dimension, after the sharing bound. + +## What the selection report shows + +- The components found. +- For each query: how many candidates steps 2 and 3 discarded, and why. +- For each component: how many candidates were priced, and whether the search was truncated. + +## Validation + +- **Equivalence.** On the planner-layering examples, and on randomly generated workloads small enough to search exhaustively, the selected plan's cost equals the exhaustive search's. Each step is tested on its own. +- **Scale.** Every workload of ASAPQuery's random mixed workloads, and the Kubernetes recording-rule set, plan jointly. The target is seconds and under 1 GB each. The report records time and peak memory. + +## Relation to other search approaches + +A solver-based selection (MILP over candidate, sharing and materialization choices) could replace steps 3 to 6 later. Steps 1 and 2 would still apply, because they shrink the problem any solver receives. This design keeps the enumeration's exact semantics and makes it scale first. From a2db92219ccfc8b00c9210429f0a6456beefef08 Mon Sep 17 00:00:00 2001 From: zzylol <50204836+zzylol@users.noreply.github.com> Date: Tue, 20 Oct 2026 18:04:47 +0000 Subject: [PATCH 02/53] docs: specify exact joint-search pruning conditions and heuristic boundaries --- .../proposals/joint-planning-search.md | 364 ++++++++++++++---- 1 file changed, 280 insertions(+), 84 deletions(-) diff --git a/docs/design_docs/proposals/joint-planning-search.md b/docs/design_docs/proposals/joint-planning-search.md index f0e886188..c30700a79 100644 --- a/docs/design_docs/proposals/joint-planning-search.md +++ b/docs/design_docs/proposals/joint-planning-search.md @@ -1,139 +1,335 @@ -# Joint Planning Search: Pruning the Whole-Workload Candidate Space +# Joint planning search: exact reductions, bounds, and heuristic fallbacks -This document designs how ASAPPlanner selects one plan for a whole workload without building every whole-workload candidate. It builds on the planning stages in [planner-layering.md](planner-layering.md) and changes how Stage 1 to Stage 3 search, not what they decide. +Status: proposal; no new pruning rule is implemented by this document. Audience: planner designers and implementers. -## Contents +ASAPPlanner must search a workload's candidate plans without retaining or pricing every combination. The design keeps the decisions in [planner-layering.md](planner-layering.md), and changes how those decisions are explored. It incorporates the [optimizer survey in #671](https://github.com/ProjectASAP/ASAPPlanner/issues/671#issuecomment-6137638876) and revises the original exactness claims in [#776](https://github.com/ProjectASAP/ASAPPlanner/pull/776). + +**Decision:** build a streaming, memoized branch-and-bound search with conservative interaction analysis. Enable a pruning rule only when its preconditions establish that it preserves the optimum. Use greedy search to find incumbents and order exploration; discarding alternatives by a heuristic is a separately reported approximation mode. A memo, dynamic programming, or a MILP solver does not by itself make workload sharing exact. -- [Problem](#problem) -- [Goals and non-goals](#goals-and-non-goals) -- [Where the candidates come from](#where-the-candidates-come-from) -- [Design](#design) - - [1. Independent components](#1-independent-components) - - [2. Per-query validity before composition](#2-per-query-validity-before-composition) - - [3. Lower-bound dominance](#3-lower-bound-dominance) - - [4. Local materialization choice](#4-local-materialization-choice) - - [5. Sharing that never costs more](#5-sharing-that-never-costs-more) - - [6. Streaming selection and a search budget](#6-streaming-selection-and-a-search-budget) -- [Cost model 3](#cost-model-3) -- [What the selection report shows](#what-the-selection-report-shows) -- [Validation](#validation) -- [Relation to other search approaches](#relation-to-other-search-approaches) +## Contents -## Problem +- [Problem and scope of the guarantee](#problem-and-scope-of-the-guarantee) +- [Lessons from the survey](#lessons-from-the-survey) +- [Review of the original six rules](#review-of-the-original-six-rules) +- [Search representation and interaction analysis](#search-representation-and-interaction-analysis) +- [Exact pruning rules and their proof conditions](#exact-pruning-rules-and-their-proof-conditions) +- [Bounds and branch-and-bound](#bounds-and-branch-and-bound) +- [Composition under the three cost models](#composition-under-the-three-cost-models) +- [Heuristics and what they give up](#heuristics-and-what-they-give-up) +- [Search procedure and stopping contract](#search-procedure-and-stopping-contract) +- [MILP as a relaxation and an alternative search](#milp-as-a-relaxation-and-an-alternative-search) +- [Validation and implementation order](#validation-and-implementation-order) +- [References](#references) -Planning a workload jointly lets queries share computation: one summary answers several queries, identical sub-expressions run once, panes are reused across evaluations. Today the planner finds the best joint plan by enumerating every whole-workload candidate and pricing each one, and it keeps all of them in memory. +## Problem and scope of the guarantee -That does not scale past a handful of queries. Planning a growing prefix of one 10-query workload of ASAPQuery's random mixed workloads: +The original motivating measurement was a growing prefix of one 10-query ASAPQuery mixed workload: -| Queries | Time | Peak memory | +| Queries | Planning time | Peak memory | | ---: | ---: | ---: | | 3 | 0.4 s | 0.14 GB | | 4 | 5 s | 1.5 GB | | 5 | 129 s | 24.7 GB | -| 6 | stopped at ~228 s | ~100 GB | +| 6 | stopped around 228 s | around 100 GB | -Each added query multiplies the work by roughly 20. None of the 30 workloads of that set plan jointly, and neither does the Kubernetes recording-rule set. Planned one query at a time, every query plans in milliseconds, but sharing across queries is lost. +These are motivating observations, not performance guarantees for this proposal. The Cartesian product of per-target alternatives, subsets of sharing opportunities, and materialization assignments makes enumeration exponential. Streaming reduces retained candidates; it does not remove that time complexity. No exact algorithm here promises seconds on every workload. -## Goals and non-goals +Let `F` be the finite set of complete plans the reference generator admits for a fixed workload, rule inventory, parameter inventory, deployment capabilities, and evidence snapshot. For scalar minimization, exact search returns a feasible `P*` with `C(P*) = min(C(P) for P in F)`. For a Pareto request, it returns the complete nondominated objective set, with explicitly defined handling of equivalent plans. -**Goals** +This guarantee has three separate boundaries: -- Joint planning of realistic workloads (tens of queries) in seconds and under a few GB. -- **The same answer as exhaustive search.** Every pruning step below is exact under an additive cost: it only discards candidates that cannot be selected. The selected plan's cost equals the exhaustive search's. -- When a search must stop early, it says so. Truncation is never silent. +- **Search completeness:** no plan in `F` that could improve the requested result is lost. It says nothing about rewrites, sketch parameters, or window realizations absent from `F`. +- **Model optimality:** `C` is the configured planner objective. An exact search can still choose a slow runtime plan when its cost or cardinality model is inaccurate. Empirical estimates are inputs to that model, not measured-runtime guarantees. +- **Validity:** accepted plans must satisfy the existing semantic, output-accuracy, capability, and executability contracts. An empirical accuracy acceptance is exact relative to that declared evidence policy, not a new theoretical error guarantee. -**Non-goals** +Fix query recurrence, time ranges, pricing, measured costs, accuracy evidence, and tie handling for a search. A changed model invalidates cached costs and proof certificates. Nonfinite or unknown estimates never justify pruning. Unsupported cost models fall back to a weaker bound, including zero for a nonnegative scalar objective. -- Changing what the stages decide: candidate generation rules, sharing rules, materializations, accuracy and cost checks stay as they are. -- A solver-based formulation (MILP). That remains an option; see the last section. -- Re-planning or incremental planning when the workload changes. +**Tie contract:** preserving optimal cost need not preserve the same candidate ID. Define a deterministic total ranking for the new search, with stable reference rank for exact ties. The legacy relative-tolerance comparison is not transitive; stable IDs alone cannot reproduce its sequential winner after reordering. If legacy winner identity is required, replay the reference-order comparison for all potentially winning near-tie candidates, or keep that path exhaustive until an equivalent rule is proved. A Pareto report must say whether it preserves all equivalent plan identities or only one representative per objective vector. Changing the tie policy is a compatibility decision, not an exact-search optimization. -## Where the candidates come from +## Lessons from the survey + +| Approach | Useful idea for ASAPPlanner | Exactness boundary | +| --- | --- | --- | +| System R-style DP; DPccp/DPhyp [1] | Memoize subproblems; exploit sparse interaction graphs; retain properties a parent needs. | A single cheapest subplan is safe only with optimal substructure. DPccp's cross-product-free join space is a restriction, not proof that arbitrary workload sharing is separable. | +| Volcano/Cascades [2] | Groups of equivalent computations, required physical properties, cost limits, and memoized exploration. | A memo shares optimizer work. It does not automatically decide runtime sharing, materialization, or a correct state key. | +| MQO and view selection [3, 4] | Model build-once/read-many decisions jointly; use greedy materialization as a practical incumbent. | Greedy MQO is heuristic. Approximation guarantees for a particular data-cube lattice or benefit function do not transfer to arbitrary ASAP DAGs. | +| MILP [5] | Explicit coupled decisions, relaxations as lower bounds, solver incumbents and gaps. | Exact only for the encoded choices and objective. Linear cost approximations and omitted sharing modes can change the optimum. | +| Rule passes and bounded/randomized search | Cheap simplification, branch ordering, and a predictable planning budget. | Restricting exploration is heuristic unless omitted alternatives are proved dominated. PostgreSQL explicitly documents this tradeoff for GEQO [6]. | -The number of whole-workload candidates is the product of three factors: +The survey's `global_selection` discussion describes an earlier implementation, not an API this design assumes still exists. The proposal branch's current stage pipeline is documented in [`stage1_to_stage3.rs`](../../../crates/plan-selection/src/pass/stage1_to_stage3.rs); sharing rules in [`asap_aware_cse.rs`](../../../crates/logical-asap-optimization/src/asap_aware_cse.rs); and resource aggregation and ranking in [`plan-selection/src/lib.rs`](../../../crates/plan-selection/src/lib.rs). Audit those contracts when implementing each rule. The architecture below is an ASAPPlanner design derived from the survey, not a guarantee supplied by those papers. -| Stage | Factor | Grows as | +## Review of the original six rules + +| Original rule | Correct classification | Required change | | --- | --- | --- | -| Stage 1, Pass 1 | One per-computation candidate per target, combined across targets | product of the targets' candidate counts | -| Stage 1, Pass 2 | Sharing alternatives of each composed candidate: identical expressions, summary capability, window composition, rollup, top-k reuse | subsets of the sharing opportunities | -| Stage 2 | One materialization per summary build: at ingestion, at query time, at query time kept across evaluations | three per build | +| Split queries by possible sharing | Conditionally exact. | Include every possible sharing/reuse edge and shared resource-accounting factor. Independent computation alone is insufficient with global budgets. | +| Check each query's validity early | Exact for a rejection invariant under every remaining completion. | Defer failures that resizing, replacement, sharing, or later lowering can repair. Missing evidence for a provisional expression is not such a proof. | +| Prune if `alone(c) - shareable(c) > best_alone(q)` | **Not generally exact**, even with additive operator costs. | A saving on one query is not a bound on the effect on all consumers. Replace with a workload branch bound or a completion-preserving replacement proof. | +| Choose the cheapest materialization for unshared builds | Conditionally exact. | Prove isolation from other choices, including ancestors, shared inputs, memory/disk budgets, recurrence, and output lifetime. Otherwise keep a frontier. | +| Always apply identical-expression sharing | Semantically safe only through the existing sharing legality checks; cost dominance is conditional. | Distinguish canonical representation from executing once. Retention, parallelism, copying, synchronization, and physical properties can favor recomputation. | +| Stream candidates and stop at a budget | Streaming is exact if enumeration and selection are equivalent. Early stopping is not generally exact. | Keep unresolved work represented in the lower bound; report a feasible incumbent, gap, and completion status. | + +### Why the standalone subtraction rule is unsafe + +Consider two queries. Query A can choose candidate `c`, containing a shareable producer costing 10 and private work costing 6, or candidate `a`, costing 5. Query B alone costs 100. A permitted cross-query rewrite can answer B from `c`'s producer with 1 unit of adapter work; the producer is offered through `c`, and that rewrite is lost if `c` is removed. + +The old test uses `alone(c) = 16`, `shareable(c) = 10`, and `best_alone(A) = 5`. It discards `c` because `16 - 10 > 5`. Yet the joint plan using `c` costs `10 + 6 + 1 = 17`, versus `5 + 100 = 105` without that opportunity. The missing quantity is the benefit to B, not the part of A's own bill it might avoid. A reference generator that independently offers the producer even after replacing A might admit a different repair; that repair must be demonstrated, not assumed. Rollup rules that only reuse an already present finer computation make this distinction relevant. + +Even without this donor effect, whole-workload memory constraints can invalidate a standalone replacement that uses more memory. More generally, marginal cost cannot be assigned independently to each query when a shared build is paid once. + +## Search representation and interaction analysis + +### Memo groups and boundary state + +A memo group represents a computation with the same observable semantics. Its alternatives describe exact or summary implementations, their output guarantees, and legal physical realizations. Similar source names, matching hashes, or equal output column names do not establish equivalence. + +A search state records the unresolved choices and the interface through which already chosen work can affect them. Include, when observable: + +- source identity, predicates, grouping, value/weight semantics, schema and row identity; +- summary family, parameters, composed output guarantees, and consumer requirements; +- evaluation frequencies, aligned windows, offsets, pane layout, and retention horizons; +- ingestion/query phase, retained versus temporary output, storage medium, and required physical properties; +- offered shared producers and consumer bindings, their adapters, and which costs are already counted; +- boundary resource demands and active source-retention horizons needed for later composition. + +Consumers with different accuracy targets may need a shared producer sized for all of them. Sharing opportunities can be partial, overlap, or arise after an earlier rewrite. The state cannot be just a query ID, a count of consumers, or a scalar cost. + +Two states may reuse a cached optimum only when every future completion observes the same interface and the same cost/evidence context. Hashes only find possible matches; verify structural and semantic equality. A bounded cache may evict entries and recompute later without losing exactness; a bounded beam discards search alternatives and does lose it. + +### Conservative interaction hypergraph + +Represent decisions as variables, and interactions as factors/hyperedges. Examples are a producer and all its potential readers, a summary shared by three consumers, a raw source's retention charge, and a workload resource constraint. Pairwise projection is sufficient to find connected components if it conservatively includes all members of every hyperedge. + +Build potential sharing from **all alternatives and the registered rewrite closure**, not from the independent winner of each query. Include identical expression, capability, rollup, top-k, window/pane reuse, and within-query dependencies. A common metric name is neither necessary nor sufficient: aliases can name one source, and different predicates can prevent sharing. + +Each sharing rule must expose a conservative summary of the sources, producers, consumers, and properties it might connect. If the analyzer cannot rule out an interaction, keep the decisions connected. New rules default to “may interact” until their interaction summary is implemented. False positives cost planning time; false negatives invalidate exactness. + +Whole-workload factors can sometimes be composed after solving computational components, but are not ignored. For example, raw disk retention is the **maximum** requested horizon per source in the current cost model, not the sum of each query's standalone disk use. Preserve that source interface or join its components. The analyzer must be conservative over transformations that introduce a new compatible window or producer, not merely direct matches in the initial graph. + +## Exact pruning rules and their proof conditions + +Here “exact” means the stated conditions are satisfied. It is a correctness property, not a promise that a rule removes many candidates or makes the workload fast. + +The safest first implementation keeps exhaustive generation semantics and enables only duplicate elimination, completed-plan rejection, and an initially zero lower bound. Those preserve the result without assuming useful sharing savings. Component splitting, local DP, and forced sharing require additional certificates and remain disabled when their preconditions are unknown. Streaming is an execution strategy, not a new pruning theorem. + +### E1. Remove duplicate search states and equivalent complete plans + +If two states have the same full interface, remaining legal choices, and resource/cost obligations, expand one and reuse its result for the other. Canonical node IDs or traversal order must not affect identity. Preserve alias/provenance information required by explanations and tie handling. + +**Proof:** each discarded state has exactly the same completions and costs as its representative. This avoids repeated optimizer work without forcing shared runtime execution. Registering a legal common expression in a memo is different from deciding to materialize it once. + +### E2. Reject permanently invalid branches + +Prune when no remaining completion can meet semantics, deployment capability, executability, or the query's output-accuracy policy. Examples are a fixed operator with no supported implementation and no remaining replacement, or a fixed unrepairable type error. A fixed raw-data requirement with raw retention forbidden is another example, provided no unresolved materialization can remove the query-time read. + +Per-query checks may run early only for this invariant subset. Otherwise carry obligations forward: shared resizing can improve accuracy; a different family can supply evidence; ingestion materialization can remove a raw read; later lowering can change an operator shape. Recheck final shared output guarantees and executor support on completed plans. + +Constraint propagation is another exact case: remove a domain value only after proving it has no support in a required constraint, even in a relaxation of the other decisions. An empty domain then rejects the branch. Pairwise arc consistency can be a cheap necessary check, but passing it does not prove that a globally feasible completion exists. + +**Proof:** the branch's feasible completion set is empty. Treating absent evidence as rejection of a completed plan can follow the existing policy; treating it as permanent rejection before all relevant transformations is a different and generally unsafe step. + +### E3. Prune by an admissible workload lower bound + +For a partial state `S`, compute `LB(S)` such that `LB(S) <= min C(P)` over its feasible completions. If a fully validated incumbent has objective `U`, discard `S` when its bound cannot beat the incumbent under the chosen tie rule. The default conservative scalar comparison is strict `LB(S) > U`, with numerical margins described below. -Most of these candidates can never win. Either they lose to another candidate of the same query regardless of sharing, or their choices in one part of the workload do not affect any other part. +A currently composed partial plan's cost is **not** automatically a lower bound: later sharing can remove work, and later rewrites can change the physical implementation. Only irrevocable, nonnegative charges can be accumulated without a proof of the remainder. The next section defines progressively stronger relaxations. -## Design +**Proof:** every completion costs at least the bound, so none improves the incumbent. Prune also when a conservative lower bound on a hard resource exceeds its limit. A sum that double-counts possibly shared memory or CPU is not a valid resource lower bound. -Six steps, applied in this order. Steps 1 to 5 are exact. Step 6 bounds the work and reports when it does. +### E4. Replace a state by a contextually dominating state -### 1. Independent components +For every completion of state B, there must be a feasible completion of A that preserves each query's output contract, exposes compatible boundary producers/properties, and is no worse on the requested objective and constrained resources. Then B can be removed. In a complete-plan comparison there is no unresolved context; use the actual whole-plan ranking directly. -Two queries interact only through sharing. A Pass 2 rule can link them only if they read the same source and have a sharing opportunity between them: an identical sub-expression, a summary able to answer both, a rollup, or a top-k reuse. +A practical sufficient certificate for partial states is the same boundary interface and a resource/cost vector no larger in every coordinate consumed by the monotone composition function. Keep alternatives with different useful orderings, summary capabilities, source horizons, recurrence/retention requirements, or accuracy guarantees unless the substitution proof covers those differences. A cheaper standalone candidate with fewer reusable outputs is not a dominating state. -- Build a graph with one vertex per query and an edge for each possible sharing opportunity. -- Plan each connected component on its own. -- Because cost is additive across queries that share nothing, the workload's best plan is the union of each component's best plan. +**Proof:** mapping every potentially winning completion of B to a no-worse completion of A preserves the optimum. Equal vectors are deduplicable only under the declared representative/tie policy. Context-free per-query Pareto pruning is not justified just because several metrics are recorded. -A workload over many metrics, such as a set of recording rules, splits into many small components. ASAPQuery's mixed workloads read two metrics, so they split into at most two components per workload. +### E5. Solve truly independent components separately -### 2. Per-query validity before composition +If `F = F1 × ... × Fk`, feasibility factors, and `C(P) = sum Ci(Pi)`, choose a scalar optimum independently in each component. Potential interactions must have been analyzed conservatively as above. -Some checks depend on one query's candidate only: +When feasibility is globally coupled, keep sufficient component alternatives and combine them through a global resource DP or search. A scalar cheapest plan per component can be infeasible in combination. If source-retention horizons or other properties cross the boundary, retain those properties or merge the components before applying the rule. -- whether the deployment supports the candidate's summaries; -- whether the executor can run it; -- whether its accuracy guarantee, or the deployment's evidence, meets the query's target. +**Proof:** the product of independent local optima minimizes the separable sum. With constraints, the same proof applies only after conditioning on a sufficient resource/boundary assignment. Merely reading different metrics does not prove independent capacity constraints. -These run on each query's candidates before they are combined. Today they run in Stage 3, after composition, so invalid candidates still multiply the product. +### E6. Eliminate local materialization choices only behind a complete interface -### 3. Lower-bound dominance +An unshared build may still share a raw scan, overlap ancestor working memory, change required raw retention, or affect the latency path. Fix its boundary inputs, outputs, lifetimes, and physical requirements first. With no remaining interactions and a scalar separable objective, choose the cheapest legal local realization. With resource tradeoffs or an observable output property, retain the nondominated alternatives for that interface. -For each candidate *c* of query *q*: +For example, local option A costs 1 and uses 10 memory units, while B costs 2 and uses 1. Under a memory limit of 2, discarding B for A is incorrect even when no other query reads this build. A subtree DP is exact only after all externally visible dependencies are represented in its state. -- **alone(c):** the cost of *c* planned by itself, with its best materialization. -- **shareable(c):** an upper bound on what sharing can save *c*. This is the cost of the parts of *c* that another query in the component could share. +### E7. Force identical-expression sharing only with a physical dominance proof -No joint plan containing *c* costs *q* less than **alone(c) − shareable(c)**. So *c* is discarded when +The existing [`ir/cse` legality checks](../../../crates/types/src/ir/cse.rs) establish which expressions may share without changing results; structural equality alone is insufficient for some unkeyed outputs. They do not prove that every shared physical realization is cheaper. -> alone(c) − shareable(c) > min over the other candidates c′ of q of alone(c′). +A sufficient restricted case is a pure deterministic computation with the same inputs, evaluation domain, schema, precision, and required properties; executing it once preserves consumer scheduling, and every relevant build/read/storage/latency/resource charge is no larger. No extra retention, copying, synchronization, or incompatible placement may be introduced. Under that cost model, remove duplicate execution choices. Otherwise retain shared and recomputed alternatives, while still memoizing their common semantics. -This also decides which sharing candidates are kept: a candidate whose sharing could still make it win stays, and every other candidate goes. A candidate with nothing to share has shareable(c) = 0, so only the query's cheapest candidates survive. +**Counterexample:** two evaluations can each compute a temporary result in memory 1, while a shared retained realization holds that result across a much longer horizon or must span multiple time slices. Saving CPU does not dominate retained memory. Different media, concurrency, and consumer schedules also require comparison. -### 4. Local materialization choice +### E8. Condition on a separator, then optimize the remainder -A summary build that no other query reads affects only its own query's cost. Its materialization is chosen locally: the cheapest of the three. Only builds read by several queries are chosen jointly. The Stage 2 factor shrinks from three per build to three per shared build. +The survey's “enumerate shared nodes, then DP the rest” is exact when the separator fixes **all** coupling: producer choices, consumer bindings, sizes/guarantees, materialization, windows/retention, shared raw charges, and relevant resource interfaces. Given that assignment, solve conditionally independent residual subproblems and compose their resource profiles. -### 5. Sharing that never costs more +Keeping only a shared node's presence bit is insufficient. Use variable elimination or tree-decomposition DP with full separator states. Complexity depends on domain sizes and induced width of the entire interaction graph, not simply the number of shared nodes or queries. Global resource constraints can enlarge the interface. This is an optional second search engine after conservative branch-and-bound, not a prerequisite for the first implementation. -Running an identical sub-expression once is never more expensive than running it twice. So the identical-expression rule is always applied, and its subsets are not enumerated. +## Bounds and branch-and-bound -Two kinds of sharing can cost more, and keep both alternatives: +### Safe bounds before sophisticated costing -- **Summary capability.** A shared summary is sized for its strictest reader, which can make it larger than either reader's own summary. -- **Rollup.** Answering a coarse query from a finer summary can cost more than a summary of its own. +Start with bounds whose validity is easy to audit: -### 6. Streaming selection and a search budget +1. **Zero:** valid for the configured nonnegative scalar objective. Weak but safe; reject negative or undefined inputs rather than assuming positivity. +2. **Mandatory exclusive work:** charge a proven lower bound for operations that every completion must execute and cannot share or replace. Charge each distinct operation once. Set potentially shareable producers, adapters with unknown implementations, and unresolved materializations to zero. +3. **Independent relaxed subproblems:** drop selected coupling constraints, allow conflicting choices to coexist, and optimize the remaining relaxed domains. The relaxation must contain every legal completion. Only sum terms whose accounting prevents double-counting; a sum of standalone query optima generally does not meet that requirement. +4. **LP relaxation:** encode candidate/build/binding choices and valid cost/resource under-estimators, then relax integer decisions to continuous ones. The relaxation optimum is a bound on the encoded exact objective when every original feasible plan maps into the relaxed model. A surrogate cost with no one-sided relation supplies no certified bound. -- **Streaming.** Candidates are priced as they are generated, and only the best so far is kept. For cost model 3, the Pareto front is kept instead. The full candidate lists are kept only when a developer tool asks for them. -- **Budget.** A budget on candidates priced, or on time, guards against a component that is still too large after steps 1 to 5. -- **When the budget runs out:** selection returns the best plan found so far and marks the result as truncated, with how much of the space was searched. The default budget is large, so workloads that fit are never truncated. +If a “sharing saving” is used, it must upper-bound the **total workload** saving under all completions, include benefits to other consumers, and count overlapping savings conservatively. That proof is harder than using an explicit relaxation. Do not implement the original per-query subtraction as an exact shortcut. -## Cost model 3 +An admissible bound need not be consistent between parent and child states for branch-and-bound correctness. A weak or zero bound can mean exhaustive exploration. Prefer a cheaply evaluated bound unless stronger pruning pays for its own planning overhead. -Cost model 3 (the deployment's own cluster) is not additive: memory, disk and cores are hard limits, and the objective chooses among the candidates that fit. Steps 1 to 6 apply with these changes: +### Incumbents and resource bounds -- **Step 1:** each component keeps its Pareto front over memory, disk and latency, not a single best plan. The workload's plans are the combinations of component fronts that fit the limits. -- **Step 3:** a candidate is dominated only if another candidate of the same query is at least as good on every dimension, after the sharing bound. +Construct an initial feasible plan using standalone choices, then try compatible sharing and local improvement. A heuristic is useful here: each validated plan is a legitimate upper bound, even if it is far from optimal. If standalone choices violate a global budget, they are not an incumbent; search for a feasible one and report none if no plan has yet been found. -## What the selection report shows +For cluster constraints, derive lower bounds on peak memory, disk, and cores using the same accounting contracts as the final model. Drop positive terms or relax dependencies when safe; do not add mutually exclusive configurations. A latency bound can come from an unavoidable dependency path with lower-bounded operator costs, not from summing work that can run in parallel. -- The components found. -- For each query: how many candidates steps 2 and 3 discarded, and why. -- For each component: how many candidates were priced, and whether the search was truncated. +For a requested Pareto front, an incumbent front may prune a branch only when every completion is dominated under the final strict-dominance/equality policy. A componentwise lower-bound vector helps: a feasible incumbent strictly dominating that bound dominates all completions under monotone aggregation. Equality is insufficient if equal-vector plan identities must be returned. A bound gap on one scalar objective does not certify an entire Pareto front. -## Validation +### Numerical comparisons and ties -- **Equivalence.** On the planner-layering examples, and on randomly generated workloads small enough to search exhaustively, the selected plan's cost equals the exhaustive search's. Each step is tested on its own. -- **Scale.** Every workload of ASAPQuery's random mixed workloads, and the Kubernetes recording-rule set, plan jointly. The target is seconds and under 1 GB each. The report records time and peak memory. +Current scalar ranking uses relative floating-point tolerance and retains an earlier candidate on a tie. Honor the tie contract above; do not let a different summation order create artificial dominance. Lower-bound arithmetic must round conservatively downward, or carry a justified error interval and prune only when its lower endpoint is strictly worse than the incumbent's upper endpoint, allowing for the ranking tolerance. An arbitrary epsilon is not a proof of numerical safety. Near ties and unknown bounds stay in the search. Stable ranks suffice for exact ties under a total ordering; the legacy tolerance case requires the additional compatibility handling described above. -## Relation to other search approaches +## Composition under the three cost models -A solver-based selection (MILP over candidate, sharing and materialization choices) could replace steps 3 to 6 later. Steps 1 and 2 would still apply, because they shrink the problem any solver receives. This design keeps the enumeration's exact semantics and makes it scale first. +The implementation's `ResourceUsage` is the authority; cost model numbers are not evidence of additivity. In particular, model 1 can include a shared raw-retention disk charge and model 2 explicitly prices peaks. + +| Model | Final choice | State needed for safe composition | +| --- | --- | --- | +| 1: `UsageOverTime` | Price CPU, scans, average memory, and configured disk use. | Charge each physical producer and execution frequency once; preserve per-source raw-retention horizons and other nonadditive accounting. Scalar component optima require proven separability, including the active prices. | +| 2: `ProvisionedCapacity` | Price peak memory, peak cores, and disk. | Keep peak/resource alternatives. Follow the current model's overlap assumptions; summing standalone price is not generally valid. | +| 3: `ClusterCapacity` | Reject memory/disk/core violations; choose lowest latency, least memory, or a Pareto front. | Carry constrained dimensions including **cores**, objective tie dimensions, and any boundary source/physical properties needed by composition. A front only over memory, disk, and latency can discard the sole core-feasible plan. | + +A vector is sufficient only with a proved monotone composition rule for that interface. Under the current conservative all-evaluations-overlap memory model, some memory terms add. Disk retention combines by maximum per source; other retained state terms add. Worst-query latency combines by maximum for independent query groups, but changing a shared producer changes consumer paths. If a later model uses actual scheduling, an aggregate peak number may no longer suffice; retain a lifetime profile or disable that reduction. + +For truly independent components with additive resource coordinates, combine their frontiers by resource DP (or exact enumeration) and discard dominated combined states. Use the same boundary signature during dominance comparisons. Exact continuous/large-integer frontiers may be large: rounding to buckets or capping a frontier is heuristic unless it has a separately proved approximation guarantee. Peak cores can be a feasibility dimension even when the displayed final front is over memory, disk, and latency only. + +## Heuristics and what they give up + +These are useful strategies, but their role must be explicit: + +| Strategy | Safe use in an exact search | Approximate use that loses completeness | +| --- | --- | --- | +| Best standalone / greedy MQO / local improvement | Produce and validate incumbents. | Return the heuristic winner without finishing or certifying the remaining search. | +| Estimated savings, learned cost, query “difficulty” | Choose variable and alternative order; keep every unresolved branch. | Prune on a predicted saving or a learned lower bound without a validity certificate. | +| Per-query top-k, beam search, capped memo/front | Retain all alternatives externally and revisit them, or use only for ordering. | Drop all but k alternatives, potentially losing a donor producer or resource tradeoff. | +| Split by metric, observed sharing, or connected components of current winners | Schedule exploration in groups. | Assume those groups cannot interact under unexamined alternatives. | +| Always share equal expressions; always use the cheapest local materialization | Apply only after E6/E7 conditions are proved. | Force those choices on CPU-only intuition while ignoring retention and resource effects. | +| Epsilon-dominance, resource buckets, coarser sketch/window parameters | Order candidates or supply a known relaxation with one-sided bounds. | Remove near-equivalent states or configurations; exactness then holds only for a deliberately restricted domain, if at all. | +| Time, candidate, or memory budget | Pause with the full unresolved frontier represented; resume later. | Discard unresolved branches and claim completion, or report a gap that ignores dropped work. | + +Greedy benefit does not automatically have diminishing returns here: resizing for additional consumers, overlapping reuse choices, and raw-input alternatives can create complements and conflicts. Do not quote a `1 - 1/e` approximation factor without proving the necessary objective and constraint structure for this model. Randomized search must record its seed; reproducibility is not optimality. + +## Search procedure and stopping contract + +The initial implementation uses depth-first branch-and-bound with a memo and lazy candidate construction. Best-first search can be added later, but keeping every frontier state in RAM can recreate the original memory problem. Depth-first search must retain the bounds of pending siblings and not silently drop work under memory pressure. + +```text +freeze workload, rewrite/parameter inventory, deployment inputs, and evidence +build conservative interaction analysis and candidate groups lazily +validate heuristic completions to seed an incumbent (or no incumbent yet) +search(partial state): + reuse a memo result only under an equivalent full interface + reject permanently invalid branches (E2) + bound all legal completions with a conservative relaxation + reject branches certified infeasible or unable to improve (E3/E4) + if complete: + run the authoritative validity, resource, and ranking checks + update the incumbent or requested Pareto front + otherwise: + choose an unresolved decision; enumerate all legal alternatives lazily + explore promising alternatives first; record bounds for pending siblings + stop or suspend only with the unresolved work represented in the report +``` + +Completeness must hold across all stages. A streaming Stage 3 is insufficient if Stage 1 first constructs a whole Cartesian product or Stage 2 first constructs every materialization. Share unchanged IR through immutable references; instantiate full physical plans only when needed. Diagnostic candidate dumps are an opt-in stream with their own storage limits, not mandatory retained search state. + +Do not enumerate a full product to calculate a search percentage. After duplicate elimination and context-dependent rewriting, its size may be unknown. Report concrete counts and bounds rather than a fabricated fraction. + +For scalar minimization report: + +- the fixed rule/parameter/model/evidence versions and whether the candidate universe was restricted; +- feasible incumbent and its objective `U`, or “no feasible plan found”; +- a global lower bound `L`, or “bound unavailable”; when an incumbent exists, `L` must cover it and every unresolved branch, so use `min(U, all unresolved bounds)`; +- additive gap `U - L`; relative gap `(U - L) / abs(U)` when `U != 0`; an explicit zero-objective convention rather than division by zero; +- status: optimum certified, stopped with incumbent, stopped without incumbent, or infeasibility proved; +- stop reason, time/peak memory, generated/expanded/pruned counts, rule-specific certificates, memo hits, and remaining work; +- whether a heuristic **discarded** states, as distinct from merely ordering them. Include those states in a valid bound or mark the gap unavailable. + +A budget can end with an optimality certificate if all remaining bounds rule out improvement; exhaustion is not required. Conversely, “no feasible plan found” after a budget is not proof of infeasibility. A Pareto request reports front completeness separately from the status of any selected representative. Gap claims are with respect to the frozen planner model and `F`, not real runtime or a larger omitted rewrite space. + +## MILP as a relaxation and an alternative search + +A facility-selection-shaped model is a useful starting point: choose one complete query alternative per query, activate required producer configurations once, and bind every consumer to a compatible producer or recomputation alternative. Include adapters and consumer-specific readout costs; require the relevant provider whenever its consumer is selected; exclude cycles and incompatible bindings. Producer configurations must encode family, size, window, phase, and retention, not merely “a summary exists.” + +For a finite configuration inventory, binary choices can encode exact costs for those configurations without approximating the cost curve. The formulation is exact relative to `F` only if every legal generated plan is representable and every accepted solver solution is legal. Prove those two mappings on the oracle workloads. A compact model containing only some sharing patterns solves a different, restricted problem. + +Hard constraints must include memory, disk, cores, and per-query accuracy/latency requirements where configured. For example, one raw-source retention variable can represent the maximum selected horizon using constraints for every active reader and a nonnegative disk price; additional constraints or a canonical minimum assignment are needed when the price is zero and the horizon affects feasibility/reporting. Critical-path latency and peak memory need explicit encodings consistent with the current model, not a sum substituted for them. Reprice and validate extracted plans through the existing selection path. + +Relaxing a faithful formulation supplies lower bounds for E3. An integer solver provides incumbents and bounds for its model, subject to numerical tolerances. Piecewise approximations, continuous sketch sizing, and unsupported nonlinear costs require one-sided error bounds before their solver gap certifies anything about the original objective. A final original-model reprice validates the incumbent; it does not turn an invalid surrogate lower bound into a certificate. + +Prototype MILP beside the separator/branch-and-bound engine on small exact instances, and compare objective, feasibility, bound strength, planning time, and memory. Do not make a solver dependency mandatory before the simpler exact reductions have measured benefits. + +## Validation and implementation order + +### Acceptance criteria + +Use an independently retained exhaustive enumerator as the small-instance oracle. Freeze its candidate universe and compare optimum objective, feasibility, and the full requested Pareto vector set. Compare winner IDs only under the explicit stable tie contract. Search-node counts and memory are measurements, not equivalence evidence. + +Each implemented pruning rule must have: + +1. a statement of its preconditions and a proof or replacement witness; +2. a failing-before counterexample for the unsafe version, plus a positive case where the safe rule prunes; +3. exhaustive equivalence tests with the rule alone and in combination; +4. counters showing its actual planning-time and memory benefit; +5. a safe fallback when its assumptions are unknown. + +### Required adversarial workloads + +| Fixture | Property it tests | +| --- | --- | +| Donor producer costs 10, private work 6, standalone alternative 5; another consumer costs 100 or reads producer for 1. | Rejects the old standalone subtraction rule; preserves donor-dependent reuse. | +| Unshared choices `(cost 1, memory 10)` and `(cost 2, memory 1)`, memory budget 2. | Cheapest-local pruning must retain the feasible tradeoff. | +| Two disconnected computation groups each choose cheap/high-memory versus dear/low-memory. | Global capacity requires frontier combination, not independent scalar winners. | +| Equal displayed memory/disk/latency, different core requirements, one over the core limit. | Cores remain a partial-state feasibility dimension. | +| Same raw source with horizons 5 and 10; a rewrite exposes another compatible reader. | Per-source maximum is charged once; interaction analysis covers rewrite closure. | +| A shared producer meets a stricter reader only after resizing; an unsupported provisional form has a supported replacement. | Provisional validity failures are deferred; completed output accuracy is rechecked. | +| Same expression but different recurrence, lifetime, or physical property; sharing increases retained state. | Semantic equality alone cannot force a physical shared realization. | +| Three consumers, partial sharing subsets, overlapping rollup/capability/pane opportunities. | Pairwise savings and consumer counts do not stand in for full boundary state. | +| Structurally similar DAGs with different predicates, weights, unique keys, or guarantees; injected hash collisions. | Memo equality and CSE legality are checked beyond hashes. | +| Multiple traversal orders, equal costs, zero costs, near floating-point ties, and unknown model terms. | Stable ranking, conservative bounds, and no pruning from undefined estimates. | +| Stops before the first feasible plan, after an incumbent, and while pending branches have lower bounds below it. | Honest completion/infeasibility status and global bound accounting. | +| Exact/approximate scalar search and full Pareto reporting with the same incumbent. | A scalar certificate is not mistaken for complete Pareto enumeration. | + +Generate small workloads with mixed recurrence, parameter sizes, accuracy evidence, capacities, and sharing patterns; for every visited branch check `LB <= oracle best completion` when a feasible completion exists. Also compare reported resource lower bounds with oracle completions and validate replacement witnesses. Test that changing a registered sharing rule or an evidence snapshot invalidates affected caches. For bounded search, increasing the expansion budget must never worsen the best objective in a deterministic resumed run; certificates must remain valid at every stop. + +### Implementation sequence + +| Step | Deliverable and guarantee | Measurement | +| --- | --- | --- | +| 1 | Lazy generation and streaming scalar/Pareto selection, unchanged feasible universe, explicit legacy-compatible or total-order tie handling, budget statuses. | Time to first feasible plan, total time, retained candidates, peak memory. | +| 2 | Duplicate-state memoization, permanent-invalidity checks, and initial zero/exclusive-work bounds. | Rule-specific pruning, bound cost, cache hits, oracle equality. | +| 3 | Conservative interaction analysis; separable components or exact component-frontier composition. | Component/separator sizes, frontier sizes, capacity cases, analysis cost. | +| 4 | Validated heuristic incumbents and stronger relaxation bounds; all heuristics used for ordering first. | Bound versus incumbent trajectory, time to certificate, expansion count. | +| 5 | Conditional local DP and sharing dominance only for proven interfaces; optional separator DP or MILP prototype. | Full oracle agreement, interface sizes, time/memory versus simpler search. | +| 6 | Explicit approximate mode with beam/front caps or restricted domains, only if exact mode remains too expensive. | Optimality gap against the oracle, available certificates, reproducible budget/seed. | + +Run the planner-layering Examples 1–4, ASAPQuery's 30 mixed workloads, and the Kubernetes recording-rule workload. Report both exhaustive small-instance quality and large-instance scaling, including timeouts, missing coverage, and no-feasible-plan outcomes. “Tens of queries in seconds under a few GB” is a benchmark target; a giant interaction component or resource frontier may defeat it. Distinguish candidate-generation coverage, pruning speed, cost-model accuracy, and runtime correctness rather than attributing all four to search quality. + +## References + +1. [Moerkotte and Neumann, VLDB 2006: optimal bushy join trees without cross products](https://www.vldb.org/conf/2006/p930-moerkotte.pdf). Connected-subgraph enumeration within a defined join space; it does not justify deleting cross-query sharing choices. +2. [Graefe, 1995: The Cascades Framework for Query Optimization](https://15721.courses.cs.cmu.edu/spring2019/papers/22-optimizer1/graefe-ieee1995.pdf). Memoization, properties, and guided search. +3. [Roy, Seshadri, Sudarshan, and Bhobe, SIGMOD 2000: Efficient and Extensible Algorithms for Multi Query Optimization](https://www.cse.iitb.ac.in/~sudarsha/Pubs-dir/mqo-sigmod00.pdf). Practical cost-based MQO heuristics, including greedy materialization. +4. [The #671 survey and view-selection references](https://github.com/ProjectASAP/ASAPPlanner/issues/671#issuecomment-6137638876). Background and candidate directions, not proofs of this proposal's pruning rules. +5. [Trummer and Koch: Solving the Join Ordering Problem via Mixed Integer Linear Programming](https://arxiv.org/abs/1511.02071). A finite join-order formulation with approximated costs; distinguish solver optimality from original-model optimality. +6. [PostgreSQL documentation: genetic query optimization](https://www.postgresql.org/docs/17/runtime-config-query.html#RUNTIME-CONFIG-QUERY-GEQO). An explicit example of trading exhaustive search for planning time. From 7e5094832f35a0a5209a78090bc2bb4639b24ec7 Mon Sep 17 00:00:00 2001 From: zzylol <50204836+zzylol@users.noreply.github.com> Date: Tue, 20 Oct 2026 18:09:50 +0000 Subject: [PATCH 03/53] docs: make survey citations clickable and use stable MQO source --- docs/design_docs/proposals/joint-planning-search.md | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/docs/design_docs/proposals/joint-planning-search.md b/docs/design_docs/proposals/joint-planning-search.md index c30700a79..48a19f4e2 100644 --- a/docs/design_docs/proposals/joint-planning-search.md +++ b/docs/design_docs/proposals/joint-planning-search.md @@ -50,11 +50,11 @@ Fix query recurrence, time ranges, pricing, measured costs, accuracy evidence, a | Approach | Useful idea for ASAPPlanner | Exactness boundary | | --- | --- | --- | -| System R-style DP; DPccp/DPhyp [1] | Memoize subproblems; exploit sparse interaction graphs; retain properties a parent needs. | A single cheapest subplan is safe only with optimal substructure. DPccp's cross-product-free join space is a restriction, not proof that arbitrary workload sharing is separable. | -| Volcano/Cascades [2] | Groups of equivalent computations, required physical properties, cost limits, and memoized exploration. | A memo shares optimizer work. It does not automatically decide runtime sharing, materialization, or a correct state key. | -| MQO and view selection [3, 4] | Model build-once/read-many decisions jointly; use greedy materialization as a practical incumbent. | Greedy MQO is heuristic. Approximation guarantees for a particular data-cube lattice or benefit function do not transfer to arbitrary ASAP DAGs. | -| MILP [5] | Explicit coupled decisions, relaxations as lower bounds, solver incumbents and gaps. | Exact only for the encoded choices and objective. Linear cost approximations and omitted sharing modes can change the optimum. | -| Rule passes and bounded/randomized search | Cheap simplification, branch ordering, and a predictable planning budget. | Restricting exploration is heuristic unless omitted alternatives are proved dominated. PostgreSQL explicitly documents this tradeoff for GEQO [6]. | +| System R-style DP; DPccp/DPhyp [1](https://www.vldb.org/conf/2006/p930-moerkotte.pdf) | Memoize subproblems; exploit sparse interaction graphs; retain properties a parent needs. | A single cheapest subplan is safe only with optimal substructure. DPccp's cross-product-free join space is a restriction, not proof that arbitrary workload sharing is separable. | +| Volcano/Cascades [2](https://15721.courses.cs.cmu.edu/spring2019/papers/22-optimizer1/graefe-ieee1995.pdf) | Groups of equivalent computations, required physical properties, cost limits, and memoized exploration. | A memo shares optimizer work. It does not automatically decide runtime sharing, materialization, or a correct state key. | +| MQO and view selection [3](https://arxiv.org/abs/cs/9910021), [4](https://github.com/ProjectASAP/ASAPPlanner/issues/671#issuecomment-6137638876) | Model build-once/read-many decisions jointly; use greedy materialization as a practical incumbent. | Greedy MQO is heuristic. Approximation guarantees for a particular data-cube lattice or benefit function do not transfer to arbitrary ASAP DAGs. | +| MILP [5](https://arxiv.org/abs/1511.02071) | Explicit coupled decisions, relaxations as lower bounds, solver incumbents and gaps. | Exact only for the encoded choices and objective. Linear cost approximations and omitted sharing modes can change the optimum. | +| Rule passes and bounded/randomized search | Cheap simplification, branch ordering, and a predictable planning budget. | Restricting exploration is heuristic unless omitted alternatives are proved dominated. PostgreSQL explicitly documents this tradeoff for GEQO [6](https://www.postgresql.org/docs/17/runtime-config-query.html#RUNTIME-CONFIG-QUERY-GEQO). | The survey's `global_selection` discussion describes an earlier implementation, not an API this design assumes still exists. The proposal branch's current stage pipeline is documented in [`stage1_to_stage3.rs`](../../../crates/plan-selection/src/pass/stage1_to_stage3.rs); sharing rules in [`asap_aware_cse.rs`](../../../crates/logical-asap-optimization/src/asap_aware_cse.rs); and resource aggregation and ranking in [`plan-selection/src/lib.rs`](../../../crates/plan-selection/src/lib.rs). Audit those contracts when implementing each rule. The architecture below is an ASAPPlanner design derived from the survey, not a guarantee supplied by those papers. @@ -329,7 +329,7 @@ Run the planner-layering Examples 1–4, ASAPQuery's 30 mixed workloads, and the 1. [Moerkotte and Neumann, VLDB 2006: optimal bushy join trees without cross products](https://www.vldb.org/conf/2006/p930-moerkotte.pdf). Connected-subgraph enumeration within a defined join space; it does not justify deleting cross-query sharing choices. 2. [Graefe, 1995: The Cascades Framework for Query Optimization](https://15721.courses.cs.cmu.edu/spring2019/papers/22-optimizer1/graefe-ieee1995.pdf). Memoization, properties, and guided search. -3. [Roy, Seshadri, Sudarshan, and Bhobe, SIGMOD 2000: Efficient and Extensible Algorithms for Multi Query Optimization](https://www.cse.iitb.ac.in/~sudarsha/Pubs-dir/mqo-sigmod00.pdf). Practical cost-based MQO heuristics, including greedy materialization. +3. [Roy, Seshadri, Sudarshan, and Bhobe, SIGMOD 2000: Efficient and Extensible Algorithms for Multi Query Optimization](https://arxiv.org/abs/cs/9910021). Practical cost-based MQO heuristics, including greedy materialization. 4. [The #671 survey and view-selection references](https://github.com/ProjectASAP/ASAPPlanner/issues/671#issuecomment-6137638876). Background and candidate directions, not proofs of this proposal's pruning rules. 5. [Trummer and Koch: Solving the Join Ordering Problem via Mixed Integer Linear Programming](https://arxiv.org/abs/1511.02071). A finite join-order formulation with approximated costs; distinguish solver optimality from original-model optimality. 6. [PostgreSQL documentation: genetic query optimization](https://www.postgresql.org/docs/17/runtime-config-query.html#RUNTIME-CONFIG-QUERY-GEQO). An explicit example of trading exhaustive search for planning time. From 26285640ae8a4363b86923fdefb3e655d7b94110 Mon Sep 17 00:00:00 2001 From: zzylol <50204836+zzylol@users.noreply.github.com> Date: Tue, 20 Oct 2026 18:10:31 +0000 Subject: [PATCH 04/53] docs: present joint search as a standalone design --- .../proposals/joint-planning-search.md | 34 +++++-------------- 1 file changed, 8 insertions(+), 26 deletions(-) diff --git a/docs/design_docs/proposals/joint-planning-search.md b/docs/design_docs/proposals/joint-planning-search.md index 48a19f4e2..185082f0f 100644 --- a/docs/design_docs/proposals/joint-planning-search.md +++ b/docs/design_docs/proposals/joint-planning-search.md @@ -2,7 +2,7 @@ Status: proposal; no new pruning rule is implemented by this document. Audience: planner designers and implementers. -ASAPPlanner must search a workload's candidate plans without retaining or pricing every combination. The design keeps the decisions in [planner-layering.md](planner-layering.md), and changes how those decisions are explored. It incorporates the [optimizer survey in #671](https://github.com/ProjectASAP/ASAPPlanner/issues/671#issuecomment-6137638876) and revises the original exactness claims in [#776](https://github.com/ProjectASAP/ASAPPlanner/pull/776). +ASAPPlanner must search a workload's candidate plans without retaining or pricing every combination. The design keeps the decisions in [planner-layering.md](planner-layering.md), and changes how those decisions are explored. It builds on the [optimizer survey in #671](https://github.com/ProjectASAP/ASAPPlanner/issues/671#issuecomment-6137638876). **Decision:** build a streaming, memoized branch-and-bound search with conservative interaction analysis. Enable a pruning rule only when its preconditions establish that it preserves the optimum. Use greedy search to find incumbents and order exploration; discarding alternatives by a heuristic is a separately reported approximation mode. A memo, dynamic programming, or a MILP solver does not by itself make workload sharing exact. @@ -10,7 +10,6 @@ ASAPPlanner must search a workload's candidate plans without retaining or pricin - [Problem and scope of the guarantee](#problem-and-scope-of-the-guarantee) - [Lessons from the survey](#lessons-from-the-survey) -- [Review of the original six rules](#review-of-the-original-six-rules) - [Search representation and interaction analysis](#search-representation-and-interaction-analysis) - [Exact pruning rules and their proof conditions](#exact-pruning-rules-and-their-proof-conditions) - [Bounds and branch-and-bound](#bounds-and-branch-and-bound) @@ -23,7 +22,7 @@ ASAPPlanner must search a workload's candidate plans without retaining or pricin ## Problem and scope of the guarantee -The original motivating measurement was a growing prefix of one 10-query ASAPQuery mixed workload: +A motivating measurement was a growing prefix of one 10-query ASAPQuery mixed workload: | Queries | Planning time | Peak memory | | ---: | ---: | ---: | @@ -56,26 +55,7 @@ Fix query recurrence, time ranges, pricing, measured costs, accuracy evidence, a | MILP [5](https://arxiv.org/abs/1511.02071) | Explicit coupled decisions, relaxations as lower bounds, solver incumbents and gaps. | Exact only for the encoded choices and objective. Linear cost approximations and omitted sharing modes can change the optimum. | | Rule passes and bounded/randomized search | Cheap simplification, branch ordering, and a predictable planning budget. | Restricting exploration is heuristic unless omitted alternatives are proved dominated. PostgreSQL explicitly documents this tradeoff for GEQO [6](https://www.postgresql.org/docs/17/runtime-config-query.html#RUNTIME-CONFIG-QUERY-GEQO). | -The survey's `global_selection` discussion describes an earlier implementation, not an API this design assumes still exists. The proposal branch's current stage pipeline is documented in [`stage1_to_stage3.rs`](../../../crates/plan-selection/src/pass/stage1_to_stage3.rs); sharing rules in [`asap_aware_cse.rs`](../../../crates/logical-asap-optimization/src/asap_aware_cse.rs); and resource aggregation and ranking in [`plan-selection/src/lib.rs`](../../../crates/plan-selection/src/lib.rs). Audit those contracts when implementing each rule. The architecture below is an ASAPPlanner design derived from the survey, not a guarantee supplied by those papers. - -## Review of the original six rules - -| Original rule | Correct classification | Required change | -| --- | --- | --- | -| Split queries by possible sharing | Conditionally exact. | Include every possible sharing/reuse edge and shared resource-accounting factor. Independent computation alone is insufficient with global budgets. | -| Check each query's validity early | Exact for a rejection invariant under every remaining completion. | Defer failures that resizing, replacement, sharing, or later lowering can repair. Missing evidence for a provisional expression is not such a proof. | -| Prune if `alone(c) - shareable(c) > best_alone(q)` | **Not generally exact**, even with additive operator costs. | A saving on one query is not a bound on the effect on all consumers. Replace with a workload branch bound or a completion-preserving replacement proof. | -| Choose the cheapest materialization for unshared builds | Conditionally exact. | Prove isolation from other choices, including ancestors, shared inputs, memory/disk budgets, recurrence, and output lifetime. Otherwise keep a frontier. | -| Always apply identical-expression sharing | Semantically safe only through the existing sharing legality checks; cost dominance is conditional. | Distinguish canonical representation from executing once. Retention, parallelism, copying, synchronization, and physical properties can favor recomputation. | -| Stream candidates and stop at a budget | Streaming is exact if enumeration and selection are equivalent. Early stopping is not generally exact. | Keep unresolved work represented in the lower bound; report a feasible incumbent, gap, and completion status. | - -### Why the standalone subtraction rule is unsafe - -Consider two queries. Query A can choose candidate `c`, containing a shareable producer costing 10 and private work costing 6, or candidate `a`, costing 5. Query B alone costs 100. A permitted cross-query rewrite can answer B from `c`'s producer with 1 unit of adapter work; the producer is offered through `c`, and that rewrite is lost if `c` is removed. - -The old test uses `alone(c) = 16`, `shareable(c) = 10`, and `best_alone(A) = 5`. It discards `c` because `16 - 10 > 5`. Yet the joint plan using `c` costs `10 + 6 + 1 = 17`, versus `5 + 100 = 105` without that opportunity. The missing quantity is the benefit to B, not the part of A's own bill it might avoid. A reference generator that independently offers the producer even after replacing A might admit a different repair; that repair must be demonstrated, not assumed. Rollup rules that only reuse an already present finer computation make this distinction relevant. - -Even without this donor effect, whole-workload memory constraints can invalidate a standalone replacement that uses more memory. More generally, marginal cost cannot be assigned independently to each query when a shared build is paid once. +The implementation boundaries are documented in [`stage1_to_stage3.rs`](../../../crates/plan-selection/src/pass/stage1_to_stage3.rs); sharing rules in [`asap_aware_cse.rs`](../../../crates/logical-asap-optimization/src/asap_aware_cse.rs); and resource aggregation and ranking in [`plan-selection/src/lib.rs`](../../../crates/plan-selection/src/lib.rs). Audit those contracts when implementing each rule. The following sections define the search state, pruning conditions, and execution procedure for ASAPPlanner. ## Search representation and interaction analysis @@ -144,6 +124,8 @@ A practical sufficient certificate for partial states is the same boundary inter **Proof:** mapping every potentially winning completion of B to a no-worse completion of A preserves the optimum. Equal vectors are deduplicable only under the declared representative/tie policy. Context-free per-query Pareto pruning is not justified just because several metrics are recorded. +**Example: preserve benefits to other queries.** Query A can use a producer costing 10 plus private work costing 6, or an independent alternative costing 5. Query B costs 100 independently, but a permitted reuse rewrite can answer it from A's producer with 1 unit of adapter work. Keeping the producer gives a joint cost of `10 + 6 + 1 = 17`; removing that opportunity gives `5 + 100 = 105`. A replacement of A must preserve B's access to that producer, or account for the loss in the whole-workload comparison. The producer's own cost is not an upper bound on the saving it enables for other consumers. + ### E5. Solve truly independent components separately If `F = F1 × ... × Fk`, feasibility factors, and `C(P) = sum Ci(Pi)`, choose a scalar optimum independently in each component. Potential interactions must have been analyzed conservatively as above. @@ -183,7 +165,7 @@ Start with bounds whose validity is easy to audit: 3. **Independent relaxed subproblems:** drop selected coupling constraints, allow conflicting choices to coexist, and optimize the remaining relaxed domains. The relaxation must contain every legal completion. Only sum terms whose accounting prevents double-counting; a sum of standalone query optima generally does not meet that requirement. 4. **LP relaxation:** encode candidate/build/binding choices and valid cost/resource under-estimators, then relax integer decisions to continuous ones. The relaxation optimum is a bound on the encoded exact objective when every original feasible plan maps into the relaxed model. A surrogate cost with no one-sided relation supplies no certified bound. -If a “sharing saving” is used, it must upper-bound the **total workload** saving under all completions, include benefits to other consumers, and count overlapping savings conservatively. That proof is harder than using an explicit relaxation. Do not implement the original per-query subtraction as an exact shortcut. +If a “sharing saving” is used, it must upper-bound the **total workload** saving under all completions, include benefits to other consumers, and count overlapping savings conservatively. That proof is harder than using an explicit relaxation. Use workload-level bounds rather than subtracting a query-local sharing estimate from its standalone cost. An admissible bound need not be consistent between parent and child states for branch-and-bound correctness. A weak or zero bound can mean exhaustive exploration. Prefer a cheaply evaluated bound unless stronger pruning pays for its own planning overhead. @@ -288,7 +270,7 @@ Use an independently retained exhaustive enumerator as the small-instance oracle Each implemented pruning rule must have: 1. a statement of its preconditions and a proof or replacement witness; -2. a failing-before counterexample for the unsafe version, plus a positive case where the safe rule prunes; +2. an adversarial case where its preconditions do not hold, plus a positive case where the rule prunes; 3. exhaustive equivalence tests with the rule alone and in combination; 4. counters showing its actual planning-time and memory benefit; 5. a safe fallback when its assumptions are unknown. @@ -297,7 +279,7 @@ Each implemented pruning rule must have: | Fixture | Property it tests | | --- | --- | -| Donor producer costs 10, private work 6, standalone alternative 5; another consumer costs 100 or reads producer for 1. | Rejects the old standalone subtraction rule; preserves donor-dependent reuse. | +| Donor producer costs 10, private work 6, standalone alternative 5; another consumer costs 100 or reads producer for 1. | Preserves donor-dependent reuse when evaluating candidate replacements. | | Unshared choices `(cost 1, memory 10)` and `(cost 2, memory 1)`, memory budget 2. | Cheapest-local pruning must retain the feasible tradeoff. | | Two disconnected computation groups each choose cheap/high-memory versus dear/low-memory. | Global capacity requires frontier combination, not independent scalar winners. | | Equal displayed memory/disk/latency, different core requirements, one over the core limit. | Cores remain a partial-state feasibility dimension. | From 41dc9b285e853b2a91ddf06bb0269f6663b09c71 Mon Sep 17 00:00:00 2001 From: zzylol <50204836+zzylol@users.noreply.github.com> Date: Tue, 20 Oct 2026 18:12:16 +0000 Subject: [PATCH 05/53] docs: present optimizer survey before design and source memo terminology --- .../proposals/joint-planning-search.md | 56 +++++++++++++++++-- 1 file changed, 51 insertions(+), 5 deletions(-) diff --git a/docs/design_docs/proposals/joint-planning-search.md b/docs/design_docs/proposals/joint-planning-search.md index 185082f0f..7cf493ed6 100644 --- a/docs/design_docs/proposals/joint-planning-search.md +++ b/docs/design_docs/proposals/joint-planning-search.md @@ -4,12 +4,11 @@ Status: proposal; no new pruning rule is implemented by this document. Audience: ASAPPlanner must search a workload's candidate plans without retaining or pricing every combination. The design keeps the decisions in [planner-layering.md](planner-layering.md), and changes how those decisions are explored. It builds on the [optimizer survey in #671](https://github.com/ProjectASAP/ASAPPlanner/issues/671#issuecomment-6137638876). -**Decision:** build a streaming, memoized branch-and-bound search with conservative interaction analysis. Enable a pruning rule only when its preconditions establish that it preserves the optimum. Use greedy search to find incumbents and order exploration; discarding alternatives by a heuristic is a separately reported approximation mode. A memo, dynamic programming, or a MILP solver does not by itself make workload sharing exact. - ## Contents - [Problem and scope of the guarantee](#problem-and-scope-of-the-guarantee) -- [Lessons from the survey](#lessons-from-the-survey) +- [Survey of optimizer search approaches](#survey-of-optimizer-search-approaches) +- [Implications for ASAPPlanner and proposed approach](#implications-for-asapplanner-and-proposed-approach) - [Search representation and interaction analysis](#search-representation-and-interaction-analysis) - [Exact pruning rules and their proof conditions](#exact-pruning-rules-and-their-proof-conditions) - [Bounds and branch-and-bound](#bounds-and-branch-and-bound) @@ -45,7 +44,47 @@ Fix query recurrence, time ranges, pricing, measured costs, accuracy evidence, a **Tie contract:** preserving optimal cost need not preserve the same candidate ID. Define a deterministic total ranking for the new search, with stable reference rank for exact ties. The legacy relative-tolerance comparison is not transitive; stable IDs alone cannot reproduce its sequential winner after reordering. If legacy winner identity is required, replay the reference-order comparison for all potentially winning near-tie candidates, or keep that path exhaustive until an equivalent rule is proved. A Pareto report must say whether it preserves all equivalent plan identities or only one representative per objective vector. Changing the tie policy is a compatibility decision, not an exact-search optimization. -## Lessons from the survey +## Survey of optimizer search approaches + +The approaches below differ in what they enumerate, what they memoize, and which alternatives they discard. Their guarantees apply to their specified plan spaces and cost models. This survey presents those mechanisms before choosing how to use them in ASAPPlanner. + +### Bottom-up dynamic programming: System R, DPccp, and DPhyp + +[System R](https://research.ibm.com/publications/access-path-selection-in-a-relational-database-management-system) builds plans for progressively larger relation subsets. For each subproblem it retains cheap alternatives and useful physical properties, notably interesting orders that can reduce later sorting costs. The classic search concentrates on left-deep join plans. The recurrence relies on optimal substructure: once the subset and relevant properties are fixed, a more expensive equivalent subplan need not be reconsidered in every parent. + +[DPccp](https://www.vldb.org/conf/2006/p930-moerkotte.pdf) enumerates connected subgraphs and their connected complements to generate bushy join trees without cross products. [DPhyp](https://15721.courses.cs.cmu.edu/spring2020/papers/20-optimizer2/p539-moerkotte.pdf) extends the graph treatment to hypergraphs and more complex join predicates. These methods reduce enumeration work by exploiting query structure; they do not make arbitrary plan spaces polynomial. Their exactness is relative to the join alternatives and properties they retain. + +### Top-down memoization: Volcano and Cascades + +[Volcano/Cascades](https://15721.courses.cs.cmu.edu/spring2019/papers/22-optimizer1/graefe-ieee1995.pdf) organizes equivalent expressions into memo groups. Transformation rules add equivalent expressions; implementation rules supply physical alternatives. Optimization requests include required properties, so the same logical group can have different winners for different contexts. A parent requests child implementations, combines their costs, and can stop exploring an alternative when a valid cost limit rules it out. + +The memo avoids optimizing the same subproblem repeatedly. Branch-and-bound avoids completing alternatives that cannot improve a known result. These mechanisms depend on sufficient property keys and valid bounds; storing equivalent expressions together does not itself decide whether their runtime results should be computed once and shared. + +### Ordered rule passes: DataFusion + +[DataFusion's optimizer documentation](https://datafusion.apache.org/library-user-guide/query-optimizer.html) describes logical optimizer rules, configurable rule sets, and repeated passes. Rules simplify expressions, push predicates and projections, and rewrite plan structure. Physical optimization is a separate phase concerned with executable plans and their requirements. + +Rule application can reduce work without explicitly searching every equivalent plan. A pass limit or fixed point describes when rewriting stops; it is not a certificate that the globally cheapest plan has been examined. Correctness of each rewrite and completeness of cost-based search are separate properties. A rule pipeline can also serve as preprocessing for an enumerator or memo-based search. + +### Joint sharing and materialization: multi-query optimization + +[Multi-query optimization by Roy et al.](https://arxiv.org/abs/cs/9910021) extends optimizer reasoning to common subexpressions across queries. A shared expression can be built once, materialized, and read by several consumers. Its benefit depends on the readers, their alternatives, and the cost of materialization and reuse. The paper presents Volcano-based and greedy cost-based heuristics rather than a general exhaustive-optimum guarantee. + +[Materialized-view selection in data cubes](https://dsf.berkeley.edu/cs286/papers/datacube-sigmod1996.pdf) studies which aggregate views to retain under a space constraint, exploiting relationships between views in a lattice. Its greedy results rely on that problem's benefit and constraint structure. General workload DAGs can have overlapping sharing opportunities and incompatible configurations, so the same approximation guarantee requires a separate proof. + +### Mathematical optimization: MILP + +[Trummer and Koch](https://arxiv.org/abs/1511.02071) encode join-order and implementation decisions as binary variables with constraints restricting legal plans. Their formulation searches left-deep plans and approximates operation costs with linear functions. A solver can return a feasible incumbent, a bound, and an optimality gap for the formulation. + +This separates the combinatorial search from its implementation, but the encoding must cover the intended alternatives and resource constraints. Solver optimality concerns the encoded objective. Cost approximations and omitted physical choices determine how that result relates to the original planning problem. + +### Bounded and randomized search + +Greedy choice, local improvement, beam search, and randomized exploration limit the planning effort spent on large spaces. [PostgreSQL documents GEQO](https://www.postgresql.org/docs/17/runtime-config-query.html#RUNTIME-CONFIG-QUERY-GEQO) as a heuristic alternative for large join problems that can produce plans inferior to exhaustive search. + +These techniques can find useful plans quickly. Used only to order a complete search or provide feasible incumbents, they do not remove its eventual optimality guarantee. Used to discard alternatives or stop without a certificate, they trade completeness for planning time. + +## Implications for ASAPPlanner and proposed approach | Approach | Useful idea for ASAPPlanner | Exactness boundary | | --- | --- | --- | @@ -55,13 +94,15 @@ Fix query recurrence, time ranges, pricing, measured costs, accuracy evidence, a | MILP [5](https://arxiv.org/abs/1511.02071) | Explicit coupled decisions, relaxations as lower bounds, solver incumbents and gaps. | Exact only for the encoded choices and objective. Linear cost approximations and omitted sharing modes can change the optimum. | | Rule passes and bounded/randomized search | Cheap simplification, branch ordering, and a predictable planning budget. | Restricting exploration is heuristic unless omitted alternatives are proved dominated. PostgreSQL explicitly documents this tradeoff for GEQO [6](https://www.postgresql.org/docs/17/runtime-config-query.html#RUNTIME-CONFIG-QUERY-GEQO). | +**Decision:** build a streaming, memoized branch-and-bound search with conservative interaction analysis. Enable a pruning rule only when its preconditions establish that it preserves the optimum. Use greedy search to find incumbents and order exploration; discarding alternatives by a heuristic is a separately reported approximation mode. A memo, dynamic programming, or a MILP solver does not by itself make workload sharing exact. + The implementation boundaries are documented in [`stage1_to_stage3.rs`](../../../crates/plan-selection/src/pass/stage1_to_stage3.rs); sharing rules in [`asap_aware_cse.rs`](../../../crates/logical-asap-optimization/src/asap_aware_cse.rs); and resource aggregation and ranking in [`plan-selection/src/lib.rs`](../../../crates/plan-selection/src/lib.rs). Audit those contracts when implementing each rule. The following sections define the search state, pruning conditions, and execution procedure for ASAPPlanner. ## Search representation and interaction analysis ### Memo groups and boundary state -A memo group represents a computation with the same observable semantics. Its alternatives describe exact or summary implementations, their output guarantees, and legal physical realizations. Similar source names, matching hashes, or equal output column names do not establish equivalence. +**Memo group** is established database-optimizer terminology: a group of logically equivalent expressions in the optimizer's memo. See [Graefe's Cascades paper, Section 2](https://15721.courses.cs.cmu.edu/spring2019/papers/22-optimizer1/graefe-ieee1995.pdf) and [CockroachDB's optimizer documentation, “Memo”](https://github.com/cockroachdb/cockroach/blob/master/pkg/sql/opt/doc.go#memo). Here, a group represents a computation with the same observable semantics. Its alternatives describe exact or summary implementations, their output guarantees, and legal physical realizations. Similar source names, matching hashes, or equal output column names do not establish equivalence. A search state records the unresolved choices and the interface through which already chosen work can affect them. Include, when observable: @@ -315,3 +356,8 @@ Run the planner-layering Examples 1–4, ASAPQuery's 30 mixed workloads, and the 4. [The #671 survey and view-selection references](https://github.com/ProjectASAP/ASAPPlanner/issues/671#issuecomment-6137638876). Background and candidate directions, not proofs of this proposal's pruning rules. 5. [Trummer and Koch: Solving the Join Ordering Problem via Mixed Integer Linear Programming](https://arxiv.org/abs/1511.02071). A finite join-order formulation with approximated costs; distinguish solver optimality from original-model optimality. 6. [PostgreSQL documentation: genetic query optimization](https://www.postgresql.org/docs/17/runtime-config-query.html#RUNTIME-CONFIG-QUERY-GEQO). An explicit example of trading exhaustive search for planning time. +7. [Selinger et al., SIGMOD 1979: Access Path Selection in a Relational Database Management System](https://research.ibm.com/publications/access-path-selection-in-a-relational-database-management-system). System R's cost-based access-path and join planning. +8. [Moerkotte and Neumann, SIGMOD 2008: Dynamic Programming Strikes Back](https://15721.courses.cs.cmu.edu/spring2020/papers/20-optimizer2/p539-moerkotte.pdf). Hypergraph-based join enumeration with DPhyp. +9. [Apache DataFusion: Query Optimizer](https://datafusion.apache.org/library-user-guide/query-optimizer.html). Logical rule passes and optimizer customization. +10. [Harinarayan, Rajaraman, and Ullman, SIGMOD 1996: Implementing Data Cubes Efficiently](https://dsf.berkeley.edu/cs286/papers/datacube-sigmod1996.pdf). Materialized-view selection over a data-cube lattice. +11. [CockroachDB optimizer package documentation: Memo](https://github.com/cockroachdb/cockroach/blob/master/pkg/sql/opt/doc.go#memo). Defines memo groups and illustrates equivalent join expressions in the same group. From 714e14ad4ac1bcbbd2031db33663901c71a63f32 Mon Sep 17 00:00:00 2001 From: zzylol <50204836+zzylol@users.noreply.github.com> Date: Tue, 20 Oct 2026 18:15:52 +0000 Subject: [PATCH 06/53] docs: focus joint search proposal on exact pruning --- .../proposals/joint-planning-search.md | 371 +++--------------- 1 file changed, 47 insertions(+), 324 deletions(-) diff --git a/docs/design_docs/proposals/joint-planning-search.md b/docs/design_docs/proposals/joint-planning-search.md index 7cf493ed6..297cae326 100644 --- a/docs/design_docs/proposals/joint-planning-search.md +++ b/docs/design_docs/proposals/joint-planning-search.md @@ -1,363 +1,86 @@ -# Joint planning search: exact reductions, bounds, and heuristic fallbacks +# Joint planning search and exact pruning -Status: proposal; no new pruning rule is implemented by this document. Audience: planner designers and implementers. +Status: proposal; this document does not implement new pruning. Audience: planner designers and implementers. -ASAPPlanner must search a workload's candidate plans without retaining or pricing every combination. The design keeps the decisions in [planner-layering.md](planner-layering.md), and changes how those decisions are explored. It builds on the [optimizer survey in #671](https://github.com/ProjectASAP/ASAPPlanner/issues/671#issuecomment-6137638876). +ASAPPlanner searches per-query alternatives, sharing choices, and materializations to select a whole-workload plan. The design preserves the stages in [planner-layering.md](planner-layering.md) while reducing redundant search. It follows the [survey in #671](https://github.com/ProjectASAP/ASAPPlanner/issues/671#issuecomment-6137638876). -## Contents +## Optimizer search survey -- [Problem and scope of the guarantee](#problem-and-scope-of-the-guarantee) -- [Survey of optimizer search approaches](#survey-of-optimizer-search-approaches) -- [Implications for ASAPPlanner and proposed approach](#implications-for-asapplanner-and-proposed-approach) -- [Search representation and interaction analysis](#search-representation-and-interaction-analysis) -- [Exact pruning rules and their proof conditions](#exact-pruning-rules-and-their-proof-conditions) -- [Bounds and branch-and-bound](#bounds-and-branch-and-bound) -- [Composition under the three cost models](#composition-under-the-three-cost-models) -- [Heuristics and what they give up](#heuristics-and-what-they-give-up) -- [Search procedure and stopping contract](#search-procedure-and-stopping-contract) -- [MILP as a relaxation and an alternative search](#milp-as-a-relaxation-and-an-alternative-search) -- [Validation and implementation order](#validation-and-implementation-order) -- [References](#references) - -## Problem and scope of the guarantee - -A motivating measurement was a growing prefix of one 10-query ASAPQuery mixed workload: - -| Queries | Planning time | Peak memory | -| ---: | ---: | ---: | -| 3 | 0.4 s | 0.14 GB | -| 4 | 5 s | 1.5 GB | -| 5 | 129 s | 24.7 GB | -| 6 | stopped around 228 s | around 100 GB | - -These are motivating observations, not performance guarantees for this proposal. The Cartesian product of per-target alternatives, subsets of sharing opportunities, and materialization assignments makes enumeration exponential. Streaming reduces retained candidates; it does not remove that time complexity. No exact algorithm here promises seconds on every workload. - -Let `F` be the finite set of complete plans the reference generator admits for a fixed workload, rule inventory, parameter inventory, deployment capabilities, and evidence snapshot. For scalar minimization, exact search returns a feasible `P*` with `C(P*) = min(C(P) for P in F)`. For a Pareto request, it returns the complete nondominated objective set, with explicitly defined handling of equivalent plans. - -This guarantee has three separate boundaries: - -- **Search completeness:** no plan in `F` that could improve the requested result is lost. It says nothing about rewrites, sketch parameters, or window realizations absent from `F`. -- **Model optimality:** `C` is the configured planner objective. An exact search can still choose a slow runtime plan when its cost or cardinality model is inaccurate. Empirical estimates are inputs to that model, not measured-runtime guarantees. -- **Validity:** accepted plans must satisfy the existing semantic, output-accuracy, capability, and executability contracts. An empirical accuracy acceptance is exact relative to that declared evidence policy, not a new theoretical error guarantee. - -Fix query recurrence, time ranges, pricing, measured costs, accuracy evidence, and tie handling for a search. A changed model invalidates cached costs and proof certificates. Nonfinite or unknown estimates never justify pruning. Unsupported cost models fall back to a weaker bound, including zero for a nonnegative scalar objective. - -**Tie contract:** preserving optimal cost need not preserve the same candidate ID. Define a deterministic total ranking for the new search, with stable reference rank for exact ties. The legacy relative-tolerance comparison is not transitive; stable IDs alone cannot reproduce its sequential winner after reordering. If legacy winner identity is required, replay the reference-order comparison for all potentially winning near-tie candidates, or keep that path exhaustive until an equivalent rule is proved. A Pareto report must say whether it preserves all equivalent plan identities or only one representative per objective vector. Changing the tie policy is a compatibility decision, not an exact-search optimization. - -## Survey of optimizer search approaches - -The approaches below differ in what they enumerate, what they memoize, and which alternatives they discard. Their guarantees apply to their specified plan spaces and cost models. This survey presents those mechanisms before choosing how to use them in ASAPPlanner. - -### Bottom-up dynamic programming: System R, DPccp, and DPhyp - -[System R](https://research.ibm.com/publications/access-path-selection-in-a-relational-database-management-system) builds plans for progressively larger relation subsets. For each subproblem it retains cheap alternatives and useful physical properties, notably interesting orders that can reduce later sorting costs. The classic search concentrates on left-deep join plans. The recurrence relies on optimal substructure: once the subset and relevant properties are fixed, a more expensive equivalent subplan need not be reconsidered in every parent. - -[DPccp](https://www.vldb.org/conf/2006/p930-moerkotte.pdf) enumerates connected subgraphs and their connected complements to generate bushy join trees without cross products. [DPhyp](https://15721.courses.cs.cmu.edu/spring2020/papers/20-optimizer2/p539-moerkotte.pdf) extends the graph treatment to hypergraphs and more complex join predicates. These methods reduce enumeration work by exploiting query structure; they do not make arbitrary plan spaces polynomial. Their exactness is relative to the join alternatives and properties they retain. - -### Top-down memoization: Volcano and Cascades - -[Volcano/Cascades](https://15721.courses.cs.cmu.edu/spring2019/papers/22-optimizer1/graefe-ieee1995.pdf) organizes equivalent expressions into memo groups. Transformation rules add equivalent expressions; implementation rules supply physical alternatives. Optimization requests include required properties, so the same logical group can have different winners for different contexts. A parent requests child implementations, combines their costs, and can stop exploring an alternative when a valid cost limit rules it out. - -The memo avoids optimizing the same subproblem repeatedly. Branch-and-bound avoids completing alternatives that cannot improve a known result. These mechanisms depend on sufficient property keys and valid bounds; storing equivalent expressions together does not itself decide whether their runtime results should be computed once and shared. - -### Ordered rule passes: DataFusion - -[DataFusion's optimizer documentation](https://datafusion.apache.org/library-user-guide/query-optimizer.html) describes logical optimizer rules, configurable rule sets, and repeated passes. Rules simplify expressions, push predicates and projections, and rewrite plan structure. Physical optimization is a separate phase concerned with executable plans and their requirements. - -Rule application can reduce work without explicitly searching every equivalent plan. A pass limit or fixed point describes when rewriting stops; it is not a certificate that the globally cheapest plan has been examined. Correctness of each rewrite and completeness of cost-based search are separate properties. A rule pipeline can also serve as preprocessing for an enumerator or memo-based search. - -### Joint sharing and materialization: multi-query optimization - -[Multi-query optimization by Roy et al.](https://arxiv.org/abs/cs/9910021) extends optimizer reasoning to common subexpressions across queries. A shared expression can be built once, materialized, and read by several consumers. Its benefit depends on the readers, their alternatives, and the cost of materialization and reuse. The paper presents Volcano-based and greedy cost-based heuristics rather than a general exhaustive-optimum guarantee. - -[Materialized-view selection in data cubes](https://dsf.berkeley.edu/cs286/papers/datacube-sigmod1996.pdf) studies which aggregate views to retain under a space constraint, exploiting relationships between views in a lattice. Its greedy results rely on that problem's benefit and constraint structure. General workload DAGs can have overlapping sharing opportunities and incompatible configurations, so the same approximation guarantee requires a separate proof. - -### Mathematical optimization: MILP - -[Trummer and Koch](https://arxiv.org/abs/1511.02071) encode join-order and implementation decisions as binary variables with constraints restricting legal plans. Their formulation searches left-deep plans and approximates operation costs with linear functions. A solver can return a feasible incumbent, a bound, and an optimality gap for the formulation. - -This separates the combinatorial search from its implementation, but the encoding must cover the intended alternatives and resource constraints. Solver optimality concerns the encoded objective. Cost approximations and omitted physical choices determine how that result relates to the original planning problem. - -### Bounded and randomized search - -Greedy choice, local improvement, beam search, and randomized exploration limit the planning effort spent on large spaces. [PostgreSQL documents GEQO](https://www.postgresql.org/docs/17/runtime-config-query.html#RUNTIME-CONFIG-QUERY-GEQO) as a heuristic alternative for large join problems that can produce plans inferior to exhaustive search. - -These techniques can find useful plans quickly. Used only to order a complete search or provide feasible incumbents, they do not remove its eventual optimality guarantee. Used to discard alternatives or stop without a certificate, they trade completeness for planning time. - -## Implications for ASAPPlanner and proposed approach - -| Approach | Useful idea for ASAPPlanner | Exactness boundary | +| Approach | How it searches | Guarantee | | --- | --- | --- | -| System R-style DP; DPccp/DPhyp [1](https://www.vldb.org/conf/2006/p930-moerkotte.pdf) | Memoize subproblems; exploit sparse interaction graphs; retain properties a parent needs. | A single cheapest subplan is safe only with optimal substructure. DPccp's cross-product-free join space is a restriction, not proof that arbitrary workload sharing is separable. | -| Volcano/Cascades [2](https://15721.courses.cs.cmu.edu/spring2019/papers/22-optimizer1/graefe-ieee1995.pdf) | Groups of equivalent computations, required physical properties, cost limits, and memoized exploration. | A memo shares optimizer work. It does not automatically decide runtime sharing, materialization, or a correct state key. | -| MQO and view selection [3](https://arxiv.org/abs/cs/9910021), [4](https://github.com/ProjectASAP/ASAPPlanner/issues/671#issuecomment-6137638876) | Model build-once/read-many decisions jointly; use greedy materialization as a practical incumbent. | Greedy MQO is heuristic. Approximation guarantees for a particular data-cube lattice or benefit function do not transfer to arbitrary ASAP DAGs. | -| MILP [5](https://arxiv.org/abs/1511.02071) | Explicit coupled decisions, relaxations as lower bounds, solver incumbents and gaps. | Exact only for the encoded choices and objective. Linear cost approximations and omitted sharing modes can change the optimum. | -| Rule passes and bounded/randomized search | Cheap simplification, branch ordering, and a predictable planning budget. | Restricting exploration is heuristic unless omitted alternatives are proved dominated. PostgreSQL explicitly documents this tradeoff for GEQO [6](https://www.postgresql.org/docs/17/runtime-config-query.html#RUNTIME-CONFIG-QUERY-GEQO). | - -**Decision:** build a streaming, memoized branch-and-bound search with conservative interaction analysis. Enable a pruning rule only when its preconditions establish that it preserves the optimum. Use greedy search to find incumbents and order exploration; discarding alternatives by a heuristic is a separately reported approximation mode. A memo, dynamic programming, or a MILP solver does not by itself make workload sharing exact. - -The implementation boundaries are documented in [`stage1_to_stage3.rs`](../../../crates/plan-selection/src/pass/stage1_to_stage3.rs); sharing rules in [`asap_aware_cse.rs`](../../../crates/logical-asap-optimization/src/asap_aware_cse.rs); and resource aggregation and ranking in [`plan-selection/src/lib.rs`](../../../crates/plan-selection/src/lib.rs). Audit those contracts when implementing each rule. The following sections define the search state, pruning conditions, and execution procedure for ASAPPlanner. - -## Search representation and interaction analysis - -### Memo groups and boundary state - -**Memo group** is established database-optimizer terminology: a group of logically equivalent expressions in the optimizer's memo. See [Graefe's Cascades paper, Section 2](https://15721.courses.cs.cmu.edu/spring2019/papers/22-optimizer1/graefe-ieee1995.pdf) and [CockroachDB's optimizer documentation, “Memo”](https://github.com/cockroachdb/cockroach/blob/master/pkg/sql/opt/doc.go#memo). Here, a group represents a computation with the same observable semantics. Its alternatives describe exact or summary implementations, their output guarantees, and legal physical realizations. Similar source names, matching hashes, or equal output column names do not establish equivalence. - -A search state records the unresolved choices and the interface through which already chosen work can affect them. Include, when observable: - -- source identity, predicates, grouping, value/weight semantics, schema and row identity; -- summary family, parameters, composed output guarantees, and consumer requirements; -- evaluation frequencies, aligned windows, offsets, pane layout, and retention horizons; -- ingestion/query phase, retained versus temporary output, storage medium, and required physical properties; -- offered shared producers and consumer bindings, their adapters, and which costs are already counted; -- boundary resource demands and active source-retention horizons needed for later composition. - -Consumers with different accuracy targets may need a shared producer sized for all of them. Sharing opportunities can be partial, overlap, or arise after an earlier rewrite. The state cannot be just a query ID, a count of consumers, or a scalar cost. - -Two states may reuse a cached optimum only when every future completion observes the same interface and the same cost/evidence context. Hashes only find possible matches; verify structural and semantic equality. A bounded cache may evict entries and recompute later without losing exactness; a bounded beam discards search alternatives and does lose it. - -### Conservative interaction hypergraph - -Represent decisions as variables, and interactions as factors/hyperedges. Examples are a producer and all its potential readers, a summary shared by three consumers, a raw source's retention charge, and a workload resource constraint. Pairwise projection is sufficient to find connected components if it conservatively includes all members of every hyperedge. - -Build potential sharing from **all alternatives and the registered rewrite closure**, not from the independent winner of each query. Include identical expression, capability, rollup, top-k, window/pane reuse, and within-query dependencies. A common metric name is neither necessary nor sufficient: aliases can name one source, and different predicates can prevent sharing. - -Each sharing rule must expose a conservative summary of the sources, producers, consumers, and properties it might connect. If the analyzer cannot rule out an interaction, keep the decisions connected. New rules default to “may interact” until their interaction summary is implemented. False positives cost planning time; false negatives invalidate exactness. - -Whole-workload factors can sometimes be composed after solving computational components, but are not ignored. For example, raw disk retention is the **maximum** requested horizon per source in the current cost model, not the sum of each query's standalone disk use. Preserve that source interface or join its components. The analyzer must be conservative over transformations that introduce a new compatible window or producer, not merely direct matches in the initial graph. - -## Exact pruning rules and their proof conditions - -Here “exact” means the stated conditions are satisfied. It is a correctness property, not a promise that a rule removes many candidates or makes the workload fast. +| [System R-style DP](https://research.ibm.com/publications/access-path-selection-in-a-relational-database-management-system) | Builds larger plans from smaller subproblems, retaining alternatives with useful physical properties. | Exact within its enumerated plan space when the state captures every property needed by a parent. | +| [Volcano/Cascades](https://15721.courses.cs.cmu.edu/spring2019/papers/22-optimizer1/graefe-ieee1995.pdf) | Stores equivalent expressions in memo groups; explores alternatives on demand with cost limits. | Requires sufficient context in memo keys and valid pruning bounds. | +| [Multi-query optimization](https://arxiv.org/abs/cs/9910021) | Chooses which common computations to build, materialize, and reuse across queries. | Greedy sharing is practical but generally heuristic; consumers' choices are coupled. | +| [MILP](https://arxiv.org/abs/1511.02071) | Encodes plan choices and constraints for a solver. | Optimality applies to the encoded alternatives and objective, including any cost approximations. | +| [Bounded/randomized search](https://www.postgresql.org/docs/17/runtime-config-query.html#RUNTIME-CONFIG-QUERY-GEQO) | Limits exploration to control planning time. | Can miss the exhaustive optimum. | -The safest first implementation keeps exhaustive generation semantics and enables only duplicate elimination, completed-plan rejection, and an initially zero lower bound. Those preserve the result without assuming useful sharing savings. Component splitting, local DP, and forced sharing require additional certificates and remain disabled when their preconditions are unknown. Streaming is an execution strategy, not a new pruning theorem. +For ASAPPlanner, retain equivalent search states, reject impossible branches, and prune only with a valid workload-level bound or replacement proof. Sharing prevents a simple “keep the cheapest plan per query” DP from being exact. -### E1. Remove duplicate search states and equivalent complete plans +## Exactness contract -If two states have the same full interface, remaining legal choices, and resource/cost obligations, expand one and reuse its result for the other. Canonical node IDs or traversal order must not affect identity. Preserve alias/provenance information required by explanations and tie handling. +“Exact” means preserving the optimum of the existing finite candidate space under the configured cost, accuracy, and capability models. It does not guarantee the fastest measured runtime or cover alternatives the generator never offers. Freeze the workload, rule/parameter inventory, and model/evidence inputs for each search. -**Proof:** each discarded state has exactly the same completions and costs as its representative. This avoids repeated optimizer work without forcing shared runtime execution. Registering a legal common expression in a memo is different from deciding to materialize it once. +A memo group is standard optimizer terminology for logically equivalent expressions; see [Cascades, Section 2](https://15721.courses.cs.cmu.edu/spring2019/papers/22-optimizer1/graefe-ieee1995.pdf) and [CockroachDB's optimizer documentation](https://github.com/cockroachdb/cockroach/blob/master/pkg/sql/opt/doc.go#memo). Approximate alternatives must also carry their output-accuracy obligations. -### E2. Reject permanently invalid branches +A reusable search state must expose everything later choices can observe: source and expression semantics, schema, summary parameters and guarantees, consumers, windows, recurrence, retention, materialization, physical properties, and resource obligations. Unknown interactions remain connected; a matching hash alone never proves equivalent states. -Prune when no remaining completion can meet semantics, deployment capability, executability, or the query's output-accuracy policy. Examples are a fixed operator with no supported implementation and no remaining replacement, or a fixed unrepairable type error. A fixed raw-data requirement with raw retention forbidden is another example, provided no unresolved materialization can remove the query-time read. +## Exact pruning methods -Per-query checks may run early only for this invariant subset. Otherwise carry obligations forward: shared resizing can improve accuracy; a different family can supply evidence; ingestion materialization can remove a raw read; later lowering can change an operator shape. Recheck final shared output guarantees and executor support on completed plans. +Each method applies only when its stated conditions hold. Otherwise retain the alternative. These are correctness conditions, not promises of a particular pruning rate. -Constraint propagation is another exact case: remove a domain value only after proving it has no support in a required constraint, even in a relaxation of the other decisions. An empty domain then rejects the branch. Pairwise arc consistency can be a cheap necessary check, but passing it does not prove that a globally feasible completion exists. +### 1. Eliminate duplicate states -**Proof:** the branch's feasible completion set is empty. Treating absent evidence as rejection of a completed plan can follow the existing policy; treating it as permanent rejection before all relevant transformations is a different and generally unsafe step. +Expand one representative when two states have the same observable interface, remaining choices, and cost/resource obligations. Verify equality after hashing and retain required provenance and tie information. -### E3. Prune by an admissible workload lower bound +This is exact because both states have the same completions. Memoizing equivalent computations does not force their runtime results to be shared. -For a partial state `S`, compute `LB(S)` such that `LB(S) <= min C(P)` over its feasible completions. If a fully validated incumbent has objective `U`, discard `S` when its bound cannot beat the incumbent under the chosen tie rule. The default conservative scalar comparison is strict `LB(S) > U`, with numerical margins described below. +### 2. Reject permanently invalid branches -A currently composed partial plan's cost is **not** automatically a lower bound: later sharing can remove work, and later rewrites can change the physical implementation. Only irrevocable, nonnegative charges can be accumulated without a proof of the remainder. The next section defines progressively stronger relaxations. +Reject a branch only when no remaining choice can satisfy semantics, capabilities, executor support, or output accuracy. Examples include an unrepairable type error or a fixed unsupported operator with no legal replacement. -**Proof:** every completion costs at least the bound, so none improves the incumbent. Prune also when a conservative lower bound on a hard resource exceeds its limit. A sum that double-counts possibly shared memory or CPU is not a valid resource lower bound. +Defer failures that another choice can repair: resizing can improve accuracy, changing summary family can supply evidence, and ingestion materialization can remove a query-time raw-data requirement. Recheck completed plans with the authoritative validator. -### E4. Replace a state by a contextually dominating state +### 3. Prune using a workload lower bound -For every completion of state B, there must be a feasible completion of A that preserves each query's output contract, exposes compatible boundary producers/properties, and is no worse on the requested objective and constrained resources. Then B can be removed. In a complete-plan comparison there is no unresolved context; use the actual whole-plan ranking directly. +For a partial state `S`, require `LB(S) <= cost(P)` for every feasible completion `P`. With a validated incumbent of cost `U`, prune only when `LB(S) > U`, conservatively accounting for numeric error and the selection tie policy. A resource lower bound exceeding a hard limit also rejects the branch. -A practical sufficient certificate for partial states is the same boundary interface and a resource/cost vector no larger in every coordinate consumed by the monotone composition function. Keep alternatives with different useful orderings, summary capabilities, source horizons, recurrence/retention requirements, or accuracy guarantees unless the substitution proof covers those differences. A cheaper standalone candidate with fewer reusable outputs is not a dominating state. +Start with zero for nonnegative objectives, then add proven unavoidable, nonshareable charges, counted once. Potentially shareable or replaceable work contributes zero until a stronger bound is proved. A partial plan's current cost, or a sum of standalone query costs, is not automatically a lower bound. -**Proof:** mapping every potentially winning completion of B to a no-worse completion of A preserves the optimum. Equal vectors are deduplicable only under the declared representative/tie policy. Context-free per-query Pareto pruning is not justified just because several metrics are recorded. +### 4. Remove contextually dominated alternatives -**Example: preserve benefits to other queries.** Query A can use a producer costing 10 plus private work costing 6, or an independent alternative costing 5. Query B costs 100 independently, but a permitted reuse rewrite can answer it from A's producer with 1 unit of adapter work. Keeping the producer gives a joint cost of `10 + 6 + 1 = 17`; removing that opportunity gives `5 + 100 = 105`. A replacement of A must preserve B's access to that producer, or account for the loss in the whole-workload comparison. The producer's own cost is not an upper bound on the saving it enables for other consumers. +Discard B only if every completion of B can be replaced by a feasible completion of A with compatible outputs and reusable producers, no higher objective, and no worse constrained resources. A sufficient case is an identical boundary interface with a componentwise no-worse resource/cost vector under a monotone composition rule. -### E5. Solve truly independent components separately +Keep useful resource tradeoffs and sharing capabilities. For example, a producer costing 10 plus private work costing 6 can let another query cost 1 instead of 100. Replacing that first query with a standalone plan costing 5 can lose the joint plan costing 17. Standalone price alone does not establish dominance. -If `F = F1 × ... × Fk`, feasibility factors, and `C(P) = sum Ci(Pi)`, choose a scalar optimum independently in each component. Potential interactions must have been analyzed conservatively as above. +### 5. Split independent components -When feasibility is globally coupled, keep sufficient component alternatives and combine them through a global resource DP or search. A scalar cheapest plan per component can be infeasible in combination. If source-retention horizons or other properties cross the boundary, retain those properties or merge the components before applying the rule. +Analyze possible interactions across **all** alternatives: identical computations, capability sharing, rollup, top-k reuse, windows, and shared resource charges. Include interactions exposed by later rewrites. If independence is unknown, keep the components together. -**Proof:** the product of independent local optima minimizes the separable sum. With constraints, the same proof applies only after conditioning on a sufficient resource/boundary assignment. Merely reading different metrics does not prove independent capacity constraints. +Choose each component's scalar optimum separately only when feasibility factors and the objective is additive. With global capacity limits, retain and combine resource frontiers instead. Include cores as well as memory, disk, and objective dimensions. Raw retention uses the maximum horizon per source, so source accounting can couple otherwise separate computations. -### E6. Eliminate local materialization choices only behind a complete interface +### 6. Resolve isolated materialization choices locally -An unshared build may still share a raw scan, overlap ancestor working memory, change required raw retention, or affect the latency path. Fix its boundary inputs, outputs, lifetimes, and physical requirements first. With no remaining interactions and a scalar separable objective, choose the cheapest legal local realization. With resource tradeoffs or an observable output property, retain the nondominated alternatives for that interface. +Choose the cheapest local realization only after fixing its inputs, outputs, lifetimes, and required properties, and proving that it has no remaining cost or resource interaction. Otherwise retain nondominated realizations for that interface. -For example, local option A costs 1 and uses 10 memory units, while B costs 2 and uses 1. Under a memory limit of 2, discarding B for A is incorrect even when no other query reads this build. A subtree DP is exact only after all externally visible dependencies are represented in its state. +An unshared build is not automatically isolated: it can share a scan or affect ancestor working memory and raw retention. Under a memory limit of 2, alternatives `(cost 1, memory 10)` and `(cost 2, memory 1)` require keeping the second option. -### E7. Force identical-expression sharing only with a physical dominance proof +### 7. Force sharing only when it dominates recomputation -The existing [`ir/cse` legality checks](../../../crates/types/src/ir/cse.rs) establish which expressions may share without changing results; structural equality alone is insufficient for some unkeyed outputs. They do not prove that every shared physical realization is cheaper. +First apply the existing [CSE legality checks](../../../crates/types/src/ir/cse.rs). Then eliminate recomputation only if the shared physical realization is no worse on every relevant cost, resource, and consumer property. -A sufficient restricted case is a pure deterministic computation with the same inputs, evaluation domain, schema, precision, and required properties; executing it once preserves consumer scheduling, and every relevant build/read/storage/latency/resource charge is no larger. No extra retention, copying, synchronization, or incompatible placement may be introduced. Under that cost model, remove duplicate execution choices. Otherwise retain shared and recomputed alternatives, while still memoizing their common semantics. +Equal expressions alone do not prove this: sharing can extend retention or introduce copying, synchronization, or scheduling costs. When dominance is unproved, retain both physical alternatives and memoize their common semantics. -**Counterexample:** two evaluations can each compute a temporary result in memory 1, while a shared retained realization holds that result across a much longer horizon or must span multiple time slices. Saving CPU does not dominate retained memory. Different media, concurrency, and consumer schedules also require comparison. +## Search and validation -### E8. Condition on a separator, then optimize the remainder +Generate candidates lazily and stream completed plans through the existing validation and ranking path. Apply the rules above as soon as their preconditions are known. Unknown bounds and numerical near-ties remain searchable. Preserve legacy selection order for tolerance-sensitive ties unless a new total-order policy is explicitly adopted; stable IDs alone do not reproduce a nontransitive tolerance comparison. -The survey's “enumerate shared nodes, then DP the rest” is exact when the separator fixes **all** coupling: producer choices, consumer bindings, sizes/guarantees, materialization, windows/retention, shared raw charges, and relevant resource interfaces. Given that assignment, solve conditionally independent residual subproblems and compose their resource profiles. +Exact search finishes when every remaining branch has been explored or certified unable to improve the result. A budget stop reports the best feasible plan found, unresolved work, and whether optimality is certified. No incumbent means “no feasible plan found,” not “infeasible.” Report a scalar gap only with a bound covering all unresolved branches; report Pareto-front completeness separately. -Keeping only a shared node's presence bit is insufficient. Use variable elimination or tree-decomposition DP with full separator states. Complexity depends on domain sizes and induced width of the entire interaction graph, not simply the number of shared nodes or queries. Global resource constraints can enlarge the interface. This is an optional second search engine after conservative branch-and-bound, not a prerequisite for the first implementation. +Validate each rule against exhaustive enumeration on small workloads, alone and in combination. Include donor-dependent sharing, global memory/core limits, overlapping reuse, per-source retention, repairable accuracy failures, and near ties. Verify bounds against the best exhaustive completion of each tested partial state. Measure planning time and peak memory on Examples 1–4, the mixed workloads, and Kubernetes recording rules without claiming a universal scaling guarantee. -## Bounds and branch-and-bound +## Future work -### Safe bounds before sophisticated costing - -Start with bounds whose validity is easy to audit: - -1. **Zero:** valid for the configured nonnegative scalar objective. Weak but safe; reject negative or undefined inputs rather than assuming positivity. -2. **Mandatory exclusive work:** charge a proven lower bound for operations that every completion must execute and cannot share or replace. Charge each distinct operation once. Set potentially shareable producers, adapters with unknown implementations, and unresolved materializations to zero. -3. **Independent relaxed subproblems:** drop selected coupling constraints, allow conflicting choices to coexist, and optimize the remaining relaxed domains. The relaxation must contain every legal completion. Only sum terms whose accounting prevents double-counting; a sum of standalone query optima generally does not meet that requirement. -4. **LP relaxation:** encode candidate/build/binding choices and valid cost/resource under-estimators, then relax integer decisions to continuous ones. The relaxation optimum is a bound on the encoded exact objective when every original feasible plan maps into the relaxed model. A surrogate cost with no one-sided relation supplies no certified bound. - -If a “sharing saving” is used, it must upper-bound the **total workload** saving under all completions, include benefits to other consumers, and count overlapping savings conservatively. That proof is harder than using an explicit relaxation. Use workload-level bounds rather than subtracting a query-local sharing estimate from its standalone cost. - -An admissible bound need not be consistent between parent and child states for branch-and-bound correctness. A weak or zero bound can mean exhaustive exploration. Prefer a cheaply evaluated bound unless stronger pruning pays for its own planning overhead. - -### Incumbents and resource bounds - -Construct an initial feasible plan using standalone choices, then try compatible sharing and local improvement. A heuristic is useful here: each validated plan is a legitimate upper bound, even if it is far from optimal. If standalone choices violate a global budget, they are not an incumbent; search for a feasible one and report none if no plan has yet been found. - -For cluster constraints, derive lower bounds on peak memory, disk, and cores using the same accounting contracts as the final model. Drop positive terms or relax dependencies when safe; do not add mutually exclusive configurations. A latency bound can come from an unavoidable dependency path with lower-bounded operator costs, not from summing work that can run in parallel. - -For a requested Pareto front, an incumbent front may prune a branch only when every completion is dominated under the final strict-dominance/equality policy. A componentwise lower-bound vector helps: a feasible incumbent strictly dominating that bound dominates all completions under monotone aggregation. Equality is insufficient if equal-vector plan identities must be returned. A bound gap on one scalar objective does not certify an entire Pareto front. - -### Numerical comparisons and ties - -Current scalar ranking uses relative floating-point tolerance and retains an earlier candidate on a tie. Honor the tie contract above; do not let a different summation order create artificial dominance. Lower-bound arithmetic must round conservatively downward, or carry a justified error interval and prune only when its lower endpoint is strictly worse than the incumbent's upper endpoint, allowing for the ranking tolerance. An arbitrary epsilon is not a proof of numerical safety. Near ties and unknown bounds stay in the search. Stable ranks suffice for exact ties under a total ordering; the legacy tolerance case requires the additional compatibility handling described above. - -## Composition under the three cost models - -The implementation's `ResourceUsage` is the authority; cost model numbers are not evidence of additivity. In particular, model 1 can include a shared raw-retention disk charge and model 2 explicitly prices peaks. - -| Model | Final choice | State needed for safe composition | -| --- | --- | --- | -| 1: `UsageOverTime` | Price CPU, scans, average memory, and configured disk use. | Charge each physical producer and execution frequency once; preserve per-source raw-retention horizons and other nonadditive accounting. Scalar component optima require proven separability, including the active prices. | -| 2: `ProvisionedCapacity` | Price peak memory, peak cores, and disk. | Keep peak/resource alternatives. Follow the current model's overlap assumptions; summing standalone price is not generally valid. | -| 3: `ClusterCapacity` | Reject memory/disk/core violations; choose lowest latency, least memory, or a Pareto front. | Carry constrained dimensions including **cores**, objective tie dimensions, and any boundary source/physical properties needed by composition. A front only over memory, disk, and latency can discard the sole core-feasible plan. | - -A vector is sufficient only with a proved monotone composition rule for that interface. Under the current conservative all-evaluations-overlap memory model, some memory terms add. Disk retention combines by maximum per source; other retained state terms add. Worst-query latency combines by maximum for independent query groups, but changing a shared producer changes consumer paths. If a later model uses actual scheduling, an aggregate peak number may no longer suffice; retain a lifetime profile or disable that reduction. - -For truly independent components with additive resource coordinates, combine their frontiers by resource DP (or exact enumeration) and discard dominated combined states. Use the same boundary signature during dominance comparisons. Exact continuous/large-integer frontiers may be large: rounding to buckets or capping a frontier is heuristic unless it has a separately proved approximation guarantee. Peak cores can be a feasibility dimension even when the displayed final front is over memory, disk, and latency only. - -## Heuristics and what they give up - -These are useful strategies, but their role must be explicit: - -| Strategy | Safe use in an exact search | Approximate use that loses completeness | -| --- | --- | --- | -| Best standalone / greedy MQO / local improvement | Produce and validate incumbents. | Return the heuristic winner without finishing or certifying the remaining search. | -| Estimated savings, learned cost, query “difficulty” | Choose variable and alternative order; keep every unresolved branch. | Prune on a predicted saving or a learned lower bound without a validity certificate. | -| Per-query top-k, beam search, capped memo/front | Retain all alternatives externally and revisit them, or use only for ordering. | Drop all but k alternatives, potentially losing a donor producer or resource tradeoff. | -| Split by metric, observed sharing, or connected components of current winners | Schedule exploration in groups. | Assume those groups cannot interact under unexamined alternatives. | -| Always share equal expressions; always use the cheapest local materialization | Apply only after E6/E7 conditions are proved. | Force those choices on CPU-only intuition while ignoring retention and resource effects. | -| Epsilon-dominance, resource buckets, coarser sketch/window parameters | Order candidates or supply a known relaxation with one-sided bounds. | Remove near-equivalent states or configurations; exactness then holds only for a deliberately restricted domain, if at all. | -| Time, candidate, or memory budget | Pause with the full unresolved frontier represented; resume later. | Discard unresolved branches and claim completion, or report a gap that ignores dropped work. | - -Greedy benefit does not automatically have diminishing returns here: resizing for additional consumers, overlapping reuse choices, and raw-input alternatives can create complements and conflicts. Do not quote a `1 - 1/e` approximation factor without proving the necessary objective and constraint structure for this model. Randomized search must record its seed; reproducibility is not optimality. - -## Search procedure and stopping contract - -The initial implementation uses depth-first branch-and-bound with a memo and lazy candidate construction. Best-first search can be added later, but keeping every frontier state in RAM can recreate the original memory problem. Depth-first search must retain the bounds of pending siblings and not silently drop work under memory pressure. - -```text -freeze workload, rewrite/parameter inventory, deployment inputs, and evidence -build conservative interaction analysis and candidate groups lazily -validate heuristic completions to seed an incumbent (or no incumbent yet) -search(partial state): - reuse a memo result only under an equivalent full interface - reject permanently invalid branches (E2) - bound all legal completions with a conservative relaxation - reject branches certified infeasible or unable to improve (E3/E4) - if complete: - run the authoritative validity, resource, and ranking checks - update the incumbent or requested Pareto front - otherwise: - choose an unresolved decision; enumerate all legal alternatives lazily - explore promising alternatives first; record bounds for pending siblings - stop or suspend only with the unresolved work represented in the report -``` - -Completeness must hold across all stages. A streaming Stage 3 is insufficient if Stage 1 first constructs a whole Cartesian product or Stage 2 first constructs every materialization. Share unchanged IR through immutable references; instantiate full physical plans only when needed. Diagnostic candidate dumps are an opt-in stream with their own storage limits, not mandatory retained search state. - -Do not enumerate a full product to calculate a search percentage. After duplicate elimination and context-dependent rewriting, its size may be unknown. Report concrete counts and bounds rather than a fabricated fraction. - -For scalar minimization report: - -- the fixed rule/parameter/model/evidence versions and whether the candidate universe was restricted; -- feasible incumbent and its objective `U`, or “no feasible plan found”; -- a global lower bound `L`, or “bound unavailable”; when an incumbent exists, `L` must cover it and every unresolved branch, so use `min(U, all unresolved bounds)`; -- additive gap `U - L`; relative gap `(U - L) / abs(U)` when `U != 0`; an explicit zero-objective convention rather than division by zero; -- status: optimum certified, stopped with incumbent, stopped without incumbent, or infeasibility proved; -- stop reason, time/peak memory, generated/expanded/pruned counts, rule-specific certificates, memo hits, and remaining work; -- whether a heuristic **discarded** states, as distinct from merely ordering them. Include those states in a valid bound or mark the gap unavailable. - -A budget can end with an optimality certificate if all remaining bounds rule out improvement; exhaustion is not required. Conversely, “no feasible plan found” after a budget is not proof of infeasibility. A Pareto request reports front completeness separately from the status of any selected representative. Gap claims are with respect to the frozen planner model and `F`, not real runtime or a larger omitted rewrite space. - -## MILP as a relaxation and an alternative search - -A facility-selection-shaped model is a useful starting point: choose one complete query alternative per query, activate required producer configurations once, and bind every consumer to a compatible producer or recomputation alternative. Include adapters and consumer-specific readout costs; require the relevant provider whenever its consumer is selected; exclude cycles and incompatible bindings. Producer configurations must encode family, size, window, phase, and retention, not merely “a summary exists.” - -For a finite configuration inventory, binary choices can encode exact costs for those configurations without approximating the cost curve. The formulation is exact relative to `F` only if every legal generated plan is representable and every accepted solver solution is legal. Prove those two mappings on the oracle workloads. A compact model containing only some sharing patterns solves a different, restricted problem. - -Hard constraints must include memory, disk, cores, and per-query accuracy/latency requirements where configured. For example, one raw-source retention variable can represent the maximum selected horizon using constraints for every active reader and a nonnegative disk price; additional constraints or a canonical minimum assignment are needed when the price is zero and the horizon affects feasibility/reporting. Critical-path latency and peak memory need explicit encodings consistent with the current model, not a sum substituted for them. Reprice and validate extracted plans through the existing selection path. - -Relaxing a faithful formulation supplies lower bounds for E3. An integer solver provides incumbents and bounds for its model, subject to numerical tolerances. Piecewise approximations, continuous sketch sizing, and unsupported nonlinear costs require one-sided error bounds before their solver gap certifies anything about the original objective. A final original-model reprice validates the incumbent; it does not turn an invalid surrogate lower bound into a certificate. - -Prototype MILP beside the separator/branch-and-bound engine on small exact instances, and compare objective, feasibility, bound strength, planning time, and memory. Do not make a solver dependency mandatory before the simpler exact reductions have measured benefits. - -## Validation and implementation order - -### Acceptance criteria - -Use an independently retained exhaustive enumerator as the small-instance oracle. Freeze its candidate universe and compare optimum objective, feasibility, and the full requested Pareto vector set. Compare winner IDs only under the explicit stable tie contract. Search-node counts and memory are measurements, not equivalence evidence. - -Each implemented pruning rule must have: - -1. a statement of its preconditions and a proof or replacement witness; -2. an adversarial case where its preconditions do not hold, plus a positive case where the rule prunes; -3. exhaustive equivalence tests with the rule alone and in combination; -4. counters showing its actual planning-time and memory benefit; -5. a safe fallback when its assumptions are unknown. - -### Required adversarial workloads - -| Fixture | Property it tests | -| --- | --- | -| Donor producer costs 10, private work 6, standalone alternative 5; another consumer costs 100 or reads producer for 1. | Preserves donor-dependent reuse when evaluating candidate replacements. | -| Unshared choices `(cost 1, memory 10)` and `(cost 2, memory 1)`, memory budget 2. | Cheapest-local pruning must retain the feasible tradeoff. | -| Two disconnected computation groups each choose cheap/high-memory versus dear/low-memory. | Global capacity requires frontier combination, not independent scalar winners. | -| Equal displayed memory/disk/latency, different core requirements, one over the core limit. | Cores remain a partial-state feasibility dimension. | -| Same raw source with horizons 5 and 10; a rewrite exposes another compatible reader. | Per-source maximum is charged once; interaction analysis covers rewrite closure. | -| A shared producer meets a stricter reader only after resizing; an unsupported provisional form has a supported replacement. | Provisional validity failures are deferred; completed output accuracy is rechecked. | -| Same expression but different recurrence, lifetime, or physical property; sharing increases retained state. | Semantic equality alone cannot force a physical shared realization. | -| Three consumers, partial sharing subsets, overlapping rollup/capability/pane opportunities. | Pairwise savings and consumer counts do not stand in for full boundary state. | -| Structurally similar DAGs with different predicates, weights, unique keys, or guarantees; injected hash collisions. | Memo equality and CSE legality are checked beyond hashes. | -| Multiple traversal orders, equal costs, zero costs, near floating-point ties, and unknown model terms. | Stable ranking, conservative bounds, and no pruning from undefined estimates. | -| Stops before the first feasible plan, after an incumbent, and while pending branches have lower bounds below it. | Honest completion/infeasibility status and global bound accounting. | -| Exact/approximate scalar search and full Pareto reporting with the same incumbent. | A scalar certificate is not mistaken for complete Pareto enumeration. | - -Generate small workloads with mixed recurrence, parameter sizes, accuracy evidence, capacities, and sharing patterns; for every visited branch check `LB <= oracle best completion` when a feasible completion exists. Also compare reported resource lower bounds with oracle completions and validate replacement witnesses. Test that changing a registered sharing rule or an evidence snapshot invalidates affected caches. For bounded search, increasing the expansion budget must never worsen the best objective in a deterministic resumed run; certificates must remain valid at every stop. - -### Implementation sequence - -| Step | Deliverable and guarantee | Measurement | -| --- | --- | --- | -| 1 | Lazy generation and streaming scalar/Pareto selection, unchanged feasible universe, explicit legacy-compatible or total-order tie handling, budget statuses. | Time to first feasible plan, total time, retained candidates, peak memory. | -| 2 | Duplicate-state memoization, permanent-invalidity checks, and initial zero/exclusive-work bounds. | Rule-specific pruning, bound cost, cache hits, oracle equality. | -| 3 | Conservative interaction analysis; separable components or exact component-frontier composition. | Component/separator sizes, frontier sizes, capacity cases, analysis cost. | -| 4 | Validated heuristic incumbents and stronger relaxation bounds; all heuristics used for ordering first. | Bound versus incumbent trajectory, time to certificate, expansion count. | -| 5 | Conditional local DP and sharing dominance only for proven interfaces; optional separator DP or MILP prototype. | Full oracle agreement, interface sizes, time/memory versus simpler search. | -| 6 | Explicit approximate mode with beam/front caps or restricted domains, only if exact mode remains too expensive. | Optimality gap against the oracle, available certificates, reproducible budget/seed. | - -Run the planner-layering Examples 1–4, ASAPQuery's 30 mixed workloads, and the Kubernetes recording-rule workload. Report both exhaustive small-instance quality and large-instance scaling, including timeouts, missing coverage, and no-feasible-plan outcomes. “Tens of queries in seconds under a few GB” is a benchmark target; a giant interaction component or resource frontier may defeat it. Distinguish candidate-generation coverage, pruning speed, cost-model accuracy, and runtime correctness rather than attributing all four to search quality. - -## References - -1. [Moerkotte and Neumann, VLDB 2006: optimal bushy join trees without cross products](https://www.vldb.org/conf/2006/p930-moerkotte.pdf). Connected-subgraph enumeration within a defined join space; it does not justify deleting cross-query sharing choices. -2. [Graefe, 1995: The Cascades Framework for Query Optimization](https://15721.courses.cs.cmu.edu/spring2019/papers/22-optimizer1/graefe-ieee1995.pdf). Memoization, properties, and guided search. -3. [Roy, Seshadri, Sudarshan, and Bhobe, SIGMOD 2000: Efficient and Extensible Algorithms for Multi Query Optimization](https://arxiv.org/abs/cs/9910021). Practical cost-based MQO heuristics, including greedy materialization. -4. [The #671 survey and view-selection references](https://github.com/ProjectASAP/ASAPPlanner/issues/671#issuecomment-6137638876). Background and candidate directions, not proofs of this proposal's pruning rules. -5. [Trummer and Koch: Solving the Join Ordering Problem via Mixed Integer Linear Programming](https://arxiv.org/abs/1511.02071). A finite join-order formulation with approximated costs; distinguish solver optimality from original-model optimality. -6. [PostgreSQL documentation: genetic query optimization](https://www.postgresql.org/docs/17/runtime-config-query.html#RUNTIME-CONFIG-QUERY-GEQO). An explicit example of trading exhaustive search for planning time. -7. [Selinger et al., SIGMOD 1979: Access Path Selection in a Relational Database Management System](https://research.ibm.com/publications/access-path-selection-in-a-relational-database-management-system). System R's cost-based access-path and join planning. -8. [Moerkotte and Neumann, SIGMOD 2008: Dynamic Programming Strikes Back](https://15721.courses.cs.cmu.edu/spring2020/papers/20-optimizer2/p539-moerkotte.pdf). Hypergraph-based join enumeration with DPhyp. -9. [Apache DataFusion: Query Optimizer](https://datafusion.apache.org/library-user-guide/query-optimizer.html). Logical rule passes and optimizer customization. -10. [Harinarayan, Rajaraman, and Ullman, SIGMOD 1996: Implementing Data Cubes Efficiently](https://dsf.berkeley.edu/cs286/papers/datacube-sigmod1996.pdf). Materialized-view selection over a data-cube lattice. -11. [CockroachDB optimizer package documentation: Memo](https://github.com/cockroachdb/cockroach/blob/master/pkg/sql/opt/doc.go#memo). Defines memo groups and illustrates equivalent join expressions in the same group. +- Stronger bounds from LP relaxations; separator/tree-decomposition DP or MILP as alternative search engines. +- Greedy incumbents and learned branch ordering to accelerate exact search. +- Explicit approximate modes such as beam search, top-k candidates, capped resource frontiers, and coarser parameter domains. These may lose the optimum and must be labeled accordingly. +- Incremental replanning and broader candidate/rewrite coverage. From b05b4db8158f5af94f66b87e342c97269fad0c54 Mon Sep 17 00:00:00 2001 From: zzylol <50204836+zzylol@users.noreply.github.com> Date: Tue, 20 Oct 2026 19:13:31 +0000 Subject: [PATCH 07/53] docs: add problem, goals, design overview, interfaces and a worked example to the joint search design --- .../proposals/joint-planning-search.md | 119 ++++++++++++++++++ 1 file changed, 119 insertions(+) diff --git a/docs/design_docs/proposals/joint-planning-search.md b/docs/design_docs/proposals/joint-planning-search.md index 297cae326..df4c97da9 100644 --- a/docs/design_docs/proposals/joint-planning-search.md +++ b/docs/design_docs/proposals/joint-planning-search.md @@ -4,6 +4,59 @@ Status: proposal; this document does not implement new pruning. Audience: planne ASAPPlanner searches per-query alternatives, sharing choices, and materializations to select a whole-workload plan. The design preserves the stages in [planner-layering.md](planner-layering.md) while reducing redundant search. It follows the [survey in #671](https://github.com/ProjectASAP/ASAPPlanner/issues/671#issuecomment-6137638876). +## Contents + +- [Problem](#problem) +- [Goals and non-goals](#goals-and-non-goals) +- [Where the search space comes from](#where-the-search-space-comes-from) +- [Optimizer search survey](#optimizer-search-survey) +- [Exactness contract](#exactness-contract) +- [Design overview](#design-overview) +- [Exact pruning methods](#exact-pruning-methods) +- [Interfaces](#interfaces) +- [Worked example](#worked-example) +- [Search and validation](#search-and-validation) +- [Future work](#future-work) + +## Problem + +Planning a workload jointly lets queries share computation: one summary answers several queries, identical sub-expressions run once, and panes are reused across evaluations. Today Stage 1 to Stage 3 enumerate every whole-workload candidate, price each one, and keep all of them in memory. + +That does not scale past a handful of queries. Planning a growing prefix of one 10-query workload of ASAPQuery's random mixed workloads: + +| Queries | Time | Peak memory | +| ---: | ---: | ---: | +| 3 | 0.4 s | 0.14 GB | +| 4 | 5 s | 1.5 GB | +| 5 | 129 s | 24.7 GB | +| 6 | stopped at about 228 s | about 100 GB | + +Each added query multiplies the work by roughly 20. None of those workloads, and not the Kubernetes recording-rule set, can be planned as a whole. Each query planned alone takes milliseconds, but then sharing across queries is lost. + +## Goals and non-goals + +**Goals** + +- Joint planning of realistic workloads (tens of queries) within seconds and a few GB. +- The same selected plan as exhaustive search over the same candidate space (see [Exactness contract](#exactness-contract)). +- Bounded search that says what it did: when a budget stops the search, the result reports it and whether optimality is certified. + +**Non-goals** + +- Changing what the stages decide. Candidate generation, sharing rules, materializations, and the accuracy and cost checks stay as they are. +- Approximate search modes. They are future work and must be labeled as such. +- Incremental replanning. + +## Where the search space comes from + +| Stage | Choice | Grows as | +| --- | --- | --- | +| Stage 1, Pass 1 | One per-computation candidate per target | the product of the targets' candidate counts | +| Stage 1, Pass 2 | Sharing alternatives of each composed candidate: identical expressions, summary capability, window composition, rollup, top-k reuse | the subsets of the sharing opportunities | +| Stage 2 | One materialization per summary build: at ingestion, at query time, or at query time kept across evaluations | three per build | + +Most combinations cannot be selected. Either a cheaper alternative replaces them in every completion, or their choices in one part of the workload do not interact with another part. The methods below remove those combinations without removing the optimum. + ## Optimizer search survey | Approach | How it searches | Guarantee | @@ -24,6 +77,36 @@ A memo group is standard optimizer terminology for logically equivalent expressi A reusable search state must expose everything later choices can observe: source and expression semantics, schema, summary parameters and guarantees, consumers, windows, recurrence, retention, materialization, physical properties, and resource obligations. Unknown interactions remain connected; a matching hash alone never proves equivalent states. +## Design overview + +The search keeps the stages' semantics and changes only how their choices are explored. + +**Components** + +| Component | Role | +| --- | --- | +| Search state | A partial whole-workload plan: the choices made so far, plus everything later choices can observe (see [Exactness contract](#exactness-contract)). | +| Memo | Equivalent search states, grouped so that each is expanded once (method 1). | +| Interaction graph | Which computations can interact through sharing or shared resource charges. Its connected components are searched separately when that is exact (method 5). | +| Lower bound | A valid lower bound on the cost of every completion of a state (method 3). | +| Incumbent | The best validated complete plan so far, or a resource frontier when global limits apply. | +| Budget | Limits on states expanded or time, with a report of unresolved work when it stops the search. | + +**Search loop** + +1. Build the interaction graph from all alternatives, and split the workload into components where independence is proven (method 5). +2. For each component, generate states lazily, in the existing enumeration order. +3. For each new state, apply in order: + - duplicate elimination (method 1); + - permanent-invalidity rejection (method 2); + - lower-bound pruning against the incumbent (method 3); + - contextual dominance (method 4). +4. When a materialization or sharing choice becomes isolated, resolve it locally (methods 6 and 7) instead of branching on it. +5. Complete plans go through the existing validation and ranking path, and update the incumbent. +6. Combine the components' results: the sum of their optima when the objective is additive and feasibility factors, otherwise their resource frontiers. + +Every pruning step requires its stated precondition. When the precondition is unknown, the alternative stays in the search. + ## Exact pruning methods Each method applies only when its stated conditions hold. Otherwise retain the alternative. These are correctness conditions, not promises of a particular pruning rate. @@ -70,6 +153,42 @@ First apply the existing [CSE legality checks](../../../crates/types/src/ir/cse. Equal expressions alone do not prove this: sharing can extend retention or introduce copying, synchronization, or scheduling costs. When dominance is unproved, retain both physical alternatives and memoize their common semantics. +## Interfaces + +These are design-level interfaces; names in the implementation may differ. + +**Inputs** +- The existing Stage 1 to Stage 3 inputs: workload, planning models, plan selection settings. +- A search budget in plan selection settings: limits on expanded states and on time, with a large default. +- An exhaustive mode, used by tests to compare results. + +**Outputs** +- The selected plan, as today. +- A search report: + - the components found; + - per component: states generated, states pruned by each method, and plans validated; + - whether the search completed; + - if a budget stopped it: the best feasible plan found, the unresolved work, and whether optimality is certified. +- Developer tools can still request the full candidate lists; normal planning keeps only the incumbent. + +## Worked example + +A workload of three recurring queries: + +```promql +q1: sum by (job) (rate(http_requests_total[5m])) +q2: topk(5, sum by (job) (rate(http_requests_total[5m]))) +q3: quantile_over_time(0.99, latency_ms[5m]) +``` + +- **Method 5.** q1 and q2 read the same source and can share the per-job rate sum (identical computation and top-k reuse), so they form one component. q3 reads another source. It has no sharing opportunity with q1 or q2, so under an additive objective with no global limit it is searched on its own. With a global memory or core limit, the two components keep resource frontiers and are combined at the end. +- **Method 1.** In the {q1, q2} component, the composed candidates that differ only in how the identical rate sum is numbered are one state. +- **Method 7.** Sharing the per-job rate sum is forced only if the shared realization is no worse on every relevant cost, resource and consumer property. Otherwise both the shared and the separate realization stay. +- **Method 6.** q3's per-series KLL panes have no interaction with the other component. Their materialization can be chosen locally once their inputs, lifetimes and raw-retention effect are fixed. If the choice changes raw retention for `latency_ms`, the nondominated realizations are kept instead. +- **Method 3.** Once a validated plan for {q1, q2} exists, a partial state whose unavoidable, nonshareable charges already exceed it is pruned. + +The selected plan is the same as exhaustive search's. The number of states expanded drops from the product over all three queries to roughly the sum over the components. + ## Search and validation Generate candidates lazily and stream completed plans through the existing validation and ranking path. Apply the rules above as soon as their preconditions are known. Unknown bounds and numerical near-ties remain searchable. Preserve legacy selection order for tolerance-sensitive ties unless a new total-order policy is explicitly adopted; stable IDs alone do not reproduce a nontransitive tolerance comparison. From 907790abc0160f12a8bb8821ad0b7d85221814f5 Mon Sep 17 00:00:00 2001 From: zzylol <50204836+zzylol@users.noreply.github.com> Date: Thu, 22 Oct 2026 15:08:22 +0000 Subject: [PATCH 08/53] docs: memo-group search design and interfaces for the joint search --- .../proposals/joint-planning-search.md | 107 ++++++++++++++++++ 1 file changed, 107 insertions(+) diff --git a/docs/design_docs/proposals/joint-planning-search.md b/docs/design_docs/proposals/joint-planning-search.md index df4c97da9..8d9b7ab27 100644 --- a/docs/design_docs/proposals/joint-planning-search.md +++ b/docs/design_docs/proposals/joint-planning-search.md @@ -16,6 +16,7 @@ ASAPPlanner searches per-query alternatives, sharing choices, and materializatio - [Interfaces](#interfaces) - [Worked example](#worked-example) - [Search and validation](#search-and-validation) +- [Memo-group search](#memo-group-search) - [Future work](#future-work) ## Problem @@ -197,6 +198,112 @@ Exact search finishes when every remaining branch has been explored or certified Validate each rule against exhaustive enumeration on small workloads, alone and in combination. Include donor-dependent sharing, global memory/core limits, overlapping reuse, per-source retention, repairable accuracy failures, and near ties. Verify bounds against the best exhaustive completion of each tested partial state. Measure planning time and peak memory on Examples 1–4, the mixed workloads, and Kubernetes recording rules without claiming a universal scaling guarantee. +## Memo-group search + +The rules above prune a search that still builds and prices whole-workload candidates, one at a time, at about a millisecond each. RangeHeavy's 24-hour quantiles component checks 153,086 of them in 92 s even with interchangeable queries collapsed, and whole workloads link most queries over one source into one component (shared scans, SQL windows as time ranges, sketch rollup). The memo-group search replaces whole-candidate enumeration: each group of queries enumerates and prices its own alternatives once, keeps the cheapest per sharing signature, and only groups that can share a node make a joint choice. + +### Components + +| Component | What it is | +| --- | --- | +| Memo group | The queries planned together as one unit: one query, every query of a class of interchangeable queries, or queries that read a common Pass 1 target. Identical queries are one group. | +| Group candidate | One alternative of one group, planned alone: a choice of a Pass 1 candidate for each of its targets, the Pass 2 rewrites that touch only its own nodes (panes, sharing inside the group), and one Stage 2 materialization of each build only it can read. | +| Sharing site | A node another group could also read: a summary build (whole, or a pane chain with its width), a shared scan or time range, a rollup source, a top-k ranking. Keyed so that two groups' sites meet exactly when a Pass 2 rule could make them one node. | +| Sharing signature | The sites a group candidate exposes, each with what decides the shared node's cost: family, size parameters, pane grid and materialization. | +| Local usage | A group candidate's usage priced by Stage 3's per-node pricing with the group alone: CPU operations, scanned bytes, memory byte-seconds, peak memory, disk, each query's latency. Priced once per candidate and cached. | +| Sharing cluster | The groups whose signatures can meet, directly or through other groups. Clusters are searched separately (method 5). | +| Cluster selection | One kept candidate per group of a cluster, with the sites they make one node. | + +### Algorithm + +1. **Groups.** Split the workload into memo groups (hash-consing identical queries; classes of interchangeable queries; queries sharing a Pass 1 target). +2. **Enumerate and price each group alone.** For each group, enumerate its candidates and price each with Stage 3 as a workload of that group alone. A candidate that fails a check that sharing cannot repair is dropped (method 2). +3. **Keep the cheapest per signature.** Within a group, among candidates with the same sharing signature, drop a candidate whose local usage is no smaller on every component and larger on one (method 4, local dominance). Ties are kept. A candidate whose sites can meet no other group's has an empty signature, so it competes with every such candidate of the group: the group's overall cheapest. +4. **Clusters.** Link two groups when one kept site of each can meet. Each connected cluster is searched on its own. +5. **Joint choice in a cluster.** Branch and bound over the groups of a cluster, one kept candidate each. A complete assignment is priced exactly: its groups' roots are composed, the meeting sites are made one node by the Pass 2 rule that applies (identical expression, summary capability sized for the strictest reader, shared pane chain), and Stage 2 and Stage 3 price the cluster. The lower bound of a partial assignment is the sum, over its groups, of local usage minus the cost of its sites, plus each distinct site's cost once, sized for its least demanding reader: sharing can only remove repeated site charges. +6. **Whole plan once.** The cluster selections are composed into one plan, checked by Stage 3 (including the executor's compile check) and returned. Runner-ups are built only when the report asks for them. + +The search is exact when every interaction between two groups goes through a sharing site (step 4), and when sharing changes nothing outside the sites it merges. Charges that couple everything (raw data on disk, cost models 2 and 3, a measured cost table) keep the current search. + +### Interfaces + +The memo-group search lives in `crates/plan-selection/src/pass/memo/`, behind the current search until it matches it on every equivalence test. + +```rust +/// One memo group: queries planned together as one unit. +pub(crate) struct MemoGroup { + /// Its queries, by index in the workload, in workload order. + pub queries: Vec, + /// The Pass 1 targets its queries read (digits of a whole-workload choice). + pub targets: Vec, +} + +/// One candidate of a memo group, planned alone. +pub(crate) struct GroupCandidate { + /// Its group, by index. + pub group: usize, + /// The group's Pass 1 choice: one candidate per target, over the group's targets. + pub choice: Vec, + /// The group's roots, in its queries' order, with Pass 2 rewrites inside the group. + pub roots: Vec, + /// One materialization per build only this group reads, in discovery order. + pub materialization: Vec, + /// The sites another group could share. + pub signature: SharingSignature, + /// The group's usage planned alone, priced by Stage 3. + pub usage: LocalUsage, +} + +/// The sites a group candidate exposes, sorted; equal signatures compete. +#[derive(Clone, PartialEq, Eq, Hash, PartialOrd, Ord)] +pub(crate) struct SharingSignature(pub Vec); + +/// A node another group could read once it is shared. +#[derive(Clone, PartialEq, Eq, Hash, PartialOrd, Ord)] +pub(crate) struct SharingSite { + /// Equal keys meet: the shared node's input and computation, sizes and ranks masked. + pub meeting_key: u64, + /// What decides the shared node's cost: family and size, pane grid, materialization. + pub cost_key: u64, +} + +/// One group candidate's usage priced alone. +pub(crate) struct LocalUsage { + pub cpu_ops_per_second: f64, + pub scanned_bytes_per_second: f64, + pub average_memory_bytes: f64, + pub peak_memory_bytes: u64, + pub disk_bytes: u64, + /// Each of the group's queries' latency per evaluation, in ms. + pub latency_ms: Vec, + /// The price under the deployment's cost model, when it has one. + pub cost_per_second: Option, +} + +/// Step 1: the memo groups of a workload. +pub(crate) fn memo_groups(per_computation: &PerComputationCandidates, entries: &[QueryWorkloadEntry], models: &PlanningModels<'_>) -> Vec; + +/// Steps 2 and 3: a group's candidates, priced alone, the dominated ones per signature dropped. +pub(crate) fn group_candidates(per_computation: &PerComputationCandidates, group: &MemoGroup, entries: &[QueryWorkloadEntry], data: &DataWorkload, models: PlanningModels<'_>) -> Vec; + +/// Step 3 alone: `candidates` with every one another of the same signature dominates dropped. +pub(crate) fn keep_undominated(candidates: Vec) -> Vec; + +/// Steps 4 and 5: the clusters of groups whose kept sites can meet, each selected exactly. +pub(crate) fn select_clusters(groups: &[MemoGroup], candidates: &[Vec], entries: &[QueryWorkloadEntry], data: &DataWorkload, models: PlanningModels<'_>) -> Result, SelectionError>; + +/// One cluster's selection: a kept candidate per group, and the plan they compose. +pub(crate) struct ClusterSelection { + pub groups: Vec, + /// Per group, the index of its selected candidate. + pub selected: Vec, + /// The cluster's physical candidate, as Stage 2 built it, priced by Stage 3. + pub physical: PhysicalCandidate, +} +``` + +Step 6 composes the cluster selections the way independent components are combined today, and the facade (`run_stage1_to_stage3_from_pass1`) switches to the memo-group search when its preconditions hold. + ## Future work - Stronger bounds from LP relaxations; separator/tree-decomposition DP or MILP as alternative search engines. From b3ba6bbd759622cabe7897f19764c26866e377e2 Mon Sep 17 00:00:00 2001 From: zzylol <50204836+zzylol@users.noreply.github.com> Date: Thu, 22 Oct 2026 16:38:28 +0000 Subject: [PATCH 09/53] docs: memo over logical sub-expressions, coupled charges and stage interfaces --- .../proposals/joint-planning-search.md | 139 +++++++----------- 1 file changed, 50 insertions(+), 89 deletions(-) diff --git a/docs/design_docs/proposals/joint-planning-search.md b/docs/design_docs/proposals/joint-planning-search.md index 8d9b7ab27..491189ad4 100644 --- a/docs/design_docs/proposals/joint-planning-search.md +++ b/docs/design_docs/proposals/joint-planning-search.md @@ -200,110 +200,71 @@ Validate each rule against exhaustive enumeration on small workloads, alone and ## Memo-group search -The rules above prune a search that still builds and prices whole-workload candidates, one at a time, at about a millisecond each. RangeHeavy's 24-hour quantiles component checks 153,086 of them in 92 s even with interchangeable queries collapsed, and whole workloads link most queries over one source into one component (shared scans, SQL windows as time ranges, sketch rollup). The memo-group search replaces whole-candidate enumeration: each group of queries enumerates and prices its own alternatives once, keeps the cheapest per sharing signature, and only groups that can share a node make a joint choice. +The rules above prune a search that still builds and prices whole-workload candidates, one at a time, at about a millisecond each: RangeHeavy's 24-hour quantiles component checks 153,086 of them in 92 s, and whole workloads link most queries over one source into one component (shared scans, SQL windows as time ranges, sketch rollup). The memo-group search replaces whole-candidate enumeration with a Cascades-style memo over the workload's logical sub-expressions. The stages keep their semantics and code boundaries; what changes is the unit they exchange. ### Components | Component | What it is | | --- | --- | -| Memo group | The queries planned together as one unit: one query, every query of a class of interchangeable queries, or queries that read a common Pass 1 target. Identical queries are one group. | -| Group candidate | One alternative of one group, planned alone: a choice of a Pass 1 candidate for each of its targets, the Pass 2 rewrites that touch only its own nodes (panes, sharing inside the group), and one Stage 2 materialization of each build only it can read. | -| Sharing site | A node another group could also read: a summary build (whole, or a pane chain with its width), a shared scan or time range, a rollup source, a top-k ranking. Keyed so that two groups' sites meet exactly when a Pass 2 rule could make them one node. | -| Sharing signature | The sites a group candidate exposes, each with what decides the shared node's cost: family, size parameters, pane grid and materialization. | -| Local usage | A group candidate's usage priced by Stage 3's per-node pricing with the group alone: CPU operations, scanned bytes, memory byte-seconds, peak memory, disk, each query's latency. Priced once per candidate and cached. | -| Sharing cluster | The groups whose signatures can meet, directly or through other groups. Clusters are searched separately (method 5). | -| Cluster selection | One kept candidate per group of a cluster, with the sites they make one node. | +| Memo group | One distinct logical sub-expression of the workload: the logical DAG hash-consed, so identical expressions of several queries are one group. Identical-expression sharing is group identity. | +| Implementation candidate | One way to implement one group: exact, exact accumulator, a summary family with its size parameters, tumbling panes, or a read of another group (newest panes of a shared chain, a rollup regrouping, a top-k ranking). It references child groups, not child nodes. A fused computation (e.g. a heap that absorbs a sum and a top-k) is one candidate that implements several groups; it records which. | +| Materialization | `NotMaterialized`, `MatAtQuery`, at ingestion, or kept across evaluations: chosen once per implementation candidate of a group, however many parents read it. | +| Shared group | A group read by two or more parents. Its choice (implementation, size, materialization, pane grid) couples its parents. | +| Coupling site | A choice several groups' costs depend on: a shared group's size (sized for its strictest reader), a shared pane chain's grid, longest window and size, a metric's scan label set, a source's raw-data horizon. | +| Local price | One implementation candidate's usage, priced by Stage 3's per-node pricing given its context (its readers' requirements, its coupling sites' thresholds): CPU operations, scanned bytes, memory byte-seconds, peak memory, disk, each reader's latency. Cached per (group, candidate, context). | +| Frontier | Per group and subproblem, the candidates no other dominates in usage: the cheapest under cost model 1, a Pareto front over (cost, peak memory, peak cores, disk, worst latency) under cost models 2 and 3. | + +### Stage interfaces + +- **Pass 1** emits, per group, its implementation candidates (exact, exact accumulators, each summary family and size, pane variants), each referencing child groups. It no longer emits a whole sub-DAG per target. +- **Pass 2 rules** add implementation candidates to groups: + - summary capability: one build serving several reader groups, sized for the strictest reader; + - window composition: pane implementations, a newest-n read of another group's chain, and the gcd grid; + - rollup: a coarse group implemented by regrouping a finer group; + - exact top-k reuse: a top-k implemented from another group's ranking. +- **Stage 2** chooses each candidate's materialization; a group with several parents chooses once. +- **Stage 3** prices per group with the cache. The executor's compile check and the full physical DAG are built only for the selection and the runner-ups a report asks for. ### Algorithm -1. **Groups.** Split the workload into memo groups (hash-consing identical queries; classes of interchangeable queries; queries sharing a Pass 1 target). -2. **Enumerate and price each group alone.** For each group, enumerate its candidates and price each with Stage 3 as a workload of that group alone. A candidate that fails a check that sharing cannot repair is dropped (method 2). -3. **Keep the cheapest per signature.** Within a group, among candidates with the same sharing signature, drop a candidate whose local usage is no smaller on every component and larger on one (method 4, local dominance). Ties are kept. A candidate whose sites can meet no other group's has an empty signature, so it competes with every such candidate of the group: the group's overall cheapest. -4. **Clusters.** Link two groups when one kept site of each can meet. Each connected cluster is searched on its own. -5. **Joint choice in a cluster.** Branch and bound over the groups of a cluster, one kept candidate each. A complete assignment is priced exactly: its groups' roots are composed, the meeting sites are made one node by the Pass 2 rule that applies (identical expression, summary capability sized for the strictest reader, shared pane chain), and Stage 2 and Stage 3 price the cluster. The lower bound of a partial assignment is the sum, over its groups, of local usage minus the cost of its sites, plus each distinct site's cost once, sized for its least demanding reader: sharing can only remove repeated site charges. -6. **Whole plan once.** The cluster selections are composed into one plan, checked by Stage 3 (including the executor's compile check) and returned. Runner-ups are built only when the report asks for them. +1. **Hash-cons** the workload's logical DAG into groups. +2. **Generate** each group's implementation candidates (Pass 1, then the Pass 2 transformations), each with its child groups and its coupling sites. +3. **Clusters.** Groups linked by a shared group or a coupling site form a cluster; clusters are solved separately. +4. **Enumerate the coupling choices of a cluster, with bounds.** A shared group's size depends only on its strictest reader, so instead of reader subsets the search enumerates the strictness threshold (one value per distinct reader requirement, the per-parameter maximum when a size has several parameters); a reader no stricter than the threshold may read it, and does iff that is cheapest for it. Pane chains add the grid (the closure of the readers' widths under gcd) and the longest window. A metric's scans enumerate the label sets of their readers. A source's raw retention enumerates the horizon threshold among its raw-reading readers. Branch and bound prunes threshold combinations whose lower bound exceeds the incumbent. +5. **Dynamic programming** over the rest of the group DAG, given the coupling choices: each group keeps its frontier of candidates (cheapest, or Pareto under cost models 2 and 3), computed bottom-up from its children's frontiers with the cached local prices. +6. **Combine** the clusters' frontiers; peaks and capacity limits (cost models 2 and 3) are checked when frontiers combine. Latency and repeat-interval checks are checked when a reader attaches to a shared group, at that group's threshold. +7. **Build late.** Compose the selected plan once, run Stage 3's checks and the executor's compile check on it, and report it. -The search is exact when every interaction between two groups goes through a sharing site (step 4), and when sharing changes nothing outside the sites it merges. Charges that couple everything (raw data on disk, cost models 2 and 3, a measured cost table) keep the current search. +### Coupled charges -### Interfaces +- **Raw data on disk**: a per-source coupling site whose size is the longest horizon among the readers that scan raw data, enumerated as a threshold (step 4). +- **Cost models 2 and 3**: frontiers instead of minima (step 5), peaks and limits at combination (step 6). +- **A measured cost table**: the calibration factor is fitted once, before the search, over every candidate node the generation produced (before pruning), and frozen; every group is priced with it. -The memo-group search lives in `crates/plan-selection/src/pass/memo/`, behind the current search until it matches it on every equivalence test. +### Exactness + +The search is exact when every interaction between groups is a shared group or a coupling site, and each coupling site's cost depends only on its threshold. When a cluster has a construct no threshold captures, that cluster is searched exhaustively over its frontiers and the report names it; no cap is applied. Each rule is checked against exhaustive enumeration on small workloads (`joint_search_equivalence`), including raw-retention coupling, a cost model 2 peak, a cost model 3 capacity limit and a partial measured table. + +### Code layout + +`crates/plan-selection/src/pass/memo/`, behind `Stage1ToStage3Search::memo_groups` until it matches the current search on every equivalence test: ```rust -/// One memo group: queries planned together as one unit. -pub(crate) struct MemoGroup { - /// Its queries, by index in the workload, in workload order. - pub queries: Vec, - /// The Pass 1 targets its queries read (digits of a whole-workload choice). - pub targets: Vec, -} - -/// One candidate of a memo group, planned alone. -pub(crate) struct GroupCandidate { - /// Its group, by index. - pub group: usize, - /// The group's Pass 1 choice: one candidate per target, over the group's targets. - pub choice: Vec, - /// The group's roots, in its queries' order, with Pass 2 rewrites inside the group. - pub roots: Vec, - /// One materialization per build only this group reads, in discovery order. - pub materialization: Vec, - /// The sites another group could share. - pub signature: SharingSignature, - /// The group's usage planned alone, priced by Stage 3. - pub usage: LocalUsage, -} - -/// The sites a group candidate exposes, sorted; equal signatures compete. -#[derive(Clone, PartialEq, Eq, Hash, PartialOrd, Ord)] -pub(crate) struct SharingSignature(pub Vec); - -/// A node another group could read once it is shared. -#[derive(Clone, PartialEq, Eq, Hash, PartialOrd, Ord)] -pub(crate) struct SharingSite { - /// Equal keys meet: the shared node's input and computation, sizes and ranks masked. - pub meeting_key: u64, - /// What decides the shared node's cost: family and size, pane grid, materialization. - pub cost_key: u64, -} - -/// One group candidate's usage priced alone. -pub(crate) struct LocalUsage { - pub cpu_ops_per_second: f64, - pub scanned_bytes_per_second: f64, - pub average_memory_bytes: f64, - pub peak_memory_bytes: u64, - pub disk_bytes: u64, - /// Each of the group's queries' latency per evaluation, in ms. - pub latency_ms: Vec, - /// The price under the deployment's cost model, when it has one. - pub cost_per_second: Option, -} - -/// Step 1: the memo groups of a workload. -pub(crate) fn memo_groups(per_computation: &PerComputationCandidates, entries: &[QueryWorkloadEntry], models: &PlanningModels<'_>) -> Vec; - -/// Steps 2 and 3: a group's candidates, priced alone, the dominated ones per signature dropped. -pub(crate) fn group_candidates(per_computation: &PerComputationCandidates, group: &MemoGroup, entries: &[QueryWorkloadEntry], data: &DataWorkload, models: PlanningModels<'_>) -> Vec; - -/// Step 3 alone: `candidates` with every one another of the same signature dominates dropped. -pub(crate) fn keep_undominated(candidates: Vec) -> Vec; - -/// Steps 4 and 5: the clusters of groups whose kept sites can meet, each selected exactly. -pub(crate) fn select_clusters(groups: &[MemoGroup], candidates: &[Vec], entries: &[QueryWorkloadEntry], data: &DataWorkload, models: PlanningModels<'_>) -> Result, SelectionError>; - -/// One cluster's selection: a kept candidate per group, and the plan they compose. -pub(crate) struct ClusterSelection { - pub groups: Vec, - /// Per group, the index of its selected candidate. - pub selected: Vec, - /// The cluster's physical candidate, as Stage 2 built it, priced by Stage 3. - pub physical: PhysicalCandidate, -} +/// One hash-consed logical sub-expression. +pub(crate) struct MemoGroup { pub node: Rc, pub parents: Vec, pub queries: Vec } +/// One implementation of one group (or of several, for a fused computation). +pub(crate) struct ImplementationCandidate { pub implements: Vec, pub node: Rc, pub children: Vec, pub coupling_sites: Vec } +/// A choice several groups' costs depend on, keyed so that equal keys are one site. +pub(crate) struct CouplingSite { pub key: u64, pub kind: CouplingKind } +pub(crate) enum CouplingKind { SharedGroupSize, PaneChain, ScanLabels, RawHorizon } +/// The cached local price of a candidate in a context (thresholds of its sites). +pub(crate) fn local_price(group: GroupId, candidate: &ImplementationCandidate, context: &Context, cache: &mut PriceCache) -> Result; +/// The frontier of a group given its children's frontiers and a context. +pub(crate) fn frontier(group: GroupId, context: &Context, memo: &Memo, cache: &mut PriceCache) -> Frontier; +/// The cluster's selection: thresholds enumerated with bounds, the DP below them. +pub(crate) fn solve_cluster(cluster: &Cluster, memo: &Memo, cache: &mut PriceCache) -> Result; ``` -Step 6 composes the cluster selections the way independent components are combined today, and the facade (`run_stage1_to_stage3_from_pass1`) switches to the memo-group search when its preconditions hold. - ## Future work - Stronger bounds from LP relaxations; separator/tree-decomposition DP or MILP as alternative search engines. From cfb01a7bdbcd8e8b5fb90c554db9ed958e38f91d Mon Sep 17 00:00:00 2001 From: zzylol <50204836+zzylol@users.noreply.github.com> Date: Thu, 22 Oct 2026 16:40:49 +0000 Subject: [PATCH 10/53] docs: show how the stages and the memo groups work together, with a worked example --- .../proposals/joint-planning-search.md | 74 +++++++++++++++++++ 1 file changed, 74 insertions(+) diff --git a/docs/design_docs/proposals/joint-planning-search.md b/docs/design_docs/proposals/joint-planning-search.md index 491189ad4..078dfcf77 100644 --- a/docs/design_docs/proposals/joint-planning-search.md +++ b/docs/design_docs/proposals/joint-planning-search.md @@ -214,6 +214,80 @@ The rules above prune a search that still builds and prices whole-workload candi | Local price | One implementation candidate's usage, priced by Stage 3's per-node pricing given its context (its readers' requirements, its coupling sites' thresholds): CPU operations, scanned bytes, memory byte-seconds, peak memory, disk, each reader's latency. Cached per (group, candidate, context). | | Frontier | Per group and subproblem, the candidates no other dominates in usage: the cheapest under cost model 1, a Pareto front over (cost, peak memory, peak cores, disk, worst latency) under cost models 2 and 3. | +### How the stages and the memo work together + +Today the stages run one after another over the whole workload, and each hands the next every whole-workload candidate; the candidate count is the product of every query's choices: + +```text +Stage 0 frontend -> pre-ASAP logical DAG +Stage 1 Pass 1 -> per target, whole sub-DAG candidates (no cost) + Pass 2 -> one whole-workload candidate per sharing subset (no cost) +Stage 2 materialization -> every candidate x every materialization +Stage 3 selection -> every physical candidate built, checked, priced (~1 ms each) +Stage 4 executor +``` + +The memo keeps each stage's semantics and code, and changes what they exchange: groups and implementation candidates instead of whole plans. Each stage runs per group, and only the choices that couple groups are made jointly: + +```text + group A group B group C (shares nothing) +Stage 1 Pass 1 A's implementations B's implementations C's implementations +Stage 1 Pass 2 transformations add implementations that read another group (capability, panes, rollup, top-k) +Stage 2 materialization per implementation, once per group however many parents read it +Stage 3 local price per (group, implementation, context), cached; frontier per group +------------------------- coupling sites of the cluster {A, B} ------------------------- + thresholds enumerated with bounds: a shared group's size (strictest reader), + a pane chain's grid and longest window, a metric's scan labels, a source's raw horizon; + dynamic programming below them; C is solved on its own +------------------------------------------------------------------------------------------- +Selected plan composed once, then Stage 3's checks and the executor's compile check +``` + +Example: two queries over one metric. + +```promql +Q1: quantile_over_time(0.9, data[5m]) +Q2: topk(5, sum by (job) (sum_over_time(data[5m]))) +``` + +Hash-consing their logical DAGs gives six groups; `G2` is one group read by both queries: + +```text +G1 = Scan(data) +G2 = TimeRange(5m) over G1 <- read by Q1 and Q2 +G3 = quantile_over_time(0.9) over G2 (Q1's root) +G4 = sum_over_time over G2 +G5 = sum by (job) over G4 +G6 = topk(5) over G5 (Q2's root) +``` + +Pass 1 and the Pass 2 transformations give each group the implementations of that node alone, referencing child groups: + +```text +G3: exact quantile (reads G2) | KLL(k) (reads G2) | DDSketch(alpha) (reads G2) + | newest-n panes of a shared chain (reads another group's pane chain) +G4: exact sum accumulator (reads G2) +G5: exact group-by sum (reads G4) +G6: exact top-k (reads G5) + | Count-Min heap (reads G2) <- fused: one implementation of G4, G5 and G6 +G1, G2: each reader computes its own (NotMaterialized) | built once per evaluation (MatAtQuery) +``` + +Before, the same `Scan -> TimeRange(5m)` was repeated inside every per-target candidate, and Pass 2 had to discover it again; candidates multiplied across targets. In the memo, `G2` appears once, `G3` has a handful of implementations, and the search is bottom-up: `G2`'s choice is a coupling choice of the cluster `{Q1, Q2}`, and every other group keeps its frontier given it. + +Where each part of the stage design lives in the memo: + +| Stage design | In the memo | +| --- | --- | +| Pass 1 computation candidates | implementation candidates per group, referencing child groups | +| Identical-expression rule | group identity (hash-consing) | +| Summary-capability rule | an implementation serving several reader groups; its size is a coupling site (strictest reader) | +| Window-composition rule | pane implementations in a group; a shared chain's grid and longest window are a coupling site; a reader reads its newest panes | +| Rollup and exact top-k reuse | an implementation of a coarse group (or a top-k) that reads a finer group (or a ranking) | +| Stage 2 materialization | chosen per implementation, once per group | +| Stage 3 pricing and checks | local price per (group, implementation, context), cached; the full plan is checked once | +| Raw retention, cost models 2 and 3, measured table | a per-source horizon site; frontiers; a calibration factor fitted once and frozen | + ### Stage interfaces - **Pass 1** emits, per group, its implementation candidates (exact, exact accumulators, each summary family and size, pane variants), each referencing child groups. It no longer emits a whole sub-DAG per target. From 9a95da6d9758f5b3b04742e4ce31057bbc607892 Mon Sep 17 00:00:00 2001 From: zzylol <50204836+zzylol@users.noreply.github.com> Date: Thu, 22 Oct 2026 16:44:18 +0000 Subject: [PATCH 11/53] docs: end-to-end walk-through of the memo with the sharing strategies --- .../proposals/joint-planning-search.md | 62 ++++++++++++++++++- 1 file changed, 61 insertions(+), 1 deletion(-) diff --git a/docs/design_docs/proposals/joint-planning-search.md b/docs/design_docs/proposals/joint-planning-search.md index 078dfcf77..cdf36360c 100644 --- a/docs/design_docs/proposals/joint-planning-search.md +++ b/docs/design_docs/proposals/joint-planning-search.md @@ -16,7 +16,7 @@ ASAPPlanner searches per-query alternatives, sharing choices, and materializatio - [Interfaces](#interfaces) - [Worked example](#worked-example) - [Search and validation](#search-and-validation) -- [Memo-group search](#memo-group-search) +- [Memo-group search](#memo-group-search): how the stages and the memo work together, and an end-to-end walk-through with the sharing strategies - [Future work](#future-work) ## Problem @@ -288,6 +288,66 @@ Where each part of the stage design lives in the memo: | Stage 3 pricing and checks | local price per (group, implementation, context), cached; the full plan is checked once | | Raw retention, cost models 2 and 3, measured table | a per-source horizon site; frontiers; a calibration factor fitted once and frozen | +### End to end: the memo with the sharing strategies + +This walks one workload through the memo, showing what each sharing strategy (the identical-expression, summary-capability and window-composition rules, rollup and exact top-k reuse) contributes and where each decision is made. All queries read `data`, a gauge. + +```promql +Q1: quantile_over_time(0.9, data[5m]) every 10 s, relative error 0.01 +Q2: quantile_over_time(0.5, data[1m]) every 10 s, relative error 0.05 +Q3: topk(5, sum by (job) (sum_over_time(data[5m]))) every 60 s +Q4: sum by (job, instance) (sum_over_time(data[5m])) every 60 s +``` + +**1. Hash-cons the logical DAGs (identical-expression rule).** Each distinct sub-expression is one group. `Scan(data)` and `TimeRange(5m)` appear once, so every reader of them reads the same group; the identical-expression rule needs no separate pass. + +```text +G1 = Scan(data) read by all four queries +G2 = TimeRange(5m) over G1 read by Q1, Q3, Q4 +G3 = TimeRange(1m) over G1 read by Q2 +G4 = quantile_over_time(0.9) over G2 Q1 +G5 = quantile_over_time(0.5) over G3 Q2 +G6 = sum_over_time over G2 read by Q3, Q4 +G7 = sum by (job, instance) over G6 Q4 +G8 = sum by (job) over G6 read by Q3 +G9 = topk(5) over G8 Q3 +``` + +**2. Pass 1: each group's own implementations.** Pass 1 gives every group the implementations of its node alone: `G4` and `G5` get exact, KLL and DDSketch, each whole-window or in tumbling panes; `G6` to `G9` get exact operators; `G9` also gets a fused Count-Min heap that implements `G6`, `G8` and `G9` at once. + +**3. Pass 2: the sharing strategies add implementations that read other groups.** + +| Strategy | Implementation it adds | Coupling site it creates | +| --- | --- | --- | +| Summary capability | `G4` and `G5` may read one DDSketch build over `data`, sized for the stricter reader (`G4`, error 0.01) | the shared build's size | +| Window composition | that build as one chain of 10 s panes over 5 m; `G5` reads its newest 6 panes, `G4` all 30 | the chain's grid (10 s, the gcd of the readers' widths), its longest window (5 m) and its size | +| Rollup | `G8` (`by (job)`) implemented by regrouping `G7` (`by (job, instance)`) | none beyond `G7` being read twice | +| Exact top-k reuse | where another query keeps the same ranking, `G9` reads it | the ranking's group | +| (scan widening) | `G1`'s readers read one scan with the union of their labels | the metric's label set | + +Nothing here builds a whole-workload candidate; each strategy only adds alternatives to a group. + +**4. Stage 2: materialization per implementation.** For example, the shared pane chain may be kept at ingestion time or rebuilt per evaluation; `G2` may be computed by each reader or built once per evaluation. A group with several parents chooses once. + +**5. Stage 3: local prices, frontiers, and clusters.** Each implementation is priced once per context and cached. Groups linked by a shared group or a coupling site form a cluster; here all nine groups form one cluster through `G1`, `G2` and the shared quantile build. A workload with another metric would form a second cluster, solved independently. + +**6. Joint choice in the cluster.** The coupling choices are enumerated as thresholds, with branch and bound: + +```text +shared quantile build: size threshold in {for G4's 0.01, for G5's 0.05, no shared build} + at 0.01: G5 may read it (0.05 >= 0.01) and does if that is cheaper for G5 + at 0.05: G4 cannot read it (0.01 is stricter), so G4 uses its own build +pane chain: grid in {10 s}, longest window in {5 m, 1 m} +G2, G6: per reader or built once per evaluation +raw retention of data: horizon threshold among the readers that scan raw data +``` + +For each combination that survives the bound, dynamic programming chooses every other group's implementation bottom-up from the cached frontiers. For example, it chooses between `G8` exact and `G8` rolled up from `G7`, and between `G9` exact and the fused heap, given the chosen `G2` and `G6`. The threshold enumeration replaces enumerating subsets of readers; dominance inside each group keeps the frontiers small. + +**7. Build late.** The selected choices are composed into one physical plan. Stage 3 checks it once (guarantees, latency against each repeat interval, capacity) and the executor's compile check runs on it alone. The runner-ups are built only if a report asks for them. + +The result is the plan the exhaustive search would select. It is checked by `joint_search_equivalence` on workloads small enough for `--exhaustive`, one per strategy and per coupling site. + ### Stage interfaces - **Pass 1** emits, per group, its implementation candidates (exact, exact accumulators, each summary family and size, pane variants), each referencing child groups. It no longer emits a whole sub-DAG per target. From e55c6ef44628c5bd3c8c9720897df766a86343ff Mon Sep 17 00:00:00 2001 From: zzylol <50204836+zzylol@users.noreply.github.com> Date: Thu, 22 Oct 2026 17:06:41 +0000 Subject: [PATCH 12/53] docs: explain the memo search's threshold enumeration, branch and bound and dynamic programming --- .../proposals/joint-planning-search.md | 49 +++++++++++++++++++ 1 file changed, 49 insertions(+) diff --git a/docs/design_docs/proposals/joint-planning-search.md b/docs/design_docs/proposals/joint-planning-search.md index cdf36360c..959db6744 100644 --- a/docs/design_docs/proposals/joint-planning-search.md +++ b/docs/design_docs/proposals/joint-planning-search.md @@ -369,6 +369,55 @@ The result is the plan the exhaustive search would select. It is checked by `joi 6. **Combine** the clusters' frontiers; peaks and capacity limits (cost models 2 and 3) are checked when frontiers combine. Latency and repeat-interval checks are checked when a reader attaches to a shared group, at that group's threshold. 7. **Build late.** Compose the selected plan once, run Stage 3's checks and the executor's compile check on it, and report it. +### The search: threshold enumeration, branch and bound, dynamic programming + +Steps 4 and 5 combine three techniques. One example runs through all three: three quantile queries over one metric that may share one DDSketch build, `Qa` with relative error 0.01, `Qb` with 0.02 and `Qc` with 0.05. + +**Threshold enumeration: which readers join a shared node, and how large it is.** Choosing which readers join is a subset choice: 2³ = 8 subsets here, 2ⁿ for n readers. The shared node's size depends only on its strictest member, and a reader no stricter than that size can join without enlarging it (it only adds its own read). So the search enumerates the size, as a threshold set by one reader's requirement, instead of the subset: + +| Threshold (sized for) | Readers that may join | Readers that cannot | +| --- | --- | --- | +| 0.01 (`Qa`) | `Qa`, `Qb`, `Qc` | none | +| 0.02 (`Qb`) | `Qb`, `Qc` | `Qa` builds its own | +| 0.05 (`Qc`) | `Qc` | `Qa`, `Qb` build their own | +| no shared node | none | all build their own | + +Given a threshold, each reader that may join decides on its own, by comparing its read of the shared node with its cheapest own implementation. That is n + 1 thresholds with n independent decisions each, instead of 2ⁿ subsets. A size with several parameters (a KLL's `k` and failure probability) takes each reader's requirement vector as a candidate threshold and drops dominated ones. The other coupling sites follow the same pattern: +- a pane chain enumerates its grid (the gcd closure of its readers' widths) and its longest window; +- a source's raw retention enumerates the horizon; +- a metric's scan enumerates its label set. + +**Branch and bound: which combinations of coupling choices to try.** A cluster usually has several coupling sites, so their thresholds still multiply. The search assigns one site at a time, in a search tree, and prunes: +1. **Branch**: pick the next coupling site and try each of its thresholds. +2. **Bound**: for a partial assignment, compute a lower bound on any completion. Assigned sites are priced exactly. Every unassigned site is priced once, at its least demanding reader's size. Every group is priced at its cheapest frontier candidate given the sites so far. Sharing only removes repeated charges, so no completion is cheaper than this bound. +3. **Prune**: when the bound exceeds the best complete assignment found so far (the incumbent), skip the whole subtree. + +For example, if an incumbent costs 0.50 per second and a branch fixes the source's raw horizon at 24 hours, whose disk charge alone is 0.60 per second, none of that branch's completions can win, so none of them is expanded. Only provably worse branches are pruned, so the result stays exact. + +**Dynamic programming: everything else, once per group.** With the coupling choices fixed, the remaining groups no longer affect each other's cost. Each group's frontier is computed bottom-up from its children's, once, and cached: + +```text +G2 = TimeRange(5m) coupling choice fixed: built once per evaluation +G6 = sum_over_time over G2 exact accumulator only: c6 +G8 = sum by (job) over G6 min(exact over G6: c6 + x, rollup of G7: c7 + y) +G9 = topk(5) over G8 min(exact top-k: best(G8) + z, fused heap over G2: w) +``` + +A parent reads its children's best values; it never multiplies their alternatives with its own. This is where the memo's speed comes from: work proportional to the number of groups and their implementations, not to the product of every query's choices. + +**How the three fit together:** + +```text +for each sharing cluster: + branch and bound over the cluster's coupling sites + each site's candidate values come from threshold enumeration (n + 1, not 2^n) + each complete assignment: dynamic programming over the remaining groups (cached) + keep the cheapest assignment (the incumbent) +compose the clusters' selections into one plan; check and compile it once +``` + +None of the three discards a plan that could be optimal, so the selection equals the exhaustive search's. `joint_search_equivalence` checks this on workloads small enough for `--exhaustive`. + ### Coupled charges - **Raw data on disk**: a per-source coupling site whose size is the longest horizon among the readers that scan raw data, enumerated as a threshold (step 4). From 79f76998d02257c05f6841a4ca8e1d85d2b63ce7 Mon Sep 17 00:00:00 2001 From: zzylol <50204836+zzylol@users.noreply.github.com> Date: Thu, 22 Oct 2026 18:13:11 +0000 Subject: [PATCH 13/53] docs: the memo's solver is a MILP; threshold enumeration gives its discrete site variables --- .../proposals/joint-planning-search.md | 91 +++++-------------- 1 file changed, 21 insertions(+), 70 deletions(-) diff --git a/docs/design_docs/proposals/joint-planning-search.md b/docs/design_docs/proposals/joint-planning-search.md index 959db6744..b73793b4f 100644 --- a/docs/design_docs/proposals/joint-planning-search.md +++ b/docs/design_docs/proposals/joint-planning-search.md @@ -363,90 +363,41 @@ The result is the plan the exhaustive search would select. It is checked by `joi 1. **Hash-cons** the workload's logical DAG into groups. 2. **Generate** each group's implementation candidates (Pass 1, then the Pass 2 transformations), each with its child groups and its coupling sites. -3. **Clusters.** Groups linked by a shared group or a coupling site form a cluster; clusters are solved separately. -4. **Enumerate the coupling choices of a cluster, with bounds.** A shared group's size depends only on its strictest reader, so instead of reader subsets the search enumerates the strictness threshold (one value per distinct reader requirement, the per-parameter maximum when a size has several parameters); a reader no stricter than the threshold may read it, and does iff that is cheapest for it. Pane chains add the grid (the closure of the readers' widths under gcd) and the longest window. A metric's scans enumerate the label sets of their readers. A source's raw retention enumerates the horizon threshold among its raw-reading readers. Branch and bound prunes threshold combinations whose lower bound exceeds the incumbent. -5. **Dynamic programming** over the rest of the group DAG, given the coupling choices: each group keeps its frontier of candidates (cheapest, or Pareto under cost models 2 and 3), computed bottom-up from its children's frontiers with the cached local prices. -6. **Combine** the clusters' frontiers; peaks and capacity limits (cost models 2 and 3) are checked when frontiers combine. Latency and repeat-interval checks are checked when a reader attaches to a shared group, at that group's threshold. -7. **Build late.** Compose the selected plan once, run Stage 3's checks and the executor's compile check on it, and report it. +3. **Coefficients.** Price each candidate's private nodes with Stage 3's per-node pricing, per consumer rate and per materialization, and each coupling site's node at each of its thresholds. Stage 3 stays the only source of prices. +4. **Solve one MILP over the memo** (below): one choice per needed group, its materializations and the coupling sites' thresholds. +5. **Lazy cuts.** Compose the solver's plan, recompose its guarantees and price it with Stage 3. If Stage 3's price exceeds the solver's objective (the held-memory term), or a composed guarantee fails, add a cut that corrects or excludes that plan, and solve again. Stop when Stage 3's price equals the objective and every check passes. The gap must be 0 to report the plan as optimal. +6. **Build late.** Run the executor's compile check on the selected plan only, and report it. -### The search: threshold enumeration, branch and bound, dynamic programming +### The solver: MILP over the memo -Steps 4 and 5 combine three techniques. One example runs through all three: three quantile queries over one metric that may share one DDSketch build, `Qa` with relative error 0.01, `Qb` with 0.02 and `Qc` with 0.05. +Threshold enumeration still turns each coupling site into discrete choices: a shared size set by one reader's requirement, a pane chain's grid and longest window, a scan's label set, a source's raw horizon. The MILP chooses among them instead of a hand-written search. -**Threshold enumeration: which readers join a shared node, and how large it is.** Choosing which readers join is a subset choice: 2³ = 8 subsets here, 2ⁿ for n readers. The shared node's size depends only on its strictest member, and a reader no stricter than that size can join without enlarging it (it only adds its own read). So the search enumerates the size, as a threshold set by one reader's requirement, instead of the subset: +**Variables.** +- `x[g,c]`: group `g` implemented by candidate `c`. Groups a chosen candidate reads must be implemented; a fused candidate implements several. +- `m[s,o]`: materialization `o` of slot `s` (a candidate's build, or a shared group's node). +- `u[site,t]`: coupling site at threshold `t`; a reader may attach only to a threshold it may read. +- `r[k,g]`: consumer `k` (a query, or a materialized slot) reaches group `g` through unmaterialized nodes. +- `y[k,g,c] = r[k,g] ∧ x[g,c]`, linearized as `y ≥ r + x − 1`. -| Threshold (sized for) | Readers that may join | Readers that cannot | -| --- | --- | --- | -| 0.01 (`Qa`) | `Qa`, `Qb`, `Qc` | none | -| 0.02 (`Qb`) | `Qb`, `Qc` | `Qa` builds its own | -| 0.05 (`Qc`) | `Qc` | `Qa`, `Qb` build their own | -| no shared node | none | all build their own | - -Given a threshold, each reader that may join decides on its own, by comparing its read of the shared node with its cheapest own implementation. That is n + 1 thresholds with n independent decisions each, instead of 2ⁿ subsets. A size with several parameters (a KLL's `k` and failure probability) takes each reader's requirement vector as a candidate threshold and drops dominated ones. The other coupling sites follow the same pattern: -- a pane chain enumerates its grid (the gcd closure of its readers' widths) and its longest window; -- a source's raw retention enumerates the horizon; -- a metric's scan enumerates its label set. +**Objective (cost model 1).** Σ rate(k) · per-consumer cost(g, c) · y, plus each materialized slot's own cost, each threshold's site cost, raw retention per source (`R_s ≥ bytes · query-time raw scan`), and pane retention (`keep ≥ span · read`). A slot built once per evaluation for readers on different grids takes the union rate through one indicator per distinct grid set. -**Branch and bound: which combinations of coupling choices to try.** A cluster usually has several coupling sites, so their thresholds still multiply. The search assigns one site at a time, in a search tree, and prunes: -1. **Branch**: pick the next coupling site and try each of its thresholds. -2. **Bound**: for a partial assignment, compute a lower bound on any completion. Assigned sites are priced exactly. Every unassigned site is priced once, at its least demanding reader's size. Every group is priced at its cheapest frontier candidate given the sites so far. Sharing only removes repeated charges, so no completion is cheaper than this bound. -3. **Prune**: when the bound exceeds the best complete assignment found so far (the incumbent), skip the whole subtree. +**Constraints.** One candidate per needed group; down-closure (ingestion needs ingestion below); the reach propagation; attachment only to admitted thresholds; the combination rules (top-k reuse never with panes; rollup source rules); each query's latency (the sum of the work it reaches) within its bound and its repeat interval. -For example, if an incumbent costs 0.50 per second and a branch fixes the source's raw horizon at 24 hours, whose disk charge alone is 0.60 per second, none of that branch's completions can win, so none of them is expanded. Only provably worse branches are pruned, so the result stays exact. +**Lazy terms.** Memory held while a query runs costs bytes × that query's latency, a product of two sums over its reach. The model omits it, which gives a lower bound, and step 5 adds it back through cuts on the incumbent. Composed accuracy guarantees depend on the chosen children; step 5 checks them on the incumbent and excludes failing combinations with no-good cuts. -**Dynamic programming: everything else, once per group.** With the coupling choices fixed, the remaining groups no longer affect each other's cost. Each group's frontier is computed bottom-up from its children's, once, and cached: +**Cost models 2 and 3.** The same variables. Peaks and capacity limits become linear constraints over the per-node peak contributions, and each objective is as specified. -```text -G2 = TimeRange(5m) coupling choice fixed: built once per evaluation -G6 = sum_over_time over G2 exact accumulator only: c6 -G8 = sum by (job) over G6 min(exact over G6: c6 + x, rollup of G7: c7 + y) -G9 = topk(5) over G8 min(exact top-k: best(G8) + z, fused heap over G2: w) -``` - -A parent reads its children's best values; it never multiplies their alternatives with its own. This is where the memo's speed comes from: work proportional to the number of groups and their implementations, not to the product of every query's choices. - -**How the three fit together:** - -```text -for each sharing cluster: - branch and bound over the cluster's coupling sites - each site's candidate values come from threshold enumeration (n + 1, not 2^n) - each complete assignment: dynamic programming over the remaining groups (cached) - keep the cheapest assignment (the incumbent) -compose the clusters' selections into one plan; check and compile it once -``` - -None of the three discards a plan that could be optimal, so the selection equals the exhaustive search's. `joint_search_equivalence` checks this on workloads small enough for `--exhaustive`. +**Validation.** The brute force over the memo is a second oracle. `joint_search_equivalence` checks the MILP, the brute force and `--exhaustive` against each other on small workloads under all three cost models: a peak case, a capacity case and a raw-horizon case. ### Coupled charges -- **Raw data on disk**: a per-source coupling site whose size is the longest horizon among the readers that scan raw data, enumerated as a threshold (step 4). -- **Cost models 2 and 3**: frontiers instead of minima (step 5), peaks and limits at combination (step 6). -- **A measured cost table**: the calibration factor is fitted once, before the search, over every candidate node the generation produced (before pruning), and frozen; every group is priced with it. - -### Exactness - -The search is exact when every interaction between groups is a shared group or a coupling site, and each coupling site's cost depends only on its threshold. When a cluster has a construct no threshold captures, that cluster is searched exhaustively over its frontiers and the report names it; no cap is applied. Each rule is checked against exhaustive enumeration on small workloads (`joint_search_equivalence`), including raw-retention coupling, a cost model 2 peak, a cost model 3 capacity limit and a partial measured table. +- **Raw data on disk**: a per-source horizon threshold, linear in the MILP. +- **Cost models 2 and 3**: peaks and limits as linear constraints. +- **A measured cost table**: the calibration factor is fitted once, before the search, over every candidate node the generation produced (before pruning), and frozen; every coefficient is priced with it. ### Code layout -`crates/plan-selection/src/pass/memo/`, behind `Stage1ToStage3Search::memo_groups` until it matches the current search on every equivalence test: - -```rust -/// One hash-consed logical sub-expression. -pub(crate) struct MemoGroup { pub node: Rc, pub parents: Vec, pub queries: Vec } -/// One implementation of one group (or of several, for a fused computation). -pub(crate) struct ImplementationCandidate { pub implements: Vec, pub node: Rc, pub children: Vec, pub coupling_sites: Vec } -/// A choice several groups' costs depend on, keyed so that equal keys are one site. -pub(crate) struct CouplingSite { pub key: u64, pub kind: CouplingKind } -pub(crate) enum CouplingKind { SharedGroupSize, PaneChain, ScanLabels, RawHorizon } -/// The cached local price of a candidate in a context (thresholds of its sites). -pub(crate) fn local_price(group: GroupId, candidate: &ImplementationCandidate, context: &Context, cache: &mut PriceCache) -> Result; -/// The frontier of a group given its children's frontiers and a context. -pub(crate) fn frontier(group: GroupId, context: &Context, memo: &Memo, cache: &mut PriceCache) -> Frontier; -/// The cluster's selection: thresholds enumerated with bounds, the DP below them. -pub(crate) fn solve_cluster(cluster: &Cluster, memo: &Memo, cache: &mut PriceCache) -> Result; -``` +`crates/plan-selection/src/pass/subexpr/`: `groups.rs` (hash-consing, Pass 1), `capability.rs`, `windows.rs`, `scans.rs` (the Pass 2 transformations), `pricing.rs` (per-node prices in a consumer context), `search.rs` (the memo and the brute-force reference), `milp.rs` (the model, the lazy-cut loop). The search is chosen by `--search legacy|memo-brute-force|memo-milp`; the MILP becomes the default once it matches `--exhaustive` on every equivalence test. ## Future work From d0ee6f1025134091fde63804be2e39e6104f3f91 Mon Sep 17 00:00:00 2001 From: zzylol <50204836+zzylol@users.noreply.github.com> Date: Thu, 22 Oct 2026 18:16:31 +0000 Subject: [PATCH 14/53] docs: MILP solver rationale, the encoding of each non-additive term, the lazy-cut loop, and the multi-query optimization view --- .../proposals/joint-planning-search.md | 62 ++++++++++++++++--- 1 file changed, 52 insertions(+), 10 deletions(-) diff --git a/docs/design_docs/proposals/joint-planning-search.md b/docs/design_docs/proposals/joint-planning-search.md index b73793b4f..3fc04e8c4 100644 --- a/docs/design_docs/proposals/joint-planning-search.md +++ b/docs/design_docs/proposals/joint-planning-search.md @@ -212,7 +212,7 @@ The rules above prune a search that still builds and prices whole-workload candi | Shared group | A group read by two or more parents. Its choice (implementation, size, materialization, pane grid) couples its parents. | | Coupling site | A choice several groups' costs depend on: a shared group's size (sized for its strictest reader), a shared pane chain's grid, longest window and size, a metric's scan label set, a source's raw-data horizon. | | Local price | One implementation candidate's usage, priced by Stage 3's per-node pricing given its context (its readers' requirements, its coupling sites' thresholds): CPU operations, scanned bytes, memory byte-seconds, peak memory, disk, each reader's latency. Cached per (group, candidate, context). | -| Frontier | Per group and subproblem, the candidates no other dominates in usage: the cheapest under cost model 1, a Pareto front over (cost, peak memory, peak cores, disk, worst latency) under cost models 2 and 3. | +| MILP | One mixed-integer linear program over the memo whose variables are the groups' candidate choices, their materializations and the coupling sites' thresholds, with Stage 3's prices as coefficients. It replaces enumerating whole-workload candidates. | ### How the stages and the memo work together @@ -234,11 +234,11 @@ The memo keeps each stage's semantics and code, and changes what they exchange: Stage 1 Pass 1 A's implementations B's implementations C's implementations Stage 1 Pass 2 transformations add implementations that read another group (capability, panes, rollup, top-k) Stage 2 materialization per implementation, once per group however many parents read it -Stage 3 local price per (group, implementation, context), cached; frontier per group +Stage 3 local price per (group, implementation, context), cached: the MILP's coefficients ------------------------- coupling sites of the cluster {A, B} ------------------------- - thresholds enumerated with bounds: a shared group's size (strictest reader), + discrete threshold variables: a shared group's size (strictest reader), a pane chain's grid and longest window, a metric's scan labels, a source's raw horizon; - dynamic programming below them; C is solved on its own + one MILP chooses them together with every group's candidate; C's choice is independent ------------------------------------------------------------------------------------------- Selected plan composed once, then Stage 3's checks and the executor's compile check ``` @@ -273,7 +273,7 @@ G6: exact top-k (reads G5) G1, G2: each reader computes its own (NotMaterialized) | built once per evaluation (MatAtQuery) ``` -Before, the same `Scan -> TimeRange(5m)` was repeated inside every per-target candidate, and Pass 2 had to discover it again; candidates multiplied across targets. In the memo, `G2` appears once, `G3` has a handful of implementations, and the search is bottom-up: `G2`'s choice is a coupling choice of the cluster `{Q1, Q2}`, and every other group keeps its frontier given it. +Before, the same `Scan -> TimeRange(5m)` was repeated inside every per-target candidate, and Pass 2 had to discover it again; candidates multiplied across targets. In the memo, `G2` appears once, `G3` has a handful of implementations, and the MILP has one variable per (group, implementation): `G2`'s materialization is a coupling choice that both queries' costs depend on. Where each part of the stage design lives in the memo: @@ -286,7 +286,7 @@ Where each part of the stage design lives in the memo: | Rollup and exact top-k reuse | an implementation of a coarse group (or a top-k) that reads a finer group (or a ranking) | | Stage 2 materialization | chosen per implementation, once per group | | Stage 3 pricing and checks | local price per (group, implementation, context), cached; the full plan is checked once | -| Raw retention, cost models 2 and 3, measured table | a per-source horizon site; frontiers; a calibration factor fitted once and frozen | +| Raw retention, cost models 2 and 3, measured table | a per-source horizon threshold; linear peak and capacity constraints; a calibration factor fitted once and frozen | ### End to end: the memo with the sharing strategies @@ -329,9 +329,9 @@ Nothing here builds a whole-workload candidate; each strategy only adds alternat **4. Stage 2: materialization per implementation.** For example, the shared pane chain may be kept at ingestion time or rebuilt per evaluation; `G2` may be computed by each reader or built once per evaluation. A group with several parents chooses once. -**5. Stage 3: local prices, frontiers, and clusters.** Each implementation is priced once per context and cached. Groups linked by a shared group or a coupling site form a cluster; here all nine groups form one cluster through `G1`, `G2` and the shared quantile build. A workload with another metric would form a second cluster, solved independently. +**5. Stage 3: local prices as coefficients.** Each implementation is priced once per consumer rate, materialization and threshold, and cached; these are the MILP's coefficients. Here all nine groups are coupled through `G1`, `G2` and the shared quantile build. A second metric's groups would share no variable with them, and the solver separates such blocks on its own. -**6. Joint choice in the cluster.** The coupling choices are enumerated as thresholds, with branch and bound: +**6. One MILP chooses everything.** Threshold enumeration gives each coupling site its discrete values, which become MILP variables: ```text shared quantile build: size threshold in {for G4's 0.01, for G5's 0.05, no shared build} @@ -342,7 +342,7 @@ G2, G6: per reader or built once per evaluation raw retention of data: horizon threshold among the readers that scan raw data ``` -For each combination that survives the bound, dynamic programming chooses every other group's implementation bottom-up from the cached frontiers. For example, it chooses between `G8` exact and `G8` rolled up from `G7`, and between `G9` exact and the fused heap, given the chosen `G2` and `G6`. The threshold enumeration replaces enumerating subsets of readers; dominance inside each group keeps the frontiers small. +The same MILP also chooses every other group's implementation: `G8` exact or rolled up from `G7`, `G9` exact or the fused heap. The threshold variables replace subsets of readers. The lazy-cut loop (below) then prices the solver's plan with Stage 3 and adds the held-memory term back until the two agree. **7. Build late.** The selected choices are composed into one physical plan. Stage 3 checks it once (guarantees, latency against each repeat interval, capacity) and the executor's compile check runs on it alone. The runner-ups are built only if a report asks for them. @@ -387,6 +387,48 @@ Threshold enumeration still turns each coupling site into discrete choices: a sh **Cost models 2 and 3.** The same variables. Peaks and capacity limits become linear constraints over the per-node peak contributions, and each objective is as specified. +**Why a MILP rather than a hand-written search.** The memo is small: the whole paper workloads have 74 groups and 284 candidates (W-RangeHeavy) and 158 groups and 584 candidates (W-RollupHeavy), each built in under 0.3 s. Cost model 1 is not additive per node: +- memory held while a query runs costs bytes × that query's latency; +- how often a node runs depends on the materialization choices above it. + +A branch and bound with dynamic programming would need custom bounds for both, plus Pareto frontiers for cost models 2 and 3. In a MILP each of these is a standard exact linearization or a linear constraint, and the solver supplies the bounds, the search and an optimality certificate. + +This is the multi-query optimization problem. The memo is its AND-OR DAG: a group is an equivalence node and a candidate is an operation node. Greedy multi-query optimizers (for example Roy et al., SIGMOD 2000) give up optimality for speed. Exact formulations are integer programs. Our shared sizes depend on the strictest reader, accuracy guarantees compose, and time enters through panes and retention, so we need the exact form. + +**How each non-additive term is encoded.** + +| Term (cost model 1) | Encoding | Exact | +| --- | --- | --- | +| How often a node runs: once per consumer whose path reaches it through unmaterialized nodes | reach indicators `r[k,g]` propagated through `x` and `m`; cost on `y = r ∧ x` | linear | +| A node built once per evaluation for readers on different grids | one indicator per distinct set of reader grids; cost at that set's union rate | linear | +| A shared size set by the strictest reader (capability builds, pane chains, rollup state) | one indicator per threshold; a reader attaches only to a threshold it may read | linear | +| A scan's label set | one indicator per label set | linear | +| Raw retention: the longest horizon among query-time raw scans of a source | `R_s ≥ bytes(n) · query-time(n)` | linear | +| Pane retention: the longest span among a chain's readers | `keep ≥ span(q) · read(q, chain)` | linear | +| Ingestion compaction: depends on the ingest worker count, a ceiling | integer workers `≥ work / capacity` | linear | +| Latency within the bound and the repeat interval | the sum of the reached work, over `y` | linear | +| Memory held while a query runs: bytes × latency | omitted from the model; added back by lazy cuts | lazy | +| Composed accuracy guarantees | checked on the incumbent; failing combinations excluded | lazy | +| A measured table's calibration factor | fitted once before the search | constant | + +An exact linearization of the held-memory product needs a variable for each query and each pair of groups it reaches (about 3M on W-RollupHeavy with 8 users), so it is generated lazily instead. + +**The lazy-cut loop.** The model without the lazy terms never overestimates a plan's cost, so its optimum is a lower bound on the true optimum. The loop: +1. Solve the model and take its plan `P`, with objective `c_milp`. +2. Compose `P` and price it with Stage 3, which includes held memory and the composed guarantees, giving `c_true`. +3. If `c_true = c_milp` and every check passes, `P` is optimal: its true cost equals a lower bound on every plan's true cost. +4. Otherwise add a cut and go to step 1: + - a combination whose guarantee fails is excluded; + - a combination whose cost was underestimated gets "choosing this combination costs at least `c_true`". + +Each cut removes or corrects at least one of finitely many combinations, so the loop ends, and it ends at the optimum. For example: +- Round 1 picks `P1` at 0.40 per second, and Stage 3 prices it at 0.47. A cut makes `P1`'s combination cost at least 0.47. +- Round 2 picks `P2` at 0.45, and Stage 3 agrees. `P2` is optimal. + +(These numbers are illustrative.) The cost of the loop is the number of rounds. Tight McCormick bounds on held memory (from the memo's byte and latency ranges) and several cuts per round keep it low; the report gives the rounds per workload. + +**Reporting.** The result states the solver gap. A time limit that ends with a nonzero gap is reported with that gap, never as optimal. + **Validation.** The brute force over the memo is a second oracle. `joint_search_equivalence` checks the MILP, the brute force and `--exhaustive` against each other on small workloads under all three cost models: a peak case, a capacity case and a raw-horizon case. ### Coupled charges @@ -401,7 +443,7 @@ Threshold enumeration still turns each coupling site into discrete choices: a sh ## Future work -- Stronger bounds from LP relaxations; separator/tree-decomposition DP or MILP as alternative search engines. +- Stronger LP relaxations and McCormick bounds for the held-memory term; tree-decomposition dynamic programming as an alternative engine for the MILP. - Greedy incumbents and learned branch ordering to accelerate exact search. - Explicit approximate modes such as beam search, top-k candidates, capped resource frontiers, and coarser parameter domains. These may lose the optimum and must be labeled accordingly. - Incremental replanning and broader candidate/rewrite coverage. From 52a94a19af5bd4fb9bc07cff265902e14b8adbae Mon Sep 17 00:00:00 2001 From: zzylol <50204836+zzylol@users.noreply.github.com> Date: Thu, 22 Oct 2026 18:22:06 +0000 Subject: [PATCH 15/53] docs: the memo MILP's formulation: sets, parameters, variables, objective, constraints and lazy cuts --- .../proposals/joint-planning-search.md | 77 +++++++++++++++++++ 1 file changed, 77 insertions(+) diff --git a/docs/design_docs/proposals/joint-planning-search.md b/docs/design_docs/proposals/joint-planning-search.md index 3fc04e8c4..21d1ecb2a 100644 --- a/docs/design_docs/proposals/joint-planning-search.md +++ b/docs/design_docs/proposals/joint-planning-search.md @@ -387,6 +387,83 @@ Threshold enumeration still turns each coupling site into discrete choices: a sh **Cost models 2 and 3.** The same variables. Peaks and capacity limits become linear constraints over the per-node peak contributions, and each objective is as specified. +**Formulation.** + +Sets: +- `Q`: queries; `G`: memo groups; `C(g)`: candidates of group `g`. +- `ch(g,c)`: child groups that candidate `(g,c)` reads; `impl(g,c)`: groups it implements (more than `{g}` for a fused candidate). +- `O = {nm, mq, ing, kept}`: materializations (not materialized, built once per evaluation, at ingestion, kept across evaluations). +- `S`: coupling sites; `T(s)`: thresholds of site `s`. +- `K = Q ∪ {(g,c,o) : o ≠ nm}`: consumers, the queries and the materialized slots. + +Parameters, all priced by Stage 3, in cost per second unless noted: +- `rate(k)`: evaluations per second of consumer `k`. +- `w(g,c)`: cost of one computation of `(g,c)`'s own nodes. +- `a(g,c,o)`: standing cost of slot `(g,c,o)` (ingestion upkeep, kept state). +- `σ(s,t)`: cost of site `s` at threshold `t`. +- `may(s,t,g)`: whether group `g` may read site `s` at threshold `t`. +- `ℓ(g,c)`: latency of one computation, in ms. +- `B(q)`: the latency limit of query `q`, `min(max_latency_ms, repeat interval)`. +- `hz(n)`: raw horizon bytes of raw scan `n`; `span(q,h)`: retention that reader `q` needs from pane chain `h`; `β(h)`: cost per second of keeping one span unit of `h`; `p_disk`: disk price. + +Variables: + +```text +x[g,c] ∈ {0,1} group g is implemented by candidate c +m[g,c,o] ∈ {0,1} candidate (g,c) is materialized as o +u[s,t] ∈ {0,1} coupling site s takes threshold t +r[k,g] ∈ {0,1} consumer k computes group g (reaches it through unmaterialized nodes) +y[k,g,c] ∈ {0,1} r[k,g] ∧ x[g,c] +λ[q,g] ∈ {0,1} g is computed within q's evaluation (reached through nm or mq nodes) +R[s] ≥ 0 raw horizon kept for source s +keep[h] ≥ 0 retention kept for pane chain h +θ ≥ 0 held memory (memory × latency), added by lazy cuts +``` + +Objective, cost model 1: + +```text +min Σ_k Σ_(g,c) rate(k) · w(g,c) · y[k,g,c] computation, once per consumer that reaches it + + Σ_(g,c,o) a(g,c,o) · m[g,c,o] standing cost of materialized slots + + Σ_(s,t) σ(s,t) · u[s,t] coupling sites at their thresholds + + p_disk · Σ_s R[s] + Σ_h β(h) · keep[h] raw and pane retention + + θ held memory (lazy) +``` + +Constraints: + +```text +(1) cover Σ_{(h,c): g ∈ impl(h,c)} x[h,c] = need[g] each needed group covered once + need[root(q)] = 1; need[g] ≥ x[p,c] for g ∈ ch(p,c) +(2) materialize Σ_o m[g,c,o] = x[g,c] + m[g,c,ing] ≤ Σ_c' m[g',c',ing] + rawsource(g') for g' ∈ ch(g,c) (ingestion needs ingestion below) +(3) reach r[q,root(q)] = 1; r[(g,c,o),g'] ≥ m[g,c,o] for g' ∈ ch(g,c) + r[k,g'] ≥ r[k,p] + x[p,c] − Σ_{o≠nm} m[p,c,o] − 1 for g' ∈ ch(p,c) +(4) product y[k,g,c] ≥ r[k,g] + x[g,c] − 1 +(5) sites Σ_t u[s,t] ≤ 1; x[g,c] ≤ Σ_{t: may(s,t,g)} u[s,t] for each candidate (g,c) that reads site s +(6) retention R[s] ≥ hz(n) · λ[q,n] for each raw scan n of s and each q + keep[h] ≥ span(q,h) · x[q's reader of h] +(7) latency λ[q,root(q)] = 1; λ[q,g'] ≥ λ[q,p] + x[p,c] − m[p,c,ing] − m[p,c,kept] − 1 + Σ_(g,c) ℓ(g,c) · (λ[q,g] + x[g,c] − 1)^+ + compaction(q) ≤ B(q) + (the product linearized as in (4)) +(8) rules combination limits, e.g. x[top-k reuse] + x[pane candidate of the same group] ≤ 1 +``` + +Minimization keeps `r`, `y`, `λ`, `R` and `keep` at their smallest feasible values, so their lower-bound constraints make them exact. + +Lazy cuts, added by the loop for an incumbent plan `P` (a set of chosen `(g,c)` and `(g,c,o)`): + +```text +held memory θ ≥ H(P) · ( Σ_{(g,c) ∈ P} x[g,c] − |P| + 1 ) H(P) = Stage 3's held-memory cost of P +accuracy Σ_{(g,c) ∈ B} x[g,c] ≤ |B| − 1 B = a combination whose composed guarantee fails +``` + +Each held-memory cut is binding only when every choice of `P` is made, so it never cuts off another plan, and its bound is exact for `P`. + +Cost models 2 and 3 keep (1)–(8) and the cuts, and change the objective: +- **Cost model 2:** `min π_mem · M + π_cpu · N + π_disk · D`, with `M ≥ Σ peak contributions`, `N ≥` ingestion cores `+` concurrently evaluating queries, and `D ≥` retention. +- **Cost model 3:** no prices. Add `M ≤ cap_mem`, `D ≤ cap_disk` (and `N ≤ cap_cores`), then minimize the chosen objective: the worst latency `L ≥ L_q` for every `q`, or `M`. + **Why a MILP rather than a hand-written search.** The memo is small: the whole paper workloads have 74 groups and 284 candidates (W-RangeHeavy) and 158 groups and 584 candidates (W-RollupHeavy), each built in under 0.3 s. Cost model 1 is not additive per node: - memory held while a query runs costs bytes × that query's latency; - how often a node runs depends on the materialization choices above it. From 158a6c9e2aab0c3f2418589a584f30e3bf14a337 Mon Sep 17 00:00:00 2001 From: zzylol <50204836+zzylol@users.noreply.github.com> Date: Thu, 22 Oct 2026 18:27:11 +0000 Subject: [PATCH 16/53] docs: why dynamic programming alone does not apply to the memo search --- .../proposals/joint-planning-search.md | 24 +++++++++++++++++++ 1 file changed, 24 insertions(+) diff --git a/docs/design_docs/proposals/joint-planning-search.md b/docs/design_docs/proposals/joint-planning-search.md index 21d1ecb2a..ba5e20542 100644 --- a/docs/design_docs/proposals/joint-planning-search.md +++ b/docs/design_docs/proposals/joint-planning-search.md @@ -470,6 +470,30 @@ Cost models 2 and 3 keep (1)–(8) and the cuts, and change the objective: A branch and bound with dynamic programming would need custom bounds for both, plus Pareto frontiers for cost models 2 and 3. In a MILP each of these is a standard exact linearization or a linear constraint, and the solver supplies the bounds, the search and an optimality certificate. +**Why dynamic programming alone does not apply.** Dynamic programming over the memo needs optimal substructure: a group's best implementation depends only on what lies below it, and the plan's cost is the sum of its parts. This workload breaks that in three ways: + +1. **The memo is a DAG, not a tree.** + - A shared group (a scan, a time range, a shared sketch) has several parents. + - Its size is set by its strictest reader, and which parents read it depends on their own choices. + - Its cost is paid once and split among the readers. + - So its best choice depends on the choices above it, not only on its subtree. This is the core difficulty of multi-query optimization, which is NP-hard. +2. **Costs depend on the context above.** + - How often a node runs depends on its ancestors' materialization: below a node built once per evaluation or maintained at ingestion time, the node is not recomputed per reader. + - Cost therefore flows top-down, while dynamic programming works bottom-up. + - Cascades handles this by memoizing per (group, required properties). Here the context is the reader set, the thresholds and the materializations above, and their combinations bring the blow-up back. +3. **The objective is not additive.** + - Held memory costs bytes × the query's latency, and the latency is the sum of the work along the query's whole path. This is a product of sums, not a sum. + - A subtree that is cheapest on its own is not always part of the optimum: a slightly costlier but faster subtree lowers the query's latency and so its held-memory cost elsewhere. + - Each subproblem would have to keep a Pareto front over (cost, latency) rather than one minimum, and the fronts can grow large. + - Raw retention (the longest horizon among readers) and pane retention (the longest span) are maxima, which are not additive either. + +A MILP needs none of these properties. It solves all choices jointly: +- sharing and context are linear constraints (reach `r`, products `y`, thresholds `u`); +- maxima are lower-bound variables (`R`, `keep`); +- the product term is added back by lazy cuts. + +Dynamic programming remains useful only for the parts that do decompose, for example independent blocks with no shared group. + This is the multi-query optimization problem. The memo is its AND-OR DAG: a group is an equivalence node and a candidate is an operation node. Greedy multi-query optimizers (for example Roy et al., SIGMOD 2000) give up optimality for speed. Exact formulations are integer programs. Our shared sizes depend on the strictest reader, accuracy guarantees compose, and time enters through panes and retention, so we need the exact form. **How each non-additive term is encoded.** From 236965ef72f2152fe6ce205dea8025bb125213b0 Mon Sep 17 00:00:00 2001 From: zzylol <50204836+zzylol@users.noreply.github.com> Date: Thu, 22 Oct 2026 20:07:40 +0000 Subject: [PATCH 17/53] docs: how the memo MILP's coefficients are derived; lower-bound coefficients and term cuts --- .../proposals/joint-planning-search.md | 37 +++++++++++++++++++ 1 file changed, 37 insertions(+) diff --git a/docs/design_docs/proposals/joint-planning-search.md b/docs/design_docs/proposals/joint-planning-search.md index ba5e20542..5eaa69792 100644 --- a/docs/design_docs/proposals/joint-planning-search.md +++ b/docs/design_docs/proposals/joint-planning-search.md @@ -532,6 +532,43 @@ Each cut removes or corrects at least one of finitely many combinations, so the **Validation.** The brute force over the memo is a second oracle. `joint_search_equivalence` checks the MILP, the brute force and `--exhaustive` against each other on small workloads under all three cost models: a peak case, a capacity case and a raw-horizon case. +### Coefficients + +**Principle: Stage 3 is the only source of prices, and the model only needs a lower bound.** +- Every coefficient comes from Stage 3's per-node pricing (`price_node`), evaluated in a hypothetical context. The MILP has no cost formula of its own. +- The selected plan is always re-priced by Stage 3 as a whole, and its cost is that price. So a coefficient may be any value **no greater than** the true cost of the choice it prices. +- Exact coefficients make the first bound tight. Lower-bound coefficients keep the result exact at the cost of more lazy-cut rounds. + +**Decomposition.** A plan's cost is the sum over its nodes of `price_node(node, context)`. A node's context is who computes it, how often, how it is materialized, at which threshold it is sized, and how large its input is. When a cost depends only on discrete context, each value of the context gets its own coefficient, selected by a 0/1 variable: + +| Cost depends on | Variable | Coefficient | +| --- | --- | --- | +| the chosen implementation | `x[g,c]` | `price_node` of the candidate's own nodes | +| the materialization | `m[g,c,o]` | standing cost `a(g,c,o)` and per-computation cost `w(g,c)` for that `o` | +| which consumers compute it, how often | `y[k,g,c]` | `rate(k) · w(g,c)`, with `w` priced at one computation | +| a shared node's size | `u[s,t]` | `σ(s,t)`: the node priced at threshold `t` | + +Example: `quantile_over_time(0.9, data[5m])` as a KLL built once per evaluation at threshold 0.01 is the choice (group, KLL, `mq`, `t = 0.01`). Its `w` is Stage 3's price of one KLL sized for 0.01 over five minutes of rows, and its objective term is `rate(Q1) · w · y[Q1, G4, KLL]`. Coefficients are generated per candidate, per combination of its coupling-site thresholds, and per materialization of its private builds. + +**Terms that do not decompose, and their lower bounds:** + +| Term | Why it does not decompose | Coefficient used | Corrected by | +| --- | --- | --- | --- | +| Held memory, bytes × latency | the latency is a sum over the whole path | 0, or a McCormick bound from the memo's byte and latency ranges | a cut on the term | +| A shared scan's window (the longest reader's) | its input depends on other groups' choices | priced at the reader's own window | a cut when a longer reader is chosen | +| A child consumed by an ingestion-time build | the child runs at ingestion time instead | the cheaper of the two pricings | a cut on the pair | +| A chain kept across evaluations, charged per request | depends on the request count | the per-evaluation price | a cut | +| Compaction with several ingest workers | depends on the global worker count | 0 | a cut | +| Composed accuracy | depends on the chosen children | not priced | a no-good cut on a failing combination | + +Validation: for every plan of the brute force over the memo, the sum of its coefficients is at most Stage 3's price, and equal wherever the term decomposes exactly. + +**Cuts.** When Stage 3's price of the incumbent exceeds the bound, the loop adds: +- a **term cut** for each underestimated term, conditioned only on the choices that term depends on, for example "if `m[n,o]` then held memory of `n` ≥ H" or "if `x[g,c]` and `x[g',c']` then the scan window cost ≥ W". One term cut corrects every plan sharing those choices. +- a **no-good cut** on the whole combination, as the fallback that guarantees termination. + +The report gives the number of rounds and the first round's bound against the final price (the tightness of the model). + ### Coupled charges - **Raw data on disk**: a per-source horizon threshold, linear in the MILP. From 022f30c37d8ea9652c649b96f29c1abce3b244ee Mon Sep 17 00:00:00 2001 From: zzylol <50204836+zzylol@users.noreply.github.com> Date: Fri, 23 Oct 2026 01:01:17 +0000 Subject: [PATCH 18/53] docs: name coefficient derivation, solution decoding and incumbent evaluation apart; one solution decodes to one plan --- .../proposals/joint-planning-search.md | 44 +++++++++++++------ 1 file changed, 31 insertions(+), 13 deletions(-) diff --git a/docs/design_docs/proposals/joint-planning-search.md b/docs/design_docs/proposals/joint-planning-search.md index 5eaa69792..0d0ecae93 100644 --- a/docs/design_docs/proposals/joint-planning-search.md +++ b/docs/design_docs/proposals/joint-planning-search.md @@ -211,7 +211,7 @@ The rules above prune a search that still builds and prices whole-workload candi | Materialization | `NotMaterialized`, `MatAtQuery`, at ingestion, or kept across evaluations: chosen once per implementation candidate of a group, however many parents read it. | | Shared group | A group read by two or more parents. Its choice (implementation, size, materialization, pane grid) couples its parents. | | Coupling site | A choice several groups' costs depend on: a shared group's size (sized for its strictest reader), a shared pane chain's grid, longest window and size, a metric's scan label set, a source's raw-data horizon. | -| Local price | One implementation candidate's usage, priced by Stage 3's per-node pricing given its context (its readers' requirements, its coupling sites' thresholds): CPU operations, scanned bytes, memory byte-seconds, peak memory, disk, each reader's latency. Cached per (group, candidate, context). | +| Local cost | One implementation candidate's usage, evaluated by Stage 3's per-node cost function given its context (its readers' requirements, its coupling sites' thresholds): CPU operations, scanned bytes, memory byte-seconds, peak memory, disk, each reader's latency. Cached per (group, candidate, context). | | MILP | One mixed-integer linear program over the memo whose variables are the groups' candidate choices, their materializations and the coupling sites' thresholds, with Stage 3's prices as coefficients. It replaces enumerating whole-workload candidates. | ### How the stages and the memo work together @@ -329,7 +329,7 @@ Nothing here builds a whole-workload candidate; each strategy only adds alternat **4. Stage 2: materialization per implementation.** For example, the shared pane chain may be kept at ingestion time or rebuilt per evaluation; `G2` may be computed by each reader or built once per evaluation. A group with several parents chooses once. -**5. Stage 3: local prices as coefficients.** Each implementation is priced once per consumer rate, materialization and threshold, and cached; these are the MILP's coefficients. Here all nine groups are coupled through `G1`, `G2` and the shared quantile build. A second metric's groups would share no variable with them, and the solver separates such blocks on its own. +**5. Stage 3: coefficient derivation.** Each implementation's cost is derived once per consumer rate, materialization and threshold, and cached; these are the MILP's coefficients. Here all nine groups are coupled through `G1`, `G2` and the shared quantile build. A second metric's groups would share no variable with them, and the solver separates such blocks on its own. **6. One MILP chooses everything.** Threshold enumeration gives each coupling site its discrete values, which become MILP variables: @@ -363,9 +363,9 @@ The result is the plan the exhaustive search would select. It is checked by `joi 1. **Hash-cons** the workload's logical DAG into groups. 2. **Generate** each group's implementation candidates (Pass 1, then the Pass 2 transformations), each with its child groups and its coupling sites. -3. **Coefficients.** Price each candidate's private nodes with Stage 3's per-node pricing, per consumer rate and per materialization, and each coupling site's node at each of its thresholds. Stage 3 stays the only source of prices. +3. **Coefficient derivation.** Derive each choice's objective coefficients from Stage 3's per-node cost function (`price_node`): a candidate's private nodes per consumer rate and per materialization, and each coupling site's node at each of its thresholds. Stage 3 stays the only source of costs. 4. **Solve one MILP over the memo** (below): one choice per needed group, its materializations and the coupling sites' thresholds. -5. **Lazy cuts.** Compose the solver's plan, recompose its guarantees and price it with Stage 3. If Stage 3's price exceeds the solver's objective (the held-memory term), or a composed guarantee fails, add a cut that corrects or excludes that plan, and solve again. Stop when Stage 3's price equals the objective and every check passes. The gap must be 0 to report the plan as optimal. +5. **Solution decoding, incumbent evaluation, lazy cuts.** Decode the solver's solution into its one physical plan, recompose its guarantees, and evaluate the plan's exact cost with Stage 3 (the incumbent cost). If the incumbent cost exceeds the model bound (the held-memory term), or a composed guarantee fails, add a cut that corrects or excludes that plan, and solve again. Stop when the best incumbent cost equals the model bound and every check passes. The gap must be 0 to report the plan as optimal. 6. **Build late.** Run the executor's compile check on the selected plan only, and report it. ### The solver: MILP over the memo @@ -396,7 +396,7 @@ Sets: - `S`: coupling sites; `T(s)`: thresholds of site `s`. - `K = Q ∪ {(g,c,o) : o ≠ nm}`: consumers, the queries and the materialized slots. -Parameters, all priced by Stage 3, in cost per second unless noted: +Parameters, all derived from Stage 3's per-node cost function, in cost per second unless noted: - `rate(k)`: evaluations per second of consumer `k`. - `w(g,c)`: cost of one computation of `(g,c)`'s own nodes. - `a(g,c,o)`: standing cost of slot `(g,c,o)` (ingestion upkeep, kept state). @@ -516,7 +516,7 @@ An exact linearization of the held-memory product needs a variable for each quer **The lazy-cut loop.** The model without the lazy terms never overestimates a plan's cost, so its optimum is a lower bound on the true optimum. The loop: 1. Solve the model and take its plan `P`, with objective `c_milp`. -2. Compose `P` and price it with Stage 3, which includes held memory and the composed guarantees, giving `c_true`. +2. Decode `P` into its physical plan and evaluate it with Stage 3, which includes held memory and the composed guarantees, giving the incumbent cost `c_true`. 3. If `c_true = c_milp` and every check passes, `P` is optimal: its true cost equals a lower bound on every plan's true cost. 4. Otherwise add a cut and go to step 1: - a combination whose guarantee fails is excluded; @@ -534,9 +534,27 @@ Each cut removes or corrects at least one of finitely many combinations, so the ### Coefficients -**Principle: Stage 3 is the only source of prices, and the model only needs a lower bound.** +**Terms.** Three distinct steps use Stage 3's cost function, and this document keeps their names apart: + +| Term | What it is | When | How often | +| --- | --- | --- | --- | +| **Coefficient derivation** | Computing the objective coefficient of each local choice (a group's implementation, materialization, threshold) from Stage 3's per-node cost function. The result is a lower bound on that choice's true cost. | before solving | once per choice, cached | +| **Solution decoding** | Turning a MILP solution (the values of `x`, `m`, `u`, …) into the one physical plan it describes. | after each solve | once per round | +| **Incumbent evaluation** | Running Stage 3 on that decoded plan as a whole: guarantees, latency checks and its exact cost (the **incumbent cost**). | after decoding | once per round | + +The **model bound** is the solver's objective, a lower bound on every plan's cost. The **gap** is (incumbent cost − model bound) / incumbent cost. + +**One solution, one plan.** Every choice that changes a plan's cost is a model variable, so a solution decodes to exactly one plan, and incumbent evaluation runs Stage 3 once per round: +- sharing an identical expression or computing it per reader is the materialization `m` (`mq` vs `nm`); +- a shared node's size is the threshold `u`; +- a metric's scan label set is a threshold variable of the scan's coupling site (the union of the labels of the readers attached to it); +- pane grids and raw horizons are threshold variables of their sites. + +A solution must never stand for several plans whose costs are then enumerated: that enumeration (thresholds × identical-expression variants × materializations) is what exhausts memory on whole workloads. If a variant cannot be a variable yet, its alternatives are evaluated as a small set local to that one site, never as a product with other sites, and the design records it as a gap to close. + +**Principle: Stage 3 is the only source of costs, and the coefficients only need to be lower bounds.** - Every coefficient comes from Stage 3's per-node pricing (`price_node`), evaluated in a hypothetical context. The MILP has no cost formula of its own. -- The selected plan is always re-priced by Stage 3 as a whole, and its cost is that price. So a coefficient may be any value **no greater than** the true cost of the choice it prices. +- The selected plan is always evaluated by Stage 3 as a whole (incumbent evaluation), and its cost is the incumbent cost. So a coefficient may be any value **no greater than** the true cost of the choice it stands for. - Exact coefficients make the first bound tight. Lower-bound coefficients keep the result exact at the cost of more lazy-cut rounds. **Decomposition.** A plan's cost is the sum over its nodes of `price_node(node, context)`. A node's context is who computes it, how often, how it is materialized, at which threshold it is sized, and how large its input is. When a cost depends only on discrete context, each value of the context gets its own coefficient, selected by a 0/1 variable: @@ -545,8 +563,8 @@ Each cut removes or corrects at least one of finitely many combinations, so the | --- | --- | --- | | the chosen implementation | `x[g,c]` | `price_node` of the candidate's own nodes | | the materialization | `m[g,c,o]` | standing cost `a(g,c,o)` and per-computation cost `w(g,c)` for that `o` | -| which consumers compute it, how often | `y[k,g,c]` | `rate(k) · w(g,c)`, with `w` priced at one computation | -| a shared node's size | `u[s,t]` | `σ(s,t)`: the node priced at threshold `t` | +| which consumers compute it, how often | `y[k,g,c]` | `rate(k) · w(g,c)`, with `w` the cost of one computation | +| a shared node's size | `u[s,t]` | `σ(s,t)`: the node's cost at threshold `t` | Example: `quantile_over_time(0.9, data[5m])` as a KLL built once per evaluation at threshold 0.01 is the choice (group, KLL, `mq`, `t = 0.01`). Its `w` is Stage 3's price of one KLL sized for 0.01 over five minutes of rows, and its objective term is `rate(Q1) · w · y[Q1, G4, KLL]`. Coefficients are generated per candidate, per combination of its coupling-site thresholds, and per materialization of its private builds. @@ -555,11 +573,11 @@ Example: `quantile_over_time(0.9, data[5m])` as a KLL built once per evaluation | Term | Why it does not decompose | Coefficient used | Corrected by | | --- | --- | --- | --- | | Held memory, bytes × latency | the latency is a sum over the whole path | 0, or a McCormick bound from the memo's byte and latency ranges | a cut on the term | -| A shared scan's window (the longest reader's) | its input depends on other groups' choices | priced at the reader's own window | a cut when a longer reader is chosen | +| A shared scan's window (the longest reader's) | its input depends on other groups' choices | derived at the reader's own window | a cut when a longer reader is chosen | | A child consumed by an ingestion-time build | the child runs at ingestion time instead | the cheaper of the two pricings | a cut on the pair | | A chain kept across evaluations, charged per request | depends on the request count | the per-evaluation price | a cut | | Compaction with several ingest workers | depends on the global worker count | 0 | a cut | -| Composed accuracy | depends on the chosen children | not priced | a no-good cut on a failing combination | +| Composed accuracy | depends on the chosen children | no coefficient | a no-good cut on a failing combination | Validation: for every plan of the brute force over the memo, the sum of its coefficients is at most Stage 3's price, and equal wherever the term decomposes exactly. @@ -573,7 +591,7 @@ The report gives the number of rounds and the first round's bound against the fi - **Raw data on disk**: a per-source horizon threshold, linear in the MILP. - **Cost models 2 and 3**: peaks and limits as linear constraints. -- **A measured cost table**: the calibration factor is fitted once, before the search, over every candidate node the generation produced (before pruning), and frozen; every coefficient is priced with it. +- **A measured cost table**: the calibration factor is fitted once, before the search, over every candidate node the generation produced (before pruning), and frozen; every coefficient is derived with it. ### Code layout From 2f961b1ec1908606e99eb7b4d6c5b3ecb9fa45d0 Mon Sep 17 00:00:00 2001 From: zzylol <50204836+zzylol@users.noreply.github.com> Date: Fri, 23 Oct 2026 01:06:45 +0000 Subject: [PATCH 19/53] docs: name the MILP lower bound and the report's milp fields --- docs/design_docs/proposals/joint-planning-search.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/design_docs/proposals/joint-planning-search.md b/docs/design_docs/proposals/joint-planning-search.md index 0d0ecae93..16318626c 100644 --- a/docs/design_docs/proposals/joint-planning-search.md +++ b/docs/design_docs/proposals/joint-planning-search.md @@ -365,7 +365,7 @@ The result is the plan the exhaustive search would select. It is checked by `joi 2. **Generate** each group's implementation candidates (Pass 1, then the Pass 2 transformations), each with its child groups and its coupling sites. 3. **Coefficient derivation.** Derive each choice's objective coefficients from Stage 3's per-node cost function (`price_node`): a candidate's private nodes per consumer rate and per materialization, and each coupling site's node at each of its thresholds. Stage 3 stays the only source of costs. 4. **Solve one MILP over the memo** (below): one choice per needed group, its materializations and the coupling sites' thresholds. -5. **Solution decoding, incumbent evaluation, lazy cuts.** Decode the solver's solution into its one physical plan, recompose its guarantees, and evaluate the plan's exact cost with Stage 3 (the incumbent cost). If the incumbent cost exceeds the model bound (the held-memory term), or a composed guarantee fails, add a cut that corrects or excludes that plan, and solve again. Stop when the best incumbent cost equals the model bound and every check passes. The gap must be 0 to report the plan as optimal. +5. **Solution decoding, incumbent evaluation, lazy cuts.** Decode the solver's solution into its one physical plan, recompose its guarantees, and evaluate the plan's exact cost with Stage 3 (the incumbent cost). If the incumbent cost exceeds the MILP lower bound (the held-memory term), or a composed guarantee fails, add a cut that corrects or excludes that plan, and solve again. Stop when the best incumbent cost equals the MILP lower bound and every check passes. The gap must be 0 to report the plan as optimal. 6. **Build late.** Run the executor's compile check on the selected plan only, and report it. ### The solver: MILP over the memo @@ -542,7 +542,7 @@ Each cut removes or corrects at least one of finitely many combinations, so the | **Solution decoding** | Turning a MILP solution (the values of `x`, `m`, `u`, …) into the one physical plan it describes. | after each solve | once per round | | **Incumbent evaluation** | Running Stage 3 on that decoded plan as a whole: guarantees, latency checks and its exact cost (the **incumbent cost**). | after decoding | once per round | -The **model bound** is the solver's objective, a lower bound on every plan's cost. The **gap** is (incumbent cost − model bound) / incumbent cost. +The **MILP lower bound** is the solver's objective, a lower bound on every plan's cost. The **gap** is (incumbent cost − MILP lower bound) / incumbent cost. The report keeps `incumbent_cost` at the search level and the MILP-only figures in a `milp` object: `lower_bound`, `first_lower_bound`, `gap`, `rounds`, `variable_count`, `constraint_count`, `solver_time_ms`, `time_limit_reached`. **One solution, one plan.** Every choice that changes a plan's cost is a model variable, so a solution decodes to exactly one plan, and incumbent evaluation runs Stage 3 once per round: - sharing an identical expression or computing it per reader is the materialization `m` (`mq` vs `nm`); From 8ed66a7d7cda07d97f8286b7ea6ab2c8489f47fe Mon Sep 17 00:00:00 2001 From: zzylol <50204836+zzylol@users.noreply.github.com> Date: Fri, 23 Oct 2026 03:10:28 +0000 Subject: [PATCH 20/53] docs: sharing as MQO, NP-hardness, solution families, LBBD algorithm --- .../proposals/joint-planning-search.md | 71 +++++++++++++++++++ 1 file changed, 71 insertions(+) diff --git a/docs/design_docs/proposals/joint-planning-search.md b/docs/design_docs/proposals/joint-planning-search.md index 16318626c..03e8df5f0 100644 --- a/docs/design_docs/proposals/joint-planning-search.md +++ b/docs/design_docs/proposals/joint-planning-search.md @@ -198,6 +198,77 @@ Exact search finishes when every remaining branch has been explored or certified Validate each rule against exhaustive enumeration on small workloads, alone and in combination. Include donor-dependent sharing, global memory/core limits, overlapping reuse, per-source retention, repairable accuracy failures, and near ties. Verify bounds against the best exhaustive completion of each tested partial state. Measure planning time and peak memory on Examples 1–4, the mixed workloads, and Kubernetes recording rules without claiming a universal scaling guarantee. +## The problem: sharing is a decision + +The main difficulty: whether to share, and at what granularity, is a decision the solver must make, and this is multi-query optimization (MQO). + +The problem we solve. Given a workload of queries, each with an accuracy target and a repeat interval, choose for every query one way to compute it, and for every computation it reads, whether it is computed alone or shared with other queries, and at which granularity (a sketch, a pane chain, a scan, a finer grouping rolled up). The choice must meet every query's accuracy and latency, fit the deployment's memory and cores, and minimize the workload's cost under the chosen cost model. + +Three properties make the decision hard. The cost of sharing depends on who shares: + +1. **A query can read several shared units.** Its latency is a sum over the units it reads, and its held memory is that sum times the latency. For example, `max_over_time(m[5m]) - min_over_time(m[5m])` together with `max_over_time(m[5m])`: the first query reads two units, one shared with the second. Pricing each unit alone overestimates the pair (0.4617 against an exact 0.4165). +2. **A unit's cost depends on its reader set.** A shared unit runs at the union of its readers' repeat intervals (60 s and 90 s → 30 s), keeps data for the longest reader's window (5 min vs 24 h), and is sized for the strictest reader's accuracy (0.01 vs 0.05). Adding one reader can make a unit more expensive for all the others. +3. **Units interact.** A shared scan reads the longest reader's window, which changes the inputs, and so the costs, of the units above it. + +### Is this MQO? + +Yes. The memo is MQO's AND-OR DAG: groups are the OR nodes (alternative ways to compute one sub-expression), implementation candidates are the AND nodes (one way, reading its child groups), and choosing a sub-DAG that covers every query, with shared nodes counted once, is the classic MQO problem. + +It is NP-hard. Classic MQO is NP-hard (Sellis 1988): choosing which shared nodes to compute is at least as hard as set cover, since each shared node covers the queries that read it. Our problem contains classic MQO as the special case where sizes, accuracy and time do not depend on the reader set, so it is NP-hard too, and no exact algorithm is polynomial in the worst case. Our problem goes beyond classic MQO in four ways: a shared node's size depends on its strictest reader; accuracy composes across the nodes a query reads; time matters (pane grids, retention, repeat intervals); and memory held during a query multiplies by its latency. + +### How MQO is solved + +| Family | Idea | Fit here | +|---|---|---| +| Greedy materialization | Add the shared node with the best benefit until none helps. | No optimality bound; misses sharing that pays only in combination. | +| A* / branch and bound | Search plan combinations, prune by a cost bound. | Exact, but the current enumerator is this, and it does not finish on whole workloads. | +| Integer programming | Binary variables per node and per choice; a solver finds the optimum. | Exact, but reader-set-dependent and multiplicative costs are not linear. | +| Logic-based Benders decomposition | A MILP master chooses with linear lower-bound costs; a subproblem prices the choice exactly and returns cuts. | Fits: Stage 3 is the exact subproblem. | +| Column generation (branch and price) | Each column is a whole sharing pattern, priced exactly; a pricing problem finds new columns. | Fits reader-set-dependent costs exactly; columns grow with reader subsets. | + +### How we solve it + +Logic-based Benders decomposition. The MILP over the memo is the master: it chooses candidates and sharing with linear coefficients that are lower bounds of the true cost. Stage 3 evaluates the decoded plan exactly. When the exact cost exceeds the master's estimate, a cut returns to the master. The loop ends when the incumbent's cost is within the gap of the master's lower bound, so the result is optimal within that gap. + +The algorithm: + +```text +input: memo M, cost model C, gap tolerance g, time limit T +output: plan P*, its exact cost U, lower bound L + +build master MILP from M: + variables x (candidate chosen), m (group materialized), + u (shared unit used), r (reader of unit), ... + constraints: every query covered; a chosen candidate's child groups + are computed; validity (accuracy, latency, capacity) + objective: sum of lower-bound coefficients of the chosen terms +cuts := {} +U := +inf; P* := none +loop: + solve master with cuts -> solution s, objective L + if master infeasible: return no plan + P := decode(s) # exactly one plan + c := Stage 3 exact cost of P under C + if c < U: U := c; P* := P # incumbent + if (U - L) / U <= g: return (P*, U, L) # optimal within g + if time limit T reached: return (P*, U, L) # reported with its gap + for each term t of P whose exact cost exceeds its coefficient: + add cut: (t chosen) => (objective >= exact cost of t) + if no term cut applies: + add no-good cut excluding s +``` + +Two properties make it correct. Every coefficient is a lower bound of the exact cost of its term, so L never exceeds the optimum. Every cut is valid: it removes no plan whose exact cost is below what the cut states. Together, when the loop stops at gap g, P* is within g of the optimum. + +The number of rounds depends on the strength of the master and the cuts, so the work goes there, in the standard ways: + +- **Validity before cost.** Invalid candidates are pruned per candidate; composed accuracy and path latency are MILP constraints, not cuts. +- **Stronger cuts.** A cut prices the terms that were underestimated (a reader set's union rate, retention, size; a query's held memory), so one cut excludes every plan sharing that term, not just one plan. +- **Decomposition into clusters.** Groups that share nothing are solved as separate MILPs. +- **Decoding.** One master solution decodes to exactly one plan. + +For a small cluster whose costs depend heavily on the reader set, column generation prices each sharing pattern exactly and needs no cuts for it. It is a complement for those clusters, not a replacement for the master. No heuristic replaces the exact search; a run that hits its time limit reports its gap. + ## Memo-group search The rules above prune a search that still builds and prices whole-workload candidates, one at a time, at about a millisecond each: RangeHeavy's 24-hour quantiles component checks 153,086 of them in 92 s, and whole workloads link most queries over one source into one component (shared scans, SQL windows as time ranges, sketch rollup). The memo-group search replaces whole-candidate enumeration with a Cascades-style memo over the workload's logical sub-expressions. The stages keep their semantics and code boundaries; what changes is the unit they exchange. From c8400cfbafc75c7eb4d74118088c111b0316cf3c Mon Sep 17 00:00:00 2001 From: Zeying Zhu <50204836+zzylol@users.noreply.github.com> Date: Fri, 23 Oct 2026 14:25:07 -0400 Subject: [PATCH 21/53] Update joint-planning-search.md --- .../proposals/joint-planning-search.md | 31 +++---------------- 1 file changed, 4 insertions(+), 27 deletions(-) diff --git a/docs/design_docs/proposals/joint-planning-search.md b/docs/design_docs/proposals/joint-planning-search.md index 03e8df5f0..566aa798b 100644 --- a/docs/design_docs/proposals/joint-planning-search.md +++ b/docs/design_docs/proposals/joint-planning-search.md @@ -1,24 +1,7 @@ # Joint planning search and exact pruning -Status: proposal; this document does not implement new pruning. Audience: planner designers and implementers. - -ASAPPlanner searches per-query alternatives, sharing choices, and materializations to select a whole-workload plan. The design preserves the stages in [planner-layering.md](planner-layering.md) while reducing redundant search. It follows the [survey in #671](https://github.com/ProjectASAP/ASAPPlanner/issues/671#issuecomment-6137638876). - ## Contents -- [Problem](#problem) -- [Goals and non-goals](#goals-and-non-goals) -- [Where the search space comes from](#where-the-search-space-comes-from) -- [Optimizer search survey](#optimizer-search-survey) -- [Exactness contract](#exactness-contract) -- [Design overview](#design-overview) -- [Exact pruning methods](#exact-pruning-methods) -- [Interfaces](#interfaces) -- [Worked example](#worked-example) -- [Search and validation](#search-and-validation) -- [Memo-group search](#memo-group-search): how the stages and the memo work together, and an end-to-end walk-through with the sharing strategies -- [Future work](#future-work) - ## Problem Planning a workload jointly lets queries share computation: one summary answers several queries, identical sub-expressions run once, and panes are reused across evaluations. Today Stage 1 to Stage 3 enumerate every whole-workload candidate, price each one, and keep all of them in memory. @@ -68,15 +51,11 @@ Most combinations cannot be selected. Either a cheaper alternative replaces them | [MILP](https://arxiv.org/abs/1511.02071) | Encodes plan choices and constraints for a solver. | Optimality applies to the encoded alternatives and objective, including any cost approximations. | | [Bounded/randomized search](https://www.postgresql.org/docs/17/runtime-config-query.html#RUNTIME-CONFIG-QUERY-GEQO) | Limits exploration to control planning time. | Can miss the exhaustive optimum. | -For ASAPPlanner, retain equivalent search states, reject impossible branches, and prune only with a valid workload-level bound or replacement proof. Sharing prevents a simple “keep the cheapest plan per query” DP from being exact. - -## Exactness contract + + +A memo group is standard optimizer terminology for logically equivalent expressions; see [Cascades, Section 2](https://15721.courses.cs.cmu.edu/spring2019/papers/22-optimizer1/graefe-ieee1995.pdf) and [CockroachDB's optimizer documentation](https://github.com/cockroachdb/cockroach/blob/master/pkg/sql/opt/doc.go#memo). Memorization group was proposed in the paper *[The Cascades Framework for Query Optimization](https://15721.courses.cs.cmu.edu/spring2019/papers/22-optimizer1/graefe-ieee1995.pdf)*, which is the central data structure used to compactly represent and explore the huge search space of equivalent query execution plans. The Memo group data structure is compact representation of the search space as a DAG, instead of building the full DAG(s) of a batch of input queries. It can reuse the common sub-expressions and tracks their costs and physical properties. Each Group represents a set of logically equivalent relational expressions (sub-queries or relations) that all produce the same output result. -“Exact” means preserving the optimum of the existing finite candidate space under the configured cost, accuracy, and capability models. It does not guarantee the fastest measured runtime or cover alternatives the generator never offers. Freeze the workload, rule/parameter inventory, and model/evidence inputs for each search. - -A memo group is standard optimizer terminology for logically equivalent expressions; see [Cascades, Section 2](https://15721.courses.cs.cmu.edu/spring2019/papers/22-optimizer1/graefe-ieee1995.pdf) and [CockroachDB's optimizer documentation](https://github.com/cockroachdb/cockroach/blob/master/pkg/sql/opt/doc.go#memo). Approximate alternatives must also carry their output-accuracy obligations. - -A reusable search state must expose everything later choices can observe: source and expression semantics, schema, summary parameters and guarantees, consumers, windows, recurrence, retention, materialization, physical properties, and resource obligations. Unknown interactions remain connected; a matching hash alone never proves equivalent states. + ## Design overview @@ -271,8 +250,6 @@ For a small cluster whose costs depend heavily on the reader set, column generat ## Memo-group search -The rules above prune a search that still builds and prices whole-workload candidates, one at a time, at about a millisecond each: RangeHeavy's 24-hour quantiles component checks 153,086 of them in 92 s, and whole workloads link most queries over one source into one component (shared scans, SQL windows as time ranges, sketch rollup). The memo-group search replaces whole-candidate enumeration with a Cascades-style memo over the workload's logical sub-expressions. The stages keep their semantics and code boundaries; what changes is the unit they exchange. - ### Components | Component | What it is | From 12bb072624ec58599ada7db8f5d920a05cd5bda0 Mon Sep 17 00:00:00 2001 From: zzylol <50204836+zzylol@users.noreply.github.com> Date: Fri, 23 Oct 2026 22:32:10 +0000 Subject: [PATCH 22/53] docs: rewrite joint planning as memo group and MILP design --- docs/design_docs/proposals/README.md | 2 +- .../proposals/joint-planning-search.md | 969 +++++++----------- 2 files changed, 390 insertions(+), 581 deletions(-) diff --git a/docs/design_docs/proposals/README.md b/docs/design_docs/proposals/README.md index 1c10845d3..3075e1945 100644 --- a/docs/design_docs/proposals/README.md +++ b/docs/design_docs/proposals/README.md @@ -11,5 +11,5 @@ extensions. A design document is not a promise of downstream runtime support. - [Operator sharing](operator-sharing.md) - [Decoupling operators from scalar expressions](decoupling_op_and_expr.md) - [ASAPPlanner layering](planner-layering.md) -- [Joint planning search](joint-planning-search.md) +- [Memo group and MILP design](joint-planning-search.md) - [Schema and physical data for ASAP primitives](asap-primitive-schema.md) diff --git a/docs/design_docs/proposals/joint-planning-search.md b/docs/design_docs/proposals/joint-planning-search.md index 566aa798b..ff11a138f 100644 --- a/docs/design_docs/proposals/joint-planning-search.md +++ b/docs/design_docs/proposals/joint-planning-search.md @@ -1,653 +1,462 @@ -# Joint planning search and exact pruning +# Memo group and MILP design -## Contents +Status: design proposal. Audience: planner designers and implementers. -## Problem +ASAP primitives introduce sharing opportunities across queries, but also enlarge +ASAPPlanner's optimization search space. The optimizer must represent these +opportunities compactly and select a low-cost, valid physical plan for the whole +workload. This design uses memo groups for candidate generation and a mixed-integer +linear program (MILP) for joint selection. It preserves the responsibilities of +the stages in [planner-layering.md](planner-layering.md). -Planning a workload jointly lets queries share computation: one summary answers several queries, identical sub-expressions run once, and panes are reused across evaluations. Today Stage 1 to Stage 3 enumerate every whole-workload candidate, price each one, and keep all of them in memory. +## Challenges and goals -That does not scale past a handful of queries. Planning a growing prefix of one 10-query workload of ASAPQuery's random mixed workloads: +Whole-workload enumeration multiplies per-computation candidates, sharing +configurations, and materialization assignments. The earlier joint-search +measurements on a growing prefix of one random mixed workload illustrate this +problem: -| Queries | Time | Peak memory | +| Queries | Planning time | Peak memory | | ---: | ---: | ---: | | 3 | 0.4 s | 0.14 GB | | 4 | 5 s | 1.5 GB | | 5 | 129 s | 24.7 GB | | 6 | stopped at about 228 s | about 100 GB | -Each added query multiplies the work by roughly 20. None of those workloads, and not the Kubernetes recording-rule set, can be planned as a whole. Each query planned alone takes milliseconds, but then sharing across queries is lost. - -## Goals and non-goals - -**Goals** - -- Joint planning of realistic workloads (tens of queries) within seconds and a few GB. -- The same selected plan as exhaustive search over the same candidate space (see [Exactness contract](#exactness-contract)). -- Bounded search that says what it did: when a budget stops the search, the result reports it and whether optimality is certified. - -**Non-goals** - -- Changing what the stages decide. Candidate generation, sharing rules, materializations, and the accuracy and cost checks stay as they are. -- Approximate search modes. They are future work and must be labeled as such. -- Incremental replanning. - -## Where the search space comes from - -| Stage | Choice | Grows as | -| --- | --- | --- | -| Stage 1, Pass 1 | One per-computation candidate per target | the product of the targets' candidate counts | -| Stage 1, Pass 2 | Sharing alternatives of each composed candidate: identical expressions, summary capability, window composition, rollup, top-k reuse | the subsets of the sharing opportunities | -| Stage 2 | One materialization per summary build: at ingestion, at query time, or at query time kept across evaluations | three per build | - -Most combinations cannot be selected. Either a cheaper alternative replaces them in every completion, or their choices in one part of the workload do not interact with another part. The methods below remove those combinations without removing the optimum. - -## Optimizer search survey - -| Approach | How it searches | Guarantee | -| --- | --- | --- | -| [System R-style DP](https://research.ibm.com/publications/access-path-selection-in-a-relational-database-management-system) | Builds larger plans from smaller subproblems, retaining alternatives with useful physical properties. | Exact within its enumerated plan space when the state captures every property needed by a parent. | -| [Volcano/Cascades](https://15721.courses.cs.cmu.edu/spring2019/papers/22-optimizer1/graefe-ieee1995.pdf) | Stores equivalent expressions in memo groups; explores alternatives on demand with cost limits. | Requires sufficient context in memo keys and valid pruning bounds. | -| [Multi-query optimization](https://arxiv.org/abs/cs/9910021) | Chooses which common computations to build, materialize, and reuse across queries. | Greedy sharing is practical but generally heuristic; consumers' choices are coupled. | -| [MILP](https://arxiv.org/abs/1511.02071) | Encodes plan choices and constraints for a solver. | Optimality applies to the encoded alternatives and objective, including any cost approximations. | -| [Bounded/randomized search](https://www.postgresql.org/docs/17/runtime-config-query.html#RUNTIME-CONFIG-QUERY-GEQO) | Limits exploration to control planning time. | Can miss the exhaustive optimum. | - - - -A memo group is standard optimizer terminology for logically equivalent expressions; see [Cascades, Section 2](https://15721.courses.cs.cmu.edu/spring2019/papers/22-optimizer1/graefe-ieee1995.pdf) and [CockroachDB's optimizer documentation](https://github.com/cockroachdb/cockroach/blob/master/pkg/sql/opt/doc.go#memo). Memorization group was proposed in the paper *[The Cascades Framework for Query Optimization](https://15721.courses.cs.cmu.edu/spring2019/papers/22-optimizer1/graefe-ieee1995.pdf)*, which is the central data structure used to compactly represent and explore the huge search space of equivalent query execution plans. The Memo group data structure is compact representation of the search space as a DAG, instead of building the full DAG(s) of a batch of input queries. It can reuse the common sub-expressions and tracks their costs and physical properties. Each Group represents a set of logically equivalent relational expressions (sub-queries or relations) that all produce the same output result. - - - -## Design overview - -The search keeps the stages' semantics and changes only how their choices are explored. - -**Components** - -| Component | Role | +These are historical measurements, not a benchmark of the proposed design. + +The goals are to reduce duplicate candidate generation, avoid enumerating the +Cartesian product of whole-workload plans, and preserve the minimum cost over the +configured finite candidate space. A time-limited search must return its best +validated plan and report whether optimality is proved. Incremental replanning +and new window algorithms are separate work. + +## Memo groups and candidate generation + +A memo group identifies one logical computation and its output contract. Exact +candidates produce the same logical result. Approximate candidates implement that +result subject to explicit accuracy obligations; they are not exactly equivalent +expressions. A group is not a sketch family or a collection of all computations +that could share a sketch. + +Build the initial memo after Stage 0, before Stage 1 Pass 1: + +- Create a group for each distinct logical sub-expression. +- Merge identical expressions only where semantic identity and sharing legality + are established; verify equality after hashing. Preserve schema, grouping, + source-language semantics, and time semantics. Keyless nodes can require + separate identities per reader. +- Record group dependencies, query roots, and reader requirements. + +Pass 1 generates candidates directly against these group identities. Cache +candidate generation by group and all relevant context, including accuracy +requirements. Different readers may need different summary sizes; merging their +logical identity must not discard candidates needed by a reader. This avoids +repeating generation for identical computations without first composing every +whole-query plan. + +### Memo grouping versus CSE + +Memo grouping represents the search space. Common subexpression elimination +(CSE) identifies computations that can reuse work. A repeated identical +expression can have one memo group with multiple parents, but that identity alone +does not require one materialized execution at runtime. + +ASAP-aware CSE also identifies sharing between different logical computations: +summary capability sharing, pane composition, rollup, top-k reuse, and scan +widening. These rules add candidates and dependencies between groups; they do not +merge groups whose requested results differ. + +For example, p95 and p99 over the same input window are separate result groups. +Their KLL candidates may share one build, with separate estimation nodes. A +coupling site identifies the shared configuration decisions, such as build size +or a pane chain's grid and retention span. The solver must select compatible +configurations for all attached readers. A shared build can instead be represented +as an explicit intermediate group; either representation must preserve the same +choices and charge ownership. + +### Responsibilities of the stages + +| Stage | Decisions represented in the memo | | --- | --- | -| Search state | A partial whole-workload plan: the choices made so far, plus everything later choices can observe (see [Exactness contract](#exactness-contract)). | -| Memo | Equivalent search states, grouped so that each is expanded once (method 1). | -| Interaction graph | Which computations can interact through sharing or shared resource charges. Its connected components are searched separately when that is exact (method 5). | -| Lower bound | A valid lower bound on the cost of every completion of a state (method 3). | -| Incumbent | The best validated complete plan so far, or a resource frontier when global limits apply. | -| Budget | Limits on states expanded or time, with a report of unresolved work when it stops the search. | - -**Search loop** - -1. Build the interaction graph from all alternatives, and split the workload into components where independence is proven (method 5). -2. For each component, generate states lazily, in the existing enumeration order. -3. For each new state, apply in order: - - duplicate elimination (method 1); - - permanent-invalidity rejection (method 2); - - lower-bound pruning against the incumbent (method 3); - - contextual dominance (method 4). -4. When a materialization or sharing choice becomes isolated, resolve it locally (methods 6 and 7) instead of branching on it. -5. Complete plans go through the existing validation and ranking path, and update the incumbent. -6. Combine the components' results: the sum of their optima when the objective is additive and feasibility factors, otherwise their resource frontiers. - -Every pruning step requires its stated precondition. When the precondition is unknown, the alternative stays in the search. - -## Exact pruning methods - -Each method applies only when its stated conditions hold. Otherwise retain the alternative. These are correctness conditions, not promises of a particular pruning rate. - -### 1. Eliminate duplicate states - -Expand one representative when two states have the same observable interface, remaining choices, and cost/resource obligations. Verify equality after hashing and retain required provenance and tie information. - -This is exact because both states have the same completions. Memoizing equivalent computations does not force their runtime results to be shared. - -### 2. Reject permanently invalid branches - -Reject a branch only when no remaining choice can satisfy semantics, capabilities, executor support, or output accuracy. Examples include an unrepairable type error or a fixed unsupported operator with no legal replacement. - -Defer failures that another choice can repair: resizing can improve accuracy, changing summary family can supply evidence, and ingestion materialization can remove a query-time raw-data requirement. Recheck completed plans with the authoritative validator. - -### 3. Prune using a workload lower bound - -For a partial state `S`, require `LB(S) <= cost(P)` for every feasible completion `P`. With a validated incumbent of cost `U`, prune only when `LB(S) > U`, conservatively accounting for numeric error and the selection tie policy. A resource lower bound exceeding a hard limit also rejects the branch. - -Start with zero for nonnegative objectives, then add proven unavoidable, nonshareable charges, counted once. Potentially shareable or replaceable work contributes zero until a stronger bound is proved. A partial plan's current cost, or a sum of standalone query costs, is not automatically a lower bound. - -### 4. Remove contextually dominated alternatives - -Discard B only if every completion of B can be replaced by a feasible completion of A with compatible outputs and reusable producers, no higher objective, and no worse constrained resources. A sufficient case is an identical boundary interface with a componentwise no-worse resource/cost vector under a monotone composition rule. - -Keep useful resource tradeoffs and sharing capabilities. For example, a producer costing 10 plus private work costing 6 can let another query cost 1 instead of 100. Replacing that first query with a standalone plan costing 5 can lose the joint plan costing 17. Standalone price alone does not establish dominance. - -### 5. Split independent components - -Analyze possible interactions across **all** alternatives: identical computations, capability sharing, rollup, top-k reuse, windows, and shared resource charges. Include interactions exposed by later rewrites. If independence is unknown, keep the components together. - -Choose each component's scalar optimum separately only when feasibility factors and the objective is additive. With global capacity limits, retain and combine resource frontiers instead. Include cores as well as memory, disk, and objective dimensions. Raw retention uses the maximum horizon per source, so source accounting can couple otherwise separate computations. - -### 6. Resolve isolated materialization choices locally - -Choose the cheapest local realization only after fixing its inputs, outputs, lifetimes, and required properties, and proving that it has no remaining cost or resource interaction. Otherwise retain nondominated realizations for that interface. - -An unshared build is not automatically isolated: it can share a scan or affect ancestor working memory and raw retention. Under a memory limit of 2, alternatives `(cost 1, memory 10)` and `(cost 2, memory 1)` require keeping the second option. - -### 7. Force sharing only when it dominates recomputation - -First apply the existing [CSE legality checks](../../../crates/types/src/ir/cse.rs). Then eliminate recomputation only if the shared physical realization is no worse on every relevant cost, resource, and consumer property. - -Equal expressions alone do not prove this: sharing can extend retention or introduce copying, synchronization, or scheduling costs. When dominance is unproved, retain both physical alternatives and memoize their common semantics. - -## Interfaces - -These are design-level interfaces; names in the implementation may differ. - -**Inputs** -- The existing Stage 1 to Stage 3 inputs: workload, planning models, plan selection settings. -- A search budget in plan selection settings: limits on expanded states and on time, with a large default. -- An exhaustive mode, used by tests to compare results. - -**Outputs** -- The selected plan, as today. -- A search report: - - the components found; - - per component: states generated, states pruned by each method, and plans validated; - - whether the search completed; - - if a budget stopped it: the best feasible plan found, the unresolved work, and whether optimality is certified. -- Developer tools can still request the full candidate lists; normal planning keeps only the incumbent. - -## Worked example - -A workload of three recurring queries: - -```promql -q1: sum by (job) (rate(http_requests_total[5m])) -q2: topk(5, sum by (job) (rate(http_requests_total[5m]))) -q3: quantile_over_time(0.99, latency_ms[5m]) -``` - -- **Method 5.** q1 and q2 read the same source and can share the per-job rate sum (identical computation and top-k reuse), so they form one component. q3 reads another source. It has no sharing opportunity with q1 or q2, so under an additive objective with no global limit it is searched on its own. With a global memory or core limit, the two components keep resource frontiers and are combined at the end. -- **Method 1.** In the {q1, q2} component, the composed candidates that differ only in how the identical rate sum is numbered are one state. -- **Method 7.** Sharing the per-job rate sum is forced only if the shared realization is no worse on every relevant cost, resource and consumer property. Otherwise both the shared and the separate realization stay. -- **Method 6.** q3's per-series KLL panes have no interaction with the other component. Their materialization can be chosen locally once their inputs, lifetimes and raw-retention effect are fixed. If the choice changes raw retention for `latency_ms`, the nondominated realizations are kept instead. -- **Method 3.** Once a validated plan for {q1, q2} exists, a partial state whose unavoidable, nonshareable charges already exceed it is pruned. - -The selected plan is the same as exhaustive search's. The number of states expanded drops from the product over all three queries to roughly the sum over the components. - -## Search and validation - -Generate candidates lazily and stream completed plans through the existing validation and ranking path. Apply the rules above as soon as their preconditions are known. Unknown bounds and numerical near-ties remain searchable. Preserve legacy selection order for tolerance-sensitive ties unless a new total-order policy is explicitly adopted; stable IDs alone do not reproduce a nontransitive tolerance comparison. - -Exact search finishes when every remaining branch has been explored or certified unable to improve the result. A budget stop reports the best feasible plan found, unresolved work, and whether optimality is certified. No incumbent means “no feasible plan found,” not “infeasible.” Report a scalar gap only with a bound covering all unresolved branches; report Pareto-front completeness separately. - -Validate each rule against exhaustive enumeration on small workloads, alone and in combination. Include donor-dependent sharing, global memory/core limits, overlapping reuse, per-source retention, repairable accuracy failures, and near ties. Verify bounds against the best exhaustive completion of each tested partial state. Measure planning time and peak memory on Examples 1–4, the mixed workloads, and Kubernetes recording rules without claiming a universal scaling guarantee. - -## The problem: sharing is a decision - -The main difficulty: whether to share, and at what granularity, is a decision the solver must make, and this is multi-query optimization (MQO). - -The problem we solve. Given a workload of queries, each with an accuracy target and a repeat interval, choose for every query one way to compute it, and for every computation it reads, whether it is computed alone or shared with other queries, and at which granularity (a sketch, a pane chain, a scan, a finer grouping rolled up). The choice must meet every query's accuracy and latency, fit the deployment's memory and cores, and minimize the workload's cost under the chosen cost model. - -Three properties make the decision hard. The cost of sharing depends on who shares: - -1. **A query can read several shared units.** Its latency is a sum over the units it reads, and its held memory is that sum times the latency. For example, `max_over_time(m[5m]) - min_over_time(m[5m])` together with `max_over_time(m[5m])`: the first query reads two units, one shared with the second. Pricing each unit alone overestimates the pair (0.4617 against an exact 0.4165). -2. **A unit's cost depends on its reader set.** A shared unit runs at the union of its readers' repeat intervals (60 s and 90 s → 30 s), keeps data for the longest reader's window (5 min vs 24 h), and is sized for the strictest reader's accuracy (0.01 vs 0.05). Adding one reader can make a unit more expensive for all the others. -3. **Units interact.** A shared scan reads the longest reader's window, which changes the inputs, and so the costs, of the units above it. - -### Is this MQO? - -Yes. The memo is MQO's AND-OR DAG: groups are the OR nodes (alternative ways to compute one sub-expression), implementation candidates are the AND nodes (one way, reading its child groups), and choosing a sub-DAG that covers every query, with shared nodes counted once, is the classic MQO problem. - -It is NP-hard. Classic MQO is NP-hard (Sellis 1988): choosing which shared nodes to compute is at least as hard as set cover, since each shared node covers the queries that read it. Our problem contains classic MQO as the special case where sizes, accuracy and time do not depend on the reader set, so it is NP-hard too, and no exact algorithm is polynomial in the worst case. Our problem goes beyond classic MQO in four ways: a shared node's size depends on its strictest reader; accuracy composes across the nodes a query reads; time matters (pane grids, retention, repeat intervals); and memory held during a query multiplies by its latency. +| Stage 0 | Parse and lower queries into logical DAGs; establish query semantics. | +| Initial memo construction | Establish logical group identities, dependencies, and query roots. | +| Stage 1, Pass 1 | Add exact, exact-accumulator, summary-family, summary-size, and fused-computation candidates. | +| Stage 1, Pass 2 | Add ASAP-aware CSE candidates, shared configurations, and cross-group dependencies. | +| Stage 2 | Generate legal physical implementation and materialization candidates, including assignments to internal nodes. | +| Stage 3 | Jointly select candidates under the configured objective and constraints; validate the decoded plan. | + +Candidate generation can be lazy. Only provably invalid or contextually dominated +candidates may be removed without changing the finite-space optimum. A higher +standalone cost is not enough to discard a candidate that enables sharing. + +## Cost-based joint optimization + +Assign resource usage to the nodes of the selected physical DAG using the +configured cost model. The objective can minimize billed cost, including CPU, +memory, storage, and other priced resources. In a cluster without billing prices, +select an explicit objective, such as latency or memory, subject to capacity +limits. Capacity limits alone do not define an optimum. + +A shared computation is charged once per actual execution, not once per reader. +An ingestion-maintained build has update and retained-state costs; query-time +work has costs at its actual trigger rate. Multiple consumers can still cause +multiple executions of an unmaterialized node. Estimation and merge work private +to each reader is charged separately. + +### Why joint selection is needed + +Consider two result groups, a quantile at 0.5 and a quantile at 0.9, over the same +input window. Assume their private KLL implementations cost 4 and 6 units per +second. A shared build satisfying both readers costs 7, with each estimator +costing 0.1. All configurations in this example meet accuracy requirements. + +Optimizing the groups independently selects the private implementations for a +combined cost of 10. Joint selection can choose the shared build for 7.2. The +incremental cost of attaching a reader depends on whether another selected +candidate already enables the build. It can also depend on whether that reader +requires a larger build, a longer retention horizon, or more frequent evaluation. + +A dynamic program retaining only the cheapest candidate per group loses this +context. DP is possible if its state includes the relevant reader sets, shared +configurations, materializations, and resource tradeoffs, but those states can +grow combinatorially. MILP provides a general engine for these joint discrete +choices. Local DP and caching remain useful where dependencies are limited. + +## Window model + +Distinguish three dimensions: + +1. **Window semantics:** width, evaluation step, time origin/alignment, and endpoint + conventions. A 15m trailing window evaluated every 1m is a sliding-window query. + With width and step both 15m and aligned boundaries, windows are tumbling. +2. **Window implementation:** how the required data is represented and combined, + such as raw recomputation, direct sliding-state maintenance, fixed-width panes, + or exponential histogram (EH) buckets. +3. **Materialization:** when each internal state is computed and how long it is + retained. Sliding-window semantics do not prevent ingestion-time maintenance. + +### Panes and EH buckets + +**Panes** partition a timeline into aligned, non-overlapping intervals of a fixed +width. Each pane contains data or aggregate/summary state for its interval. A +window combines the required panes, with explicit boundary handling when its +endpoints do not align. An implementation may maintain several pane widths or a +hierarchy where larger panes are formed from smaller ones. + +**EH buckets** follow an exponential histogram algorithm: eligible buckets are +merged into geometrically increasing size levels and expired buckets are +removed. Standard EH sizes commonly measure represented item counts, rather than +fixed time durations. The algorithm must define the statistic it supports and +bound error from partially overlapping the oldest bucket. EH cannot be assumed +to work with every summary family, including an arbitrary KLL composition. + +Pane width and EH bucket size are separate from the summary-size parameters +controlling the state within each pane or bucket. + +### Window configurations + +For a 15m window, candidate pane compositions might include 15 × 1m, 5 × 3m, +3 × 5m, or a hierarchy combining fine boundary panes and coarser interior panes. +These are candidates only when they match the query's evaluation boundaries or +include a valid boundary treatment. For a 15m window evaluated every 1m, complete +5m panes alone cannot generally represent every evaluation exactly. + +The solver may enable multiple pane widths simultaneously. It chooses which +states to maintain, whether they are built from raw input or finer panes, and +which composition each query reads. Do not impose a global constraint selecting +only one pane width. Shared configuration sites must distinguish incompatible +chains rather than collapse every pane computation into one site. + +| Implementation | Ingestion-time work, if maintained there | Evaluation work | +| --- | --- | --- | +| Direct sliding state | Insert new data and expire old contributions, where the statistic supports it. | Read/finalize current state. | +| Pane-based | Update the active pane and maintain retention. | Read, merge, and estimate from the required panes, unless those results are also maintained. | +| EH-based | Update, merge, and expire buckets according to the supported algorithm. | Apply its window estimator. | -### How MQO is solved +This table describes design options, not an implementation-completeness claim. +The inspected memo implementation provides tumbling-pane and pane-chain +candidates and summary computation support. It does not establish a general +direct sliding-state implementation or EH support. Pane input binding, +scheduling, and retention also require deployment support. Unsupported strategies +must not be offered as executable candidates. -| Family | Idea | Fit here | -|---|---|---| -| Greedy materialization | Add the shared node with the best benefit until none helps. | No optimality bound; misses sharing that pays only in combination. | -| A* / branch and bound | Search plan combinations, prune by a cost bound. | Exact, but the current enumerator is this, and it does not finish on whole workloads. | -| Integer programming | Binary variables per node and per choice; a solver finds the optimum. | Exact, but reader-set-dependent and multiplicative costs are not linear. | -| Logic-based Benders decomposition | A MILP master chooses with linear lower-bound costs; a subproblem prices the choice exactly and returns cuts. | Fits: Stage 3 is the exact subproblem. | -| Column generation (branch and price) | Each column is a whole sharing pattern, priced exactly; a pricing problem finds new columns. | Fits reader-set-dependent costs exactly; columns grow with reader subsets. | +## MILP formulation -### How we solve it +The following is a design-level formulation. The implementation may expand a +choice into several internal-node decisions rather than enumerate all local +assignments. It must not enumerate the product of complete workload plans. -Logic-based Benders decomposition. The MILP over the memo is the master: it chooses candidates and sharing with linear coefficients that are lower bounds of the true cost. Stage 3 evaluates the decoded plan exactly. When the exact cost exceeds the master's estimate, a cut returns to the master. The loop ends when the incumbent's cost is within the gap of the master's lower bound, so the result is optimal within that gap. +### Choices and variables -The algorithm: +Let `G` be the logical groups and `C(g)` the legal physical choices for group `g`. +A choice includes its implementation, window configuration, private-node summary +parameters, and legal materialization assignment. Shared configurations are +represented separately so their costs are not duplicated. ```text -input: memo M, cost model C, gap tolerance g, time limit T -output: plan P*, its exact cost U, lower bound L - -build master MILP from M: - variables x (candidate chosen), m (group materialized), - u (shared unit used), r (reader of unit), ... - constraints: every query covered; a chosen candidate's child groups - are computed; validity (accuracy, latency, capacity) - objective: sum of lower-bound coefficients of the chosen terms -cuts := {} -U := +inf; P* := none -loop: - solve master with cuts -> solution s, objective L - if master infeasible: return no plan - P := decode(s) # exactly one plan - c := Stage 3 exact cost of P under C - if c < U: U := c; P* := P # incumbent - if (U - L) / U <= g: return (P*, U, L) # optimal within g - if time limit T reached: return (P*, U, L) # reported with its gap - for each term t of P whose exact cost exceeds its coefficient: - add cut: (t chosen) => (objective >= exact cost of t) - if no term cut applies: - add no-good cut excluding s +n[g] ∈ {0,1} logical group g is required +x[g,c] ∈ {0,1} select physical choice c for group g +y[b,t] ∈ {0,1} enable shared computation instance b at configuration t ``` -Two properties make it correct. Every coefficient is a lower bound of the exact cost of its term, so L never exceeds the optimum. Every cut is valid: it removes no plan whose exact cost is below what the cut states. Together, when the loop stops at gap g, P* is within g of the optimum. - -The number of rounds depends on the strength of the master and the cuts, so the work goes there, in the standard ways: - -- **Validity before cost.** Invalid candidates are pruned per candidate; composed accuracy and path latency are MILP constraints, not cuts. -- **Stronger cuts.** A cut prices the terms that were underestimated (a reader set's union rate, retention, size; a query's held memory), so one cut excludes every plan sharing that term, not just one plan. -- **Decomposition into clusters.** Groups that share nothing are solved as separate MILPs. -- **Decoding.** One master solution decodes to exactly one plan. +A shared configuration `t` can include summary size `s`, pane width, retention, +and materialization. Thus `y[b,s]` is the special case where only summary size +varies. Materialization can also be explicit as `x[g,c,m]`; when a candidate has +several internal nodes, `m` denotes an assignment to those nodes, not one label +for the entire query. -For a small cluster whose costs depend heavily on the reader set, column generation prices each sharing pattern exactly and needs no cuts for it. It is a complement for those clusters, not a replacement for the master. No heuristic replaces the exact search; a run that hits its time limit reports its gap. +The materialization vocabulary is: -## Memo-group search - -### Components - -| Component | What it is | +| Option | Meaning | | --- | --- | -| Memo group | One distinct logical sub-expression of the workload: the logical DAG hash-consed, so identical expressions of several queries are one group. Identical-expression sharing is group identity. | -| Implementation candidate | One way to implement one group: exact, exact accumulator, a summary family with its size parameters, tumbling panes, or a read of another group (newest panes of a shared chain, a rollup regrouping, a top-k ranking). It references child groups, not child nodes. A fused computation (e.g. a heap that absorbs a sum and a top-k) is one candidate that implements several groups; it records which. | -| Materialization | `NotMaterialized`, `MatAtQuery`, at ingestion, or kept across evaluations: chosen once per implementation candidate of a group, however many parents read it. | -| Shared group | A group read by two or more parents. Its choice (implementation, size, materialization, pane grid) couples its parents. | -| Coupling site | A choice several groups' costs depend on: a shared group's size (sized for its strictest reader), a shared pane chain's grid, longest window and size, a metric's scan label set, a source's raw-data horizon. | -| Local cost | One implementation candidate's usage, evaluated by Stage 3's per-node cost function given its context (its readers' requirements, its coupling sites' thresholds): CPU operations, scanned bytes, memory byte-seconds, peak memory, disk, each reader's latency. Cached per (group, candidate, context). | -| MILP | One mixed-integer linear program over the memo whose variables are the groups' candidate choices, their materializations and the coupling sites' thresholds, with Stage 3's prices as coefficients. It replaces enumerating whole-workload candidates. | +| `NotMaterialized` | Compute through a consumer without retaining an independently reusable result. | +| `MatAtQuery` | Build at query time and reuse within the applicable evaluation. | +| `MatAtIngestion` | Maintain state as input arrives and retain it for readers. | +| `MatAtQueryKeptAcrossEvaluations` | Build at query time and retain across evaluations, with valid freshness and maintenance behavior. | -### How the stages and the memo work together +Generate only legal assignments. A reader can use ingestion-maintained panes +while performing merge and estimation at query time. -Today the stages run one after another over the whole workload, and each hands the next every whole-workload candidate; the candidate count is the product of every query's choices: +Enumerate a finite set of supported size/configuration values and derive costs +for each. The mapping from size to cost need not be linear: a precomputed cost +is a constant coefficient multiplying a binary configuration variable. This +approach is exact only over the generated configurations; exhaustive configuration +enumeration can itself become expensive. -```text -Stage 0 frontend -> pre-ASAP logical DAG -Stage 1 Pass 1 -> per target, whole sub-DAG candidates (no cost) - Pass 2 -> one whole-workload candidate per sharing subset (no cost) -Stage 2 materialization -> every candidate x every materialization -Stage 3 selection -> every physical candidate built, checked, priced (~1 ms each) -Stage 4 executor -``` +### Coverage and dependencies -The memo keeps each stage's semantics and code, and changes what they exchange: groups and implementation candidates instead of whole plans. Each stage runs per group, and only the choices that couple groups are made jointly: +Each query root must be required: ```text - group A group B group C (shares nothing) -Stage 1 Pass 1 A's implementations B's implementations C's implementations -Stage 1 Pass 2 transformations add implementations that read another group (capability, panes, rollup, top-k) -Stage 2 materialization per implementation, once per group however many parents read it -Stage 3 local price per (group, implementation, context), cached: the MILP's coefficients -------------------------- coupling sites of the cluster {A, B} ------------------------- - discrete threshold variables: a shared group's size (strictest reader), - a pane chain's grid and longest window, a metric's scan labels, a source's raw horizon; - one MILP chooses them together with every group's candidate; C's choice is independent -------------------------------------------------------------------------------------------- -Selected plan composed once, then Stage 3's checks and the executor's compile check +n[root(q)] = 1 ``` -Example: two queries over one metric. - -```promql -Q1: quantile_over_time(0.9, data[5m]) -Q2: topk(5, sum by (job) (sum_over_time(data[5m]))) -``` - -Hash-consing their logical DAGs gives six groups; `G2` is one group read by both queries: +Let `implements(h,c)` contain all groups implemented by choice `(h,c)`, including +groups absorbed by fusion. Require each needed group to be covered once: ```text -G1 = Scan(data) -G2 = TimeRange(5m) over G1 <- read by Q1 and Q2 -G3 = quantile_over_time(0.9) over G2 (Q1's root) -G4 = sum_over_time over G2 -G5 = sum by (job) over G4 -G6 = topk(5) over G5 (Q2's root) +Σ_{(h,c): g ∈ implements(h,c)} x[h,c] = n[g] ``` -Pass 1 and the Pass 2 transformations give each group the implementations of that node alone, referencing child groups: +If choice `(h,c)` reads logical group `d`, it requires that group: ```text -G3: exact quantile (reads G2) | KLL(k) (reads G2) | DDSketch(alpha) (reads G2) - | newest-n panes of a shared chain (reads another group's pane chain) -G4: exact sum accumulator (reads G2) -G5: exact group-by sum (reads G4) -G6: exact top-k (reads G5) - | Count-Min heap (reads G2) <- fused: one implementation of G4, G5 and G6 -G1, G2: each reader computes its own (NotMaterialized) | built once per evaluation (MatAtQuery) +x[h,c] ≤ n[d] ``` -Before, the same `Scan -> TimeRange(5m)` was repeated inside every per-target candidate, and Pass 2 had to discover it again; candidates multiplied across targets. In the memo, `G2` appears once, `G3` has a handful of implementations, and the MILP has one variable per (group, implementation): `G2`'s materialization is a coupling choice that both queries' costs depend on. +Candidates must also satisfy exact-input, state-exposure, and other reuse +requirements. Dependencies must be acyclic, either by candidate construction or +by explicit ordering constraints. Sharing a result and computing separate +physical copies may be distinct choices while preserving one logical identity. -Where each part of the stage design lives in the memo: - -| Stage design | In the memo | -| --- | --- | -| Pass 1 computation candidates | implementation candidates per group, referencing child groups | -| Identical-expression rule | group identity (hash-consing) | -| Summary-capability rule | an implementation serving several reader groups; its size is a coupling site (strictest reader) | -| Window-composition rule | pane implementations in a group; a shared chain's grid and longest window are a coupling site; a reader reads its newest panes | -| Rollup and exact top-k reuse | an implementation of a coarse group (or a top-k) that reads a finer group (or a ranking) | -| Stage 2 materialization | chosen per implementation, once per group | -| Stage 3 pricing and checks | local price per (group, implementation, context), cached; the full plan is checked once | -| Raw retention, cost models 2 and 3, measured table | a per-source horizon threshold; linear peak and capacity constraints; a calibration factor fitted once and frozen | - -### End to end: the memo with the sharing strategies - -This walks one workload through the memo, showing what each sharing strategy (the identical-expression, summary-capability and window-composition rules, rollup and exact top-k reuse) contributes and where each decision is made. All queries read `data`, a gauge. - -```promql -Q1: quantile_over_time(0.9, data[5m]) every 10 s, relative error 0.01 -Q2: quantile_over_time(0.5, data[1m]) every 10 s, relative error 0.05 -Q3: topk(5, sum by (job) (sum_over_time(data[5m]))) every 60 s -Q4: sum by (job, instance) (sum_over_time(data[5m])) every 60 s -``` +### Shared producers and compatibility -**1. Hash-cons the logical DAGs (identical-expression rule).** Each distinct sub-expression is one group. `Scan(data)` and `TimeRange(5m)` appear once, so every reader of them reads the same group; the identical-expression rule needs no separate pass. +For each reader choice using shared instance `b`, let `A(g,c,b)` be the compatible +configurations: ```text -G1 = Scan(data) read by all four queries -G2 = TimeRange(5m) over G1 read by Q1, Q3, Q4 -G3 = TimeRange(1m) over G1 read by Q2 -G4 = quantile_over_time(0.9) over G2 Q1 -G5 = quantile_over_time(0.5) over G3 Q2 -G6 = sum_over_time over G2 read by Q3, Q4 -G7 = sum by (job, instance) over G6 Q4 -G8 = sum by (job) over G6 read by Q3 -G9 = topk(5) over G8 Q3 +x[g,c] ≤ Σ_{t ∈ A(g,c,b)} y[b,t] +Σ_t y[b,t] ≤ 1 for one shared instance b ``` -**2. Pass 1: each group's own implementations.** Pass 1 gives every group the implementations of its node alone: `G4` and `G5` get exact, KLL and DDSketch, each whole-window or in tumbling panes; `G6` to `G9` get exact operators; `G9` also gets a fused Count-Min heap that implements `G6`, `G8` and `G9` at once. - -**3. Pass 2: the sharing strategies add implementations that read other groups.** - -| Strategy | Implementation it adds | Coupling site it creates | -| --- | --- | --- | -| Summary capability | `G4` and `G5` may read one DDSketch build over `data`, sized for the stricter reader (`G4`, error 0.01) | the shared build's size | -| Window composition | that build as one chain of 10 s panes over 5 m; `G5` reads its newest 6 panes, `G4` all 30 | the chain's grid (10 s, the gcd of the readers' widths), its longest window (5 m) and its size | -| Rollup | `G8` (`by (job)`) implemented by regrouping `G7` (`by (job, instance)`) | none beyond `G7` being read twice | -| Exact top-k reuse | where another query keeps the same ranking, `G9` reads it | the ranking's group | -| (scan widening) | `G1`'s readers read one scan with the union of their labels | the metric's label set | - -Nothing here builds a whole-workload candidate; each strategy only adds alternatives to a group. +Compatibility includes input, family, grouping, window boundaries, summary size, +and materialization. If multiple configurations must coexist, represent separate +instances or an explicit multi-configuration structure. Each enabled producer +must have its own input dependencies and legal production implementation. -**4. Stage 2: materialization per implementation.** For example, the shared pane chain may be kept at ingestion time or rebuilt per evaluation; `G2` may be computed by each reader or built once per evaluation. A group with several parents chooses once. - -**5. Stage 3: coefficient derivation.** Each implementation's cost is derived once per consumer rate, materialization and threshold, and cached; these are the MILP's coefficients. Here all nine groups are coupled through `G1`, `G2` and the shared quantile build. A second metric's groups would share no variable with them, and the solver separates such blocks on its own. - -**6. One MILP chooses everything.** Threshold enumeration gives each coupling site its discrete values, which become MILP variables: +Where activation means a producer serves at least one reader, add: ```text -shared quantile build: size threshold in {for G4's 0.01, for G5's 0.05, no shared build} - at 0.01: G5 may read it (0.05 >= 0.01) and does if that is cheaper for G5 - at 0.05: G4 cannot read it (0.01 is stricter), so G4 uses its own build -pane chain: grid in {10 s}, longest window in {5 m, 1 m} -G2, G6: per reader or built once per evaluation -raw retention of data: horizon threshold among the readers that scan raw data +y[b,t] ≤ Σ_{(g,c) able to read (b,t)} x[g,c] ``` -The same MILP also chooses every other group's implementation: `G8` exact or rolled up from `G7`, `G9` exact or the fused heap. The threshold variables replace subsets of readers. The lazy-cut loop (below) then prices the solver's plan with Stage 3 and adds the held-memory term back until the two agree. - -**7. Build late.** The selected choices are composed into one physical plan. Stage 3 checks it once (guarantees, latency against each repeat interval, capacity) and the executor's compile check runs on it alone. The runner-ups are built only if a report asks for them. - -The result is the plan the exhaustive search would select. It is checked by `joint_search_equivalence` on workloads small enough for `--exhaustive`, one per strategy and per coupling site. +Do not rely solely on a positive objective coefficient to suppress unused +producers; zero-price resources are possible. -### Stage interfaces +### Accuracy, windows, and physical validity -- **Pass 1** emits, per group, its implementation candidates (exact, exact accumulators, each summary family and size, pane variants), each referencing child groups. It no longer emits a whole sub-DAG per target. -- **Pass 2 rules** add implementation candidates to groups: - - summary capability: one build serving several reader groups, sized for the strictest reader; - - window composition: pane implementations, a newest-n read of another group's chain, and the gcd grid; - - rollup: a coarse group implemented by regrouping a finer group; - - exact top-k reuse: a top-k implemented from another group's ranking. -- **Stage 2** chooses each candidate's materialization; a group with several parents chooses once. -- **Stage 3** prices per group with the cache. The executor's compile check and the full physical DAG are built only for the selection and the runner-ups a report asks for. +- Every query's composed guarantee must satisfy its target, including applicable + confidence requirements. Summary error and window-representation error require + a valid composition rule, not an assumed arithmetic sum. +- Every window read must cover the correct data at each modeled evaluation, + without omission or double counting. Regular-grid checks can establish this + symbolically without expanding every timestamp. +- Every selected operation must have a supported physical implementation. +- A selected materialization must provide fresh, available state when its readers + execute. Enforce the actual physical compatibility rules rather than assuming + that every ancestor forces the same materialization on all descendants. +- Raw and intermediate-state retention must cover every selected reader's needs. -### Algorithm +Reject provably invalid choices during generation. Encode remaining joint checks +in the model, or use a sound validation/refinement mechanism described below. -1. **Hash-cons** the workload's logical DAG into groups. -2. **Generate** each group's implementation candidates (Pass 1, then the Pass 2 transformations), each with its child groups and its coupling sites. -3. **Coefficient derivation.** Derive each choice's objective coefficients from Stage 3's per-node cost function (`price_node`): a candidate's private nodes per consumer rate and per materialization, and each coupling site's node at each of its thresholds. Stage 3 stays the only source of costs. -4. **Solve one MILP over the memo** (below): one choice per needed group, its materializations and the coupling sites' thresholds. -5. **Solution decoding, incumbent evaluation, lazy cuts.** Decode the solver's solution into its one physical plan, recompose its guarantees, and evaluate the plan's exact cost with Stage 3 (the incumbent cost). If the incumbent cost exceeds the MILP lower bound (the held-memory term), or a composed guarantee fails, add a cut that corrects or excludes that plan, and solve again. Stop when the best incumbent cost equals the MILP lower bound and every check passes. The gap must be 0 to report the plan as optimal. -6. **Build late.** Run the executor's compile check on the selected plan only, and report it. +### Consumer rates and resource accounting -### The solver: MILP over the memo +Introduce consumer-reach and activation variables where a node's execution rate +depends on selected ancestors, readers, and materialization. For a binary product +`v = a AND b`, use the complete linearization: -Threshold enumeration still turns each coupling site into discrete choices: a shared size set by one reader's requirement, a pane chain's grid and longest window, a scan's label set, a source's raw horizon. The MILP chooses among them instead of a hand-written search. - -**Variables.** -- `x[g,c]`: group `g` implemented by candidate `c`. Groups a chosen candidate reads must be implemented; a fused candidate implements several. -- `m[s,o]`: materialization `o` of slot `s` (a candidate's build, or a shared group's node). -- `u[site,t]`: coupling site at threshold `t`; a reader may attach only to a threshold it may read. -- `r[k,g]`: consumer `k` (a query, or a materialized slot) reaches group `g` through unmaterialized nodes. -- `y[k,g,c] = r[k,g] ∧ x[g,c]`, linearized as `y ≥ r + x − 1`. - -**Objective (cost model 1).** Σ rate(k) · per-consumer cost(g, c) · y, plus each materialized slot's own cost, each threshold's site cost, raw retention per source (`R_s ≥ bytes · query-time raw scan`), and pane retention (`keep ≥ span · read`). A slot built once per evaluation for readers on different grids takes the union rate through one indicator per distinct grid set. - -**Constraints.** One candidate per needed group; down-closure (ingestion needs ingestion below); the reach propagation; attachment only to admitted thresholds; the combination rules (top-k reuse never with panes; rollup source rules); each query's latency (the sum of the work it reaches) within its bound and its repeat interval. - -**Lazy terms.** Memory held while a query runs costs bytes × that query's latency, a product of two sums over its reach. The model omits it, which gives a lower bound, and step 5 adds it back through cuts on the incumbent. Composed accuracy guarantees depend on the chosen children; step 5 checks them on the incumbent and excludes failing combinations with no-good cuts. - -**Cost models 2 and 3.** The same variables. Peaks and capacity limits become linear constraints over the per-node peak contributions, and each objective is as specified. +```text +v ≤ a +v ≤ b +v ≥ a + b - 1 +``` -**Formulation.** +A query-time shared build runs at the union of actual reader triggers, accounting +for time alignment and reuse scope. Two readers on 60s and 90s schedules aligned +at zero have union rate `1/60 + 1/90 - 1/180`, not the rate of the 30s gcd grid. +A gcd grid may describe legal pane alignment without describing execution count. -Sets: -- `Q`: queries; `G`: memo groups; `C(g)`: candidates of group `g`. -- `ch(g,c)`: child groups that candidate `(g,c)` reads; `impl(g,c)`: groups it implements (more than `{g}` for a fused candidate). -- `O = {nm, mq, ing, kept}`: materializations (not materialized, built once per evaluation, at ingestion, kept across evaluations). -- `S`: coupling sites; `T(s)`: thresholds of site `s`. -- `K = Q ∪ {(g,c,o) : o ≠ nm}`: consumers, the queries and the materialized slots. +For one raw source, retention follows the longest required raw horizon under +the selected storage policy, rather than the sum of per-query copies. Pane +retention similarly follows the selected readers and maintained layers. If a +maximum is modeled only with lower bounds, positivity/minimization may make it +tight; otherwise add constraints enforcing its intended value. -Parameters, all derived from Stage 3's per-node cost function, in cost per second unless noted: -- `rate(k)`: evaluations per second of consumer `k`. -- `w(g,c)`: cost of one computation of `(g,c)`'s own nodes. -- `a(g,c,o)`: standing cost of slot `(g,c,o)` (ingestion upkeep, kept state). -- `σ(s,t)`: cost of site `s` at threshold `t`. -- `may(s,t,g)`: whether group `g` may read site `s` at threshold `t`. -- `ℓ(g,c)`: latency of one computation, in ms. -- `B(q)`: the latency limit of query `q`, `min(max_latency_ms, repeat interval)`. -- `hz(n)`: raw horizon bytes of raw scan `n`; `span(q,h)`: retention that reader `q` needs from pane chain `h`; `β(h)`: cost per second of keeping one span unit of `h`; `p_disk`: disk price. +### Objective and optional capacity constraints -Variables: +The billing objective is: ```text -x[g,c] ∈ {0,1} group g is implemented by candidate c -m[g,c,o] ∈ {0,1} candidate (g,c) is materialized as o -u[s,t] ∈ {0,1} coupling site s takes threshold t -r[k,g] ∈ {0,1} consumer k computes group g (reaches it through unmaterialized nodes) -y[k,g,c] ∈ {0,1} r[k,g] ∧ x[g,c] -λ[q,g] ∈ {0,1} g is computed within q's evaluation (reached through nm or mq nodes) -R[s] ≥ 0 raw horizon kept for source s -keep[h] ≥ 0 retention kept for pane chain h -θ ≥ 0 held memory (memory × latency), added by lazy cuts +min selected private-node execution costs + + enabled shared-producer execution costs + + retained raw and intermediate-state costs + + temporary-state and other plan-level costs ``` -Objective, cost model 1: +Give each charge one owner. A shared build must not be included both in every +reader's private coefficient and in `y[b,t]`. Account for update, merge, +estimation, expiration, and compaction where the selected implementation needs +them. Freeze calibration and model inputs before solving. + +If requested, constrain CPU, memory, disk, and per-query latency. Resource +coefficients must match their meaning: average memory is not peak memory, and +summing query work does not establish concurrent CPU or memory demand without +an execution/concurrency model. A cluster objective can minimize latency or +memory under these constraints. A Pareto request requires a frontier search, +not a claim that one scalar solve returns every nondominated plan. + +## Solving and validation + +Start with a complete MILP for the supported finite candidate space where the +costs and constraints can be encoded accurately and compactly. MILP does not +require logic-based Benders decomposition (LBBD). + +A separate exact evaluator remains useful for checking a decoded plan. Validation +alone cannot certify optimality if the model omits costs or legal candidates. +If difficult composed checks or costs are deliberately omitted, the model must +be a valid relaxation, and refinement is required: + +1. Solve the relaxed model and retain its certified solver lower bound. +2. Decode a complete physical plan and validate its accuracy, timing, resources, + and full cost. A validated plan supplies an incumbent upper bound. +3. Exclude a proved-infeasible combination, or add a proved-valid cost bound. +4. Repeat until the certified bound closes the target gap or the budget expires. + +A no-good cut must include every decision needed to prove infeasibility. A cost +cut must be conditioned on all choices affecting that charge; applying a +whole-plan cost to an incomplete candidate signature can exclude a cheaper plan. +Excluding a feasible, evaluated configuration is safe only if its validated plan +is retained and the proof accounts for that excluded region. Do not confuse a +solver's current feasible objective with its certified global lower bound. + +LBBD is one possible master/subproblem design for this refinement, not a +requirement. An iterative cut loop is not automatically LBBD. Select a +decomposition only when measured model size or difficult checks justify it. + +Split workload components only when all potential sharing, retention, and +feasibility interactions are absent. Global capacity constraints can couple +otherwise separate components: keep the joint model or combine complete resource +frontiers. Unknown interactions remain connected. + +### Reporting and exactness + +Report candidate/configuration counts, model size, generation and solver time, +validation effort, incumbent cost, certified lower bound, gap, and stop reason. +The gap is `(U-L)/U` for positive incumbent cost `U`; define zero-cost handling +explicitly. A time limit with no incumbent means no feasible plan found, not +proved infeasible. If refinement has exhausted the remaining region but retained +an incumbent, return that incumbent rather than reporting the workload infeasible. + +Optimality is relative to the generated finite candidates and configured models. +It is not a guarantee of measured runtime, completeness over all pane widths or +summary sizes, or support for unimplemented algorithms. Ties require an explicit +selection policy; equal-cost plans need not have identical structure. + +## Workflow and example ```text -min Σ_k Σ_(g,c) rate(k) · w(g,c) · y[k,g,c] computation, once per consumer that reaches it - + Σ_(g,c,o) a(g,c,o) · m[g,c,o] standing cost of materialized slots - + Σ_(s,t) σ(s,t) · u[s,t] coupling sites at their thresholds - + p_disk · Σ_s R[s] + Σ_h β(h) · keep[h] raw and pane retention - + θ held memory (lazy) +Stage 0: queries → logical DAGs + ↓ +Build initial memo: identities, dependencies, query roots + ↓ +Stage 1, Pass 1: generate per-group computation candidates + ↓ +Stage 1, Pass 2: add ASAP-aware CSE candidates and shared configurations + ↓ +Stage 2: generate legal physical and materialization candidates + ↓ +Stage 3: build and solve the joint MILP + ↓ +Decode and validate the selected physical DAG + ↓ +Return a validated plan and search report; refine the model if required ``` -Constraints: +Consider: ```text -(1) cover Σ_{(h,c): g ∈ impl(h,c)} x[h,c] = need[g] each needed group covered once - need[root(q)] = 1; need[g] ≥ x[p,c] for g ∈ ch(p,c) -(2) materialize Σ_o m[g,c,o] = x[g,c] - m[g,c,ing] ≤ Σ_c' m[g',c',ing] + rawsource(g') for g' ∈ ch(g,c) (ingestion needs ingestion below) -(3) reach r[q,root(q)] = 1; r[(g,c,o),g'] ≥ m[g,c,o] for g' ∈ ch(g,c) - r[k,g'] ≥ r[k,p] + x[p,c] − Σ_{o≠nm} m[p,c,o] − 1 for g' ∈ ch(p,c) -(4) product y[k,g,c] ≥ r[k,g] + x[g,c] − 1 -(5) sites Σ_t u[s,t] ≤ 1; x[g,c] ≤ Σ_{t: may(s,t,g)} u[s,t] for each candidate (g,c) that reads site s -(6) retention R[s] ≥ hz(n) · λ[q,n] for each raw scan n of s and each q - keep[h] ≥ span(q,h) · x[q's reader of h] -(7) latency λ[q,root(q)] = 1; λ[q,g'] ≥ λ[q,p] + x[p,c] − m[p,c,ing] − m[p,c,kept] − 1 - Σ_(g,c) ℓ(g,c) · (λ[q,g] + x[g,c] − 1)^+ + compaction(q) ≤ B(q) - (the product linearized as in (4)) -(8) rules combination limits, e.g. x[top-k reuse] + x[pane candidate of the same group] ≤ 1 +Q1: p95(latency[15m]), evaluated every 1m +Q2: p99(latency[15m]), evaluated every 1m ``` -Minimization keeps `r`, `y`, `λ`, `R` and `keep` at their smallest feasible values, so their lower-bound constraints make them exact. - -Lazy cuts, added by the loop for an incumbent plan `P` (a set of chosen `(g,c)` and `(g,c,o)`): - -```text -held memory θ ≥ H(P) · ( Σ_{(g,c) ∈ P} x[g,c] − |P| + 1 ) H(P) = Stage 3's held-memory cost of P -accuracy Σ_{(g,c) ∈ B} x[g,c] ≤ |B| − 1 B = a combination whose composed guarantee fails -``` - -Each held-memory cut is binding only when every choice of `P` is made, so it never cuts off another plan, and its bound is exact for `P`. - -Cost models 2 and 3 keep (1)–(8) and the cuts, and change the objective: -- **Cost model 2:** `min π_mem · M + π_cpu · N + π_disk · D`, with `M ≥ Σ peak contributions`, `N ≥` ingestion cores `+` concurrently evaluating queries, and `D ≥` retention. -- **Cost model 3:** no prices. Add `M ≤ cap_mem`, `D ≤ cap_disk` (and `N ≤ cap_cores`), then minimize the chosen objective: the worst latency `L ≥ L_q` for every `q`, or `M`. - -**Why a MILP rather than a hand-written search.** The memo is small: the whole paper workloads have 74 groups and 284 candidates (W-RangeHeavy) and 158 groups and 584 candidates (W-RollupHeavy), each built in under 0.3 s. Cost model 1 is not additive per node: -- memory held while a query runs costs bytes × that query's latency; -- how often a node runs depends on the materialization choices above it. - -A branch and bound with dynamic programming would need custom bounds for both, plus Pareto frontiers for cost models 2 and 3. In a MILP each of these is a standard exact linearization or a linear constraint, and the solver supplies the bounds, the search and an optimality certificate. - -**Why dynamic programming alone does not apply.** Dynamic programming over the memo needs optimal substructure: a group's best implementation depends only on what lies below it, and the plan's cost is the sum of its parts. This workload breaks that in three ways: - -1. **The memo is a DAG, not a tree.** - - A shared group (a scan, a time range, a shared sketch) has several parents. - - Its size is set by its strictest reader, and which parents read it depends on their own choices. - - Its cost is paid once and split among the readers. - - So its best choice depends on the choices above it, not only on its subtree. This is the core difficulty of multi-query optimization, which is NP-hard. -2. **Costs depend on the context above.** - - How often a node runs depends on its ancestors' materialization: below a node built once per evaluation or maintained at ingestion time, the node is not recomputed per reader. - - Cost therefore flows top-down, while dynamic programming works bottom-up. - - Cascades handles this by memoizing per (group, required properties). Here the context is the reader set, the thresholds and the materializations above, and their combinations bring the blow-up back. -3. **The objective is not additive.** - - Held memory costs bytes × the query's latency, and the latency is the sum of the work along the query's whole path. This is a product of sums, not a sum. - - A subtree that is cheapest on its own is not always part of the optimum: a slightly costlier but faster subtree lowers the query's latency and so its held-memory cost elsewhere. - - Each subproblem would have to keep a Pareto front over (cost, latency) rather than one minimum, and the fronts can grow large. - - Raw retention (the longest horizon among readers) and pane retention (the longest span) are maxima, which are not additive either. - -A MILP needs none of these properties. It solves all choices jointly: -- sharing and context are linear constraints (reach `r`, products `y`, thresholds `u`); -- maxima are lower-bound variables (`R`, `keep`); -- the product term is added back by lazy cuts. - -Dynamic programming remains useful only for the parts that do decompose, for example independent blocks with no shared group. - -This is the multi-query optimization problem. The memo is its AND-OR DAG: a group is an equivalence node and a candidate is an operation node. Greedy multi-query optimizers (for example Roy et al., SIGMOD 2000) give up optimality for speed. Exact formulations are integer programs. Our shared sizes depend on the strictest reader, accuracy guarantees compose, and time enters through panes and retention, so we need the exact form. - -**How each non-additive term is encoded.** - -| Term (cost model 1) | Encoding | Exact | -| --- | --- | --- | -| How often a node runs: once per consumer whose path reaches it through unmaterialized nodes | reach indicators `r[k,g]` propagated through `x` and `m`; cost on `y = r ∧ x` | linear | -| A node built once per evaluation for readers on different grids | one indicator per distinct set of reader grids; cost at that set's union rate | linear | -| A shared size set by the strictest reader (capability builds, pane chains, rollup state) | one indicator per threshold; a reader attaches only to a threshold it may read | linear | -| A scan's label set | one indicator per label set | linear | -| Raw retention: the longest horizon among query-time raw scans of a source | `R_s ≥ bytes(n) · query-time(n)` | linear | -| Pane retention: the longest span among a chain's readers | `keep ≥ span(q) · read(q, chain)` | linear | -| Ingestion compaction: depends on the ingest worker count, a ceiling | integer workers `≥ work / capacity` | linear | -| Latency within the bound and the repeat interval | the sum of the reached work, over `y` | linear | -| Memory held while a query runs: bytes × latency | omitted from the model; added back by lazy cuts | lazy | -| Composed accuracy guarantees | checked on the incumbent; failing combinations excluded | lazy | -| A measured table's calibration factor | fitted once before the search | constant | - -An exact linearization of the held-memory product needs a variable for each query and each pair of groups it reaches (about 3M on W-RollupHeavy with 8 users), so it is generated lazily instead. - -**The lazy-cut loop.** The model without the lazy terms never overestimates a plan's cost, so its optimum is a lower bound on the true optimum. The loop: -1. Solve the model and take its plan `P`, with objective `c_milp`. -2. Decode `P` into its physical plan and evaluate it with Stage 3, which includes held memory and the composed guarantees, giving the incumbent cost `c_true`. -3. If `c_true = c_milp` and every check passes, `P` is optimal: its true cost equals a lower bound on every plan's true cost. -4. Otherwise add a cut and go to step 1: - - a combination whose guarantee fails is excluded; - - a combination whose cost was underestimated gets "choosing this combination costs at least `c_true`". - -Each cut removes or corrects at least one of finitely many combinations, so the loop ends, and it ends at the optimum. For example: -- Round 1 picks `P1` at 0.40 per second, and Stage 3 prices it at 0.47. A cut makes `P1`'s combination cost at least 0.47. -- Round 2 picks `P2` at 0.45, and Stage 3 agrees. `P2` is optimal. - -(These numbers are illustrative.) The cost of the loop is the number of rounds. Tight McCormick bounds on held memory (from the memo's byte and latency ranges) and several cuts per round keep it low; the report gives the rounds per workload. - -**Reporting.** The result states the solver gap. A time limit that ends with a nonzero gap is reported with that gap, never as optimal. - -**Validation.** The brute force over the memo is a second oracle. `joint_search_equivalence` checks the MILP, the brute force and `--exhaustive` against each other on small workloads under all three cost models: a peak case, a capacity case and a raw-horizon case. - -### Coefficients - -**Terms.** Three distinct steps use Stage 3's cost function, and this document keeps their names apart: - -| Term | What it is | When | How often | -| --- | --- | --- | --- | -| **Coefficient derivation** | Computing the objective coefficient of each local choice (a group's implementation, materialization, threshold) from Stage 3's per-node cost function. The result is a lower bound on that choice's true cost. | before solving | once per choice, cached | -| **Solution decoding** | Turning a MILP solution (the values of `x`, `m`, `u`, …) into the one physical plan it describes. | after each solve | once per round | -| **Incumbent evaluation** | Running Stage 3 on that decoded plan as a whole: guarantees, latency checks and its exact cost (the **incumbent cost**). | after decoding | once per round | - -The **MILP lower bound** is the solver's objective, a lower bound on every plan's cost. The **gap** is (incumbent cost − MILP lower bound) / incumbent cost. The report keeps `incumbent_cost` at the search level and the MILP-only figures in a `milp` object: `lower_bound`, `first_lower_bound`, `gap`, `rounds`, `variable_count`, `constraint_count`, `solver_time_ms`, `time_limit_reached`. - -**One solution, one plan.** Every choice that changes a plan's cost is a model variable, so a solution decodes to exactly one plan, and incumbent evaluation runs Stage 3 once per round: -- sharing an identical expression or computing it per reader is the materialization `m` (`mq` vs `nm`); -- a shared node's size is the threshold `u`; -- a metric's scan label set is a threshold variable of the scan's coupling site (the union of the labels of the readers attached to it); -- pane grids and raw horizons are threshold variables of their sites. - -A solution must never stand for several plans whose costs are then enumerated: that enumeration (thresholds × identical-expression variants × materializations) is what exhausts memory on whole workloads. If a variant cannot be a variable yet, its alternatives are evaluated as a small set local to that one site, never as a product with other sites, and the design records it as a gap to close. - -**Principle: Stage 3 is the only source of costs, and the coefficients only need to be lower bounds.** -- Every coefficient comes from Stage 3's per-node pricing (`price_node`), evaluated in a hypothetical context. The MILP has no cost formula of its own. -- The selected plan is always evaluated by Stage 3 as a whole (incumbent evaluation), and its cost is the incumbent cost. So a coefficient may be any value **no greater than** the true cost of the choice it stands for. -- Exact coefficients make the first bound tight. Lower-bound coefficients keep the result exact at the cost of more lazy-cut rounds. - -**Decomposition.** A plan's cost is the sum over its nodes of `price_node(node, context)`. A node's context is who computes it, how often, how it is materialized, at which threshold it is sized, and how large its input is. When a cost depends only on discrete context, each value of the context gets its own coefficient, selected by a 0/1 variable: - -| Cost depends on | Variable | Coefficient | -| --- | --- | --- | -| the chosen implementation | `x[g,c]` | `price_node` of the candidate's own nodes | -| the materialization | `m[g,c,o]` | standing cost `a(g,c,o)` and per-computation cost `w(g,c)` for that `o` | -| which consumers compute it, how often | `y[k,g,c]` | `rate(k) · w(g,c)`, with `w` the cost of one computation | -| a shared node's size | `u[s,t]` | `σ(s,t)`: the node's cost at threshold `t` | - -Example: `quantile_over_time(0.9, data[5m])` as a KLL built once per evaluation at threshold 0.01 is the choice (group, KLL, `mq`, `t = 0.01`). Its `w` is Stage 3's price of one KLL sized for 0.01 over five minutes of rows, and its objective term is `rate(Q1) · w · y[Q1, G4, KLL]`. Coefficients are generated per candidate, per combination of its coupling-site thresholds, and per materialization of its private builds. - -**Terms that do not decompose, and their lower bounds:** - -| Term | Why it does not decompose | Coefficient used | Corrected by | -| --- | --- | --- | --- | -| Held memory, bytes × latency | the latency is a sum over the whole path | 0, or a McCormick bound from the memo's byte and latency ranges | a cut on the term | -| A shared scan's window (the longest reader's) | its input depends on other groups' choices | derived at the reader's own window | a cut when a longer reader is chosen | -| A child consumed by an ingestion-time build | the child runs at ingestion time instead | the cheaper of the two pricings | a cut on the pair | -| A chain kept across evaluations, charged per request | depends on the request count | the per-evaluation price | a cut | -| Compaction with several ingest workers | depends on the global worker count | 0 | a cut | -| Composed accuracy | depends on the chosen children | no coefficient | a no-good cut on a failing combination | - -Validation: for every plan of the brute force over the memo, the sum of its coefficients is at most Stage 3's price, and equal wherever the term decomposes exactly. - -**Cuts.** When Stage 3's price of the incumbent exceeds the bound, the loop adds: -- a **term cut** for each underestimated term, conditioned only on the choices that term depends on, for example "if `m[n,o]` then held memory of `n` ≥ H" or "if `x[g,c]` and `x[g',c']` then the scan window cost ≥ W". One term cut corrects every plan sharing those choices. -- a **no-good cut** on the whole combination, as the fallback that guarantees termination. - -The report gives the number of rounds and the first round's bound against the final price (the tightness of the model). - -### Coupled charges - -- **Raw data on disk**: a per-source horizon threshold, linear in the MILP. -- **Cost models 2 and 3**: peaks and limits as linear constraints. -- **A measured cost table**: the calibration factor is fitted once, before the search, over every candidate node the generation produced (before pruning), and frozen; every coefficient is derived with it. - -### Code layout - -`crates/plan-selection/src/pass/subexpr/`: `groups.rs` (hash-consing, Pass 1), `capability.rs`, `windows.rs`, `scans.rs` (the Pass 2 transformations), `pricing.rs` (per-node prices in a consumer context), `search.rs` (the memo and the brute-force reference), `milp.rs` (the model, the lazy-cut loop). The search is chosen by `--search legacy|memo-brute-force|memo-milp`; the MILP becomes the default once it matches `--exhaustive` on every equivalence test. - -## Future work - -- Stronger LP relaxations and McCormick bounds for the held-memory term; tree-decomposition dynamic programming as an alternative engine for the MILP. -- Greedy incumbents and learned branch ordering to accelerate exact search. -- Explicit approximate modes such as beam search, top-k candidates, capped resource frontiers, and coarser parameter domains. These may lose the optimum and must be labeled accordingly. -- Incremental replanning and broader candidate/rewrite coverage. +Stage 0 establishes two different quantile results with the same window width +and evaluation step. Initial memo construction keeps separate result groups and +shares eligible identical inputs. Pass 1 adds exact and supported KLL candidates, +with sizes derived from the readers' accuracy requirements. + +Pass 2 adds candidates for a shared whole-window KLL build and supported pane +compositions. One possible configuration maintains 1m KLL panes and reads the +latest 15 panes. The two estimates can share the build, and, where legal, a merge +result. Other widths and hierarchies remain candidates only when generated and +supported; 5m panes alone cannot cover every 1m-aligned evaluation. + +Stage 2 supplies legal assignments: for example, pane builds at ingestion and +merge/estimate at query time, or query-time construction from retained raw data. +Stage 3 compares their full costs. A shared build may be sized for both readers; +private builds remain candidates. The selected plan meets both accuracy targets, +window semantics, latency limits, and any configured resource capacities. + +## Implementation and verification + +The inspected memo implementation in +`crates/plan-selection/src/pass/subexpr/` separates group construction, +summary capability sharing, window composition, scan sharing, pricing, and search. +Its memo builder currently receives Pass 1 candidates. Moving group construction +after Stage 0 and adapting Pass 1 to generate directly into groups is the target +architecture, not an assertion about the current call order. + +Likewise, the inspected MILP path supports usage-over-time pricing and routes +unsupported cost models to legacy search. Cluster-capacity and provisioned-cost +formulations in this document are design requirements, not claims of current +MILP coverage. EH, general direct sliding-state maintenance, and arbitrary +multi-layer pane generation require their own implementation work. + +Verify the design against exhaustive enumeration on small workloads and a +separate brute-force search over the memo. Compare minimum validated cost under +the same candidate inventory and models, not merely whether both return a plan. +Include: + +- Sharing that is profitable only when selected jointly. +- Reader-dependent summary sizes, evaluation rates, and retention. +- Multiple pane widths, alignment failures, and overlapping windows. +- Fused candidates, rollup dependencies, and cycles introduced by reuse. +- All supported materializations and internal-node assignments. +- Composed accuracy, shared scans, raw horizons, and global capacity limits. +- Zero-price resources, numeric tolerances, budget stops, and valid solver bounds. + +For any relaxed formulation, check on enumerated plans that its assigned cost +never exceeds full cost and that every generated cut is valid. Measure planning +time, peak memory, model size, solver gap, and validation rounds on representative +workloads before choosing stronger decomposition or configuration restrictions. From 570beec3e439783626dac932686593df826c46bf Mon Sep 17 00:00:00 2001 From: zzylol <50204836+zzylol@users.noreply.github.com> Date: Fri, 23 Oct 2026 22:58:50 +0000 Subject: [PATCH 23/53] docs: align memo MILP proposal with planner stages and current architecture --- docs/design_docs/proposals/README.md | 2 +- .../proposals/joint-planning-search.md | 888 ++++++++++-------- 2 files changed, 495 insertions(+), 395 deletions(-) diff --git a/docs/design_docs/proposals/README.md b/docs/design_docs/proposals/README.md index 3075e1945..913cd6ff6 100644 --- a/docs/design_docs/proposals/README.md +++ b/docs/design_docs/proposals/README.md @@ -11,5 +11,5 @@ extensions. A design document is not a promise of downstream runtime support. - [Operator sharing](operator-sharing.md) - [Decoupling operators from scalar expressions](decoupling_op_and_expr.md) - [ASAPPlanner layering](planner-layering.md) -- [Memo group and MILP design](joint-planning-search.md) +- [Memo-based joint optimization with MILP](joint-planning-search.md) - [Schema and physical data for ASAP primitives](asap-primitive-schema.md) diff --git a/docs/design_docs/proposals/joint-planning-search.md b/docs/design_docs/proposals/joint-planning-search.md index ff11a138f..f75d830cf 100644 --- a/docs/design_docs/proposals/joint-planning-search.md +++ b/docs/design_docs/proposals/joint-planning-search.md @@ -1,462 +1,562 @@ -# Memo group and MILP design - -Status: design proposal. Audience: planner designers and implementers. - -ASAP primitives introduce sharing opportunities across queries, but also enlarge -ASAPPlanner's optimization search space. The optimizer must represent these -opportunities compactly and select a low-cost, valid physical plan for the whole -workload. This design uses memo groups for candidate generation and a mixed-integer -linear program (MILP) for joint selection. It preserves the responsibilities of -the stages in [planner-layering.md](planner-layering.md). - -## Challenges and goals - -Whole-workload enumeration multiplies per-computation candidates, sharing -configurations, and materialization assignments. The earlier joint-search -measurements on a growing prefix of one random mixed workload illustrate this -problem: - -| Queries | Planning time | Peak memory | -| ---: | ---: | ---: | -| 3 | 0.4 s | 0.14 GB | -| 4 | 5 s | 1.5 GB | -| 5 | 129 s | 24.7 GB | -| 6 | stopped at about 228 s | about 100 GB | - -These are historical measurements, not a benchmark of the proposed design. - -The goals are to reduce duplicate candidate generation, avoid enumerating the -Cartesian product of whole-workload plans, and preserve the minimum cost over the -configured finite candidate space. A time-limited search must return its best -validated plan and report whether optimality is proved. Incremental replanning -and new window algorithms are separate work. - -## Memo groups and candidate generation - -A memo group identifies one logical computation and its output contract. Exact -candidates produce the same logical result. Approximate candidates implement that -result subject to explicit accuracy obligations; they are not exactly equivalent -expressions. A group is not a sketch family or a collection of all computations -that could share a sketch. - -Build the initial memo after Stage 0, before Stage 1 Pass 1: - -- Create a group for each distinct logical sub-expression. -- Merge identical expressions only where semantic identity and sharing legality - are established; verify equality after hashing. Preserve schema, grouping, - source-language semantics, and time semantics. Keyless nodes can require - separate identities per reader. -- Record group dependencies, query roots, and reader requirements. - -Pass 1 generates candidates directly against these group identities. Cache -candidate generation by group and all relevant context, including accuracy -requirements. Different readers may need different summary sizes; merging their -logical identity must not discard candidates needed by a reader. This avoids -repeating generation for identical computations without first composing every -whole-query plan. - -### Memo grouping versus CSE - -Memo grouping represents the search space. Common subexpression elimination -(CSE) identifies computations that can reuse work. A repeated identical -expression can have one memo group with multiple parents, but that identity alone -does not require one materialized execution at runtime. - -ASAP-aware CSE also identifies sharing between different logical computations: -summary capability sharing, pane composition, rollup, top-k reuse, and scan -widening. These rules add candidates and dependencies between groups; they do not -merge groups whose requested results differ. - -For example, p95 and p99 over the same input window are separate result groups. -Their KLL candidates may share one build, with separate estimation nodes. A -coupling site identifies the shared configuration decisions, such as build size -or a pane chain's grid and retention span. The solver must select compatible -configurations for all attached readers. A shared build can instead be represented -as an explicit intermediate group; either representation must preserve the same -choices and charge ownership. - -### Responsibilities of the stages - -| Stage | Decisions represented in the memo | +# ASAPPlanner: Memo-Based Joint Optimization with MILP + +**Status:** Design proposal + +**Scope:** Multi-query optimization across exact operators, ASAP primitives, window implementations, sharing, and materialization + +**Audience:** Planner designers and implementers + +## 1. Motivation and Problem Statement + +ASAP primitives introduce optimization opportunities: a summary can replace expensive exact computation, one summary can answer several queries, and compatible queries can reuse panes, rollups, scans, or retained raw data. These candidates can reduce CPU, memory, storage, and evaluation latency. + +They also enlarge the search space. Enumerating and pricing every complete workload plan produces combinatorial growth. Planning each query independently can miss sharing and account for a common physical producer multiple times. + +**ASAPPlanner should represent candidates compactly and select a minimum-cost valid physical plan for the workload.** Optimality is relative to the generated candidate set, configured models, and solver tolerances. + +### Goals + +1. Avoid repeated exploration of identical or proven-equivalent logical sub-expressions. +2. Represent ASAP-aware reuse explicitly, including reuse between nonidentical logical expressions. +3. Optimize workloads under resource prices or a specified objective with hard resource constraints. + +The coupled decisions include: + +- Exact computation, exact accumulator state, or approximate summaries such as KLL. +- Legal window implementations and their configurations. +- Shared builds, pane chains, scans, rollups, and per-source raw retention. +- Materialization timing, storage medium, and lifetime. +- End-to-end accuracy, latency, and deployment constraints. +- Shared-producer costs counted once per actual execution or retained state, with reader-specific work counted separately. + +### Non-goals + +- Inventing arbitrary sketches or window algorithms. +- Assuming a closed-form linear cost/size/accuracy relationship. +- Substituting MILP estimates for physical-plan validation. +- Proving optimality over ungenerated configurations or all imaginable plans. +- Implementing runtime scheduling or retention services inside the optimizer. + +## 2. Terminology + +### 2.1 Logical expressions, memo groups, and candidates + +Terminology follows [planner-layering.md](planner-layering.md): + +| Term | Meaning | | --- | --- | -| Stage 0 | Parse and lower queries into logical DAGs; establish query semantics. | -| Initial memo construction | Establish logical group identities, dependencies, and query roots. | -| Stage 1, Pass 1 | Add exact, exact-accumulator, summary-family, summary-size, and fused-computation candidates. | -| Stage 1, Pass 2 | Add ASAP-aware CSE candidates, shared configurations, and cross-group dependencies. | -| Stage 2 | Generate legal physical implementation and materialization candidates, including assignments to internal nodes. | -| Stage 3 | Jointly select candidates under the configured objective and constraints; validate the decoded plan. | - -Candidate generation can be lazy. Only provably invalid or contextually dominated -candidates may be removed without changing the finite-space optimum. A higher -standalone cost is not enough to discard a candidate that enables sharing. - -## Cost-based joint optimization - -Assign resource usage to the nodes of the selected physical DAG using the -configured cost model. The objective can minimize billed cost, including CPU, -memory, storage, and other priced resources. In a cluster without billing prices, -select an explicit objective, such as latency or memory, subject to capacity -limits. Capacity limits alone do not define an optimum. - -A shared computation is charged once per actual execution, not once per reader. -An ingestion-maintained build has update and retained-state costs; query-time -work has costs at its actual trigger rate. Multiple consumers can still cause -multiple executions of an unmaterialized node. Estimation and merge work private -to each reader is charged separately. - -### Why joint selection is needed - -Consider two result groups, a quantile at 0.5 and a quantile at 0.9, over the same -input window. Assume their private KLL implementations cost 4 and 6 units per -second. A shared build satisfying both readers costs 7, with each estimator -costing 0.1. All configurations in this example meet accuracy requirements. - -Optimizing the groups independently selects the private implementations for a -combined cost of 10. Joint selection can choose the shared build for 7.2. The -incremental cost of attaching a reader depends on whether another selected -candidate already enables the build. It can also depend on whether that reader -requires a larger build, a longer retention horizon, or more frequent evaluation. - -A dynamic program retaining only the cheapest candidate per group loses this -context. DP is possible if its state includes the relevant reader sets, shared -configurations, materializations, and resource tradeoffs, but those states can -grow combinatorially. MILP provides a general engine for these joint discrete -choices. Local DP and caching remain useful where dependencies are limited. - -## Window model - -Distinguish three dimensions: - -1. **Window semantics:** width, evaluation step, time origin/alignment, and endpoint - conventions. A 15m trailing window evaluated every 1m is a sliding-window query. - With width and step both 15m and aligned boundaries, windows are tumbling. -2. **Window implementation:** how the required data is represented and combined, - such as raw recomputation, direct sliding-state maintenance, fixed-width panes, - or exponential histogram (EH) buckets. -3. **Materialization:** when each internal state is computed and how long it is - retained. Sliding-window semantics do not prevent ingestion-time maintenance. +| **Logical expression** | A node and its input sub-expressions in the language-independent `LogicalDAG` (Pre-ASAP IR). It specifies selectors, transformations, predicates, aggregations, grouping, window semantics, and output schema, while preserving source-language behavior such as series identity, evaluation timing, and missing-data semantics. It contains no summary or materialization choice. | +| **Memo group** | One distinct logical sub-expression and its output contract. Proven-equivalent logical expressions can have the same group identity. The current implementation hash-conses eligible identical expressions; it does not implement a general equivalence saturation engine. | +| **Implementation candidate** | One way to implement a group over separately chosen child groups. It owns private nodes, records dependencies and rule origin, and can carry coupling sites and accuracy obligations. A fused candidate can implement several groups. This corresponds to `ImplementationCandidate` in the memo code. | +| **Whole-workload candidate** | A DAG for the whole workload, as defined by the layering document: a `LogicalDAG`, logical `LogicalASAPDAG` (Post-ASAP IR), or timed `PhysicalASAPDAG`, depending on the stage. Its candidate sets are `CandidateLogicalDAGs`, `CandidateLogicalASAPDAGs`, and `CandidatePhysicalASAPDAGs`. | +| **Shared producer** | A physical computation or maintained intermediate state that compatible selected readers can reuse, including its upstream dependencies. Examples are a summary build, pane chain, scan, or finer result used by rollup. It need not itself be a query result. | +| **Coupling site** | A shared decision affecting several implementation candidates. Equal site keys identify one decision; thresholds enumerate legal configurations. Existing kinds include `SharedBuildSize`, `PaneChain`, `ScanLabels`, and `RawHorizon`. | +| **Physical plan** | One selected whole-workload `PhysicalASAPDAG`, produced by physical ASAP-aware optimization and selected at Stage 3. It has physical operator structure, execution timing, materialization assignments, and query roots. Shared physical nodes have multiple consumers. Execution remains downstream of ASAPPlanner. | + +Below, “candidate” means a local implementation candidate when discussing groups, and a whole-workload candidate when discussing stage outputs. The memo compactly represents combinations of local candidates instead of constructing every whole-workload DAG. + +A group's logical result remains fixed. Its exact candidates preserve that result exactly; approximate candidates implement it under explicit accuracy contracts. An approximate estimate is not an exactly equivalent logical expression. Different reader accuracy requirements can require different candidates without changing the requested statistic. + +### 2.2 Three distinct kinds of reuse + +| Mechanism | Condition | Representation | Example | +| --- | --- | --- | --- | +| Memo equivalence | Expressions have the same logical semantics. | One logical identity with implementation candidates. | Two occurrences of the same filtered sum over the same window. | +| Conventional CSE | Identical computation has compatible semantics and execution context. | One physical producer with multiple consumers, when the selected physical realization reuses it. | A common scan or identical exact aggregate. | +| ASAP-aware sharing | Different requested results can consume compatible intermediate state. | Separate result groups linked to a common producer/configuration. | P95 and P99 read one KLL; different window lengths reuse panes. | + +Memo grouping deduplicates logical identities and associated search work. It does not automatically deduplicate arbitrary search states with different readers, materializations, or accuracy obligations, and it does not force runtime materialization. CSE and ASAP-aware sharing expose physical reuse candidates without equating different outputs. + +## 3. ASAPPlanner Workflow + +This is the target workflow. The current memo builder receives Pass 1 candidates; moving initial group construction before Pass 1 is a proposed change, not a description of today's call order. + +```text +Stage 0: frontends -> language-independent LogicalDAGs + | +Build initial memo + - Intern eligible identical / proven-equivalent sub-expressions + - Record roots, group dependencies, semantics, and reader requirements + | +Stage 1 / Pass 1: generate per-computation candidates + - Exact, exact accumulator state, summary families/sizes, fusion + - Preserve logical output and approximation contracts + | +Stage 1 / Pass 2: add ASAP-aware CSE candidates + - Identical expressions, summary capability, window composition + - Rollup, exact top-k reuse, scan widening, raw-horizon interactions + - Record compatible producers/configurations and dependencies + | +Stage 2: generate legal physical candidates + - Physical operators, materialization timing, medium, and lifetime + - Check window configurations against physical capabilities + | +Analyze coupling across all generated candidates + - Separate components only when objectives and feasibility factor + | +Stage 3: joint MILP selection + - Choose candidates, producer configurations, and reader attachments + - Enforce accuracy, latency, dependencies, and resource constraints + | +Reconstruct and validate the selected PhysicalASAPDAG + - Refine/re-solve when required, or return a validated fallback/status +``` + +### 3.1 Stage 0: Normalize and build the initial memo + +Frontends preserve source-language semantics. Canonicalize only where equivalence is established: names, predicates, operand order, or windows cannot be rewritten merely because they look similar. Floating-point behavior, nulls, missing samples, and evaluation context can prevent apparently commutative rewrites. + +Intern distinct logical sub-expressions, verifying equality after hashing. Keys cover relevant source/input identity, arguments, filters, grouping, schema, series identity, window width/offset/alignment, and time semantics. Include late-data or ordering policies where they affect results. Current identity rules leave keyless nodes, such as range windows, separate per reader. + +Store accuracy targets and evaluation requirements as reader/root obligations. Include a requirement in identity only if it changes the logical semantics; otherwise retain it in candidate-generation/cache context. Do not merge P95 with P99 merely because both can use KLL. + +### 3.2 Stage 1, Pass 1: Generate local candidates + +Generate per-computation candidates against group identities rather than composing complete workload plans. For eligible aggregates, include the exact implementation: exact accumulator state when supported and constructible, otherwise the original exact computation. Add supported summary families and sizes and fused-computation candidates. + +Record each candidate's private nodes, logical output, child groups, guarantees, and accuracy obligations. Cache generation by group and every relevant context value, including reader requirements. This avoids repeated generation without dropping sizes needed by stricter readers. It also avoids the Cartesian product across unrelated groups. + +### 3.3 Stage 1, Pass 2: Discover sharing opportunities + +Use the ASAP-aware CSE rules from the layering design. Identical expressions are already group identity in the memo. Summary capability, window composition, rollup, exact top-k reuse, and scan widening add implementation candidates or dependencies between groups. + +Match producer signatures using summary input data, family, reduction, grouping, item/weight, filters, and applicable window semantics. Requested quantile levels differ at estimation, but their build may match. Window compatibility is checked separately from summary capability. + +For each opportunity, record: + +- Producer identity, input/update semantics, and upstream dependencies. +- Finite configurations: summary size, pane width, retention, or scan labels. +- Compatible reader candidates and any required exposed state or exact input. +- Legal materializations to be resolved by Stage 2. +- Usage inputs and contracts needed for Stage 3 pricing, accuracy, and latency. + +Pass 2 generates logical window-composition candidates; Stage 2 does not invent pane algorithms. Raw retention is a shared accounting dependency, not necessarily an executable operator. Sharing remains optional: independent physical instances remain selectable when supported and useful. + +### 3.4 Stage 2: Physical candidates and materialization + +Stage 2 makes physical operator and materialization decisions; it does not select the cheapest valid candidate. For example, exact top-k may lower to sort and limit. Materialization is assigned to individual eligible nodes/sub-DAGs, rather than one label for an entire query. + +| Mode | Meaning | Cost drivers | +| --- | --- | --- | +| `NotMaterialized` | Run at query time without retaining an independently reusable output. | CPU and transient memory at the actual execution rate. | +| `MatAtQuery(Memory)` | Build once for applicable readers within an evaluation. | Per-evaluation build and memory held during the evaluation. | +| `MatAtIngestion(Memory)` | Maintain as input arrives and retain state for readers. | Updates, retained state, and reads. | +| `MatAtQueryKeptAcrossEvaluations(Memory)` | Build at query time and keep state across evaluations. | Build, subsequent updates/refresh, retention, and reads. | + +Use the actual `Materialization` enum and `Medium`. Ingestion eligibility depends on `DataWorkload`, recurrence, predictability, input support, and window/phase legality. Current Stage 2 requires ingestion-time builds below ingestion-time builds; shared non-summary nodes below them use the permitted unmaterialized form. Distinguish execution at ingestion time from forcing every upstream output to be materialized. + +The current retained-query-time mode requires deployment capability and an eligible tumbling-pane chain building one new pane per evaluation. It is not a generic cache mode for arbitrary nodes. Current physical generation also offers query-time sharing for eligible non-summary sub-DAGs. Disk placement, arbitrary cache policies, parallelism, partitioning, and resource management must not be claimed implemented merely because they appear in the broader design. + +Following the layering document, earlier stages prune provably invalid candidates with reasons; Stage 3 selects among valid candidates. Cost-based dominance is a Stage 3 search optimization and needs a proof preserving every external sharing and resource obligation. + +## 4. Window Semantics, Algorithms, and Materialization + +Window semantics, window implementation, and materialization are separate dimensions, subject to compatibility constraints. + +- **Semantics:** width, offset, evaluation timestamps, endpoint convention, and time alignment. +- **Implementation:** raw recomputation, a supported direct sliding-state algorithm, fixed-width panes, or a supported EH algorithm. +- **Materialization:** when internal state is built, maintained, retained, and discarded. + +A 15m trailing window evaluated every 1m is sliding-window semantics. It may be implemented using 1m tumbling panes maintained at ingestion, with query-time merge and estimation. Width and step both 15m give tumbling semantics when aligned. + +Workload vocabulary matters: `QueryRecurrence` specifies request cadence. `RangeEvaluation { step_ms, span_ms }` specifies evaluation steps inside one request. A range request can contain many evaluation timestamps. Neither request rate nor a legal pane grid alone gives the number of computations; pricing must account for both and any reuse. ### Panes and EH buckets -**Panes** partition a timeline into aligned, non-overlapping intervals of a fixed -width. Each pane contains data or aggregate/summary state for its interval. A -window combines the required panes, with explicit boundary handling when its -endpoints do not align. An implementation may maintain several pane widths or a -hierarchy where larger panes are formed from smaller ones. - -**EH buckets** follow an exponential histogram algorithm: eligible buckets are -merged into geometrically increasing size levels and expired buckets are -removed. Standard EH sizes commonly measure represented item counts, rather than -fixed time durations. The algorithm must define the statistic it supports and -bound error from partially overlapping the oldest bucket. EH cannot be assumed -to work with every summary family, including an arbitrary KLL composition. - -Pane width and EH bucket size are separate from the summary-size parameters -controlling the state within each pane or bucket. - -### Window configurations - -For a 15m window, candidate pane compositions might include 15 × 1m, 5 × 3m, -3 × 5m, or a hierarchy combining fine boundary panes and coarser interior panes. -These are candidates only when they match the query's evaluation boundaries or -include a valid boundary treatment. For a 15m window evaluated every 1m, complete -5m panes alone cannot generally represent every evaluation exactly. - -The solver may enable multiple pane widths simultaneously. It chooses which -states to maintain, whether they are built from raw input or finer panes, and -which composition each query reads. Do not impose a global constraint selecting -only one pane width. Shared configuration sites must distinguish incompatible -chains rather than collapse every pane computation into one site. - -| Implementation | Ingestion-time work, if maintained there | Evaluation work | +A **pane** is an aligned, non-overlapping fixed-width interval and its data or aggregate/summary state. A configuration defines width, origin, retention, and reconstruction rules. Partial boundary panes require explicit treatment: finer panes, raw boundary data, or an authorized approximation. Divisibility alone is insufficient when boundaries are misaligned. + +An **EH bucket** summarizes a contiguous portion of a stream according to a specific exponential histogram algorithm. Size levels typically grow geometrically; standard EH sizes commonly count represented items rather than fixed time durations. Merging, expiration, partial-boundary treatment, and error guarantees are algorithm- and operator-specific. EH is not a generic replacement for panes or an arbitrary KLL. + +Pane width, EH bucket size, and summary capacity are different parameters. Multiple pane widths may coexist, including a hierarchy of fine panes merged into coarser panes. The candidate generator defines which hierarchies exist in the search space; the solver cannot invent an absent composition. + +For example, 15 × 1m, 5 × 3m, and 3 × 5m can represent different candidates for an aligned 15m interval. Complete 5m panes alone cannot generally reconstruct a window whose endpoint moves every 1m. A multi-width composition needs explicit coverage and no-double-counting checks. + +| Window implementation | Ingestion work, when maintained there | Evaluation work | | --- | --- | --- | -| Direct sliding state | Insert new data and expire old contributions, where the statistic supports it. | Read/finalize current state. | -| Pane-based | Update the active pane and maintain retention. | Read, merge, and estimate from the required panes, unless those results are also maintained. | -| EH-based | Update, merge, and expire buckets according to the supported algorithm. | Apply its window estimator. | +| Direct sliding state | Add records and expire contributions using a supported algorithm. | Read/finalize state. | +| Pane-based | Update active panes and maintain retention. | Select and merge panes, then finalize/estimate, unless those results are also maintained. | +| EH-based | Update, merge, and expire buckets. | Apply the supported window estimator. | -This table describes design options, not an implementation-completeness claim. -The inspected memo implementation provides tumbling-pane and pane-chain -candidates and summary computation support. It does not establish a general -direct sliding-state implementation or EH support. Pane input binding, -scheduling, and retention also require deployment support. Unsupported strategies -must not be offered as executable candidates. +This table describes possible implementations, not a support matrix. Current code supports tumbling-pane/pane-chain generation and summary computation. General direct sliding-state algorithms and EH are not implemented by that support. Arbitrary multi-layer pane generation also remains an extension. The downstream deployment supplies runtime scheduling, input binding, and retention behavior. -## MILP formulation +Legality checks include mergeability, alignment/phase, coverage, ingestion support, historical evaluations, retention, and composed error. KLL pane merging does not provide reversible deletion of arbitrary old samples. -The following is a design-level formulation. The implementation may expand a -choice into several internal-node decisions rather than enumerate all local -assignments. It must not enumerate the product of complete workload plans. +## 5. Cost and Accuracy Model + +### 5.1 Cost attribution + +Stage 3 measures `ResourceUsage`, then applies `PlanSelectionSettings.cost_model`. The objective must agree with those configured models, rather than introducing an independent MILP pricing model. + +| Existing cost model | Objective | +| --- | --- | +| `UsageOverTime(ResourceUnitCost)` | Price resource usage per second, including occupancy over time. | +| `ProvisionedCapacity(ResourceUnitCost)` | Price provisioned peak memory, peak CPU cores, and disk. | +| `ClusterCapacity(ClusterLimits)` | Enforce memory/disk/optional-core limits and use `LowestLatency`, `LeastMemory`, or `ParetoFront`. | -### Choices and variables +A Pareto request requires a frontier computation; one scalar MILP solution does not return the whole frontier. Current memo MILP supports usage-over-time pricing; unsupported cost models route to legacy search. The other formulations below are extensions, not existing MILP capabilities. -Let `G` be the logical groups and `C(g)` the legal physical choices for group `g`. -A choice includes its implementation, window configuration, private-node summary -parameters, and legal materialization assignment. Shared configurations are -represented separately so their costs are not duplicated. +Assign each charge to one physical owner: + +1. Producer build/update, retained state, and maintenance. +2. Reader-specific merge, read, estimation, and finalization. +3. Nonshared physical instances. +4. Source-level retention and other plan-level charges. + +Use consistent per-second rates, as in current code, or integrate all terms over one explicit billing horizon. Conceptually: ```text -n[g] ∈ {0,1} logical group g is required -x[g,c] ∈ {0,1} select physical choice c for group g -y[b,t] ∈ {0,1} enable shared computation instance b at configuration t +C = p_cpu × CPU + p_mem × MemoryTime + p_disk × DiskTime + + p_io × IO + other explicitly priced usage ``` -A shared configuration `t` can include summary size `s`, pane width, retention, -and materialization. Thus `y[b,s]` is the special case where only summary size -varies. Materialization can also be explicit as `x[g,c,m]`; when a candidate has -several internal nodes, `m` denotes an assignment to those nodes, not one label -for the entire query. +Occupancy over time is different from peak capacity. Without prices, use a specified objective under quotas; do not add CPU-seconds and bytes without defined weights. + +“Charge once” means once per actual execution or retained state. Query-time recomputation can occur repeatedly. A shared build's size, retention, and execution rate depend on its selected readers and ancestors' materializations. Constant coefficients are sufficient only when that context is fixed or explicitly represented by configuration/reach variables. + +Model the union of actual reader triggers, including phases and reuse scope. Readers on aligned 60s and 90s schedules have union rate `1/60 + 1/90 - 1/180`; a 30s gcd grid is not their execution rate. Current pricing remains authoritative if its schedule approximation differs; changing that policy requires a separately validated model change. + +Per-source raw retention uses the selected storage policy and longest required horizon, not duplicated per-query copies. Longer retained state and lifetime-dependent temporary memory can couple otherwise separate computations. Freeze measured calibration inputs before search. + +### 5.2 Accuracy + +Accuracy is an end-to-end query-root requirement evaluated by the configured accuracy model. Preserve error type, probability/failure parameters, evidence, and composition rules. A locally admissible sketch does not establish a valid ratio, arithmetic expression, or window approximation. + +Filter known-incompatible configurations, then encode proven composition bounds or finite feasible-combination tables where practical. Otherwise use sound validation and refinement. Summary error and window error cannot be assumed to add without a valid bound. Exact-only obligations must remain enforceable. -The materialization vocabulary is: +### 5.3 Latency -| Option | Meaning | +Use the configured latency model. Current Stage 3 models per-evaluation latency as the sum of reached query-time node work, counted once per node for that query, plus the longest applicable ingestion-state compaction. It assumes elastic CPU without queueing; fixed recurrence also imposes a repeat-interval check. Range-request validation follows the evaluator's declared per-step/request semantics. + +This is not a sum of all active workload costs and is not a general parallel-runtime critical-path model. If a future deployment uses parallel branches, queueing, or visibility delays, encode its precedence/scheduling bounds and validate them with the corresponding evaluator. Do not silently change latency semantics in the MILP. + +## 6. MILP Formulation + +This base formulation explains ownership, coverage, and reader attachments. Current code additionally models consumer reach, private materialization slots, per-reader copies, and context-dependent charges. These extensions are required whenever the base model's constant costs do not describe the physical plan. + +### 6.1 Sets and parameters + +| Symbol | Definition | | --- | --- | -| `NotMaterialized` | Compute through a consumer without retaining an independently reusable result. | -| `MatAtQuery` | Build at query time and reuse within the applicable evaluation. | -| `MatAtIngestion` | Maintain state as input arrives and retain it for readers. | -| `MatAtQueryKeptAcrossEvaluations` | Build at query time and retain across evaluations, with valid freshness and maintenance behavior. | +| `Q` | Workload queries. | +| `G` | Logical memo groups. | +| `C_g` | Legal physical candidates owned by group `g`, with local materialization assignments. | +| `B` | Potential shared producer instances or explicit shared accounting states. | +| `S_b` | Finite configurations of `b`, such as summary size, pane width, retention, and materialization. | +| `I(g,c)` | Groups implemented by candidate `(g,c)`, including fusion coverage. | +| `R(g,c)` | Required input groups, conditional on choosing `(g,c)`. | +| `J(g,c)` | Producer roles required by `(g,c)`. | +| `A(g,c,j)` | Compatible producer/configuration pairs for role `j`. | +| `d[g,c]` | Private incremental cost, excluding shared producers. | +| `f[b,s]` | Producer configuration's owned build/update/retention cost. | +| `r[g,c,j,b,s]` | Role-specific reader cost not already included elsewhere. | + +Costs use one common horizon. Finite `S_b` permits lookup-table costs without a linear size/cost function. Configuration enumeration is itself a search-space cost. If `f` depends on reader sets, trigger rates, or ancestors' choices, represent those conditions explicitly or compute the charge with reach/auxiliary variables; it is not an unconditional constant. + +### 6.2 Decision variables -Generate only legal assignments. A reader can use ingestion-maintained panes -while performing merge and estimation at query time. +```text +a[g] ∈ {0,1} group g is required +x[g,c] ∈ {0,1} candidate c is selected for group g +y[b,s] ∈ {0,1} producer b is enabled at configuration s +z[g,c,j,b,s] ∈ {0,1} candidate (g,c) uses (b,s) for role j +``` -Enumerate a finite set of supported size/configuration values and derive costs -for each. The mapping from size to cost need not be linear: a precomputed cost -is a constant coefficient multiplying a binary configuration variable. This -approach is exact only over the generated configurations; exhaustive configuration -enumeration can itself become expensive. +Materialization is part of `c` or `s`, or represented by explicit legal assignment variables such as `x[g,c,m]`. Do not maintain two representations that can disagree. A multi-node implementation can have several materialization slots rather than one mode for every private node. -### Coverage and dependencies +### 6.3 Query coverage and group selection -Each query root must be required: +Let `Roots(q)` be the group references needed for query `q`, including multiple references in a scalar query root: ```text -n[root(q)] = 1 +a[g] = 1 for g ∈ Roots(q), q ∈ Q ``` -Let `implements(h,c)` contain all groups implemented by choice `(h,c)`, including -groups absorbed by fusion. Require each needed group to be covered once: +Without fusion, select one implementation for an active group and none for an inactive group: ```text -Σ_{(h,c): g ∈ implements(h,c)} x[h,c] = n[g] +Σ_{c ∈ C_g} x[g,c] = a[g] ``` -If choice `(h,c)` reads logical group `d`, it requires that group: +Current candidates can fuse several logical computations. The general form is: ```text -x[h,c] ≤ n[d] +Σ_{(h,c): g ∈ I(h,c)} x[h,c] = a[g] ``` -Candidates must also satisfy exact-input, state-exposure, and other reuse -requirements. Dependencies must be acyclic, either by candidate construction or -by explicit ordering constraints. Sharing a result and computing separate -physical copies may be distinct choices while preserving one logical identity. +A fused candidate covers absorbed groups; do not additionally select standalone implementations for them. Every selected candidate requires its inputs: -### Shared producers and compatibility +```text +a[h] ≥ x[g,c] for h ∈ R(g,c) +``` -For each reader choice using shared instance `b`, let `A(g,c,b)` be the compatible -configurations: +An active group must be justified by a query root, selected consuming candidate, upstream producer requirement, or fusion coverage. Add reverse support/flow constraints so unused groups cannot activate merely because prices are zero. Support must include all valid dependency kinds, not only original logical parents. Enforce acyclicity or construct a candidate dependency space with a proved ordering. + +### 6.4 Sharing and configuration compatibility + +For each required producer role: + +```text +Σ_{(b,s) ∈ A(g,c,j)} z[g,c,j,b,s] = x[g,c] for j ∈ J(g,c) +z[g,c,j,b,s] ≤ y[b,s] +``` + +Use a separate equality per role: a candidate might need both a scan and a pane chain. Candidates running locally have no shared role, or use explicit local-fallback variables. An empty compatibility set makes that required-role candidate invalid. + +If producer instance `b` can have only one physical configuration: ```text -x[g,c] ≤ Σ_{t ∈ A(g,c,b)} y[b,t] -Σ_t y[b,t] ≤ 1 for one shared instance b +Σ_{s ∈ S_b} y[b,s] ≤ 1 ``` -Compatibility includes input, family, grouping, window boundaries, summary size, -and materialization. If multiple configurations must coexist, represent separate -instances or an explicit multi-configuration structure. Each enabled producer -must have its own input dependencies and legal production implementation. +Represent coexisting states as separate instances or explicitly permit multiple configurations. In particular, this is not a global one-pane-width constraint. -Where activation means a producer serves at least one reader, add: +Prevent unused producers: ```text -y[b,t] ≤ Σ_{(g,c) able to read (b,t)} x[g,c] +y[b,s] ≤ Σ selected consumer attachments requiring (b,s) ``` -Do not rely solely on a positive objective coefficient to suppress unused -producers; zero-price resources are possible. +The sum includes downstream producer attachments where producers consume other producers. Their upstream dependencies, exact-input requirements, and exposed-state requirements must also be encoded. Compatibility includes grouping, reduction, inputs, windows, family, capacity, freshness, and timing. -### Accuracy, windows, and physical validity +### 6.5 Objective: pay once for shared producers -- Every query's composed guarantee must satisfy its target, including applicable - confidence requirements. Summary error and window-representation error require - a valid composition rule, not an assumed arithmetic sum. -- Every window read must cover the correct data at each modeled evaluation, - without omission or double counting. Regular-grid checks can establish this - symbolically without expanding every timestamp. -- Every selected operation must have a supported physical implementation. -- A selected materialization must provide fresh, available state when its readers - execute. Enforce the actual physical compatibility rules rather than assuming - that every ancestor forces the same materialization on all descendants. -- Raw and intermediate-state retention must cover every selected reader's needs. +For the base constant-cost case: -Reject provably invalid choices during generation. Encode remaining joint checks -in the model, or use a sound validation/refinement mechanism described below. +```text +min Σ_{g,c} d[g,c] × x[g,c] + + Σ_{b,s} f[b,s] × y[b,s] + + Σ_{g,c,j,b,s} r[g,c,j,b,s] × z[g,c,j,b,s] +``` -### Consumer rates and resource accounting +The producer's owned charge appears once, regardless of reader count. Exclude its cost from readers' private coefficients. Reader-specific work is charged per actual computation, not blindly per syntactic occurrence; another shared merge result needs its own ownership if reused. -Introduce consumer-reach and activation variables where a node's execution rate -depends on selected ancestors, readers, and materialization. For a binary product -`v = a AND b`, use the complete linearization: +For context-dependent rates and memory lifetimes, extend the model using consumer-reach variables, rate/configuration variables, and bounded linearizations. For example, binary `v = u AND w` requires: ```text -v ≤ a -v ≤ b -v ≥ a + b - 1 +v ≤ u +v ≤ w +v ≥ u + w - 1 ``` -A query-time shared build runs at the union of actual reader triggers, accounting -for time alignment and reuse scope. Two readers on 60s and 90s schedules aligned -at zero have union rate `1/60 + 1/90 - 1/180`, not the rate of the 30s gcd grid. -A gcd grid may describe legal pane alignment without describing execution count. +Raw-retention and pane-retention maxima need bounds linked to selected readers. Lower-bound-only maximum constraints are tight only when the objective or other constraints enforce minimization of that variable; otherwise encode its intended value explicitly. -For one raw source, retention follows the longest required raw horizon under -the selected storage policy, rather than the sum of per-query copies. Pane -retention similarly follows the selected readers and maintained layers. If a -maximum is modeled only with lower bounds, positivity/minimization may make it -tight; otherwise add constraints enforcing its intended value. +### 6.6 Accuracy, validity, and resource limits -### Objective and optional capacity constraints +- **Accuracy:** compatibility filtering plus valid root-level composition constraints or a sound refinement loop. +- **Physical support:** selected operations must have supported physical implementations. +- **Materialization:** freshness, lifetime, update/refresh behavior, storage capability, and upstream timing must be consistent. +- **Windows:** validate alignment, pane coverage, offsets, historical reads, retention, and any algorithm-specific error contracts. +- **Reuse:** enforce exact-child/state-exposure conditions for rollup and top-k reuse. +- **Resources:** constrain CPU, memory, disk, and other requested budgets with the configured concurrency and resource model. +- **Latency:** enforce the configured per-query bounds and applicable recurrence checks. -The billing objective is: +For truly additive resource `k`, a capacity constraint can take the form: ```text -min selected private-node execution costs - + enabled shared-producer execution costs - + retained raw and intermediate-state costs - + temporary-state and other plan-level costs +Σ u_x[k,g,c] × x[g,c] + Σ u_y[k,b,s] × y[b,s] + + Σ u_z[k,g,c,j,b,s] × z[g,c,j,b,s] ≤ U[k] ``` -Give each charge one owner. A shared build must not be included both in every -reader's private coefficient and in `y[b,t]`. Account for update, merge, -estimation, expiration, and compaction where the selected implementation needs -them. Freeze calibration and model inputs before solving. - -If requested, constrain CPU, memory, disk, and per-query latency. Resource -coefficients must match their meaning: average memory is not peak memory, and -summing query work does not establish concurrent CPU or memory demand without -an execution/concurrency model. A cluster objective can minimize latency or -memory under these constraints. A Pareto request requires a frontier search, -not a claim that one scalar solve returns every nondominated plan. - -## Solving and validation - -Start with a complete MILP for the supported finite candidate space where the -costs and constraints can be encoded accurately and compactly. MILP does not -require logic-based Benders decomposition (LBBD). - -A separate exact evaluator remains useful for checking a decoded plan. Validation -alone cannot certify optimality if the model omits costs or legal candidates. -If difficult composed checks or costs are deliberately omitted, the model must -be a valid relaxation, and refinement is required: - -1. Solve the relaxed model and retain its certified solver lower bound. -2. Decode a complete physical plan and validate its accuracy, timing, resources, - and full cost. A validated plan supplies an incumbent upper bound. -3. Exclude a proved-infeasible combination, or add a proved-valid cost bound. -4. Repeat until the certified bound closes the target gap or the budget expires. - -A no-good cut must include every decision needed to prove infeasibility. A cost -cut must be conditioned on all choices affecting that charge; applying a -whole-plan cost to an incomplete candidate signature can exclude a cheaper plan. -Excluding a feasible, evaluated configuration is safe only if its validated plan -is retained and the proof accounts for that excluded region. Do not confuse a -solver's current feasible objective with its certified global lower bound. - -LBBD is one possible master/subproblem design for this refinement, not a -requirement. An iterative cut loop is not automatically LBBD. Select a -decomposition only when measured model size or difficult checks justify it. - -Split workload components only when all potential sharing, retention, and -feasibility interactions are absent. Global capacity constraints can couple -otherwise separate components: keep the joint model or combine complete resource -frontiers. Unknown interactions remain connected. - -### Reporting and exactness - -Report candidate/configuration counts, model size, generation and solver time, -validation effort, incumbent cost, certified lower bound, gap, and stop reason. -The gap is `(U-L)/U` for positive incumbent cost `U`; define zero-cost handling -explicitly. A time limit with no incumbent means no feasible plan found, not -proved infeasible. If refinement has exhausted the remaining region but retained -an incumbent, return that incumbent rather than reporting the workload infeasible. - -Optimality is relative to the generated finite candidates and configured models. -It is not a guarantee of measured runtime, completeness over all pane widths or -summary sizes, or support for unimplemented algorithms. Ties require an explicit -selection policy; equal-cost plans need not have identical structure. - -## Workflow and example +Peak memory and concurrent CPU generally need more than a sum of average-use coefficients. Distinguish resident state from temporary lifetimes and overlapping tasks. Extend cost models only with validated usage accounting. + +### 6.7 Why not optimize each group independently? + +Two groups can each run a private KLL or read one common KLL build. A reader's incremental cost depends on whether another candidate already enables that build and whether it changes the required size, trigger rate, or retention. + +A DP retaining only the cheapest local candidate per group can lose the joint optimum. DP with complete sharing context is possible, but its states can become large. MILP jointly selects discrete configurations; local caching or DP remains useful for parts that genuinely decompose. + +## 7. Worked Example: P95 and P99 Quantiles + +Assume identical filtered input, grouping, and aligned windows: ```text -Stage 0: queries → logical DAGs - ↓ -Build initial memo: identities, dependencies, query roots - ↓ -Stage 1, Pass 1: generate per-group computation candidates - ↓ -Stage 1, Pass 2: add ASAP-aware CSE candidates and shared configurations - ↓ -Stage 2: generate legal physical and materialization candidates - ↓ -Stage 3: build and solve the joint MILP - ↓ -Decode and validate the selected physical DAG - ↓ -Return a validated plan and search report; refine the model if required +Q1: quantile_over_time(0.95, stream[15m]), every 1m +Q2: quantile_over_time(0.99, stream[15m]), every 1m + +G95: Quantile(q=0.95, input=X, window=15m) +G99: Quantile(q=0.99, input=X, window=15m) ``` -Consider: +The result groups differ. Pass 1 adds exact and supported KLL candidates. Pass 2 discovers compatible shared builds: ```text -Q1: p95(latency[15m]), evaluated every 1m -Q2: p99(latency[15m]), evaluated every 1m + KLL build(X, 15m, size=s) + / \ + estimate P95 estimate P99 + | | + Q1 Q2 ``` -Stage 0 establishes two different quantile results with the same window width -and evaluation step. Initial memo construction keeps separate result groups and -shares eligible identical inputs. Pass 1 adds exact and supported KLL candidates, -with sizes derived from the readers' accuracy requirements. - -Pass 2 adds candidates for a shared whole-window KLL build and supported pane -compositions. One possible configuration maintains 1m KLL panes and reads the -latest 15 panes. The two estimates can share the build, and, where legal, a merge -result. Other widths and hierarchies remain candidates only when generated and -supported; 5m panes alone cannot cover every 1m-aligned evaluation. - -Stage 2 supplies legal assignments: for example, pane builds at ingestion and -merge/estimate at query time, or query-time construction from retained raw data. -Stage 3 compares their full costs. A shared build may be sized for both readers; -private builds remain candidates. The selected plan meets both accuracy targets, -window semantics, latency limits, and any configured resource capacities. - -## Implementation and verification - -The inspected memo implementation in -`crates/plan-selection/src/pass/subexpr/` separates group construction, -summary capability sharing, window composition, scan sharing, pricing, and search. -Its memo builder currently receives Pass 1 candidates. Moving group construction -after Stage 0 and adapting Pass 1 to generate directly into groups is the target -architecture, not an assertion about the current call order. - -Likewise, the inspected MILP path supports usage-over-time pricing and routes -unsupported cost models to legacy search. Cluster-capacity and provisioned-cost -formulations in this document are design requirements, not claims of current -MILP coverage. EH, general direct sliding-state maintenance, and arbitrary -multi-layer pane generation require their own implementation work. - -Verify the design against exhaustive enumeration on small workloads and a -separate brute-force search over the memo. Compare minimum validated cost under -the same candidate inventory and models, not merely whether both return a plan. -Include: - -- Sharing that is profitable only when selected jointly. -- Reader-dependent summary sizes, evaluation rates, and retention. -- Multiple pane widths, alignment failures, and overlapping windows. -- Fused candidates, rollup dependencies, and cycles introduced by reuse. -- All supported materializations and internal-node assignments. -- Composed accuracy, shared scans, raw horizons, and global capacity limits. -- Zero-price resources, numeric tolerances, budget stops, and valid solver bounds. - -For any relaxed formulation, check on enumerated plans that its assigned cost -never exceeds full cost and that every generated cut is valid. Measure planning -time, peak memory, model size, solver gap, and validation rounds on representative -workloads before choosing stronger decomposition or configuration restrictions. +Illustrative costs per common horizon, assuming all listed choices meet requirements: + +| Choice | Build cost | Reader costs | Total | +| --- | ---: | ---: | ---: | +| Separate KLL builds | 8 + 8 | 1 + 1 | 18 | +| One shared KLL build | 8 | 1 + 1 | 10 | +| Separate exact computations | 12 + 12 | included | 24 | + +If a reader's target requires more capacity, choose a compatible larger shared build, supported private builds, or an exact candidate. Quantile level alone does not determine required KLL capacity; the accuracy model and target do. + +### 7.1 Extending the example to panes + +A supported candidate maintains 1m KLL panes at ingestion and merges the latest 15 at evaluation. Both readers can use the same panes and, if eligible, the same merge result. Their estimation remains distinct. Window composition is generated in Pass 2; Stage 2 determines its materializations and physical validity. + +For different window lengths, the chain must cover the longest selected reader, and each reads the appropriate panes. Other pane widths or hierarchies require explicit generated candidates and valid boundary handling. Window compatibility is independent of sketch compatibility. + +### 7.2 Example binary decisions + +With one producer role named `summary` and one compatible configuration `s`: + +```text +x[G95, KLL-reader] = 1 +x[G99, KLL-reader] = 1 +y[Shared-KLL(X,W), s] = 1 +z[G95, KLL-reader, summary, Shared-KLL, s] = 1 +z[G99, KLL-reader, summary, Shared-KLL, s] = 1 +``` + +The objective charges the producer once and adds the two readers' incremental costs. Configuration `s` includes or consistently references materialization. A query-time rebuilt full-window KLL is distinct from an ingestion-maintained pane chain; do not assume a full 15m sketch can directly delete expired samples every minute. + +## 8. Decomposition and Search-Space Control + +Build a coupling graph across all candidates, including child dependencies, fusion, shared producer choices, state exposure, common raw sources, and plan-level charges. Solve independent components only if objectives and feasibility factor. A shared capacity constraint or workload-wide calibration can prevent independent scalar optimization; retain the joint model or combine sufficient resource frontiers. + +This component decomposition differs from master/subproblem decomposition such as LBBD. + +1. Intern logical identities and compatible producer signatures with verified equality. +2. Remove provably invalid algorithm/window/materialization combinations with reasons. +3. Apply Stage 3 dominance only with the same external sharing interface and no worse relevant obligations; standalone price is insufficient. +4. Enumerate supported pane widths, capacities, retention configurations, and assignments without building complete-plan products. +5. Use a validated independent-query baseline as an incumbent where available; keep its true resource and accuracy checks. +6. Set solver time/gap limits and report incomplete search honestly. +7. Track candidate-generation, coefficient-generation, solver, and validation costs separately. + +Representative parameter subsets or adaptive enrichment can reduce model size, but narrow the finite candidate space. A solver certificate applies only to the current inventory. Expanding that inventory requires solving the expanded problem; validation does not prove omitted configurations are irrelevant. + +Start with a direct MILP where costs and checks can be encoded compactly. The current implementation uses a lower-bound model with lazy cost/validity refinement for difficult terms. That is an implementation strategy, not a mathematical requirement of MILP. LBBD is one possible formally justified master/subproblem design; an arbitrary validate-and-add-cuts loop is not automatically LBBD. + +## 9. Plan Reconstruction and Verification + +1. Traverse query roots, selected candidates, role attachments, and producer dependencies. +2. Construct one physical DAG, respecting timing-specific copies and selected materializations. Deduplicate only compatible physical instances and reject cycles. +3. Check root coverage, dependency support, execution capabilities, and state availability. +4. Run the configured accuracy evaluator on each complete query result. +5. Run the configured latency evaluator under its timing and resource assumptions. +6. Run the full cost/resource evaluator, including retention, temporary memory, and shared charges. +7. If validation fails or cost was underestimated, refine and re-solve under a bounded budget, or return a validated baseline with an explicit reason. + +An invalid incumbent does not prove the entire workload infeasible. No plan found before timeout is also not an infeasibility proof. If a validated plan exists but the remaining model is exhausted, return that plan rather than reporting infeasibility. + +For a relaxed model, keep a certified solver lower bound `L` and best validated cost `U`. Use `(U-L)/U` for positive `U`, with explicit zero-cost handling and numerical tolerances. A feasible solver objective is not automatically a certified lower bound when the solve stops early. + +Feasibility cuts must exclude only proved-invalid combinations. Cost cuts must be conditioned on every decision affecting their charge, including materializations, reader attachments, and configurations. A cut based only on group candidates can be invalid if those candidates permit several physical realizations. Retain any feasible configuration excluded after evaluation and include its region in the optimality argument. Refine until the valid global bound closes the target gap or the budget expires. + +A direct exact MILP and a validated matching evaluator can certify the finite-space optimum. A relaxed MILP plus validation of one plan cannot do so without sound refinement/bounds. Report modeled objective, validated cost, solver gap, global validated bound/gap where available, stop reason, and fallback routing distinctly. + +## 10. Data Structures and Interfaces + +Reuse the current architecture rather than adding a parallel optimizer framework. The following names map the conceptual model to the code inspected on `origin/main` at `900800fd`: + +| Design concept | Current representation / boundary | +| --- | --- | +| Logical identity | `MemoGroup { node, children, parents, queries, roots_of }` and `GroupId`. | +| Group candidate inventory | `Memo { groups, candidates }`. | +| Implementation candidate | `ImplementationCandidate { implements, node, children, origin, coupling_sites, budget_share, requires_exact_children, rollup_of, materializations, requires_sites }`. | +| Shared configuration | `CouplingSite { key, kind }`, `CouplingKind`, and rule-specific thresholds/reader checks. | +| Physical choice | MILP `Choice`, including candidate identity, materialization assignment, coefficients, and copy/scan context. | +| Per-node usage and costs | Stage 3 pricing, `ResourceUsage`, `PlanningModels`, `PlanSelectionSettings`, and derived choice coefficients. | +| Selected output | `PhysicalCandidate` / `PhysicalASAPDAG`, `Selection`, and search/MILP reports. | + +`SharedProducer`, `ProducerConfiguration`, and role attachments are formulation concepts. Existing sites and shared-node keys provide much of this representation. Introduce explicit records only where needed to express multiple suppliers, upstream producers, or ownership that existing structures cannot represent; do not duplicate identity or charge ownership in two catalogs. + +A conceptual extension, when required, is: + +```text +ProducerConfiguration { + producer_identity, configuration_id + family_or_operator, input_and_update_signature + window_parameters, summary_parameters + materialization_assignment, retention_and_freshness_contract + required_inputs, compatible_reader_roles + resource_profile, accuracy_contract +} + +SelectedChoices { + group_candidates, producer_configurations, reader_attachments + reconstructed_PhysicalASAPDAG + modeled_objective, certified_bound, validated_cost + solver_status, validation_status, search_report +} +``` + +These are design sketches, not new public APIs or substitutes for `Selection`. + +### Current module boundaries + +Current main has moved the earlier `pass/subexpr` implementation into: + +| Responsibility | Module | +| --- | --- | +| Group construction and memo data | `crates/plan-selection/src/search/memo/{mod,groups}.rs` | +| Pass 2 memo rules | `crates/plan-selection/src/search/memo/rules/{capability,windows,scans}.rs` | +| Reference memo search and construction | `crates/plan-selection/src/search/memo/brute_force.rs` | +| MILP model, coefficients, cuts, decoding, reporting | `crates/plan-selection/src/search/milp/{model,coefficients,cuts,decode,report}.rs` | +| Independent memo clusters | `crates/plan-selection/src/search/milp/clusters.rs` | +| Stage orchestration and legacy routing | `crates/plan-selection/src/search/legacy/stage1_to_stage3.rs` | +| Physical generation | `crates/physical-asap-optimization/src/lib.rs` | +| Local logical candidates | `crates/logical-asap-optimization/src/per_computation_candidates.rs` | + +The PR's older base may not contain the refactored paths. See the inspected [memo code](https://github.com/ProjectASAP/ASAPPlanner/tree/900800fd8a7e57c35c84d365c209bd687407f146/crates/plan-selection/src/search/memo), [MILP code](https://github.com/ProjectASAP/ASAPPlanner/tree/900800fd8a7e57c35c84d365c209bd687407f146/crates/plan-selection/src/search/milp), and [current layering design](https://github.com/ProjectASAP/ASAPPlanner/blob/900800fd8a7e57c35c84d365c209bd687407f146/docs/design_docs/proposals/planner-layering.md). + +Current `brute_force::memo` receives `PerComputationCandidates`, builds groups, and adds Pass 2 candidates. Search modes are `Legacy` (default), `MemoBruteForce`, and `MemoMilp`. The memo MILP uses `good_lp` with HiGHS, clusters under supported conditions, decodes choices, and evaluates complete plans through Stage 3 with lazy refinement. Initial construction before Pass 1, arbitrary pane hierarchies, EH/direct sliding algorithms, and broader MILP cost-model support remain proposed extensions. + +## 11. Implementation Plan + +These phases distinguish existing foundations from proposed changes. + +| Phase | Deliverable | Exit criterion | +| --- | --- | --- | +| 1. Early memo foundation | Separate logical identity construction from Pass 1 attachment; move initial construction after Stage 0. | Eligible identical expressions intern once without changing semantics or losing reader obligations. | +| 2. Direct group generation | Adapt existing per-computation generation to emit group candidates, including fusion. | Explicit dependencies and contracts; no complete-workload Cartesian product. | +| 3. Sharing representation | Reuse sites/thresholds and shared-node identity; add role records only where necessary. | P95/P99 share a compatible build without merging result groups or duplicating charges. | +| 4. Physical choices and costs | Reuse legal materializations and Stage 3 accounting; extend supported configurations explicitly. | Same pricing and eligibility as authoritative physical generation/evaluation. | +| 5. MILP coverage | Maintain current selection/refinement path; extend role/dependency/resource coverage where needed. | Matches exhaustive and memo brute force on the same finite inventory. | +| 6. Scaling | Proven independent clusters, safe Stage 3 dominance, solver budgets, optional configuration enrichment. | Reports generation/model/solve costs without storing every workload candidate. | +| 7. Validation and window extensions | Reconstruct/validate every selected plan; implement new algorithms only with runtime support and guarantees. | Plans meet declared constraints or return a validated fallback/explicit failure status. | + +## 12. Testing and Evaluation + +### Correctness + +- Eligible identical expressions share a group; P95 and P99 remain separate. +- Different accuracy obligations preserve the candidates needed by each reader. +- Compatible KLL readers share a build; incompatible inputs, grouping, filters, or windows do not share without an explicit valid conversion. +- Shared builds and shared merges are counted once at actual execution rates; reader-specific work is counted appropriately. +- Selected dependencies are active, unused groups/producers cannot activate, and fusion does not double-implement absorbed groups. +- Multiple producer roles, upstream producer dependencies, and cycles are handled correctly. +- Pane widths, phases, partial boundaries, offsets, and historical reads satisfy coverage; future EH candidates are tested only after supported contracts exist. +- Unsupported materializations are excluded; retained state obeys timing, freshness, and lifetime requirements. +- Tiny-instance MILP, memo brute force, and exhaustive search agree on minimum validated cost for the same inventory/models, with an explicit tie policy. +- Zero prices, capacities, numeric tolerances, timeout/no-incumbent cases, and fallback routes have correct status. +- For relaxed models, every tested plan's lower-bound price is no greater than full price, and generated cuts preserve all feasible cheaper configurations. + +### Performance + +Measure workload size against groups, candidates, producer/configuration identities, reader edges, variables, constraints, generation/solve/validation time, and peak planner memory. Compare independent-query planning with joint planning on modeled and measured resource savings. Vary query overlap, sketch sizes, pane widths, evaluation schedules, and materializations. + +Report time-limited incumbent quality and certified gaps, modeled versus validated cost, and planning overhead versus execution savings. Evaluate billing and resource-constrained deployments only through supported engines; do not label a legacy fallback as MILP coverage. + +## 13. Open Design Questions + +1. Which accuracy compositions admit compact constraints, and which need sound evaluator-driven refinement? +2. Which operators support pane merge, direct expiration, or EH with the required guarantees and boundary behavior? +3. What is the shared-instance identity across evaluation rounds, and which freshness/update policies are executable? +4. Should resource and latency models retain current assumptions or add validated concurrency/critical-path models? +5. How should finite capacities, pane widths, retention horizons, and adaptive inventory expansion be chosen? +6. Which incompatible windows, update rates, or groupings can be connected through an explicit conversion candidate? +7. How should billing objectives, cluster objectives, accuracy margins, and planning budgets be prioritized? +8. When do model size and refinement measurements justify a formal LBBD or another decomposition, rather than the direct model? + +## 14. Design Outcome + +The memo represents logical identities and per-group implementation candidates. ASAP-aware CSE exposes shared computations through producer identities, coupling sites, and reader dependencies without conflating different results. MILP selects a consistent physical DAG and attributes each charge to the work or state that incurs it. The selected plan remains subject to the existing accuracy, physical, latency, and cost evaluators; optimality claims require a complete finite-space model or a sound refinement proof. From 8fc506233c9dd2a0ac9ec183d814d014c2f4bbc7 Mon Sep 17 00:00:00 2001 From: zzylol <50204836+zzylol@users.noreply.github.com> Date: Fri, 23 Oct 2026 23:12:10 +0000 Subject: [PATCH 24/53] docs: focus memo MILP proposal into eight sections --- .../proposals/joint-planning-search.md | 551 +++++------------- 1 file changed, 138 insertions(+), 413 deletions(-) diff --git a/docs/design_docs/proposals/joint-planning-search.md b/docs/design_docs/proposals/joint-planning-search.md index f75d830cf..2aaec6aa9 100644 --- a/docs/design_docs/proposals/joint-planning-search.md +++ b/docs/design_docs/proposals/joint-planning-search.md @@ -2,325 +2,166 @@ **Status:** Design proposal -**Scope:** Multi-query optimization across exact operators, ASAP primitives, window implementations, sharing, and materialization +**Scope:** Multi-query optimization across exact operators, ASAP primitives, window implementations, sharing, and materialization. -**Audience:** Planner designers and implementers +## 1. Motivation and Challenges -## 1. Motivation and Problem Statement +ASAP primitives create opportunities to replace expensive exact computations and share summaries, panes, rollups, scans, or retained raw data. They also enlarge the optimization search space. Enumerating complete workload candidates multiplies implementation, sharing, window, and materialization choices. Optimizing queries independently avoids that enumeration but can miss joint savings. -ASAP primitives introduce optimization opportunities: a summary can replace expensive exact computation, one summary can answer several queries, and compatible queries can reuse panes, rollups, scans, or retained raw data. These candidates can reduce CPU, memory, storage, and evaluation latency. +The challenge is cross-query dependence: a candidate's incremental cost changes when another candidate enables a shared producer, requires a larger summary, or extends retention. Selecting the cheapest candidate for each group independently can therefore lose the workload optimum. -They also enlarge the search space. Enumerating and pricing every complete workload plan produces combinatorial growth. Planning each query independently can miss sharing and account for a common physical producer multiple times. +The design uses a memo to represent candidates compactly and a mixed-integer linear program (MILP) to select a consistent physical plan. The goals are to avoid repeated candidate generation, expose ASAP-aware sharing, and minimize the configured objective subject to accuracy, latency, and resource constraints. Optimality applies to the generated finite candidate set and configured models, not every imaginable plan. New sketch/window algorithms and runtime scheduling are outside this proposal. -**ASAPPlanner should represent candidates compactly and select a minimum-cost valid physical plan for the workload.** Optimality is relative to the generated candidate set, configured models, and solver tolerances. - -### Goals - -1. Avoid repeated exploration of identical or proven-equivalent logical sub-expressions. -2. Represent ASAP-aware reuse explicitly, including reuse between nonidentical logical expressions. -3. Optimize workloads under resource prices or a specified objective with hard resource constraints. - -The coupled decisions include: - -- Exact computation, exact accumulator state, or approximate summaries such as KLL. -- Legal window implementations and their configurations. -- Shared builds, pane chains, scans, rollups, and per-source raw retention. -- Materialization timing, storage medium, and lifetime. -- End-to-end accuracy, latency, and deployment constraints. -- Shared-producer costs counted once per actual execution or retained state, with reader-specific work counted separately. - -### Non-goals - -- Inventing arbitrary sketches or window algorithms. -- Assuming a closed-form linear cost/size/accuracy relationship. -- Substituting MILP estimates for physical-plan validation. -- Proving optimality over ungenerated configurations or all imaginable plans. -- Implementing runtime scheduling or retention services inside the optimizer. - -## 2. Terminology - -### 2.1 Logical expressions, memo groups, and candidates +## 2. Memo-Based Candidate Representation Terminology follows [planner-layering.md](planner-layering.md): -| Term | Meaning | -| --- | --- | -| **Logical expression** | A node and its input sub-expressions in the language-independent `LogicalDAG` (Pre-ASAP IR). It specifies selectors, transformations, predicates, aggregations, grouping, window semantics, and output schema, while preserving source-language behavior such as series identity, evaluation timing, and missing-data semantics. It contains no summary or materialization choice. | -| **Memo group** | One distinct logical sub-expression and its output contract. Proven-equivalent logical expressions can have the same group identity. The current implementation hash-conses eligible identical expressions; it does not implement a general equivalence saturation engine. | -| **Implementation candidate** | One way to implement a group over separately chosen child groups. It owns private nodes, records dependencies and rule origin, and can carry coupling sites and accuracy obligations. A fused candidate can implement several groups. This corresponds to `ImplementationCandidate` in the memo code. | -| **Whole-workload candidate** | A DAG for the whole workload, as defined by the layering document: a `LogicalDAG`, logical `LogicalASAPDAG` (Post-ASAP IR), or timed `PhysicalASAPDAG`, depending on the stage. Its candidate sets are `CandidateLogicalDAGs`, `CandidateLogicalASAPDAGs`, and `CandidatePhysicalASAPDAGs`. | -| **Shared producer** | A physical computation or maintained intermediate state that compatible selected readers can reuse, including its upstream dependencies. Examples are a summary build, pane chain, scan, or finer result used by rollup. It need not itself be a query result. | -| **Coupling site** | A shared decision affecting several implementation candidates. Equal site keys identify one decision; thresholds enumerate legal configurations. Existing kinds include `SharedBuildSize`, `PaneChain`, `ScanLabels`, and `RawHorizon`. | -| **Physical plan** | One selected whole-workload `PhysicalASAPDAG`, produced by physical ASAP-aware optimization and selected at Stage 3. It has physical operator structure, execution timing, materialization assignments, and query roots. Shared physical nodes have multiple consumers. Execution remains downstream of ASAPPlanner. | - -Below, “candidate” means a local implementation candidate when discussing groups, and a whole-workload candidate when discussing stage outputs. The memo compactly represents combinations of local candidates instead of constructing every whole-workload DAG. - -A group's logical result remains fixed. Its exact candidates preserve that result exactly; approximate candidates implement it under explicit accuracy contracts. An approximate estimate is not an exactly equivalent logical expression. Different reader accuracy requirements can require different candidates without changing the requested statistic. +- **Memo group:** One distinct logical sub-expression, preserving source-language semantics, inputs, filters, grouping, window semantics, and output contract. Proven-equivalent expressions can share an identity. Current code interns eligible identical expressions rather than discovering arbitrary equivalences. +- **Implementation candidate:** One way to implement a group over child groups, using exact operators, exact accumulator state, or ASAP primitives. Logical candidates record dependencies and accuracy contracts; physical choices additionally specify supported operators and materialization assignments. A fused candidate can implement several groups. +- **Shared producer:** A physical computation or state reusable by compatible candidates, such as a KLL build, pane chain, scan, or finer aggregation result. A producer includes its input dependencies and legal configurations. -### 2.2 Three distinct kinds of reuse +Memo grouping avoids duplicate logical exploration. Conventional CSE exposes reuse of identical computations; ASAP-aware CSE additionally exposes shared intermediate state between different results. P95 and P99 remain separate result groups but can read one KLL. Sharing is represented between their candidates, not by merging their output semantics. One memo identity does not itself force runtime materialization. -| Mechanism | Condition | Representation | Example | -| --- | --- | --- | --- | -| Memo equivalence | Expressions have the same logical semantics. | One logical identity with implementation candidates. | Two occurrences of the same filtered sum over the same window. | -| Conventional CSE | Identical computation has compatible semantics and execution context. | One physical producer with multiple consumers, when the selected physical realization reuses it. | A common scan or identical exact aggregate. | -| ASAP-aware sharing | Different requested results can consume compatible intermediate state. | Separate result groups linked to a common producer/configuration. | P95 and P99 read one KLL; different window lengths reuse panes. | +In the layering document, a stage-level candidate is a DAG for the **whole workload**. Local `ImplementationCandidate` records compactly represent its component choices, avoiding construction of every complete DAG. Approximate implementations retain the original logical result as their target and carry an accuracy contract; they are not exactly equivalent expressions. -Memo grouping deduplicates logical identities and associated search work. It does not automatically deduplicate arbitrary search states with different readers, materializations, or accuracy obligations, and it does not force runtime materialization. CSE and ASAP-aware sharing expose physical reuse candidates without equating different outputs. - -## 3. ASAPPlanner Workflow - -This is the target workflow. The current memo builder receives Pass 1 candidates; moving initial group construction before Pass 1 is a proposed change, not a description of today's call order. +## 3. ASAPPlanner Optimization Workflow ```text Stage 0: frontends -> language-independent LogicalDAGs - | -Build initial memo - - Intern eligible identical / proven-equivalent sub-expressions - - Record roots, group dependencies, semantics, and reader requirements - | -Stage 1 / Pass 1: generate per-computation candidates - - Exact, exact accumulator state, summary families/sizes, fusion - - Preserve logical output and approximation contracts - | -Stage 1 / Pass 2: add ASAP-aware CSE candidates - - Identical expressions, summary capability, window composition - - Rollup, exact top-k reuse, scan widening, raw-horizon interactions - - Record compatible producers/configurations and dependencies - | -Stage 2: generate legal physical candidates - - Physical operators, materialization timing, medium, and lifetime - - Check window configurations against physical capabilities - | -Analyze coupling across all generated candidates - - Separate components only when objectives and feasibility factor - | -Stage 3: joint MILP selection - - Choose candidates, producer configurations, and reader attachments - - Enforce accuracy, latency, dependencies, and resource constraints - | -Reconstruct and validate the selected PhysicalASAPDAG - - Refine/re-solve when required, or return a validated fallback/status + ↓ +Initial memo: logical identities, dependencies, query roots + ↓ +Stage 1 / Pass 1: per-computation candidates + exact / exact accumulator / summary family and size / fusion + ↓ +Stage 1 / Pass 2: ASAP-aware CSE candidates + shared builds / window composition / rollup / top-k reuse / scans + ↓ +Stage 2: legal physical implementations and materialization assignments + ↓ +Stage 3: joint MILP selection -> reconstruct and validate PhysicalASAPDAG ``` -### 3.1 Stage 0: Normalize and build the initial memo - -Frontends preserve source-language semantics. Canonicalize only where equivalence is established: names, predicates, operand order, or windows cannot be rewritten merely because they look similar. Floating-point behavior, nulls, missing samples, and evaluation context can prevent apparently commutative rewrites. - -Intern distinct logical sub-expressions, verifying equality after hashing. Keys cover relevant source/input identity, arguments, filters, grouping, schema, series identity, window width/offset/alignment, and time semantics. Include late-data or ordering policies where they affect results. Current identity rules leave keyless nodes, such as range windows, separate per reader. - -Store accuracy targets and evaluation requirements as reader/root obligations. Include a requirement in identity only if it changes the logical semantics; otherwise retain it in candidate-generation/cache context. Do not merge P95 with P99 merely because both can use KLL. - -### 3.2 Stage 1, Pass 1: Generate local candidates - -Generate per-computation candidates against group identities rather than composing complete workload plans. For eligible aggregates, include the exact implementation: exact accumulator state when supported and constructible, otherwise the original exact computation. Add supported summary families and sizes and fused-computation candidates. - -Record each candidate's private nodes, logical output, child groups, guarantees, and accuracy obligations. Cache generation by group and every relevant context value, including reader requirements. This avoids repeated generation without dropping sizes needed by stricter readers. It also avoids the Cartesian product across unrelated groups. - -### 3.3 Stage 1, Pass 2: Discover sharing opportunities - -Use the ASAP-aware CSE rules from the layering design. Identical expressions are already group identity in the memo. Summary capability, window composition, rollup, exact top-k reuse, and scan widening add implementation candidates or dependencies between groups. +Build initial identities after Stage 0. Intern expressions only when semantics and sharing legality match; verify equality after hashing. Preserve reader-specific requirements and keyless-node identity rules. Cache candidate generation by group and relevant context, including accuracy obligations. Current memo construction receives Pass 1 candidates; moving it earlier is a proposed change. -Match producer signatures using summary input data, family, reduction, grouping, item/weight, filters, and applicable window semantics. Requested quantile levels differ at estimation, but their build may match. Window compatibility is checked separately from summary capability. +Pass 1 generates each computation's candidates without composing their workload-wide Cartesian product. Pass 2 matches producer signatures and adds compatible sharing candidates, including window composition. Stage 2 determines physical operators and materialization; Stage 3 is responsible for cost-based selection. Earlier stages prune provably invalid candidates with reasons. -For each opportunity, record: +### Window and materialization choices -- Producer identity, input/update semantics, and upstream dependencies. -- Finite configurations: summary size, pane width, retention, or scan labels. -- Compatible reader candidates and any required exposed state or exact input. -- Legal materializations to be resolved by Stage 2. -- Usage inputs and contracts needed for Stage 3 pricing, accuracy, and latency. +Keep these dimensions separate: -Pass 2 generates logical window-composition candidates; Stage 2 does not invent pane algorithms. Raw retention is a shared accounting dependency, not necessarily an executable operator. Sharing remains optional: independent physical instances remain selectable when supported and useful. +- **Semantics:** requested window width, offset, evaluation timestamps, and alignment. +- **Implementation:** raw recomputation, supported direct sliding state, fixed-width panes, or an EH algorithm. +- **Materialization:** when each internal state is computed and how long it is retained. -### 3.4 Stage 2: Physical candidates and materialization +A pane is an aligned, non-overlapping fixed-width interval and its aggregate/summary state. An EH bucket represents a portion of the stream under an exponential histogram algorithm, with algorithm-specific merge, expiration, and error rules. -Stage 2 makes physical operator and materialization decisions; it does not select the cheapest valid candidate. For example, exact top-k may lower to sort and limit. Materialization is assigned to individual eligible nodes/sub-DAGs, rather than one label for an entire query. - -| Mode | Meaning | Cost drivers | +| Implementation | Ingestion, when maintained there | Evaluation | | --- | --- | --- | -| `NotMaterialized` | Run at query time without retaining an independently reusable output. | CPU and transient memory at the actual execution rate. | -| `MatAtQuery(Memory)` | Build once for applicable readers within an evaluation. | Per-evaluation build and memory held during the evaluation. | -| `MatAtIngestion(Memory)` | Maintain as input arrives and retain state for readers. | Updates, retained state, and reads. | -| `MatAtQueryKeptAcrossEvaluations(Memory)` | Build at query time and keep state across evaluations. | Build, subsequent updates/refresh, retention, and reads. | - -Use the actual `Materialization` enum and `Medium`. Ingestion eligibility depends on `DataWorkload`, recurrence, predictability, input support, and window/phase legality. Current Stage 2 requires ingestion-time builds below ingestion-time builds; shared non-summary nodes below them use the permitted unmaterialized form. Distinguish execution at ingestion time from forcing every upstream output to be materialized. - -The current retained-query-time mode requires deployment capability and an eligible tumbling-pane chain building one new pane per evaluation. It is not a generic cache mode for arbitrary nodes. Current physical generation also offers query-time sharing for eligible non-summary sub-DAGs. Disk placement, arbitrary cache policies, parallelism, partitioning, and resource management must not be claimed implemented merely because they appear in the broader design. - -Following the layering document, earlier stages prune provably invalid candidates with reasons; Stage 3 selects among valid candidates. Cost-based dominance is a Stage 3 search optimization and needs a proof preserving every external sharing and resource obligation. - -## 4. Window Semantics, Algorithms, and Materialization - -Window semantics, window implementation, and materialization are separate dimensions, subject to compatibility constraints. +| Direct sliding-window state | Add data and remove expired contributions. | Read state and compute result. | +| Pane-based | Update panes and evict expired panes. | Merge required panes and compute result. | +| EH-based | Update, merge, and evict buckets. | Estimate the window result. | -- **Semantics:** width, offset, evaluation timestamps, endpoint convention, and time alignment. -- **Implementation:** raw recomputation, a supported direct sliding-state algorithm, fixed-width panes, or a supported EH algorithm. -- **Materialization:** when internal state is built, maintained, retained, and discarded. +This table describes possible strategies. Current pane support does not imply general direct sliding-state or EH support. Offer only executable candidates. Multiple pane widths can coexist; alignment, coverage, mergeability, retention, and error contracts determine legality. A 15m window evaluated every 1m can use 1m panes; complete 5m panes alone generally cannot cover every endpoint. -A 15m trailing window evaluated every 1m is sliding-window semantics. It may be implemented using 1m tumbling panes maintained at ingestion, with query-time merge and estimation. Width and step both 15m give tumbling semantics when aligned. +Use existing `Materialization` options: `NotMaterialized`, `MatAtQuery(Memory)`, `MatAtIngestion(Memory)`, and `MatAtQueryKeptAcrossEvaluations(Memory)`. Assign them to eligible internal nodes, not necessarily the entire query. For example, maintain panes at ingestion and merge at query time. Generate only supported assignments: retained query-time state currently requires deployment capability and an eligible pane chain. Ingestion timing and upstream dependencies must satisfy Stage 2 rules. -Workload vocabulary matters: `QueryRecurrence` specifies request cadence. `RangeEvaluation { step_ms, span_ms }` specifies evaluation steps inside one request. A range request can contain many evaluation timestamps. Neither request rate nor a legal pane grid alone gives the number of computations; pricing must account for both and any reuse. +## 4. Cost Model and Optimization Objective -### Panes and EH buckets +Attribute resource usage to physical nodes in the selected DAG, then apply the existing Stage 3 cost model: -A **pane** is an aligned, non-overlapping fixed-width interval and its data or aggregate/summary state. A configuration defines width, origin, retention, and reconstruction rules. Partial boundary panes require explicit treatment: finer panes, raw boundary data, or an authorized approximation. Divisibility alone is insufficient when boundaries are misaligned. - -An **EH bucket** summarizes a contiguous portion of a stream according to a specific exponential histogram algorithm. Size levels typically grow geometrically; standard EH sizes commonly count represented items rather than fixed time durations. Merging, expiration, partial-boundary treatment, and error guarantees are algorithm- and operator-specific. EH is not a generic replacement for panes or an arbitrary KLL. - -Pane width, EH bucket size, and summary capacity are different parameters. Multiple pane widths may coexist, including a hierarchy of fine panes merged into coarser panes. The candidate generator defines which hierarchies exist in the search space; the solver cannot invent an absent composition. - -For example, 15 × 1m, 5 × 3m, and 3 × 5m can represent different candidates for an aligned 15m interval. Complete 5m panes alone cannot generally reconstruct a window whose endpoint moves every 1m. A multi-width composition needs explicit coverage and no-double-counting checks. - -| Window implementation | Ingestion work, when maintained there | Evaluation work | -| --- | --- | --- | -| Direct sliding state | Add records and expire contributions using a supported algorithm. | Read/finalize state. | -| Pane-based | Update active panes and maintain retention. | Select and merge panes, then finalize/estimate, unless those results are also maintained. | -| EH-based | Update, merge, and expire buckets. | Apply the supported window estimator. | - -This table describes possible implementations, not a support matrix. Current code supports tumbling-pane/pane-chain generation and summary computation. General direct sliding-state algorithms and EH are not implemented by that support. Arbitrary multi-layer pane generation also remains an extension. The downstream deployment supplies runtime scheduling, input binding, and retention behavior. - -Legality checks include mergeability, alignment/phase, coverage, ingestion support, historical evaluations, retention, and composed error. KLL pane merging does not provide reversible deletion of arbitrary old samples. - -## 5. Cost and Accuracy Model - -### 5.1 Cost attribution - -Stage 3 measures `ResourceUsage`, then applies `PlanSelectionSettings.cost_model`. The objective must agree with those configured models, rather than introducing an independent MILP pricing model. - -| Existing cost model | Objective | +| Model | Selection policy | | --- | --- | -| `UsageOverTime(ResourceUnitCost)` | Price resource usage per second, including occupancy over time. | -| `ProvisionedCapacity(ResourceUnitCost)` | Price provisioned peak memory, peak CPU cores, and disk. | -| `ClusterCapacity(ClusterLimits)` | Enforce memory/disk/optional-core limits and use `LowestLatency`, `LeastMemory`, or `ParetoFront`. | - -A Pareto request requires a frontier computation; one scalar MILP solution does not return the whole frontier. Current memo MILP supports usage-over-time pricing; unsupported cost models route to legacy search. The other formulations below are extensions, not existing MILP capabilities. - -Assign each charge to one physical owner: +| `UsageOverTime(ResourceUnitCost)` | Minimize priced usage per second. | +| `ProvisionedCapacity(ResourceUnitCost)` | Minimize priced provisioned memory, cores, and disk. | +| `ClusterCapacity(ClusterLimits)` | Enforce capacities and use `LowestLatency`, `LeastMemory`, or `ParetoFront`. | -1. Producer build/update, retained state, and maintenance. -2. Reader-specific merge, read, estimation, and finalization. -3. Nonshared physical instances. -4. Source-level retention and other plan-level charges. +Current memo MILP supports usage-over-time pricing; unsupported models route to legacy search. Broader MILP coverage is an extension. A Pareto objective requires frontier computation rather than one scalar solve. -Use consistent per-second rates, as in current code, or integrate all terms over one explicit billing horizon. Conceptually: +Give every charge one owner: -```text -C = p_cpu × CPU + p_mem × MemoryTime + p_disk × DiskTime - + p_io × IO + other explicitly priced usage -``` +- **Producer:** build/update, maintenance, and retained state. +- **Reader:** private merge, extraction, read, and finalization. +- **Other physical work:** private operators and temporary state. +- **Shared accounting:** per-source raw retention and other plan-level charges. -Occupancy over time is different from peak capacity. Without prices, use a specified objective under quotas; do not add CPU-seconds and bytes without defined weights. +Charge shared work once per actual execution or maintained state, not once per query path. Reader-specific work remains separately charged. Use a common horizon or consistent per-second rates; distinguish memory/storage occupancy over time from peak capacity. Producer costs can depend on selected readers, evaluation schedules, upstream materialization, and retention. Model that context rather than assuming every producer has a constant cost. -“Charge once” means once per actual execution or retained state. Query-time recomputation can occur repeatedly. A shared build's size, retention, and execution rate depend on its selected readers and ancestors' materializations. Constant coefficients are sufficient only when that context is fixed or explicitly represented by configuration/reach variables. +Accuracy is an end-to-end root requirement. Compose summary and window error only with valid contracts; locally admissible sketches do not prove a composed query accurate. Latency and resource limits must use the configured evaluator's timing and concurrency assumptions. Current latency accounting sums reached query-time work plus applicable compaction; this proposal does not silently replace it with a general parallel scheduler. Keep calibrated model inputs fixed during a solve. -Model the union of actual reader triggers, including phases and reuse scope. Readers on aligned 60s and 90s schedules have union rate `1/60 + 1/90 - 1/180`; a 30s gcd grid is not their execution rate. Current pricing remains authoritative if its schedule approximation differs; changing that policy requires a separately validated model change. +## 5. MILP Formulation -Per-source raw retention uses the selected storage policy and longest required horizon, not duplicated per-query copies. Longer retained state and lifetime-dependent temporary memory can couple otherwise separate computations. Freeze measured calibration inputs before search. +The base formulation below separates logical coverage, shared producers, and reader attachments. Extend it with consumer-reach variables and context-dependent charges where necessary, as current code does. -### 5.2 Accuracy +### 5.1 Sets and variables -Accuracy is an end-to-end query-root requirement evaluated by the configured accuracy model. Preserve error type, probability/failure parameters, evidence, and composition rules. A locally admissible sketch does not establish a valid ratio, arithmetic expression, or window approximation. - -Filter known-incompatible configurations, then encode proven composition bounds or finite feasible-combination tables where practical. Otherwise use sound validation and refinement. Summary error and window error cannot be assumed to add without a valid bound. Exact-only obligations must remain enforceable. - -### 5.3 Latency - -Use the configured latency model. Current Stage 3 models per-evaluation latency as the sum of reached query-time node work, counted once per node for that query, plus the longest applicable ingestion-state compaction. It assumes elastic CPU without queueing; fixed recurrence also imposes a repeat-interval check. Range-request validation follows the evaluator's declared per-step/request semantics. - -This is not a sum of all active workload costs and is not a general parallel-runtime critical-path model. If a future deployment uses parallel branches, queueing, or visibility delays, encode its precedence/scheduling bounds and validate them with the corresponding evaluator. Do not silently change latency semantics in the MILP. - -## 6. MILP Formulation - -This base formulation explains ownership, coverage, and reader attachments. Current code additionally models consumer reach, private materialization slots, per-reader copies, and context-dependent charges. These extensions are required whenever the base model's constant costs do not describe the physical plan. - -### 6.1 Sets and parameters - -| Symbol | Definition | +| Symbol | Meaning | | --- | --- | -| `Q` | Workload queries. | -| `G` | Logical memo groups. | -| `C_g` | Legal physical candidates owned by group `g`, with local materialization assignments. | -| `B` | Potential shared producer instances or explicit shared accounting states. | -| `S_b` | Finite configurations of `b`, such as summary size, pane width, retention, and materialization. | -| `I(g,c)` | Groups implemented by candidate `(g,c)`, including fusion coverage. | -| `R(g,c)` | Required input groups, conditional on choosing `(g,c)`. | -| `J(g,c)` | Producer roles required by `(g,c)`. | +| `Q`, `G` | Queries and memo groups. | +| `C_g` | Legal physical candidates owned by group `g`, including local materialization assignments. | +| `B`, `S_b` | Potential producer instances and their finite configurations. | +| `I(g,c)` | Groups implemented by candidate `(g,c)`, including fusion. | +| `R(g,c)` | Input groups required when `(g,c)` is selected. | +| `J(g,c)` | Shared-producer input roles required by `(g,c)`. | | `A(g,c,j)` | Compatible producer/configuration pairs for role `j`. | -| `d[g,c]` | Private incremental cost, excluding shared producers. | -| `f[b,s]` | Producer configuration's owned build/update/retention cost. | -| `r[g,c,j,b,s]` | Role-specific reader cost not already included elsewhere. | - -Costs use one common horizon. Finite `S_b` permits lookup-table costs without a linear size/cost function. Configuration enumeration is itself a search-space cost. If `f` depends on reader sets, trigger rates, or ancestors' choices, represent those conditions explicitly or compute the charge with reach/auxiliary variables; it is not an unconditional constant. -### 6.2 Decision variables +A role is an input slot, such as “pane chain”; it is filled by one compatible producer/configuration. `R` references logical results, while `J` references separately represented shared state. Do not encode the same dependency twice. If a producer is an ordinary memo group, its dependency can instead use `R`. ```text a[g] ∈ {0,1} group g is required -x[g,c] ∈ {0,1} candidate c is selected for group g -y[b,s] ∈ {0,1} producer b is enabled at configuration s -z[g,c,j,b,s] ∈ {0,1} candidate (g,c) uses (b,s) for role j +x[g,c] ∈ {0,1} select candidate c for g +y[b,s] ∈ {0,1} enable producer b at configuration s +z[g,c,j,b,s] ∈ {0,1} attach role j of (g,c) to (b,s) ``` -Materialization is part of `c` or `s`, or represented by explicit legal assignment variables such as `x[g,c,m]`. Do not maintain two representations that can disagree. A multi-node implementation can have several materialization slots rather than one mode for every private node. +Configuration `s` can include summary capacity, pane width, retention, and materialization. Discrete configurations allow measured lookup-table costs without assuming a linear size/cost function. Materialization can instead use explicit legal `x[g,c,m]` variables; avoid conflicting representations. Several internal nodes can require separate assignment slots. -### 6.3 Query coverage and group selection +### 5.2 Query coverage and dependencies -Let `Roots(q)` be the group references needed for query `q`, including multiple references in a scalar query root: +Let `Roots(q)` contain the group references needed for query `q`, including scalar roots with multiple references: ```text -a[g] = 1 for g ∈ Roots(q), q ∈ Q +a[g] = 1 for g ∈ Roots(q) +Σ_{c ∈ C_g} x[g,c] = a[g] without fusion ``` -Without fusion, select one implementation for an active group and none for an inactive group: - -```text -Σ_{c ∈ C_g} x[g,c] = a[g] -``` - -Current candidates can fuse several logical computations. The general form is: +With fused candidates, use the general coverage constraint: ```text Σ_{(h,c): g ∈ I(h,c)} x[h,c] = a[g] ``` -A fused candidate covers absorbed groups; do not additionally select standalone implementations for them. Every selected candidate requires its inputs: +Every required group is implemented once, possibly by fusion; absorbed groups do not also select standalone implementations. Selected candidates require their inputs: ```text a[h] ≥ x[g,c] for h ∈ R(g,c) ``` -An active group must be justified by a query root, selected consuming candidate, upstream producer requirement, or fusion coverage. Add reverse support/flow constraints so unused groups cannot activate merely because prices are zero. Support must include all valid dependency kinds, not only original logical parents. Enforce acyclicity or construct a candidate dependency space with a proved ordering. +Add reverse support constraints: an active group must be justified by a root, selected consuming candidate/producer, or fusion coverage. This prevents unused activation even at zero prices. Dependencies are conditional on selected candidates, not every possible child. Enforce acyclicity, exact-input requirements, and exposed-state requirements for reuse. -### 6.4 Sharing and configuration compatibility +### 5.3 Shared producers and compatibility -For each required producer role: +For each required role: ```text Σ_{(b,s) ∈ A(g,c,j)} z[g,c,j,b,s] = x[g,c] for j ∈ J(g,c) z[g,c,j,b,s] ≤ y[b,s] ``` -Use a separate equality per role: a candidate might need both a scan and a pane chain. Candidates running locally have no shared role, or use explicit local-fallback variables. An empty compatibility set makes that required-role candidate invalid. +Use one equality per role: a candidate needing a scan and a pane chain needs both inputs. Local execution is a separate candidate or explicit fallback choice. -If producer instance `b` can have only one physical configuration: +For one configurable producer instance: ```text Σ_{s ∈ S_b} y[b,s] ≤ 1 +y[b,s] ≤ Σ selected attachments requiring (b,s) ``` -Represent coexisting states as separate instances or explicitly permit multiple configurations. In particular, this is not a global one-pane-width constraint. - -Prevent unused producers: - -```text -y[b,s] ≤ Σ selected consumer attachments requiring (b,s) -``` - -The sum includes downstream producer attachments where producers consume other producers. Their upstream dependencies, exact-input requirements, and exposed-state requirements must also be encoded. Compatibility includes grouping, reduction, inputs, windows, family, capacity, freshness, and timing. +Represent coexisting configurations as separate instances or explicitly permit them. This is not a global restriction to one pane width. Attachments include downstream producers; encode producers' upstream dependencies too. Compatibility covers inputs, grouping, family, reduction, windows, capacity, freshness, and timing. -### 6.5 Objective: pay once for shared producers +### 5.4 Objective and cost attribution -For the base constant-cost case: +For fixed-context coefficients: ```text min Σ_{g,c} d[g,c] × x[g,c] @@ -328,9 +169,11 @@ min Σ_{g,c} d[g,c] × x[g,c] + Σ_{g,c,j,b,s} r[g,c,j,b,s] × z[g,c,j,b,s] ``` -The producer's owned charge appears once, regardless of reader count. Exclude its cost from readers' private coefficients. Reader-specific work is charged per actual computation, not blindly per syntactic occurrence; another shared merge result needs its own ownership if reused. +Here `d` is private incremental cost, `f` owns producer build/update/retention costs, and `r` is reader-specific work. All use the same horizon. Exclude shared costs from `d` to avoid duplicate charges; reuse of a merge result also needs explicit ownership. + +When execution rates depend on selected readers or ancestors, add consumer-reach/rate variables rather than using unconditional `f` or `r`. Reader recurrence, phase, and range-evaluation steps affect actual work; a pane-alignment gcd grid is not automatically an execution rate. Raw and pane retention depend on the longest required horizons. -For context-dependent rates and memory lifetimes, extend the model using consumer-reach variables, rate/configuration variables, and bounded linearizations. For example, binary `v = u AND w` requires: +For binary `v = u AND w`, use: ```text v ≤ u @@ -338,225 +181,107 @@ v ≤ w v ≥ u + w - 1 ``` -Raw-retention and pane-retention maxima need bounds linked to selected readers. Lower-bound-only maximum constraints are tight only when the objective or other constraints enforce minimization of that variable; otherwise encode its intended value explicitly. +Use valid bounds for other products and maxima. A lower-bound-only maximum is exact only when minimization or other constraints make it tight. -### 6.6 Accuracy, validity, and resource limits +### 5.5 Accuracy, validity, and budgets -- **Accuracy:** compatibility filtering plus valid root-level composition constraints or a sound refinement loop. -- **Physical support:** selected operations must have supported physical implementations. -- **Materialization:** freshness, lifetime, update/refresh behavior, storage capability, and upstream timing must be consistent. -- **Windows:** validate alignment, pane coverage, offsets, historical reads, retention, and any algorithm-specific error contracts. -- **Reuse:** enforce exact-child/state-exposure conditions for rollup and top-k reuse. -- **Resources:** constrain CPU, memory, disk, and other requested budgets with the configured concurrency and resource model. -- **Latency:** enforce the configured per-query bounds and applicable recurrence checks. +- **Accuracy:** filter incompatible configurations and enforce proven per-query composition bounds, feasible-combination tables, or sound refinement. +- **Physical validity:** require supported operators and legal materialization assignments, including freshness, upstream timing, and state lifetime. +- **Window correctness:** require alignment, coverage without omission/double counting, retention, and algorithm-specific guarantees. +- **Resource limits:** constrain requested CPU, memory, disk, and other resources under the configured concurrency model. +- **Latency:** enforce configured query bounds and applicable recurrence checks. -For truly additive resource `k`, a capacity constraint can take the form: +For an additive resource `k`: ```text Σ u_x[k,g,c] × x[g,c] + Σ u_y[k,b,s] × y[b,s] + Σ u_z[k,g,c,j,b,s] × z[g,c,j,b,s] ≤ U[k] ``` -Peak memory and concurrent CPU generally need more than a sum of average-use coefficients. Distinguish resident state from temporary lifetimes and overlapping tasks. Extend cost models only with validated usage accounting. - -### 6.7 Why not optimize each group independently? - -Two groups can each run a private KLL or read one common KLL build. A reader's incremental cost depends on whether another candidate already enables that build and whether it changes the required size, trigger rate, or retention. +Do not use average occupancy as peak capacity or sum unrelated workloads as query latency. Resident state and concurrent temporaries may need separate constraints. -A DP retaining only the cheapest local candidate per group can lose the joint optimum. DP with complete sharing context is possible, but its states can become large. MILP jointly selects discrete configurations; local caching or DP remains useful for parts that genuinely decompose. +A DP retaining one cheapest candidate per group can discard sharing benefits. DP with full sharing context remains possible, but its state can grow combinatorially; MILP handles these discrete choices jointly. -## 7. Worked Example: P95 and P99 Quantiles - -Assume identical filtered input, grouping, and aligned windows: +## 6. Worked Example: Shared KLL for P95/P99 ```text Q1: quantile_over_time(0.95, stream[15m]), every 1m Q2: quantile_over_time(0.99, stream[15m]), every 1m - -G95: Quantile(q=0.95, input=X, window=15m) -G99: Quantile(q=0.99, input=X, window=15m) ``` -The result groups differ. Pass 1 adds exact and supported KLL candidates. Pass 2 discovers compatible shared builds: +The memo contains separate P95 and P99 result groups over the same eligible input. Pass 1 adds exact and KLL candidates. Pass 2 identifies one compatible shared KLL build with separate estimation nodes: ```text - KLL build(X, 15m, size=s) - / \ - estimate P95 estimate P99 - | | - Q1 Q2 + Shared KLL producer + / \ + extract P95 extract P99 + | | + Q1 Q2 ``` -Illustrative costs per common horizon, assuming all listed choices meet requirements: +Illustrative costs per common horizon, assuming all listed candidates satisfy requirements: | Choice | Build cost | Reader costs | Total | | --- | ---: | ---: | ---: | | Separate KLL builds | 8 + 8 | 1 + 1 | 18 | -| One shared KLL build | 8 | 1 + 1 | 10 | +| Shared KLL build | 8 | 1 + 1 | 10 | | Separate exact computations | 12 + 12 | included | 24 | -If a reader's target requires more capacity, choose a compatible larger shared build, supported private builds, or an exact candidate. Quantile level alone does not determine required KLL capacity; the accuracy model and target do. - -### 7.1 Extending the example to panes - -A supported candidate maintains 1m KLL panes at ingestion and merges the latest 15 at evaluation. Both readers can use the same panes and, if eligible, the same merge result. Their estimation remains distinct. Window composition is generated in Pass 2; Stage 2 determines its materializations and physical validity. - -For different window lengths, the chain must cover the longest selected reader, and each reads the appropriate panes. Other pane widths or hierarchies require explicit generated candidates and valid boundary handling. Window compatibility is independent of sketch compatibility. - -### 7.2 Example binary decisions - -With one producer role named `summary` and one compatible configuration `s`: +The shared choice selects both reader candidates and attachments but one producer configuration: ```text -x[G95, KLL-reader] = 1 -x[G99, KLL-reader] = 1 -y[Shared-KLL(X,W), s] = 1 +x[G95, KLL-reader] = x[G99, KLL-reader] = 1 +y[Shared-KLL, s] = 1 z[G95, KLL-reader, summary, Shared-KLL, s] = 1 z[G99, KLL-reader, summary, Shared-KLL, s] = 1 ``` -The objective charges the producer once and adds the two readers' incremental costs. Configuration `s` includes or consistently references materialization. A query-time rebuilt full-window KLL is distinct from an ingestion-maintained pane chain; do not assume a full 15m sketch can directly delete expired samples every minute. - -## 8. Decomposition and Search-Space Control - -Build a coupling graph across all candidates, including child dependencies, fusion, shared producer choices, state exposure, common raw sources, and plan-level charges. Solve independent components only if objectives and feasibility factor. A shared capacity constraint or workload-wide calibration can prevent independent scalar optimization; retain the joint model or combine sufficient resource frontiers. - -This component decomposition differs from master/subproblem decomposition such as LBBD. +If accuracy requires larger state, the solver compares compatible shared capacity, private states, and exact candidates. Quantile level alone does not determine capacity. -1. Intern logical identities and compatible producer signatures with verified equality. -2. Remove provably invalid algorithm/window/materialization combinations with reasons. -3. Apply Stage 3 dominance only with the same external sharing interface and no worse relevant obligations; standalone price is insufficient. -4. Enumerate supported pane widths, capacities, retention configurations, and assignments without building complete-plan products. -5. Use a validated independent-query baseline as an incumbent where available; keep its true resource and accuracy checks. -6. Set solver time/gap limits and report incomplete search honestly. -7. Track candidate-generation, coefficient-generation, solver, and validation costs separately. +For ingestion-time reuse, a supported implementation maintains 1m KLL panes and merges the latest 15 at evaluation. Both readers can share panes and an eligible merge result. A direct whole-window sketch cannot be assumed to delete expired samples. Different window lengths or pane widths require separate compatibility and coverage checks. -Representative parameter subsets or adaptive enrichment can reduce model size, but narrow the finite candidate space. A solver certificate applies only to the current inventory. Expanding that inventory requires solving the expanded problem; validation does not prove omitted configurations are irrelevant. +## 7. Search-Space Reduction and Validation -Start with a direct MILP where costs and checks can be encoded compactly. The current implementation uses a lower-bound model with lazy cost/validity refinement for difficult terms. That is an implementation strategy, not a mathematical requirement of MILP. LBBD is one possible formally justified master/subproblem design; an arbitrary validate-and-add-cuts loop is not automatically LBBD. +Reduce search without losing valid joint choices: -## 9. Plan Reconstruction and Verification +1. Intern duplicate identities and cache generation with complete relevant context. +2. Remove provably invalid candidates early; apply cost dominance at Stage 3 only when external sharing interfaces and resource obligations are preserved. +3. Split coupling components only when all possible sharing, dependencies, raw retention, and constraints separate. Global capacities or calibration can prevent independent optimization. +4. Use a validated baseline incumbent and solver time/gap limits. Restricted parameter inventories narrow the optimality claim; enrichment requires solving the expanded inventory. -1. Traverse query roots, selected candidates, role attachments, and producer dependencies. -2. Construct one physical DAG, respecting timing-specific copies and selected materializations. Deduplicate only compatible physical instances and reject cycles. -3. Check root coverage, dependency support, execution capabilities, and state availability. -4. Run the configured accuracy evaluator on each complete query result. -5. Run the configured latency evaluator under its timing and resource assumptions. -6. Run the full cost/resource evaluator, including retention, temporary memory, and shared charges. -7. If validation fails or cost was underestimated, refine and re-solve under a bounded budget, or return a validated baseline with an explicit reason. +Reconstruct selected candidates, producer attachments, and timing-specific copies into one `PhysicalASAPDAG`. Check dependencies and cycles, then run the configured accuracy, latency, and cost/resource evaluators. Return a validated plan, fallback, or explicit search failure status. -An invalid incumbent does not prove the entire workload infeasible. No plan found before timeout is also not an infeasibility proof. If a validated plan exists but the remaining model is exhausted, return that plan rather than reporting infeasibility. +A complete MILP does not require LBBD. Current code uses a lower-bound model with lazy refinement for difficult costs/checks. When refinement is needed, add only sound feasibility or cost cuts conditioned on every relevant decision. Validation of one plan alone does not prove a relaxed model optimal. Report the best validated cost, certified lower bound/gap, solver status, and stop reason. Timeout without a plan is not proof of infeasibility. -For a relaxed model, keep a certified solver lower bound `L` and best validated cost `U`. Use `(U-L)/U` for positive `U`, with explicit zero-cost handling and numerical tolerances. A feasible solver objective is not automatically a certified lower bound when the solve stops early. +## 8. Implementation and Evaluation -Feasibility cuts must exclude only proved-invalid combinations. Cost cuts must be conditioned on every decision affecting their charge, including materializations, reader attachments, and configurations. A cut based only on group candidates can be invalid if those candidates permit several physical realizations. Retain any feasible configuration excluded after evaluation and include its region in the optimality argument. Refine until the valid global bound closes the target gap or the budget expires. +Reuse current structures rather than create a parallel optimizer framework: -A direct exact MILP and a validated matching evaluator can certify the finite-space optimum. A relaxed MILP plus validation of one plan cannot do so without sound refinement/bounds. Report modeled objective, validated cost, solver gap, global validated bound/gap where available, stop reason, and fallback routing distinctly. - -## 10. Data Structures and Interfaces - -Reuse the current architecture rather than adding a parallel optimizer framework. The following names map the conceptual model to the code inspected on `origin/main` at `900800fd`: - -| Design concept | Current representation / boundary | +| Structure | Responsibility | | --- | --- | -| Logical identity | `MemoGroup { node, children, parents, queries, roots_of }` and `GroupId`. | -| Group candidate inventory | `Memo { groups, candidates }`. | -| Implementation candidate | `ImplementationCandidate { implements, node, children, origin, coupling_sites, budget_share, requires_exact_children, rollup_of, materializations, requires_sites }`. | -| Shared configuration | `CouplingSite { key, kind }`, `CouplingKind`, and rule-specific thresholds/reader checks. | -| Physical choice | MILP `Choice`, including candidate identity, materialization assignment, coefficients, and copy/scan context. | -| Per-node usage and costs | Stage 3 pricing, `ResourceUsage`, `PlanningModels`, `PlanSelectionSettings`, and derived choice coefficients. | -| Selected output | `PhysicalCandidate` / `PhysicalASAPDAG`, `Selection`, and search/MILP reports. | - -`SharedProducer`, `ProducerConfiguration`, and role attachments are formulation concepts. Existing sites and shared-node keys provide much of this representation. Introduce explicit records only where needed to express multiple suppliers, upstream producers, or ownership that existing structures cannot represent; do not duplicate identity or charge ownership in two catalogs. - -A conceptual extension, when required, is: +| `MemoGroup`, `Memo` | Logical identity, dependencies, roots, and per-group candidates. | +| `ImplementationCandidate` | Private nodes, fusion coverage, child groups, contracts, and reuse requirements. | +| `CouplingSite` and thresholds | Shared configurations and compatible readers. | +| MILP `Choice` | Physical/materialization assignments, consumer context, and coefficients. | +| `PhysicalASAPDAG`, `Selection`, reports | Selected output, validation, bounds, and search status. | -```text -ProducerConfiguration { - producer_identity, configuration_id - family_or_operator, input_and_update_signature - window_parameters, summary_parameters - materialization_assignment, retention_and_freshness_contract - required_inputs, compatible_reader_roles - resource_profile, accuracy_contract -} - -SelectedChoices { - group_candidates, producer_configurations, reader_attachments - reconstructed_PhysicalASAPDAG - modeled_objective, certified_bound, validated_cost - solver_status, validation_status, search_report -} -``` - -These are design sketches, not new public APIs or substitutes for `Selection`. +Shared-producer and role notation maps onto existing sites and shared-node identities; it does not imply a separate catalog already exists. Add explicit records only where the existing representation cannot express ownership or dependencies. -### Current module boundaries +Current main places memo/rules in `search/memo` and solver code in `search/milp`, with Stage 2 physical generation and Stage 3 pricing remaining authoritative. `Legacy` is the default; `MemoBruteForce` is the reference and `MemoMilp` the solver path. Early memo construction, broader cost-model support, EH/direct sliding algorithms, and arbitrary pane hierarchies are proposed extensions. -Current main has moved the earlier `pass/subexpr` implementation into: +### Implementation phases -| Responsibility | Module | +| Phase | Deliverable and acceptance criterion | | --- | --- | -| Group construction and memo data | `crates/plan-selection/src/search/memo/{mod,groups}.rs` | -| Pass 2 memo rules | `crates/plan-selection/src/search/memo/rules/{capability,windows,scans}.rs` | -| Reference memo search and construction | `crates/plan-selection/src/search/memo/brute_force.rs` | -| MILP model, coefficients, cuts, decoding, reporting | `crates/plan-selection/src/search/milp/{model,coefficients,cuts,decode,report}.rs` | -| Independent memo clusters | `crates/plan-selection/src/search/milp/clusters.rs` | -| Stage orchestration and legacy routing | `crates/plan-selection/src/search/legacy/stage1_to_stage3.rs` | -| Physical generation | `crates/physical-asap-optimization/src/lib.rs` | -| Local logical candidates | `crates/logical-asap-optimization/src/per_computation_candidates.rs` | +| 1. Memo generation | Move identity construction before Pass 1 and generate candidates directly into groups without losing semantics, reader obligations, or fusion. | +| 2. Sharing and physical choices | Reuse sites and materialization rules; represent producers/roles and shared charges consistently. | +| 3. Joint selection | Extend MILP coverage as required; match exhaustive and memo brute force under the same inventory/models. | +| 4. Validation and scaling | Validate reconstructed plans, prove safe reductions, and measure generation/model/solve costs under budgets. | -The PR's older base may not contain the refactored paths. See the inspected [memo code](https://github.com/ProjectASAP/ASAPPlanner/tree/900800fd8a7e57c35c84d365c209bd687407f146/crates/plan-selection/src/search/memo), [MILP code](https://github.com/ProjectASAP/ASAPPlanner/tree/900800fd8a7e57c35c84d365c209bd687407f146/crates/plan-selection/src/search/milp), and [current layering design](https://github.com/ProjectASAP/ASAPPlanner/blob/900800fd8a7e57c35c84d365c209bd687407f146/docs/design_docs/proposals/planner-layering.md). +### Evaluation -Current `brute_force::memo` receives `PerComputationCandidates`, builds groups, and adds Pass 2 candidates. Search modes are `Legacy` (default), `MemoBruteForce`, and `MemoMilp`. The memo MILP uses `good_lp` with HiGHS, clusters under supported conditions, decodes choices, and evaluates complete plans through Stage 3 with lazy refinement. Initial construction before Pass 1, arbitrary pane hierarchies, EH/direct sliding algorithms, and broader MILP cost-model support remain proposed extensions. +**Correctness:** test identity versus sharing, producer compatibility, exact-once cost ownership, active dependencies, fusion, multiple roles, zero-price resources, pane boundaries, legal materializations, end-to-end accuracy, and capacity/status handling. Validate any relaxed bounds and cuts against enumerated plans. -## 11. Implementation Plan +**Optimality:** tiny workloads must match exhaustive enumeration and memo brute force on minimum validated cost, with an explicit numerical/tie policy and the same candidate inventory. -These phases distinguish existing foundations from proposed changes. - -| Phase | Deliverable | Exit criterion | -| --- | --- | --- | -| 1. Early memo foundation | Separate logical identity construction from Pass 1 attachment; move initial construction after Stage 0. | Eligible identical expressions intern once without changing semantics or losing reader obligations. | -| 2. Direct group generation | Adapt existing per-computation generation to emit group candidates, including fusion. | Explicit dependencies and contracts; no complete-workload Cartesian product. | -| 3. Sharing representation | Reuse sites/thresholds and shared-node identity; add role records only where necessary. | P95/P99 share a compatible build without merging result groups or duplicating charges. | -| 4. Physical choices and costs | Reuse legal materializations and Stage 3 accounting; extend supported configurations explicitly. | Same pricing and eligibility as authoritative physical generation/evaluation. | -| 5. MILP coverage | Maintain current selection/refinement path; extend role/dependency/resource coverage where needed. | Matches exhaustive and memo brute force on the same finite inventory. | -| 6. Scaling | Proven independent clusters, safe Stage 3 dominance, solver budgets, optional configuration enrichment. | Reports generation/model/solve costs without storing every workload candidate. | -| 7. Validation and window extensions | Reconstruct/validate every selected plan; implement new algorithms only with runtime support and guarantees. | Plans meet declared constraints or return a validated fallback/explicit failure status. | - -## 12. Testing and Evaluation - -### Correctness - -- Eligible identical expressions share a group; P95 and P99 remain separate. -- Different accuracy obligations preserve the candidates needed by each reader. -- Compatible KLL readers share a build; incompatible inputs, grouping, filters, or windows do not share without an explicit valid conversion. -- Shared builds and shared merges are counted once at actual execution rates; reader-specific work is counted appropriately. -- Selected dependencies are active, unused groups/producers cannot activate, and fusion does not double-implement absorbed groups. -- Multiple producer roles, upstream producer dependencies, and cycles are handled correctly. -- Pane widths, phases, partial boundaries, offsets, and historical reads satisfy coverage; future EH candidates are tested only after supported contracts exist. -- Unsupported materializations are excluded; retained state obeys timing, freshness, and lifetime requirements. -- Tiny-instance MILP, memo brute force, and exhaustive search agree on minimum validated cost for the same inventory/models, with an explicit tie policy. -- Zero prices, capacities, numeric tolerances, timeout/no-incumbent cases, and fallback routes have correct status. -- For relaxed models, every tested plan's lower-bound price is no greater than full price, and generated cuts preserve all feasible cheaper configurations. - -### Performance - -Measure workload size against groups, candidates, producer/configuration identities, reader edges, variables, constraints, generation/solve/validation time, and peak planner memory. Compare independent-query planning with joint planning on modeled and measured resource savings. Vary query overlap, sketch sizes, pane widths, evaluation schedules, and materializations. - -Report time-limited incumbent quality and certified gaps, modeled versus validated cost, and planning overhead versus execution savings. Evaluate billing and resource-constrained deployments only through supported engines; do not label a legacy fallback as MILP coverage. - -## 13. Open Design Questions - -1. Which accuracy compositions admit compact constraints, and which need sound evaluator-driven refinement? -2. Which operators support pane merge, direct expiration, or EH with the required guarantees and boundary behavior? -3. What is the shared-instance identity across evaluation rounds, and which freshness/update policies are executable? -4. Should resource and latency models retain current assumptions or add validated concurrency/critical-path models? -5. How should finite capacities, pane widths, retention horizons, and adaptive inventory expansion be chosen? -6. Which incompatible windows, update rates, or groupings can be connected through an explicit conversion candidate? -7. How should billing objectives, cluster objectives, accuracy margins, and planning budgets be prioritized? -8. When do model size and refinement measurements justify a formal LBBD or another decomposition, rather than the direct model? - -## 14. Design Outcome - -The memo represents logical identities and per-group implementation candidates. ASAP-aware CSE exposes shared computations through producer identities, coupling sites, and reader dependencies without conflating different results. MILP selects a consistent physical DAG and attributes each charge to the work or state that incurs it. The selected plan remains subject to the existing accuracy, physical, latency, and cost evaluators; optimality claims require a complete finite-space model or a sound refinement proof. +**Scalability:** measure groups, candidates, configurations, attachments, variables/constraints, generation/solve/validation time, and peak planner memory. Compare joint versus independent planning savings, time-limited gaps, and modeled versus validated costs across query overlap, window configurations, and materializations. From 0f6b9e928a3886e3dd5dc5dac2304346a87cac6e Mon Sep 17 00:00:00 2001 From: zzylol <50204836+zzylol@users.noreply.github.com> Date: Fri, 23 Oct 2026 23:15:53 +0000 Subject: [PATCH 25/53] docs: call shared producer roles producer inputs --- .../proposals/joint-planning-search.md | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-) diff --git a/docs/design_docs/proposals/joint-planning-search.md b/docs/design_docs/proposals/joint-planning-search.md index 2aaec6aa9..8688df914 100644 --- a/docs/design_docs/proposals/joint-planning-search.md +++ b/docs/design_docs/proposals/joint-planning-search.md @@ -102,16 +102,16 @@ The base formulation below separates logical coverage, shared producers, and rea | `B`, `S_b` | Potential producer instances and their finite configurations. | | `I(g,c)` | Groups implemented by candidate `(g,c)`, including fusion. | | `R(g,c)` | Input groups required when `(g,c)` is selected. | -| `J(g,c)` | Shared-producer input roles required by `(g,c)`. | -| `A(g,c,j)` | Compatible producer/configuration pairs for role `j`. | +| `J(g,c)` | Producer inputs required by `(g,c)`. | +| `A(g,c,j)` | Compatible producer/configuration pairs for producer input `j`. | -A role is an input slot, such as “pane chain”; it is filled by one compatible producer/configuration. `R` references logical results, while `J` references separately represented shared state. Do not encode the same dependency twice. If a producer is an ordinary memo group, its dependency can instead use `R`. +A producer input is an input slot, such as “pane chain”; it is filled by one compatible producer/configuration. `R` references logical results, while `J` references separately represented shared state. Do not encode the same dependency twice. If a producer is an ordinary memo group, its dependency can instead use `R`. ```text a[g] ∈ {0,1} group g is required x[g,c] ∈ {0,1} select candidate c for g y[b,s] ∈ {0,1} enable producer b at configuration s -z[g,c,j,b,s] ∈ {0,1} attach role j of (g,c) to (b,s) +z[g,c,j,b,s] ∈ {0,1} connect producer input j of (g,c) to (b,s) ``` Configuration `s` can include summary capacity, pane width, retention, and materialization. Discrete configurations allow measured lookup-table costs without assuming a linear size/cost function. Materialization can instead use explicit legal `x[g,c,m]` variables; avoid conflicting representations. Several internal nodes can require separate assignment slots. @@ -141,14 +141,14 @@ Add reverse support constraints: an active group must be justified by a root, se ### 5.3 Shared producers and compatibility -For each required role: +For each required producer input: ```text Σ_{(b,s) ∈ A(g,c,j)} z[g,c,j,b,s] = x[g,c] for j ∈ J(g,c) z[g,c,j,b,s] ≤ y[b,s] ``` -Use one equality per role: a candidate needing a scan and a pane chain needs both inputs. Local execution is a separate candidate or explicit fallback choice. +Use one equality per producer input: a candidate needing a scan and a pane chain needs both inputs. Local execution is a separate candidate or explicit fallback choice. For one configurable producer instance: @@ -265,7 +265,7 @@ Reuse current structures rather than create a parallel optimizer framework: | MILP `Choice` | Physical/materialization assignments, consumer context, and coefficients. | | `PhysicalASAPDAG`, `Selection`, reports | Selected output, validation, bounds, and search status. | -Shared-producer and role notation maps onto existing sites and shared-node identities; it does not imply a separate catalog already exists. Add explicit records only where the existing representation cannot express ownership or dependencies. +Shared-producer and producer-input notation maps onto existing sites and shared-node identities; it does not imply a separate catalog already exists. Add explicit records only where the existing representation cannot express ownership or dependencies. Current main places memo/rules in `search/memo` and solver code in `search/milp`, with Stage 2 physical generation and Stage 3 pricing remaining authoritative. `Legacy` is the default; `MemoBruteForce` is the reference and `MemoMilp` the solver path. Early memo construction, broader cost-model support, EH/direct sliding algorithms, and arbitrary pane hierarchies are proposed extensions. @@ -274,13 +274,13 @@ Current main places memo/rules in `search/memo` and solver code in `search/milp` | Phase | Deliverable and acceptance criterion | | --- | --- | | 1. Memo generation | Move identity construction before Pass 1 and generate candidates directly into groups without losing semantics, reader obligations, or fusion. | -| 2. Sharing and physical choices | Reuse sites and materialization rules; represent producers/roles and shared charges consistently. | +| 2. Sharing and physical choices | Reuse sites and materialization rules; represent producers, producer inputs, and shared charges consistently. | | 3. Joint selection | Extend MILP coverage as required; match exhaustive and memo brute force under the same inventory/models. | | 4. Validation and scaling | Validate reconstructed plans, prove safe reductions, and measure generation/model/solve costs under budgets. | ### Evaluation -**Correctness:** test identity versus sharing, producer compatibility, exact-once cost ownership, active dependencies, fusion, multiple roles, zero-price resources, pane boundaries, legal materializations, end-to-end accuracy, and capacity/status handling. Validate any relaxed bounds and cuts against enumerated plans. +**Correctness:** test identity versus sharing, producer compatibility, exact-once cost ownership, active dependencies, fusion, multiple producer inputs, zero-price resources, pane boundaries, legal materializations, end-to-end accuracy, and capacity/status handling. Validate any relaxed bounds and cuts against enumerated plans. **Optimality:** tiny workloads must match exhaustive enumeration and memo brute force on minimum validated cost, with an explicit numerical/tie policy and the same candidate inventory. From 8acc8d1320b99c3e29f435e1e792557ea5a975a1 Mon Sep 17 00:00:00 2001 From: zzylol <50204836+zzylol@users.noreply.github.com> Date: Fri, 23 Oct 2026 23:16:29 +0000 Subject: [PATCH 26/53] docs: separate MILP variables and constraints --- .../proposals/joint-planning-search.md | 42 +++++++++++-------- 1 file changed, 25 insertions(+), 17 deletions(-) diff --git a/docs/design_docs/proposals/joint-planning-search.md b/docs/design_docs/proposals/joint-planning-search.md index 8688df914..e293b46d3 100644 --- a/docs/design_docs/proposals/joint-planning-search.md +++ b/docs/design_docs/proposals/joint-planning-search.md @@ -93,7 +93,9 @@ Accuracy is an end-to-end root requirement. Compose summary and window error onl The base formulation below separates logical coverage, shared producers, and reader attachments. Extend it with consumer-reach variables and context-dependent charges where necessary, as current code does. -### 5.1 Sets and variables +### 5.1 Variables + +#### Sets and notation | Symbol | Meaning | | --- | --- | @@ -107,6 +109,8 @@ The base formulation below separates logical coverage, shared producers, and rea A producer input is an input slot, such as “pane chain”; it is filled by one compatible producer/configuration. `R` references logical results, while `J` references separately represented shared state. Do not encode the same dependency twice. If a producer is an ordinary memo group, its dependency can instead use `R`. +#### Decision variables + ```text a[g] ∈ {0,1} group g is required x[g,c] ∈ {0,1} select candidate c for g @@ -116,7 +120,23 @@ z[g,c,j,b,s] ∈ {0,1} connect producer input j of (g,c) to (b,s) Configuration `s` can include summary capacity, pane width, retention, and materialization. Discrete configurations allow measured lookup-table costs without assuming a linear size/cost function. Materialization can instead use explicit legal `x[g,c,m]` variables; avoid conflicting representations. Several internal nodes can require separate assignment slots. -### 5.2 Query coverage and dependencies +#### Objective and cost attribution + +For fixed-context coefficients: + +```text +min Σ_{g,c} d[g,c] × x[g,c] + + Σ_{b,s} f[b,s] × y[b,s] + + Σ_{g,c,j,b,s} r[g,c,j,b,s] × z[g,c,j,b,s] +``` + +Here `d` is private incremental cost, `f` owns producer build/update/retention costs, and `r` is reader-specific work. All use the same horizon. Exclude shared costs from `d` to avoid duplicate charges; reuse of a merge result also needs explicit ownership. + +When execution rates depend on selected readers or ancestors, add consumer-reach/rate variables rather than using unconditional `f` or `r`. Reader recurrence, phase, and range-evaluation steps affect actual work; a pane-alignment gcd grid is not automatically an execution rate. Raw and pane retention depend on the longest required horizons. + +### 5.2 Constraints + +#### Query coverage and dependencies Let `Roots(q)` contain the group references needed for query `q`, including scalar roots with multiple references: @@ -139,7 +159,7 @@ a[h] ≥ x[g,c] for h ∈ R(g,c) Add reverse support constraints: an active group must be justified by a root, selected consuming candidate/producer, or fusion coverage. This prevents unused activation even at zero prices. Dependencies are conditional on selected candidates, not every possible child. Enforce acyclicity, exact-input requirements, and exposed-state requirements for reuse. -### 5.3 Shared producers and compatibility +#### Shared producers and compatibility For each required producer input: @@ -159,19 +179,7 @@ y[b,s] ≤ Σ selected attachments requiring (b,s) Represent coexisting configurations as separate instances or explicitly permit them. This is not a global restriction to one pane width. Attachments include downstream producers; encode producers' upstream dependencies too. Compatibility covers inputs, grouping, family, reduction, windows, capacity, freshness, and timing. -### 5.4 Objective and cost attribution - -For fixed-context coefficients: - -```text -min Σ_{g,c} d[g,c] × x[g,c] - + Σ_{b,s} f[b,s] × y[b,s] - + Σ_{g,c,j,b,s} r[g,c,j,b,s] × z[g,c,j,b,s] -``` - -Here `d` is private incremental cost, `f` owns producer build/update/retention costs, and `r` is reader-specific work. All use the same horizon. Exclude shared costs from `d` to avoid duplicate charges; reuse of a merge result also needs explicit ownership. - -When execution rates depend on selected readers or ancestors, add consumer-reach/rate variables rather than using unconditional `f` or `r`. Reader recurrence, phase, and range-evaluation steps affect actual work; a pane-alignment gcd grid is not automatically an execution rate. Raw and pane retention depend on the longest required horizons. +#### Auxiliary-variable constraints For binary `v = u AND w`, use: @@ -183,7 +191,7 @@ v ≥ u + w - 1 Use valid bounds for other products and maxima. A lower-bound-only maximum is exact only when minimization or other constraints make it tight. -### 5.5 Accuracy, validity, and budgets +#### Accuracy, validity, and budgets - **Accuracy:** filter incompatible configurations and enforce proven per-query composition bounds, feasible-combination tables, or sound refinement. - **Physical validity:** require supported operators and legal materialization assignments, including freshness, upstream timing, and state lifetime. From 598f61477160308ee0d990de8feb16fb3cfbb6d9 Mon Sep 17 00:00:00 2001 From: zzylol <50204836+zzylol@users.noreply.github.com> Date: Fri, 23 Oct 2026 23:24:54 +0000 Subject: [PATCH 27/53] docs: represent shared intermediate states as memo groups --- .../proposals/joint-planning-search.md | 119 ++++++++---------- 1 file changed, 53 insertions(+), 66 deletions(-) diff --git a/docs/design_docs/proposals/joint-planning-search.md b/docs/design_docs/proposals/joint-planning-search.md index e293b46d3..8b078cac9 100644 --- a/docs/design_docs/proposals/joint-planning-search.md +++ b/docs/design_docs/proposals/joint-planning-search.md @@ -16,13 +16,13 @@ The design uses a memo to represent candidates compactly and a mixed-integer lin Terminology follows [planner-layering.md](planner-layering.md): -- **Memo group:** One distinct logical sub-expression, preserving source-language semantics, inputs, filters, grouping, window semantics, and output contract. Proven-equivalent expressions can share an identity. Current code interns eligible identical expressions rather than discovering arbitrary equivalences. +- **Memo group:** One distinct computation and its output contract: either a query-result expression or a reusable intermediate state. Query-result identity preserves source-language semantics, inputs, filters, grouping, and windows. An intermediate-state group additionally specifies its state family/interface and the guarantees readers may rely on. Current code interns eligible identical expressions; explicitly representing shared states as groups is the proposed extension. - **Implementation candidate:** One way to implement a group over child groups, using exact operators, exact accumulator state, or ASAP primitives. Logical candidates record dependencies and accuracy contracts; physical choices additionally specify supported operators and materialization assignments. A fused candidate can implement several groups. -- **Shared producer:** A physical computation or state reusable by compatible candidates, such as a KLL build, pane chain, scan, or finer aggregation result. A producer includes its input dependencies and legal configurations. +- **Shared producer:** The selected implementation of a memo group read by multiple candidates, such as a KLL-state group, pane-chain group, scan group, or finer aggregation group. It uses the same group dependencies and candidate-selection mechanism as other computations, rather than a separate producer catalog. -Memo grouping avoids duplicate logical exploration. Conventional CSE exposes reuse of identical computations; ASAP-aware CSE additionally exposes shared intermediate state between different results. P95 and P99 remain separate result groups but can read one KLL. Sharing is represented between their candidates, not by merging their output semantics. One memo identity does not itself force runtime materialization. +Memo grouping avoids duplicate logical exploration. Conventional CSE exposes reuse of identical computations; ASAP-aware CSE additionally exposes shared intermediate state between different results. P95 and P99 remain separate result groups but can read one KLL. ASAP-aware CSE introduces or reuses an intermediate-state group and adds reader candidates depending on it; it does not merge different output semantics. One memo identity does not itself force runtime materialization. -In the layering document, a stage-level candidate is a DAG for the **whole workload**. Local `ImplementationCandidate` records compactly represent its component choices, avoiding construction of every complete DAG. Approximate implementations retain the original logical result as their target and carry an accuracy contract; they are not exactly equivalent expressions. +In the layering document, a stage-level candidate is a DAG for the **whole workload**. Local `ImplementationCandidate` records compactly represent its component choices, avoiding construction of every complete DAG. Approximate implementations retain the original logical result as their target and carry an accuracy contract; they are not exactly equivalent expressions. Likewise, intermediate-state candidates must satisfy their group contract; different capacities or merge layouts are not automatically interchangeable. Reader compatibility constrains which state candidate can serve each reader. ## 3. ASAPPlanner Optimization Workflow @@ -35,7 +35,7 @@ Stage 1 / Pass 1: per-computation candidates exact / exact accumulator / summary family and size / fusion ↓ Stage 1 / Pass 2: ASAP-aware CSE candidates - shared builds / window composition / rollup / top-k reuse / scans + shared-state groups + reader candidates / windows / rollup / scans ↓ Stage 2: legal physical implementations and materialization assignments ↓ @@ -44,7 +44,7 @@ Stage 3: joint MILP selection -> reconstruct and validate PhysicalASAPDAG Build initial identities after Stage 0. Intern expressions only when semantics and sharing legality match; verify equality after hashing. Preserve reader-specific requirements and keyless-node identity rules. Cache candidate generation by group and relevant context, including accuracy obligations. Current memo construction receives Pass 1 candidates; moving it earlier is a proposed change. -Pass 1 generates each computation's candidates without composing their workload-wide Cartesian product. Pass 2 matches producer signatures and adds compatible sharing candidates, including window composition. Stage 2 determines physical operators and materialization; Stage 3 is responsible for cost-based selection. Earlier stages prune provably invalid candidates with reasons. +Pass 1 generates each computation's candidates without composing their workload-wide Cartesian product. Pass 2 matches intermediate-state signatures, introduces or reuses their groups, and adds reader candidates referencing them, including window composition. Groups can therefore be added after initial memo construction. Reusable states needed by a single candidate can also be factored out; reuse does not require two readers in advance. Stage 2 determines physical operators and materialization; Stage 3 is responsible for cost-based selection. Earlier stages prune provably invalid candidates with reasons. ### Window and materialization choices @@ -91,7 +91,7 @@ Accuracy is an end-to-end root requirement. Compose summary and window error onl ## 5. MILP Formulation -The base formulation below separates logical coverage, shared producers, and reader attachments. Extend it with consumer-reach variables and context-dependent charges where necessary, as current code does. +Query results and shared intermediate states use the same group/candidate model. There are no separate producer-selection or producer-input variables. Extend the base formulation with consumer-reach, physical-copy, and lifetime variables when execution costs depend on the selected context. ### 5.1 Variables @@ -99,40 +99,34 @@ The base formulation below separates logical coverage, shared producers, and rea | Symbol | Meaning | | --- | --- | -| `Q`, `G` | Queries and memo groups. | -| `C_g` | Legal physical candidates owned by group `g`, including local materialization assignments. | -| `B`, `S_b` | Potential producer instances and their finite configurations. | +| `Q`, `G` | Queries and memo groups, including intermediate-state groups. | +| `C_g` | Legal physical candidates for group `g`, including state configurations and local materialization assignments. | | `I(g,c)` | Groups implemented by candidate `(g,c)`, including fusion. | -| `R(g,c)` | Input groups required when `(g,c)` is selected. | -| `J(g,c)` | Producer inputs required by `(g,c)`. | -| `A(g,c,j)` | Compatible producer/configuration pairs for producer input `j`. | +| `R(g,c)` | Input groups required when `(g,c)` is selected, whether they produce results or intermediate state. | +| `K(g,c,h)` | Candidates of input group `h` compatible with reader candidate `(g,c)`. | -A producer input is an input slot, such as “pane chain”; it is filled by one compatible producer/configuration. `R` references logical results, while `J` references separately represented shared state. Do not encode the same dependency twice. If a producer is an ordinary memo group, its dependency can instead use `R`. +For example, a P95 estimator depends on a KLL-state group through `R`; `K` identifies that state's candidates with compatible family, window, capacity, and timing. Several required input groups create several dependencies, all of which must be satisfied. #### Decision variables ```text -a[g] ∈ {0,1} group g is required -x[g,c] ∈ {0,1} select candidate c for g -y[b,s] ∈ {0,1} enable producer b at configuration s -z[g,c,j,b,s] ∈ {0,1} connect producer input j of (g,c) to (b,s) +a[g] ∈ {0,1} group g is required +x[g,c] ∈ {0,1} select candidate c for group g ``` -Configuration `s` can include summary capacity, pane width, retention, and materialization. Discrete configurations allow measured lookup-table costs without assuming a linear size/cost function. Materialization can instead use explicit legal `x[g,c,m]` variables; avoid conflicting representations. Several internal nodes can require separate assignment slots. +Summary capacity, pane width, retention, and materialization are candidate configurations, not a second set of producer decisions. Discrete configurations permit lookup-table costs without assuming a linear size/cost function. Materialization can instead use explicit legal assignment variables; do not maintain two representations that can disagree. Several internal nodes can require separate assignment slots. #### Objective and cost attribution -For fixed-context coefficients: +For fixed-context costs: ```text -min Σ_{g,c} d[g,c] × x[g,c] - + Σ_{b,s} f[b,s] × y[b,s] - + Σ_{g,c,j,b,s} r[g,c,j,b,s] × z[g,c,j,b,s] +min Σ_{g,c} cost[g,c] × x[g,c] + plan-level charges ``` -Here `d` is private incremental cost, `f` owns producer build/update/retention costs, and `r` is reader-specific work. All use the same horizon. Exclude shared costs from `d` to avoid duplicate charges; reuse of a merge result also needs explicit ownership. +`cost[g,c]` owns only the candidate's private work/state, down to its input-group boundaries. A KLL-state candidate owns its build and retention; an estimator candidate owns extraction. A shared merge has its own group if readers reuse its result. Never include an input group's costs again in each parent. -When execution rates depend on selected readers or ancestors, add consumer-reach/rate variables rather than using unconditional `f` or `r`. Reader recurrence, phase, and range-evaluation steps affect actual work; a pane-alignment gcd grid is not automatically an execution rate. Raw and pane retention depend on the longest required horizons. +Selecting one state candidate does not by itself mean one execution. Materialization and consumer context determine whether it runs once per evaluation, per reader, or at ingestion. Add reach/rate and physical-copy variables for these cases; charge each actual instance correctly. Reader recurrence, phase, and range-evaluation steps affect work; a pane-alignment gcd grid is not automatically an execution rate. Raw retention and other non-computation charges remain resource/accounting variables rather than artificial computation groups. ### 5.2 Constraints @@ -145,39 +139,31 @@ a[g] = 1 for g ∈ Roots(q) Σ_{c ∈ C_g} x[g,c] = a[g] without fusion ``` -With fused candidates, use the general coverage constraint: +With fused candidates, use: ```text Σ_{(h,c): g ∈ I(h,c)} x[h,c] = a[g] ``` -Every required group is implemented once, possibly by fusion; absorbed groups do not also select standalone implementations. Selected candidates require their inputs: +Every required group is implemented once, possibly by fusion; absorbed groups do not also select standalone implementations. Every selected candidate requires its inputs, including shared states: ```text a[h] ≥ x[g,c] for h ∈ R(g,c) ``` -Add reverse support constraints: an active group must be justified by a root, selected consuming candidate/producer, or fusion coverage. This prevents unused activation even at zero prices. Dependencies are conditional on selected candidates, not every possible child. Enforce acyclicity, exact-input requirements, and exposed-state requirements for reuse. +Add reverse support constraints: an active group must be justified by a root, selected consumer, or fusion coverage. This prevents unused activation even at zero prices. Dependencies are conditional on selected candidates, not every possible child. Enforce acyclicity and reuse requirements such as exact inputs or exposed state. -#### Shared producers and compatibility +#### Reader/state compatibility -For each required producer input: +Requiring an input group is insufficient if only some of its candidates can serve the reader. For each input group `h`: ```text -Σ_{(b,s) ∈ A(g,c,j)} z[g,c,j,b,s] = x[g,c] for j ∈ J(g,c) -z[g,c,j,b,s] ≤ y[b,s] +x[g,c] ≤ Σ_{d ∈ K(g,c,h)} x[h,d] for h ∈ R(g,c) ``` -Use one equality per producer input: a candidate needing a scan and a pane chain needs both inputs. Local execution is a separate candidate or explicit fallback choice. +This form assumes nonfused input candidates. If another candidate implements `h` by fusion, use its compatible implementing choices in the sum instead. Compatibility covers inputs, grouping, family, reduction, windows, capacity, freshness, and timing. If several readers select the same state group, its chosen candidate must satisfy all of them. -For one configurable producer instance: - -```text -Σ_{s ∈ S_b} y[b,s] ≤ 1 -y[b,s] ≤ Σ selected attachments requiring (b,s) -``` - -Represent coexisting configurations as separate instances or explicitly permit them. This is not a global restriction to one pane width. Attachments include downstream producers; encode producers' upstream dependencies too. Compatibility covers inputs, grouping, family, reduction, windows, capacity, freshness, and timing. +Different pane widths can coexist as distinct intermediate groups. A group-level configuration choice must not become a global one-pane-width restriction. Retain private computation through private-state candidates/instances or the existing per-reader-copy/materialization mechanism; neither interning nor CSE should force a shared execution. Shared-instance scope is explicit where readers cannot use one configuration or lifetime. #### Auxiliary-variable constraints @@ -193,22 +179,21 @@ Use valid bounds for other products and maxima. A lower-bound-only maximum is ex #### Accuracy, validity, and budgets -- **Accuracy:** filter incompatible configurations and enforce proven per-query composition bounds, feasible-combination tables, or sound refinement. +- **Accuracy:** preserve state contracts and enforce proven per-query composition bounds, feasible-combination tables, or sound refinement. - **Physical validity:** require supported operators and legal materialization assignments, including freshness, upstream timing, and state lifetime. - **Window correctness:** require alignment, coverage without omission/double counting, retention, and algorithm-specific guarantees. - **Resource limits:** constrain requested CPU, memory, disk, and other resources under the configured concurrency model. - **Latency:** enforce configured query bounds and applicable recurrence checks. -For an additive resource `k`: +For an additive resource `k` with fixed-context usage: ```text -Σ u_x[k,g,c] × x[g,c] + Σ u_y[k,b,s] × y[b,s] - + Σ u_z[k,g,c,j,b,s] × z[g,c,j,b,s] ≤ U[k] +Σ_{g,c} usage[k,g,c] × x[g,c] + plan-level usage[k] ≤ capacity[k] ``` -Do not use average occupancy as peak capacity or sum unrelated workloads as query latency. Resident state and concurrent temporaries may need separate constraints. +Extend this for context-dependent rates and lifetimes. Do not use average occupancy as peak capacity or sum unrelated workloads as query latency. Resident state and concurrent temporaries may need separate constraints. -A DP retaining one cheapest candidate per group can discard sharing benefits. DP with full sharing context remains possible, but its state can grow combinatorially; MILP handles these discrete choices jointly. +Representing sharing as group dependencies simplifies the graph but does not make costs independent. DP with complete sharing context remains possible; retaining one cheapest candidate per group can still discard the joint optimum. ## 6. Worked Example: Shared KLL for P95/P99 @@ -217,14 +202,14 @@ Q1: quantile_over_time(0.95, stream[15m]), every 1m Q2: quantile_over_time(0.99, stream[15m]), every 1m ``` -The memo contains separate P95 and P99 result groups over the same eligible input. Pass 1 adds exact and KLL candidates. Pass 2 identifies one compatible shared KLL build with separate estimation nodes: +The memo contains separate P95 and P99 result groups over the same eligible input. Pass 1 adds exact and KLL candidates. Pass 2 introduces a KLL-state group and reader candidates with ordinary group dependencies: ```text - Shared KLL producer - / \ - extract P95 extract P99 - | | - Q1 Q2 +G_input ──→ G_KLL: build KLL state + / \ + G95: estimate P95 G99: estimate P99 + | | + Q1 Q2 ``` Illustrative costs per common horizon, assuming all listed candidates satisfy requirements: @@ -235,16 +220,18 @@ Illustrative costs per common horizon, assuming all listed candidates satisfy re | Shared KLL build | 8 | 1 + 1 | 10 | | Separate exact computations | 12 + 12 | included | 24 | -The shared choice selects both reader candidates and attachments but one producer configuration: +The shared choice selects both reader candidates and one KLL-state candidate: ```text +R(G95, KLL-reader) = {G_KLL} +R(G99, KLL-reader) = {G_KLL} +R(G_KLL, build-s) = {G_input} + x[G95, KLL-reader] = x[G99, KLL-reader] = 1 -y[Shared-KLL, s] = 1 -z[G95, KLL-reader, summary, Shared-KLL, s] = 1 -z[G99, KLL-reader, summary, Shared-KLL, s] = 1 +x[G_KLL, build-s] = 1 ``` -If accuracy requires larger state, the solver compares compatible shared capacity, private states, and exact candidates. Quantile level alone does not determine capacity. +The build cost is owned by `G_KLL` and appears once for a shared execution; each result group owns its estimation cost. If accuracy requires larger state, the solver compares compatible shared capacity, private states, and exact candidates. Quantile level alone does not determine capacity. For ingestion-time reuse, a supported implementation maintains 1m KLL panes and merges the latest 15 at evaluation. Both readers can share panes and an eligible merge result. A direct whole-window sketch cannot be assumed to delete expired samples. Different window lengths or pane widths require separate compatibility and coverage checks. @@ -257,7 +244,7 @@ Reduce search without losing valid joint choices: 3. Split coupling components only when all possible sharing, dependencies, raw retention, and constraints separate. Global capacities or calibration can prevent independent optimization. 4. Use a validated baseline incumbent and solver time/gap limits. Restricted parameter inventories narrow the optimality claim; enrichment requires solving the expanded inventory. -Reconstruct selected candidates, producer attachments, and timing-specific copies into one `PhysicalASAPDAG`. Check dependencies and cycles, then run the configured accuracy, latency, and cost/resource evaluators. Return a validated plan, fallback, or explicit search failure status. +Reconstruct selected candidates, their group dependencies, and timing-specific copies into one `PhysicalASAPDAG`. Check dependencies and cycles, then run the configured accuracy, latency, and cost/resource evaluators. Return a validated plan, fallback, or explicit search failure status. A complete MILP does not require LBBD. Current code uses a lower-bound model with lazy refinement for difficult costs/checks. When refinement is needed, add only sound feasibility or cost cuts conditioned on every relevant decision. Validation of one plan alone does not prove a relaxed model optimal. Report the best validated cost, certified lower bound/gap, solver status, and stop reason. Timeout without a plan is not proof of infeasibility. @@ -267,29 +254,29 @@ Reuse current structures rather than create a parallel optimizer framework: | Structure | Responsibility | | --- | --- | -| `MemoGroup`, `Memo` | Logical identity, dependencies, roots, and per-group candidates. | +| `MemoGroup`, `Memo` | Result/state contracts, dependencies, roots, and per-group candidates. | | `ImplementationCandidate` | Private nodes, fusion coverage, child groups, contracts, and reuse requirements. | -| `CouplingSite` and thresholds | Shared configurations and compatible readers. | +| `CouplingSite` and thresholds | Existing shared-configuration representation; migrate state choices into intermediate-group candidates, retaining genuinely cross-group constraints. | | MILP `Choice` | Physical/materialization assignments, consumer context, and coefficients. | | `PhysicalASAPDAG`, `Selection`, reports | Selected output, validation, bounds, and search status. | -Shared-producer and producer-input notation maps onto existing sites and shared-node identities; it does not imply a separate catalog already exists. Add explicit records only where the existing representation cannot express ownership or dependencies. +This is an extension of the current representation: shared builds are primarily coordinated through coupling sites today. Introduce intermediate-state group contracts and reader compatibility, then migrate corresponding state choices and charge ownership into group candidates. This requires changes to generation, selection, decoding, and pricing; it is not merely renaming coupling sites. Raw retention and global capacities remain accounting constraints. -Current main places memo/rules in `search/memo` and solver code in `search/milp`, with Stage 2 physical generation and Stage 3 pricing remaining authoritative. `Legacy` is the default; `MemoBruteForce` is the reference and `MemoMilp` the solver path. Early memo construction, broader cost-model support, EH/direct sliding algorithms, and arbitrary pane hierarchies are proposed extensions. +Current main places memo/rules in `search/memo` and solver code in `search/milp`, with Stage 2 physical generation and Stage 3 pricing remaining authoritative. `Legacy` is the default; `MemoBruteForce` is the reference and `MemoMilp` the solver path. Early memo construction, unified shared-state groups, broader cost-model support, EH/direct sliding algorithms, and arbitrary pane hierarchies are proposed extensions. ### Implementation phases | Phase | Deliverable and acceptance criterion | | --- | --- | | 1. Memo generation | Move identity construction before Pass 1 and generate candidates directly into groups without losing semantics, reader obligations, or fusion. | -| 2. Sharing and physical choices | Reuse sites and materialization rules; represent producers, producer inputs, and shared charges consistently. | +| 2. Shared-state groups | Extend group contracts; generate intermediate groups and reader candidates with compatible dependencies, legal materializations, and exact-once charge ownership. | | 3. Joint selection | Extend MILP coverage as required; match exhaustive and memo brute force under the same inventory/models. | | 4. Validation and scaling | Validate reconstructed plans, prove safe reductions, and measure generation/model/solve costs under budgets. | ### Evaluation -**Correctness:** test identity versus sharing, producer compatibility, exact-once cost ownership, active dependencies, fusion, multiple producer inputs, zero-price resources, pane boundaries, legal materializations, end-to-end accuracy, and capacity/status handling. Validate any relaxed bounds and cuts against enumerated plans. +**Correctness:** test identity versus sharing, producer compatibility, exact-once cost ownership, active dependencies, fusion, multiple input groups, shared/private instances, zero-price resources, pane boundaries, legal materializations, end-to-end accuracy, and capacity/status handling. Validate any relaxed bounds and cuts against enumerated plans. **Optimality:** tiny workloads must match exhaustive enumeration and memo brute force on minimum validated cost, with an explicit numerical/tie policy and the same candidate inventory. -**Scalability:** measure groups, candidates, configurations, attachments, variables/constraints, generation/solve/validation time, and peak planner memory. Compare joint versus independent planning savings, time-limited gaps, and modeled versus validated costs across query overlap, window configurations, and materializations. +**Scalability:** measure groups, candidates, state configurations, dependency edges, variables/constraints, generation/solve/validation time, and peak planner memory. Compare joint versus independent planning savings, time-limited gaps, and modeled versus validated costs across query overlap, window configurations, and materializations. From 177ff8fea9a7a3f0793daa69cd7e1f91a3c4861b Mon Sep 17 00:00:00 2001 From: zzylol <50204836+zzylol@users.noreply.github.com> Date: Fri, 23 Oct 2026 23:25:42 +0000 Subject: [PATCH 28/53] docs: explain traditional memo logical and physical rule expansion --- .../proposals/joint-planning-search.md | 78 ++++++++++++++++--- 1 file changed, 68 insertions(+), 10 deletions(-) diff --git a/docs/design_docs/proposals/joint-planning-search.md b/docs/design_docs/proposals/joint-planning-search.md index 8b078cac9..f9b91539f 100644 --- a/docs/design_docs/proposals/joint-planning-search.md +++ b/docs/design_docs/proposals/joint-planning-search.md @@ -4,7 +4,65 @@ **Scope:** Multi-query optimization across exact operators, ASAP primitives, window implementations, sharing, and materialization. -## 1. Motivation and Challenges +## 1. Traditional Memo Groups and Rule Expansion + +A traditional memo, as used by Cascades-style optimizers, compactly represents equivalent query expressions. A **memo group** contains expressions that produce the same logical result under the query semantics. Each expression references child groups instead of copying complete input plans. Different parent expressions can therefore reuse the same search work. + +### Logical rule expansion + +Consider an inner join with a filter on `A`: + +```sql +SELECT * +FROM A JOIN B ON A.id = B.id +WHERE A.x > 10; +``` + +Initially, the memo can contain: + +```text +G1 = { Scan(A) } +G2 = { Scan(B) } +G3 = { Join(G1, G2, A.id = B.id) } +G5 = { Filter(A.x > 10, G3) } query result +``` + +A filter-pushdown rule matches the filter above the inner join. Because the predicate depends only on `A` and pushing it down preserves semantics, it adds a filtered-input group and another expression to the existing result group: + +```text +G4 = { Filter(A.x > 10, G1) } + +G5 = { + Filter(A.x > 10, G3), + Join(G4, G2, A.id = B.id) +} +``` + +Both members of `G5` produce the same result. The rule expands the group rather than immediately replacing its existing expression. Newly introduced computations, such as the filtered `A`, can create new groups. Further rules can match newly inserted expressions; interning avoids repeating identical expressions. Exploration may be demand-driven and interleaved with costing rather than saturating every possible rewrite. + +### Physical implementation expansion + +Implementation rules turn logical expressions into physical execution candidates. For the join expression in `G5`, eligible candidates can include: + +```text +HashJoin(G4, G2) +MergeJoin(G4, G2) +NestedLoopJoin(G4, G2) +``` + +All implement the same logical join result, but impose different input requirements and costs. A merge join may require inputs ordered by the join key; a hash join does not require that order. Physical expressions continue to reference child groups, whose implementations are chosen separately. The precise storage of logical and physical expressions varies by optimizer. + +The optimizer typically searches and caches: + +```text +Best(group, required_physical_properties) +``` + +For each physical candidate, it derives child requirements, optimizes the child groups under those requirements, and combines their costs with the candidate's own cost. The cheapest implementation for unordered output may differ from the cheapest one for ordered output; sorting or other property-enforcing operators may be needed. Rules generate candidates; costing and valid bounds guide exploration and selection. + +Sharing a memo group shares optimization representation, not necessarily runtime computation. Explicit materialization/reuse decisions are still needed to execute a common producer once. This distinction becomes central to ASAPPlanner's cross-query optimization below. + +## 2. Motivation and Challenges ASAP primitives create opportunities to replace expensive exact computations and share summaries, panes, rollups, scans, or retained raw data. They also enlarge the optimization search space. Enumerating complete workload candidates multiplies implementation, sharing, window, and materialization choices. Optimizing queries independently avoids that enumeration but can miss joint savings. @@ -12,7 +70,7 @@ The challenge is cross-query dependence: a candidate's incremental cost changes The design uses a memo to represent candidates compactly and a mixed-integer linear program (MILP) to select a consistent physical plan. The goals are to avoid repeated candidate generation, expose ASAP-aware sharing, and minimize the configured objective subject to accuracy, latency, and resource constraints. Optimality applies to the generated finite candidate set and configured models, not every imaginable plan. New sketch/window algorithms and runtime scheduling are outside this proposal. -## 2. Memo-Based Candidate Representation +## 3. Memo-Based Candidate Representation Terminology follows [planner-layering.md](planner-layering.md): @@ -24,7 +82,7 @@ Memo grouping avoids duplicate logical exploration. Conventional CSE exposes reu In the layering document, a stage-level candidate is a DAG for the **whole workload**. Local `ImplementationCandidate` records compactly represent its component choices, avoiding construction of every complete DAG. Approximate implementations retain the original logical result as their target and carry an accuracy contract; they are not exactly equivalent expressions. Likewise, intermediate-state candidates must satisfy their group contract; different capacities or merge layouts are not automatically interchangeable. Reader compatibility constrains which state candidate can serve each reader. -## 3. ASAPPlanner Optimization Workflow +## 4. ASAPPlanner Optimization Workflow ```text Stage 0: frontends -> language-independent LogicalDAGs @@ -66,7 +124,7 @@ This table describes possible strategies. Current pane support does not imply ge Use existing `Materialization` options: `NotMaterialized`, `MatAtQuery(Memory)`, `MatAtIngestion(Memory)`, and `MatAtQueryKeptAcrossEvaluations(Memory)`. Assign them to eligible internal nodes, not necessarily the entire query. For example, maintain panes at ingestion and merge at query time. Generate only supported assignments: retained query-time state currently requires deployment capability and an eligible pane chain. Ingestion timing and upstream dependencies must satisfy Stage 2 rules. -## 4. Cost Model and Optimization Objective +## 5. Cost Model and Optimization Objective Attribute resource usage to physical nodes in the selected DAG, then apply the existing Stage 3 cost model: @@ -89,11 +147,11 @@ Charge shared work once per actual execution or maintained state, not once per q Accuracy is an end-to-end root requirement. Compose summary and window error only with valid contracts; locally admissible sketches do not prove a composed query accurate. Latency and resource limits must use the configured evaluator's timing and concurrency assumptions. Current latency accounting sums reached query-time work plus applicable compaction; this proposal does not silently replace it with a general parallel scheduler. Keep calibrated model inputs fixed during a solve. -## 5. MILP Formulation +## 6. MILP Formulation Query results and shared intermediate states use the same group/candidate model. There are no separate producer-selection or producer-input variables. Extend the base formulation with consumer-reach, physical-copy, and lifetime variables when execution costs depend on the selected context. -### 5.1 Variables +### 6.1 Variables #### Sets and notation @@ -128,7 +186,7 @@ min Σ_{g,c} cost[g,c] × x[g,c] + plan-level charges Selecting one state candidate does not by itself mean one execution. Materialization and consumer context determine whether it runs once per evaluation, per reader, or at ingestion. Add reach/rate and physical-copy variables for these cases; charge each actual instance correctly. Reader recurrence, phase, and range-evaluation steps affect work; a pane-alignment gcd grid is not automatically an execution rate. Raw retention and other non-computation charges remain resource/accounting variables rather than artificial computation groups. -### 5.2 Constraints +### 6.2 Constraints #### Query coverage and dependencies @@ -195,7 +253,7 @@ Extend this for context-dependent rates and lifetimes. Do not use average occupa Representing sharing as group dependencies simplifies the graph but does not make costs independent. DP with complete sharing context remains possible; retaining one cheapest candidate per group can still discard the joint optimum. -## 6. Worked Example: Shared KLL for P95/P99 +## 7. Worked Example: Shared KLL for P95/P99 ```text Q1: quantile_over_time(0.95, stream[15m]), every 1m @@ -235,7 +293,7 @@ The build cost is owned by `G_KLL` and appears once for a shared execution; each For ingestion-time reuse, a supported implementation maintains 1m KLL panes and merges the latest 15 at evaluation. Both readers can share panes and an eligible merge result. A direct whole-window sketch cannot be assumed to delete expired samples. Different window lengths or pane widths require separate compatibility and coverage checks. -## 7. Search-Space Reduction and Validation +## 8. Search-Space Reduction and Validation Reduce search without losing valid joint choices: @@ -248,7 +306,7 @@ Reconstruct selected candidates, their group dependencies, and timing-specific c A complete MILP does not require LBBD. Current code uses a lower-bound model with lazy refinement for difficult costs/checks. When refinement is needed, add only sound feasibility or cost cuts conditioned on every relevant decision. Validation of one plan alone does not prove a relaxed model optimal. Report the best validated cost, certified lower bound/gap, solver status, and stop reason. Timeout without a plan is not proof of infeasibility. -## 8. Implementation and Evaluation +## 9. Implementation and Evaluation Reuse current structures rather than create a parallel optimizer framework: From f2008ec392232fc1469f79b7661d0ffd01c191f4 Mon Sep 17 00:00:00 2001 From: zzylol <50204836+zzylol@users.noreply.github.com> Date: Fri, 23 Oct 2026 23:26:42 +0000 Subject: [PATCH 29/53] docs: lead memo design with motivation and challenges --- .../proposals/joint-planning-search.md | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-) diff --git a/docs/design_docs/proposals/joint-planning-search.md b/docs/design_docs/proposals/joint-planning-search.md index f9b91539f..93d7cbcec 100644 --- a/docs/design_docs/proposals/joint-planning-search.md +++ b/docs/design_docs/proposals/joint-planning-search.md @@ -4,7 +4,15 @@ **Scope:** Multi-query optimization across exact operators, ASAP primitives, window implementations, sharing, and materialization. -## 1. Traditional Memo Groups and Rule Expansion +## 1. Motivation and Challenges + +ASAP primitives create opportunities to replace expensive exact computations and share summaries, panes, rollups, scans, or retained raw data. They also enlarge the optimization search space. Enumerating complete workload candidates multiplies implementation, sharing, window, and materialization choices. Optimizing queries independently avoids that enumeration but can miss joint savings. + +The challenge is cross-query dependence: a candidate's incremental cost changes when another candidate enables a shared producer, requires a larger summary, or extends retention. Selecting the cheapest candidate for each group independently can therefore lose the workload optimum. + +The design uses a memo to represent candidates compactly and a mixed-integer linear program (MILP) to select a consistent physical plan. The goals are to avoid repeated candidate generation, expose ASAP-aware sharing, and minimize the configured objective subject to accuracy, latency, and resource constraints. Optimality applies to the generated finite candidate set and configured models, not every imaginable plan. New sketch/window algorithms and runtime scheduling are outside this proposal. + +## 2. Traditional Memo Groups and Rule Expansion A traditional memo, as used by Cascades-style optimizers, compactly represents equivalent query expressions. A **memo group** contains expressions that produce the same logical result under the query semantics. Each expression references child groups instead of copying complete input plans. Different parent expressions can therefore reuse the same search work. @@ -62,14 +70,6 @@ For each physical candidate, it derives child requirements, optimizes the child Sharing a memo group shares optimization representation, not necessarily runtime computation. Explicit materialization/reuse decisions are still needed to execute a common producer once. This distinction becomes central to ASAPPlanner's cross-query optimization below. -## 2. Motivation and Challenges - -ASAP primitives create opportunities to replace expensive exact computations and share summaries, panes, rollups, scans, or retained raw data. They also enlarge the optimization search space. Enumerating complete workload candidates multiplies implementation, sharing, window, and materialization choices. Optimizing queries independently avoids that enumeration but can miss joint savings. - -The challenge is cross-query dependence: a candidate's incremental cost changes when another candidate enables a shared producer, requires a larger summary, or extends retention. Selecting the cheapest candidate for each group independently can therefore lose the workload optimum. - -The design uses a memo to represent candidates compactly and a mixed-integer linear program (MILP) to select a consistent physical plan. The goals are to avoid repeated candidate generation, expose ASAP-aware sharing, and minimize the configured objective subject to accuracy, latency, and resource constraints. Optimality applies to the generated finite candidate set and configured models, not every imaginable plan. New sketch/window algorithms and runtime scheduling are outside this proposal. - ## 3. Memo-Based Candidate Representation Terminology follows [planner-layering.md](planner-layering.md): From 8418f53fad6d21f9c902852d2c772197a1db9ba9 Mon Sep 17 00:00:00 2001 From: zzylol <50204836+zzylol@users.noreply.github.com> Date: Fri, 23 Oct 2026 23:29:51 +0000 Subject: [PATCH 30/53] docs: explain ASAP memo groups through staged rule expansion --- .../proposals/joint-planning-search.md | 101 ++++++++++++------ 1 file changed, 70 insertions(+), 31 deletions(-) diff --git a/docs/design_docs/proposals/joint-planning-search.md b/docs/design_docs/proposals/joint-planning-search.md index 93d7cbcec..bbfd6d581 100644 --- a/docs/design_docs/proposals/joint-planning-search.md +++ b/docs/design_docs/proposals/joint-planning-search.md @@ -70,39 +70,78 @@ For each physical candidate, it derives child requirements, optimizes the child Sharing a memo group shares optimization representation, not necessarily runtime computation. Explicit materialization/reuse decisions are still needed to execute a common producer once. This distinction becomes central to ASAPPlanner's cross-query optimization below. -## 3. Memo-Based Candidate Representation +## 3. ASAPPlanner Memo Groups and Rule Expansion -Terminology follows [planner-layering.md](planner-layering.md): +ASAPPlanner follows the logical/physical stage responsibilities in [planner-layering.md](planner-layering.md), but must also represent approximate implementations and reuse between different results. -- **Memo group:** One distinct computation and its output contract: either a query-result expression or a reusable intermediate state. Query-result identity preserves source-language semantics, inputs, filters, grouping, and windows. An intermediate-state group additionally specifies its state family/interface and the guarantees readers may rely on. Current code interns eligible identical expressions; explicitly representing shared states as groups is the proposed extension. -- **Implementation candidate:** One way to implement a group over child groups, using exact operators, exact accumulator state, or ASAP primitives. Logical candidates record dependencies and accuracy contracts; physical choices additionally specify supported operators and materialization assignments. A fused candidate can implement several groups. -- **Shared producer:** The selected implementation of a memo group read by multiple candidates, such as a KLL-state group, pane-chain group, scan group, or finer aggregation group. It uses the same group dependencies and candidate-selection mechanism as other computations, rather than a separate producer catalog. +A **memo group** identifies a computation and its output contract: a query-result expression or a reusable intermediate state. A result group preserves input, filters, grouping, window semantics, and requested output. A state group additionally specifies its state family/interface and guarantees. **Implementation candidates** provide exact computation, accumulators, or summaries and reference input groups. Physical candidates add legal operators and materialization assignments. A **shared producer** is a selected group implementation read by several consumers, not a separately selected catalog entry. -Memo grouping avoids duplicate logical exploration. Conventional CSE exposes reuse of identical computations; ASAP-aware CSE additionally exposes shared intermediate state between different results. P95 and P99 remain separate result groups but can read one KLL. ASAP-aware CSE introduces or reuses an intermediate-state group and adds reader candidates depending on it; it does not merge different output semantics. One memo identity does not itself force runtime materialization. +Unlike exact expressions in a traditional equivalence group, approximate candidates target the same requested result under explicit accuracy obligations. State candidates must satisfy their group's contract, and readers can impose stricter compatibility requirements. One group identity does not force one runtime execution. A stage-level candidate remains a whole-workload DAG; local group candidates compactly represent its component choices. -In the layering document, a stage-level candidate is a DAG for the **whole workload**. Local `ImplementationCandidate` records compactly represent its component choices, avoiding construction of every complete DAG. Approximate implementations retain the original logical result as their target and carry an accuracy contract; they are not exactly equivalent expressions. Likewise, intermediate-state candidates must satisfy their group contract; different capacities or merge layouts are not automatically interchangeable. Reader compatibility constrains which state candidate can serve each reader. +The following example traces how the proposed memo expands: -## 4. ASAPPlanner Optimization Workflow +```text +Q1: quantile_over_time(0.95, X[15m]), every 1m +Q2: quantile_over_time(0.99, X[15m]), every 1m +``` + +### Stage 0 and initial memo: establish logical identity + +Frontends produce language-independent `LogicalDAGs`. Build groups for distinct logical sub-expressions and record query roots and reader requirements. Intern eligible identical inputs only after semantic equality and sharing legality are established; preserve keyless-node rules. P95 and P99 are different computations, so their result groups remain separate: + +```text +G_input: common eligible input computation X +G95: P95 over G_input in the last 15m root of Q1 +G99: P99 over G_input in the last 15m root of Q2 +``` + +Current memo construction receives Pass 1 candidates. Building initial identities before Pass 1 is the proposed workflow, allowing candidate generation to reuse group identity from the start. + +### Stage 1, Pass 1: add per-computation candidates + +Apply computation rules to each group independently. In this example, each group receives an exact candidate and supported KLL candidates: ```text -Stage 0: frontends -> language-independent LogicalDAGs - ↓ -Initial memo: logical identities, dependencies, query roots - ↓ -Stage 1 / Pass 1: per-computation candidates - exact / exact accumulator / summary family and size / fusion - ↓ -Stage 1 / Pass 2: ASAP-aware CSE candidates - shared-state groups + reader candidates / windows / rollup / scans - ↓ -Stage 2: legal physical implementations and materialization assignments - ↓ -Stage 3: joint MILP selection -> reconstruct and validate PhysicalASAPDAG +G95 = { ExactQuantile(0.95, G_input, 15m), PrivateKLLQuantile(0.95, G_input, 15m, s) } +G99 = { ExactQuantile(0.99, G_input, 15m), PrivateKLLQuantile(0.99, G_input, 15m, s) } ``` -Build initial identities after Stage 0. Intern expressions only when semantics and sharing legality match; verify equality after hashing. Preserve reader-specific requirements and keyless-node identity rules. Cache candidate generation by group and relevant context, including accuracy obligations. Current memo construction receives Pass 1 candidates; moving it earlier is a proposed change. +Here `s` ranges over generated capacities; each candidate records its accuracy contract and dependencies. Other computations can receive exact accumulator or fused candidates. Cache generation with relevant reader requirements and avoid composing the Cartesian product of complete workload plans. + +### Stage 1, Pass 2: add intermediate groups and sharing candidates + +ASAP-aware CSE recognizes that compatible KLL state supports both quantiles. Introduce or reuse a state group and add reader candidates: + +```text +G_KLL: KLL state over X[15m] + candidates: BuildKLL(G_input, window=15m, capacity=s) + +G95 += EstimateQuantile(0.95, G_KLL) +G99 += EstimateQuantile(0.99, G_KLL) +``` + +The result groups are not merged. Their new candidates depend on the same state group, whose selected capacity must satisfy both readers. Independent candidates remain available. Conventional CSE can similarly reuse identical computations; ASAP-aware rules also expose window composition, rollup, top-k reuse, and scan widening. + +For pane-based reuse, window-composition rules can add: + +```text +G_panes: chain of aligned 1m KLL panes over G_input, covering 15m +G_KLL += MergeLatest15(G_panes) +``` + +This is another way to produce compatible window state. Other generated pane widths or compositions require coverage, merge, and accuracy checks. Groups can be added throughout rule expansion; they are not fixed after Stage 0. + +### Stage 2: add physical and materialization candidates + +Generate supported physical operators and assignments for these candidates. For example, an eligible `G_panes` implementation can maintain panes at ingestion while merge and estimation execute at query time. Query-time construction and supported retention modes remain candidates where legal. + +Stage 2 checks deployment capabilities, ingestion eligibility, and upstream timing. It does not choose the cheapest candidate. Assignments can differ between internal nodes; sharing a logical state does not imply a shared physical execution under every assignment. + +### Stage 3: select a joint physical plan + +The MILP chooses candidates for both query roots and their required input groups, including any intermediate states. It may select exact/private computation, a shared whole-window build, or a legal pane-based candidate. It enforces reader/state compatibility, accuracy, latency, and resource constraints, then reconstructs and validates one `PhysicalASAPDAG`. -Pass 1 generates each computation's candidates without composing their workload-wide Cartesian product. Pass 2 matches intermediate-state signatures, introduces or reuses their groups, and adds reader candidates referencing them, including window composition. Groups can therefore be added after initial memo construction. Reusable states needed by a single candidate can also be factored out; reuse does not require two readers in advance. Stage 2 determines physical operators and materialization; Stage 3 is responsible for cost-based selection. Earlier stages prune provably invalid candidates with reasons. +Shared execution is charged once, with merge and estimation charges owned by their selected groups. A higher standalone cost can enable a cheaper joint plan, so earlier stages retain valid candidates and Stage 3 performs cost-based selection. ### Window and materialization choices @@ -124,7 +163,7 @@ This table describes possible strategies. Current pane support does not imply ge Use existing `Materialization` options: `NotMaterialized`, `MatAtQuery(Memory)`, `MatAtIngestion(Memory)`, and `MatAtQueryKeptAcrossEvaluations(Memory)`. Assign them to eligible internal nodes, not necessarily the entire query. For example, maintain panes at ingestion and merge at query time. Generate only supported assignments: retained query-time state currently requires deployment capability and an eligible pane chain. Ingestion timing and upstream dependencies must satisfy Stage 2 rules. -## 5. Cost Model and Optimization Objective +## 4. Cost Model and Optimization Objective Attribute resource usage to physical nodes in the selected DAG, then apply the existing Stage 3 cost model: @@ -147,11 +186,11 @@ Charge shared work once per actual execution or maintained state, not once per q Accuracy is an end-to-end root requirement. Compose summary and window error only with valid contracts; locally admissible sketches do not prove a composed query accurate. Latency and resource limits must use the configured evaluator's timing and concurrency assumptions. Current latency accounting sums reached query-time work plus applicable compaction; this proposal does not silently replace it with a general parallel scheduler. Keep calibrated model inputs fixed during a solve. -## 6. MILP Formulation +## 5. MILP Formulation Query results and shared intermediate states use the same group/candidate model. There are no separate producer-selection or producer-input variables. Extend the base formulation with consumer-reach, physical-copy, and lifetime variables when execution costs depend on the selected context. -### 6.1 Variables +### 5.1 Variables #### Sets and notation @@ -186,7 +225,7 @@ min Σ_{g,c} cost[g,c] × x[g,c] + plan-level charges Selecting one state candidate does not by itself mean one execution. Materialization and consumer context determine whether it runs once per evaluation, per reader, or at ingestion. Add reach/rate and physical-copy variables for these cases; charge each actual instance correctly. Reader recurrence, phase, and range-evaluation steps affect work; a pane-alignment gcd grid is not automatically an execution rate. Raw retention and other non-computation charges remain resource/accounting variables rather than artificial computation groups. -### 6.2 Constraints +### 5.2 Constraints #### Query coverage and dependencies @@ -253,14 +292,14 @@ Extend this for context-dependent rates and lifetimes. Do not use average occupa Representing sharing as group dependencies simplifies the graph but does not make costs independent. DP with complete sharing context remains possible; retaining one cheapest candidate per group can still discard the joint optimum. -## 7. Worked Example: Shared KLL for P95/P99 +## 6. Worked Example: Shared KLL for P95/P99 ```text Q1: quantile_over_time(0.95, stream[15m]), every 1m Q2: quantile_over_time(0.99, stream[15m]), every 1m ``` -The memo contains separate P95 and P99 result groups over the same eligible input. Pass 1 adds exact and KLL candidates. Pass 2 introduces a KLL-state group and reader candidates with ordinary group dependencies: +Using the groups introduced in Section 3, compare the cost of selecting shared versus private KLL candidates: ```text G_input ──→ G_KLL: build KLL state @@ -293,7 +332,7 @@ The build cost is owned by `G_KLL` and appears once for a shared execution; each For ingestion-time reuse, a supported implementation maintains 1m KLL panes and merges the latest 15 at evaluation. Both readers can share panes and an eligible merge result. A direct whole-window sketch cannot be assumed to delete expired samples. Different window lengths or pane widths require separate compatibility and coverage checks. -## 8. Search-Space Reduction and Validation +## 7. Search-Space Reduction and Validation Reduce search without losing valid joint choices: @@ -306,7 +345,7 @@ Reconstruct selected candidates, their group dependencies, and timing-specific c A complete MILP does not require LBBD. Current code uses a lower-bound model with lazy refinement for difficult costs/checks. When refinement is needed, add only sound feasibility or cost cuts conditioned on every relevant decision. Validation of one plan alone does not prove a relaxed model optimal. Report the best validated cost, certified lower bound/gap, solver status, and stop reason. Timeout without a plan is not proof of infeasibility. -## 9. Implementation and Evaluation +## 8. Implementation and Evaluation Reuse current structures rather than create a parallel optimizer framework: From daad3793df8ce7d50778cb23144ade91112f3007 Mon Sep 17 00:00:00 2001 From: zzylol <50204836+zzylol@users.noreply.github.com> Date: Fri, 23 Oct 2026 23:31:49 +0000 Subject: [PATCH 31/53] docs: call per-computation KLL candidates local --- .../proposals/joint-planning-search.md | 22 +++++++++---------- 1 file changed, 11 insertions(+), 11 deletions(-) diff --git a/docs/design_docs/proposals/joint-planning-search.md b/docs/design_docs/proposals/joint-planning-search.md index bbfd6d581..1c4529b18 100644 --- a/docs/design_docs/proposals/joint-planning-search.md +++ b/docs/design_docs/proposals/joint-planning-search.md @@ -102,8 +102,8 @@ Current memo construction receives Pass 1 candidates. Building initial identitie Apply computation rules to each group independently. In this example, each group receives an exact candidate and supported KLL candidates: ```text -G95 = { ExactQuantile(0.95, G_input, 15m), PrivateKLLQuantile(0.95, G_input, 15m, s) } -G99 = { ExactQuantile(0.99, G_input, 15m), PrivateKLLQuantile(0.99, G_input, 15m, s) } +G95 = { ExactQuantile(0.95, G_input, 15m), LocalKLLQuantile(0.95, G_input, 15m, s) } +G99 = { ExactQuantile(0.99, G_input, 15m), LocalKLLQuantile(0.99, G_input, 15m, s) } ``` Here `s` ranges over generated capacities; each candidate records its accuracy contract and dependencies. Other computations can receive exact accumulator or fused candidates. Cache generation with relevant reader requirements and avoid composing the Cartesian product of complete workload plans. @@ -139,7 +139,7 @@ Stage 2 checks deployment capabilities, ingestion eligibility, and upstream timi ### Stage 3: select a joint physical plan -The MILP chooses candidates for both query roots and their required input groups, including any intermediate states. It may select exact/private computation, a shared whole-window build, or a legal pane-based candidate. It enforces reader/state compatibility, accuracy, latency, and resource constraints, then reconstructs and validates one `PhysicalASAPDAG`. +The MILP chooses candidates for both query roots and their required input groups, including any intermediate states. It may select exact/local computation, a shared whole-window build, or a legal pane-based candidate. It enforces reader/state compatibility, accuracy, latency, and resource constraints, then reconstructs and validates one `PhysicalASAPDAG`. Shared execution is charged once, with merge and estimation charges owned by their selected groups. A higher standalone cost can enable a cheaper joint plan, so earlier stages retain valid candidates and Stage 3 performs cost-based selection. @@ -178,8 +178,8 @@ Current memo MILP supports usage-over-time pricing; unsupported models route to Give every charge one owner: - **Producer:** build/update, maintenance, and retained state. -- **Reader:** private merge, extraction, read, and finalization. -- **Other physical work:** private operators and temporary state. +- **Reader:** local merge, extraction, read, and finalization. +- **Other physical work:** local operators and temporary state. - **Shared accounting:** per-source raw retention and other plan-level charges. Charge shared work once per actual execution or maintained state, not once per query path. Reader-specific work remains separately charged. Use a common horizon or consistent per-second rates; distinguish memory/storage occupancy over time from peak capacity. Producer costs can depend on selected readers, evaluation schedules, upstream materialization, and retention. Model that context rather than assuming every producer has a constant cost. @@ -221,7 +221,7 @@ For fixed-context costs: min Σ_{g,c} cost[g,c] × x[g,c] + plan-level charges ``` -`cost[g,c]` owns only the candidate's private work/state, down to its input-group boundaries. A KLL-state candidate owns its build and retention; an estimator candidate owns extraction. A shared merge has its own group if readers reuse its result. Never include an input group's costs again in each parent. +`cost[g,c]` owns only the candidate's local work/state, down to its input-group boundaries. A KLL-state candidate owns its build and retention; an estimator candidate owns extraction. A shared merge has its own group if readers reuse its result. Never include an input group's costs again in each parent. Selecting one state candidate does not by itself mean one execution. Materialization and consumer context determine whether it runs once per evaluation, per reader, or at ingestion. Add reach/rate and physical-copy variables for these cases; charge each actual instance correctly. Reader recurrence, phase, and range-evaluation steps affect work; a pane-alignment gcd grid is not automatically an execution rate. Raw retention and other non-computation charges remain resource/accounting variables rather than artificial computation groups. @@ -260,7 +260,7 @@ x[g,c] ≤ Σ_{d ∈ K(g,c,h)} x[h,d] for h ∈ R(g,c) This form assumes nonfused input candidates. If another candidate implements `h` by fusion, use its compatible implementing choices in the sum instead. Compatibility covers inputs, grouping, family, reduction, windows, capacity, freshness, and timing. If several readers select the same state group, its chosen candidate must satisfy all of them. -Different pane widths can coexist as distinct intermediate groups. A group-level configuration choice must not become a global one-pane-width restriction. Retain private computation through private-state candidates/instances or the existing per-reader-copy/materialization mechanism; neither interning nor CSE should force a shared execution. Shared-instance scope is explicit where readers cannot use one configuration or lifetime. +Different pane widths can coexist as distinct intermediate groups. A group-level configuration choice must not become a global one-pane-width restriction. Retain local computation through local-state candidates/instances or the existing per-reader-copy/materialization mechanism; neither interning nor CSE should force a shared execution. Shared-instance scope is explicit where readers cannot use one configuration or lifetime. #### Auxiliary-variable constraints @@ -299,7 +299,7 @@ Q1: quantile_over_time(0.95, stream[15m]), every 1m Q2: quantile_over_time(0.99, stream[15m]), every 1m ``` -Using the groups introduced in Section 3, compare the cost of selecting shared versus private KLL candidates: +Using the groups introduced in Section 3, compare the cost of selecting shared versus local KLL candidates: ```text G_input ──→ G_KLL: build KLL state @@ -328,7 +328,7 @@ x[G95, KLL-reader] = x[G99, KLL-reader] = 1 x[G_KLL, build-s] = 1 ``` -The build cost is owned by `G_KLL` and appears once for a shared execution; each result group owns its estimation cost. If accuracy requires larger state, the solver compares compatible shared capacity, private states, and exact candidates. Quantile level alone does not determine capacity. +The build cost is owned by `G_KLL` and appears once for a shared execution; each result group owns its estimation cost. If accuracy requires larger state, the solver compares compatible shared capacity, local states, and exact candidates. Quantile level alone does not determine capacity. For ingestion-time reuse, a supported implementation maintains 1m KLL panes and merges the latest 15 at evaluation. Both readers can share panes and an eligible merge result. A direct whole-window sketch cannot be assumed to delete expired samples. Different window lengths or pane widths require separate compatibility and coverage checks. @@ -352,7 +352,7 @@ Reuse current structures rather than create a parallel optimizer framework: | Structure | Responsibility | | --- | --- | | `MemoGroup`, `Memo` | Result/state contracts, dependencies, roots, and per-group candidates. | -| `ImplementationCandidate` | Private nodes, fusion coverage, child groups, contracts, and reuse requirements. | +| `ImplementationCandidate` | Local nodes, fusion coverage, child groups, contracts, and reuse requirements. | | `CouplingSite` and thresholds | Existing shared-configuration representation; migrate state choices into intermediate-group candidates, retaining genuinely cross-group constraints. | | MILP `Choice` | Physical/materialization assignments, consumer context, and coefficients. | | `PhysicalASAPDAG`, `Selection`, reports | Selected output, validation, bounds, and search status. | @@ -372,7 +372,7 @@ Current main places memo/rules in `search/memo` and solver code in `search/milp` ### Evaluation -**Correctness:** test identity versus sharing, producer compatibility, exact-once cost ownership, active dependencies, fusion, multiple input groups, shared/private instances, zero-price resources, pane boundaries, legal materializations, end-to-end accuracy, and capacity/status handling. Validate any relaxed bounds and cuts against enumerated plans. +**Correctness:** test identity versus sharing, producer compatibility, exact-once cost ownership, active dependencies, fusion, multiple input groups, shared/local instances, zero-price resources, pane boundaries, legal materializations, end-to-end accuracy, and capacity/status handling. Validate any relaxed bounds and cuts against enumerated plans. **Optimality:** tiny workloads must match exhaustive enumeration and memo brute force on minimum validated cost, with an explicit numerical/tie policy and the same candidate inventory. From 161dfef188956b04c05d283c5f66730ac08778fd Mon Sep 17 00:00:00 2001 From: zzylol <50204836+zzylol@users.noreply.github.com> Date: Fri, 23 Oct 2026 23:32:52 +0000 Subject: [PATCH 32/53] docs: extract window and materialization choices design --- docs/design_docs/proposals/README.md | 1 + .../proposals/joint-planning-search.md | 20 +------------- .../window-materialization-choices.md | 26 +++++++++++++++++++ 3 files changed, 28 insertions(+), 19 deletions(-) create mode 100644 docs/design_docs/proposals/window-materialization-choices.md diff --git a/docs/design_docs/proposals/README.md b/docs/design_docs/proposals/README.md index 913cd6ff6..eeadf4b4a 100644 --- a/docs/design_docs/proposals/README.md +++ b/docs/design_docs/proposals/README.md @@ -12,4 +12,5 @@ extensions. A design document is not a promise of downstream runtime support. - [Decoupling operators from scalar expressions](decoupling_op_and_expr.md) - [ASAPPlanner layering](planner-layering.md) - [Memo-based joint optimization with MILP](joint-planning-search.md) +- [Window and materialization choices](window-materialization-choices.md) - [Schema and physical data for ASAP primitives](asap-primitive-schema.md) diff --git a/docs/design_docs/proposals/joint-planning-search.md b/docs/design_docs/proposals/joint-planning-search.md index 1c4529b18..8625ba3ab 100644 --- a/docs/design_docs/proposals/joint-planning-search.md +++ b/docs/design_docs/proposals/joint-planning-search.md @@ -143,25 +143,7 @@ The MILP chooses candidates for both query roots and their required input groups Shared execution is charged once, with merge and estimation charges owned by their selected groups. A higher standalone cost can enable a cheaper joint plan, so earlier stages retain valid candidates and Stage 3 performs cost-based selection. -### Window and materialization choices - -Keep these dimensions separate: - -- **Semantics:** requested window width, offset, evaluation timestamps, and alignment. -- **Implementation:** raw recomputation, supported direct sliding state, fixed-width panes, or an EH algorithm. -- **Materialization:** when each internal state is computed and how long it is retained. - -A pane is an aligned, non-overlapping fixed-width interval and its aggregate/summary state. An EH bucket represents a portion of the stream under an exponential histogram algorithm, with algorithm-specific merge, expiration, and error rules. - -| Implementation | Ingestion, when maintained there | Evaluation | -| --- | --- | --- | -| Direct sliding-window state | Add data and remove expired contributions. | Read state and compute result. | -| Pane-based | Update panes and evict expired panes. | Merge required panes and compute result. | -| EH-based | Update, merge, and evict buckets. | Estimate the window result. | - -This table describes possible strategies. Current pane support does not imply general direct sliding-state or EH support. Offer only executable candidates. Multiple pane widths can coexist; alignment, coverage, mergeability, retention, and error contracts determine legality. A 15m window evaluated every 1m can use 1m panes; complete 5m panes alone generally cannot cover every endpoint. - -Use existing `Materialization` options: `NotMaterialized`, `MatAtQuery(Memory)`, `MatAtIngestion(Memory)`, and `MatAtQueryKeptAcrossEvaluations(Memory)`. Assign them to eligible internal nodes, not necessarily the entire query. For example, maintain panes at ingestion and merge at query time. Generate only supported assignments: retained query-time state currently requires deployment capability and an eligible pane chain. Ingestion timing and upstream dependencies must satisfy Stage 2 rules. +Window semantics, implementation candidates, and materialization are defined separately in [Window and Materialization Choices](window-materialization-choices.md). That design also describes pane/EH terminology, supported configurations, and materialization eligibility. ## 4. Cost Model and Optimization Objective diff --git a/docs/design_docs/proposals/window-materialization-choices.md b/docs/design_docs/proposals/window-materialization-choices.md new file mode 100644 index 000000000..da284fb73 --- /dev/null +++ b/docs/design_docs/proposals/window-materialization-choices.md @@ -0,0 +1,26 @@ +# Window and Materialization Choices + +**Status:** Design proposal + +**Audience:** Planner designers and implementers. + +This document separates window semantics, implementation, and materialization when generating candidates for ASAPPlanner. Stage responsibilities follow [planner-layering.md](planner-layering.md); joint selection is described in [Memo-Based Joint Optimization with MILP](joint-planning-search.md). + +Keep these dimensions separate: + +- **Semantics:** requested window width, offset, evaluation timestamps, and alignment. +- **Implementation:** raw recomputation, supported direct sliding state, fixed-width panes, or an EH algorithm. +- **Materialization:** when each internal state is computed and how long it is retained. + +A pane is an aligned, non-overlapping fixed-width interval and its aggregate/summary state. An EH bucket represents a portion of the stream under an exponential histogram algorithm, with algorithm-specific merge, expiration, and error rules. + +| Implementation | Ingestion, when maintained there | Evaluation | +| --- | --- | --- | +| Direct sliding-window state | Add data and remove expired contributions. | Read state and compute result. | +| Pane-based | Update panes and evict expired panes. | Merge required panes and compute result. | +| EH-based | Update, merge, and evict buckets. | Estimate the window result. | + +This table describes possible strategies. Current pane support does not imply general direct sliding-state or EH support. Offer only executable candidates. Multiple pane widths can coexist; alignment, coverage, mergeability, retention, and error contracts determine legality. A 15m window evaluated every 1m can use 1m panes; complete 5m panes alone generally cannot cover every endpoint. + +Use existing `Materialization` options: `NotMaterialized`, `MatAtQuery(Memory)`, `MatAtIngestion(Memory)`, and `MatAtQueryKeptAcrossEvaluations(Memory)`. Assign them to eligible internal nodes, not necessarily the entire query. For example, maintain panes at ingestion and merge at query time. Generate only supported assignments: retained query-time state currently requires deployment capability and an eligible pane chain. Ingestion timing and upstream dependencies must satisfy Stage 2 rules. + From 8005d2f3eb969683b1541796ed137d82e55408e7 Mon Sep 17 00:00:00 2001 From: zzylol <50204836+zzylol@users.noreply.github.com> Date: Fri, 23 Oct 2026 23:42:58 +0000 Subject: [PATCH 33/53] docs: select physical instances within memo groups --- .../proposals/joint-planning-search.md | 161 +++++++++--------- 1 file changed, 84 insertions(+), 77 deletions(-) diff --git a/docs/design_docs/proposals/joint-planning-search.md b/docs/design_docs/proposals/joint-planning-search.md index 8625ba3ab..f8a673564 100644 --- a/docs/design_docs/proposals/joint-planning-search.md +++ b/docs/design_docs/proposals/joint-planning-search.md @@ -74,9 +74,9 @@ Sharing a memo group shares optimization representation, not necessarily runtime ASAPPlanner follows the logical/physical stage responsibilities in [planner-layering.md](planner-layering.md), but must also represent approximate implementations and reuse between different results. -A **memo group** identifies a computation and its output contract: a query-result expression or a reusable intermediate state. A result group preserves input, filters, grouping, window semantics, and requested output. A state group additionally specifies its state family/interface and guarantees. **Implementation candidates** provide exact computation, accumulators, or summaries and reference input groups. Physical candidates add legal operators and materialization assignments. A **shared producer** is a selected group implementation read by several consumers, not a separately selected catalog entry. +A **memo group** identifies a computation and its output contract: a query-result expression or a reusable intermediate state. A result group preserves input, filters, grouping, window semantics, and requested output. A state group additionally specifies its state family/interface and guarantees. **Implementation candidates** provide exact computation, accumulators, or summaries and reference input groups. Physical candidates add legal operators and materialization assignments. A **physical instance** is one enabled realization of a group, with its own selected candidate, materialization, reader set, and lifetime. Several instances of the same group can coexist. A **shared producer** is an instance read by several consumers, not a separately selected catalog entry. -Unlike exact expressions in a traditional equivalence group, approximate candidates target the same requested result under explicit accuracy obligations. State candidates must satisfy their group's contract, and readers can impose stricter compatibility requirements. One group identity does not force one runtime execution. A stage-level candidate remains a whole-workload DAG; local group candidates compactly represent its component choices. +Unlike exact expressions in a traditional equivalence group, approximate candidates target the same requested result under explicit accuracy obligations. State candidates must satisfy their group's contract, and readers can impose stricter compatibility requirements. One group identity neither forces shared execution nor limits the plan to one physical instance. A stage-level candidate remains a whole-workload DAG; local group candidates compactly represent its component choices. The following example traces how the proposed memo expands: @@ -120,7 +120,7 @@ G95 += EstimateQuantile(0.95, G_KLL) G99 += EstimateQuantile(0.99, G_KLL) ``` -The result groups are not merged. Their new candidates depend on the same state group, whose selected capacity must satisfy both readers. Independent candidates remain available. Conventional CSE can similarly reuse identical computations; ASAP-aware rules also expose window composition, rollup, top-k reuse, and scan widening. +The result groups are not merged. Their new candidates depend on the same state group. If they attach to one physical instance, that instance's selected capacity must satisfy both; they can instead attach to different instances with different configurations. Independent candidates remain available. Conventional CSE can similarly reuse identical computations; ASAP-aware rules also expose window composition, rollup, top-k reuse, and scan widening. For pane-based reuse, window-composition rules can add: @@ -135,13 +135,13 @@ This is another way to produce compatible window state. Other generated pane wid Generate supported physical operators and assignments for these candidates. For example, an eligible `G_panes` implementation can maintain panes at ingestion while merge and estimation execute at query time. Query-time construction and supported retention modes remain candidates where legal. -Stage 2 checks deployment capabilities, ingestion eligibility, and upstream timing. It does not choose the cheapest candidate. Assignments can differ between internal nodes; sharing a logical state does not imply a shared physical execution under every assignment. +Stage 2 checks deployment capabilities, ingestion eligibility, and upstream timing. It does not choose the cheapest candidate. Assignments can differ between internal nodes. Prepare finite instance slots for each group so readers can share one instance, form several sharing subsets, or compute separately; do not enumerate all reader partitions as complete plans. Reader-dependent eligibility must use the selected instance's readers; defer unknown joint checks instead of pruning a mode based on every potential reader of the logical group. ### Stage 3: select a joint physical plan -The MILP chooses candidates for both query roots and their required input groups, including any intermediate states. It may select exact/local computation, a shared whole-window build, or a legal pane-based candidate. It enforces reader/state compatibility, accuracy, latency, and resource constraints, then reconstructs and validates one `PhysicalASAPDAG`. +The MILP chooses which group instances to enable, one candidate per enabled instance, and the instance serving each selected input edge and query-root reference. It may select exact/local computation, a shared whole-window build, or a legal pane-based candidate. It enforces reader/state compatibility, accuracy, latency, and resource constraints, then reconstructs and validates one `PhysicalASAPDAG`. -Shared execution is charged once, with merge and estimation charges owned by their selected groups. A higher standalone cost can enable a cheaper joint plan, so earlier stages retain valid candidates and Stage 3 performs cost-based selection. +Each physical instance owns its work/state charges; an instance shared by several readers is not charged once per reader. A higher standalone cost can enable a cheaper joint plan, so earlier stages retain valid candidates and Stage 3 performs cost-based selection. Window semantics, implementation candidates, and materialization are defined separately in [Window and Materialization Choices](window-materialization-choices.md). That design also describes pane/EH terminology, supported configurations, and materialization eligibility. @@ -170,7 +170,7 @@ Accuracy is an end-to-end root requirement. Compose summary and window error onl ## 5. MILP Formulation -Query results and shared intermediate states use the same group/candidate model. There are no separate producer-selection or producer-input variables. Extend the base formulation with consumer-reach, physical-copy, and lifetime variables when execution costs depend on the selected context. +Groups represent computation contracts; instances represent physical realizations. A group can have several enabled instances with different candidates. Each enabled instance chooses exactly one candidate. Reuse is expressed by multiple consumers connecting to the same instance, without a separate producer catalog. ### 5.1 Variables @@ -178,101 +178,97 @@ Query results and shared intermediate states use the same group/candidate model. | Symbol | Meaning | | --- | --- | -| `Q`, `G` | Queries and memo groups, including intermediate-state groups. | -| `C_g` | Legal physical candidates for group `g`, including state configurations and local materialization assignments. | -| `I(g,c)` | Groups implemented by candidate `(g,c)`, including fusion. | -| `R(g,c)` | Input groups required when `(g,c)` is selected, whether they produce results or intermediate state. | -| `K(g,c,h)` | Candidates of input group `h` compatible with reader candidate `(g,c)`. | +| `Q`, `G` | Queries and memo groups, including intermediate states. | +| `P_g` | Finite physical-instance slots for group `g`. | +| `C_g` | Legal physical candidates, including state configuration and materialization assignment. | +| `R(g,c)` | Indexed input-edge occurrences required by candidate `(g,c)`. | +| `H(g,c,e)` | Group required by input edge `e ∈ R(g,c)`. | +| `K(g,c,e)` | Candidates of that input group compatible with this edge. | +| `T(q)` | Query-root references; each reference `t` identifies group `root(q,t)`. | -For example, a P95 estimator depends on a KLL-state group through `R`; `K` identifies that state's candidates with compatible family, window, capacity, and timing. Several required input groups create several dependencies, all of which must be satisfied. +Input edges reference groups but connect to particular instances. Preserve distinct edge occurrences even when two inputs reference the same group. Capacity, pane width, retention, and materialization are candidate configurations, not separate producer decisions. Finite configurations permit lookup-table costs without assuming a linear size/cost function. #### Decision variables ```text -a[g] ∈ {0,1} group g is required -x[g,c] ∈ {0,1} select candidate c for group g +u[g,i] ∈ {0,1} enable instance i of group g +x[g,i,c] ∈ {0,1} instance (g,i) selects candidate c +b[g,i,c,e,j] ∈ {0,1} input edge e connects to instance j of H(g,c,e) +r[q,t,i] ∈ {0,1} root reference t of query q uses instance i ``` -Summary capacity, pane width, retention, and materialization are candidate configurations, not a second set of producer decisions. Discrete configurations permit lookup-table costs without assuming a linear size/cost function. Materialization can instead use explicit legal assignment variables; do not maintain two representations that can disagree. Several internal nodes can require separate assignment slots. +Each instance's materialization can be bundled into its candidate or represented by consistent legal assignment variables. Several internal nodes can have separate assignment slots. Two enabled instances may use the same candidate but have different readers or execution scopes. #### Objective and cost attribution For fixed-context costs: ```text -min Σ_{g,c} cost[g,c] × x[g,c] + plan-level charges +min Σ_{g,i,c} cost[g,i,c] × x[g,i,c] + plan-level charges ``` -`cost[g,c]` owns only the candidate's local work/state, down to its input-group boundaries. A KLL-state candidate owns its build and retention; an estimator candidate owns extraction. A shared merge has its own group if readers reuse its result. Never include an input group's costs again in each parent. +The cost belongs to the instance's local work/state, excluding input instances. A KLL-state instance owns build and retention; estimator instances own extraction. A reused merge result also has its own instance. Two readers sharing one state instance incur its owned charge once; two enabled state instances incur two charges. -Selecting one state candidate does not by itself mean one execution. Materialization and consumer context determine whether it runs once per evaluation, per reader, or at ingestion. Add reach/rate and physical-copy variables for these cases; charge each actual instance correctly. Reader recurrence, phase, and range-evaluation steps affect work; a pane-alignment gcd grid is not automatically an execution rate. Raw retention and other non-computation charges remain resource/accounting variables rather than artificial computation groups. +An enabled instance is not necessarily executed only once: query-time work can repeat. Its materialization, reader schedules, phases, range steps, and upstream consumers determine the execution rate and state lifetime. Extend the base model with reach/rate/lifetime variables or finite context configurations constrained to match the selected graph. Do not assign a fixed cost independent of that context. Raw retention and other non-computation charges remain shared resource/accounting variables. ### 5.2 Constraints -#### Query coverage and dependencies - -Let `Roots(q)` contain the group references needed for query `q`, including scalar roots with multiple references: +#### Instance selection and query coverage ```text -a[g] = 1 for g ∈ Roots(q) -Σ_{c ∈ C_g} x[g,c] = a[g] without fusion +Σ_{c ∈ C_g} x[g,i,c] = u[g,i] for each instance (g,i) +Σ_{i ∈ P_root(q,t)} r[q,t,i] = 1 for each root reference (q,t) +r[q,t,i] ≤ u[root(q,t),i] ``` -With fused candidates, use: +There is no constraint limiting an entire group to one enabled instance. Each query-root reference must connect to an available implementation. -```text -Σ_{(h,c): g ∈ I(h,c)} x[h,c] = a[g] -``` +#### Input dependencies and compatibility -Every required group is implemented once, possibly by fusion; absorbed groups do not also select standalone implementations. Every selected candidate requires its inputs, including shared states: +For each input edge of a selected candidate, choose one compatible input instance. Let `h = H(g,c,e)`: ```text -a[h] ≥ x[g,c] for h ∈ R(g,c) +Σ_{j ∈ P_h} b[g,i,c,e,j] = x[g,i,c] +b[g,i,c,e,j] ≤ u[h,j] +b[g,i,c,e,j] ≤ Σ_{d ∈ K(g,c,e)} x[h,j,d] ``` -Add reverse support constraints: an active group must be justified by a root, selected consumer, or fusion coverage. This prevents unused activation even at zero prices. Dependencies are conditional on selected candidates, not every possible child. Enforce acyclicity and reuse requirements such as exact inputs or exposed state. - -#### Reader/state compatibility +Several consumers can choose the same `j`; other consumers can choose another instance of the same group. Compatibility covers state family, grouping, reduction, window coverage, capacity, timing, and freshness. If two readers attach to one instance, its selected candidate must satisfy both readers. -Requiring an input group is insufficient if only some of its candidates can serve the reader. For each input group `h`: +Every enabled instance must have a selected root reference or incoming input edge: ```text -x[g,c] ≤ Σ_{d ∈ K(g,c,h)} x[h,d] for h ∈ R(g,c) +u[h,j] ≤ Σ root references using (h,j) + + Σ selected input edges using (h,j) ``` -This form assumes nonfused input candidates. If another candidate implements `h` by fusion, use its compatible implementing choices in the sum instead. Compatibility covers inputs, grouping, family, reduction, windows, capacity, freshness, and timing. If several readers select the same state group, its chosen candidate must satisfy all of them. - -Different pane widths can coexist as distinct intermediate groups. A group-level configuration choice must not become a global one-pane-width restriction. Retain local computation through local-state candidates/instances or the existing per-reader-copy/materialization mechanism; neither interning nor CSE should force a shared execution. Shared-instance scope is explicit where readers cannot use one configuration or lifetime. - -#### Auxiliary-variable constraints - -For binary `v = u AND w`, use: +Together with acyclicity, this prevents disconnected instances even when prices are zero. Enforce an acyclic physical instance graph, not a requirement that every possible memo dependency be acyclic. For example, with `N` total slots and order variables `0 ≤ order[g,i] ≤ N−1`: ```text -v ≤ u -v ≤ w -v ≥ u + w - 1 +order[h,j] + 1 ≤ order[g,i] + N × (1 − b[g,i,c,e,j]) ``` -Use valid bounds for other products and maxima. A lower-bound-only maximum is exact only when minimization or other constraints make it tight. +This orders producers before their selected consumers. Interchangeable slots may additionally use symmetry-breaking constraints such as `u[g,i] ≥ u[g,i+1]`. + +#### Fusion and available outputs -#### Accuracy, validity, and budgets +A fused candidate can eliminate internal computation boundaries without exposing those intermediate results. In the base model it implements its owning group; only explicitly available outputs can serve external readers. For example, a fused `topk(sum(...))` does not automatically supply another query's full `sum(...)` result. That query can require a separate sum instance. Multi-output fusion needs explicit output mappings before it can satisfy several groups; absorption alone is not coverage. -- **Accuracy:** preserve state contracts and enforce proven per-query composition bounds, feasible-combination tables, or sound refinement. -- **Physical validity:** require supported operators and legal materialization assignments, including freshness, upstream timing, and state lifetime. -- **Window correctness:** require alignment, coverage without omission/double counting, retention, and algorithm-specific guarantees. -- **Resource limits:** constrain requested CPU, memory, disk, and other resources under the configured concurrency model. -- **Latency:** enforce configured query bounds and applicable recurrence checks. +#### Accuracy, materialization, and resources -For an additive resource `k` with fixed-context usage: +- **Accuracy:** preserve state contracts and enforce root-level composition bounds, feasible-combination tables, or sound refinement over the selected instance graph. +- **Physical validity:** require supported operators, legal materializations, available upstream state, and valid freshness/lifetimes. +- **Windows:** validate alignment, coverage without omission/double counting, retention, and algorithm-specific guarantees. +- **Resources and latency:** apply configured CPU/memory/disk limits, query deadlines, and recurrence checks using actual instance demand and timing. + +For additive, fixed-context resource usage: ```text -Σ_{g,c} usage[k,g,c] × x[g,c] + plan-level usage[k] ≤ capacity[k] +Σ_{g,i,c} usage[k,g,i,c] × x[g,i,c] + + plan-level usage[k] ≤ capacity[k] ``` -Extend this for context-dependent rates and lifetimes. Do not use average occupancy as peak capacity or sum unrelated workloads as query latency. Resident state and concurrent temporaries may need separate constraints. - -Representing sharing as group dependencies simplifies the graph but does not make costs independent. DP with complete sharing context remains possible; retaining one cheapest candidate per group can still discard the joint optimum. +Peak capacity is not average occupancy; temporary lifetimes and concurrency need their own accounting. Use bounded auxiliary-variable linearizations where needed. A sum of fixed candidate costs is only the base case, not a substitute for the configured evaluator. ## 6. Worked Example: Shared KLL for P95/P99 @@ -284,11 +280,11 @@ Q2: quantile_over_time(0.99, stream[15m]), every 1m Using the groups introduced in Section 3, compare the cost of selecting shared versus local KLL candidates: ```text -G_input ──→ G_KLL: build KLL state - / \ - G95: estimate P95 G99: estimate P99 - | | - Q1 Q2 +G_input[i0] → G_KLL[i1]: build KLL state + / \ + G95[i1]: estimate P95 G99[i1]: estimate P99 + | | + Q1 Q2 ``` Illustrative costs per common horizon, assuming all listed candidates satisfy requirements: @@ -299,18 +295,25 @@ Illustrative costs per common horizon, assuming all listed candidates satisfy re | Shared KLL build | 8 | 1 + 1 | 10 | | Separate exact computations | 12 + 12 | included | 24 | -The shared choice selects both reader candidates and one KLL-state candidate: +The shared choice enables one state instance, and both estimator instances connect to it: ```text -R(G95, KLL-reader) = {G_KLL} -R(G99, KLL-reader) = {G_KLL} -R(G_KLL, build-s) = {G_input} +u[G_KLL,1] = x[G_KLL,1,build-s] = 1 +x[G95,1,KLL-reader] = x[G99,1,KLL-reader] = 1 +b[G95,1,KLL-reader,state-input,1] = 1 +b[G99,1,KLL-reader,state-input,1] = 1 +``` + +The build cost is owned by `G_KLL[1]`; each estimator instance owns its extraction cost. If accuracy requires larger state, the solver compares compatible shared capacity, local states, and exact candidates. Quantile level alone does not determine capacity. -x[G95, KLL-reader] = x[G99, KLL-reader] = 1 -x[G_KLL, build-s] = 1 +With four readers, the same group can also have two shared instances: + +```text +G_KLL[1]: small-capacity query-time state ← readers A, B +G_KLL[2]: large-capacity query-time state ← readers C, D ``` -The build cost is owned by `G_KLL` and appears once for a shared execution; each result group owns its estimation cost. If accuracy requires larger state, the solver compares compatible shared capacity, local states, and exact candidates. Quantile level alone does not determine capacity. +Both can be enabled if their combined cost is lower or one instance cannot satisfy all requirements. Their selected candidates can differ in capacity, materialization, retention, and execution frequency. Each instance must satisfy every attached reader; readers do not all have to share one configuration. For ingestion-time reuse, a supported implementation maintains 1m KLL panes and merges the latest 15 at evaluation. Both readers can share panes and an eligible merge result. A direct whole-window sketch cannot be assumed to delete expired samples. Different window lengths or pane widths require separate compatibility and coverage checks. @@ -323,7 +326,11 @@ Reduce search without losing valid joint choices: 3. Split coupling components only when all possible sharing, dependencies, raw retention, and constraints separate. Global capacities or calibration can prevent independent optimization. 4. Use a validated baseline incumbent and solver time/gap limits. Restricted parameter inventories narrow the optimality claim; enrichment requires solving the expanded inventory. -Reconstruct selected candidates, their group dependencies, and timing-specific copies into one `PhysicalASAPDAG`. Check dependencies and cycles, then run the configured accuracy, latency, and cost/resource evaluators. Return a validated plan, fallback, or explicit search failure status. +Define a finite rule/configuration inventory and process new groups/candidates through a worklist, including newly enabled matches with existing candidates, until no eligible rule produces a new item. Deduplicate rule applications and do not discard candidates merely because they are more expensive alone. If generation stops early, report the restricted inventory rather than claiming rule closure. + +Instance slots also bound the search space. Use an upper bound justified by the unshared expansion of the permitted finite, acyclic derivations and evaluation scopes, or explicitly declare a slot cap as a restriction. An arbitrary cap can omit profitable sharing partitions. Rule closure establishes generation completeness only within this declared inventory; MILP optimality establishes selection quality within the generated model. Validate both separately. + +Reconstruct enabled instances, their candidates, and selected input/root connections into one `PhysicalASAPDAG`. Do not merge distinct instances solely because they share a group identity. Check dependencies and cycles, then run the configured accuracy, latency, and cost/resource evaluators. Return a validated plan, fallback, or explicit search failure status. A complete MILP does not require LBBD. Current code uses a lower-bound model with lazy refinement for difficult costs/checks. When refinement is needed, add only sound feasibility or cost cuts conditioned on every relevant decision. Validation of one plan alone does not prove a relaxed model optimal. Report the best validated cost, certified lower bound/gap, solver status, and stop reason. Timeout without a plan is not proof of infeasibility. @@ -334,28 +341,28 @@ Reuse current structures rather than create a parallel optimizer framework: | Structure | Responsibility | | --- | --- | | `MemoGroup`, `Memo` | Result/state contracts, dependencies, roots, and per-group candidates. | -| `ImplementationCandidate` | Local nodes, fusion coverage, child groups, contracts, and reuse requirements. | +| `ImplementationCandidate` | Local nodes, fusion absorption/output metadata, child groups, contracts, and reuse requirements. | | `CouplingSite` and thresholds | Existing shared-configuration representation; migrate state choices into intermediate-group candidates, retaining genuinely cross-group constraints. | -| MILP `Choice` | Physical/materialization assignments, consumer context, and coefficients. | +| MILP `Choice` and proposed instance records | Instance identity, selected physical/materialization assignment, reader connections, and cost context. | | `PhysicalASAPDAG`, `Selection`, reports | Selected output, validation, bounds, and search status. | -This is an extension of the current representation: shared builds are primarily coordinated through coupling sites today. Introduce intermediate-state group contracts and reader compatibility, then migrate corresponding state choices and charge ownership into group candidates. This requires changes to generation, selection, decoding, and pricing; it is not merely renaming coupling sites. Raw retention and global capacities remain accounting constraints. +This is an extension of the current representation: shared builds are primarily coordinated through coupling sites today. Introduce intermediate-state contracts, finite instance slots, and reader compatibility; migrate state choices into instance candidates and assign costs to physical instances. This requires changes to generation, selection, decoding, and pricing; it is not merely renaming coupling sites. Raw retention and global capacities remain accounting constraints. -Current main places memo/rules in `search/memo` and solver code in `search/milp`, with Stage 2 physical generation and Stage 3 pricing remaining authoritative. `Legacy` is the default; `MemoBruteForce` is the reference and `MemoMilp` the solver path. Early memo construction, unified shared-state groups, broader cost-model support, EH/direct sliding algorithms, and arbitrary pane hierarchies are proposed extensions. +Current main places memo/rules in `search/memo` and solver code in `search/milp`, with Stage 2 physical generation and Stage 3 pricing remaining authoritative. `Legacy` is the default; `MemoBruteForce` is the reference and `MemoMilp` the solver path. Early memo construction, unified shared-state groups with selectable physical instances, broader cost-model support, EH/direct sliding algorithms, and arbitrary pane hierarchies are proposed extensions. ### Implementation phases | Phase | Deliverable and acceptance criterion | | --- | --- | | 1. Memo generation | Move identity construction before Pass 1 and generate candidates directly into groups without losing semantics, reader obligations, or fusion. | -| 2. Shared-state groups | Extend group contracts; generate intermediate groups and reader candidates with compatible dependencies, legal materializations, and exact-once charge ownership. | +| 2. Shared-state groups | Extend group contracts; generate intermediate groups, instance slots, and compatible reader connections with legal materializations and per-instance cost ownership. | | 3. Joint selection | Extend MILP coverage as required; match exhaustive and memo brute force under the same inventory/models. | | 4. Validation and scaling | Validate reconstructed plans, prove safe reductions, and measure generation/model/solve costs under budgets. | ### Evaluation -**Correctness:** test identity versus sharing, producer compatibility, exact-once cost ownership, active dependencies, fusion, multiple input groups, shared/local instances, zero-price resources, pane boundaries, legal materializations, end-to-end accuracy, and capacity/status handling. Validate any relaxed bounds and cuts against enumerated plans. +**Correctness:** test identity versus sharing, producer compatibility, exact-once cost ownership, active dependencies, fusion with external intermediate readers, multiple input edges, multiple configurations/partial-sharing subsets of one group, zero-price resources, pane boundaries, legal materializations, end-to-end accuracy, and capacity/status handling. Validate any relaxed bounds and cuts against enumerated plans. **Optimality:** tiny workloads must match exhaustive enumeration and memo brute force on minimum validated cost, with an explicit numerical/tie policy and the same candidate inventory. -**Scalability:** measure groups, candidates, state configurations, dependency edges, variables/constraints, generation/solve/validation time, and peak planner memory. Compare joint versus independent planning savings, time-limited gaps, and modeled versus validated costs across query overlap, window configurations, and materializations. +**Scalability:** measure groups, candidates, instance slots/enabled instances, state configurations, input connections, variables/constraints, generation/solve/validation time, and peak planner memory. Compare joint versus independent planning savings, time-limited gaps, and modeled versus validated costs across query overlap, window configurations, and materializations. From 4304de0d3c0c37936544b63784ed38d7c7e3461a Mon Sep 17 00:00:00 2001 From: zzylol <50204836+zzylol@users.noreply.github.com> Date: Sat, 24 Oct 2026 00:08:47 +0000 Subject: [PATCH 34/53] docs: frame joint planning as covering queries with an execution DAG --- .../proposals/joint-planning-search.md | 26 +++++++++++++++---- 1 file changed, 21 insertions(+), 5 deletions(-) diff --git a/docs/design_docs/proposals/joint-planning-search.md b/docs/design_docs/proposals/joint-planning-search.md index f8a673564..c830aae0b 100644 --- a/docs/design_docs/proposals/joint-planning-search.md +++ b/docs/design_docs/proposals/joint-planning-search.md @@ -10,7 +10,7 @@ ASAP primitives create opportunities to replace expensive exact computations and The challenge is cross-query dependence: a candidate's incremental cost changes when another candidate enables a shared producer, requires a larger summary, or extends retention. Selecting the cheapest candidate for each group independently can therefore lose the workload optimum. -The design uses a memo to represent candidates compactly and a mixed-integer linear program (MILP) to select a consistent physical plan. The goals are to avoid repeated candidate generation, expose ASAP-aware sharing, and minimize the configured objective subject to accuracy, latency, and resource constraints. Optimality applies to the generated finite candidate set and configured models, not every imaginable plan. New sketch/window algorithms and runtime scheduling are outside this proposal. +The optimization problem is to select and connect physical implementation instances into a minimum-cost execution DAG covering every query. The memo generates and organizes the supported candidates; a mixed-integer linear program (MILP) selects instance activation, implementations, and compatible connections jointly. The goals are to avoid repeated candidate generation, expose ASAP-aware sharing, and minimize the configured objective subject to accuracy, latency, and resource constraints. Optimality applies to the generated finite candidate set and configured models, not every imaginable plan. New sketch/window algorithms and runtime scheduling are outside this proposal. ## 2. Traditional Memo Groups and Rule Expansion @@ -139,7 +139,7 @@ Stage 2 checks deployment capabilities, ingestion eligibility, and upstream timi ### Stage 3: select a joint physical plan -The MILP chooses which group instances to enable, one candidate per enabled instance, and the instance serving each selected input edge and query-root reference. It may select exact/local computation, a shared whole-window build, or a legal pane-based candidate. It enforces reader/state compatibility, accuracy, latency, and resource constraints, then reconstructs and validates one `PhysicalASAPDAG`. +Stage 3 covers all query roots by choosing which group instances to enable, one candidate per enabled instance, and the supplier instance for every selected input edge and query-root reference. It may select exact/local computation, a shared whole-window build, or a legal pane-based candidate. It enforces reader/state compatibility, accuracy, latency, and resource constraints, then reconstructs and validates one `PhysicalASAPDAG`. Each physical instance owns its work/state charges; an instance shared by several readers is not charged once per reader. A higher standalone cost can enable a cheaper joint plan, so earlier stages retain valid candidates and Stage 3 performs cost-based selection. @@ -147,6 +147,22 @@ Window semantics, implementation candidates, and materialization are defined sep ## 4. Cost Model and Optimization Objective +### Covering queries with an execution DAG + +A query is covered only when its root references connect to enabled instances, every required input is recursively supplied by a compatible instance, and the resulting DAG satisfies accuracy, timing, and deployment constraints. Selecting a KLL build alone does not cover a quantile query: the estimator and the build's upstream dependencies must also be provided. + +The decisions are instance activation, implementation selection, and input assignment. Sharing occurs when several consumers connect to the same instance. The objective is the total cost of those instances under their assigned demand, not the sum of independently priced query paths. + +### Why MILP rather than bipartite matching + +A bipartite compatibility graph captures which suppliers can serve each input, but ordinary matching cannot express the whole problem. Suppliers can serve multiple consumers, enabling a supplier has a pay-once cost, candidates can require several inputs together, and suppliers themselves depend on upstream instances. Accuracy and deployment limits further couple the choices. + +For example, enabling a KLL build costs 8 and each of two estimators costs 1. Sharing costs 10. Charging the build on both assignment edges incorrectly gives 18; omitting it leaves activation unpriced. Allowing multiple assignments through b-matching or min-cost flow does not by itself model that activation cost or the dependency DAG. Facility location captures activation and assignment more closely, but must still be extended for upstream dependencies and demand-dependent costs. + +MILP is the proposed joint solver, not a claim that it is the only exact method. Matching or min-cost flow can solve a restricted assignment subproblem after supplier activation, implementations, rates, and costs are fixed, provided its remaining capacities and constraints admit that formulation. Without assignment coupling, choosing each input's cheapest compatible supplier may suffice. Any local assignment must preserve the global demand and feasibility assumptions. + +### Resource pricing and demand + Attribute resource usage to physical nodes in the selected DAG, then apply the existing Stage 3 cost model: | Model | Selection policy | @@ -170,7 +186,7 @@ Accuracy is an end-to-end root requirement. Compose summary and window error onl ## 5. MILP Formulation -Groups represent computation contracts; instances represent physical realizations. A group can have several enabled instances with different candidates. Each enabled instance chooses exactly one candidate. Reuse is expressed by multiple consumers connecting to the same instance, without a separate producer catalog. +The formulation selects a covering execution DAG. Groups represent computation contracts; instances represent physical realizations. A group can have several enabled instances with different candidates. Each enabled instance chooses exactly one candidate. Reuse is expressed by multiple consumers connecting to the same instance, without a separate producer catalog. ### 5.1 Variables @@ -356,13 +372,13 @@ Current main places memo/rules in `search/memo` and solver code in `search/milp` | --- | --- | | 1. Memo generation | Move identity construction before Pass 1 and generate candidates directly into groups without losing semantics, reader obligations, or fusion. | | 2. Shared-state groups | Extend group contracts; generate intermediate groups, instance slots, and compatible reader connections with legal materializations and per-instance cost ownership. | -| 3. Joint selection | Extend MILP coverage as required; match exhaustive and memo brute force under the same inventory/models. | +| 3. Joint selection | Encode covering-DAG activation, implementations, assignments, and demand; match an independent tiny-instance enumerator under the same inventory/models. | | 4. Validation and scaling | Validate reconstructed plans, prove safe reductions, and measure generation/model/solve costs under budgets. | ### Evaluation **Correctness:** test identity versus sharing, producer compatibility, exact-once cost ownership, active dependencies, fusion with external intermediate readers, multiple input edges, multiple configurations/partial-sharing subsets of one group, zero-price resources, pane boundaries, legal materializations, end-to-end accuracy, and capacity/status handling. Validate any relaxed bounds and cuts against enumerated plans. -**Optimality:** tiny workloads must match exhaustive enumeration and memo brute force on minimum validated cost, with an explicit numerical/tie policy and the same candidate inventory. +**Optimality:** compare MILP with an independent tiny-instance enumerator covering instance activation, candidate selection, root/input assignments, and dependency validity under the same inventory/models. The existing group-level brute-force reference must be extended before it can verify arbitrary instance partitions. Test candidate-generation coverage separately, including cases where two independently shared instances outperform full sharing and separate computation. Use an explicit numerical/tie policy. **Scalability:** measure groups, candidates, instance slots/enabled instances, state configurations, input connections, variables/constraints, generation/solve/validation time, and peak planner memory. Compare joint versus independent planning savings, time-limited gaps, and modeled versus validated costs across query overlap, window configurations, and materializations. From 9f94389866019583dc94761f3e2591f21e35b016 Mon Sep 17 00:00:00 2001 From: zzylol <50204836+zzylol@users.noreply.github.com> Date: Sat, 24 Oct 2026 00:11:49 +0000 Subject: [PATCH 35/53] docs: state the optimization problem before the proposed solution --- docs/design_docs/proposals/joint-planning-search.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/design_docs/proposals/joint-planning-search.md b/docs/design_docs/proposals/joint-planning-search.md index c830aae0b..f4b86d6f0 100644 --- a/docs/design_docs/proposals/joint-planning-search.md +++ b/docs/design_docs/proposals/joint-planning-search.md @@ -10,7 +10,7 @@ ASAP primitives create opportunities to replace expensive exact computations and The challenge is cross-query dependence: a candidate's incremental cost changes when another candidate enables a shared producer, requires a larger summary, or extends retention. Selecting the cheapest candidate for each group independently can therefore lose the workload optimum. -The optimization problem is to select and connect physical implementation instances into a minimum-cost execution DAG covering every query. The memo generates and organizes the supported candidates; a mixed-integer linear program (MILP) selects instance activation, implementations, and compatible connections jointly. The goals are to avoid repeated candidate generation, expose ASAP-aware sharing, and minimize the configured objective subject to accuracy, latency, and resource constraints. Optimality applies to the generated finite candidate set and configured models, not every imaginable plan. New sketch/window algorithms and runtime scheduling are outside this proposal. +The problem addressed by this document is to find a minimum-cost physical execution DAG for a workload of queries under the deployment's cost model. The plan must produce every requested result, satisfy each query's accuracy and latency requirements, respect resource limits, and account for shared computation and retained state without duplicate charges. This requires jointly deciding which supported implementations to use, which computations to share, and when to materialize their outputs, while keeping planning time and memory manageable. Optimality is relative to the supported finite candidate space and configured models. Developing new sketch/window algorithms and runtime scheduling policies is outside the scope. ## 2. Traditional Memo Groups and Rule Expansion From b400fa031300784fad7f5ce19a8607a47d639dac Mon Sep 17 00:00:00 2001 From: Zeying Zhu <50204836+zzylol@users.noreply.github.com> Date: Fri, 23 Oct 2026 20:15:06 -0400 Subject: [PATCH 36/53] Update joint-planning-search.md --- docs/design_docs/proposals/joint-planning-search.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/design_docs/proposals/joint-planning-search.md b/docs/design_docs/proposals/joint-planning-search.md index f4b86d6f0..0b9781b0c 100644 --- a/docs/design_docs/proposals/joint-planning-search.md +++ b/docs/design_docs/proposals/joint-planning-search.md @@ -10,7 +10,7 @@ ASAP primitives create opportunities to replace expensive exact computations and The challenge is cross-query dependence: a candidate's incremental cost changes when another candidate enables a shared producer, requires a larger summary, or extends retention. Selecting the cheapest candidate for each group independently can therefore lose the workload optimum. -The problem addressed by this document is to find a minimum-cost physical execution DAG for a workload of queries under the deployment's cost model. The plan must produce every requested result, satisfy each query's accuracy and latency requirements, respect resource limits, and account for shared computation and retained state without duplicate charges. This requires jointly deciding which supported implementations to use, which computations to share, and when to materialize their outputs, while keeping planning time and memory manageable. Optimality is relative to the supported finite candidate space and configured models. Developing new sketch/window algorithms and runtime scheduling policies is outside the scope. +This doc aims to address the problem of finding a minimum-cost physical execution DAG for a workload of queries under the deployment's cost model. The plan must produce every requested result, satisfy each query's accuracy and latency requirements, respect resource limits, and account for shared computation and retained state without duplicate cost charging. This requires jointly deciding which supported implementations to use, which computations to share, and when to materialize their outputs, while keeping ASAPPlanner's planning time and planning runtime memory reasonable. Optimality is relative to the supported finite candidate space and configured models. Developing new sketch/window algorithms and runtime deployment is outside the scope. ## 2. Traditional Memo Groups and Rule Expansion From 28e50818ddcfc86b7dff1723e60961c3f1366a13 Mon Sep 17 00:00:00 2001 From: Zeying Zhu <50204836+zzylol@users.noreply.github.com> Date: Fri, 23 Oct 2026 20:17:25 -0400 Subject: [PATCH 37/53] Update joint-planning-search.md --- docs/design_docs/proposals/joint-planning-search.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/design_docs/proposals/joint-planning-search.md b/docs/design_docs/proposals/joint-planning-search.md index 0b9781b0c..14dac15b5 100644 --- a/docs/design_docs/proposals/joint-planning-search.md +++ b/docs/design_docs/proposals/joint-planning-search.md @@ -72,9 +72,9 @@ Sharing a memo group shares optimization representation, not necessarily runtime ## 3. ASAPPlanner Memo Groups and Rule Expansion -ASAPPlanner follows the logical/physical stage responsibilities in [planner-layering.md](planner-layering.md), but must also represent approximate implementations and reuse between different results. +Given the benefits of Memo groups, ASAPPlanner should introduce the Memo group while representing summaries and reuse strategies. -A **memo group** identifies a computation and its output contract: a query-result expression or a reusable intermediate state. A result group preserves input, filters, grouping, window semantics, and requested output. A state group additionally specifies its state family/interface and guarantees. **Implementation candidates** provide exact computation, accumulators, or summaries and reference input groups. Physical candidates add legal operators and materialization assignments. A **physical instance** is one enabled realization of a group, with its own selected candidate, materialization, reader set, and lifetime. Several instances of the same group can coexist. A **shared producer** is an instance read by several consumers, not a separately selected catalog entry. +In ASAPPlanner, we extend the definition of a **memo group**, whose output can be a query-result expression or a reusable intermediate summary state. A result group preserves input, filters, grouping, window semantics, and requested output. A state group additionally specifies its state family/interface and guarantees. **Implementation candidates** provide exact computation, accumulators, or summaries and reference input groups. Physical candidates add legal operators and materialization assignments. A **physical instance** is one enabled realization of a group, with its own selected candidate, materialization, reader set, and lifetime. Several instances of the same group can coexist. A **shared producer** is an instance read by several consumers, not a separately selected catalog entry. Unlike exact expressions in a traditional equivalence group, approximate candidates target the same requested result under explicit accuracy obligations. State candidates must satisfy their group's contract, and readers can impose stricter compatibility requirements. One group identity neither forces shared execution nor limits the plan to one physical instance. A stage-level candidate remains a whole-workload DAG; local group candidates compactly represent its component choices. From 07e44cea0f618f6f217c6366e5908c90d90625af Mon Sep 17 00:00:00 2001 From: zzylol <50204836+zzylol@users.noreply.github.com> Date: Sat, 24 Oct 2026 00:22:14 +0000 Subject: [PATCH 38/53] docs: clarify shared decision and instance terminology --- .../proposals/joint-planning-search.md | 68 ++++++++++--------- 1 file changed, 35 insertions(+), 33 deletions(-) diff --git a/docs/design_docs/proposals/joint-planning-search.md b/docs/design_docs/proposals/joint-planning-search.md index 14dac15b5..f60308e39 100644 --- a/docs/design_docs/proposals/joint-planning-search.md +++ b/docs/design_docs/proposals/joint-planning-search.md @@ -74,9 +74,9 @@ Sharing a memo group shares optimization representation, not necessarily runtime Given the benefits of Memo groups, ASAPPlanner should introduce the Memo group while representing summaries and reuse strategies. -In ASAPPlanner, we extend the definition of a **memo group**, whose output can be a query-result expression or a reusable intermediate summary state. A result group preserves input, filters, grouping, window semantics, and requested output. A state group additionally specifies its state family/interface and guarantees. **Implementation candidates** provide exact computation, accumulators, or summaries and reference input groups. Physical candidates add legal operators and materialization assignments. A **physical instance** is one enabled realization of a group, with its own selected candidate, materialization, reader set, and lifetime. Several instances of the same group can coexist. A **shared producer** is an instance read by several consumers, not a separately selected catalog entry. +In ASAPPlanner, we extend the definition of a **memo group**, whose output can be a query or intermediate relational result, or reusable intermediate state. A result group preserves input, filters, grouping, window semantics, and requested output. A state group additionally specifies its state family/interface and guarantees. **Implementation candidates** provide exact computation, accumulators, or summaries and reference input groups. Physical candidates add legal operators and materialization assignments. A **physical instance** is one enabled realization of a group, with its own selected candidate, materialization, consumer set, and lifetime. Several instances of the same group can coexist. A **shared producer** is an instance read by several consumers, not a separately selected catalog entry. -Unlike exact expressions in a traditional equivalence group, approximate candidates target the same requested result under explicit accuracy obligations. State candidates must satisfy their group's contract, and readers can impose stricter compatibility requirements. One group identity neither forces shared execution nor limits the plan to one physical instance. A stage-level candidate remains a whole-workload DAG; local group candidates compactly represent its component choices. +Unlike exact expressions in a traditional equivalence group, approximate candidates target the same requested result under explicit accuracy obligations. State candidates must satisfy their group's contract, and consumers can impose stricter compatibility requirements. One group identity neither forces shared execution nor limits the plan to one physical instance. A stage-level candidate remains a whole-workload DAG; local group candidates compactly represent its component choices. The following example traces how the proposed memo expands: @@ -87,7 +87,7 @@ Q2: quantile_over_time(0.99, X[15m]), every 1m ### Stage 0 and initial memo: establish logical identity -Frontends produce language-independent `LogicalDAGs`. Build groups for distinct logical sub-expressions and record query roots and reader requirements. Intern eligible identical inputs only after semantic equality and sharing legality are established; preserve keyless-node rules. P95 and P99 are different computations, so their result groups remain separate: +Frontends produce language-independent `LogicalDAGs`. Build groups for distinct logical sub-expressions and record query roots and consumer requirements. Intern eligible identical inputs only after semantic equality and sharing legality are established; preserve keyless-node rules. P95 and P99 are different computations, so their result groups remain separate: ```text G_input: common eligible input computation X @@ -106,11 +106,11 @@ G95 = { ExactQuantile(0.95, G_input, 15m), LocalKLLQuantile(0.95, G_input, 15m, G99 = { ExactQuantile(0.99, G_input, 15m), LocalKLLQuantile(0.99, G_input, 15m, s) } ``` -Here `s` ranges over generated capacities; each candidate records its accuracy contract and dependencies. Other computations can receive exact accumulator or fused candidates. Cache generation with relevant reader requirements and avoid composing the Cartesian product of complete workload plans. +Here `s` ranges over generated capacities; each candidate records its accuracy contract and dependencies. Other computations can receive exact accumulator or fused candidates. Cache generation with relevant consumer requirements and avoid composing the Cartesian product of complete workload plans. ### Stage 1, Pass 2: add intermediate groups and sharing candidates -ASAP-aware CSE recognizes that compatible KLL state supports both quantiles. Introduce or reuse a state group and add reader candidates: +ASAP-aware CSE recognizes that compatible KLL state supports both quantiles. Introduce or reuse a state group and add consumer candidates: ```text G_KLL: KLL state over X[15m] @@ -135,13 +135,13 @@ This is another way to produce compatible window state. Other generated pane wid Generate supported physical operators and assignments for these candidates. For example, an eligible `G_panes` implementation can maintain panes at ingestion while merge and estimation execute at query time. Query-time construction and supported retention modes remain candidates where legal. -Stage 2 checks deployment capabilities, ingestion eligibility, and upstream timing. It does not choose the cheapest candidate. Assignments can differ between internal nodes. Prepare finite instance slots for each group so readers can share one instance, form several sharing subsets, or compute separately; do not enumerate all reader partitions as complete plans. Reader-dependent eligibility must use the selected instance's readers; defer unknown joint checks instead of pruning a mode based on every potential reader of the logical group. +Stage 2 checks deployment capabilities, ingestion eligibility, and upstream timing. It does not choose the cheapest candidate. Assignments can differ between internal nodes. Prepare finite instance slots for each group so consumers can share one instance, form several sharing subsets, or compute separately; do not enumerate all consumer partitions as complete plans. Consumer-dependent eligibility must use the selected instance's consumers; defer unknown joint checks instead of pruning a mode based on every potential consumer of the logical group. ### Stage 3: select a joint physical plan -Stage 3 covers all query roots by choosing which group instances to enable, one candidate per enabled instance, and the supplier instance for every selected input edge and query-root reference. It may select exact/local computation, a shared whole-window build, or a legal pane-based candidate. It enforces reader/state compatibility, accuracy, latency, and resource constraints, then reconstructs and validates one `PhysicalASAPDAG`. +Stage 3 covers all query roots by choosing which group instances to enable, one candidate per enabled instance, and the supplier instance for every selected input edge and query-root reference. It may select exact/local computation, a shared whole-window build, or a legal pane-based candidate. It enforces consumer/state compatibility, accuracy, latency, and resource constraints, then reconstructs and validates one `PhysicalASAPDAG`. -Each physical instance owns its work/state charges; an instance shared by several readers is not charged once per reader. A higher standalone cost can enable a cheaper joint plan, so earlier stages retain valid candidates and Stage 3 performs cost-based selection. +Each physical instance owns its work/state charges; an instance shared by several consumers is not charged once per consumer. A higher standalone cost can enable a cheaper joint plan, so earlier stages retain valid candidates and Stage 3 performs cost-based selection. Window semantics, implementation candidates, and materialization are defined separately in [Window and Materialization Choices](window-materialization-choices.md). That design also describes pane/EH terminology, supported configurations, and materialization eligibility. @@ -176,11 +176,11 @@ Current memo MILP supports usage-over-time pricing; unsupported models route to Give every charge one owner: - **Producer:** build/update, maintenance, and retained state. -- **Reader:** local merge, extraction, read, and finalization. +- **Consumer:** local merge, extraction, read, and finalization. - **Other physical work:** local operators and temporary state. - **Shared accounting:** per-source raw retention and other plan-level charges. -Charge shared work once per actual execution or maintained state, not once per query path. Reader-specific work remains separately charged. Use a common horizon or consistent per-second rates; distinguish memory/storage occupancy over time from peak capacity. Producer costs can depend on selected readers, evaluation schedules, upstream materialization, and retention. Model that context rather than assuming every producer has a constant cost. +Charge shared work once per actual execution or maintained state, not once per query path. Consumer-specific work remains separately charged. Use a common horizon or consistent per-second rates; distinguish memory/storage occupancy over time from peak capacity. Producer costs can depend on selected consumers, evaluation schedules, upstream materialization, and retention. Model that context rather than assuming every producer has a constant cost. Accuracy is an end-to-end root requirement. Compose summary and window error only with valid contracts; locally admissible sketches do not prove a composed query accurate. Latency and resource limits must use the configured evaluator's timing and concurrency assumptions. Current latency accounting sums reached query-time work plus applicable compaction; this proposal does not silently replace it with a general parallel scheduler. Keep calibrated model inputs fixed during a solve. @@ -213,7 +213,7 @@ b[g,i,c,e,j] ∈ {0,1} input edge e connects to instance j of H(g,c,e) r[q,t,i] ∈ {0,1} root reference t of query q uses instance i ``` -Each instance's materialization can be bundled into its candidate or represented by consistent legal assignment variables. Several internal nodes can have separate assignment slots. Two enabled instances may use the same candidate but have different readers or execution scopes. +Each instance's materialization can be bundled into its candidate or represented by consistent legal assignment variables. Several internal nodes can have separate assignment slots. Two enabled instances may use the same candidate but have different consumers or execution scopes. #### Objective and cost attribution @@ -223,9 +223,9 @@ For fixed-context costs: min Σ_{g,i,c} cost[g,i,c] × x[g,i,c] + plan-level charges ``` -The cost belongs to the instance's local work/state, excluding input instances. A KLL-state instance owns build and retention; estimator instances own extraction. A reused merge result also has its own instance. Two readers sharing one state instance incur its owned charge once; two enabled state instances incur two charges. +The cost belongs to the instance's local work/state, excluding input instances. A KLL-state instance owns build and retention; estimator instances own extraction. A reused merge result also has its own instance. Two consumers sharing one state instance incur its owned charge once; two enabled state instances incur two charges. -An enabled instance is not necessarily executed only once: query-time work can repeat. Its materialization, reader schedules, phases, range steps, and upstream consumers determine the execution rate and state lifetime. Extend the base model with reach/rate/lifetime variables or finite context configurations constrained to match the selected graph. Do not assign a fixed cost independent of that context. Raw retention and other non-computation charges remain shared resource/accounting variables. +An enabled instance is not necessarily executed only once: query-time work can repeat. Its materialization, consumer schedules, phases, range steps, and upstream consumers determine the execution rate and state lifetime. Extend the base model with reach/rate/lifetime variables or finite context configurations constrained to match the selected graph. Do not assign a fixed cost independent of that context. Raw retention and other non-computation charges remain shared resource/accounting variables. ### 5.2 Constraints @@ -249,7 +249,7 @@ b[g,i,c,e,j] ≤ u[h,j] b[g,i,c,e,j] ≤ Σ_{d ∈ K(g,c,e)} x[h,j,d] ``` -Several consumers can choose the same `j`; other consumers can choose another instance of the same group. Compatibility covers state family, grouping, reduction, window coverage, capacity, timing, and freshness. If two readers attach to one instance, its selected candidate must satisfy both readers. +Several consumers can choose the same `j`; other consumers can choose another instance of the same group. Compatibility covers state family, grouping, reduction, window coverage, capacity, timing, and freshness. If two consumers attach to one instance, its selected candidate must satisfy both consumers. Every enabled instance must have a selected root reference or incoming input edge: @@ -268,7 +268,7 @@ This orders producers before their selected consumers. Interchangeable slots may #### Fusion and available outputs -A fused candidate can eliminate internal computation boundaries without exposing those intermediate results. In the base model it implements its owning group; only explicitly available outputs can serve external readers. For example, a fused `topk(sum(...))` does not automatically supply another query's full `sum(...)` result. That query can require a separate sum instance. Multi-output fusion needs explicit output mappings before it can satisfy several groups; absorption alone is not coverage. +A fused candidate can eliminate internal computation boundaries without exposing those intermediate results. In the base model it implements its owning group; only explicitly available outputs can serve external consumers. For example, a fused `topk(sum(...))` does not automatically supply another query's full `sum(...)` result. That query can require a separate sum instance. Multi-output fusion needs explicit output mappings before it can satisfy several groups; absorption alone is not coverage. #### Accuracy, materialization, and resources @@ -305,7 +305,7 @@ G_input[i0] → G_KLL[i1]: build KLL state Illustrative costs per common horizon, assuming all listed candidates satisfy requirements: -| Choice | Build cost | Reader costs | Total | +| Choice | Build cost | Consumer costs | Total | | --- | ---: | ---: | ---: | | Separate KLL builds | 8 + 8 | 1 + 1 | 18 | | Shared KLL build | 8 | 1 + 1 | 10 | @@ -315,23 +315,23 @@ The shared choice enables one state instance, and both estimator instances conne ```text u[G_KLL,1] = x[G_KLL,1,build-s] = 1 -x[G95,1,KLL-reader] = x[G99,1,KLL-reader] = 1 -b[G95,1,KLL-reader,state-input,1] = 1 -b[G99,1,KLL-reader,state-input,1] = 1 +x[G95,1,KLL-estimator] = x[G99,1,KLL-estimator] = 1 +b[G95,1,KLL-estimator,state-input,1] = 1 +b[G99,1,KLL-estimator,state-input,1] = 1 ``` The build cost is owned by `G_KLL[1]`; each estimator instance owns its extraction cost. If accuracy requires larger state, the solver compares compatible shared capacity, local states, and exact candidates. Quantile level alone does not determine capacity. -With four readers, the same group can also have two shared instances: +With four consumers, the same group can also have two shared instances: ```text -G_KLL[1]: small-capacity query-time state ← readers A, B -G_KLL[2]: large-capacity query-time state ← readers C, D +G_KLL[1]: small-capacity query-time state ← consumers A, B +G_KLL[2]: large-capacity query-time state ← consumers C, D ``` -Both can be enabled if their combined cost is lower or one instance cannot satisfy all requirements. Their selected candidates can differ in capacity, materialization, retention, and execution frequency. Each instance must satisfy every attached reader; readers do not all have to share one configuration. +Both can be enabled if their combined cost is lower or one instance cannot satisfy all requirements. Their selected candidates can differ in capacity, materialization, retention, and execution frequency. Each instance must satisfy every attached consumer; consumers do not all have to share one configuration. -For ingestion-time reuse, a supported implementation maintains 1m KLL panes and merges the latest 15 at evaluation. Both readers can share panes and an eligible merge result. A direct whole-window sketch cannot be assumed to delete expired samples. Different window lengths or pane widths require separate compatibility and coverage checks. +For ingestion-time reuse, a supported implementation maintains 1m KLL panes and merges the latest 15 at evaluation. Both consumers can share panes and an eligible merge result. A direct whole-window sketch cannot be assumed to delete expired samples. Different window lengths or pane widths require separate compatibility and coverage checks. ## 7. Search-Space Reduction and Validation @@ -339,7 +339,7 @@ Reduce search without losing valid joint choices: 1. Intern duplicate identities and cache generation with complete relevant context. 2. Remove provably invalid candidates early; apply cost dominance at Stage 3 only when external sharing interfaces and resource obligations are preserved. -3. Split coupling components only when all possible sharing, dependencies, raw retention, and constraints separate. Global capacities or calibration can prevent independent optimization. +3. Split independent workload components only when all possible sharing, dependencies, raw retention, and constraints separate. Global capacities or calibration can prevent independent optimization. 4. Use a validated baseline incumbent and solver time/gap limits. Restricted parameter inventories narrow the optimality claim; enrichment requires solving the expanded inventory. Define a finite rule/configuration inventory and process new groups/candidates through a worklist, including newly enabled matches with existing candidates, until no eligible rule produces a new item. Deduplicate rule applications and do not discard candidates merely because they are more expensive alone. If generation stops early, report the restricted inventory rather than claiming rule closure. @@ -352,17 +352,19 @@ A complete MILP does not require LBBD. Current code uses a lower-bound model wit ## 8. Implementation and Evaluation -Reuse current structures rather than create a parallel optimizer framework: +Reuse existing structures rather than create a parallel optimizer framework; proposed names and extensions are marked below: | Structure | Responsibility | | --- | --- | | `MemoGroup`, `Memo` | Result/state contracts, dependencies, roots, and per-group candidates. | | `ImplementationCandidate` | Local nodes, fusion absorption/output metadata, child groups, contracts, and reuse requirements. | -| `CouplingSite` and thresholds | Existing shared-configuration representation; migrate state choices into intermediate-group candidates, retaining genuinely cross-group constraints. | -| MILP `Choice` and proposed instance records | Instance identity, selected physical/materialization assignment, reader connections, and cost context. | +| `SharedDecision`, `SharedDecisionKind` (proposed names) | Configuration choices coordinated across candidates; migrate state-local choices into instance candidates and retain cross-instance decisions where needed. | +| MILP `Choice` and proposed instance records | Instance identity, selected physical/materialization assignment, consumer connections, and cost context. | | `PhysicalASAPDAG`, `Selection`, reports | Selected output, validation, bounds, and search status. | -This is an extension of the current representation: shared builds are primarily coordinated through coupling sites today. Introduce intermediate-state contracts, finite instance slots, and reader compatibility; migrate state choices into instance candidates and assign costs to physical instances. This requires changes to generation, selection, decoding, and pricing; it is not merely renaming coupling sites. Raw retention and global capacities remain accounting constraints. +`SharedDecision` replaces the design term `CouplingSite`; `SharedDecisionKind` replaces `CouplingKind`. Current Rust code still uses the latter names. A shared decision has an identity and kind, with legal configurations and compatibility checks. Use **configuration**, rather than **threshold**, because pane layouts and scan-label sets need not be scalar or ordered. This is a proposed naming change, not a claim that the Rust types have already been renamed. + +This design extends the current shared-decision representation. Introduce intermediate-state contracts, finite instance slots, and consumer compatibility; migrate state choices into instance candidates and assign costs to physical instances. This requires changes to generation, selection, decoding, and pricing; it also changes behavior beyond the proposed type rename. Raw retention and global capacities remain accounting constraints. Current main places memo/rules in `search/memo` and solver code in `search/milp`, with Stage 2 physical generation and Stage 3 pricing remaining authoritative. `Legacy` is the default; `MemoBruteForce` is the reference and `MemoMilp` the solver path. Early memo construction, unified shared-state groups with selectable physical instances, broader cost-model support, EH/direct sliding algorithms, and arbitrary pane hierarchies are proposed extensions. @@ -370,14 +372,14 @@ Current main places memo/rules in `search/memo` and solver code in `search/milp` | Phase | Deliverable and acceptance criterion | | --- | --- | -| 1. Memo generation | Move identity construction before Pass 1 and generate candidates directly into groups without losing semantics, reader obligations, or fusion. | -| 2. Shared-state groups | Extend group contracts; generate intermediate groups, instance slots, and compatible reader connections with legal materializations and per-instance cost ownership. | -| 3. Joint selection | Encode covering-DAG activation, implementations, assignments, and demand; match an independent tiny-instance enumerator under the same inventory/models. | +| 1. Memo generation | Move identity construction before Pass 1 and generate candidates directly into groups without losing semantics, consumer obligations, or fusion. | +| 2. Shared-state groups | Extend group contracts; generate intermediate groups, instance slots, and compatible consumer connections with legal materializations and per-instance cost ownership. | +| 3. Joint selection | Encode instance activation, implementation selection, input connections, and demand to cover all queries; match an independent tiny-instance enumerator under the same inventory/models. | | 4. Validation and scaling | Validate reconstructed plans, prove safe reductions, and measure generation/model/solve costs under budgets. | ### Evaluation -**Correctness:** test identity versus sharing, producer compatibility, exact-once cost ownership, active dependencies, fusion with external intermediate readers, multiple input edges, multiple configurations/partial-sharing subsets of one group, zero-price resources, pane boundaries, legal materializations, end-to-end accuracy, and capacity/status handling. Validate any relaxed bounds and cuts against enumerated plans. +**Correctness:** test identity versus sharing, producer compatibility, per-instance cost attribution, active dependencies, fusion with external intermediate consumers, multiple input edges, multiple configurations/partial-sharing subsets of one group, zero-price resources, pane boundaries, legal materializations, end-to-end accuracy, and capacity/status handling. Validate any relaxed bounds and cuts against enumerated plans. **Optimality:** compare MILP with an independent tiny-instance enumerator covering instance activation, candidate selection, root/input assignments, and dependency validity under the same inventory/models. The existing group-level brute-force reference must be extended before it can verify arbitrary instance partitions. Test candidate-generation coverage separately, including cases where two independently shared instances outperform full sharing and separate computation. Use an explicit numerical/tie policy. From 436a04e04c1764bded4274c64d954068971839a9 Mon Sep 17 00:00:00 2001 From: zzylol <50204836+zzylol@users.noreply.github.com> Date: Sat, 24 Oct 2026 00:44:07 +0000 Subject: [PATCH 39/53] docs: clarify instance notation and shared resource charges --- .../proposals/joint-planning-search.md | 53 ++++++++++++++----- 1 file changed, 39 insertions(+), 14 deletions(-) diff --git a/docs/design_docs/proposals/joint-planning-search.md b/docs/design_docs/proposals/joint-planning-search.md index f60308e39..923f4c28d 100644 --- a/docs/design_docs/proposals/joint-planning-search.md +++ b/docs/design_docs/proposals/joint-planning-search.md @@ -178,7 +178,8 @@ Give every charge one owner: - **Producer:** build/update, maintenance, and retained state. - **Consumer:** local merge, extraction, read, and finalization. - **Other physical work:** local operators and temporary state. -- **Shared accounting:** per-source raw retention and other plan-level charges. +- **Shared resources:** raw-data retention charged once per source. +- **Query overhead:** configured work charged once per request. Charge shared work once per actual execution or maintained state, not once per query path. Consumer-specific work remains separately charged. Use a common horizon or consistent per-second rates; distinguish memory/storage occupancy over time from peak capacity. Producer costs can depend on selected consumers, evaluation schedules, upstream materialization, and retention. Model that context rather than assuming every producer has a constant cost. @@ -192,17 +193,38 @@ The formulation selects a covering execution DAG. Groups represent computation c #### Sets and notation -| Symbol | Meaning | +First distinguish **how many copies may exist** from **how each copy is implemented**. Consider a group `G_KLL` representing KLL state for a specified input and window: + +```text +Potential instances: P_G_KLL = {1, 2} +Candidate implementations: C_G_KLL = {small-KLL, large-KLL} + +One possible selection: + instance 1 → small-KLL → consumers A, B + instance 2 → large-KLL → consumers C, D +``` + +The instance numbers carry no implementation choice. They are potential copies that the solver may enable or leave unused. Each enabled instance selects one candidate from the same implementation set; two instances may also select the same candidate. Thus `P_g` bounds the number of copies, while `C_g` defines the supported ways to implement each copy. + +| Set | Meaning | | --- | --- | -| `Q`, `G` | Queries and memo groups, including intermediate states. | -| `P_g` | Finite physical-instance slots for group `g`. | -| `C_g` | Legal physical candidates, including state configuration and materialization assignment. | -| `R(g,c)` | Indexed input-edge occurrences required by candidate `(g,c)`. | -| `H(g,c,e)` | Group required by input edge `e ∈ R(g,c)`. | -| `K(g,c,e)` | Candidates of that input group compatible with this edge. | -| `T(q)` | Query-root references; each reference `t` identifies group `root(q,t)`. | +| `Q` | The workload queries. | +| `G` | Memo groups: result computations and reusable intermediate states. | +| `P_g` | Indices of potential physical instances of group `g`, such as `{1,2}`. An index identifies a copy, not its algorithm or configuration. | +| `C_g` | Supported implementation candidates for group `g`. Each candidate specifies its computation, configuration, input dependencies, and legal materialization assignment. | -Input edges reference groups but connect to particular instances. Preserve distinct edge occurrences even when two inputs reference the same group. Capacity, pane width, retention, and materialization are candidate configurations, not separate producer decisions. Finite configurations permit lookup-table costs without assuming a linear size/cost function. +Candidates refer to input **groups**; the solver later chooses the specific input **instances**. The following notation describes those dependencies: + +| Notation | Meaning | +| --- | --- | +| `R(g,c)` | Input positions needed by candidate `c` of group `g`. Keep two positions distinct even if both require the same group. | +| `H(g,c,e)` | The group whose output is needed at input position `e`. | +| `K(g,c,e)` | Candidates of that input group that can supply the required output, configuration, and guarantees. | +| `T(q)` | References to computation results needed to produce query `q`'s output. `root(q,t)` identifies the group for reference `t`. A scalar query can reference several computation groups. | + +For example, a KLL estimator has an input position `state`. `H(G95, estimator, state) = G_KLL`; `K(G95, estimator, state)` lists the KLL-state candidates that meet this estimator's requirements. Its input may connect to instance 1 or 2, provided that instance selects a compatible candidate. Likewise, a selected KLL build has its own input position referring to the input-data group. + +Capacity, pane width, retention, and materialization are candidate configurations. Finite configurations permit lookup-table costs without assuming a linear size/cost function. #### Decision variables @@ -220,12 +242,15 @@ Each instance's materialization can be bundled into its candidate or represented For fixed-context costs: ```text -min Σ_{g,i,c} cost[g,i,c] × x[g,i,c] + plan-level charges +min Σ_{g,i,c} cost[g,i,c] × x[g,i,c] + shared_resource_cost + query_overhead ``` The cost belongs to the instance's local work/state, excluding input instances. A KLL-state instance owns build and retention; estimator instances own extraction. A reused merge result also has its own instance. Two consumers sharing one state instance incur its owned charge once; two enabled state instances incur two charges. -An enabled instance is not necessarily executed only once: query-time work can repeat. Its materialization, consumer schedules, phases, range steps, and upstream consumers determine the execution rate and state lifetime. Extend the base model with reach/rate/lifetime variables or finite context configurations constrained to match the selected graph. Do not assign a fixed cost independent of that context. Raw retention and other non-computation charges remain shared resource/accounting variables. +`shared_resource_cost` accounts for resources owned outside computation instances, such as retained raw data charged once per source. If selected scans require 5m and 15m of the same raw source, that source's required horizon is 15m, not 20m; derive its storage cost from a retention variable constrained by the selected scans. Intermediate-state retention already charged to an instance is excluded here. `query_overhead` is any configured once-per-request charge, counted at each query's request rate and excluded from instance costs; it is zero when the model has no such charge. Both terms must have explicit model expressions, not arbitrary residual adjustments after solving. + + +An enabled instance is not necessarily executed only once: query-time work can repeat. Its materialization, consumer schedules, phases, range steps, and upstream consumers determine the execution rate and state lifetime. Extend the base model with reach/rate/lifetime variables or finite context configurations constrained to match the selected graph. Do not assign a fixed cost independent of that context. Raw retention uses shared resource variables; request overhead uses query-level demand. ### 5.2 Constraints @@ -281,10 +306,10 @@ For additive, fixed-context resource usage: ```text Σ_{g,i,c} usage[k,g,i,c] × x[g,i,c] - + plan-level usage[k] ≤ capacity[k] + + shared_resource_usage[k] ≤ capacity[k] ``` -Peak capacity is not average occupancy; temporary lifetimes and concurrency need their own accounting. Use bounded auxiliary-variable linearizations where needed. A sum of fixed candidate costs is only the base case, not a substitute for the configured evaluator. +`shared_resource_usage[k]` covers non-instance resources such as per-source raw-data storage, without repeating instance-owned usage. Peak capacity is not average occupancy; temporary lifetimes and concurrency need their own accounting. Use bounded auxiliary-variable linearizations where needed. A sum of fixed candidate costs is only the base case, not a substitute for the configured evaluator. ## 6. Worked Example: Shared KLL for P95/P99 From ad24a0e26911e8d1418cad89c68b8599cd23914d Mon Sep 17 00:00:00 2001 From: zzylol <50204836+zzylol@users.noreply.github.com> Date: Sat, 24 Oct 2026 00:55:20 +0000 Subject: [PATCH 40/53] docs: simplify memo design to candidate selection and connections --- .../proposals/joint-planning-search.md | 403 ++++++------------ 1 file changed, 122 insertions(+), 281 deletions(-) diff --git a/docs/design_docs/proposals/joint-planning-search.md b/docs/design_docs/proposals/joint-planning-search.md index 923f4c28d..92a1dfce8 100644 --- a/docs/design_docs/proposals/joint-planning-search.md +++ b/docs/design_docs/proposals/joint-planning-search.md @@ -1,411 +1,252 @@ # ASAPPlanner: Memo-Based Joint Optimization with MILP -**Status:** Design proposal - -**Scope:** Multi-query optimization across exact operators, ASAP primitives, window implementations, sharing, and materialization. +**Status:** Design proposal. The staged memo and selection model below includes proposed extensions. ## 1. Motivation and Challenges -ASAP primitives create opportunities to replace expensive exact computations and share summaries, panes, rollups, scans, or retained raw data. They also enlarge the optimization search space. Enumerating complete workload candidates multiplies implementation, sharing, window, and materialization choices. Optimizing queries independently avoids that enumeration but can miss joint savings. - -The challenge is cross-query dependence: a candidate's incremental cost changes when another candidate enables a shared producer, requires a larger summary, or extends retention. Selecting the cheapest candidate for each group independently can therefore lose the workload optimum. +ASAP primitives offer more ways to compute and share query results: exact operators, summaries, panes, rollups, and scans. These choices enlarge the search space and couple query costs. A computation that is expensive for one query may be economical when shared by several queries. -This doc aims to address the problem of finding a minimum-cost physical execution DAG for a workload of queries under the deployment's cost model. The plan must produce every requested result, satisfy each query's accuracy and latency requirements, respect resource limits, and account for shared computation and retained state without duplicate cost charging. This requires jointly deciding which supported implementations to use, which computations to share, and when to materialize their outputs, while keeping ASAPPlanner's planning time and planning runtime memory reasonable. Optimality is relative to the supported finite candidate space and configured models. Developing new sketch/window algorithms and runtime deployment is outside the scope. +The problem is to find a minimum-cost execution DAG covering every query, subject to accuracy, latency, and resource requirements, while keeping planning time and memory manageable. Shared computation and retained data must not be charged repeatedly along different query paths. Optimality is limited to the generated finite candidate set and configured models. New sketch/window algorithms and runtime scheduling are outside this proposal. ## 2. Traditional Memo Groups and Rule Expansion -A traditional memo, as used by Cascades-style optimizers, compactly represents equivalent query expressions. A **memo group** contains expressions that produce the same logical result under the query semantics. Each expression references child groups instead of copying complete input plans. Different parent expressions can therefore reuse the same search work. - -### Logical rule expansion +A **memo group** contains expressions producing the same logical result. Expressions reference input groups rather than copying complete input plans. Rules add equivalent expressions and physical candidates; the optimizer compares their costs. -Consider an inner join with a filter on `A`: +### Logical rules -```sql -SELECT * -FROM A JOIN B ON A.id = B.id -WHERE A.x > 10; -``` - -Initially, the memo can contain: +Consider `Filter(A.x > 10, Join(A, B))`, where the inner join allows this filter to move onto `A`. ```text -G1 = { Scan(A) } -G2 = { Scan(B) } -G3 = { Join(G1, G2, A.id = B.id) } -G5 = { Filter(A.x > 10, G3) } query result -``` +Initially: + G_A = { Scan(A) } + G_B = { Scan(B) } + G_join = { Join(G_A, G_B) } + G_out = { Filter(A.x > 10, G_join) } -A filter-pushdown rule matches the filter above the inner join. Because the predicate depends only on `A` and pushing it down preserves semantics, it adds a filtered-input group and another expression to the existing result group: - -```text -G4 = { Filter(A.x > 10, G1) } - -G5 = { - Filter(A.x > 10, G3), - Join(G4, G2, A.id = B.id) -} +After filter pushdown: + G_filtered = { Filter(A.x > 10, G_A) } + G_out += Join(G_filtered, G_B) ``` -Both members of `G5` produce the same result. The rule expands the group rather than immediately replacing its existing expression. Newly introduced computations, such as the filtered `A`, can create new groups. Further rules can match newly inserted expressions; interning avoids repeating identical expressions. Exploration may be demand-driven and interleaved with costing rather than saturating every possible rewrite. +The new expression belongs to `G_out` because it produces the same result. The filtered input gets its own group. Further rules can expand newly added expressions; interning avoids duplicates. -### Physical implementation expansion +### Physical rules -Implementation rules turn logical expressions into physical execution candidates. For the join expression in `G5`, eligible candidates can include: +Implementation rules add supported physical candidates for a logical expression: ```text -HashJoin(G4, G2) -MergeJoin(G4, G2) -NestedLoopJoin(G4, G2) +Join(G_filtered, G_B) + → HashJoin(G_filtered, G_B) + → MergeJoin(G_filtered, G_B) + → NestedLoopJoin(G_filtered, G_B) ``` -All implement the same logical join result, but impose different input requirements and costs. A merge join may require inputs ordered by the join key; a hash join does not require that order. Physical expressions continue to reference child groups, whose implementations are chosen separately. The precise storage of logical and physical expressions varies by optimizer. +Candidates have different costs and input requirements. For example, merge join requires ordered inputs. The optimizer typically caches `Best(group, required_properties)` and combines local operator cost with child costs. -The optimizer typically searches and caches: - -```text -Best(group, required_physical_properties) -``` - -For each physical candidate, it derives child requirements, optimizes the child groups under those requirements, and combines their costs with the candidate's own cost. The cheapest implementation for unordered output may differ from the cheapest one for ordered output; sorting or other property-enforcing operators may be needed. Rules generate candidates; costing and valid bounds guide exploration and selection. - -Sharing a memo group shares optimization representation, not necessarily runtime computation. Explicit materialization/reuse decisions are still needed to execute a common producer once. This distinction becomes central to ASAPPlanner's cross-query optimization below. +A shared memo group avoids repeated optimization work. It does not by itself mean that execution computes the result once and shares it. ## 3. ASAPPlanner Memo Groups and Rule Expansion -Given the benefits of Memo groups, ASAPPlanner should introduce the Memo group while representing summaries and reuse strategies. +ASAPPlanner uses two concepts: -In ASAPPlanner, we extend the definition of a **memo group**, whose output can be a query or intermediate relational result, or reusable intermediate state. A result group preserves input, filters, grouping, window semantics, and requested output. A state group additionally specifies its state family/interface and guarantees. **Implementation candidates** provide exact computation, accumulators, or summaries and reference input groups. Physical candidates add legal operators and materialization assignments. A **physical instance** is one enabled realization of a group, with its own selected candidate, materialization, consumer set, and lifetime. Several instances of the same group can coexist. A **shared producer** is an instance read by several consumers, not a separately selected catalog entry. +- **Group:** a requested result or intermediate-state contract, including input, filters, grouping, and window semantics. State groups also specify the state interface and guarantees. +- **Candidate:** a supported way to produce that group's output. A physical candidate specifies operators, configuration, materialization, execution scope, and required inputs. -Unlike exact expressions in a traditional equivalence group, approximate candidates target the same requested result under explicit accuracy obligations. State candidates must satisfy their group's contract, and consumers can impose stricter compatibility requirements. One group identity neither forces shared execution nor limits the plan to one physical instance. A stage-level candidate remains a whole-workload DAG; local group candidates compactly represent its component choices. +Approximate candidates target the requested result subject to accuracy requirements; they are not asserted to return exactly the same values as exact computation. **Several physical candidates in one group may be selected.** Each selected candidate becomes a node in the execution DAG. Consumers connect to specific selected candidates; sharing means using the same node. -The following example traces how the proposed memo expands: +For example: ```text -Q1: quantile_over_time(0.95, X[15m]), every 1m -Q2: quantile_over_time(0.99, X[15m]), every 1m +G_KLL + C_small: small-capacity KLL ← consumers A, B + C_large: large-capacity KLL ← consumers C, D ``` -### Stage 0 and initial memo: establish logical identity +Both candidates may be selected. There is no separate physical-instance index. If two computations need separate execution scopes or incompatible input bindings, generation must provide distinct candidates that can coexist. A single selected candidate cannot have different bindings for different consumers. This is a candidate-generation requirement, not an assumption that every duplicate computation can safely be shared. + +The workflow follows [planner layering](planner-layering.md). Local group candidates represent components of the workload DAGs described there. + +### Stage 0: build initial groups -Frontends produce language-independent `LogicalDAGs`. Build groups for distinct logical sub-expressions and record query roots and consumer requirements. Intern eligible identical inputs only after semantic equality and sharing legality are established; preserve keyless-node rules. P95 and P99 are different computations, so their result groups remain separate: +Parse and normalize the workload, intern equivalent logical sub-expressions, and record query roots. ```text -G_input: common eligible input computation X -G95: P95 over G_input in the last 15m root of Q1 -G99: P99 over G_input in the last 15m root of Q2 +Q1: P95(X[15m]), every 1m +Q2: P99(X[15m]), every 1m + +G_input: input X +G95: P95 over X[15m] root of Q1 +G99: P99 over X[15m] root of Q2 ``` -Current memo construction receives Pass 1 candidates. Building initial identities before Pass 1 is the proposed workflow, allowing candidate generation to reuse group identity from the start. +P95 and P99 remain different groups. Initial grouping before Pass 1 is proposed; current memo construction receives Pass 1 candidates. -### Stage 1, Pass 1: add per-computation candidates +### Stage 1, Pass 1: expand computation candidates -Apply computation rules to each group independently. In this example, each group receives an exact candidate and supported KLL candidates: +Apply supported computation rules per group: ```text -G95 = { ExactQuantile(0.95, G_input, 15m), LocalKLLQuantile(0.95, G_input, 15m, s) } -G99 = { ExactQuantile(0.99, G_input, 15m), LocalKLLQuantile(0.99, G_input, 15m, s) } +G95 = { ExactQuantile(0.95, G_input), LocalKLLQuantile(0.95, G_input, s) } +G99 = { ExactQuantile(0.99, G_input), LocalKLLQuantile(0.99, G_input, s) } ``` -Here `s` ranges over generated capacities; each candidate records its accuracy contract and dependencies. Other computations can receive exact accumulator or fused candidates. Cache generation with relevant consumer requirements and avoid composing the Cartesian product of complete workload plans. +Here the window remains 15m, and `s` ranges over generated capacities. Each candidate records dependencies and accuracy obligations. Rules can add groups and candidates without enumerating complete workload combinations. -### Stage 1, Pass 2: add intermediate groups and sharing candidates +### Stage 1, Pass 2: expose shared state -ASAP-aware CSE recognizes that compatible KLL state supports both quantiles. Introduce or reuse a state group and add consumer candidates: +ASAP-aware CSE recognizes that both quantiles can consume compatible KLL state: ```text G_KLL: KLL state over X[15m] - candidates: BuildKLL(G_input, window=15m, capacity=s) + candidates: BuildKLL(G_input, capacity=s) G95 += EstimateQuantile(0.95, G_KLL) G99 += EstimateQuantile(0.99, G_KLL) ``` -The result groups are not merged. Their new candidates depend on the same state group. If they attach to one physical instance, that instance's selected capacity must satisfy both; they can instead attach to different instances with different configurations. Independent candidates remain available. Conventional CSE can similarly reuse identical computations; ASAP-aware rules also expose window composition, rollup, top-k reuse, and scan widening. +Conventional CSE discovers identical computations. ASAP-aware CSE also exposes common intermediate state for different results. It creates or reuses a state group without merging `G95` and `G99`. -For pane-based reuse, window-composition rules can add: +Window rules can similarly introduce a pane-state group and a merge candidate: ```text -G_panes: chain of aligned 1m KLL panes over G_input, covering 15m +G_panes: aligned 1m KLL panes covering 15m G_KLL += MergeLatest15(G_panes) ``` -This is another way to produce compatible window state. Other generated pane widths or compositions require coverage, merge, and accuracy checks. Groups can be added throughout rule expansion; they are not fixed after Stage 0. - -### Stage 2: add physical and materialization candidates - -Generate supported physical operators and assignments for these candidates. For example, an eligible `G_panes` implementation can maintain panes at ingestion while merge and estimation execute at query time. Query-time construction and supported retention modes remain candidates where legal. - -Stage 2 checks deployment capabilities, ingestion eligibility, and upstream timing. It does not choose the cheapest candidate. Assignments can differ between internal nodes. Prepare finite instance slots for each group so consumers can share one instance, form several sharing subsets, or compute separately; do not enumerate all consumer partitions as complete plans. Consumer-dependent eligibility must use the selected instance's consumers; defer unknown joint checks instead of pruning a mode based on every potential consumer of the logical group. - -### Stage 3: select a joint physical plan +### Stage 2: expand physical candidates -Stage 3 covers all query roots by choosing which group instances to enable, one candidate per enabled instance, and the supplier instance for every selected input edge and query-root reference. It may select exact/local computation, a shared whole-window build, or a legal pane-based candidate. It enforces consumer/state compatibility, accuracy, latency, and resource constraints, then reconstructs and validates one `PhysicalASAPDAG`. +Enumerate executable operators, configurations, and materialization assignments. For example, maintain eligible panes at ingestion and merge them at evaluation. Retain candidate-specific compatibility and timing requirements; checks depending on actual consumers must use the selected connections. -Each physical instance owns its work/state charges; an instance shared by several consumers is not charged once per consumer. A higher standalone cost can enable a cheaper joint plan, so earlier stages retain valid candidates and Stage 3 performs cost-based selection. +Window semantics, algorithms, and materialization remain separate dimensions. Their definitions and support restrictions belong in [Window and Materialization Choices](window-materialization-choices.md). -Window semantics, implementation candidates, and materialization are defined separately in [Window and Materialization Choices](window-materialization-choices.md). That design also describes pane/EH terminology, supported configurations, and materialization eligibility. +### Stage 3: select and connect candidates -## 4. Cost Model and Optimization Objective +Select physical candidates, connect each required input to a compatible selected candidate, and cover every query root. Several consumers may share one KLL candidate or use different candidates from `G_KLL`. Reconstruct and validate the resulting `PhysicalASAPDAG`. -### Covering queries with an execution DAG +## 4. Cost Model -A query is covered only when its root references connect to enabled instances, every required input is recursively supplied by a compatible instance, and the resulting DAG satisfies accuracy, timing, and deployment constraints. Selecting a KLL build alone does not cover a quantile query: the estimator and the build's upstream dependencies must also be provided. +Charge each selected node for its local work and state, excluding its input nodes. A KLL build owns construction and retained state; an estimator owns extraction. Sharing a build avoids repeating its charge, while each estimator still incurs its own cost. -The decisions are instance activation, implementation selection, and input assignment. Sharing occurs when several consumers connect to the same instance. The objective is the total cost of those instances under their assigned demand, not the sum of independently priced query paths. +Costs use a common horizon or consistent rates. A selected node can execute repeatedly: materialization, evaluation schedules, and selected consumers determine its execution frequency and lifetime. A constant candidate cost is valid only when that context is fixed. -### Why MILP rather than bipartite matching +Shared raw-data retention has one owner per source. Consumers needing 5m and 15m of the same source require 15m retention, not 20m. Add configured query-request overhead separately. Do not double-count state already owned by a candidate. -A bipartite compatibility graph captures which suppliers can serve each input, but ordinary matching cannot express the whole problem. Suppliers can serve multiple consumers, enabling a supplier has a pay-once cost, candidates can require several inputs together, and suppliers themselves depend on upstream instances. Accuracy and deployment limits further couple the choices. - -For example, enabling a KLL build costs 8 and each of two estimators costs 1. Sharing costs 10. Charging the build on both assignment edges incorrectly gives 18; omitting it leaves activation unpriced. Allowing multiple assignments through b-matching or min-cost flow does not by itself model that activation cost or the dependency DAG. Facility location captures activation and assignment more closely, but must still be extended for upstream dependencies and demand-dependent costs. - -MILP is the proposed joint solver, not a claim that it is the only exact method. Matching or min-cost flow can solve a restricted assignment subproblem after supplier activation, implementations, rates, and costs are fixed, provided its remaining capacities and constraints admit that formulation. Without assignment coupling, choosing each input's cheapest compatible supplier may suffice. Any local assignment must preserve the global demand and feasibility assumptions. - -### Resource pricing and demand - -Attribute resource usage to physical nodes in the selected DAG, then apply the existing Stage 3 cost model: - -| Model | Selection policy | -| --- | --- | -| `UsageOverTime(ResourceUnitCost)` | Minimize priced usage per second. | -| `ProvisionedCapacity(ResourceUnitCost)` | Minimize priced provisioned memory, cores, and disk. | -| `ClusterCapacity(ClusterLimits)` | Enforce capacities and use `LowestLatency`, `LeastMemory`, or `ParetoFront`. | - -Current memo MILP supports usage-over-time pricing; unsupported models route to legacy search. Broader MILP coverage is an extension. A Pareto objective requires frontier computation rather than one scalar solve. - -Give every charge one owner: - -- **Producer:** build/update, maintenance, and retained state. -- **Consumer:** local merge, extraction, read, and finalization. -- **Other physical work:** local operators and temporary state. -- **Shared resources:** raw-data retention charged once per source. -- **Query overhead:** configured work charged once per request. - -Charge shared work once per actual execution or maintained state, not once per query path. Consumer-specific work remains separately charged. Use a common horizon or consistent per-second rates; distinguish memory/storage occupancy over time from peak capacity. Producer costs can depend on selected consumers, evaluation schedules, upstream materialization, and retention. Model that context rather than assuming every producer has a constant cost. - -Accuracy is an end-to-end root requirement. Compose summary and window error only with valid contracts; locally admissible sketches do not prove a composed query accurate. Latency and resource limits must use the configured evaluator's timing and concurrency assumptions. Current latency accounting sums reached query-time work plus applicable compaction; this proposal does not silently replace it with a general parallel scheduler. Keep calibrated model inputs fixed during a solve. +Use the configured Stage 3 objective and evaluators. Current memo MILP supports usage-over-time pricing; broader cost-model support is an extension. Accuracy is checked at query outputs, and latency and resource limits must respect the configured timing model. ## 5. MILP Formulation -The formulation selects a covering execution DAG. Groups represent computation contracts; instances represent physical realizations. A group can have several enabled instances with different candidates. Each enabled instance chooses exactly one candidate. Reuse is expressed by multiple consumers connecting to the same instance, without a separate producer catalog. +MILP jointly selects candidates and their connections. Ordinary bipartite matching does not capture pay-once producer activation, multiple consumers, and recursively required inputs. The formulation below is a fixed-context base; demand-dependent costs require corresponding rate/lifetime variables or finite context configurations. ### 5.1 Variables #### Sets and notation -First distinguish **how many copies may exist** from **how each copy is implemented**. Consider a group `G_KLL` representing KLL state for a specified input and window: - -```text -Potential instances: P_G_KLL = {1, 2} -Candidate implementations: C_G_KLL = {small-KLL, large-KLL} - -One possible selection: - instance 1 → small-KLL → consumers A, B - instance 2 → large-KLL → consumers C, D -``` - -The instance numbers carry no implementation choice. They are potential copies that the solver may enable or leave unused. Each enabled instance selects one candidate from the same implementation set; two instances may also select the same candidate. Thus `P_g` bounds the number of copies, while `C_g` defines the supported ways to implement each copy. - -| Set | Meaning | -| --- | --- | -| `Q` | The workload queries. | -| `G` | Memo groups: result computations and reusable intermediate states. | -| `P_g` | Indices of potential physical instances of group `g`, such as `{1,2}`. An index identifies a copy, not its algorithm or configuration. | -| `C_g` | Supported implementation candidates for group `g`. Each candidate specifies its computation, configuration, input dependencies, and legal materialization assignment. | - -Candidates refer to input **groups**; the solver later chooses the specific input **instances**. The following notation describes those dependencies: - -| Notation | Meaning | +| Symbol | Meaning | | --- | --- | -| `R(g,c)` | Input positions needed by candidate `c` of group `g`. Keep two positions distinct even if both require the same group. | -| `H(g,c,e)` | The group whose output is needed at input position `e`. | -| `K(g,c,e)` | Candidates of that input group that can supply the required output, configuration, and guarantees. | -| `T(q)` | References to computation results needed to produce query `q`'s output. `root(q,t)` identifies the group for reference `t`. A scalar query can reference several computation groups. | +| `G` | Result and state groups. | +| `C_g` | Finite physical candidates belonging to group `g`. | +| `E(c)` | Required input positions of candidate `c`. | +| `S(c,e)` | Compatible supplier candidates for input `e` of candidate `c`. | +| `T` | Required query-output references. | +| `S(t)` | Candidates compatible with output reference `t`. | -For example, a KLL estimator has an input position `state`. `H(G95, estimator, state) = G_KLL`; `K(G95, estimator, state)` lists the KLL-state candidates that meet this estimator's requirements. Its input may connect to instance 1 or 2, provided that instance selects a compatible candidate. Likewise, a selected KLL build has its own input position referring to the input-data group. - -Capacity, pane width, retention, and materialization are candidate configurations. Finite configurations permit lookup-table costs without assuming a linear size/cost function. - -#### Decision variables +Candidate IDs are globally unique; `C` is the union of all `C_g`. Input positions stay distinct even when they request the same group. For a P95 estimator, the `state` input's suppliers are compatible candidates from `G_KLL`. A query may have several output references. ```text -u[g,i] ∈ {0,1} enable instance i of group g -x[g,i,c] ∈ {0,1} instance (g,i) selects candidate c -b[g,i,c,e,j] ∈ {0,1} input edge e connects to instance j of H(g,c,e) -r[q,t,i] ∈ {0,1} root reference t of query q uses instance i +x[c] ∈ {0,1} select physical candidate c +b[c,e,d] ∈ {0,1} input e of c uses supplier candidate d +r[t,c] ∈ {0,1} query-output reference t uses candidate c ``` -Each instance's materialization can be bundled into its candidate or represented by consistent legal assignment variables. Several internal nodes can have separate assignment slots. Two enabled instances may use the same candidate but have different consumers or execution scopes. - -#### Objective and cost attribution +Create connection variables only for compatible pairs. Capacity and materialization are included in candidate configuration; no separate copy or instance variables are needed. -For fixed-context costs: +#### Objective ```text -min Σ_{g,i,c} cost[g,i,c] × x[g,i,c] + shared_resource_cost + query_overhead +min Σ_c local_cost[c] × x[c] + + shared_resource_cost + + query_overhead ``` -The cost belongs to the instance's local work/state, excluding input instances. A KLL-state instance owns build and retention; estimator instances own extraction. A reused merge result also has its own instance. Two consumers sharing one state instance incur its owned charge once; two enabled state instances incur two charges. - -`shared_resource_cost` accounts for resources owned outside computation instances, such as retained raw data charged once per source. If selected scans require 5m and 15m of the same raw source, that source's required horizon is 15m, not 20m; derive its storage cost from a retention variable constrained by the selected scans. Intermediate-state retention already charged to an instance is excluded here. `query_overhead` is any configured once-per-request charge, counted at each query's request rate and excluded from instance costs; it is zero when the model has no such charge. Both terms must have explicit model expressions, not arbitrary residual adjustments after solving. - - -An enabled instance is not necessarily executed only once: query-time work can repeat. Its materialization, consumer schedules, phases, range steps, and upstream consumers determine the execution rate and state lifetime. Extend the base model with reach/rate/lifetime variables or finite context configurations constrained to match the selected graph. Do not assign a fixed cost independent of that context. Raw retention uses shared resource variables; request overhead uses query-level demand. +`local_cost` excludes suppliers. `shared_resource_cost` explicitly models resources such as per-source raw retention; `query_overhead` accounts for configured request charges. Both are zero when absent. If cost depends on selected consumers, model that dependence instead of assigning an incorrect constant. ### 5.2 Constraints -#### Instance selection and query coverage +#### Cover every query output ```text -Σ_{c ∈ C_g} x[g,i,c] = u[g,i] for each instance (g,i) -Σ_{i ∈ P_root(q,t)} r[q,t,i] = 1 for each root reference (q,t) -r[q,t,i] ≤ u[root(q,t),i] +Σ_{c ∈ S(t)} r[t,c] = 1 for each output reference t +r[t,c] ≤ x[c] ``` -There is no constraint limiting an entire group to one enabled instance. Each query-root reference must connect to an available implementation. - -#### Input dependencies and compatibility - -For each input edge of a selected candidate, choose one compatible input instance. Let `h = H(g,c,e)`: +#### Supply every selected candidate's inputs ```text -Σ_{j ∈ P_h} b[g,i,c,e,j] = x[g,i,c] -b[g,i,c,e,j] ≤ u[h,j] -b[g,i,c,e,j] ≤ Σ_{d ∈ K(g,c,e)} x[h,j,d] +Σ_{d ∈ S(c,e)} b[c,e,d] = x[c] for each required input (c,e) +b[c,e,d] ≤ x[d] ``` -Several consumers can choose the same `j`; other consumers can choose another instance of the same group. Compatibility covers state family, grouping, reduction, window coverage, capacity, timing, and freshness. If two consumers attach to one instance, its selected candidate must satisfy both consumers. +Several consumers may connect to the same supplier. There is **no** constraint `Σ_{c ∈ C_g} x[c] ≤ 1`: multiple candidates in one group may be needed. -Every enabled instance must have a selected root reference or incoming input edge: +#### Keep only connected, acyclic computation ```text -u[h,j] ≤ Σ root references using (h,j) - + Σ selected input edges using (h,j) +x[d] ≤ Σ_t r[t,d] + Σ_{c,e} b[c,e,d] ``` -Together with acyclicity, this prevents disconnected instances even when prices are zero. Enforce an acyclic physical instance graph, not a requirement that every possible memo dependency be acyclic. For example, with `N` total slots and order variables `0 ≤ order[g,i] ≤ N−1`: - -```text -order[h,j] + 1 ≤ order[g,i] + N × (1 − b[g,i,c,e,j]) -``` +Only defined connections appear in these sums. Every selected candidate must serve a query output or another selected candidate. Together with acyclicity, this excludes unused computation. -This orders producers before their selected consumers. Interchangeable slots may additionally use symmetry-breaking constraints such as `u[g,i] ≥ u[g,i+1]`. - -#### Fusion and available outputs - -A fused candidate can eliminate internal computation boundaries without exposing those intermediate results. In the base model it implements its owning group; only explicitly available outputs can serve external consumers. For example, a fused `topk(sum(...))` does not automatically supply another query's full `sum(...)` result. That query can require a separate sum instance. Multi-output fusion needs explicit output mappings before it can satisfy several groups; absorption alone is not coverage. - -#### Accuracy, materialization, and resources - -- **Accuracy:** preserve state contracts and enforce root-level composition bounds, feasible-combination tables, or sound refinement over the selected instance graph. -- **Physical validity:** require supported operators, legal materializations, available upstream state, and valid freshness/lifetimes. -- **Windows:** validate alignment, coverage without omission/double counting, retention, and algorithm-specific guarantees. -- **Resources and latency:** apply configured CPU/memory/disk limits, query deadlines, and recurrence checks using actual instance demand and timing. - -For additive, fixed-context resource usage: +If candidate generation does not already guarantee acyclicity, introduce `0 ≤ order[c] ≤ N−1`, where `N = |C|`: ```text -Σ_{g,i,c} usage[k,g,i,c] × x[g,i,c] - + shared_resource_usage[k] ≤ capacity[k] +order[d] + 1 ≤ order[c] + N × (1 − b[c,e,d]) ``` -`shared_resource_usage[k]` covers non-instance resources such as per-source raw-data storage, without repeating instance-owned usage. Peak capacity is not average occupancy; temporary lifetimes and concurrency need their own accounting. Use bounded auxiliary-variable linearizations where needed. A sum of fixed candidate costs is only the base case, not a substitute for the configured evaluator. +#### Enforce requirements -## 6. Worked Example: Shared KLL for P95/P99 +- **Accuracy:** enforce valid end-to-end error bounds or sound feasibility refinement for each query. +- **Physical validity:** require supported operators, legal materialization, compatible timing, and correct window coverage. +- **Resources:** for additive fixed-context usage, constrain `Σ_c usage[k,c] × x[c] + shared_usage[k] ≤ capacity[k]`. Peak and temporary usage require appropriate lifetime/concurrency accounting. +- **Latency:** enforce deadlines under the configured evaluator's timing assumptions. -```text -Q1: quantile_over_time(0.95, stream[15m]), every 1m -Q2: quantile_over_time(0.99, stream[15m]), every 1m -``` - -Using the groups introduced in Section 3, compare the cost of selecting shared versus local KLL candidates: +Only exposed candidate outputs can supply consumers. A fused `topk(sum(...))` candidate does not automatically supply another query's full `sum(...)`; that result needs a separate candidate unless explicitly exposed. -```text -G_input[i0] → G_KLL[i1]: build KLL state - / \ - G95[i1]: estimate P95 G99[i1]: estimate P99 - | | - Q1 Q2 -``` +## 6. Example: Shared KLL -Illustrative costs per common horizon, assuming all listed candidates satisfy requirements: - -| Choice | Build cost | Consumer costs | Total | -| --- | ---: | ---: | ---: | -| Separate KLL builds | 8 + 8 | 1 + 1 | 18 | -| Shared KLL build | 8 | 1 + 1 | 10 | -| Separate exact computations | 12 + 12 | included | 24 | - -The shared choice enables one state instance, and both estimator instances connect to it: +For the P95/P99 workload, a selected plan can be: ```text -u[G_KLL,1] = x[G_KLL,1,build-s] = 1 -x[G95,1,KLL-estimator] = x[G99,1,KLL-estimator] = 1 -b[G95,1,KLL-estimator,state-input,1] = 1 -b[G99,1,KLL-estimator,state-input,1] = 1 +Input candidate → KLL build candidate + ├→ P95 estimator → Q1 + └→ P99 estimator → Q2 ``` -The build cost is owned by `G_KLL[1]`; each estimator instance owns its extraction cost. If accuracy requires larger state, the solver compares compatible shared capacity, local states, and exact candidates. Quantile level alone does not determine capacity. - -With four consumers, the same group can also have two shared instances: - -```text -G_KLL[1]: small-capacity query-time state ← consumers A, B -G_KLL[2]: large-capacity query-time state ← consumers C, D -``` +Illustrative costs over the same horizon, assuming all candidates meet requirements: -Both can be enabled if their combined cost is lower or one instance cannot satisfy all requirements. Their selected candidates can differ in capacity, materialization, retention, and execution frequency. Each instance must satisfy every attached consumer; consumers do not all have to share one configuration. +| Selection | Build | Estimation | Total | +| --- | ---: | ---: | ---: | +| Local KLL computation for each query | 8 + 8 | 1 + 1 | 18 | +| One shared KLL build | 8 | 1 + 1 | 10 | -For ingestion-time reuse, a supported implementation maintains 1m KLL panes and merges the latest 15 at evaluation. Both consumers can share panes and an eligible merge result. A direct whole-window sketch cannot be assumed to delete expired samples. Different window lengths or pane widths require separate compatibility and coverage checks. +Both estimators connect to the same selected build candidate, so its cost appears once. If one configuration cannot serve all consumers economically, select multiple candidates from `G_KLL`, as in the small/large example in Section 3. ## 7. Search-Space Reduction and Validation -Reduce search without losing valid joint choices: - -1. Intern duplicate identities and cache generation with complete relevant context. -2. Remove provably invalid candidates early; apply cost dominance at Stage 3 only when external sharing interfaces and resource obligations are preserved. -3. Split independent workload components only when all possible sharing, dependencies, raw retention, and constraints separate. Global capacities or calibration can prevent independent optimization. -4. Use a validated baseline incumbent and solver time/gap limits. Restricted parameter inventories narrow the optimality claim; enrichment requires solving the expanded inventory. - -Define a finite rule/configuration inventory and process new groups/candidates through a worklist, including newly enabled matches with existing candidates, until no eligible rule produces a new item. Deduplicate rule applications and do not discard candidates merely because they are more expensive alone. If generation stops early, report the restricted inventory rather than claiming rule closure. +Intern duplicate expressions and apply rules to newly generated groups/candidates. Generate only supported configurations and execution scopes. If generation is capped or stops before rule closure, report that restriction. Removing instance indices eliminates interchangeable-slot symmetry, but does not make the optimization polynomial: even a restricted pay-once supplier selection problem contains set cover. -Instance slots also bound the search space. Use an upper bound justified by the unshared expansion of the permitted finite, acyclic derivations and evaluation scopes, or explicitly declare a slot cap as a restriction. An arbitrary cap can omit profitable sharing partitions. Rule closure establishes generation completeness only within this declared inventory; MILP optimality establishes selection quality within the generated model. Validate both separately. +The base model has `|C|` selection binaries, one binary per compatible input connection, and one per compatible output connection. Reduce these counts through safe compatibility filtering, generation caching, and dominance pruning that preserves sharing opportunities. Split components only when no dependencies, sharing, or global constraints connect them. -Reconstruct enabled instances, their candidates, and selected input/root connections into one `PhysicalASAPDAG`. Do not merge distinct instances solely because they share a group identity. Check dependencies and cycles, then run the configured accuracy, latency, and cost/resource evaluators. Return a validated plan, fallback, or explicit search failure status. - -A complete MILP does not require LBBD. Current code uses a lower-bound model with lazy refinement for difficult costs/checks. When refinement is needed, add only sound feasibility or cost cuts conditioned on every relevant decision. Validation of one plan alone does not prove a relaxed model optimal. Report the best validated cost, certified lower bound/gap, solver status, and stop reason. Timeout without a plan is not proof of infeasibility. +Reconstruct the selected DAG and run accuracy, latency, and cost/resource validation. Refine and re-solve using sound cuts when necessary. MILP does not require LBBD; lazy refinement is useful when some checks cannot be encoded directly. Report validated cost, solver status, gap, and generation limits. ## 8. Implementation and Evaluation -Reuse existing structures rather than create a parallel optimizer framework; proposed names and extensions are marked below: - -| Structure | Responsibility | -| --- | --- | -| `MemoGroup`, `Memo` | Result/state contracts, dependencies, roots, and per-group candidates. | -| `ImplementationCandidate` | Local nodes, fusion absorption/output metadata, child groups, contracts, and reuse requirements. | -| `SharedDecision`, `SharedDecisionKind` (proposed names) | Configuration choices coordinated across candidates; migrate state-local choices into instance candidates and retain cross-instance decisions where needed. | -| MILP `Choice` and proposed instance records | Instance identity, selected physical/materialization assignment, consumer connections, and cost context. | -| `PhysicalASAPDAG`, `Selection`, reports | Selected output, validation, bounds, and search status. | - -`SharedDecision` replaces the design term `CouplingSite`; `SharedDecisionKind` replaces `CouplingKind`. Current Rust code still uses the latter names. A shared decision has an identity and kind, with legal configurations and compatibility checks. Use **configuration**, rather than **threshold**, because pane layouts and scan-label sets need not be scalar or ordered. This is a proposed naming change, not a claim that the Rust types have already been renamed. - -This design extends the current shared-decision representation. Introduce intermediate-state contracts, finite instance slots, and consumer compatibility; migrate state choices into instance candidates and assign costs to physical instances. This requires changes to generation, selection, decoding, and pricing; it also changes behavior beyond the proposed type rename. Raw retention and global capacities remain accounting constraints. - -Current main places memo/rules in `search/memo` and solver code in `search/milp`, with Stage 2 physical generation and Stage 3 pricing remaining authoritative. `Legacy` is the default; `MemoBruteForce` is the reference and `MemoMilp` the solver path. Early memo construction, unified shared-state groups with selectable physical instances, broader cost-model support, EH/direct sliding algorithms, and arbitrary pane hierarchies are proposed extensions. - -### Implementation phases - -| Phase | Deliverable and acceptance criterion | +| Phase | Deliverable | | --- | --- | -| 1. Memo generation | Move identity construction before Pass 1 and generate candidates directly into groups without losing semantics, consumer obligations, or fusion. | -| 2. Shared-state groups | Extend group contracts; generate intermediate groups, instance slots, and compatible consumer connections with legal materializations and per-instance cost ownership. | -| 3. Joint selection | Encode instance activation, implementation selection, input connections, and demand to cover all queries; match an independent tiny-instance enumerator under the same inventory/models. | -| 4. Validation and scaling | Validate reconstructed plans, prove safe reductions, and measure generation/model/solve costs under budgets. | - -### Evaluation - -**Correctness:** test identity versus sharing, producer compatibility, per-instance cost attribution, active dependencies, fusion with external intermediate consumers, multiple input edges, multiple configurations/partial-sharing subsets of one group, zero-price resources, pane boundaries, legal materializations, end-to-end accuracy, and capacity/status handling. Validate any relaxed bounds and cuts against enumerated plans. +| 1. Memo generation | Build initial identities and expand result/state groups with supported candidates. | +| 2. Physical candidates | Generate configurations, materializations, execution scopes, and compatible connections. | +| 3. Joint selection | Implement candidate selection, connection constraints, and consistent cost ownership. | +| 4. Validation | Compare tiny cases with exhaustive enumeration and measure scaling. | -**Optimality:** compare MILP with an independent tiny-instance enumerator covering instance activation, candidate selection, root/input assignments, and dependency validity under the same inventory/models. The existing group-level brute-force reference must be extended before it can verify arbitrary instance partitions. Test candidate-generation coverage separately, including cases where two independently shared instances outperform full sharing and separate computation. Use an explicit numerical/tie policy. +Extend existing `MemoGroup`, `ImplementationCandidate`, and MILP selection structures rather than introducing a separate instance layer. `SharedDecision` and `SharedDecisionKind` remain proposed names for current `CouplingSite` and `CouplingKind`; this document does not rename Rust types. State-local configurations belong in candidates; genuinely cross-candidate decisions remain explicit constraints. -**Scalability:** measure groups, candidates, instance slots/enabled instances, state configurations, input connections, variables/constraints, generation/solve/validation time, and peak planner memory. Compare joint versus independent planning savings, time-limited gaps, and modeled versus validated costs across query overlap, window configurations, and materializations. +Test query coverage, multiple selected candidates per group, shared-cost accounting, scope/binding distinctions, fusion outputs, and accuracy/resource feasibility. The exhaustive reference must enumerate candidate subsets and compatible connections under the same inventory. Measure candidate/edge counts, generation and solve time, planner memory, and validated execution savings. This proposal does not claim that current code already supports the full representation. From 7832eb7afd2057c41e9b31c4dbf9bef6b07e0f9a Mon Sep 17 00:00:00 2001 From: zzylol <50204836+zzylol@users.noreply.github.com> Date: Sat, 24 Oct 2026 01:01:19 +0000 Subject: [PATCH 41/53] docs: explain memo optimization through examples and design rationale --- .../proposals/joint-planning-search.md | 105 +++++++++--------- 1 file changed, 52 insertions(+), 53 deletions(-) diff --git a/docs/design_docs/proposals/joint-planning-search.md b/docs/design_docs/proposals/joint-planning-search.md index 92a1dfce8..a4d6a527d 100644 --- a/docs/design_docs/proposals/joint-planning-search.md +++ b/docs/design_docs/proposals/joint-planning-search.md @@ -4,35 +4,33 @@ ## 1. Motivation and Challenges -ASAP primitives offer more ways to compute and share query results: exact operators, summaries, panes, rollups, and scans. These choices enlarge the search space and couple query costs. A computation that is expensive for one query may be economical when shared by several queries. +ASAP primitives increase both the opportunities for query optimization and the difficulty of searching for a good plan. A quantile, for example, may be computed directly or estimated from a summary. Several quantiles may use the same summary, and several windows may use overlapping sets of panes. The cost of one query therefore depends on decisions made for other queries. Selecting the cheapest implementation of each query independently can miss the cheapest workload plan. -The problem is to find a minimum-cost execution DAG covering every query, subject to accuracy, latency, and resource requirements, while keeping planning time and memory manageable. Shared computation and retained data must not be charged repeatedly along different query paths. Optimality is limited to the generated finite candidate set and configured models. New sketch/window algorithms and runtime scheduling are outside this proposal. +The problem addressed here is to find a minimum-cost execution DAG that produces every requested result and satisfies accuracy, latency, and resource requirements. The optimizer must account for shared computation and retained data without charging them repeatedly along different query paths. It must also keep its own search time and memory manageable as the number of queries and implementation choices grows. -## 2. Traditional Memo Groups and Rule Expansion +The following sections develop a memo-based representation and a joint selection procedure for this problem. The proposal concerns the finite candidates supported by the planner and its configured cost and constraint models. It does not introduce new sketch or window algorithms, or a runtime scheduler. An optimal solution consequently means an optimum within the generated model, not among all imaginable execution plans. -A **memo group** contains expressions producing the same logical result. Expressions reference input groups rather than copying complete input plans. Rules add equivalent expressions and physical candidates; the optimizer compares their costs. +## 2. Traditional Memo Groups and Rule Expansion -### Logical rules +A memo group represents expressions with the same logical result. Expressions reference input groups, allowing the optimizer to retain several ways of computing a result without copying complete plans. Graefe's [Cascades framework](https://15721.courses.cs.cmu.edu/spring2016/papers/graefe-ieee1995.pdf) develops this representation with rule-driven exploration and optimization. -Consider `Filter(A.x > 10, Join(A, B))`, where the inner join allows this filter to move onto `A`. +Consider a filter above an inner join: ```text -Initially: - G_A = { Scan(A) } - G_B = { Scan(B) } - G_join = { Join(G_A, G_B) } - G_out = { Filter(A.x > 10, G_join) } - -After filter pushdown: - G_filtered = { Filter(A.x > 10, G_A) } - G_out += Join(G_filtered, G_B) +G_A = { Scan(A) } +G_B = { Scan(B) } +G_join = { Join(G_A, G_B) } +G_out = { Filter(A.x > 10, G_join) } ``` -The new expression belongs to `G_out` because it produces the same result. The filtered input gets its own group. Further rules can expand newly added expressions; interning avoids duplicates. +A valid filter-pushdown rule adds: -### Physical rules +```text +G_filtered = { Filter(A.x > 10, G_A) } +G_out += Join(G_filtered, G_B) +``` -Implementation rules add supported physical candidates for a logical expression: +The new join belongs to `G_out` because its result is unchanged. Implementation rules then introduce physical candidates: ```text Join(G_filtered, G_B) @@ -41,20 +39,17 @@ Join(G_filtered, G_B) → NestedLoopJoin(G_filtered, G_B) ``` -Candidates have different costs and input requirements. For example, merge join requires ordered inputs. The optimizer typically caches `Best(group, required_properties)` and combines local operator cost with child costs. +These candidates differ in cost and input requirements; merge join, for example, requires ordered inputs. Optimization can cache `Best(group, required_properties)`. Logical and physical exploration need not occur in separate exhaustive phases: Cascades explores expressions as needed and avoids repeating earlier work. -A shared memo group avoids repeated optimization work. It does not by itself mean that execution computes the result once and shares it. +This reuse concerns optimization. A shared group does not itself require shared runtime execution. ASAPPlanner needs an explicit execution-sharing decision, developed next. ## 3. ASAPPlanner Memo Groups and Rule Expansion -ASAPPlanner uses two concepts: - -- **Group:** a requested result or intermediate-state contract, including input, filters, grouping, and window semantics. State groups also specify the state interface and guarantees. -- **Candidate:** a supported way to produce that group's output. A physical candidate specifies operators, configuration, materialization, execution scope, and required inputs. +ASAPPlanner retains the distinction between a required output and the ways to produce it. A **group** describes a requested result or an intermediate-state contract. Its identity includes input, filters, grouping, and window semantics; a state group also specifies its interface and guarantees. A **candidate** describes a supported implementation of that contract. At the physical level it includes operators, configuration, materialization, execution scope, and required inputs. -Approximate candidates target the requested result subject to accuracy requirements; they are not asserted to return exactly the same values as exact computation. **Several physical candidates in one group may be selected.** Each selected candidate becomes a node in the execution DAG. Consumers connect to specific selected candidates; sharing means using the same node. +Approximation requires care in interpreting the group. An exact quantile and a KLL estimate target the same requested result, but need not return identical values. The approximate candidate is admissible only under the applicable accuracy requirements. Thus the proposed result/state contracts extend traditional exact equivalence rather than asserting equivalence between exact and approximate values. -For example: +The second extension concerns selection. Several physical candidates in one group may be selected, and each selected candidate becomes a node in the execution DAG. Consumers connect to particular candidates. For example: ```text G_KLL @@ -62,13 +57,15 @@ G_KLL C_large: large-capacity KLL ← consumers C, D ``` -Both candidates may be selected. There is no separate physical-instance index. If two computations need separate execution scopes or incompatible input bindings, generation must provide distinct candidates that can coexist. A single selected candidate cannot have different bindings for different consumers. This is a candidate-generation requirement, not an assumption that every duplicate computation can safely be shared. +Selecting both candidates permits two sharing subsets. Selecting only one is appropriate when its configuration and cost suit all consumers. This representation uses no additional physical-instance index: a candidate is the selectable unit of execution. + +That simplification places a requirement on generation. If computations must coexist with separate execution scopes or incompatible input bindings, they need distinct generated candidates, even when their algorithms and capacities match. One selected candidate has one set of input bindings. Removing instance indices therefore does not establish that every duplicate computation can safely be shared, or that an arbitrary candidate inventory is complete. The workflow follows [planner layering](planner-layering.md). Local group candidates represent components of the workload DAGs described there. ### Stage 0: build initial groups -Parse and normalize the workload, intern equivalent logical sub-expressions, and record query roots. +The frontend produces normalized logical expressions. The proposed memo builder interns equivalent sub-expressions and records their query roots. Consider two queries over the same input and evaluation schedule: ```text Q1: P95(X[15m]), every 1m @@ -83,7 +80,7 @@ P95 and P99 remain different groups. Initial grouping before Pass 1 is proposed; ### Stage 1, Pass 1: expand computation candidates -Apply supported computation rules per group: +Computation rules expand each result group with the supported ways to obtain its output: ```text G95 = { ExactQuantile(0.95, G_input), LocalKLLQuantile(0.95, G_input, s) } @@ -115,27 +112,27 @@ G_KLL += MergeLatest15(G_panes) ### Stage 2: expand physical candidates -Enumerate executable operators, configurations, and materialization assignments. For example, maintain eligible panes at ingestion and merge them at evaluation. Retain candidate-specific compatibility and timing requirements; checks depending on actual consumers must use the selected connections. +Physical rules provide executable operators, configurations, and materialization assignments for the generated computations. For example, a supported pane implementation can maintain state at ingestion and merge it at evaluation. This introduces two timing decisions within the same query, rather than one materialization choice for the whole query. Eligibility that depends on consumers is checked against the selected connections; considering every potential consumer at once could incorrectly reject a useful candidate. Window semantics, algorithms, and materialization remain separate dimensions. Their definitions and support restrictions belong in [Window and Materialization Choices](window-materialization-choices.md). ### Stage 3: select and connect candidates -Select physical candidates, connect each required input to a compatible selected candidate, and cover every query root. Several consumers may share one KLL candidate or use different candidates from `G_KLL`. Reconstruct and validate the resulting `PhysicalASAPDAG`. +Selection chooses physical candidates and connects their required inputs until every query root is covered. Several consumers may connect to one KLL candidate or to different candidates from `G_KLL`. The selected graph is reconstructed as a `PhysicalASAPDAG` and validated. This is where implementation costs are compared jointly; the earlier rules expose the choices on which that comparison depends. ## 4. Cost Model -Charge each selected node for its local work and state, excluding its input nodes. A KLL build owns construction and retained state; an estimator owns extraction. Sharing a build avoids repeating its charge, while each estimator still incurs its own cost. +The cost of a shared plan is obtained from its selected execution nodes, not by summing independently priced query paths. Each node owns its local work and state, excluding its inputs. In the running example, the KLL build owns construction and retained state, while each estimator owns extraction. Two estimators using one build therefore incur one build charge and two extraction charges. -Costs use a common horizon or consistent rates. A selected node can execute repeatedly: materialization, evaluation schedules, and selected consumers determine its execution frequency and lifetime. A constant candidate cost is valid only when that context is fixed. +A node may execute repeatedly. Its materialization, consumer schedules, and lifetime determine how frequently its work is performed and how long its state occupies resources. A constant candidate cost is appropriate only when this context is fixed. All terms use a common horizon or consistent rates, so ingestion work and evaluation work can be compared. -Shared raw-data retention has one owner per source. Consumers needing 5m and 15m of the same source require 15m retention, not 20m. Add configured query-request overhead separately. Do not double-count state already owned by a candidate. +Some charges belong to a resource shared beyond one computation. If two consumers need 5m and 15m of the same raw source, the required retention horizon is 15m, not 20m. Such retention is charged once per source and excludes intermediate state already owned by a candidate. Configured query-request overhead is accounted for separately. -Use the configured Stage 3 objective and evaluators. Current memo MILP supports usage-over-time pricing; broader cost-model support is an extension. Accuracy is checked at query outputs, and latency and resource limits must respect the configured timing model. +The configured Stage 3 objective and evaluators remain authoritative. Current memo MILP supports usage-over-time pricing; broader cost-model support is an extension. Accuracy is an output requirement, since locally acceptable errors may compose into an unacceptable query result. Latency and resource limits likewise depend on the configured timing and capacity assumptions. ## 5. MILP Formulation -MILP jointly selects candidates and their connections. Ordinary bipartite matching does not capture pay-once producer activation, multiple consumers, and recursively required inputs. The formulation below is a fixed-context base; demand-dependent costs require corresponding rate/lifetime variables or finite context configurations. +The selection problem combines supplier activation with input assignment. Ordinary bipartite matching describes compatible pairs, but does not directly capture a producer paid for once, used by several consumers, and itself dependent on other producers. A MILP expresses these decisions together. The following formulation gives the fixed-context base; demand-dependent costs require rate/lifetime variables or finite context configurations consistent with the selected graph. ### 5.1 Variables @@ -158,7 +155,7 @@ b[c,e,d] ∈ {0,1} input e of c uses supplier candidate d r[t,c] ∈ {0,1} query-output reference t uses candidate c ``` -Create connection variables only for compatible pairs. Capacity and materialization are included in candidate configuration; no separate copy or instance variables are needed. +Connection variables exist only for compatible pairs. Capacity and materialization belong to candidate configuration, so selection needs no separate copy or instance variables. #### Objective @@ -168,7 +165,7 @@ min Σ_c local_cost[c] × x[c] + query_overhead ``` -`local_cost` excludes suppliers. `shared_resource_cost` explicitly models resources such as per-source raw retention; `query_overhead` accounts for configured request charges. Both are zero when absent. If cost depends on selected consumers, model that dependence instead of assigning an incorrect constant. +The first term charges each selected candidate for its local work, excluding suppliers. The remaining terms explicitly model shared resources such as source retention and configured request charges; either is zero when absent. Where selected consumers change cost, additional expressions model that dependence rather than treating it as a constant. ### 5.2 Constraints @@ -204,10 +201,15 @@ order[d] + 1 ≤ order[c] + N × (1 − b[c,e,d]) #### Enforce requirements -- **Accuracy:** enforce valid end-to-end error bounds or sound feasibility refinement for each query. -- **Physical validity:** require supported operators, legal materialization, compatible timing, and correct window coverage. -- **Resources:** for additive fixed-context usage, constrain `Σ_c usage[k,c] × x[c] + shared_usage[k] ≤ capacity[k]`. Peak and temporary usage require appropriate lifetime/concurrency accounting. -- **Latency:** enforce deadlines under the configured evaluator's timing assumptions. +Candidate and connection compatibility eliminate unsupported operators, illegal materializations, and invalid window combinations. End-to-end accuracy still requires composition bounds, feasible-combination constraints, or sound feasibility refinement. Query deadlines are enforced under the configured latency model. + +For additive resource usage with fixed execution context, capacity constraints take the form: + +```text +Σ_c usage[k,c] × x[c] + shared_usage[k] ≤ capacity[k] +``` + +Peak usage and temporary allocations need lifetime and concurrency accounting when a simple sum is inappropriate. These extensions preserve the same ownership rule: suppliers and shared resources are not charged again through each consumer. Only exposed candidate outputs can supply consumers. A fused `topk(sum(...))` candidate does not automatically supply another query's full `sum(...)`; that result needs a separate candidate unless explicitly exposed. @@ -232,21 +234,18 @@ Both estimators connect to the same selected build candidate, so its cost appear ## 7. Search-Space Reduction and Validation -Intern duplicate expressions and apply rules to newly generated groups/candidates. Generate only supported configurations and execution scopes. If generation is capped or stops before rule closure, report that restriction. Removing instance indices eliminates interchangeable-slot symmetry, but does not make the optimization polynomial: even a restricted pay-once supplier selection problem contains set cover. +Candidate generation maintains a worklist of new expressions and applicable rules. Interning prevents duplicate derivations, while compatibility checks eliminate unsupported connections. Generation is relative to a finite inventory of rules, configurations, and execution scopes. If exploration stops before closure or imposes a cap, the reported optimality claim must reflect that restriction. + +Let `n` be the number of physical candidates, `m` the number of compatible input connections, and `r` the number of compatible query-output connections. The base formulation has `n + m + r` binary variables, plus ordering and any cost-model auxiliaries. This counts the model size, not its solution time. Even a restricted supplier-selection case contains set cover: suppliers cover subsets of outputs and have activation costs. Removing interchangeable instance slots reduces redundant representation but does not remove this combinatorial difficulty. -The base model has `|C|` selection binaries, one binary per compatible input connection, and one per compatible output connection. Reduce these counts through safe compatibility filtering, generation caching, and dominance pruning that preserves sharing opportunities. Split components only when no dependencies, sharing, or global constraints connect them. +Search reductions must preserve joint choices. A candidate that costs more alone may enable useful sharing, so local cost dominance is insufficient. Independent components can be solved separately only when dependencies, sharing, and global constraints also separate. Generation caching and sparse compatibility edges reduce overhead without requiring enumeration of complete workload plans. -Reconstruct the selected DAG and run accuracy, latency, and cost/resource validation. Refine and re-solve using sound cuts when necessary. MILP does not require LBBD; lazy refinement is useful when some checks cannot be encoded directly. Report validated cost, solver status, gap, and generation limits. +After selection, the planner reconstructs the DAG and evaluates accuracy, latency, and resource cost. A complete MILP does not require LBBD. Lazy refinement is useful when difficult checks are left outside the initial model, provided the resulting feasibility or cost cuts are sound. The report includes the validated cost, solver status and gap, and generation limits; validating one solution alone does not prove a relaxed model optimal. ## 8. Implementation and Evaluation -| Phase | Deliverable | -| --- | --- | -| 1. Memo generation | Build initial identities and expand result/state groups with supported candidates. | -| 2. Physical candidates | Generate configurations, materializations, execution scopes, and compatible connections. | -| 3. Joint selection | Implement candidate selection, connection constraints, and consistent cost ownership. | -| 4. Validation | Compare tiny cases with exhaustive enumeration and measure scaling. | +The design extends existing memo and selection structures. `MemoGroup` stores result/state contracts, while `ImplementationCandidate` and physical generation describe supported implementations and their input requirements. Selection adds direct candidate activation and compatible connections, followed by reconstruction and pricing of the resulting graph. These are proposed extensions, not a claim that the existing implementation already supports the full representation. -Extend existing `MemoGroup`, `ImplementationCandidate`, and MILP selection structures rather than introducing a separate instance layer. `SharedDecision` and `SharedDecisionKind` remain proposed names for current `CouplingSite` and `CouplingKind`; this document does not rename Rust types. State-local configurations belong in candidates; genuinely cross-candidate decisions remain explicit constraints. +Implementation proceeds from initial memo identities and rule expansion to physical configurations and connections, then joint selection and validation. State-local choices belong in candidates. Decisions genuinely spanning candidates remain explicit constraints. `SharedDecision` and `SharedDecisionKind` are the proposed names for current `CouplingSite` and `CouplingKind`; no Rust types are renamed by this document. -Test query coverage, multiple selected candidates per group, shared-cost accounting, scope/binding distinctions, fusion outputs, and accuracy/resource feasibility. The exhaustive reference must enumerate candidate subsets and compatible connections under the same inventory. Measure candidate/edge counts, generation and solve time, planner memory, and validated execution savings. This proposal does not claim that current code already supports the full representation. +Evaluation must distinguish generation coverage from selection correctness. Small workloads are exhaustively enumerated over candidate subsets and compatible connections under the same inventory, then compared with MILP results. Cases include multiple selected candidates in one group, shared build costs, separate execution scopes, fusion outputs, and composed accuracy/resource requirements. Scaling experiments measure candidate and connection counts, generation and solve time, planner memory, and validated execution savings. Together these measurements establish whether the representation avoids repeated search work and whether joint selection justifies its planning cost. From fdd0aee1b43d4f2623a1d5821091d37ca542625b Mon Sep 17 00:00:00 2001 From: zzylol <50204836+zzylol@users.noreply.github.com> Date: Sat, 24 Oct 2026 01:06:03 +0000 Subject: [PATCH 42/53] docs: illustrate physical selection and shared cost accounting --- .../proposals/joint-planning-search.md | 49 +++++++++++++++++++ 1 file changed, 49 insertions(+) diff --git a/docs/design_docs/proposals/joint-planning-search.md b/docs/design_docs/proposals/joint-planning-search.md index a4d6a527d..2217eada9 100644 --- a/docs/design_docs/proposals/joint-planning-search.md +++ b/docs/design_docs/proposals/joint-planning-search.md @@ -114,12 +114,45 @@ G_KLL += MergeLatest15(G_panes) Physical rules provide executable operators, configurations, and materialization assignments for the generated computations. For example, a supported pane implementation can maintain state at ingestion and merge it at evaluation. This introduces two timing decisions within the same query, rather than one materialization choice for the whole query. Eligibility that depends on consumers is checked against the selected connections; considering every potential consumer at once could incorrectly reject a useful candidate. +For the running example, assume evaluations occur at aligned minute boundaries and the deployment supports ingestion-time pane maintenance. Stage 2 can expand the state groups as follows; `s` is a generated KLL capacity: + +```text +Group Physical candidate Computation Materialization +G_panes P_ingest(s) Maintain aligned 1m panes MatAtIngestion(Memory) +G_KLL K_merge(s) MergeLatest15(G_panes) MatAtQuery(Memory) +G_KLL K_build(s) BuildKLL(G_input, 15m) MatAtQuery(Memory) +G95 E95 EstimateQuantile(.95, G_KLL) NotMaterialized +G99 E99 EstimateQuantile(.99, G_KLL) NotMaterialized +``` + +`P_ingest(s)` retains the panes needed to cover the 15m window. `K_merge(s)` constructs window state from those panes at evaluation; `K_build(s)` instead builds window state from available raw input. Each candidate records its input requirements and costs. The estimators can use either compatible state candidate. Exact and local KLL candidates from Pass 1 also receive supported physical implementations and remain available. Stage 2 has generated choices; it has not yet selected a build or connected an estimator to it. + Window semantics, algorithms, and materialization remain separate dimensions. Their definitions and support restrictions belong in [Window and Materialization Choices](window-materialization-choices.md). ### Stage 3: select and connect candidates Selection chooses physical candidates and connects their required inputs until every query root is covered. Several consumers may connect to one KLL candidate or to different candidates from `G_KLL`. The selected graph is reconstructed as a `PhysicalASAPDAG` and validated. This is where implementation costs are compared jointly; the earlier rules expose the choices on which that comparison depends. +Suppose the pane-based choice has the lowest modeled cost among feasible plans, and capacity `s*` satisfies both queries after pane merges. Stage 3 selects `P_ingest(s*)`, `K_merge(s*)`, `E95`, and `E99`, together with their required upstream input, and chooses these connections: + +```text +Selected input for X + │ + ▼ +P_ingest(s*) maintain 1m panes at ingestion + │ + ▼ +K_merge(s*) merge the latest 15 panes at evaluation + ├───────────────┐ + ▼ ▼ + E95 E99 + │ │ + ▼ ▼ + Q1 Q2 +``` + +For a common evaluation of Q1 and Q2, the selected merge state serves both estimators. Pane maintenance is charged once for the shared pane chain, and merge work once per common evaluation; each estimator contributes its own extraction cost. `K_build(s*)` and the exact/local candidates are unselected in this example. If timing or accuracy prevents this sharing, Stage 3 may connect the estimators to different candidates from `G_KLL`. Thus Stage 2 enumerates executable choices, while Stage 3 determines which choices execute and which consumers share them. + ## 4. Cost Model The cost of a shared plan is obtained from its selected execution nodes, not by summing independently priced query paths. Each node owns its local work and state, excluding its inputs. In the running example, the KLL build owns construction and retained state, while each estimator owns extraction. Two estimators using one build therefore incur one build charge and two extraction charges. @@ -128,6 +161,22 @@ A node may execute repeatedly. Its materialization, consumer schedules, and life Some charges belong to a resource shared beyond one computation. If two consumers need 5m and 15m of the same raw source, the required retention horizon is 15m, not 20m. Such retention is charged once per source and excludes intermediate state already owned by a candidate. Configured query-request overhead is accounted for separately. +The following cases make the accounting concrete. Let `B` be the cost of one whole-window KLL build, `M` one pane merge, and `E95` and `E99` the extraction costs. These are illustrative work charges; retained and temporary resources are accounted for separately over the same horizon. + +| Case | Selected execution | Cost accounting | +| --- | --- | --- | +| Independent query-time builds | Q1 and Q2 each build KLL state and extract a quantile. | For one evaluation each: `2B + E95 + E99`. The two builds are separate work. | +| Shared query-time build | Q1 and Q2 use one compatible build in a common evaluation. | `B + E95 + E99`. Query-local state is charged for its actual lifetime. | +| Repeated common evaluations | Both queries evaluate together `n` times; state is discarded after each evaluation. | `n × (B + E95 + E99)`. Selecting one candidate does not mean building only once over the entire horizon. | +| Different evaluation timestamps | Q1 evaluates every minute; Q2 every five minutes on the same grid. Both use a 15m window. | Over 15 evaluation ticks, a shareable query-local build executes 15 times, with 15 P95 extractions and 3 P99 extractions: `15B + 15E95 + 3E99`. This assumes sharing at the three common timestamps; an earlier window state cannot automatically answer a later window. | +| Ingestion-maintained panes | One pane chain feeds a shared query-time merge and both estimators. | Ingestion updates and pane retention are charged once for the chain. Add `n × (M + E95 + E99)` for `n` common evaluations, plus query-local merge memory. | +| State retained across evaluations | An eligible pane-chain candidate uses `MatAtQueryKeptAcrossEvaluations(Memory)`. | Charge initial construction, required refresh/update work, retained memory/storage over its lifetime, and evaluation work. Reuse does not eliminate freshness maintenance; generate this case only when supported. | +| Multiple selected candidates in one group | Some consumers use a small KLL state and others a separate large state. | Charge each selected state's build/update and retention, then each consumer's local work. Equal group identity does not collapse two physical candidates into one charge. | +| Shared raw-data retention | Two consumers need 5m and 15m of the same raw source and compatible retention scope. | Charge source storage for the 15m horizon once, with reads charged according to actual work. Do not add 5m and 15m or include intermediate KLL state in this charge. | +| Per-request overhead | Two requests share computation but still require separate response handling. | Charge configured request overhead for each request. Compute sharing does not remove request-specific work. | + +Memory-time and capacity answer different questions. A state occupying `m` bytes for duration `t` contributes `m × t` to memory-time. A capacity limit instead considers simultaneous residency, including temporary allocations; it cannot be inferred from memory-time alone. + The configured Stage 3 objective and evaluators remain authoritative. Current memo MILP supports usage-over-time pricing; broader cost-model support is an extension. Accuracy is an output requirement, since locally acceptable errors may compose into an unacceptable query result. Latency and resource limits likewise depend on the configured timing and capacity assumptions. ## 5. MILP Formulation From ea64990712e445462138541e0811fe68fdde0bd2 Mon Sep 17 00:00:00 2001 From: zzylol <50204836+zzylol@users.noreply.github.com> Date: Sat, 24 Oct 2026 01:15:15 +0000 Subject: [PATCH 43/53] docs: explain candidate dependencies and unify resource cost ownership --- .../proposals/joint-planning-search.md | 92 +++++++++++++------ 1 file changed, 62 insertions(+), 30 deletions(-) diff --git a/docs/design_docs/proposals/joint-planning-search.md b/docs/design_docs/proposals/joint-planning-search.md index 2217eada9..cee1c6698 100644 --- a/docs/design_docs/proposals/joint-planning-search.md +++ b/docs/design_docs/proposals/joint-planning-search.md @@ -159,7 +159,9 @@ The cost of a shared plan is obtained from its selected execution nodes, not by A node may execute repeatedly. Its materialization, consumer schedules, and lifetime determine how frequently its work is performed and how long its state occupies resources. A constant candidate cost is appropriate only when this context is fixed. All terms use a common horizon or consistent rates, so ingestion work and evaluation work can be compared. -Some charges belong to a resource shared beyond one computation. If two consumers need 5m and 15m of the same raw source, the required retention horizon is 15m, not 20m. Such retention is charged once per source and excludes intermediate state already owned by a candidate. Configured query-request overhead is accounted for separately. +Some charges belong to a resource shared beyond one computation. If two consumers need 5m and 15m of the same raw source, the required retention horizon is 15m, not 20m. Represent this shared storage as a state group `G_raw[d]` for source `d` and a particular storage scope, with candidates such as `Retain5m` and `Retain15m`. The group describes access to retained raw data; each candidate advertises its coverage, and each consumer connects only to a candidate covering its required horizon. Both consumers can use `Retain15m`, whose storage cost is charged once through candidate selection. Reads and intermediate KLL state retain their own cost owners. + +Fixed request overhead does not affect plan selection. If query `q` executes `N_q` times over the accounting horizon with fixed per-request cost `h_q`, add `Σ_q N_q × h_q` to the reported total after optimization. Its resource usage still counts against applicable budgets. Any overhead that changes with implementation or connections belongs in the corresponding decision-dependent cost model instead. The following cases make the accounting concrete. Let `B` be the cost of one whole-window KLL build, `M` one pane merge, and `E95` and `E99` the extraction costs. These are illustrative work charges; retained and temporary resources are accounted for separately over the same horizon. @@ -172,8 +174,8 @@ The following cases make the accounting concrete. Let `B` be the cost of one who | Ingestion-maintained panes | One pane chain feeds a shared query-time merge and both estimators. | Ingestion updates and pane retention are charged once for the chain. Add `n × (M + E95 + E99)` for `n` common evaluations, plus query-local merge memory. | | State retained across evaluations | An eligible pane-chain candidate uses `MatAtQueryKeptAcrossEvaluations(Memory)`. | Charge initial construction, required refresh/update work, retained memory/storage over its lifetime, and evaluation work. Reuse does not eliminate freshness maintenance; generate this case only when supported. | | Multiple selected candidates in one group | Some consumers use a small KLL state and others a separate large state. | Charge each selected state's build/update and retention, then each consumer's local work. Equal group identity does not collapse two physical candidates into one charge. | -| Shared raw-data retention | Two consumers need 5m and 15m of the same raw source and compatible retention scope. | Charge source storage for the 15m horizon once, with reads charged according to actual work. Do not add 5m and 15m or include intermediate KLL state in this charge. | -| Per-request overhead | Two requests share computation but still require separate response handling. | Charge configured request overhead for each request. Compute sharing does not remove request-specific work. | +| Shared raw-data retention | Two consumers need 5m and 15m of the same raw source and compatible retention scope. | Both consumers connect to `Retain15m` in `G_raw[d]`. Charge that candidate once; reads and intermediate KLL state have separate owners. | +| Per-request overhead | Two requests share computation but still require separate response handling. | Report fixed overhead as `Σ_q N_q × h_q`; omit this constant from the optimization objective but retain its applicable resource usage. Decision-dependent overhead stays in the optimization model. | Memory-time and capacity answer different questions. A state occupying `m` bytes for duration `t` contributes `m × t` to memory-time. A capacity limit instead considers simultaneous residency, including temporary allocations; it cannot be inferred from memory-time alone. @@ -185,67 +187,97 @@ The selection problem combines supplier activation with input assignment. Ordina ### 5.1 Variables -#### Sets and notation +#### Candidates and Dependency Connections + +The optimizer needs to know which computations are available, which inputs each computation requires, and which query outputs must be produced. These are the nodes, dependencies, and roots of the candidate dependency graph. For the running example, suppose both KLL capacities can serve P95, but only the larger capacity meets P99's accuracy requirement: + +```text +Groups and their candidates: + G_KLL: {small-build, large-build} + G95: {exact-P95, estimate-P95} + G99: {exact-P99, estimate-P99} + +Allowed state-input connections (source → consumer): + small-build ────────→ estimate-P95 + large-build ────────→ estimate-P95 + large-build ────────→ estimate-P99 + +Query roots: + Q1 ← exact-P95 or estimate-P95 + Q2 ← exact-P99 or estimate-P99 +``` + +The builds and exact candidates also require raw-input dependencies, omitted from this diagram. The graph records possible connections; it does not select them. Choosing both estimators and connecting them to `large-build` produces one shared build in the selected execution DAG. The candidate dependency graph may contain cycles among possible choices; the selected execution DAG must be acyclic. + +| Name | Meaning | Example | +| --- | --- | --- | +| `Groups` | Logical result and state groups. | `{G_KLL, G95, G99, ...}` | +| `Candidates(g)` | Finite physical candidates in group `g`. | `Candidates(G95) = {exact-P95, estimate-P95}` | +| `Inputs(c)` | Inputs that candidate `c` must receive. | `Inputs(estimate-P95) = {state}` | +| `CompatibleSources(c,input)` | Candidates allowed to supply that input. | `CompatibleSources(estimate-P99,state) = {large-build}` | +| `QueryRoots` | Required query outputs. | `{Q1, Q2}` | +| `RootCandidates(q)` | Candidates allowed to provide query root `q`. | `RootCandidates(Q1) = {exact-P95, estimate-P95}` | -| Symbol | Meaning | -| --- | --- | -| `G` | Result and state groups. | -| `C_g` | Finite physical candidates belonging to group `g`. | -| `E(c)` | Required input positions of candidate `c`. | -| `S(c,e)` | Compatible supplier candidates for input `e` of candidate `c`. | -| `T` | Required query-output references. | -| `S(t)` | Candidates compatible with output reference `t`. | +Candidate IDs are globally unique; `Candidates` denotes their union across groups. Input positions remain distinct: a join needs both `left` and `right`, even if both refer to the same group. For this formulation, each query has one explicit output root. If its output combines several computations, a finalization candidate has those computations as required inputs; covering the root therefore covers all of them. -Candidate IDs are globally unique; `C` is the union of all `C_g`. Input positions stay distinct even when they request the same group. For a P95 estimator, the `state` input's suppliers are compatible candidates from `G_KLL`. A query may have several output references. +Three binary variables select nodes, dependency connections, and query-root assignments: ```text -x[c] ∈ {0,1} select physical candidate c -b[c,e,d] ∈ {0,1} input e of c uses supplier candidate d -r[t,c] ∈ {0,1} query-output reference t uses candidate c +x[c] ∈ {0,1} select candidate c +b[c,input,d] ∈ {0,1} connect source candidate d to input of c +r[q,c] ∈ {0,1} use candidate c to provide query root q ``` -Connection variables exist only for compatible pairs. Capacity and materialization belong to candidate configuration, so selection needs no separate copy or instance variables. +Connection variables exist only for compatible sources. Capacity and materialization belong to candidate configuration, so selection needs no separate copy or instance variables. #### Objective ```text min Σ_c local_cost[c] × x[c] - + shared_resource_cost - + query_overhead ``` -The first term charges each selected candidate for its local work, excluding suppliers. The remaining terms explicitly model shared resources such as source retention and configured request charges; either is zero when absent. Where selected consumers change cost, additional expressions model that dependence rather than treating it as a constant. +Each selected candidate owns its local work and state, excluding suppliers. This includes shared-resource candidates such as `Retain15m`, so no separate shared-resource charge is needed. Fixed request overhead is omitted from the objective because it is constant for the workload; Section 4 describes its inclusion in reported total cost. Where selected consumers or connections change cost, additional expressions model that dependence rather than treating it as a constant. ### 5.2 Constraints #### Cover every query output ```text -Σ_{c ∈ S(t)} r[t,c] = 1 for each output reference t -r[t,c] ≤ x[c] +Σ_{c ∈ RootCandidates(q)} r[q,c] = 1 for each q ∈ QueryRoots +r[q,c] ≤ x[c] ``` #### Supply every selected candidate's inputs ```text -Σ_{d ∈ S(c,e)} b[c,e,d] = x[c] for each required input (c,e) -b[c,e,d] ≤ x[d] +Σ_{d ∈ CompatibleSources(c,input)} b[c,input,d] = x[c] +b[c,input,d] ≤ x[d] for each input ∈ Inputs(c) +``` + +Several consumers may connect to the same supplier. There is **no general** constraint `Σ_{c ∈ Candidates(g)} x[c] ≤ 1`: multiple candidates in one group may be needed. + +#### Shared raw-retention configuration + +For one raw-storage scope, retention candidates configure the same stored copy, so they are mutually exclusive: + +```text +Σ_{c ∈ Candidates(G_raw[d])} x[c] ≤ 1 ``` -Several consumers may connect to the same supplier. There is **no** constraint `Σ_{c ∈ C_g} x[c] ≤ 1`: multiple candidates in one group may be needed. +This is a resource-specific constraint, not a restriction on every group. For a consumer needing 15m, its compatible suppliers include `Retain15m` and any generated longer-retention candidates, but exclude `Retain5m`. The input constraints above require a selected compatible supplier. A second consumer needing 5m can connect to the same `Retain15m`; storage is therefore charged once. Separate stored copies require distinct storage scopes and are charged separately. #### Keep only connected, acyclic computation ```text -x[d] ≤ Σ_t r[t,d] + Σ_{c,e} b[c,e,d] +x[d] ≤ Σ_q r[q,d] + Σ_{c,input} b[c,input,d] ``` Only defined connections appear in these sums. Every selected candidate must serve a query output or another selected candidate. Together with acyclicity, this excludes unused computation. -If candidate generation does not already guarantee acyclicity, introduce `0 ≤ order[c] ≤ N−1`, where `N = |C|`: +If candidate generation does not already guarantee acyclicity, introduce `0 ≤ order[c] ≤ N−1`, where `N = |Candidates|`: ```text -order[d] + 1 ≤ order[c] + N × (1 − b[c,e,d]) +order[d] + 1 ≤ order[c] + N × (1 − b[c,input,d]) ``` #### Enforce requirements @@ -255,10 +287,10 @@ Candidate and connection compatibility eliminate unsupported operators, illegal For additive resource usage with fixed execution context, capacity constraints take the form: ```text -Σ_c usage[k,c] × x[c] + shared_usage[k] ≤ capacity[k] +Σ_c usage[k,c] × x[c] ≤ capacity[k] − fixed_usage[k] ``` -Peak usage and temporary allocations need lifetime and concurrency accounting when a simple sum is inappropriate. These extensions preserve the same ownership rule: suppliers and shared resources are not charged again through each consumer. +Here candidate usage includes shared raw-retention candidates. `fixed_usage[k]` is usage fixed by the workload and independent of plan selection, such as fixed request handling, measured in the same units and accounting context; it is zero when absent. Peak usage and temporary allocations need lifetime and concurrency accounting when a simple sum is inappropriate. These extensions preserve the same ownership rule: suppliers and shared resources are not charged again through each consumer. Only exposed candidate outputs can supply consumers. A fused `topk(sum(...))` candidate does not automatically supply another query's full `sum(...)`; that result needs a separate candidate unless explicitly exposed. From c44b9303bc908bff976d6ba5ed62624acb639528 Mon Sep 17 00:00:00 2001 From: zzylol <50204836+zzylol@users.noreply.github.com> Date: Sat, 24 Oct 2026 01:18:54 +0000 Subject: [PATCH 44/53] docs: name producer requirements and consumer connections consistently --- .../proposals/joint-planning-search.md | 44 ++++++++++--------- 1 file changed, 24 insertions(+), 20 deletions(-) diff --git a/docs/design_docs/proposals/joint-planning-search.md b/docs/design_docs/proposals/joint-planning-search.md index cee1c6698..efab99939 100644 --- a/docs/design_docs/proposals/joint-planning-search.md +++ b/docs/design_docs/proposals/joint-planning-search.md @@ -183,7 +183,7 @@ The configured Stage 3 objective and evaluators remain authoritative. Current me ## 5. MILP Formulation -The selection problem combines supplier activation with input assignment. Ordinary bipartite matching describes compatible pairs, but does not directly capture a producer paid for once, used by several consumers, and itself dependent on other producers. A MILP expresses these decisions together. The following formulation gives the fixed-context base; demand-dependent costs require rate/lifetime variables or finite context configurations consistent with the selected graph. +The selection problem combines producer activation with dependency assignment. Ordinary bipartite matching describes compatible pairs, but does not directly capture a producer paid for once, used by several consumers, and itself dependent on other producers. A MILP expresses these decisions together. The following formulation gives the fixed-context base; demand-dependent costs require rate/lifetime variables or finite context configurations consistent with the selected graph. ### 5.1 Variables @@ -197,7 +197,7 @@ Groups and their candidates: G95: {exact-P95, estimate-P95} G99: {exact-P99, estimate-P99} -Allowed state-input connections (source → consumer): +Allowed producer–consumer connections (producer → consumer): small-build ────────→ estimate-P95 large-build ────────→ estimate-P95 large-build ────────→ estimate-P99 @@ -213,22 +213,22 @@ The builds and exact candidates also require raw-input dependencies, omitted fro | --- | --- | --- | | `Groups` | Logical result and state groups. | `{G_KLL, G95, G99, ...}` | | `Candidates(g)` | Finite physical candidates in group `g`. | `Candidates(G95) = {exact-P95, estimate-P95}` | -| `Inputs(c)` | Inputs that candidate `c` must receive. | `Inputs(estimate-P95) = {state}` | -| `CompatibleSources(c,input)` | Candidates allowed to supply that input. | `CompatibleSources(estimate-P99,state) = {large-build}` | +| `ProducerRequirements(consumer)` | Dependencies that the consumer must have supplied. | `ProducerRequirements(estimate-P95) = {KLL-state}` | +| `CompatibleProducers(consumer,requirement)` | Producer candidates allowed to satisfy that dependency. | `CompatibleProducers(estimate-P99,KLL-state) = {large-build}` | | `QueryRoots` | Required query outputs. | `{Q1, Q2}` | | `RootCandidates(q)` | Candidates allowed to provide query root `q`. | `RootCandidates(Q1) = {exact-P95, estimate-P95}` | -Candidate IDs are globally unique; `Candidates` denotes their union across groups. Input positions remain distinct: a join needs both `left` and `right`, even if both refer to the same group. For this formulation, each query has one explicit output root. If its output combines several computations, a finalization candidate has those computations as required inputs; covering the root therefore covers all of them. +Candidate IDs are globally unique; `Candidates` denotes their union across groups. A producer requirement describes what must be supplied, not a producer already selected. For example, P95 requires KLL state; either compatible build may satisfy it. Requirements remain distinct: a join needs both `left` and `right`, even if the same producer can supply both. For this formulation, each query has one explicit output root. If its output combines several computations, a finalization candidate has those computations as required inputs; covering the root therefore covers all of them. -Three binary variables select nodes, dependency connections, and query-root assignments: +A candidate can be a producer on one connection and a consumer on another. Three binary variables select nodes, producer–consumer connections, and query-root assignments: ```text -x[c] ∈ {0,1} select candidate c -b[c,input,d] ∈ {0,1} connect source candidate d to input of c -r[q,c] ∈ {0,1} use candidate c to provide query root q +x[c] ∈ {0,1} select candidate c +connect[producer,consumer,requirement] ∈ {0,1} producer satisfies this consumer requirement +r[q,c] ∈ {0,1} candidate c provides query root q ``` -Connection variables exist only for compatible sources. Capacity and materialization belong to candidate configuration, so selection needs no separate copy or instance variables. +Connection variables exist only for compatible producer–consumer requirements. Capacity and materialization belong to candidate configuration, so selection needs no separate copy or instance variables. #### Objective @@ -236,7 +236,7 @@ Connection variables exist only for compatible sources. Capacity and materializa min Σ_c local_cost[c] × x[c] ``` -Each selected candidate owns its local work and state, excluding suppliers. This includes shared-resource candidates such as `Retain15m`, so no separate shared-resource charge is needed. Fixed request overhead is omitted from the objective because it is constant for the workload; Section 4 describes its inclusion in reported total cost. Where selected consumers or connections change cost, additional expressions model that dependence rather than treating it as a constant. +Each selected candidate owns its local work and state, excluding producers. This includes shared-resource candidates such as `Retain15m`, so no separate shared-resource charge is needed. Fixed request overhead is omitted from the objective because it is constant for the workload; Section 4 describes its inclusion in reported total cost. Where selected consumers or connections change cost, additional expressions model that dependence rather than treating it as a constant. ### 5.2 Constraints @@ -247,14 +247,16 @@ Each selected candidate owns its local work and state, excluding suppliers. This r[q,c] ≤ x[c] ``` -#### Supply every selected candidate's inputs +#### Satisfy every selected consumer's producer requirements ```text -Σ_{d ∈ CompatibleSources(c,input)} b[c,input,d] = x[c] -b[c,input,d] ≤ x[d] for each input ∈ Inputs(c) +Σ_{producer ∈ CompatibleProducers(consumer,requirement)} + connect[producer,consumer,requirement] = x[consumer] + +connect[producer,consumer,requirement] ≤ x[producer] ``` -Several consumers may connect to the same supplier. There is **no general** constraint `Σ_{c ∈ Candidates(g)} x[c] ≤ 1`: multiple candidates in one group may be needed. +The equality applies to every `requirement ∈ ProducerRequirements(consumer)`: each selected consumer must have every dependency supplied by one compatible producer. Several consumers may connect to the same producer. There is **no general** constraint `Σ_{c ∈ Candidates(g)} x[c] ≤ 1`: multiple candidates in one group may be needed. #### Shared raw-retention configuration @@ -264,12 +266,13 @@ For one raw-storage scope, retention candidates configure the same stored copy, Σ_{c ∈ Candidates(G_raw[d])} x[c] ≤ 1 ``` -This is a resource-specific constraint, not a restriction on every group. For a consumer needing 15m, its compatible suppliers include `Retain15m` and any generated longer-retention candidates, but exclude `Retain5m`. The input constraints above require a selected compatible supplier. A second consumer needing 5m can connect to the same `Retain15m`; storage is therefore charged once. Separate stored copies require distinct storage scopes and are charged separately. +This is a resource-specific constraint, not a restriction on every group. For a consumer needing 15m, its compatible producers include `Retain15m` and any generated longer-retention candidates, but exclude `Retain5m`. The dependency constraints above require a selected compatible producer. A second consumer needing 5m can connect to the same `Retain15m`; storage is therefore charged once. Separate stored copies require distinct storage scopes and are charged separately. #### Keep only connected, acyclic computation ```text -x[d] ≤ Σ_q r[q,d] + Σ_{c,input} b[c,input,d] +x[producer] ≤ Σ_q r[q,producer] + + Σ_{consumer,requirement} connect[producer,consumer,requirement] ``` Only defined connections appear in these sums. Every selected candidate must serve a query output or another selected candidate. Together with acyclicity, this excludes unused computation. @@ -277,7 +280,8 @@ Only defined connections appear in these sums. Every selected candidate must ser If candidate generation does not already guarantee acyclicity, introduce `0 ≤ order[c] ≤ N−1`, where `N = |Candidates|`: ```text -order[d] + 1 ≤ order[c] + N × (1 − b[c,input,d]) +order[producer] + 1 ≤ order[consumer] + + N × (1 − connect[producer,consumer,requirement]) ``` #### Enforce requirements @@ -290,7 +294,7 @@ For additive resource usage with fixed execution context, capacity constraints t Σ_c usage[k,c] × x[c] ≤ capacity[k] − fixed_usage[k] ``` -Here candidate usage includes shared raw-retention candidates. `fixed_usage[k]` is usage fixed by the workload and independent of plan selection, such as fixed request handling, measured in the same units and accounting context; it is zero when absent. Peak usage and temporary allocations need lifetime and concurrency accounting when a simple sum is inappropriate. These extensions preserve the same ownership rule: suppliers and shared resources are not charged again through each consumer. +Here candidate usage includes shared raw-retention candidates. `fixed_usage[k]` is usage fixed by the workload and independent of plan selection, such as fixed request handling, measured in the same units and accounting context; it is zero when absent. Peak usage and temporary allocations need lifetime and concurrency accounting when a simple sum is inappropriate. These extensions preserve the same ownership rule: producers and shared resources are not charged again through each consumer. Only exposed candidate outputs can supply consumers. A fused `topk(sum(...))` candidate does not automatically supply another query's full `sum(...)`; that result needs a separate candidate unless explicitly exposed. @@ -317,7 +321,7 @@ Both estimators connect to the same selected build candidate, so its cost appear Candidate generation maintains a worklist of new expressions and applicable rules. Interning prevents duplicate derivations, while compatibility checks eliminate unsupported connections. Generation is relative to a finite inventory of rules, configurations, and execution scopes. If exploration stops before closure or imposes a cap, the reported optimality claim must reflect that restriction. -Let `n` be the number of physical candidates, `m` the number of compatible input connections, and `r` the number of compatible query-output connections. The base formulation has `n + m + r` binary variables, plus ordering and any cost-model auxiliaries. This counts the model size, not its solution time. Even a restricted supplier-selection case contains set cover: suppliers cover subsets of outputs and have activation costs. Removing interchangeable instance slots reduces redundant representation but does not remove this combinatorial difficulty. +Let `n` be the number of physical candidates, `m` the number of compatible producer–consumer connections, and `r` the number of compatible query-output connections. The base formulation has `n + m + r` binary variables, plus ordering and any cost-model auxiliaries. This counts the model size, not its solution time. Even a restricted producer-selection case contains set cover: producers cover subsets of outputs and have activation costs. Removing interchangeable instance slots reduces redundant representation but does not remove this combinatorial difficulty. Search reductions must preserve joint choices. A candidate that costs more alone may enable useful sharing, so local cost dominance is insufficient. Independent components can be solved separately only when dependencies, sharing, and global constraints also separate. Generation caching and sparse compatibility edges reduce overhead without requiring enumeration of complete workload plans. From 6bc8c25639192b919b599e6c601bb46084cffc1c Mon Sep 17 00:00:00 2001 From: zzylol <50204836+zzylol@users.noreply.github.com> Date: Sat, 24 Oct 2026 01:22:05 +0000 Subject: [PATCH 45/53] docs: define topological rank variables and acyclicity constraint --- docs/design_docs/proposals/joint-planning-search.md | 13 ++++++++++++- 1 file changed, 12 insertions(+), 1 deletion(-) diff --git a/docs/design_docs/proposals/joint-planning-search.md b/docs/design_docs/proposals/joint-planning-search.md index efab99939..7f0952477 100644 --- a/docs/design_docs/proposals/joint-planning-search.md +++ b/docs/design_docs/proposals/joint-planning-search.md @@ -230,6 +230,15 @@ r[q,c] ∈ {0,1} candidate c provides query Connection variables exist only for compatible producer–consumer requirements. Capacity and materialization belong to candidate configuration, so selection needs no separate copy or instance variables. +If the candidate dependency graph may contain cycles, the model also introduces an auxiliary variable for every candidate: + +```text +N = |Candidates| +order[c] ∈ [0, N−1] continuous topological rank of candidate c +``` + +`order[c]` witnesses dependency order in the selected execution DAG; it is not a start time, latency, or runtime schedule. Ranks need not be unique or integral. Unselected candidates' ranks have no execution meaning. These variables can be omitted when the candidate dependency graph is already acyclic. + #### Objective ```text @@ -277,13 +286,15 @@ x[producer] ≤ Σ_q r[q,producer] Only defined connections appear in these sums. Every selected candidate must serve a query output or another selected candidate. Together with acyclicity, this excludes unused computation. -If candidate generation does not already guarantee acyclicity, introduce `0 ≤ order[c] ≤ N−1`, where `N = |Candidates|`: +Using the auxiliary `order[c]` variables defined in Section 5.1, impose the following constraint for every allowed producer–consumer connection: ```text order[producer] + 1 ≤ order[consumer] + N × (1 − connect[producer,consumer,requirement]) ``` +When `connect = 1`, the consumer's rank must exceed the producer's by at least one. When `connect = 0`, the `N` term makes the constraint nonbinding under the rank bounds. For example, `build → merge → estimate` can use ranks `0, 1, 2`. Adding `estimate → build` would require `order[build] ≥ order[estimate] + 1`, making the cycle infeasible. Independent candidates may have equal ranks. + #### Enforce requirements Candidate and connection compatibility eliminate unsupported operators, illegal materializations, and invalid window combinations. End-to-end accuracy still requires composition bounds, feasible-combination constraints, or sound feasibility refinement. Query deadlines are enforced under the configured latency model. From c6581a245d6dbe604d56f92b5c54cff5b0eaa03a Mon Sep 17 00:00:00 2001 From: zzylol <50204836+zzylol@users.noreply.github.com> Date: Sat, 24 Oct 2026 01:28:11 +0000 Subject: [PATCH 46/53] docs: distinguish shared cost ownership from execution frequency --- docs/design_docs/proposals/joint-planning-search.md | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/docs/design_docs/proposals/joint-planning-search.md b/docs/design_docs/proposals/joint-planning-search.md index 7f0952477..d5ab647fb 100644 --- a/docs/design_docs/proposals/joint-planning-search.md +++ b/docs/design_docs/proposals/joint-planning-search.md @@ -157,7 +157,9 @@ For a common evaluation of Q1 and Q2, the selected merge state serves both estim The cost of a shared plan is obtained from its selected execution nodes, not by summing independently priced query paths. Each node owns its local work and state, excluding its inputs. In the running example, the KLL build owns construction and retained state, while each estimator owns extraction. Two estimators using one build therefore incur one build charge and two extraction charges. -A node may execute repeatedly. Its materialization, consumer schedules, and lifetime determine how frequently its work is performed and how long its state occupies resources. A constant candidate cost is appropriate only when this context is fixed. All terms use a common horizon or consistent rates, so ingestion work and evaluation work can be compared. +**Charging a selected candidate once does not mean executing it once.** It means assigning its cost to one owner rather than repeating that cost for every consumer. If a shared KLL build is reconstructed every minute, its charge includes every reconstruction over the accounting horizon. For 60 common evaluations of P95 and P99, the build work costs `60 × B`, where `B` is one reconstruction's cost—not `B`, and not `120 × B`. + +The candidate's materialization, selected consumer schedules, and lifetime determine its execution count and state occupancy. When those quantities depend on selected connections, its cost must depend on those connections as well; it cannot be a constant fixed before selection. For example, a query-time build serving only a five-minute consumer executes 12 times over 60 aligned minute ticks. Connecting an additional one-minute consumer can increase that count to 60, with sharing at common ticks when legal. The model must derive this demand from the selected graph using execution-count/lifetime variables or consistent finite context configurations. All terms use a common horizon or consistent rates, so ingestion work and evaluation work can be compared. Some charges belong to a resource shared beyond one computation. If two consumers need 5m and 15m of the same raw source, the required retention horizon is 15m, not 20m. Represent this shared storage as a state group `G_raw[d]` for source `d` and a particular storage scope, with candidates such as `Retain5m` and `Retain15m`. The group describes access to retained raw data; each candidate advertises its coverage, and each consumer connects only to a candidate covering its required horizon. Both consumers can use `Retain15m`, whose storage cost is charged once through candidate selection. Reads and intermediate KLL state retain their own cost owners. @@ -241,12 +243,16 @@ order[c] ∈ [0, N−1] continuous topological rank of candidate c #### Objective +For candidates whose execution demand and lifetime are fixed, the objective is: + ```text min Σ_c local_cost[c] × x[c] ``` Each selected candidate owns its local work and state, excluding producers. This includes shared-resource candidates such as `Retain15m`, so no separate shared-resource charge is needed. Fixed request overhead is omitted from the objective because it is constant for the workload; Section 4 describes its inclusion in reported total cost. Where selected consumers or connections change cost, additional expressions model that dependence rather than treating it as a constant. +Here `local_cost[c]` covers the entire accounting horizon, not just one invocation. If selected connections change demand, the fixed-coefficient expression above is insufficient. The MILP must instead include the resulting execution and retention costs, with constraints tying execution counts and lifetimes to the selected connections. Selecting `x[c] = 1` alone determines neither the number of executions nor their total cost; the same accounting applies to resource limits. + ### 5.2 Constraints #### Cover every query output From e6f4666cdfd41d17f0703be5b5d222558dbe069c Mon Sep 17 00:00:00 2001 From: zzylol <50204836+zzylol@users.noreply.github.com> Date: Sat, 24 Oct 2026 01:32:14 +0000 Subject: [PATCH 47/53] docs: model execution demand from selected producer connections --- .../proposals/joint-planning-search.md | 55 +++++++++++++++++++ 1 file changed, 55 insertions(+) diff --git a/docs/design_docs/proposals/joint-planning-search.md b/docs/design_docs/proposals/joint-planning-search.md index d5ab647fb..c52839717 100644 --- a/docs/design_docs/proposals/joint-planning-search.md +++ b/docs/design_docs/proposals/joint-planning-search.md @@ -241,6 +241,18 @@ order[c] ∈ [0, N−1] continuous topological rank of candidate c `order[c]` witnesses dependency order in the selected execution DAG; it is not a start time, latency, or runtime schedule. Ranks need not be unique or integral. Unselected candidates' ranks have no execution meaning. These variables can be omitted when the candidate dependency graph is already acyclic. +#### Execution-demand variables + +For query-time recomputation on a fixed evaluation schedule, let `Ticks` be the finite evaluation events in the accounting horizon and `evaluates[q,t]` a known binary parameter indicating whether query `q` evaluates at event `t`. A shared event means compatible evaluation context, not merely equal wall-clock timestamps. Introduce: + +```text +run[c,t] ∈ {0,1} candidate c executes at event t +need[producer,consumer,requirement,t] ∈ {0,1} + the connected consumer needs this producer at t +``` + +This demand model assumes every required producer executes at the same event as its consumer, with one execution reusable by all compatible consumers at that event. It applies to query-time recomputation with legal within-evaluation sharing. Ingestion maintenance and retained state need their own update/freshness/lifetime constraints; the model below must not be applied to them unchanged. + #### Objective For candidates whose execution demand and lifetime are fixed, the objective is: @@ -253,6 +265,15 @@ Each selected candidate owns its local work and state, excluding producers. This Here `local_cost[c]` covers the entire accounting horizon, not just one invocation. If selected connections change demand, the fixed-coefficient expression above is insufficient. The MILP must instead include the resulting execution and retention costs, with constraints tying execution counts and lifetimes to the selected connections. Selecting `x[c] = 1` alone determines neither the number of executions nor their total cost; the same accounting applies to resource limits. +For the query-time demand model, replace the fixed-context objective with: + +```text +min Σ_c standing_cost[c] × x[c] + + Σ_{c,t} execution_cost[c,t] × run[c,t] +``` + +`standing_cost` contains only horizon-wide costs independent of execution count; `execution_cost` contains local per-execution costs, excluding producers. They partition the modeled charges rather than adding them to `local_cost` again. Any connection-dependent retention or peak-memory cost requires additional constraints. + ### 5.2 Constraints #### Cover every query output @@ -273,6 +294,40 @@ connect[producer,consumer,requirement] ≤ x[producer] The equality applies to every `requirement ∈ ProducerRequirements(consumer)`: each selected consumer must have every dependency supplied by one compatible producer. Several consumers may connect to the same producer. There is **no general** constraint `Σ_{c ∈ Candidates(g)} x[c] ≤ 1`: multiple candidates in one group may be needed. +#### Derive execution demand from selected connections + +For every compatible connection and event `t`, linearize the conjunction of a selected connection and a running consumer: + +```text +need[producer,consumer,requirement,t] ≤ connect[producer,consumer,requirement] +need[producer,consumer,requirement,t] ≤ run[consumer,t] +need[producer,consumer,requirement,t] ≥ connect[producer,consumer,requirement] + + run[consumer,t] − 1 +``` + +A candidate runs if a query root or a connected, running consumer demands it. For every candidate `c` and event `t`: + +```text +run[c,t] ≤ x[c] +run[c,t] ≥ evaluates[q,t] × r[q,c] for each compatible query root q +run[c,t] ≥ need[c,consumer,requirement,t] for each outgoing connection +run[c,t] ≤ Σ_q evaluates[q,t] × r[q,c] + + Σ_{consumer,requirement} need[c,consumer,requirement,t] +``` + +Only defined assignments and connections appear in the sums; an empty sum is zero. `evaluates` is a fixed parameter, so these expressions are linear. Together with acyclicity, the constraints propagate demand from query roots through every required producer and prevent unsupported execution. Multiple consumers demanding the same candidate at one event still produce a single `run[c,t] = 1`. + +For example, Q1 evaluates every minute and Q2 every five minutes on the same grid. Over 60 minute ticks, assuming compatible evaluations and fresh query-time builds: + +| Consumers connected to one KLL build | Build executions | +| --- | ---: | +| Q1 only | 60 | +| Q2 only | 12 | +| Q1 and Q2 | 60 | +| Neither, and no other demand | 0 | + +The build's execution cost is `Σ_t execution_cost[build,t] × run[build,t]`. Additive per-execution resource usage follows the same pattern in capacity constraints. This is a joint optimization, not a circular cost definition: the solver selects connections and execution variables together, constrained by the fixed query schedule. The objective compares their resulting costs. Execution-graph cycles remain a separate issue handled by the ordering constraints. + #### Shared raw-retention configuration For one raw-storage scope, retention candidates configure the same stored copy, so they are mutually exclusive: From 0b0bc8531149b4e4315f9b7eee493c211636553a Mon Sep 17 00:00:00 2001 From: zzylol <50204836+zzylol@users.noreply.github.com> Date: Sat, 24 Oct 2026 01:33:58 +0000 Subject: [PATCH 48/53] docs: define auxiliary variable indices semantics and examples --- .../proposals/joint-planning-search.md | 40 +++++++++++++------ 1 file changed, 28 insertions(+), 12 deletions(-) diff --git a/docs/design_docs/proposals/joint-planning-search.md b/docs/design_docs/proposals/joint-planning-search.md index c52839717..a44bcfd55 100644 --- a/docs/design_docs/proposals/joint-planning-search.md +++ b/docs/design_docs/proposals/joint-planning-search.md @@ -232,26 +232,42 @@ r[q,c] ∈ {0,1} candidate c provides query Connection variables exist only for compatible producer–consumer requirements. Capacity and materialization belong to candidate configuration, so selection needs no separate copy or instance variables. -If the candidate dependency graph may contain cycles, the model also introduces an auxiliary variable for every candidate: +#### Auxiliary variables: `order`, `run`, and `need` -```text -N = |Candidates| -order[c] ∈ [0, N−1] continuous topological rank of candidate c -``` +These are solver variables, not functions to be evaluated before optimization. Square brackets identify which candidate, connection, or evaluation event a variable belongs to. Their indices are: + +| Index or parameter | Definition | +| --- | --- | +| `c`, `producer`, `consumer` | Globally unique physical candidate IDs in `Candidates`. | +| `requirement` | One dependency in `ProducerRequirements(consumer)`, such as `KLL-state` or a join's `left` input. | +| `N` | The number of physical candidates, `|Candidates|`. | +| `Ticks` | A finite set of evaluation events over the accounting horizon. | +| `t` | One event in `Ticks`; it identifies an evaluation context, not a solver-chosen start time. | +| `evaluates[q,t]` | A fixed binary parameter: 1 if query `q` evaluates at event `t`, otherwise 0. | -`order[c]` witnesses dependency order in the selected execution DAG; it is not a start time, latency, or runtime schedule. Ranks need not be unique or integral. Unselected candidates' ranks have no execution meaning. These variables can be omitted when the candidate dependency graph is already acyclic. +A shared event requires compatible evaluation contexts, not merely equal wall-clock timestamps. The variables have the following meanings: -#### Execution-demand variables +| Variable | Domain and indexing | Definition | +| --- | --- | --- | +| `order[c]` | Continuous, `0 ≤ order[c] ≤ N−1`, for each candidate `c`. | A topological rank used to require each selected producer to precede its consumers in the dependency graph. It is not execution time, latency, or a runtime schedule. | +| `run[c,t]` | Binary, for each candidate `c` and event `t`. | 1 exactly when the selected graph requires candidate `c` to execute for event `t`; 0 otherwise. This records execution demand, not measured runtime activity. | +| `need[producer,consumer,requirement,t]` | Binary, for each compatible producer–consumer requirement and event `t`. | 1 exactly when that connection is selected **and** the consumer executes for event `t`; 0 otherwise. It records demand along one connection. | -For query-time recomputation on a fixed evaluation schedule, let `Ticks` be the finite evaluation events in the accounting horizon and `evaluates[q,t]` a known binary parameter indicating whether query `q` evaluates at event `t`. A shared event means compatible evaluation context, not merely equal wall-clock timestamps. Introduce: +For example, let `build` be a KLL producer, and `P95` and `P99` its estimator candidates. At evaluation event `t5`, suppose both estimators execute and both use `build` for their `KLL-state` requirement: ```text -run[c,t] ∈ {0,1} candidate c executes at event t -need[producer,consumer,requirement,t] ∈ {0,1} - the connected consumer needs this producer at t +x[build] = 1 build belongs to the selected plan +run[P95,t5] = run[P99,t5] = 1 both estimators execute for this event +need[build,P95,KLL-state,t5] = 1 the P95 connection demands the build +need[build,P99,KLL-state,t5] = 1 the P99 connection also demands it +run[build,t5] = 1 one shared build execution, not two ``` -This demand model assumes every required producer executes at the same event as its consumer, with one execution reusable by all compatible consumers at that event. It applies to query-time recomputation with legal within-evaluation sharing. Ingestion maintenance and retained state need their own update/freshness/lifetime constraints; the model below must not be applied to them unchanged. +At another event with no query or consumer demand, `run[build,t] = 0` even though `x[build] = 1`. Thus `x` selects a plan node, `need` records per-connection demand, and `run` combines demand into one execution per compatible event. The constraints in Section 5.2 enforce these definitions, including the logical AND defining `need`. + +For the subgraph `build → P95` and `build → P99`, valid ranks are `order[build] = 0` and `order[P95] = order[P99] = 1`. Ranks need not be unique or integral. They have no execution meaning for unselected candidates. `order` variables can be omitted if the candidate dependency graph is already acyclic; otherwise the ordering constraints in Section 5.2 exclude selected cycles. + +The `run`/`need` model assumes query-time recomputation on a fixed evaluation schedule: each required producer executes for the same event as its consumer, and compatible consumers can reuse that execution. The event is a logical evaluation, not a claim that all operators start simultaneously. Ingestion maintenance and retained state need their own update/freshness/lifetime constraints; this model must not be applied to them unchanged. #### Objective From 386d13bc91875b54c6dd38114f916820964ff0a0 Mon Sep 17 00:00:00 2001 From: zzylol <50204836+zzylol@users.noreply.github.com> Date: Sat, 24 Oct 2026 01:36:07 +0000 Subject: [PATCH 49/53] docs: clarify query evaluation event terminology --- .../proposals/joint-planning-search.md | 28 +++++++++---------- 1 file changed, 14 insertions(+), 14 deletions(-) diff --git a/docs/design_docs/proposals/joint-planning-search.md b/docs/design_docs/proposals/joint-planning-search.md index a44bcfd55..7786e3079 100644 --- a/docs/design_docs/proposals/joint-planning-search.md +++ b/docs/design_docs/proposals/joint-planning-search.md @@ -234,40 +234,40 @@ Connection variables exist only for compatible producer–consumer requirements. #### Auxiliary variables: `order`, `run`, and `need` -These are solver variables, not functions to be evaluated before optimization. Square brackets identify which candidate, connection, or evaluation event a variable belongs to. Their indices are: +These are solver variables, not functions to be evaluated before optimization. Square brackets identify which candidate, connection, or query evaluation event a variable belongs to. Their indices are: | Index or parameter | Definition | | --- | --- | | `c`, `producer`, `consumer` | Globally unique physical candidate IDs in `Candidates`. | | `requirement` | One dependency in `ProducerRequirements(consumer)`, such as `KLL-state` or a join's `left` input. | | `N` | The number of physical candidates, `|Candidates|`. | -| `Ticks` | A finite set of evaluation events over the accounting horizon. | -| `t` | One event in `Ticks`; it identifies an evaluation context, not a solver-chosen start time. | -| `evaluates[q,t]` | A fixed binary parameter: 1 if query `q` evaluates at event `t`, otherwise 0. | +| `Ticks` | A finite set of query evaluation events over the accounting horizon. | +| `t` | One query evaluation event in `Ticks`; it identifies an evaluation context, not a solver-chosen start time. | +| `evaluates[q,t]` | A fixed binary parameter: 1 if query `q` evaluates at query evaluation event `t`, otherwise 0. | -A shared event requires compatible evaluation contexts, not merely equal wall-clock timestamps. The variables have the following meanings: +A shared query evaluation event requires compatible evaluation contexts, not merely equal wall-clock timestamps. The variables have the following meanings: | Variable | Domain and indexing | Definition | | --- | --- | --- | | `order[c]` | Continuous, `0 ≤ order[c] ≤ N−1`, for each candidate `c`. | A topological rank used to require each selected producer to precede its consumers in the dependency graph. It is not execution time, latency, or a runtime schedule. | -| `run[c,t]` | Binary, for each candidate `c` and event `t`. | 1 exactly when the selected graph requires candidate `c` to execute for event `t`; 0 otherwise. This records execution demand, not measured runtime activity. | -| `need[producer,consumer,requirement,t]` | Binary, for each compatible producer–consumer requirement and event `t`. | 1 exactly when that connection is selected **and** the consumer executes for event `t`; 0 otherwise. It records demand along one connection. | +| `run[c,t]` | Binary, for each candidate `c` and query evaluation event `t`. | 1 exactly when the selected graph requires candidate `c` to execute for query evaluation event `t`; 0 otherwise. This records execution demand, not measured runtime activity. | +| `need[producer,consumer,requirement,t]` | Binary, for each compatible producer–consumer requirement and query evaluation event `t`. | 1 exactly when that connection is selected **and** the consumer executes for query evaluation event `t`; 0 otherwise. It records demand along one connection. | -For example, let `build` be a KLL producer, and `P95` and `P99` its estimator candidates. At evaluation event `t5`, suppose both estimators execute and both use `build` for their `KLL-state` requirement: +For example, let `build` be a KLL producer, and `P95` and `P99` its estimator candidates. At query evaluation event `t5`, suppose both estimators execute and both use `build` for their `KLL-state` requirement: ```text x[build] = 1 build belongs to the selected plan -run[P95,t5] = run[P99,t5] = 1 both estimators execute for this event +run[P95,t5] = run[P99,t5] = 1 both estimators execute for this query evaluation event need[build,P95,KLL-state,t5] = 1 the P95 connection demands the build need[build,P99,KLL-state,t5] = 1 the P99 connection also demands it run[build,t5] = 1 one shared build execution, not two ``` -At another event with no query or consumer demand, `run[build,t] = 0` even though `x[build] = 1`. Thus `x` selects a plan node, `need` records per-connection demand, and `run` combines demand into one execution per compatible event. The constraints in Section 5.2 enforce these definitions, including the logical AND defining `need`. +At another query evaluation event with no query or consumer demand, `run[build,t] = 0` even though `x[build] = 1`. Thus `x` selects a plan node, `need` records per-connection demand, and `run` combines demand into one execution per compatible query evaluation event. The constraints in Section 5.2 enforce these definitions, including the logical AND defining `need`. For the subgraph `build → P95` and `build → P99`, valid ranks are `order[build] = 0` and `order[P95] = order[P99] = 1`. Ranks need not be unique or integral. They have no execution meaning for unselected candidates. `order` variables can be omitted if the candidate dependency graph is already acyclic; otherwise the ordering constraints in Section 5.2 exclude selected cycles. -The `run`/`need` model assumes query-time recomputation on a fixed evaluation schedule: each required producer executes for the same event as its consumer, and compatible consumers can reuse that execution. The event is a logical evaluation, not a claim that all operators start simultaneously. Ingestion maintenance and retained state need their own update/freshness/lifetime constraints; this model must not be applied to them unchanged. +The `run`/`need` model assumes query-time recomputation on a fixed evaluation schedule: each required producer executes for the same query evaluation event as its consumer, and compatible consumers can reuse that execution. The query evaluation event is a logical evaluation, not a claim that all operators start simultaneously. Ingestion maintenance and retained state need their own update/freshness/lifetime constraints; this model must not be applied to them unchanged. #### Objective @@ -312,7 +312,7 @@ The equality applies to every `requirement ∈ ProducerRequirements(consumer)`: #### Derive execution demand from selected connections -For every compatible connection and event `t`, linearize the conjunction of a selected connection and a running consumer: +For every compatible connection and query evaluation event `t`, linearize the conjunction of a selected connection and a running consumer: ```text need[producer,consumer,requirement,t] ≤ connect[producer,consumer,requirement] @@ -321,7 +321,7 @@ need[producer,consumer,requirement,t] ≥ connect[producer,consumer,requirement] + run[consumer,t] − 1 ``` -A candidate runs if a query root or a connected, running consumer demands it. For every candidate `c` and event `t`: +A candidate runs if a query root or a connected, running consumer demands it. For every candidate `c` and query evaluation event `t`: ```text run[c,t] ≤ x[c] @@ -331,7 +331,7 @@ run[c,t] ≤ Σ_q evaluates[q,t] × r[q,c] + Σ_{consumer,requirement} need[c,consumer,requirement,t] ``` -Only defined assignments and connections appear in the sums; an empty sum is zero. `evaluates` is a fixed parameter, so these expressions are linear. Together with acyclicity, the constraints propagate demand from query roots through every required producer and prevent unsupported execution. Multiple consumers demanding the same candidate at one event still produce a single `run[c,t] = 1`. +Only defined assignments and connections appear in the sums; an empty sum is zero. `evaluates` is a fixed parameter, so these expressions are linear. Together with acyclicity, the constraints propagate demand from query roots through every required producer and prevent unsupported execution. Multiple consumers demanding the same candidate at one query evaluation event still produce a single `run[c,t] = 1`. For example, Q1 evaluates every minute and Q2 every five minutes on the same grid. Over 60 minute ticks, assuming compatible evaluations and fresh query-time builds: From 8933389c87bc9678c75b75add6cee3046dd60e05 Mon Sep 17 00:00:00 2001 From: zzylol <50204836+zzylol@users.noreply.github.com> Date: Sat, 24 Oct 2026 01:39:10 +0000 Subject: [PATCH 50/53] docs: specify feasibility and resource constraints for joint optimization --- .../proposals/joint-planning-search.md | 80 +++++++++++++++++-- 1 file changed, 74 insertions(+), 6 deletions(-) diff --git a/docs/design_docs/proposals/joint-planning-search.md b/docs/design_docs/proposals/joint-planning-search.md index 7786e3079..07918b9de 100644 --- a/docs/design_docs/proposals/joint-planning-search.md +++ b/docs/design_docs/proposals/joint-planning-search.md @@ -292,6 +292,8 @@ min Σ_c standing_cost[c] × x[c] ### 5.2 Constraints +A feasible plan must cover every query, meet every query's accuracy target, provide executable implementations and valid materializations, and respect configured CPU, memory, disk, and latency limits. Dependency, execution-demand, configuration, and acyclicity constraints below make those requirements consistent across shared computation. Limits are imposed when configured; a missing limit does not mean that its resource has zero cost. + #### Cover every query output ```text @@ -372,19 +374,85 @@ order[producer] + 1 ≤ order[consumer] When `connect = 1`, the consumer's rank must exceed the producer's by at least one. When `connect = 0`, the `N` term makes the constraint nonbinding under the rank bounds. For example, `build → merge → estimate` can use ranks `0, 1, 2`. Adding `estimate → build` would require `order[build] ≥ order[estimate] + 1`, making the cycle infeasible. Independent candidates may have equal ranks. -#### Enforce requirements +#### Physical implementation for every selected operation + +A physical candidate must identify an executable operator for every operation it owns, including fused internal operations. Let `executable[c]` be a fixed Boolean obtained from the operator registry and deployment capabilities: + +```text +x[c] ≤ executable[c] for every candidate c +``` + +Unsupported candidates can equivalently be omitted. Coverage and dependency constraints then ensure that every required operation has a selected implementation; logical expressions alone cannot be selected for execution. Only exposed outputs can supply consumers. A fused `topk(sum(...))` does not supply another query's full `sum(...)` unless that output is explicitly exposed. + +#### Valid materialization and window configuration + +Materialization is bundled into physical candidates. Define fixed Boolean parameters `valid_materialization[c]` and `valid_window[c]` for locally checkable legality, and `compatible[producer,consumer,requirement]` for pairwise legality: + +```text +x[c] ≤ valid_materialization[c] +x[c] ≤ valid_window[c] +connect[producer,consumer,requirement] + ≤ compatible[producer,consumer,requirement] +``` + +These checks include supported modes, deployment capabilities, pane alignment and coverage, retention guarantees, and producer/consumer timing. Omitting invalid candidates and connections implements the same restrictions. For example, an ingestion-maintained candidate cannot depend on data available only at query time, and a 15m consumer cannot connect to a raw-retention candidate covering only 5m. + +Consumer-dependent freshness, invalidation, and lifetime checks may involve several selected connections; pairwise compatibility alone does not establish them. Such checks require a valid joint linear encoding or the feasibility cuts defined below. Retained-state candidates are eligible only with a defined refresh/update policy and its cost. The query-time `run` equations do not constitute an ingestion or retained-state model. + +#### Every query meets its accuracy target + +For each query `q`, the configured accuracy contract specifies an error metric and target, and, when probabilistic, a failure-probability target. A plan is feasible only if its composed output error and failure probability satisfy those targets. These are end-to-end checks, not a sum of arbitrary local sketch errors. + +When proven composition rules exist, encode their bounds directly. Otherwise use the exact finite-assignment exclusion below for each plan whose validated accuracy fails. For example, if a selected P99 estimator and its small KLL producer violate the target regardless of all other choices, the following focused cut is sound: + +```text +connect[small-build,estimate-P99,KLL-state] = 0 +``` + +If upstream choices or other connections affect the error, this focused cut is insufficiently justified; exclude the complete failing assignment or a proven sufficient subset instead. Each query must pass, including those sharing the same producer. An exact operator above approximate input is not automatically exact end to end. + +#### CPU, memory, and disk capacities + +Resource coefficients have explicit ownership and units. For query-time recomputation, let `cpu_exec[c,t]` be local CPU-seconds per execution, `cpu_standing[c]` local background CPU-seconds over the horizon, and `cpu_fixed` nonselectable workload CPU-seconds. With `CPU_budget` measured over the same horizon: + +```text +Σ_c cpu_standing[c] × x[c] + + Σ_{c,t} cpu_exec[c,t] × run[c,t] + + cpu_fixed ≤ CPU_budget +``` + +A core limit can give a CPU-time budget through available cores times horizon duration and an allowed utilization factor. This aggregate bound does not guarantee short-term throughput or deadlines; configured interval budgets and latency checks must also hold. Ingestion-maintained candidates need separately modeled update demand rather than the query-time `run` term. + +For a conservative capacity model, let `resident_memory[c]` and `retained_disk[c]` bound persistent allocation, and `temporary_memory[c]` and `temporary_disk[c]` bound peak local temporary allocation per candidate under the permitted concurrency. All are bytes; fixed allocations are `memory_fixed` and `disk_fixed`: + +```text +Σ_c (resident_memory[c] + temporary_memory[c]) × x[c] + + memory_fixed ≤ Memory_capacity + +Σ_c (retained_disk[c] + temporary_disk[c]) × x[c] + + disk_fixed ≤ Disk_capacity +``` + +These bounds conservatively allow all selected candidates' allocations to overlap. They are safe only when the coefficients bound each candidate's concurrent executions, not merely one invocation. A phase-aware allocation model may admit more plans, but must encode actual overlap rather than substituting average occupancy. Shared state and raw retention are counted through their owning candidates once. Memory-time and disk-time used in billing are not substitutes for capacity bounds. + +#### Every query meets its latency limit + +For each query `q`, the configured evaluator must establish latency at or below `Latency_limit[q]` for all required query evaluation events under the declared deployment assumptions. The latency quantity must match the requirement, including any freshness or ingestion-to-visibility component. CPU-time summed over the workload is not query latency, and `order[c]` is not an execution schedule. + +Where the configured latency model has a valid linear encoding, include it directly. Otherwise evaluate the selected graph and add a feasibility cut for a failing assignment. For instance, if a particular query-time build path exceeds Q1's deadline but an ingestion-maintained pane path does not, the failing path is excluded without excluding the pane choice. A focused path cut is valid only if omitted choices cannot repair its latency; otherwise use the full-assignment cut below. -Candidate and connection compatibility eliminate unsupported operators, illegal materializations, and invalid window combinations. End-to-end accuracy still requires composition bounds, feasible-combination constraints, or sound feasibility refinement. Query deadlines are enforced under the configured latency model. +#### Feasibility cuts for checks without a direct linear encoding -For additive resource usage with fixed execution context, capacity constraints take the form: +This rule makes deferred accuracy, latency, and joint materialization checks explicit. Let `v` be the vector of binary plan decisions: candidate selection, producer–consumer connections, root assignments, and any additional discrete policy configurations affecting the check. With a fixed schedule and acyclic graph, the query-time `run` and `need` values are derived from these decisions. For a rejected assignment `v_bar`, add: ```text -Σ_c usage[k,c] × x[c] ≤ capacity[k] − fixed_usage[k] +Σ_{i: v_bar[i] = 1} (1 − v[i]) + + Σ_{i: v_bar[i] = 0} v[i] ≥ 1 ``` -Here candidate usage includes shared raw-retention candidates. `fixed_usage[k]` is usage fixed by the workload and independent of plan selection, such as fixed request handling, measured in the same units and accounting context; it is zero when absent. Peak usage and temporary allocations need lifetime and concurrency accounting when a simple sum is inappropriate. These extensions preserve the same ownership rule: producers and shared resources are not charged again through each consumer. +The constraint requires at least one plan decision to change. For example, rejecting `(v1,v2,v3) = (1,1,0)` gives `(1−v1) + (1−v2) + v3 ≥ 1`. The reason must be a definite failure under fixed configured models, not evaluator timeout or numerical uncertainty. If continuous settings can repair the plan, excluding only its binary assignment is not sound; include a valid continuous constraint or discretize the relevant settings. -Only exposed candidate outputs can supply consumers. A fused `topk(sum(...))` candidate does not automatically supply another query's full `sum(...)`; that result needs a separate candidate unless explicitly exposed. +The complete finite feasibility specification contains this inequality for every assignment violating any configured accuracy, materialization, window, resource, or latency requirement. Enumerating all such assignments is generally impractical, so a solver may add them lazily: solve, validate every requirement, exclude a failing assignment, and re-solve. Every returned plan must pass all checks. Exhaustion establishes infeasibility only for the declared finite inventory and models; timeout does not. These feasibility cuts do not repair an underestimated objective—cost refinement requires separate valid bounds. ## 6. Example: Shared KLL From 633e1657b40627ab6c98ee0a355ea334473244cd Mon Sep 17 00:00:00 2001 From: zzylol <50204836+zzylol@users.noreply.github.com> Date: Sat, 24 Oct 2026 01:50:18 +0000 Subject: [PATCH 51/53] docs: motivate workload pattern and repeated query optimization --- docs/design_docs/proposals/joint-planning-search.md | 13 +++++++++++++ 1 file changed, 13 insertions(+) diff --git a/docs/design_docs/proposals/joint-planning-search.md b/docs/design_docs/proposals/joint-planning-search.md index 07918b9de..d6ab1acb7 100644 --- a/docs/design_docs/proposals/joint-planning-search.md +++ b/docs/design_docs/proposals/joint-planning-search.md @@ -6,6 +6,19 @@ ASAP primitives increase both the opportunities for query optimization and the difficulty of searching for a good plan. A quantile, for example, may be computed directly or estimated from a summary. Several quantiles may use the same summary, and several windows may use overlapping sets of panes. The cost of one query therefore depends on decisions made for other queries. Selecting the cheapest implementation of each query independently can miss the cheapest workload plan. +ASAPPlanner also optimizes for **workload patterns**, including repeated queries, their evaluation frequencies and alignment, and overlap among their windows. The unit of optimization is therefore a workload over time, not just a set of queries executed once. These patterns introduce further choices about when to recompute, which consumers can share an execution, and which intermediate states to maintain or retain across evaluations. + +For example, consider two queries over the same input and aligned 15m window: + +```text +Q1: P95(X[15m]), evaluated every 1m +Q2: P99(X[15m]), evaluated every 5m +``` + +Over 60 aligned minute ticks, separate query-time KLL builds require 60 + 12 executions. A compatible shared build can serve both queries at the 12 common query evaluation events, reducing the total to 60 builds. Another supported candidate may maintain 1m panes at ingestion and merge them at evaluation, trading update and retained-state costs against repeated whole-window construction. Eligible retained-state candidates introduce refresh and lifetime choices as well. Repetition alone does not make an earlier window's state valid for a later window; reuse must preserve coverage and freshness. + +The search space therefore grows across algorithm, configuration, consumer sharing, materialization, and temporal reuse choices together. Selecting a producer's consumers can change its execution frequency and required retention, so these decisions cannot be priced independently. Explicitly expanding every query evaluation event can further multiply the optimization model's size, even when query texts and algorithm candidates remain unchanged. Workload-pattern optimization thus adds both a combinatorial planning challenge and a need for compact representation of repeated execution demand. + The problem addressed here is to find a minimum-cost execution DAG that produces every requested result and satisfies accuracy, latency, and resource requirements. The optimizer must account for shared computation and retained data without charging them repeatedly along different query paths. It must also keep its own search time and memory manageable as the number of queries and implementation choices grows. The following sections develop a memo-based representation and a joint selection procedure for this problem. The proposal concerns the finite candidates supported by the planner and its configured cost and constraint models. It does not introduce new sketch or window algorithms, or a runtime scheduler. An optimal solution consequently means an optimum within the generated model, not among all imaginable execution plans. From ea290e944f020acc2393ebc5497ce1229e157a14 Mon Sep 17 00:00:00 2001 From: zzylol <50204836+zzylol@users.noreply.github.com> Date: Sat, 24 Oct 2026 01:53:43 +0000 Subject: [PATCH 52/53] docs: add workload-specific planning and execution expectations --- .../design_docs/proposals/joint-planning-search.md | 14 ++++++++++++++ 1 file changed, 14 insertions(+) diff --git a/docs/design_docs/proposals/joint-planning-search.md b/docs/design_docs/proposals/joint-planning-search.md index d6ab1acb7..53d279617 100644 --- a/docs/design_docs/proposals/joint-planning-search.md +++ b/docs/design_docs/proposals/joint-planning-search.md @@ -503,3 +503,17 @@ The design extends existing memo and selection structures. `MemoGroup` stores re Implementation proceeds from initial memo identities and rule expansion to physical configurations and connections, then joint selection and validation. State-local choices belong in candidates. Decisions genuinely spanning candidates remain explicit constraints. `SharedDecision` and `SharedDecisionKind` are the proposed names for current `CouplingSite` and `CouplingKind`; no Rust types are renamed by this document. Evaluation must distinguish generation coverage from selection correctness. Small workloads are exhaustively enumerated over candidate subsets and compatible connections under the same inventory, then compared with MILP results. Cases include multiple selected candidates in one group, shared build costs, separate execution scopes, fusion outputs, and composed accuracy/resource requirements. Scaling experiments measure candidate and connection counts, generation and solve time, planner memory, and validated execution savings. Together these measurements establish whether the representation avoids repeated search work and whether joint selection justifies its planning cost. + +### Expected planning performance and execution benefits + +Planning performance and the selected plan's execution performance are separate outcomes. The following expectations are hypotheses to test, not measured speedups or solver-time guarantees. + +| Workload characteristic | Expected planning performance | Expected execution benefit | +| --- | --- | --- | +| Queries have little opportunity to share. | Smaller components are easier to solve when dependencies and global constraints permit independent optimization. | Limited savings from sharing. | +| Many queries reuse a small number of states. | Memo reuse reduces repeated candidate generation; compatible connection counts still need control. | Strong opportunity to reduce repeated builds and state costs. | +| Candidates and compatible connections are dense. | MILP can become harder, especially with many possible partial-sharing configurations. | Potentially substantial savings, but proving optimality can be expensive. | +| Many window, capacity, and materialization configurations are available. | Both candidate generation and solving can grow substantially. | Benefits depend on the workload and which configurations are actually useful. | +| Accuracy and latency rely mainly on post-solve checks. | Repeated validation and refinement can make planning time highly variable. | Benefits depend on the feasible plans remaining after validation. | + +Benchmark these cases separately, recording generation, model-construction, solve, and validation time, refinement rounds, peak planner memory, and validated execution cost. A richer candidate space can improve the modeled optimum while increasing planning overhead; under a fixed solver time limit, it does not guarantee a better returned plan. From 30edb3c1719e097b16ca2c99a5fd06279c8d48da Mon Sep 17 00:00:00 2001 From: zzylol <50204836+zzylol@users.noreply.github.com> Date: Sat, 24 Oct 2026 01:57:40 +0000 Subject: [PATCH 53/53] docs: enumerate planner benchmark metrics --- .../proposals/joint-planning-search.md | 17 ++++++++++++++++- 1 file changed, 16 insertions(+), 1 deletion(-) diff --git a/docs/design_docs/proposals/joint-planning-search.md b/docs/design_docs/proposals/joint-planning-search.md index 53d279617..1198ece2d 100644 --- a/docs/design_docs/proposals/joint-planning-search.md +++ b/docs/design_docs/proposals/joint-planning-search.md @@ -516,4 +516,19 @@ Planning performance and the selected plan's execution performance are separate | Many window, capacity, and materialization configurations are available. | Both candidate generation and solving can grow substantially. | Benefits depend on the workload and which configurations are actually useful. | | Accuracy and latency rely mainly on post-solve checks. | Repeated validation and refinement can make planning time highly variable. | Benefits depend on the feasible plans remaining after validation. | -Benchmark these cases separately, recording generation, model-construction, solve, and validation time, refinement rounds, peak planner memory, and validated execution cost. A richer candidate space can improve the modeled optimum while increasing planning overhead; under a fixed solver time limit, it does not guarantee a better returned plan. +### Benchmark metrics + +Benchmark each workload case separately and report: + +| Metric | What to measure | +| --- | --- | +| Candidate generation time | Time spent building memo groups, applying rules, and generating legal physical candidates and compatible connections. | +| Model construction time | Time spent creating MILP variables, coefficients, and constraints, including rebuilding or extending the model during refinement. | +| MILP solve time | Time inside the solver, accumulated across all refinement rounds. | +| Validation/refinement rounds | Number of plans validated and refinement rounds triggered; also report validation time separately from solve time. | +| Peak planner memory | Peak memory used during planning, including candidate generation, model construction, solver execution, and validation. | +| Validated execution cost | Cost of the returned feasible plan under the configured evaluator and common accounting horizon; distinguish this estimate from measured runtime cost. | + +Report total planning wall time alongside these metrics, using consistent timing boundaries to avoid double counting. + + A richer candidate space can improve the modeled optimum while increasing planning overhead; under a fixed solver time limit, it does not guarantee a better returned plan.