[zzylol]

Landing on main (2026-11-02)

This PR now goes to main, and it makes the group MILP Stage 3's default search. The code is merged as it stands. The review of the group MILP (#1015, section "Review of the group MILP") lists what to revise next:

• note_inexact gating the cost-equality check;
• the duplicated run resolution;
• ResultSlot workarounds;
• tests against enumeration;
• a debug dump.

Changes since the draft:

• Rebase onto main. Rebased onto main after feat: refresh windows: refresh groups and a per-window ingestion charge (#1151, lands #1152–#1154 on main) #1182 (Refresh ingestion-time views that share a refresh window in one run, and charge a shared stateless ingestion node once per run #1151's refresh windows, which feat(plan-selection): one ingestion pipeline per refresh window in the selection MILP (#1095 §2.5 O3(b)) #1157 builds on), refactor: rename every remaining pane name to the tumbling-window wording (#1163, part 2) #1176 (Rename PaneChain to SummaryTumblingWindowChain, and every pane name to the tumbling-window wording #1163 part 2) and test: ignore tests that plan under a wall-clock budget (#1175) #1181.

• Two rename fix-ups. They carry Rename PaneChain to SummaryTumblingWindowChain, and every pane name to the tumbling-window wording #1163's tumbling-window names, parts 1 and 2, into this PR's code.

• New commit "the group MILP is Stage 3's default search ([Feature] Stage 3: no execution-scope copies, exact candidate-level MILP, one cost formula #1095 C6)". select_plan runs the group MILP unless ASAP_STAGE3_SEARCH=old. Inputs it does not price yet return ErrorSelectingPlan::UnsupportedCostModel, with no fallback to the old engine:

  • cost models 2 and 3;
  • a measured operation cost table without a frozen calibration factor.

  ASAP_STAGE3_SEARCH=milp is no longer needed.

Tests under the new default. Seven tests fail with the group MILP, and #1181 already ignores all seven, as nondeterministic under wall-clock budgets (#1175). Each passes with ASAP_STAGE3_SEARCH=old. The gaps they hit, also listed on #1015:

• 4 forced-heap runtime_install tests: an accuracy model stricter than the built-in rule (HeapPlansOnly) is checked only after the solve, with no re-solve.
• 2 stage_pipeline tests of cost models 2 and 3: now UnsupportedCostModel.
• raw_data_windows_of_unsupported_shapes_are_refused: the query is still refused, but the MILP reports Infeasible without the concrete raw-data-window reason.

The release-mode snapshot tests (corpus_snapshots, sketch_plan_snapshots, #[ignore]) were not regenerated under the new default. Their selected plans may change.

Why

#1095 §2.0 replaces the owner/instance selection MILP with a group-level MILP over execution contexts: each query evaluation, the ingestion pipeline I, and each selected shareable candidate m. This PR is step 1 of the design's solving strategy (one MILP with a warm start, then measure), so the design can be judged on numbers before any decomposition (3a–3c). It was first a draft for visibility; it now lands on main as the default search (see "Landing on main" above).

What

• Stage 3 solves the group MILP (the default; ASAP_STAGE3_SEARCH=old runs the old engine) once per planning round, from a warm start, decodes it to DagOfSelectedCandidates, realizes it once, and checks plan_cost and satisfies.
• On every memo example measured, the MILP's objective equals plan_cost of the decoded plan. It solves to optimality and matches today's engine. ratio-of-quantiles now plans at 2.407e-6, down from 5.27e-6 under the owner model. Planner aborts with bad_alloc under 8 GB on random-mixed-seed-862 workload-01 when raw data is retained #1036 workload-01 plans within 8 GB (0.9 GB peak) at 1.417e-6 (owner model: 2.006e-6).
• ASAP_STAGE3_REPORT=<path> appends one JSON line per Stage 3 run: size per family and per scope (the m contexts apart), timings, warm-start cost, objective, plan_cost and the dual bound. This is for measurement only.

How

• groups.rs (was owners.rs): dominance, per-candidate usage and accuracy weights come from the owner model. New in this file:

  • RunKind (per context, ingestion, shareable);
  • ResultSlot: what "one candidate per memo group per context" ranges over;
  • execution_contexts;
  • scan usage at every union of the window kinds read within one context.

• milp/model.rs: p[ctx, c], x[m], reach a[q, m], rows 1–9, slot links for inputs over several slots, order ranks inside and across contexts, latency in scaled units, and the solve.

• milp/cost_terms.rs holds the run terms:

  • plain;
  • producer-dependent, on AND(run, p[ctx, c']);
  • exact scan-window sets, δ_S.

  It also holds the schedule-group rates z, held memory (H_query, H, W), raw retention levels v and pane retention levels π.

• milp/warm_start.rs: each query's cheapest plan, then greedy sharing, over several query orders. Every candidate plan is costed exactly by the completion, which gives the cheapest continuous values for given binaries.

• milp/decode.rs: one node per (context, computed candidate) and one per selected shareable candidate.

• asap-cost, one hook (execution_demand::computations): a NotMaterialized ingestion-time node runs once per second for all consumers. This is §2.0's I. Before, it was charged once per consumer.

Key code interfaces added / modified

Crate-private except the following, all restored from tag stage3-owner-milp-redesign:

crates/plan-selection/src/report.rs

pub struct SearchReport {
    /// Set when plan_cost differs from the objective and some term was encoded inexactly.
    pub inexact_objective: Option<InexactObjective>, // new
    // other fields unchanged
}
/// The objective, plan_cost, and the inexact terms (new).
pub struct InexactObjective { pub objective_per_second: f64, pub plan_cost_per_second: f64, pub inexact_terms: BTreeMap<String, usize> }
pub enum ErrorSelectingPlan {
    /// Cost models 2/3 under the new search (unsatisfiable request).
    UnsupportedCostModel(String),                    // new
    /// Objective != plan_cost with no inexact term (internal bug).
    ObjectiveDiffersFromPlanCost { objective_per_second: f64, plan_cost_per_second: f64 }, // new
    /// The decoded plan fails accuracy or latency (internal bug).
    SelectedPlanFailedCheck { cause: RejectionCause, reason: String }, // new
    // other variants unchanged
}
pub enum RejectionCause { /// the MILP is infeasible
    Infeasible, /* new; others unchanged */ }

crates/types/src/stages/pruning.rs: PruneReason::DominatedBy(PhysicalCandidateId) and PruneReason::UsageNotCostable are new; Stage 3 prunes these before the MILP.
crates/cost/src/lib.rs: retained_panes and maintained_window_ms are pub again.

Usage:

source /mydata/asap-build-env.sh <worktree>
ASAP_STAGE3_SEARCH=milp ASAP_STAGE3_REPORT=target/meas/t5.report \
  target/release/stage_pipeline --workload tools/dag-viewer/memo-examples/promql/rangeheavy-t5-q05-q09-1h.json \
  --cost-model aws-fargate-us-east-1 --enforce-accuracy-guarantees --milp-time-limit-s 300 --no-memo \
  --deployment-capabilities '{"raw_data_retained": true, "query_time_results_retained": false}' --out t5.json
# t5.json: stage3_plan_selection.cost.cost_per_second = 3.9901e-6, search_report.status = "optimal",
# milp_variables = 6992; t5.report: per round, scopes.SharedContexts = the m contexts' share.

Major algorithm changes

Stage 3 selection: group MILP over execution contexts (behind the switch)

• Problem: the owner model gave each shareable candidate m_s instances, each owning a region. On ratio-of-quantiles it recomputed the stateless subtree per accumulator owner (5.27e-6 against 2.41e-6 for today's engine). Planner aborts with bad_alloc under 8 GB on random-mixed-seed-862 workload-01 when raw data is retained #1036 workload-01 needed 584k variables and stopped at the deadline at 2.006e-6.
• Before / after: the owner MILP had x[(s, j)] instances, per-owner links e/b and symmetry rows. Now:
  1. contexts and options (execution_contexts);
  2. p[ctx, c] with one candidate per slot per context;
  3. validity and "used only if read" per context;
  4. x[m] charged once;
  5. reach, latency (T[q, m] per query-time m) and A1 accuracy per run;
  6. the objective from asap-cost numbers;
  7. a warm start;
  8. one HiGHS solve;
  9. the incumbent's objective recomputed from its rounded binaries;
  10. decode, realize, check.
• Correctness:
  • Every cost term's auxiliary variable appears only where a larger value costs more. Scan windows are exact sets (σ exact from both sides, δ_S per set), costed by asap-cost at each union, so they need no monotonicity assumption.
  • Tests: milp::model::tests::*. Each test solves, decodes, validates, realizes and checks objective == plan_cost. Together they cover:
    • held memory with readers of different latency;
    • raw retention;
    • a scan read through two offsets;
    • a kept pane chain with readers of different spans;
    • a shared MatAtQuery result's NotMaterialized input run once;
    • one ingestion scan read by two sketch configurations, charged once;
    • the ratio-of-quantiles avg shape computed once;
    • a phantom reader that cannot satisfy MinimumQueryReaders;
    • decode node counts;
    • an infeasible latency bound;
    • cost models 2 and 3 unsupported;
    • accuracy rows;
    • the warm start's exact cost;
    • a deadline that yields the warm start;
    • the widened-scan self-read.
  • Against the reference pricer (--exhaustive), the MILP equals the best enumerated plan on example1 (4.3770), ratio-of-quantiles (2.4071e-6) and rangeheavy-t5-q05-q09 (3.9901e-6).
• Complexity: see the table below. Variables are about Σ over contexts of options, plus slot links, plus terms. The m contexts are 35–60 % of the variables.
• Assumptions and limits: see "Design points" below.

Measurements (300 s budget, accuracy enforced, aws-fargate, raw data retained; workload-01 under ulimit -v 8388608)

The facade runs Stage 3 once per pane-width round inside the 300 s budget. Each row is the round whose plan was selected; "largest round" is the biggest model of the run.

In the table, m = the shareable-context share of the variables / rows / binaries. Warm and Final are costs per second. Old engine is today's engine on this branch, same flags.

Workload	Vars / rows / bins (m)	Build / solve ms	Status, gap, dual	Warm	Final	Old engine	Peak RSS	plan_cost / satisfies
example1	501 / 897 / 323 (165 / 278 / 104)	115 / 60	optimal, 0, 4.3770	7.0125	4.3770	4.3770 (optimal)	55 MB	equal / pass
rangeheavy-t5-q05-q09	6,982 / 13,719 / 3,908 (2,445 / 4,412 / 1,094)	183 / 1,680	optimal, 0	3.9901e-6	3.9901e-6	3.9901e-6 (deadline, 584k vars, 4.7 GB)	119 MB	equal / pass
rangeheavy-t8-t5	4,013 / 8,860 / 1,749 (2,528 / 5,653 / 922); largest round 114,225 / 253,592 / 50,601 (71,676 m vars), build 45 s, no dual bound	30 / 1,475	optimal, 0	6.4825e-6	6.4822e-6	6.4822e-6 (deadline, 851k vars, 12.6 GB)	302 MB	equal / pass
rangeheavy-t7-t4	4,490 / 8,685 / 2,627 (1,720 / 3,565 / 849)	29 / 2,508	optimal, 0	2.1493e-7	2.1493e-7	2.1493e-7 (optimal, 182k vars, 1.8 GB)	108 MB	equal / pass
ratio-of-quantiles	52 / 89 / 29 (0)	1 / 89	optimal, 0	2.4071e-6	2.4071e-6	2.4071e-6	36 MB	equal / pass
rollupheavy-t11	8,788 / 16,967 / 4,990 (2,805 / 5,012 / 1,334)	161 / 3,657	optimal, 0	2.5295e-6	1.4159e-6	1.4159e-6 (deadline, 6.9 GB)	150 MB	equal / pass
#1036 workload-01 (ulimit -v 8388608)	3,856 / 7,222 / 1,281 (1,811 / 3,523 / 539); largest round 39,907 / 70,955 / 13,147 (18,932 m vars)	26 / 4,806; largest round 200 s	deadline, 7.1 %, dual 1.316e-6; largest round 70.9 % (incumbent = warm start 3.5429e-6, dual 1.032e-6)	3.5429e-6	1.4171e-6	old engine: bad_alloc at 7.7 GB (#1036)	882 MB	≤ (equal) / pass

Owner model (/mydata/milp-decode-runs, same flags), for comparison:

Workload	Owner-model cost	Owner-model vars / rows
example1	4.3770	1,350 / 2,316
t5-q05-q09	3.9901e-6	20,239 / 39,087
t8-t5	6.4822e-6 (deadline)	13,127 / 27,537
t7-t4	2.1493e-7	27,742 / 52,131
ratio-of-quantiles	5.2662e-6	194 / 270
rollupheavy-t11	1.4159e-6	24,693 / 46,669
workload-01	2.0059e-6 (deadline)	584,478 / 1,034,663, 3.76 GB

Every selected plan's plan_cost equals the MILP objective, which is recomputed from the incumbent's binaries. The only term encoded inexactly on these runs is one scan in workload-01 read at more than 4 window kinds; it is charged at the widest union, so its cost is an upper bound.

Design points

Held

• Execution contexts work. "One candidate per slot per context", charging x[m] once, I charged once, "used only if read" and per-run latency/accuracy all hold exactly against plan_cost.
• The ratio-of-quantiles avg subtree is computed once: 2.41e-6, against 5.27e-6 under the owner model.
• Every memo example solves to optimality, and t8-t5 is now optimal.
• Models are 3–7× smaller than the owner model's. workload-01 fits in 0.9 GB, against 3.8 GB.

Did not hold or was underspecified. In each case this PR takes the conservative exact option:

1. "The memo group graph is acyclic" does not hold on this base. A widened scan's Project is a root of the scan's own group; fix(memo): an unwidened scan no longer becomes a widened reader of its own group #1138 removed only the unwidened case. Under one-candidate-per-group the Project and the scan it reads would need one slot (infeasible).

   • A group's roots are split by self-read depth (0 for the scan, 1 for a Project reading its own group).
   • A depth-1 root reading another depth-1 root in one context is dropped. This loses Project∘Project plans.
   • Test: a_widened_scan_reading_its_own_memo_group_plans_exactly.

2. "One candidate per memo group": an implementation has private nodes (is_implementation_root == false) of its own group. These are their own slots.

   • A factored root's input can list private nodes of several implementations. Counterexample on workload-01: input 0 of candidate 77 reads slots 241/242/244/245.
   • There, row 4 let the factored root justify a selected producer it does not read. That phantom read made MinimumQueryReaders(2) on candidate 43 hold falsely, and DagOfSelectedCandidates::validate rejected the decoded plan.
   • Added: a slot-link binary e per multi-slot input. It is one-hot on the reader, and row 4 counts e.

3. Row 7 double-counts when L[m] includes T[m, m']: q reaches m' directly too, so T[q, m'] counts it again. Implemented L[m] = m's own run plus its context's runs, and one T[q, m] per query-time m that q reaches. This is plan_cost's "each reached query-time node once".

4. Rows 3–5 admit cross-context self-justifying cycles: x[m] ≤ p[m', m] and x[m'] ≤ p[m, m'] for two shareable candidates of one group. Order ranks were added on the graph of shareable runs and runs of I, for cyclic components only.

5. "I runs once per second for every reader" was not plan_cost's semantics: computations charged once per consumer. Hook in asap_cost::execution_demand::computations: a NotMaterialized ingestion-time node is (1, 1.0). (The slowest-reader working memory was already in fix(cost): hold a materialized consumer's working-memory copy for its longest reader (#1095) #1134.)

6. Scan windows: the design's 2-D thresholds assume monotone usage. They were replaced by an exact encoding:

   • σ_W is exact from both sides per window kind;
   • δ_S is the indicator of "exactly the kinds in S";
   • every union is costed by asap-cost, with plan_cost's rule for an unbounded read (the trailing widest bounded window).

   This needs no monotonicity, for up to 4 kinds per scan per context (beyond that the widest union is charged: one scan on workload-01).

7. "One group = one result, so input statistics do not depend on the producer" does not hold for scans (output rows follow the window) or for candidates whose outputs differ only by child ids.

   • Producer-dependent usage stays on AND(run, p[ctx, c'], producer gives o).
   • A reader only sees scan outputs at windows that contain its own.
   • Outputs are compared without child ids. Before that, 61×61 contexts on a two-offset scan exceeded the cap.

8. Numerics: with aws-fargate, latency in ms is below HiGHS's absolute tolerances. On shared_sum, HiGHS returned an objective of 9.5e-14 for a plan costing 2.97e-13, and claimed optimal. Fixed in two ways: latency variables count in scaled units (largest possible latency ≈ 1,000 units), and the incumbent's objective is recomputed from its rounded binaries.

9. "One MILP solve": the facade runs Stage 3 once per pane-width round inside the 300 s budget, which gives 11–16 solves per run. The selected round is usually an early, small one.

What a decomposition step would help with

• 3a (connected components): nothing on these workloads. Each reads one metric.
• 3c (Lagrangian / column generation) is the one the numbers point at, for the bound:
  • The final workload-01 round (39.9k vars) ran 200 s. It never improved on the warm start (3.54e-6), with dual 1.03e-6 (gap 71 %).
  • t8-t5's largest round (114k vars, 50.6k binaries) took 45 s to build and had no dual bound at all.
  • The small rounds reach optimality because the LP bound is tight there. On the large rounds the weak LP bound, not model size or memory, stops branch-and-bound.
• 3b (private subtrees as Pareto options) would shrink the query contexts. The m contexts are 47–63 % of the variables (t8-t5 largest round: 71.7k of 114k). So the open point "share one input context per operator node across its materializations" would cut size more than 3b.
• The warm start is weak on workload-01 (3.54e-6 against 1.42e-6). A 3c-style per-query pricing pass would also give a better reference plan.

Exactness checks run

• Reference pricer (--exhaustive) equal on example1, ratio-of-quantiles and t5-q05-q09.
• t7-t4's enumeration produced no reference plans, so it is not compared.
• The equivalence seeds, Two windowed quantiles over one metric: shared scan costs more, and nested-window KLL panes are never shared #839/Instant quantile by (host, job) next to windowed quantiles: no scan sharing, no panes, no ingest-time variant, 100-group default #840/q24h and the A1 ⊆ satisfies corpus gate were not run.

Before this PR

refactor/stage-boundaries has only today's engine (#1133 removed the owner-model redesign).

After this PR

ASAP_STAGE3_SEARCH=milp runs the group MILP. It is exact against plan_cost and the reference pricer on the memo examples, and it plans #1036 workload-01 within 8 GB. The old engine stays the default.

Verification

• cargo fmt --all --check; cargo clippy --workspace --all-targets -- -D warnings.
• cargo test -p asap-plan-selection --lib: 312 passed.
• cargo test -p asap-cost.
• cargo test -p asap-devtools --test devtools -- milp_search: 3 passed.
• The measurement runs above, with stage_pipeline and ASAP_STAGE3_REPORT.

🤖 Generated with Claude Code