[zzylol]

Stacked on #1221; part of #1203, tracked in #1207 (PR 2a-5).

Merge unit. This PR merges into main only together with the rest of #1207's merge unit (steps 1a–6, #1207 "Order"): interim items below are deleted by a later PR in the unit.

Why

#1203 decided one operator per candidate, every intermediate result its own memo group. Scan widening (rule 2) still adds a consumer chain down to the widened source, and interns that source with its own equality (same_source_node, which ignores guarantees).

What

• Rule 2 adds each widened consumer as one operator over groups (add_candidate_over_groups): the narrowing projection, the readout, the merge and the build beneath it are groups.
• Its widened source chains (Scan, TimeRange, TimeShift) are found or added with Stage 0's equality by add_candidate_over_groups itself (origin WidenedScans). This replaces the rule's own interning (intern_inputs / same_source_node) and its "already added" check.
• Rule 9 keeps reading a widened tumbling-window consumer as it did before: a Stage 0 group's candidate whose one build beneath its groups is a SummaryTumblingWindowChain is rewritten whole at each shared layout or size, and the rewrite is stored as one operator over groups (BuildPlace::BeneathGroups).
  • Not chosen: letting rule 9 take the merge group that reads the chain group directly (as refactor(stage1): Pass 1 adds a memo group for every intermediate result (#1203 PR 2a-1) #1215 does for a SummaryAgg build group). Each tumbling-window width's merge is then its own group, so rule 9's per-group layout deduplication never applies: on the nine-query workload below that tried variant grew Stage 2 from 3340 to 4844 candidates and the devtools tests from 63 s to 1318 s.
• Corpus snapshots unchanged: no snapshot file changes, and Stage 2 candidate counts are identical on every snapshot input. The nine-query workload keeps its selected cost with fewer Stage 2 candidates.
• One plan changes (found in review): on sum by (job) / sum by (region) of sum_over_time(latency[5m]), the two queries' equal FinalizeExactAccumulator consumers of the shared build become one group. Cost 0.003267 → 0.003250 per second. Details in "Second review fixes".

New abstractions not in #1207 or the doc: decisions (2026-11-05)

Per #1207's interface policy, these names are not defined by #1207, #1203 or section 3 of memo-groups-properties-requirements.md. #1207's steps 1a–6 merge into main as one merge unit (#1207, Order), so an interim abstraction may stay in this PR only when a later PR of the unit deletes it; none reaches main. Rows marked Pending approval are not deleted by any PR in the unit and need the user's decision.

Abstraction	What it is for	Decision
SummaryStates::stage0_group_nodes (private field)	rule 9 puts Stage 0's group nodes back after a whole rewrite	Interim, deleted by 2b (rule 9)
groups rule 9 adds beneath a widened candidate get origin SharedBuild	no origin of its own	Interim, deleted by 2b (SharedBuild, rule 9)
without_exact_guarantees_below + reuse::without_exact_guarantees	two strips	Merged in #1222 into without_exact_guarantees (interim until 2b)
same_range_input, same_local_fields (pre-existing)	a third own-fields equality for rule 9	Deleted in #1222: equal TimeRange inputs are one group
intern_inputs, same_source_node	scan widening's own interning	Deleted in #1222

How

1. The widened root is wrapped in an identity Project when a group's node equals it (Memo::group_with_equal_node on the root without exact guarantees), as before for a root that is itself a source chain.

2. One output-contract check (same_output_contract); then add_candidate_over_groups finds or adds every node beneath it, the widened source chains included.

3. Rule 9's sets takes build_of's place (refactor(stage1): Pass 1 adds a memo group for every intermediate result (#1203 PR 2a-1) #1215):

   • Private and InputGroup as before;
   • BeneathGroups only for a Stage 0 group's candidate whose one build down to Stage 0's groups is a chain.

   derive puts back only Stage 0's group nodes after the rewrite (with_child_nodes, the node set collected once per run). It stores the rewrite with operator_over_groups (added groups' origin SharedBuild).

Representation / rules / enumeration

• (a) Representation. Unchanged types. Candidates of rules 2–8 are one operator over groups; rule 1 (tumbling windows), rule 9's private-build and input-group paths, and step 10 still add chains.
• (b) Rules. Rule 2 extends the memo through find_or_add_group. Rule 9 matches a chain beneath a Stage 0 group's groups as it matched the private chain.
• (c) Enumeration. Unchanged.

Key code interfaces added / modified

crates/logical-asap-optimization/src/pass2/memo/rules/shared_configuration_readers.rs (crate-internal)

/// Where a candidate reads its one summary build: now #1215's
/// `state_groups::BuildPlace` (Private, InputGroup, BeneathGroups); rule 9's
/// own copy of the enum is gone.
struct StateNeed {
    // build_group: Option<MemoGroupId>, // removed
    /// Where the candidate reads its build. // new
    place: BuildPlace,
    /* other fields unchanged */
}
struct SummaryStates {
    // known: HashMap<(usize, String), Vec<Rc<OperatorNode>>>, // removed in #1218 (Memo::add_candidate)
    /// Stage 0's group nodes, put back after a whole-candidate rewrite;
    /// collected once per rule run. // new
    stage0_group_nodes: HashSet<*const OperatorNode>,
    /* other fields unchanged */
}

crates/logical-asap-optimization/src/pass2/memo/rules/reuse.rs (crate-internal)

/// A node with every exact guarantee removed, its own included; what Stage 0's
/// equality compares with the groups' nodes. // new (was inside find_or_add_producer)
pub(in crate::pass2::memo) fn without_exact_guarantees(node: &Rc<OperatorNode>) -> Rc<OperatorNode>;

crates/logical-asap-optimization/src/pass2/memo/rules/scan_labels.rs (crate-internal)

// fn intern_inputs(..) -> Result<Rc<OperatorNode>, Pass2RuleError>; // removed (add_candidate_over_groups finds or adds the source chains)
// fn same_source_node(left: &OperatorNode, right: &OperatorNode) -> bool; // removed

Usage. On the KLL and scan-widening examples, rule 2 widens only the source groups (Scan, TimeRange), whose candidates were already one operator: asap_planner --run-until stage1_pass2 --format text prints byte-identical output before and after. A widened consumer of a build fires in state_groups::tests::disjoint_scan_labels_share_a_union_tumbling_window_state_with_typed_readers (avg by (job) (quantile_over_time(0.95, latency[5m])) and avg by (region) (quantile_over_time(0.99, latency[5m])), max_error 0.01, with tumbling windows); its memo, printed from the test:

# before (#1221): the widened reader is a chain down to the union-label scan
G2  [k]  WidenedScans  Project ← Project ← SummaryEstimate ← SummaryMerge ← SummaryTumblingWindowChain  inputs=[G28]
G28 [0]  WidenedScans  Scan (job and region labels)                                                     queries [0, 1]
# after: one operator per candidate; the chain and its merge are groups both queries reach
G2  [k]  WidenedScans  Project                     inputs=[G39]
G39 [0]  WidenedScans  Project                     inputs=[G38]  queries [0]
G38 [0]  WidenedScans  SummaryEstimate             inputs=[G37]  queries [0]
G37 [0]  WidenedScans  SummaryMerge                inputs=[G36]  queries [0, 1]
G36 [0]  WidenedScans  SummaryTumblingWindowChain  inputs=[G28]  queries [0, 1]

Major algorithm changes

None.

Before this PR

Rule 2 added consumer chains and interned widened source chains with same_source_node; rule 9 found a widened tumbling-window consumer's chain among its private nodes.

After this PR

Rule 2 adds one operator over groups with groups found by Stage 0's equality; rule 9 finds the same chain beneath the consumer's groups and rewrites the consumer as before.

Deleted

• scan_labels.rs: intern_inputs (add_candidate_over_groups finds or adds the source chains), same_source_node, the "already added" check and its deep ==, the direct push_group of widened source groups, the boundary_groups call.
• shared_configuration_readers.rs: StateNeed::build_group (now place).
• Second review: with_source_chain_groups, scan widening's second output check, chain_beneath_groups, workload_group_nodes, rule 9's own BuildPlace, state_groups::build_down_to_workload_groups.
• Test changes (shape only, same intent): disjoint_scan_labels_share_a_union_tumbling_window_state_with_typed_readers and widened_tumbling_window_inputs_preserve_series_identity_and_keys find the union-label chain among all groups (it is now a group of rule 2, read by its merge group, not a SharedBuild group of step 10), and check every Stage 0 group's widened consumer of it.

Plan cost, Stage 2 candidates and planning time

UPDATE_SNAPSHOTS=1 left every snapshot file unchanged. Stage 2 physical candidates and --run-until stage3 wall time per snapshot input, #1221 vs this PR, release builds on one host:

snapshot	Stage 2 candidates, before	after	seconds, before	after
o11y_bench/q02	248	248	0.12	0.12
o11y_bench/q05	498	498	0.24	0.24
o11y_bench/q06	251	251	0.11	0.11
o11y_bench/q11	7	7	0.02	0.02
o11y_bench/q12	2	2	0.01	0.01
o11y_bench/q18	737	737	0.28	0.28
o11y_bench/q19	495	495	0.26	0.26
o11y_bench/q20	4	4	0.01	0.02
o11y_bench/q21	368	368	0.11	0.12
o11y_bench/q22	247	247	0.10	0.10
o11y_bench/q23	247	247	0.10	0.10
o11y_bench/q25	495	495	0.26	0.26
o11y_bench/q26	247	247	0.10	0.11
o11y_bench/q27	4	4	0.01	0.01
sketch/shared-quantiles-1h-mvp	1801	1801	0.88	0.89
sketch/nested-windows-15m-1h-mvp	2281	2281	1.24	1.25
sketch/ratio-of-quantiles-1h-mvp	1802	1802	2.52	2.53
sketch/quantile-single-mvp	201	201	0.08	0.08
sketch/quantile-grouped-mvp	108	108	0.06	0.06
sketch/topk-rate-guarantees-ignored-mvp	87	87	0.05	0.05
sketch/fuzz-issue903-raw-retained	135	135	0.07	0.08
sketch/random-mixed-seed-862-w02-q2-q7-mvp	136	136	0.09	0.09
sketch/random-mixed-seed-862-w01-q3-q8-mvp	226	226	0.11	0.11
sketch/random-mixed-seed-862-long-w01-q3-mvp	72	72	0.09	0.09
total			6.93	7.00

Not a snapshot, but where rule 2 widens builds: the nine-query workload (tools/dag-viewer/workloads/random-mixed-seed-862/workload-01.json, no raw data retained, the devtools tests' setting): selected cost 716.285846 before and after; Stage 2 candidates 3340 → 2812 (equal widened builds are one group); --run-until stage3 13.1 s → 11.4 s.

Second review fixes (folded into 470bc69)

• This PR changes one plan; the body used to say plans were unchanged. Corpus snapshots are unchanged, but this two-query workload gets cheaper:
  • Workload: sum by (job) (sum_over_time(latency[5m])) and sum by (region) (sum_over_time(latency[5m])), every 60 s, max_error 0.01.
  • Before (a2f8f18): both queries' widened consumers read the shared SummaryAgg group G16, each through its own private Project ← FinalizeExactAccumulator, so the finalization was computed twice.
  • After: the two equal FinalizeExactAccumulator nodes are one group (G17), and both consumers are Project over G17.
  • Selected cost: 0.003267 → 0.003250 per second. Selected choice: G2:c122 G6:c122 … → G2:c61 G6:c61 … G17:c0.
  • This is the intended effect of hash-consing the consumers.
• One output check in scan widening. It checked the output contract before and after wrapping the root in a projection, and add_candidate_over_groups checked it again. Now scan widening checks it once (reuse::same_output_contract), on the node it adds.
• with_source_chain_groups deleted. add_candidate_over_groups finds or adds every node beneath the widened root, source chains included. Before, a widened root that was itself a source chain became a group's node by pointer. Now the aliasing check finds that group with Memo::group_with_equal_node, on the root without exact guarantees (reuse::without_exact_guarantees, shared with find_or_add_producer).
• One build finder. chain_beneath_groups is deleted: rule 9 takes build_of's BeneathGroups place (refactor(stage1): Pass 1 adds a memo group for every intermediate result (#1203 PR 2a-1) #1215), still only for a Stage 0 group's candidate whose build is a tumbling-window chain. Also deleted:
  • rule 9's own BuildPlace enum (it uses refactor(stage1): Pass 1 adds a memo group for every intermediate result (#1203 PR 2a-1) #1215's);
  • state_groups::build_down_to_workload_groups (it was refactor(stage1): Pass 1 adds a memo group for every intermediate result (#1203 PR 2a-1) #1215's build_down_to_stage0_groups);
  • workload_group_nodes.
• Stage 0 groups computed once. Rule 9 collects Stage 0's group nodes once per run (SummaryStates::stage0_group_nodes), not once per rewrite.
• Rollup gap: nothing lost, so no fix.
  • After this PR, a widened tumbling-window consumer's chain sits beneath groups (BeneathGroups), which rule 6 does not take. Before, the chain was a private build, which rule 6 did take.
  • Measured: a temporary build let rule 6 take those consumers too. On 33 workloads this found 181–970 such consumers per workload (g1–g5 and random-mixed-seed-862 01–04), but rollup_candidate_for found no rollup for any of them.
  • New test state_groups::tests::rollups_lose_nothing_from_widened_readers_whose_chain_is_beneath_groups checks this on sum by (job) / sum by (job, region) of sum_over_time(latency[5m]) with tumbling windows.

Notation (formulation → code)

Formulation (#1203, #1201, #1207)	Code name in this PR
memo group $G$	MemoGroup, id MemoGroupId
candidate $v$ added to $G$ (over groups)	add_candidate_over_groups
expression $e_v = \operatorname{op}v(G{v,1..n})$ of a widened consumer	its root (Project) over the narrowing projection's group, down to the widened source-chain groups
equality of a node with a group's	Memo::group_with_equal_node (Stage 0's equality)

Concepts not in the formulation

• Widened scans (origin WidenedScans, rule 2): a consumer whose scans read the union of the labels its metric's consumers need, then project back. In the formulation these are memo groups for the widened scans plus candidates of the consumers' groups.
• The build a candidate reads, and where (BuildPlace::BeneathGroups): rule 9 rewrites such a candidate whole. Rule 9 is deleted in 2b (Logical memo candidate as (one operator over groups, logical properties θ) with a requirement per input; no private nodes #1203 decision 3).
• Stage 0 groups (stage0_group_nodes), added groups, output contract, producer, accuracy budget parts: see refactor(stage1): Pass 1 adds a memo group for every intermediate result (#1203 PR 2a-1) #1215 / refactor(stage1): summary readouts and inputs as one operator over groups; Memo::add_candidate; one node equality in Pass 1 (#1203 2a-2) #1218.

Second review verification (470bc69)

• cargo test -p asap-types --lib: 146 passed. cargo test -p asap-logical-asap-optimization --lib: 199 passed (new: rollups_lose_nothing_from_widened_readers_whose_chain_is_beneath_groups).
• cargo test -p asap-integration-tests --test integration -- memo_enumeration reuse_rules operator_sharing sharing_issue_regressions finer_aggregate_reuse planner_layering_example having_shared_aggregate stage1_logical_asap_optimization joint_search_equivalence sketch_rollup rollup_controlled production_rollup_workloads: 154 passed, 23 ignored (pre-existing).
• cargo test -p asap-devtools --lib: 18 passed. --test devtools -- asap_planner:: asap_planner_acceptance:: memo_pricing::: 56 passed, 1 ignored.
• UPDATE_SNAPSHOTS=1 cargo test --release -p asap-devtools --test corpus_snapshots -- --include-ignored: 9 passed, no file changed.
• asap_planner --run-until stage1_pass2 --format text: byte-identical to the pre-review head 5535471 on the KLL and scan-widening examples and on 35 other workloads (the reviewer's 20 workloads including the two-query one above, six rollup / scan-widening / avg workloads, five earlier finer-aggregate, rollup, top-k and scan-KLL examples, random-mixed-seed-862 workloads 01–04). So the cost change above was already in this PR before the review fixes.
• cargo fmt --all; cargo clippy --workspace --all-targets -- -D warnings: clean.

Restack on #1218 (b8402c7)

Rebased on #1218's interface alignment: the doc's Memo::add_candidate replaces push_candidate_if_new (same equality; returns the candidate's CandidateForMemoGroupId), and #1218's CI fix of sketch_rollup::quantiles_rolled_up_keep_the_sketch_bound. Plans unchanged.

• cargo fmt --all -- --check, cargo clippy --workspace --all-targets -- -D warnings: clean.
• cargo test -p asap-logical-asap-optimization --lib: 199 passed.
• cargo test -p asap-integration-tests --test integration -- scan widen label memo_enumeration: 28 passed, 2 ignored (pre-existing).

Restack (2026-11-05)

On #1215's consumer naming (build_of → build_of) and #1218's one node equality in Pass 1.

• One without_exact_guarantees (was also without_exact_guarantees_below); rule 9 compares build inputs by pointer, so same_range_input / same_local_fields are deleted (equal TimeRange inputs are one group). cargo test -p asap-logical-asap-optimization --lib: 199 passed, integration (12 filters as above): 154 passed, 23 ignored; UPDATE_SNAPSHOTS=1 corpus snapshots: 9 passed, no file changed.
• After the restack: cargo fmt --all -- --check; cargo clippy --workspace --all-targets -- -D warnings: clean; cargo test -p asap-logical-asap-optimization --lib: 198 passed.

Viewer

Viewer tests not updated for the group shape (#1203 decision 6); viewer_contract passes.

Verification

• cargo test -p asap-logical-asap-optimization --lib: 198 passed.
• cargo test -p asap-physical-asap-optimization: 29 passed.
• cargo test -p asap-plan-selection --lib: 142 passed.
• cargo test -p asap-devtools --lib: 18 passed.
• cargo test -p asap-integration-tests --test integration -- memo_enumeration reuse_rules operator_sharing sharing_issue_regressions finer_aggregate_reuse planner_layering_example having_shared_aggregate stage1_logical_asap_optimization joint_search_equivalence sketch_rollup rollup_controlled production_rollup_workloads: 154 passed, 23 ignored (pre-existing).
• cargo test -p asap-devtools --test devtools -- asap_planner:: asap_planner_acceptance:: memo_pricing:: viewer_contract::: 58 passed, 1 ignored (63 s).
• UPDATE_SNAPSHOTS=1 cargo test --release -p asap-devtools --test corpus_snapshots -- --include-ignored: 9 passed, no file changed.
• cargo fmt --all; cargo clippy --workspace --all-targets -- -D warnings: clean.
• KLL and scan-widening --run-until stage1_pass2 --format text: byte-identical to refactor(stage1): rollups as one operator over groups (#1203 2a-4) #1221's head.

Left for 2a-6..k

Tumbling windows (1), rule 9's own outputs and step 10, then deleting direct_input_builds, BuildPlace::Private, same_range_input, SharedBuild and SummaryResult, and the interim added_groups helpers (rule 9's BeneathGroups path uses Memo::is_stage0_group until rule 9 is deleted in 2b).

[zzylol]

Closed: replaced by #1207's new order (interfaces first, then one PR per stage), starting with PR 0 #1226. The code on this branch is reused in PRs 1–8; the branch is kept.