Skip to content

feat(types): an unmaterialized summary build runs at its consumer's timing; one α^phys error at export - #1219

Closed
zzylol wants to merge 1 commit into
refactor/1200-all-valid-statesfrom
refactor/1200-ingestion-not-kept
Closed

zzylol wants to merge 1 commit into
refactor/1200-all-valid-statesfrom
refactor/1200-ingestion-not-kept

Conversation

@zzylol

@zzylol zzylol commented Nov 5, 2026 •

Copy link
Copy Markdown
Contributor

Stacked on #1216; part of #1200, tracked in #1207. Base branch: refactor/1200-all-valid-states.

Merge unit. This PR merges into main only together with #1207's steps 1a–6 (#1207, Order): the #1200 stack, #1203 2a and 2b, #1201 and #1204 PR 1.

Representation / rules / enumeration

  • (a) Representation. Unchanged from refactor(types, stages): every valid timing × lifetime state explicit; PhysicalRequirement, satisfies_physical, no named states #1216: PhysicalProperties is the product { timing: ExecutionTiming, lifetime: Lifetime }, built by PhysicalProperties::new, which rejects (IngestionTime, OneEvaluation). Physical properties live only in PhysicalPropertiesAssignment (refactor(types, stages): physical properties off the logical IR (OperatorNode has no timing or lifetime) #1223). A summary build (or MaintainPopulation) may now be (IngestionTime, None) in a timed plan. Before this PR that state was a valid value, but the timing pass never placed a build there and export rejected it.

  • (b) Rules. No Stage 2 rule is added. apply_physical_properties changes one placement rule for a node the assignment does not list:

    • Before: a summary build or MaintainPopulation that is not materialized runs at query time.
    • After: it runs where its consumer runs, like every other node: in the query run, or inside the ingestion run of the materialized state that reads it.

    Two exceptions keep it at query time: a filtered build (raw samples carry none of the columns its filter reads), and a build over something that only exists at query time (for example an evaluation). Stage 2 still does not offer (IngestionTime, None) for summary builds: down_closed requires every build below an ingestion-time build to be materialized there. Offering it is the next PR.

  • (c) Enumeration. Unchanged. satisfies_physical with each consumer's same-evaluation PhysicalRequirement (α^phys, refactor(types, stages): every valid timing × lifetime state explicit; PhysicalRequirement, satisfies_physical, no named states #1216) now also guards export: every edge where it fails is rejected as one error, PhysicalRequirementNotSatisfied (ingestion work reading a query-time output, or a query-time node reading an ingestion-time output that is not materialized), as the timing pass already rejects it.

Why

User decision, 2026-11-05: implement (IngestionTime, None) for summary builds. #1216 listed it as the one valid state whose handling was a rule, not an implementation: the timing pass moved an unmaterialized build to query time, and export rejected the pair (UnmaterializedSummaryBuildNotAtQueryTime).

Example. sum_over_time(m[5m]) with a SummaryMerge over the 5-minute Sum build. The merge is materialized at ingestion time; the build has no entry.

  • Before: the timing pass failed. It placed the unmaterialized build at query time, below an ingestion-time merge (UpstreamNotAtIngestion).
  • After: the build is (IngestionTime, None) and runs inside the merge's ingestion run. Only the merge is stored, and the query reads the merge's view. The answers equal the whole build at query time (an_unmaterialized_build_runs_in_the_ingestion_run_of_the_state_reading_it).

What

  • Timing pass (asap_types::ir::properties::timing). A summary build or MaintainPopulation with no entry takes its consumer's timing (placement), unless it is filtered or something below it forces query time. forces_query_time follows the same rule: such a node forces query time only when it is filtered or something below it does. The module doc, PhysicalPropertiesAssignment's doc and forces_query_time's doc state the new rule.
  • Export validation (PhysicalASAPDAG::validate).
    • UnmaterializedSummaryBuildNotAtQueryTime is deleted.
    • The node loop keeps each node's PhysicalProperties in a map. The edge loop reads the producer's properties from it (no recomputation) and checks satisfies_physical(&producer, &PhysicalRequirement::new(consumer timing, Evaluation::Same)). A failed check is one error, PhysicalRequirementNotSatisfied { producer, consumer }, whether ingestion work reads a query-time output or a query-time node reads an ingestion-time output that is not materialized. It replaces QueryDependencyInIngestion.
  • Executor. No code change: a view's ingestion refresh already compiles and runs everything below the view that is not a view. The new tests show it computes the unmaterialized build there.
  • Cost. No change. refactor(types, stages): every valid timing × lifetime state explicit; PhysicalRequirement, satisfies_physical, no named states #1216 already prices (IngestionTime, None): working memory, computed once per refresh window of the views reading it (execution_demand::computations).
  • Tests find the build and merge ids of an exported DAG inline; there is no shared test helper.
  • Docs. post-asap-ir.md gets the new placement rule and the new edge check. The comment at Stage 2's generation site (enumerate_candidate_configurations) says why Stage 2 does not offer the state yet.

Deleted

  • PhysicalASAPDAGValidationError::UnmaterializedSummaryBuildNotAtQueryTime and PhysicalASAPDAGValidationError::QueryDependencyInIngestion (both now PhysicalRequirementNotSatisfied).
  • The placement rule "a summary build or MaintainPopulation that is not materialized runs at query time" (in placement and forces_query_time).
  • Tests of the old rule, replaced by tests of the new one:
    • ingestion_time_node_over_query_time_upstream_is_rejected → an_unmaterialized_build_under_an_ingestion_time_state_runs_at_ingestion and ingestion_time_state_over_a_filtered_build_is_rejected;
    • ingestion_fixed_finalize_over_query_time_state_is_upstream_not_at_ingestion → ingestion_pinned_finalize_over_an_unmaterialized_state_does_not_satisfy_a_query_time_consumer;
    • in summary_merge_timing, "a merge at ingestion time over unassigned windows is an error" now asserts that the windows run at ingestion time.

Formulation notation → code

Notation Code Status
$(\tau, \ell) = (\mathrm{IngestionTime}, \mathrm{None})$ for a summary build PhysicalProperties::NOT_MATERIALIZED_AT_INGESTION on a SummaryAgg / SummaryTumblingWindowChain / MaintainPopulation now produced by the timing pass and accepted by export
$\alpha^{\mathrm{phys}}_{u,c,i}$ (timing and lifetime part) at export satisfies_physical with PhysicalRequirement, called per edge in PhysicalASAPDAG::validate was a timing-only comparison
$\alpha^{\mathrm{phys}}_{u,c,i} = 0$ on an exported edge PhysicalASAPDAGValidationError::PhysicalRequirementNotSatisfied new; replaces QueryDependencyInIngestion

Concepts not in the formulation

  • Placement of a node with no entry. The formulation gives a candidate with $\ell$ = None no schedule: it runs inside its consumer's execution (a query evaluation $\kappa_q$, an ingestion pipeline $I_r$ or a materialized result's computation $\kappa_m$). The timing pass encodes that as "take the consumer's timing", with two code-level exceptions (filtered builds; builds over a query-time-only input) that come from what an ingestion run can read.

New abstractions not in #1207 or the doc: decisions (2026-11-05)

Abstraction What it is for Decision
PhysicalASAPDAGValidationError::PhysicalRequirementNotSatisfied { producer, consumer } export's one error for an edge where satisfies_physical is false Merged in this PR: replaces QueryDependencyInIngestion and the unmaterialized-ingestion-output case (no separate variant). Pending approval of the name
summary_agg_and_merge_ids (test-support) find the build and merge of an exported DAG Inlined in this PR into its two tests
PhysicalPropertiesAssignment and the timing pass's placement of nodes with no entry where an ℓ = None candidate runs Interim, deleted by #1201 + #1204 PR 1 (contexts κ choose where an ℓ = None candidate runs)

How

placement keeps its entry handling. For a node with no entry it returns (QueryTime, None) for SummaryEstimate / EvaluatePopulation, and for a summary build or MaintainPopulation that is filtered or has something below it that forces query time. Every other node gets (consumer timing, None). forces_query_time drops its "unlisted summary build forces query time" arm and adds "unlisted filtered build forces query time". Export's edge loop reads the producer's properties from the node loop's map and calls satisfies_physical.

Key code interfaces added / modified

crates/types/src/ir/physical_export.rs

/// Why an exported physical DAG is invalid; stays. Only the changed variants are listed
/// here; every other variant is unchanged.
pub enum PhysicalASAPDAGValidationError {
    /// A summary build not materialized and not at query time.
    UnmaterializedSummaryBuildNotAtQueryTime { node: PhysicalASAPNodeId }, // removed: the state is valid
    /// Ingestion work reads a query-time output.
    QueryDependencyInIngestion { producer: PhysicalASAPNodeId, consumer: PhysicalASAPNodeId }, // removed: PhysicalRequirementNotSatisfied
    /// An edge where α^phys fails (satisfies_physical is false): ingestion work reads a
    /// query-time output, or a query-time node reads an ingestion-time output that is not
    /// materialized. A planner bug.
    PhysicalRequirementNotSatisfied { producer: PhysicalASAPNodeId, consumer: PhysicalASAPNodeId }, // new
}

apply_physical_properties keeps its signature; only the placement rule above changes.

Usage (the end-to-end test tumbling_window_merge::an_unmaterialized_build_runs_in_the_ingestion_run_of_the_state_reading_it):

// The query, planned as one 5-minute exact Sum build.
let single = single_build("sum_over_time(m[5m])", AccuracyTarget::exact());
// The Sum build; it gets no assignment entry.
let build = summary_agg(&single);
// A merge over the build, which the plan materializes at ingestion time.
let merged = OperatorNode::new_shared(Operator::ASAP(ASAPOp::SummaryMerge {
    children: vec![Rc::clone(&build)], newest_tumbling_windows: None, group_by: None,
})).unwrap();
// The query with the build replaced by the merge.
let root = replace(&single, &build, &merged);
// Only the merge is listed: materialized across evaluations, at ingestion time.
let mut assignment = PhysicalPropertiesAssignment::default();
assignment.set(&merged, PhysicalProperties::MATERIALIZED_IN_MEMORY_ACROSS_EVALUATIONS_AT_INGESTION);
// Timing pass + export: the build comes out (IngestionTime, None), and validate() accepts the DAG.
let mut exported = export(&root, &assignment);
// The merge's ingestion refresh over [0, 5 min) computes the Sum build from raw samples and
// stores only the merge; the query binds the merge's view and answers as the whole build at
// query time.

Major algorithm changes

Timing-pass placement of an unmaterialized summary build.

  • Problem: a merge materialized at ingestion time over a build with no entry failed with UpstreamNotAtIngestion.
  • Before: a build with no entry ran at query time.
  • After: it runs at its consumer's timing, unless filtered or over a query-time-only input.
  • Correctness: the timing pass's edge checks are unchanged, and export now also checks satisfies_physical on every edge, so no query reads an ingestion output that is not materialized.
  • Complexity: unchanged (one map lookup per edge in export).
  • Limits: a filtered build still runs at query time. Stage 2 does not offer the state yet (next PR).

Plan and cost changes

None. Stage 2 never selects a build with no entry under an ingestion-time build (down_closed), so the new placement is reached only through explicit assignments (the tests). All corpus and sketch-plan snapshots pass unchanged (UPDATE_SNAPSHOTS=1, then git status clean).

Tests at each layer

Layer Test
Timing pass an_unmaterialized_build_under_an_ingestion_time_state_runs_at_ingestion, ingestion_time_state_over_a_filtered_build_is_rejected (test helper agg_with(child, family, filter); agg calls it with no filter), ingestion_pinned_finalize_over_an_unmaterialized_state_does_not_satisfy_a_query_time_consumer, summary_merge_structure::summary_merge_timing
Timing pass, population schema_rebuilding::an_unmaterialized_population_under_an_ingestion_time_consumer_runs_at_ingestion: a key Filter over a MaintainPopulation, materialized at ingestion time; the population is (IngestionTime, None)
Export validation export_validation_over_every_valid_state (the (IngestionTime, None) row is the build with no entry under a finalize materialized at ingestion time, and valid); invalid_physical_properties_are_rejected (an unmaterialized ingestion output read at query time: PhysicalRequirementNotSatisfied)
Executor physical planner every_valid_state.rs::physical_planner_over_every_valid_state: the plan has a merge over the build; for (IngestionTime, None) the query reads the merge's view and the merge's ingestion refresh computes the build
End to end tumbling_window_merge::an_unmaterialized_build_runs_in_the_ingestion_run_of_the_state_reading_it
Cost unchanged; cost_over_every_valid_state and computations_over_every_valid_state (#1216)

Before this PR

A summary build that is not materialized always ran at query time, and export rejected one at ingestion time.

After this PR

A summary build that is not materialized runs where its consumer runs, including inside a view's ingestion refresh. Export accepts it, and rejects any query-time read of an ingestion output that is not materialized.

Verification

  • Head eec9a3fa3 (on refactor(types, stages): every valid timing × lifetime state explicit; PhysicalRequirement, satisfies_physical, no named states #1216 198b9a9f1): cargo fmt --all -- --check and cargo clippy --workspace --all-targets -- -D warnings clean; cargo test -p asap-types (145 + 103 + 8), -p asap-executor physical_planner_over_every_valid_state, -p asap-integration-tests tumbling_window_merge (14): 0 failed.
  • Full run one rebase earlier (same code before the consumer renames): cargo test -p asap-types (256), -p asap-cost (46), -p asap-physical-asap-optimization (28), -p asap-executor (297), -p asap-planner (89, 8 wall-clock ignored), -p asap-plan-selection --lib (146), -p asap-devtools (109), -p asap-integration-tests (365): 0 failed.
  • Plans: devtools snapshot tests pass unchanged; Stage 2 offers no new state in this PR.

🤖 Generated with Claude Code

https://claude.ai/code/session_01K1M6br9PL4SDeVS1XpezKJ

…iming, including ingestion

A summary build or MaintainPopulation that is not materialized and has no
assignment entry now runs where its consumer runs: in the query run, or
inside the ingestion run of the materialized state that reads it
((IngestionTime, None)). A filtered build, or one over something that only
exists at query time, still runs at query time.

Export validation checks every edge with readable_at: a query-time node
reading an unmaterialized ingestion-time output is the new
UnmaterializedIngestionOutputReadAtQueryTime; the old
UnmaterializedSummaryBuildNotAtQueryTime is deleted.

Stage 2 does not offer the state yet, so plans are unchanged.

Export reports every edge where satisfies_physical fails as one
variant, PhysicalASAPDAGValidationError::PhysicalRequirementNotSatisfied,
replacing QueryDependencyInIngestion and the unmaterialized-ingestion-
output case. The tests find the build and merge ids inline.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01K1M6br9PL4SDeVS1XpezKJ
@zzylol
zzylol force-pushed the refactor/1200-ingestion-not-kept branch from cc63aac to eec9a3f Compare November 5, 2026 21:40
@zzylol zzylol changed the title feat(types): a summary build not kept runs at its consumer's timing, including ingestion feat(types): an unmaterialized summary build runs at its consumer's timing; one α^phys error at export Nov 5, 2026
@zzylol

zzylol commented Nov 6, 2026

Copy link
Copy Markdown
Contributor Author

Closed: replaced by #1207's new order (interfaces first, then one PR per stage), starting with PR 0 #1226. The code on this branch is reused in PRs 1–8; the branch is kept.

@zzylol zzylol closed this Nov 6, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request ir Pre-ASAP or post-ASAP intermediate representation optimizer Workload optimization and plan selection

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant