Repository navigation
Conversation
zzylol
force-pushed
the
refactor/1200-all-valid-states
branch
from
November 5, 2026 17:11
e5edb06 to
9eae875
Compare
zzylol
force-pushed
the
refactor/1200-all-valid-states
branch
from
November 5, 2026 20:21
9eae875 to
d32bdc6
Compare
…iming, including ingestion A summary build or MaintainPopulation that is not materialized and has no assignment entry now runs where its consumer runs: in the query run, or inside the ingestion run of the materialized state that reads it ((IngestionTime, None)). A filtered build, or one over something that only exists at query time, still runs at query time. Export validation checks every edge with readable_at: a query-time node reading an unmaterialized ingestion-time output is the new UnmaterializedIngestionOutputReadAtQueryTime; the old UnmaterializedSummaryBuildNotAtQueryTime is deleted. Stage 2 does not offer the state yet, so plans are unchanged. Export reports every edge where satisfies_physical fails as one variant, PhysicalASAPDAGValidationError::PhysicalRequirementNotSatisfied, replacing QueryDependencyInIngestion and the unmaterialized-ingestion- output case. The tests find the build and merge ids inline. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01K1M6br9PL4SDeVS1XpezKJ
zzylol
force-pushed
the
refactor/1200-ingestion-not-kept
branch
from
November 5, 2026 21:40
cc63aac to
eec9a3f
Compare
Contributor
Author
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #1216; part of #1200, tracked in #1207. Base branch:
refactor/1200-all-valid-states.Merge unit. This PR merges into main only together with #1207's steps 1a–6 (#1207, Order): the #1200 stack, #1203 2a and 2b, #1201 and #1204 PR 1.
Representation / rules / enumeration
(a) Representation. Unchanged from refactor(types, stages): every valid timing × lifetime state explicit; PhysicalRequirement, satisfies_physical, no named states #1216:
PhysicalPropertiesis the product{ timing: ExecutionTiming, lifetime: Lifetime }, built byPhysicalProperties::new, which rejects (IngestionTime, OneEvaluation). Physical properties live only inPhysicalPropertiesAssignment(refactor(types, stages): physical properties off the logical IR (OperatorNode has no timing or lifetime) #1223). A summary build (orMaintainPopulation) may now be (IngestionTime, None) in a timed plan. Before this PR that state was a valid value, but the timing pass never placed a build there and export rejected it.(b) Rules. No Stage 2 rule is added.
apply_physical_propertieschanges one placement rule for a node the assignment does not list:MaintainPopulationthat is not materialized runs at query time.Two exceptions keep it at query time: a filtered build (raw samples carry none of the columns its filter reads), and a build over something that only exists at query time (for example an evaluation). Stage 2 still does not offer (IngestionTime, None) for summary builds:
down_closedrequires every build below an ingestion-time build to be materialized there. Offering it is the next PR.(c) Enumeration. Unchanged.
satisfies_physicalwith each consumer's same-evaluationPhysicalRequirement(α^phys, refactor(types, stages): every valid timing × lifetime state explicit; PhysicalRequirement, satisfies_physical, no named states #1216) now also guards export: every edge where it fails is rejected as one error,PhysicalRequirementNotSatisfied(ingestion work reading a query-time output, or a query-time node reading an ingestion-time output that is not materialized), as the timing pass already rejects it.Why
User decision, 2026-11-05: implement (IngestionTime, None) for summary builds. #1216 listed it as the one valid state whose handling was a rule, not an implementation: the timing pass moved an unmaterialized build to query time, and export rejected the pair (
UnmaterializedSummaryBuildNotAtQueryTime).Example.
sum_over_time(m[5m])with aSummaryMergeover the 5-minute Sum build. The merge is materialized at ingestion time; the build has no entry.UpstreamNotAtIngestion).an_unmaterialized_build_runs_in_the_ingestion_run_of_the_state_reading_it).What
asap_types::ir::properties::timing). A summary build orMaintainPopulationwith no entry takes its consumer's timing (placement), unless it is filtered or something below it forces query time.forces_query_timefollows the same rule: such a node forces query time only when it is filtered or something below it does. The module doc,PhysicalPropertiesAssignment's doc andforces_query_time's doc state the new rule.PhysicalASAPDAG::validate).UnmaterializedSummaryBuildNotAtQueryTimeis deleted.PhysicalPropertiesin a map. The edge loop reads the producer's properties from it (no recomputation) and checkssatisfies_physical(&producer, &PhysicalRequirement::new(consumer timing, Evaluation::Same)). A failed check is one error,PhysicalRequirementNotSatisfied { producer, consumer }, whether ingestion work reads a query-time output or a query-time node reads an ingestion-time output that is not materialized. It replacesQueryDependencyInIngestion.execution_demand::computations).post-asap-ir.mdgets the new placement rule and the new edge check. The comment at Stage 2's generation site (enumerate_candidate_configurations) says why Stage 2 does not offer the state yet.Deleted
PhysicalASAPDAGValidationError::UnmaterializedSummaryBuildNotAtQueryTimeandPhysicalASAPDAGValidationError::QueryDependencyInIngestion(both nowPhysicalRequirementNotSatisfied).MaintainPopulationthat is not materialized runs at query time" (inplacementandforces_query_time).ingestion_time_node_over_query_time_upstream_is_rejected→an_unmaterialized_build_under_an_ingestion_time_state_runs_at_ingestionandingestion_time_state_over_a_filtered_build_is_rejected;ingestion_fixed_finalize_over_query_time_state_is_upstream_not_at_ingestion→ingestion_pinned_finalize_over_an_unmaterialized_state_does_not_satisfy_a_query_time_consumer;summary_merge_timing, "a merge at ingestion time over unassigned windows is an error" now asserts that the windows run at ingestion time.Formulation notation → code
PhysicalProperties::NOT_MATERIALIZED_AT_INGESTIONon aSummaryAgg/SummaryTumblingWindowChain/MaintainPopulationsatisfies_physicalwithPhysicalRequirement, called per edge inPhysicalASAPDAG::validatePhysicalASAPDAGValidationError::PhysicalRequirementNotSatisfiedQueryDependencyInIngestionConcepts not in the formulation
New abstractions not in #1207 or the doc: decisions (2026-11-05)
PhysicalASAPDAGValidationError::PhysicalRequirementNotSatisfied { producer, consumer }satisfies_physicalis falseQueryDependencyInIngestionand the unmaterialized-ingestion-output case (no separate variant). Pending approval of the namesummary_agg_and_merge_ids(test-support)PhysicalPropertiesAssignmentand the timing pass's placement of nodes with no entryHow
placementkeeps its entry handling. For a node with no entry it returns (QueryTime, None) forSummaryEstimate/EvaluatePopulation, and for a summary build orMaintainPopulationthat is filtered or has something below it that forces query time. Every other node gets (consumer timing, None).forces_query_timedrops its "unlisted summary build forces query time" arm and adds "unlisted filtered build forces query time". Export's edge loop reads the producer's properties from the node loop's map and callssatisfies_physical.Key code interfaces added / modified
crates/types/src/ir/physical_export.rsapply_physical_propertieskeeps its signature; only the placement rule above changes.Usage (the end-to-end test
tumbling_window_merge::an_unmaterialized_build_runs_in_the_ingestion_run_of_the_state_reading_it):Major algorithm changes
Timing-pass placement of an unmaterialized summary build.
UpstreamNotAtIngestion.satisfies_physicalon every edge, so no query reads an ingestion output that is not materialized.Plan and cost changes
None. Stage 2 never selects a build with no entry under an ingestion-time build (
down_closed), so the new placement is reached only through explicit assignments (the tests). All corpus and sketch-plan snapshots pass unchanged (UPDATE_SNAPSHOTS=1, thengit statusclean).Tests at each layer
an_unmaterialized_build_under_an_ingestion_time_state_runs_at_ingestion,ingestion_time_state_over_a_filtered_build_is_rejected(test helperagg_with(child, family, filter);aggcalls it with no filter),ingestion_pinned_finalize_over_an_unmaterialized_state_does_not_satisfy_a_query_time_consumer,summary_merge_structure::summary_merge_timingschema_rebuilding::an_unmaterialized_population_under_an_ingestion_time_consumer_runs_at_ingestion: a keyFilterover aMaintainPopulation, materialized at ingestion time; the population is (IngestionTime, None)export_validation_over_every_valid_state(the (IngestionTime, None) row is the build with no entry under a finalize materialized at ingestion time, and valid);invalid_physical_properties_are_rejected(an unmaterialized ingestion output read at query time:PhysicalRequirementNotSatisfied)every_valid_state.rs::physical_planner_over_every_valid_state: the plan has a merge over the build; for (IngestionTime, None) the query reads the merge's view and the merge's ingestion refresh computes the buildtumbling_window_merge::an_unmaterialized_build_runs_in_the_ingestion_run_of_the_state_reading_itcost_over_every_valid_stateandcomputations_over_every_valid_state(#1216)Before this PR
A summary build that is not materialized always ran at query time, and export rejected one at ingestion time.
After this PR
A summary build that is not materialized runs where its consumer runs, including inside a view's ingestion refresh. Export accepts it, and rejects any query-time read of an ingestion output that is not materialized.
Verification
eec9a3fa3(on refactor(types, stages): every valid timing × lifetime state explicit; PhysicalRequirement, satisfies_physical, no named states #1216198b9a9f1):cargo fmt --all -- --checkandcargo clippy --workspace --all-targets -- -D warningsclean;cargo test -p asap-types(145 + 103 + 8),-p asap-executorphysical_planner_over_every_valid_state,-p asap-integration-teststumbling_window_merge(14): 0 failed.cargo test -p asap-types(256),-p asap-cost(46),-p asap-physical-asap-optimization(28),-p asap-executor(297),-p asap-planner(89, 8 wall-clock ignored),-p asap-plan-selection --lib(146),-p asap-devtools(109),-p asap-integration-tests(365): 0 failed.🤖 Generated with Claude Code
https://claude.ai/code/session_01K1M6br9PL4SDeVS1XpezKJ