Skip to content

refactor(types, stages): replace Materialization with PhysicalProperties (timing, lifetime) - #1214

Closed
zzylol wants to merge 9 commits into
mainfrom
refactor/1200-timing-lifetime
Closed

zzylol wants to merge 9 commits into
mainfrom
refactor/1200-timing-lifetime

Conversation

@zzylol

@zzylol zzylol commented Nov 4, 2026 •

Copy link
Copy Markdown
Contributor

Part of #1200 (PR 1 of 2), tracked in #1207 (step 1a).

Merge unit. This PR merges into main only together with #1207's steps 1a–6 (#1207, Order): the #1200 stack, #1203 2a and 2b, #1201 and #1204 PR 1.

Representation / rules / enumeration

  • (a) Representation. Each physical candidate stores PhysicalProperties = (timing, lifetime): timing is IngestionTime or QueryTime, lifetime is None, OneEvaluation(Medium) or AcrossEvaluations(Medium). Lifetime None means the output is not materialized (discarded after its consumer runs). "Materialized" (ℓ ≠ None, Physical properties: replace Materialization with timing × lifetime (None / OneEvaluation / AcrossEvaluations) and a schedule for shareable results, with r^phys and α^phys #1200) is derived (lifetime.is_materialized()), not stored. (IngestionTime, OneEvaluation) cannot be represented: PhysicalProperties::new returns InvalidPhysicalProperties::OneEvaluationAtIngestion. The schedule σ (periodic refresh or tumbling-window grid phase of a materialized result) is not part of this PR; it comes in PR 2.

  • (b) Rules. No new rules are added; only the property type changes. Stage 2 (enumerate_candidate_configurations) still offers, for each operator node of each memo group's candidates:

    • every node: (QueryTime, None);
    • a summary build (SummaryAgg, SummaryTumblingWindowChain): also (QueryTime, OneEvaluation(Memory)); (IngestionTime, AcrossEvaluations(Memory)) when some reading query allows ingestion; and, for a tumbling-window chain only, (QueryTime, AcrossEvaluations(Memory)) when the deployment keeps query-time results and some consuming query builds one new tumbling window per evaluation;
    • any other node, unless it must run at query time (SummaryEstimate, EvaluatePopulation, MaintainPopulation): also (IngestionTime, None); and (QueryTime, OneEvaluation(Memory)) when two or more queries may reach it.

    Then, as before, it prunes PerEntityAggregateNotExecutable (all alternatives) and RawDataNotRetained (query-time Scan), and Stage 2's later pruning (TooFewQueryReaders, ReachesNoQueryOutput, InputHasNoBuildableProducer) is unchanged. The (old enum value) → (timing, lifetime) mapping is one to one: NotMaterialized at timing τ → (τ, None), MatAtQuery → (QueryTime, OneEvaluation), MatAtQueryKeptAcrossEvaluations → (QueryTime, AcrossEvaluations), MatAtIngestion → (IngestionTime, AcrossEvaluations).

  • (c) Enumeration. Stage 3's MILP picks one physical candidate per used memo group per consumer context, as before. For each consumer input, Stage 2 lists only the producers whose properties a consumer at that timing may read: PhysicalProperties::readable_at(consumer_timing) is true when the producer has the same timing, or when a query-time consumer reads an ingestion-time AcrossEvaluations result. This replaces compatible_timing with the same truth table. Stage 3's run_kind takes PhysicalProperties: (QueryTime, None) → PerContext, (IngestionTime, None) → Ingestion, materialized (OneEvaluation or AcrossEvaluations) → Shareable (renamed Materialized in refactor(types, stages): every valid timing × lifetime state explicit; PhysicalRequirement, satisfies_physical, no named states #1216).

Why

#1207's formulation gives each physical candidate the properties $p^{\mathrm{phys}}_v = (\tau_v, \ell_v)$. Before this PR the code folded both into one enum, Materialization. One variant, MatAtQueryKeptAcrossEvaluations, mixed timing and lifetime, and the timing was also stored a second time next to the enum (PhysicalCandidate::timing, output_state.timing). The two could disagree, and the export had to check them against each other (MaterializationTimingMismatch). Later steps (σ in #1200 PR 2, explicit edges in #1201, the MILP in #1204) read timing and lifetime as separate values.

What

  • New asap_types::physical::{Lifetime, PhysicalProperties, InvalidPhysicalProperties} in crates/types/src/physical/properties.rs; Medium moves there. crates/types/src/physical/materialization.rs is deleted.
  • Stage 2 candidates (PhysicalCandidate, PhysicalCandidateKey) hold properties: PhysicalProperties in place of timing + materialization.
  • Plan nodes (OperatorNode, PhysicalASAPDAGNode) hold lifetime: Lifetime; their timing stays where it was (timing / output_state.timing).
  • PhysicalPropertiesAssignment (was MaterializationAssignment) maps each assigned node to its PhysicalProperties; a node it does not list is not materialized.
  • Each value Stage 2 builds is a named associated const on PhysicalProperties (NOT_MATERIALIZED_AT_QUERY_TIME, NOT_MATERIALIZED_AT_INGESTION, MATERIALIZED_IN_MEMORY_FOR_ONE_EVALUATION, MATERIALIZED_IN_MEMORY_ACROSS_EVALUATIONS_AT_INGESTION, MATERIALIZED_IN_MEMORY_ACROSS_EVALUATIONS_AT_QUERY_TIME), used by Stage 2 and the tests.
  • An invalid timed node is reported (ExecutionDataStateError::InvalidPhysicalProperties in the timing pass, AnalyticalCostError::InvalidPhysicalProperties { node, reason } in execution_demand::computations, naming the node), never read as untimed or skipped.
  • New helpers Lifetime::is_materialized (ℓ ≠ None) and Lifetime::is_across_evaluations (replaced in refactor(types, stages): every valid timing × lifetime state explicit; PhysicalRequirement, satisfies_physical, no named states #1216 by PhysicalProperties::is_materialized, as the doc defines it). PhysicalWorkloadPlan::materializes_any (test support) now reads dag.nodes.
  • Plan nodes give their pair through one accessor each (OperatorNode::physical_properties, PhysicalASAPDAGNode::physical_properties); code that rebuilt the pair by hand, or re-implemented readable_at, uses them.
  • The plan file's node field materialization becomes lifetime ("None", {"OneEvaluation": "Memory"}, {"AcrossEvaluations": "Memory"}). INSTALLABLE_PLAN_WIRE_VERSION goes from 11 to 12. The executor, the DAG viewer (app.js, stages.js, memo.js), fixtures and snapshots all change in this PR. No consumer of the old format is kept.
  • Cost (asap-cost), Stage 3 (groups.rs, MILP cost terms, decode), materialized views, the executor's physical planner, devtools and docs match on (timing, lifetime).

Deleted: Materialization and its variants NotMaterialized, MatAtIngestion, MatAtQuery, MatAtQueryKeptAcrossEvaluations; Materialization::timing(); Materialization::is_materialized_view and test_support::is_materialized_view; compatible_timing; OperatorNode::materialization and with_materialization; PhysicalCandidate::{timing, materialization}; PhysicalCandidateKey::{timing, materialization}; PhysicalASAPDAGNode::materialization; PhysicalWorkloadPlan::materialization (a copy of the dag nodes' lifetimes; consumers read dag.nodes), and its filling code in Stage 2; PhysicalASAPDAGValidationError::MaterializationTimingMismatch (split into InvalidPhysicalProperties { node, reason } and UnmaterializedSummaryBuildNotAtQueryTime { node }); MaterializationAssignment::with_default, its unassigned field and the branch reading it (only tests used it; they now call the test-support builder materialized_at_ingestion_where_allowed); explicit not-materialized assignments (set(node, None); #1216 lets set take any valid PhysicalProperties) and the Option<Option<_>> they needed in get (now private) and in Stage 2's choice callback (now an option index); timing.rs's kept_at_ingestion (re-implemented readable_at); materialized_view.rs's is_view (now Lifetime::is_across_evaluations); Stage 2's kept_for_one_evaluation(), kept_across_evaluations_at_ingestion(), kept_across_evaluations_at_query_time(), physical_properties(t, l) and the tests' copies (now associated consts); optimize_physical_asap_assigned (a duplicate of test_support::optimize_physical_asap_chosen); the unused Serialize derive on PhysicalProperties; the tests of removed behavior (query_time_state_moves_its_consuming_summary_to_query_time, explicit_not_kept_keeps_a_fixed_timing); the integration tests' copy of the old enum, MaterializationOption { MatAtIngestion, MatAtQuery, NotMaterialized } (now Materializing { MaterializedAtIngestion, MaterializedAtQueryTime, NotMaterialized }, a test-only projection without lifetime length or medium).

Renamed (names that held the old enum's values now say what they hold; code, tests and docs):

Before After
MaterializationAssignment PhysicalPropertiesAssignment
apply_materialization_timings, apply_materialization_timings_to_roots apply_physical_properties, apply_physical_properties_to_roots
MaterializationCandidate (Stage 2, a node whose output may be materialized), materialization_candidates() MaterializedOutputCandidate, find_materialized_output_candidates()
ChooseMaterialization, materialization_assignments ChoosePhysicalProperties, physical_properties_assignments
ExecutionDataStateError::MaterializationConflictsWithFixedTiming PhysicalPropertiesConflictWithFixedTiming
is_materialization_point is_materializable_state_output
ReaderConstraint::HighestSharedQueryMaterialization HighestSharedOneEvaluation
materialize_selected_candidates realize_selected_candidates
summary_mode_allowed(mode) summary_properties_allowed(candidate_properties)
integration tests' MaterializationOption { MatAtIngestion, MatAtQuery, NotMaterialized }, materialization() Materializing { MaterializedAtIngestion, MaterializedAtQueryTime, NotMaterialized }, materializing()
tests/materialization_e2e.rs; tests and helpers named *_materialization*, *not_materialized*, *unmaterialized* tests/physical_properties_e2e.rs; *_physical_properties* / *_lifetime*, *not_materialized*

Docs (design proposals including planner-layering.md Algorithm 3 and its legend, the Example 1–4 acceptance docs, joint-planning-search.md, window-materialization-choices.md, maintained-populations.md, develop docs) use (timing, lifetime) in place of NotMaterialized / MatAtQuery / MatAtIngestion / MatAtQueryKeptAcrossEvaluations, and "physical properties assignment" / "materialized-output candidate" for the renamed types. Code says "materialized" for ℓ ≠ None, as #1200 does; "materialized view" (a node materialized across evaluations) and the SQL frontend's unrelated materialize are unchanged.

Considered, not done: storing one PhysicalProperties field on plan nodes instead of timing + lifetime (a larger wire-shape change); the accessors above give the pair without it.

Formulation notation → code

Formulation (#1200) Code
$\tau_v$ ExecutionTiming (PhysicalProperties::timing)
$\ell_v$ ∈ {None, OneEvaluation, AcrossEvaluations} Lifetime::{None, OneEvaluation(Medium), AcrossEvaluations(Medium)}
$p^{\mathrm{phys}}_v = (\tau_v, \ell_v)$ PhysicalProperties (struct of the two, checked by new)
materialized ($\ell_v \ne$ None) Lifetime::is_materialized, MaterializedOutputCandidate, materialized_tumbling_window_chains, PhysicalProperties::MATERIALIZED_IN_MEMORY_*, NOT_MATERIALIZED_AT_*
invalid combination 1 ((IngestionTime, OneEvaluation)) InvalidPhysicalProperties::OneEvaluationAtIngestion
$\alpha^{\mathrm{phys}}$ (timing and lifetime part) PhysicalProperties::readable_at

Naming (user decision, 2026-11-05). An earlier version of this PR said "kept" for ℓ ≠ None. The formulation's word is "materialized", so every name this PR introduced says "materialized" (commit "say materialized, not kept"). Prose that describes a lifetime ("kept across evaluations", "kept until every consumer of one evaluation finishes") follows #1200's own definitions of the lifetimes.

Concepts not in the formulation: Medium (Memory / Disk), carried inside Lifetime as decided; the run kinds PerContext / Ingestion (named in #1200's RunKind formula).

New abstractions not in #1207 or the doc: decisions (2026-11-05)

Per #1207's interface policy, these names are not defined by #1207, #1200 or section 3 of memo-groups-properties-requirements.md. The user decided each one on 2026-11-05. #1207's steps 1a–6 merge into main as one merge unit (#1207, Order). So an interim abstraction may stay in this PR only when a later PR of the unit deletes it; none reaches main.

Abstraction What it is for Decision
PhysicalProperties::{NOT_MATERIALIZED_AT_QUERY_TIME, NOT_MATERIALIZED_AT_INGESTION, MATERIALIZED_IN_MEMORY_FOR_ONE_EVALUATION, MATERIALIZED_IN_MEMORY_ACROSS_EVALUATIONS_AT_INGESTION, MATERIALIZED_IN_MEMORY_ACROSS_EVALUATIONS_AT_QUERY_TIME} named valid states that Stage 2 offers Deleted in #1216: every use becomes the defined PhysicalProperties::new(τ, ℓ); no not_materialized_at either
PhysicalPropertiesAssignment (was MaterializationAssignment), set, copy_assignment (new), apply_physical_properties[_to_roots] (renamed) per-node map of $p^{\mathrm{phys}}$ over the OperatorNode DAG the realizer and the export read Interim, deleted in the unit by #1201 + #1204 PR 1: the realizer then reads each node's properties from its PhysicalCandidate / DagOfSelectedCandidates (doc, "Realizer and export")
Stage 2's whole-assignment path, renamed: MaterializedOutputCandidate, find_materialized_output_candidates, ChoosePhysicalProperties, physical_properties_assignments, realize_selected_candidates, summary_properties_allowed, is_materializable_state_output, materialized_tumbling_window_chains, ReaderConstraint::HighestSharedOneEvaluation renames forced by deleting Materialization Interim, deleted with the side map by #1201 + #1204 PR 1 (Stage 2 enumerates per-candidate properties over explicit edges)
MaterializedOutputCandidate::options() -> Vec<Option<PhysicalProperties>> None = no entry: the timing pass places a not-materialized node where its consumer runs Interim, deleted by #1201 (each candidate has a full $p^{\mathrm{phys}}$, as the doc says)
PhysicalASAPDAGNode::physical_properties() the export node's (timing, lifetime) as a checked pair Deleted in #1216: inlined at its callers
AnalyticalCostError::InvalidPhysicalProperties { node, reason }, PhysicalASAPDAGValidationError::InvalidPhysicalProperties { node, reason } report an invalid pair on a plan node Merged into fewer error names in #1216: one shared variant, the export's PhysicalASAPDAGValidationError::InvalidPhysicalProperties { node, reason } carrying the defined InvalidPhysicalProperties; cost wraps it as AnalyticalCostError::InvalidPhysicalDag(_)
PhysicalASAPDAGValidationError::UnmaterializedSummaryBuildNotAtQueryTime rejects a summary build at (IngestionTime, None) Replaced in #1219 by the export's edge check, folded into the single $\alpha^{\mathrm{phys}}$-failure error (satisfies_physical false)
materialized_at_ingestion_where_allowed (test-support) test fixture, replaces main's MaterializationAssignment::with_default and the private all_at_ingestion Interim, deleted with the side map by #1201 + #1204 PR 1. Not inlined: it walks the timing pass's private rules (forces_query_time, is_filtered_build), and 8 test sites plus validate_at_ingestion / planned_data_state use it

Replaced later in this stack by the defined interface (#1216, decided 2026-11-05): PhysicalProperties::readable_at(consumer_timing) → PhysicalRequirement { consumer_timing, reads_from: Evaluation, consumer_schedule: Option<EvaluationSchedule> } + satisfies_physical (consumer_schedule is None until #1200 PR 2); Lifetime::is_materialized → PhysicalProperties::is_materialized; Lifetime::is_across_evaluations deleted. They are done in #1216 rather than here so the three stacked PRs do not each rewrite the same call sites.

Divergences from the doc's section 3: no schedule field (#1200 PR 2); ExecutionTiming stays in ir/properties/execution_data_state.rs (the doc places it in physical/properties.rs); plan nodes keep timing + lifetime fields (the export writes both, as the doc says).

How

PhysicalProperties is a pair with private fields and a checked constructor, so every value in Stage 2 and Stage 3 is a valid row of the table in properties.rs. Stage 2 builds its alternatives from PhysicalProperties' named consts, each one of the old enum's values. Code that matched on an old variant now matches on (timing, lifetime) with the mapping in (b). The plan's DAG validation rejects a node whose timing and lifetime cannot go together, or a summary build that is not materialized outside query time, which is what the old timing check accepted.

Key code interfaces added / modified

crates/types/src/physical/properties.rs (new; replaces materialization.rs)

/// Where a materialized output is stored. Stays; moved from materialization.rs.
pub enum Medium {
    Memory, // unchanged
    Disk,   // unchanged
}

/// How long a node's output lives (ℓ_v). New; replaces `Materialization`.
pub enum Lifetime {
    /// Discarded after its reader's execution completes (default). // new; was NotMaterialized
    None,
    /// Kept until every reader within one evaluation finishes. // new; was MatAtQuery
    OneEvaluation(Medium),
    /// Kept across evaluations, so later evaluations reuse it. // new; was MatAtIngestion / MatAtQueryKeptAcrossEvaluations
    AcrossEvaluations(Medium),
}
impl Lifetime {
    /// OneEvaluation or AcrossEvaluations: the output outlives its reader.
    pub fn is_materialized(self) -> bool; // new
    /// AcrossEvaluations: a materialized view later evaluations reuse.
    pub fn is_across_evaluations(self) -> bool; // new
}

/// A timing and lifetime that cannot go together; bad input from the caller.
pub enum InvalidPhysicalProperties {
    /// (IngestionTime, OneEvaluation): there is no evaluation at ingestion time.
    OneEvaluationAtIngestion, // new
}

/// p_v^phys = (τ, ℓ); only valid pairs are constructible. New.
pub struct PhysicalProperties { /* timing: ExecutionTiming, lifetime: Lifetime */ }
impl PhysicalProperties {
    /// The pair, or Err for (IngestionTime, OneEvaluation).
    pub fn new(timing: ExecutionTiming, lifetime: Lifetime) -> Result<Self, InvalidPhysicalProperties>; // new
    pub fn timing(self) -> ExecutionTiming;   // new
    pub fn lifetime(self) -> Lifetime;        // new
    /// Same timing, or a query-time consumer of an ingestion-time AcrossEvaluations output.
    pub fn readable_at(self, reader: ExecutionTiming) -> bool; // new; replaces compatible_timing
    /// The values Stage 2 builds. // new
    pub const NOT_MATERIALIZED_AT_QUERY_TIME: Self;
    pub const NOT_MATERIALIZED_AT_INGESTION: Self;
    pub const MATERIALIZED_IN_MEMORY_FOR_ONE_EVALUATION: Self;
    pub const MATERIALIZED_IN_MEMORY_ACROSS_EVALUATIONS_AT_INGESTION: Self;
    pub const MATERIALIZED_IN_MEMORY_ACROSS_EVALUATIONS_AT_QUERY_TIME: Self;
}

crates/types/src/stages/physical/memo_candidates.rs

/// One operator node with one timing and lifetime: what Stage 3 selects. Stays.
pub struct PhysicalCandidate {
    /// The operator node it runs.
    pub operator_node: OperatorNodeOfCandidateForMemoGroupId, // unchanged
    /// When it runs and how long its result lives.
    pub properties: PhysicalProperties, // new; was `timing` + `materialization`
    /// Per input, the candidates that may produce it (filtered by `readable_at`).
    pub producer_candidates_per_input: Vec<ProducerCandidates>, // unchanged
    /// Whether it may produce a query's output.
    pub may_produce_query_output: bool, // unchanged
    /// Per query, whether a query-time AcrossEvaluations chain may be kept with that query as sole reader.
    pub may_keep_across_evaluations_per_sole_reader_query: Vec<bool>, // unchanged
}

/// A pruned candidate, named by what it would have been. Stays.
pub struct PhysicalCandidateKey {
    /// Index into `operator_nodes_of_candidates_for_memo_groups`.
    pub operator_node: OperatorNodeOfCandidateForMemoGroupId, // unchanged
    /// Its timing and lifetime.
    pub properties: PhysicalProperties, // new; was `timing` + `materialization`
}

crates/types/src/ir/operator/node.rs, crates/types/src/ir/physical_export.rs, crates/types/src/physical/workload_plan.rs, crates/types/src/ir/properties/timing.rs, crates/planner/src/installable.rs

// OperatorNode (stays): how long this node's output lives; None by default.
pub lifetime: Lifetime,                                   // was `materialization: Materialization`
pub fn with_lifetime(self, lifetime: Lifetime) -> Self;   // was with_materialization
/// Its pair once timed (None before), or why timing and lifetime cannot go together.
pub fn physical_properties(&self) -> Option<Result<PhysicalProperties, InvalidPhysicalProperties>>; // new

// ExecutionDataStateError (stays): a timed node whose timing and lifetime cannot go together; a planner bug.
InvalidPhysicalProperties { operator: &'static str, #[source] reason: InvalidPhysicalProperties }, // new

// asap_cost::analytical::AnalyticalCostError (stays): the same, met while costing.
InvalidPhysicalProperties { node: PhysicalASAPNodeId, #[source] reason: InvalidPhysicalProperties }, // new; names the node

// PhysicalASAPDAGNode (stays; plan-file node): with output_state.timing, the node's PhysicalProperties.
pub lifetime: Lifetime,                                   // was `materialization: Materialization`
/// Its pair, unless timing and lifetime cannot go together.
pub fn physical_properties(&self) -> Result<PhysicalProperties, InvalidPhysicalProperties>; // new

// PhysicalASAPDAGValidationError (stays). Both are planner bugs.
/// A node whose timing and lifetime cannot go together.
InvalidPhysicalProperties { node: PhysicalASAPNodeId, #[source] reason: InvalidPhysicalProperties }, // was MaterializationTimingMismatch
/// A summary build or MaintainPopulation not materialized and not at query time.
UnmaterializedSummaryBuildNotAtQueryTime { node: PhysicalASAPNodeId }, // new; split out of MaterializationTimingMismatch

// PhysicalWorkloadPlan (stays)
pub materialization: BTreeMap<PhysicalASAPNodeId, Materialization>, // removed: read `dag.nodes`
#[cfg(test-support)] pub fn materializes_any(&self) -> bool;               // was materializes_any

// PhysicalPropertiesAssignment (was MaterializationAssignment): each assigned node's properties; unlisted = not materialized.
pub fn set(&mut self, node: &Rc<OperatorNode>, properties: PhysicalProperties); // was Option<Materialization>
pub fn copy_assignment(&mut self, original: &OperatorNode, copy: &Rc<OperatorNode>); // new: a rebuilt copy keeps its original's
#[cfg(test-support)] pub fn materialized_at_ingestion_where_allowed(roots: &[Rc<OperatorNode>]) -> Self; // new; replaces with_default in tests
pub fn with_default(materialization: Option<Materialization>) -> Self;   // removed
fn get(&self, node: &OperatorNode) -> Option<PhysicalProperties>; // was pub, Option<Option<_>>

// Plan-file format version.
pub const INSTALLABLE_PLAN_WIRE_VERSION: u32 = 12;        // was 11

Usage (end to end, run at the head commit): Example 1's two PromQL queries, sum by (job) (rate(http_requests_total[1m])) and topk by (job) (10, sum_over_time(http_requests_total[1m])), exact, every 10 s:

# The workload file the user writes; the planner runs Stages 0-3 on it and prints each stage.
cargo run --release -p asap-devtools --bin asap_planner -- \
  --workload tools/dag-viewer/workloads/planner-layering/example1.json --run-until stage3

Stage 2's output lists each physical candidate with its PhysicalProperties and its readable producers (readable_at). The scan O0 gets three candidates; an exact SummaryAgg that may run at ingestion gets (IngestionTime, AcrossEvaluations) and reads only the ingestion-time scan C1:

    C0:    O0    QueryTime      None
    C1:    O0    IngestionTime  None
    C2:    O0    QueryTime      OneEvaluation(Memory)
    C4:    O1    QueryTime      OneEvaluation(Memory)              <- {C0, C2}
    C60:   O57   QueryTime      None                               <- {C0, C2}
    C61:   O57   IngestionTime  AcrossEvaluations(Memory)          <- {C1}

Pruned candidates name their properties too (G4 O155 QueryTime OneEvaluation(Memory) TooFewQueryReaders). Stage 3 selects the shared query-time time range C4 (OneEvaluation) read by both queries, with cost_per_second: 4.377000.

Major algorithm changes

None. Candidate generation, readable_at (was compatible_timing), run_kind and the cost terms keep the same truth tables under the mapping in (b). One cost arm changes for a case Stage 2 does not generate (see Verification).

Before this PR

A candidate's timing was stored twice (PhysicalCandidate::timing and inside Materialization), and lifetime only existed as part of four enum variants. Code asking "is this kept across evaluations?" had to list MatAtIngestion | MatAtQueryKeptAcrossEvaluations; code asking "when does it run?" called Materialization::timing(). The plan file had a materialization field per node.

After this PR

Timing and lifetime are separate values, and only valid pairs exist. The next steps of #1207 (σ in #1200 PR 2, explicit edges in #1201, MILP contexts only for None candidates in #1204) can each read the value they need. The plan file has a lifetime field per node at wire version 12; older plan files are refused by version.

Verification

  • cargo check --workspace --all-targets, cargo fmt --all, cargo clippy --workspace --all-targets -- -D warnings: clean.
  • cargo test --workspace --no-fail-fast at the head (covers types, cost, physical-asap-optimization, plan-selection --lib, logical-asap-optimization, planner, executor, devtools snapshots, integration tests): 2093 passed, 0 failed, 40 ignored. An earlier run caught deployment_capabilities::query_time_results_retained_still_offers_the_chain_shared_within_one_evaluation: its test helper matched every AcrossEvaluations chain, ingestion-time ones included; it now matches (QueryTime, AcrossEvaluations).
  • logical-asap-optimization's test-support feature now enables asap-types/test-support (its at_ingestion helper calls the test-support builder), so cargo test -p asap-cost alone builds; cargo check -p <crate> --all-targets passes for every touched crate on its own.
  • After the last review round: cargo test -p asap-types (256 passed), -p asap-cost (45), -p asap-plan-selection --lib (142), -p asap-planner --test planner runtime_install (37), all 0 failed; new tests an_invalid_timed_node_reports_its_error, an_invalid_node_is_reported_not_skipped.
  • Review round 2 (head of this push): AnalyticalCostError::InvalidPhysicalProperties names its node; new timing-pass tests a_summary_build_over_an_invalid_input_reports_its_error (the validate_asap call site) and a_non_asap_node_over_an_invalid_input_reports_its_error (the validate_non_asap call site). After the kept → materialized rename: cargo test -p asap-types, -p asap-cost, -p asap-physical-asap-optimization, -p asap-executor, -p asap-planner, -p asap-plan-selection --lib, -p asap-devtools --lib, -p asap-devtools --test devtools, -p asap-integration-tests: types 258, cost 45, physical-asap-optimization 29, executor 296, planner 89 (8 wall-clock ignored), plan-selection lib 142, devtools lib 18, devtools tests 82, integration tests 364 passed, 0 failed; cargo fmt --all and cargo clippy --workspace --all-targets -- -D warnings: clean.
  • Review round 3 (head of this push, 64e721439): the remaining review fixes (kept → materialized in fields, errors, labels, test names and docs; set with lifetime None removes the entry in release builds too). cargo test -p asap-types (259), -p asap-cost (45), -p asap-physical-asap-optimization (29), -p asap-executor (296), -p asap-planner (89, 8 wall-clock ignored), -p asap-plan-selection --lib, -p asap-devtools, -p asap-integration-tests (364): 0 failed after range_evaluation::stage3_plan_selection_prices_61_evaluations_per_dashboard_request was updated for the chain's detail text "; materializes N bytes across evaluations" (was "; keeps N bytes across evaluations"); cargo fmt and cargo clippy --workspace --all-targets -- -D warnings: clean.
  • DAG viewer: node --test tools/dag-viewer/{stages,enumeration,memo}.test.mjs: 43 pass.
  • Snapshots: rename only. Corpus and sketch-plan snapshots, viewer fixtures and plan fixtures change only "materialization": "NotMaterialized" → "lifetime": "None", {"MatAtIngestion": "Memory"} → {"AcrossEvaluations": "Memory"}, {"MatAtQuery": "Memory"} → {"OneEvaluation": "Memory"}. No selected plan or cost changes in any snapshot.
  • Cost arms to check by hand (no snapshot plan reaches them):
    • asap_cost::usage::node_usage: a query-time AcrossEvaluations node that is not a tumbling-window chain is now priced as a whole materialized state (state × 2, as an ingestion-time materialized state), not with the chain's per-window retention formula. Stage 2 offers query-time AcrossEvaluations only to tumbling-window chains, so no generated plan reaches this arm; the plan-selection test helper keeping_state does.
    • maintained (the per-sample update charge) now requires ingestion timing as well as AcrossEvaluations(Memory). Before, it keyed on MatAtIngestion(Memory), which always meant ingestion timing.
  • Not run: no baseline plan comparison against main (plans are checked by hand with asap_planner). Wall-clock / time-budget tests stay ignored.

🤖 Generated with Claude Code

https://claude.ai/code/session_01K1M6br9PL4SDeVS1XpezKJ

@zzylol zzylol added cost-model Cost formulas, statistics, ranking, and selection ir Pre-ASAP or post-ASAP intermediate representation optimizer Workload optimization and plan selection refactor labels Nov 4, 2026
zzylol and others added 3 commits November 4, 2026 23:19
…fetime)

Each physical candidate stores PhysicalProperties { timing, lifetime } with
Lifetime { None, OneEvaluation(Medium), AcrossEvaluations(Medium) };
PhysicalProperties::new rejects (IngestionTime, OneEvaluation). Plan nodes
carry `lifetime` next to their timing; the plan file's node field
`materialization` becomes `lifetime` (wire version 12). compatible_timing
becomes PhysicalProperties::readable_at with the same truth table. Stage 2
offers the same alternatives as before, one to one with the old variants.

Deletes Materialization and its variants, Materialization::timing,
is_materialized_view, compatible_timing, and the old export field.

Part of #1200 (PR 1 of 2).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01K1M6br9PL4SDeVS1XpezKJ
MaterializationAssignment -> PhysicalPropertiesAssignment,
apply_materialization_timings(_to_roots) -> apply_physical_properties(_to_roots),
MaterializationCandidate -> KeptOutputCandidate (find_kept_output_candidates),
SummaryMaterializationEligibility -> SummaryKeepEligibility,
ChooseMaterialization -> ChoosePhysicalProperties,
MaterializationConflictsWithFixedTiming -> PhysicalPropertiesConflictWithFixedTiming,
is_materialization_point -> is_keepable_state_output,
HighestSharedQueryMaterialization -> HighestSharedOneEvaluation,
materialize_selected_candidates -> realize_selected_candidates, and test
names and files that said materialization / not_materialized. Docs,
including planner-layering Algorithm 3 and the Example 1-4 acceptance
docs, use (timing, lifetime) for the old enum's values.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01K1M6br9PL4SDeVS1XpezKJ
@zzylol
zzylol force-pushed the refactor/1200-timing-lifetime branch from 5ad3731 to bca3b77 Compare November 4, 2026 23:51
zzylol and others added 2 commits November 5, 2026 00:59
…sors

- PhysicalPropertiesAssignment maps kept nodes only: with_default and its
  unassigned field are deleted (tests use kept_at_ingestion_where_allowed),
  set takes PhysicalProperties, get is private, copy_assignment replaces
  split_by_timing's get/set pair; Stage 2's choice callback returns an
  option index instead of Option<Option<_>>.
- PhysicalASAPDAGValidationError::InvalidPhysicalProperties carries the
  typed reason; UnkeptSummaryBuildNotAtQueryTime is its own variant.
- Lifetime::is_across_evaluations, OperatorNode::physical_properties and
  PhysicalASAPDAGNode::physical_properties replace hand-written matches,
  is_view and kept_at_ingestion.
- PhysicalWorkloadPlan::lifetime (a copy of dag's lifetimes) is deleted.
- PhysicalProperties gets named consts for the values Stage 2 builds; the
  Stage 2 constructors and test copies are deleted, as is
  optimize_physical_asap_assigned (duplicate of the test_support entry).
- Code says kept, not materialized, for lifetime != None.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01K1M6br9PL4SDeVS1XpezKJ
…est-support

test_support::at_ingestion calls the test-support builder
PhysicalPropertiesAssignment::kept_at_ingestion_where_allowed, so a crate
built alone with logical-asap-optimization/test-support (e.g. cargo test
-p asap-cost) needs asap-types' test-support too.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01K1M6br9PL4SDeVS1XpezKJ
zzylol and others added 2 commits November 5, 2026 03:39
…timed

OperatorNode::physical_properties returns Option<Result<..>>: None before
timing, Err when timing and lifetime cannot go together. The timing pass
reports ExecutionDataStateError::InvalidPhysicalProperties, and
execution_demand::computations returns
AnalyticalCostError::InvalidPhysicalProperties instead of skipping the node.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01K1M6br9PL4SDeVS1XpezKJ
…est both timing-pass call sites

AnalyticalCostError::InvalidPhysicalProperties becomes { node, reason }, so a
costing error names the node whose timing and lifetime cannot go together.
New timing-pass tests cover the validate_asap and validate_non_asap call
sites of ExecutionDataStateError::InvalidPhysicalProperties.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01K1M6br9PL4SDeVS1XpezKJ
The formulation (#1200) calls l != None materialized. Every name this PR
introduced with 'kept' says 'materialized': Lifetime::is_materialized,
MaterializedOutputCandidate, find_materialized_output_candidates,
SummaryMaterializationEligibility, materialized_at_ingestion_where_allowed,
materializes_any, materialized_tumbling_window_chains,
PhysicalProperties::MATERIALIZED_IN_MEMORY_* and NOT_MATERIALIZED_AT_*,
Consumer::Materialized, and the tests and doc comments.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01K1M6br9PL4SDeVS1XpezKJ
zzylol added a commit that referenced this pull request Nov 5, 2026
PhysicalProperties stays the struct of (ExecutionTiming, Lifetime) from
#1214. Consumers match (timing(), lifetime()) with one arm per valid state
or arms that list their states, and one unreachable!() arm for the invalid
(IngestionTime, OneEvaluation): new, readable_at, summary_properties_allowed,
run_kind, node_usage, materialized_tumbling_window_chains (now Result, the
error names the node), materialized_run_rate (one classification shared by
computations and the MILP's context rates) and the devtools phase_label.
RunKind::Shareable and ContextKind::Shared become Materialized (kappa_m) and
carry no copy of the candidate's properties. The selector and select path
share cost_error and materialized_chains_of; Stage 3's window_read_through
uses the shared chain predicate. Table tests over every valid state
(PhysicalProperties::every_valid) for each consumer.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01K1M6br9PL4SDeVS1XpezKJ
zzylol added a commit that referenced this pull request Nov 5, 2026
…e None

Review fixes for #1214: the assignment's field, Debug label, set's
parameter and docs, error messages, the devtools and viewer labels, the
cost detail for chains across evaluations, test names, locals and the
fixture name say materialized. post-asap-ir.md documents set(node,
properties); the example 4 acceptance doc names materializing().
PhysicalPropertiesAssignment::set with a lifetime None removes the
node's assignment in release builds too, instead of only a debug_assert.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01K1M6br9PL4SDeVS1XpezKJ
…e None

Review fixes for #1214: the assignment's field, Debug label, set's
parameter and docs, error messages, the devtools and viewer labels, the
cost detail for chains across evaluations, test names, locals and the
fixture name say materialized. post-asap-ir.md documents set(node,
properties); the example 4 acceptance doc names materializing().
PhysicalPropertiesAssignment::set with a lifetime None removes the
node's assignment in release builds too, instead of only a debug_assert.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01K1M6br9PL4SDeVS1XpezKJ
@zzylol

zzylol commented Nov 6, 2026

Copy link
Copy Markdown
Contributor Author

Closed: replaced by #1207's new order (interfaces first, then one PR per stage), starting with PR 0 #1226. The code on this branch is reused in PRs 1–8; the branch is kept.

@zzylol zzylol closed this Nov 6, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

cost-model Cost formulas, statistics, ranking, and selection ir Pre-ASAP or post-ASAP intermediate representation optimizer Workload optimization and plan selection refactor

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant