Skip to content

refactor(types): properties and requirements from #1207 (#1207 PR 1) - #1227

Draft
zzylol wants to merge 2 commits into
refactor/1207-crates-per-stagefrom
refactor/1207-properties
Draft

zzylol wants to merge 2 commits into
refactor/1207-crates-per-stagefrom
refactor/1207-properties

Conversation

@zzylol

@zzylol zzylol commented Nov 6, 2026 •

Copy link
Copy Markdown
Contributor

Merge unit. This is PR 1 of #1207's merge unit (PRs 0–9). It is stacked on PR 0b (#1229, refactor/1207-crates-per-stage), which is stacked on PR 0 (#1226). The PRs merge into main together, after PR 9. This PR fills in only the section 2 functions that PR 0 left as todo!(). Planning still panics at the first todo!() of a later stage (see "Test status").

Why

Every later stage reads the property and requirement checks of the concepts doc, section 2:

  • Pass 1 (PR 3) adds one candidate per listed θ (parameter_list).
  • Pass 2 (PR 4) builds logical edges with $\alpha^{\log}$ (producer_satisfies_logical_requirements).
  • Stage 2 (PR 5) builds physical alternatives (PhysicalProperties::new) and edges with $\alpha^{\mathrm{phys}}$ (producer_satisfies_physical_requirements).
  • Stage 3 (PR 7) checks alignment per context (delivers_on).

PR 0 defined these signatures with todo!() bodies. This PR implements them, so that the stages built on them start from checked properties.

What

  • Physical properties. There is no schedule σ (decided 2026-11-06; PR 0 deletes it): $p^{\mathrm{phys}} = (\tau, \ell)$. PhysicalProperties::new rejects the one invalid combination, (IngestionTime, OneEvaluation).
  • Physical candidates.
    • PhysicalCandidate::delivers_on(memo, context) implements the per-context alignment table: a windowed AcrossEvaluations producer of width $w$ (read from its logical layout; every grid is anchored at phase 0) serves an evaluation context iff $w$ divides every evaluation time of it: the evaluation grid gcd(interval, range step, phase) of FixedIntervalAt, every Scheduled time, or a OneTime query's time. It is false when the times are not known (FixedInterval without a phase, OneTime without a time, EstimatedRate, Unknown). A None or OneEvaluation producer runs in the consumer's own context, so it is true for it. The ingestion and materialized contexts stay todo!() for PR 7.
    • Evaluation grid moved. QueryWorkloadEntry::evaluation_grid_ms moved to QueryRecurrence::evaluation_grid_ms(&self, range_evaluation) in types/src/workload/mod.rs, so the context can call it; the entry method now delegates (no duplicated logic). It also covers Scheduled (gcd of the times) and OneTime with a time; its only callers are the workload tests, whose cases are unchanged.
    • producer_satisfies_physical_requirements implements the timing × lifetime table only.
  • Logical properties.
    • LogicalProperties::new rejects a sketch build without parameters (SketchBuildWithoutParameters), parameters on an operator without a sketch family or of another family, and parameters not in the family's list (ParametersNotInList). It also rejects a window of width 0, and a window on anything but a SummaryAgg whose states merge.
    • parameter_list returns each family's predefined θ list. Heap variants take each width × depth with heap sizes from the 1-2-5 series 10 … 10 000 (70 entries); KMV's and UnivMon's distinct-count entries stop at ε = 0.01.
    • LogicalRequirements::new stores any floor of the family's parameter type, listed or not (e.g. heap_size ≥ k exactly).
    • producer_satisfies_logical_requirements compares θ per family on the parameters that set ε.
  • Accuracy crate. The sizing functions (size_params and its helpers) are now #[cfg(test)]. Their only remaining role is a test that keeps the predefined lists equal to them. sizing::default_size_params, test_support::default_size_params and test_support::cms_width are deleted (no caller).

No other stage changes. Their todo!()s stay.

Decisions and pending items

Item Decision Status
parameter_list contents Today's sizing at δ = 0.01 over ε ∈ {0.1, 0.05, 0.02, 0.01, 0.005, 0.002, 0.001}, weakest first, without repeats. HLL saturates at precision 18 (4 entries). KMV and UnivMon's distinct-count shapes stop at ε = 0.01 (decided 2026-11-06): KMV 4 entries; UnivMon 7 L2 shapes, 4 distinct-count shapes, then its one shape for other statistics (12). pending (ε range approved 2026-11-06)
Heap sizes of the heap variants Each CMS / CountSketch width × depth (the top-1 sizing over the ε grid) with heap_size from the 1-2-5 series 10, 20, 50, 100, 200, 500, 1 000, 2 000, 5 000, 10 000: 7 × 10 = 70 entries, by width then heap size. Maximum 10 000, so a top-k with k > 10 000 has no listed heap. A top-k consumer's floor is heap_size ≥ k; the next listed step up is at most 2.5× too large. approved 2026-11-06 (series and maximum 10 000)
Hydra and other families' lists Empty: Pass 1 offers no Hydra (HydraCms, HydraCountSketch, HydraKll, HydraHll, HydraUnivMon), nor HllHip, KllDynamic, … So no Hydra build is valid yet (ParametersNotInList). dominates still compares Hydra sizes per kind, and a unit test checks that. pending
What "the parameters that set ε" are KLL k and KMV k ≥; DDSketch alpha ≤; HLL precision ≥; CMS and CountSketch width, depth ≥; heap variants also heap_size ≥; UnivMon all four sizes ≥; Hydra all sizes of one kind ≥. The recorded failure_probability, relative_error and distinct_failure_probability are not compared, because every listed configuration records δ = 0.01. Another family never satisfies. pending
Which operators can be windowed Only SummaryAgg whose family has a merge (FamilyCapability::of(family).merge_inputs.is_some()). A SummaryTumblingWindowChain already carries its width, so it is rejected. pending
A sketch build with parameters: None Rejected (SketchBuildWithoutParameters). Exact operators keep parameters: None. approved 2026-11-06
θ outside the list in LogicalProperties::new Rejected (ParametersNotInList). Because Hydra lists are empty, no Hydra build is valid until they get lists. approved 2026-11-06
A floor outside the list Accepted: LogicalRequirements::new checks nothing and returns Self (PR 0). approved 2026-11-06
Schedule σ Dropped (decided 2026-11-06; PR 0). PhysicalCandidate::new and its grid variants are deleted with it, since it existed only for the grid checks; PhysicalCandidate's fields stay public as in the doc. applied
A non-windowed AcrossEvaluations producer in delivers_on false: the doc's alignment table says it is not offered yet (it needs its own delivery rule). approved 2026-11-06
producer_satisfies_physical_requirements takes no memo Dropped in PR 0: it reads only timing and lifetime. approved 2026-11-06
Memo::candidate Implemented here (one indexing line) because delivers_on reads the layout through it. PR 2 keeps find_or_add_group and add_candidate. pulled forward from PR 2
Sizing functions #[cfg(test)] estimators::size_params, topk_capacity, cms::cms_width, count_sketch::count_sketch_width, ddsketch::size_params, hll::hll_precision, and univmon's size_params, l2_shape, MAX_LAYERS, MAX_L2_ROWS, HASH_BITS, BASELINE. Production never sizes from ε any more; the lists replace it. pending

New abstractions not in #1207 or the doc (pending approval)

Item Where Why it is needed
QueryRecurrence::evaluation_grid_ms(&self, range_evaluation: Option<RangeEvaluation>) -> Option<u64> (moved from QueryWorkloadEntry::evaluation_grid_ms, which delegates) types/src/workload/mod.rs delivers_on needs the grid of an ExecutionContext, which has the recurrence and range evaluation but no entry.
private family_of_operator, family_of_parameters, can_be_windowed, dominates types/src/stages/candidates.rs The family of an operator or a θ, the windowing rule, and per-family dominance.
private constants KLL_PARAMETERS … UNIVMON_PARAMETERS and const fns kll, hll, kmv, univmon_l2, univmon_distinct same The predefined lists that parameter_list returns.
test helpers: summary_agg, sketch (candidates tests); memo, candidate, evaluation, every, constants UNWINDOWED/WINDOWS_5S/WINDOWS_10S (memo_candidates tests) test modules Build the fixtures of the table tests.

How

  • PhysicalProperties::new rejects (IngestionTime, OneEvaluation) and accepts every other (τ, ℓ).

  • delivers_on matches ℓ: None and OneEvaluation are true (they run in the consumer's context κ, $x_{\kappa,v}$); AcrossEvaluations reads ω from memo.candidate(logical_candidate_id).logical_properties; without a tumbling layout it is false. In an evaluation context, FixedInterval (no phase) is false; every other recurrence is true iff recurrence.evaluation_grid_ms(range_evaluation) is known and a multiple of $w$. Example: two 10 s queries at phases 0 s and 5 s both read a 5 s chain, not a 10 s one.

  • producer_satisfies_physical_requirements matches the producer's (τ, ℓ), with unreachable! for (IngestionTime, OneEvaluation):

    producer consumer QueryTime consumer IngestionTime
    QT, None / OneEvaluation / AcrossEvaluations ✓ ✗
    IT, None ✗ ✓
    IT, AcrossEvaluations ✓ ✓

    Alignment is not part of it: Stage 3 checks delivers_on per context.

  • parameter_list returns constants. They are not computed, because asap-types cannot depend on asap-accuracy; the heap variants' lists are built at compile time by a const fn from the CMS / CountSketch shapes and the heap series. An accuracy test keeps them equal to size_params over the grid.

  • producer_satisfies_logical_requirements: no floor, or an exact producer (parameters: None), satisfies. Otherwise dominates(θ_u, θ^min) holds, with one match arm per family pair. Different families are listed by the producer's variant, not _.

Known limitations

Key code interfaces

Bodies are elided as ;. Every signature below except those marked // new is PR 0's, and only its body changed.

crates/types/src/physical/properties.rs

impl PhysicalProperties {
    /// (τ, ℓ), unless (IngestionTime, OneEvaluation).
    pub fn new(timing: ExecutionTiming, lifetime: Lifetime)
        -> Result<Self, PhysicalPropertiesError>; // body filled in
}

// crates/types/src/workload/mod.rs
impl QueryRecurrence {
    /// The epoch-anchored grid every evaluation with this recurrence and range evaluation
    /// falls on: gcd of interval and phase, of the scheduled times, or a one-off query's
    /// time; with a range evaluation of more than one step, also its step. None when the
    /// times are not known. (Moved from `QueryWorkloadEntry`, which delegates.)
    pub fn evaluation_grid_ms(&self, range_evaluation: Option<RangeEvaluation>) -> Option<u64>; // moved
}

crates/types/src/stages/physical/memo_candidates.rs

impl PhysicalCandidate {
    /// Whether a consumer running in `context` can read this producer: w divides every
    /// evaluation time of the context for a windowed AcrossEvaluations producer; true for
    /// None and OneEvaluation.
    pub fn delivers_on(&self, memo: &Memo, context: &ExecutionContext) -> bool; // body filled in
}

/// α^phys: the timing × lifetime table only.
pub fn producer_satisfies_physical_requirements(producer: &PhysicalCandidate, consumer: &PhysicalCandidate) -> bool; // body filled in

crates/types/src/stages/candidates.rs

impl Memo {
    /// The candidate `id` (pulled forward from PR 2).
    pub fn candidate(&self, id: CandidateForMemoGroupId) -> &CandidateForMemoGroup; // body filled in
}

impl LogicalProperties {
    /// p^log = (θ, ω), unless a sketch build has no θ, θ is on an operator without a sketch family, of another family or not in the family's list, or ω has width 0 or cannot be applied.
    pub fn new(expression: &CandidateExpression, parameters: Option<SketchParams>, window_layout: Option<WindowLayout>)
        -> Result<Self, LogicalPropertiesError>; // body filled in
}

/// The family's predefined θ list, weakest first; empty for families Pass 1 does not offer.
pub fn parameter_list(family: SketchAlgorithm) -> &'static [SketchParams]; // body filled in

impl LogicalRequirements {
    /// r^log with floor θ^min: any value of the family's parameter type, listed or not.
    pub fn new(parameter_floor_for_required_accuracy: Option<SketchParams>) -> Self; // body filled in (signature from PR 0)
}

/// α^log: θ_u ⪰ θ^min per family; no floor or an exact producer always satisfies.
pub fn producer_satisfies_logical_requirements(producer: &LogicalProperties, requirement: &LogicalRequirements) -> bool; // body filled in

crates/accuracy/src/sizing.rs, crates/accuracy/src/test_support.rs

pub(crate) fn default_size_params(kind: SketchAlgorithm, intent: &AggIntent, eps: f64, delta: f64) -> SketchParams; // removed (sizing.rs; replaced by parameter_list)
pub fn default_size_params(kind: SketchAlgorithm, intent: &AggIntent, eps: f64, delta: f64) -> SketchParams; // removed (test_support.rs; no caller)
pub fn cms_width(eps: f64) -> u32; // removed (test_support.rs; no caller)

Usage

No workload reaches these functions yet: Pass 1 (PR 3), Pass 2 (PR 4) and Stage 2 (PR 5) are still todo!(). The flow below is the one those PRs will run, for the doc's running example (query A: p95 over 1 m, every 10 s). It uses the calls the table tests make at the linked commit (memo_candidates.rs and candidates.rs tests).

// Pass 1 (PR 3): one KLL build candidate of group G2 per listed θ.
// `kll_list` is [k=26, 52, 132, 269, 547, 1404, 2863].
let kll_list = parameter_list(SketchAlgorithm::Kll);
// `build` is G2's expression, SummaryAgg[KLL](G1).
let build = CandidateExpression { operator: Operator::ASAP(ASAPOp::SummaryAgg { child: g1, family: FieldDataType::Sketch(SketchAlgorithm::Kll, GroupingStrategy::default()), /* input, reduction, grouping, filter */ }) };
// `k_547` is p^log of the k=547 candidate, here with 5 s tumbling windows (Pass 2 rule 1).
let k_547 = LogicalProperties::new(&build, Some(kll_list[4].clone()), Some(WindowLayout::Tumbling { width_ms: 5_000 }))?;

// Pass 2 (PR 4): A's readout requires rank error ≤ cap, so its floor is k=269.
let floor = LogicalRequirements::new(Some(kll_list[3].clone()));
// `alpha_log` is true: 547 ≥ 269.
let alpha_log = producer_satisfies_logical_requirements(&k_547, &floor);

// Stage 2 (PR 5): the k=547 candidate materialized across evaluations on its 5 s grid (phase 0).
let state = PhysicalCandidate { logical_candidate_id: k_547_id, physical_properties: PhysicalProperties::new(
    ExecutionTiming::QueryTime, Lifetime::AcrossEvaluations(StorageMedium::Memory))? };
// A's readout, materialized for one evaluation.
let readout = PhysicalCandidate { logical_candidate_id: readout_id, physical_properties: PhysicalProperties::new(
    ExecutionTiming::QueryTime, Lifetime::OneEvaluation(StorageMedium::Memory))? };
// `alpha_phys` is true: QT→QT.
let alpha_phys = producer_satisfies_physical_requirements(&state, &readout);

// Stage 3 (PR 7): A's evaluation context, shared with B (both every 10 s at phase 0).
let every = |phase_ms| ExecutionContext::Evaluation {
    recurrence: QueryRecurrence::Repeated(RepeatedDemand::FixedIntervalAt {
        interval: RepetitionInterval(10_000),
        evaluation_phase: TimestampMs(phase_ms),
    }),
    range_evaluation: None,
};
// The 5 s grid divides the 10 s grid at phase 0.
assert!(state.delivers_on(&memo, &every(0)));
// A context at phase 2 s cannot read it: 5 s does not divide 2 s.
assert!(!state.delivers_on(&memo, &every(2_000)));

The planner's text output does not change in this PR. It still stops at Pass 1's todo!().

Major algorithm changes

None. These are new checks behind PR 0's interfaces. The θ lists replace sizing per budget share (deleted in PR 0); PR 3 uses them.

Before this PR

Every section 2 function was todo!() (PR 0). Today's sizing chose θ from ε inside Pass 1 (deleted in PR 0).

After this PR

The property and requirement checks of section 2 run, each covered by an exhaustive table test, so PRs 2–5 can build candidates and edges on them. θ comes from a fixed list per family that matches today's sizing.

Verification

Tests added. Each states one behavior:

  • physical::properties::tests
    • new_over_every_timing_and_lifetime: all 2 timings × 3 lifetimes, both media, against Ok or OneEvaluationAtIngestion; is_materialized on each valid pair.
  • stages::physical::memo_candidates::tests
    • delivers_on_follows_the_alignment_table: one case per recurrence: FixedIntervalAt (with and without a range step; phases 0 and 5 s read a 5 s chain, not a 10 s one), FixedInterval without a phase, Scheduled, OneTime with and without a time, EstimatedRate, Unknown; also non-windowed AcrossEvaluations, None and OneEvaluation.
    • producer_satisfies_physical_requirements_over_every_valid_timing_and_lifetime: 5 × 5 valid states, all medium pairs.
  • stages::candidates::tests
    • logical_properties_new_over_every_operator_parameters_and_window: 6 operators (KLL build, Hydra CMS build, exact Sum, exact IRate, KLL chain, readout) × 5 θ (none, KLL 269, CMS, Hydra CMS, unlisted KLL 270) × 3 windows.
    • parameter_list_holds_its_family_without_repeats.
    • logical_requirement_new_accepts_any_floor: listed and unlisted floors, including heap_size = 37.
    • producer_satisfies_logical_requirements_compares_the_parameters_that_set_the_error: also a top-37 floor, met by heap 50 and not by heap 20.
    • hydra_dominance_compares_every_size_of_one_kind.
  • asap-accuracy estimators::tests
    • parameter_list_is_the_sizing_over_the_error_grid: the lists equal size_params over the grid (distinct counts of KMV and UnivMon to ε = 0.01), recorded fields included; heap variants carry every heap size of the series.
    • a_configuration_satisfying_a_listed_floor_is_no_less_accurate: for every family, every pair of listed θ with θ_u ⪰ θ^min, and every statistic the family answers, θ_u's error is no larger at δ ≤ 0.01. This checks that dominance on the listed configurations is sound, including UnivMon's partial order.

Checks run:

  • cargo check --workspace --all-targets: clean.
  • cargo clippy --workspace --all-targets: clean.
  • cargo fmt --all -- --check: clean.

Test status

cargo test -p <crate> --all-features --no-fail-fast at ba60db8. Every failure panics at a later PR's todo!(); none reaches a PR 1 function. Rows other than asap-types and asap-accuracy are unchanged from PR 0b.

Crate Passed Failed Ignored Fails at Passes after
asap-types 209 0 0 — (was 201 / 2 at PR 0b; PR 0b moved DeploymentCapabilities' test here) this PR
asap-accuracy 79 0 0 — (2 tests added) —
asap-cost 31 1 0 the executor's θ check (capability.rs) PR 6
asap-stage0-memo 24 0 0 — —
asap-stage1-pass1-per-computation-candidates 12 1 0 the per-node sketch guarantee (1); PR 0b deleted the 65 tests that built Stage 0's memo or lowered PromQL PR 4
asap-stage1-pass2-reuse-candidates 0 0 0 no tests left PR 4 adds them
asap-stage2-physical-optimization 0 0 0 no tests left PR 5 adds them
asap-stage3-plan-selection 3 0 0 — PR 6–8 add the MILP tests
asap-planner 9 19 1 Pass 1 (7), the sketch and distinct-count guarantees (12) PR 3 → PR 6
asap-executor 217 56 1 θ checks (capability.rs) and the weighted-frequency kernel PR 6
asap-frontend-promql 218 0 0 — (PR 0b deleted the tests that called Stage 0, Pass 1 or the facade) —
asap-frontend-sql 174 0 0 — —
asap-frontend-metricsql 15 0 0 — —
asap-devtools 49 48 3 Pass 1 and the sketch guarantee, mostly through the asap_planner binary PR 6; text and viewer output in PR 9
asap-integration-tests 103 76 4 Pass 1, the sketch and distinct-count guarantees, the executor's θ check, and PR 0's stale fixture PR 3–6; the fixture in PR 9

@zzylol zzylol added optimizer Workload optimization and plan selection refactor labels Nov 6, 2026
@zzylol
zzylol force-pushed the refactor/1207-properties branch 4 times, most recently from 5ef1ef3 to d9d64df Compare November 6, 2026 15:37
@zzylol
zzylol force-pushed the refactor/1207-properties branch from d9d64df to 2a62109 Compare November 6, 2026 15:47
@zzylol
zzylol force-pushed the refactor/1207-properties branch from 2a62109 to b87b7c0 Compare November 6, 2026 16:30
@zzylol
zzylol changed the base branch from refactor/1207-interfaces to refactor/1207-crates-per-stage November 6, 2026 16:30
@zzylol
zzylol force-pushed the refactor/1207-properties branch 3 times, most recently from 460e9c7 to ba60db8 Compare November 6, 2026 17:12
@zzylol
zzylol force-pushed the refactor/1207-crates-per-stage branch from 4d55373 to ae4843e Compare November 6, 2026 18:56
@zzylol
zzylol force-pushed the refactor/1207-properties branch from ba60db8 to 60a16ee Compare November 6, 2026 18:56
@zzylol
zzylol force-pushed the refactor/1207-crates-per-stage branch from ae4843e to 8d020b0 Compare November 6, 2026 19:10
@zzylol
zzylol force-pushed the refactor/1207-properties branch from 60a16ee to 107a77b Compare November 6, 2026 19:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

optimizer Workload optimization and plan selection refactor

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant