Skip to content

docs: refresh the ingestion-time views of one refresh window in one run (#1151) - #1152

Merged
zzylol merged 2 commits into
feat/1095-stage3from
feat/1151-refresh-docs
Oct 31, 2026
Merged

zzylol merged 2 commits into
feat/1095-stage3from
feat/1151-refresh-docs

Conversation

@zzylol

@zzylol zzylol commented Oct 31, 2026 •

Copy link
Copy Markdown
Contributor

Why

Part of #1151. A KLL pane chain and a DDSketch pane chain with the same 10 s panes over one ingestion Scan are refreshed today in two ingestion runs, one per view, so each pane's raw data is read twice. The executor already runs a node shared by several roots once per execute call, but the deployment contract (the running example refresh = compile(dag, [], [pane_0]) in physical-planning-and-deployment.md §4 and the materialized_view.rs module doc) says to compile and run each view alone. #1151 changes the contract and the cost rule together. This PR changes the contract text first.

What

Documentation only.

  • docs/design_docs/physical-planning-and-deployment.md §4: new subsection Refresh groups.
    • It defines the refresh window of a view refreshed at ingestion: windows of width W ending o mod W after a multiple of W, from its view_window (T - o - W, T - o]. All panes of a chain have their newest pane's refresh window.
    • A refresh group is the views refreshed at ingestion that share one refresh window (W, p).
    • A runtime refreshes a group with one ingestion run per window (e - W, e], e ≡ p (mod W).
      • The run's roots are the views of the group whose state for the window ending at e is not stored yet. In steady state that is the newest pane of each chain, because older panes are the same states, stored by earlier runs.
      • A view of the group that a root reads is computed in the run. Views of other groups are materialized view slots.
    • A 10 s chain and a 60 s chain stay in separate runs. A runtime that splits a group computes the same states with more runs.
    • The sentence on how Stage 3 prices this grouping moved to the cost PR (feat: charge a shared ingestion node once per refresh window (#1151) #1154), which introduces that rule.
  • §4 running example: the KLL p50/p99 chain plus a DDSketch p99.9 chain, both with one-minute panes. Each pane's run has roots [kll_pane_0, dd_pane_0], the newest pane of each chain (the group holds ten pane views), and makes one read_raw_data call.
  • §3 note on illustrative operator names: adds NativeDDSketchBuild (implemented as SummaryAgg).
  • §5 responsibility table and text: the deployment schedules one ingestion run per refresh group and window.
  • crates/types/src/ir/materialized_view.rs module doc:
    • New section "Refresh groups", with the same run rule as the design doc.
    • The "Refreshing a view" paragraph points to it, and its slots are "each view it reads that the run does not refresh".

How

Text only. The derived API (RefreshWindow, PhysicalASAPDAG::refresh_window, ingestion_refresh_groups) arrives in the next PR of the stack (#1153), and the cost rule and its doc sentence in the PR after it (#1154). The doc names those concepts in words here, and #1153 adds the API to the plan-file block.

Key code interfaces added / modified

None (documentation only).

Major algorithm changes

None.

Before this PR

The contract refreshed each view in its own ingestion run, compiled for that view's node alone. Two equal-width chains over one Scan meant two runs and two raw reads per pane.

After this PR

The contract refreshes the views of one refresh window with one run per window, whose roots are the views without a stored state for that window. The doc states when two views can share a run, and what stays the deployment's choice (sealing, lateness, threading, grid alignment).

Verification

🤖 Generated with Claude Code

@zzylol
zzylol merged commit 060096a into feat/1095-stage3 Oct 31, 2026
2 checks passed
zzylol added a commit that referenced this pull request Nov 2, 2026
…ge (#1151, lands #1152–#1154 on main) (#1182)

* docs: refresh the ingestion-time views of one refresh window in one run (#1151) (#1152)

The deployment contract refreshes the ingestion-time views of one refresh window in one run per window end; the run's roots are the group's views whose state for that window is not yet stored. Part of #1151.

* feat: derive ingestion refresh groups and refresh each group in one run (#1151) (#1153)

RefreshWindow, PhysicalASAPDAG::refresh_window and ingestion_refresh_groups derive which ingestion-time views share one run; the install compile check and the stand-in runtimes refresh per group, and a counting-store test shows one raw read per window per group. No cost change. Part of #1151.

* feat: charge a shared ingestion node once per refresh window (#1151) (#1154)

asap_cost::execution_demand::computations counts a NotMaterialized ingestion-time node once per refresh window of its MatAtIngestion consumers instead of once per consumer, and the old Stage 3 engine (general_schedule, DirectCostProfile::selected_cost, direct_cost::encode) charges it the same way. Snapshots unchanged. Closes #1151 except the backend follow-up.

* fix: use the tumbling-window names (#1163) in the refresh-window code rebased onto main

The refresh-window commits (#1152-#1154) were written before #1163 renamed
PaneChain to SummaryTumblingWindowChain and pane_width_ms / pane_count /
newest_panes to tumbling_window_width_ms / tumbling_window_count /
newest_tumbling_windows; rename their uses so they build on main.
@zzylol
zzylol deleted the feat/1151-refresh-docs branch November 3, 2026 02:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

cost-model Cost formulas, statistics, ranking, and selection documentation Improvements or additions to documentation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant