Repository navigation
docs: refresh the ingestion-time views of one refresh window in one run (#1151) - #1152
Merged
Merged
Conversation
This was referenced Oct 31, 2026
…t moves to the cost change (#1151)
This was referenced Oct 31, 2026
zzylol
added a commit
that referenced
this pull request
Nov 2, 2026
…ge (#1151, lands #1152–#1154 on main) (#1182) * docs: refresh the ingestion-time views of one refresh window in one run (#1151) (#1152) The deployment contract refreshes the ingestion-time views of one refresh window in one run per window end; the run's roots are the group's views whose state for that window is not yet stored. Part of #1151. * feat: derive ingestion refresh groups and refresh each group in one run (#1151) (#1153) RefreshWindow, PhysicalASAPDAG::refresh_window and ingestion_refresh_groups derive which ingestion-time views share one run; the install compile check and the stand-in runtimes refresh per group, and a counting-store test shows one raw read per window per group. No cost change. Part of #1151. * feat: charge a shared ingestion node once per refresh window (#1151) (#1154) asap_cost::execution_demand::computations counts a NotMaterialized ingestion-time node once per refresh window of its MatAtIngestion consumers instead of once per consumer, and the old Stage 3 engine (general_schedule, DirectCostProfile::selected_cost, direct_cost::encode) charges it the same way. Snapshots unchanged. Closes #1151 except the backend follow-up. * fix: use the tumbling-window names (#1163) in the refresh-window code rebased onto main The refresh-window commits (#1152-#1154) were written before #1163 renamed PaneChain to SummaryTumblingWindowChain and pane_width_ms / pane_count / newest_panes to tumbling_window_width_ms / tumbling_window_count / newest_tumbling_windows; rename their uses so they build on main.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Part of #1151. A KLL pane chain and a DDSketch pane chain with the same 10 s panes over one ingestion
Scanare refreshed today in two ingestion runs, one per view, so each pane's raw data is read twice. The executor already runs a node shared by several roots once perexecutecall, but the deployment contract (the running examplerefresh = compile(dag, [], [pane_0])inphysical-planning-and-deployment.md§4 and thematerialized_view.rsmodule doc) says to compile and run each view alone. #1151 changes the contract and the cost rule together. This PR changes the contract text first.What
Documentation only.
docs/design_docs/physical-planning-and-deployment.md§4: new subsection Refresh groups.Wendingo mod Wafter a multiple ofW, from itsview_window(T - o - W, T - o]. All panes of a chain have their newest pane's refresh window.(W, p).(e - W, e],e ≡ p (mod W).eis not stored yet. In steady state that is the newest pane of each chain, because older panes are the same states, stored by earlier runs.[kll_pane_0, dd_pane_0], the newest pane of each chain (the group holds ten pane views), and makes oneread_raw_datacall.NativeDDSketchBuild(implemented asSummaryAgg).crates/types/src/ir/materialized_view.rsmodule doc:How
Text only. The derived API (
RefreshWindow,PhysicalASAPDAG::refresh_window,ingestion_refresh_groups) arrives in the next PR of the stack (#1153), and the cost rule and its doc sentence in the PR after it (#1154). The doc names those concepts in words here, and #1153 adds the API to the plan-file block.Key code interfaces added / modified
None (documentation only).
Major algorithm changes
None.
Before this PR
The contract refreshed each view in its own ingestion run, compiled for that view's node alone. Two equal-width chains over one
Scanmeant two runs and two raw reads per pane.After this PR
The contract refreshes the views of one refresh window with one run per window, whose roots are the views without a stored state for that window. The doc states when two views can share a run, and what stays the deployment's choice (sealing, lateness, threading, grid alignment).
Verification
cargo fmt --all --check: clean.materialized_view.rs). No intra-doc links were added.🤖 Generated with Claude Code