Repository navigation
feat(pass2): share scans of one metric that project different labels - #868
Merged
Merged
Conversation
The PromQL frontend projects only the labels a query uses, and structural CSE shares only nodes with equal schemas, so scans of one metric under `quantile by (host, job)` and `quantile by (job)` were never shared. The identical-expression rule now also tries the candidate with every predicate-free scan of a metric widened to the union of the labels its scans project. Time ranges, shifts and per-series builds keep the wider rows; any other reader narrows them back with a Project that keeps its own columns by name. Only the widened candidates that share a scan are kept. Addresses #840 (questions 1 and 5). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RqhsM5XsKhxjMP6J2XXXfB
zzylol
force-pushed
the
feat/840-union-scan-labels
branch
from
October 22, 2026 11:50
14e7b9a to
8c25da4
Compare
This was referenced Oct 22, 2026
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Addresses #840 (questions 1 and 5).
Why
In #840's workload, Q1/Q2 (
quantile_over_timeover 1 m and 5 m), Q3 (quantile by (host, job) (0.5, data)) and Q4 (quantile by (job) (0.5, data)) all read the rows ofdata. The PromQL frontend projects only the labels each query uses into itsScan(ts, value, $promql_series_identityfor Q1/Q2, plushost, jobfor Q3 andjobfor Q4), and structural CSE shares only nodes with equal schemas. So the identical-expression rule never offered to share Q3's or Q4's scan, nor Q3 and Q4's identical instant time ranges. Since #855, Stage 2 can build a shared scan once per evaluation, so sharing these scans can now be priced fairly.What
Projectthat restores its columns by name, as fix: one finest-grain stored summary serves several groupings (quantiles at the estimate level, sums at the state level) #804 did for per-series builds.asap_aware_csemodule doc describe it.How
with_union_scan_labels(roots)walks the DAG children first and builds, per metric, the union schema of its unfiltered scans (labels inserted before the series identity, in walk order; a label with two types disables widening for that metric). It returnsNonewhen every metric's scans already agree. Otherwise it rebuilds the DAG: widened scans;passes_labels_onnodes rebuilt over the wider children; every other node overnarrowed(child)(aProjectby name). It gives up (None) when a reader would have to narrow a summary state, or when a grouped or filtered pane chain would read aProject(plan selection reads a pane chain's window from the rows directly below it; the executor also needs a per-series build'sTimeRangedirectly below it). Guarantees are recomposed.Pass2List::identical_expressionruns the rule on the original roots as before, then on the widened roots, keeping those that share aScaninPass2List.widened, appended at the end.Key code interfaces added / modified
crates/logical-asap-optimization/src/asap_aware_cse.rscrates/logical-asap-optimization/src/workload_candidates.rsUsage, end to end: #840's 4-query workload through
stage_pipelineat this commit.Before this PR
shared Scan (Q1, Q2). Q1–Q3: 156 and 387.After this PR
shared Scan (Q1, Q2)at 0.1536/s (Q1–Q3: 0.1458/s), as Instant quantile by (host, job) next to windowed quantiles: no scan sharing, no panes, no ingest-time variant, 100-group default #840's analysis predicted.Projectis sized without the source's declared non-negativity (plan selection looks for the scan or time range directly below it). Widening is all-or-nothing per metric, so Q3 and Q4 cannot share a 15 s scan while Q1 and Q2 keep theirs. A whole-window grouped build that reads aProjectis not offered at ingestion time (Stage 2 looks for itsTimeRangedirectly below it); its unwidened counterpart keeps that option. The widened candidates roughly double the logical candidates of a workload whose scans of one metric project different labels.Verification
scans_projecting_different_labels_are_shared(crates/devtools/tests/stage_pipeline.rs, Instant quantile by (host, job) next to windowed quantiles: no scan sharing, no panes, no ingest-time variant, 100-group default #840's 4-query workload) has no "shared Scan (Q1, Q2, Q3, Q4)" candidate on main; it passes now: one scan withts, value, host, job, $promql_series_identity, multiplier 1, "scan 200000 samples; x 0.1000/s", 0.2068/s against 0.1702/s, and the Q1/Q2 shared scan selected.scans_of_one_metric_with_other_labels_share_once_widened(unit):sum by (job) (m)andmax by (host, job) (m)share nothing, then a Scan and a TimeRange once widened; each root keeps its schema; agreeing scans returnNone.joint_search_equivalence::a_shared_summary_is_combined_with_a_rollupcounts a build shared by either rule.with_union_scan_labelskeeps the scans' unique key on the series identity and ts after widening (unit test), and does not widen scans that disagree on whether they hold every label.cargo fmt --all -- --check;cargo clippy -p asap-logical-asap-optimization -p asap-devtools -p asap-integration-tests --all-targets --all-features --locked -- -D warningsclean;asap-logical-asap-optimizationlib (164),asap-integration-testsjoint_search_equivalence (12),asap-devtoolsstage_pipeline (36, including the committed fixtures) pass.