Skip to content

fix: exact top-k answers keep the chosen series' labels - #794

Merged
zzylol merged 1 commit into
mainfrom
fix/topk-series-identity
Oct 21, 2026
Merged

zzylol merged 1 commit into
mainfrom
fix/topk-series-identity

Conversation

@zzylol

@zzylol zzylol commented Oct 21, 2026

Copy link
Copy Markdown
Contributor

Why

PromQL topk and bottomk return whole input series. topk by (G) (k, …) and its count and rate forms returned the right values but only the G labels. In the seed-862 workload-01 plan q06-topk-sum.json, the root projection keeps only region, service and topk_4, so a runtime cannot tell which series each answer is.

What

Exact top-k answers keep the chosen series' identity, so each answer carries that series' full label set. This holds for every topk form, grouped or not. bottomk and generic topk lower to a plain sort and limit with no projection, so they already kept it; the new test covers them as well.

How

The column is dropped in Stage 2, not in the PromQL frontend. The frontend lowers these queries to an Aggregate{[TopK]} whose logical schema is [keys.., topk_k]. Stage 2 implements exact top-k as Limit(Sort) and then adds a Project back to [keys.., topk_k], and that Project dropped the series identity. It now also keeps $promql_series_identity whenever the ranked rows carry it. The column goes after the logical columns, so a reader of [keys.., topk_k] finds them where they were. The executor reads an answer's labels from that identity.

Key code interfaces added / modified

None. Only the exact top-k root's output schema gains a trailing $promql_series_identity column, which is visible in plan files, for example planner-layering-example1.json.

Usage (the test topk_answers_carry_the_chosen_series_labels):

// topk by (region) (2, sum_over_time(data[1m])), exact, over five series:
// east: e1/api sum 6, e2/web 60, e3/db 30; west: w1/api 9, w2/web 3.
let plan_json = plan_json_for(workload, &DeploymentCapabilities::default()).await;
// The runtime feeds the minute's samples; the executor runs the plan.
let run = run_data_plane(&plan_json, &topk_samples(), EVALUATION_MS, &mut SummaryStore::new());
// Answers (labels from the series identity):
//   {instance=e2, region=east, service=web} 60   {instance=e3, region=east, service=db} 30
//   {instance=w1, region=west, service=api} 9    {instance=w2, region=west, service=web} 3
// Equal to the query-time reference's.

Before this PR

The same query answered {region=east} 60, {region=east} 30, {region=west} 9 and {region=west} 3. The values were right, but the series labels were lost.

After this PR

Each answer is the chosen input series with its value, as in Prometheus.

Verification

  • topk_answers_carry_the_chosen_series_labels (written first, failed on main). It runs the query above end to end through plan_installable and the executor and compares against the hand-computed answers and query_time_reference. On main it failed with left: {"region": "east"}, right: {"instance": "e2", "region": "east", "service": "web"}.
  • every_topk_form_answers_whole_series. It covers topk(2, sum_over_time), topk by (region) (1, count_over_time), topk by (region) (1, rate), bottomk by (region) (1, sum_over_time) and bottomk(3, sum_over_time). Each answer must carry its instance and service, and the answers must equal the reference.
  • Updated for the new column. The committed stage-pipeline fixture tools/dag-viewer/examples/planner-layering-example1.json was regenerated with stage_pipeline --example planner-layering-1. The shape test stage2_physical_asap_optimization_exact_topk_returns_job_and_topk_10 now expects the trailing identity column.
  • Workspace. cargo test --workspace --locked --no-fail-fast with the fixture already regenerated: 149 test binaries passed, and the one failure was that shape test. After updating it, planner_layering_example1 passes (29 passed). Under the current speed-first policy, fmt and clippy were not run.
  • Not covered. The sketch-based heavy-hitter top-k path is not covered: its SummaryEstimate{TopK} output is not executable yet (fix(ir): top-k sketch readouts return selected rows; Example 1 expectations follow planner output #579).
  • Review. Not reviewed independently.

PromQL topk and bottomk answer whole input series. Stage 2's exact top-k
implementation projected the limited rows to the logical [keys.., topk_k]
columns and dropped the series identity, so `topk by (G) (k, ...)` (and
its count and rate forms) answered with the G labels only. The Project
now keeps $promql_series_identity after the logical columns when the
ranked rows carry it: each answer has the chosen series' full label set,
and readers of [keys.., topk_k] find those columns where they were.
Tests: topk by (region) (2, sum_over_time(data[1m])) over five series
answers the four chosen series with their full labels and hand-computed
sums, equal to the query-time reference; every topk/bottomk form,
grouped or not, over sums, counts and rates answers whole series. The
example-1 fixture and its exact top-k shape test now include the column.
@zzylol zzylol added bug Something isn't working promql PromQL front-end lowering (L1→L2) labels Oct 21, 2026
@zzylol
zzylol merged commit 4393316 into main Oct 21, 2026
2 of 3 checks passed
@zzylol
zzylol deleted the fix/topk-series-identity branch November 3, 2026 02:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working promql PromQL front-end lowering (L1→L2)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant