Migrate ASAPQuery-backend's DAG data plane into ASAPQuery and run ASAPPlanner plans end to end #918

Status: agreed direction (Milind, Zeying), 2026-10-27. Open questions at the end.

Context

• ASAPPlanner turns PromQL/SQL/MetricsQL workloads into one PhysicalASAPDAG across all queries. It also owns asap-executor (crates/executor, ~28k LOC): the physical operator library, sketch kernels over asap_sketchlib, DAG compiler and runtime.
• ASAPQuery-backend was rebuilt in refactor!: plan with ASAPPlanner main's stage pipeline and drop the old execution stack ASAPQuery-backend#818 through 822 (Oct 18–21) as a thin (~4k LOC) deployment of that executor: plan with asap_planner::plan_installable → plan.json → data plane ingests Prometheus Remote Write, builds summary panes, answers planned instant PromQL queries (ingestion-time and query-time plans), forwards the rest to Prometheus. 98 tests pass in-process (mocked Prometheus); Gordon's sketch-bench harness (aqpbm-planeval) runs it against real Prometheus on one query. Gaps: range queries, SQL, multi-worker precompute, persistence, scale bugs (Query runs fail with "execution memory limit exceeded" above ~2M raw samples per window ASAPQuery-backend#848, A pane never closes when its series stops, so every query over that window falls back ASAPQuery-backend#772, Series interner grows without bound under series churn ASAPQuery-backend#842). Deleted-but-not-replaced features are tracked in Capabilities removed with the old backend that are not replaced yet (#818) ASAPQuery-backend#823.
• ASAPQuery (~90k LOC) works end to end on many workloads and has the surrounding scaffolding: Prometheus/ClickHouse/Elastic query adapters with fallback, range queries, Remote Write/VictoriaMetrics/OTel/JSON ingest, promql-compliance, benchmarks/, asap-tools (CloudLab). Its plan contract (streaming_config/inference_config from asap-planner-rs) is incompatible with ASAPPlanner's DAG.

Decision

Move ASAPQuery-backend's data plane into ASAPQuery, run it side by side with the legacy engine behind --plan plan.json, reach parity on live e2e tests, switch over, then delete the legacy engine. Sunset the ASAPQuery-backend repo. asap-executor stays in ASAPPlanner as a pinned git dependency.

Target architecture

Code lives in one new dag module in asap-query-engine (sds, storage, precompute, query). A plan subcommand wraps plan_installable (the backend's control plane). No new crates.

Plan

1. Generalize the PromQL harness (legacy only). Suite YAML gains optional accuracy/cadence fields and generates both legacy config and a PlanningWorkload. Execution header and scoreboard (below). Watermark endpoint added to legacy. External perf capture.
2. Copy the backend in behind --plan. Plain copy, commit cites backend SHA. Add the DAG arm to promql-compliance, benchmarks/ and experiment_run_e2e.py. Transfer open backend issues (gh issue transfer), archive the repo; no further changes land there.
3. sql-compliance/. Mirrors promql-compliance with ClickHouse as reference. The DAG arm refuses all SQL until step 5 — that is expected and visible.
4. PromQL parity. Refactor into the five components (SDS/storage split first), range queries, scale fixes (Query runs fail with "execution memory limit exceeded" above ~2M raw samples per window ASAPQuery-backend#848, A pane never closes when its series stops, so every query over that window falls back ASAPQuery-backend#772, Series interner grows without bound under series churn ASAPQuery-backend#842), multi-worker precompute — each change checked by the harness.
5. SQL on the DAG engine. Table rows from JSON ingest into precompute; ClickHouse adapter routes to the plan. Query-time SQL plans are refused with an "unsupported plan" error; no support added.
6. Switch over. Elastic moved to the DAG engine (see open questions), legacy engine, asap-planner-rs and old configs deleted.

Testing

Goal: harnesses that expose correctness and performance bugs, for the whole repo, PromQL and SQL, covering plan → install → live data plane on real data.

• Tiers. cargo test (fast, in-process; the backend's tests) · Docker compliance suites in CI (promql-compliance, new sql-compliance, benchmarks/) · CloudLab via asap-tools for scale.
• One workload definition. The suite YAML is the source of truth; both engines are planned from it.
• One engine per process. Legacy and DAG run as separate containers next to the reference (Prometheus or ClickHouse), so neither skews the other's perf and no answer silently falls back between engines.
• Who answered. Both engines set x-asap-execution: plan | forwarded | refused plus a reason; tests run with forwarding off. A forwarded answer never counts as a pass.
• Scoreboard. Per query and engine: answered-correct / answered-wrong / not answered (reason). Correct = within the plan's accuracy guarantee for that query, against the reference. Legacy-vs-DAG diff is a diagnostic only (they use different asap_sketchlib versions).
• Deterministic time. Seed data in the past, start the engine with --stream-start-ms = seed start, send the watermark barrier after seeding instead of sleeping. Off-grid evaluation times are reported as "not answered: off grid".
• Performance. Ingest throughput, query latency, CPU, RSS, measured from outside the process (harness timings, container stats). Reported as trends (JSON artifact per run); never blocks PRs.
• Parity = switch-over criterion. For every suite, the DAG engine answers at least what legacy answers, within guarantee. Perf is reported, not gated.

Decisions log

Open questions

1. Storage form. Live in-memory states first (current backend; fast, no persistence), with serialized bytes (ASAPv1) added alongside persistence? Proposed: yes.
2. Elastic. QueryDSL, Elastic SQL, or both? ASAPPlanner has no QueryDSL frontend; Elastic SQL could use the SQL frontend.
3. RQE route (/api/v1/rqe_query) and promsketch_store. Survive or drop at switch-over? Proposed: drop unless an active eval needs them.
4. Owners and dates for each step, and target dates for PromQL and SQL parity.

Follow-up issues to file (link here as sub-issues)

• Query → plan matching by pattern and capability, falling back from exact text (ASAPQuery).
• One issue per plan step above.

References

• ASAPPlanner: crates/executor/README.md, docs/design_docs/physical-planning-and-deployment.md
• ASAPQuery-backend: README; refactor!: plan with ASAPPlanner main's stage pipeline and drop the old execution stack ASAPQuery-backend#818 through 822 (rebuild), Capabilities removed with the old backend that are not replaced yet (#818) ASAPQuery-backend#823 (removed capabilities), Query runs fail with "execution memory limit exceeded" above ~2M raw samples per window ASAPQuery-backend#848, A pane never closes when its series stops, so every query over that window falls back ASAPQuery-backend#772, Series interner grows without bound under series churn ASAPQuery-backend#842
• ASAPQuery: promql-compliance/ARCHITECTURE.md, docs/729-native-query-dag-todos.md, asap-tools/experiments/experiment_run_e2e.py
• sketch-bench phase2-ver3: aqpbm-planeval/ (backend vs Prometheus harness)