Skip to content

feat: replace legacy planning with joint candidate MILP search - #934

Open
zzylol wants to merge 49 commits into
mainfrom
feat/joint-execution-demand
Open

zzylol wants to merge 49 commits into
mainfrom
feat/joint-execution-demand

Conversation

@zzylol

@zzylol zzylol commented Oct 24, 2026 •

Copy link
Copy Markdown
Contributor

Before this PR

Separate workload enumeration and selection paths repeated equivalent work and represented sharing, materialization, and accuracy decisions inconsistently.

After this PR

One candidate-connection MILP selects a physical execution DAG. P95/P99 retain distinct results while sharing a compatible summary, or selecting separate configurations when appropriate. Legacy selectors and the global Cartesian workload enumerator are removed; modules are organized by candidate generation, costing, MILP, refinement, and pipeline scope.

Supported sharing rules, pane widths, materializations, deployment objectives, recurring execution demand, and retention use the same route. Budget stops retain only validated plans. Internal failures do not silently become fallback results.

Key code interfaces added / modified

  • Logical memo, shared-state rules, candidate dependencies/configurations, and DAG reconstruction.
  • The sole CandidateConnections route and phase-level reports, including no-incumbent budget stops.
  • Deadline-aware physical DAG export and executor compilation/installation.
  • Accuracy binding with data-domain conditions and composed DDS seed sizing.
  • Viewer reporting and the frozen-binary workload acceptance runner.

Major algorithm changes

  • Joint activation and producer connections with selected-reader legality and acyclicity.
  • Complete supported integer-millisecond pane-width domain, bounded seed/full search, safe pruning, component separation, and warm starts.
  • Per-node cost ownership, shared recurring execution demand, maximum retention, checked arithmetic, and certified bounds distinct from proposal heuristics.
  • Numerically stable cost scaling and explicit invalid-certificate reporting.
  • Linear edge-multiplicity validation, sorted disjoint pane-coverage union, and cooperative cancellation through export/compilation.

Validation

Complete CI passed on 761a019a:

  • Workspace: 2,175 passed, zero failed, 30 existing ignored.
  • PromQL-only: 64 passed, zero failed, two existing ignored.
  • Viewer Python: 102/102 passed with the real CLI, zero skips; JavaScript: 14/14 passed.
  • Formatting, strict Clippy, MetricsQL external-consumer check, and documented upstream baseline verification passed.
  • Final frozen planner binary: 30/30 ASAPQuery workloads (270 query outputs) passed enforced accuracy contracts, cost-certificate checks, and installation-metadata checks. Separate viewer workload examples: 9/9 passed.

Planning wall time was median 32.37 s, maximum 39.64 s, with peak planner RSS 2.57 GiB on a shared debug-build host running two workloads concurrently. The requested 30-second budget is cooperative; individual calls and cleanup can overrun it. None of these bounded runs proved optimality. Execution costs and accuracy guarantees are evaluator results, not measured stream-execution speedups or empirical error measurements.

Workload evidence and reproduction includes portable phase timings, costs, model sizes, solver status, and binary identity. Explicit-rule tests preserve accuracy, cost, boundary, sharing, and runtime assertions without depending on a timed search archive's visitation order.

zzylol added 25 commits October 24, 2026 03:52
@zzylol zzylol changed the title feat: add execution-aware candidate selection and cost refinement feat: replace legacy planning with joint candidate MILP search Oct 24, 2026
zzylol added 20 commits October 24, 2026 16:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant