[zzylol]

Why

#1095 §6.1 left two failures on the prototype (#1144): workload-01's largest pane-width round ends at a 70.9 % gap with the incumbent equal to the warm start, and rangeheavy-t8-t5's largest round (114,225 variables) has no dual bound after a 45 s build. §5 lists the steps to try in order. This PR takes the first one (3a, connected components) and measures where the LP relaxation loses its bound, so the next step (shared input contexts, 3b or 3c) is chosen from data.

What

• 3a, independent parts. The selection MILP is split into the connected components of the graph that joins the variables of each row. Each part is solved alone, with its share of the deadline, and the joined solution goes through the same single completion, decode, realization and plan_cost = objective check. Queries end up in one part when they may share a shareable candidate, a run of I, a source's raw retention, or anything else a row couples.
• LP diagnosis (devtools only). A new lp-diagnosis cargo feature (off in production builds; asap-devtools forwards it). With ASAP_STAGE3_LP_DIAGNOSIS=<path>, each solve appends: the pure LP bound, the root-node bound, the LP with each row family dropped, the LP with cover rows added, the objective split by variable family (at the LP, the warm start and the incumbent), and the p/x runs with the largest LP-to-incumbent difference. The MILP's deadline is pushed back by the diagnosis's time. (The diagnosis numbers below come from runs made before that change, with a 3,000 s budget.)
• Tighter big-M bounds. Ē[m] in V ≥ E − Ē (x[m] − z) is now the sum over the context's slots of its costliest run, not a sum over every candidate. N̄_q in the accuracy row is now the number of (context, slot) places, not the number of runs. The other big-M constants the task listed were already per reader in the prototype: H[m] uses each reader's L̄[q], W uses H̄[m] over m's readers, and H_query uses L̄[q] = min(bound, most).

How

independent_parts runs a union-find over columns, joining the columns of each row. A component is keyed by the first query whose output-reference choice falls in it. Components without a query are merged into one last part. solve_selection_milp renames each part's variables into its own ProblemVariables, rebuilds its rows and objective terms, and solves the parts smallest first. Each part gets (time left) × (its variables / variables not yet solved), so time a small part leaves goes to the larger ones. Each part has its own objective scale from its share of the warm start. The status is Optimal only if every part is optimal. The dual bound is the sum of the parts' bounds. With a single part, the original model is solved unchanged. The parts run one after another (HiGHS uses 4 threads per solve).

Key code interfaces added / modified

crates/plan-selection/src/milp/model.rs

/// How the MILP is built (stays).
pub(crate) struct SelectionMilpOptions {
    /// Write the accuracy rows (row 8). // unchanged
    pub(crate) enforce_accuracy: bool,
    /// Solve each independent part alone; `false` only to compare with one solve in tests. // new
    pub(crate) split_into_parts: bool,
}

/// One independent part: a connected component of the row graph, or all components without a query together. // new
pub(crate) struct IndependentPart {
    /// The queries whose output references it covers, ascending. // new
    pub(crate) queries: Vec<usize>,
    /// Its columns (variables in creation order), ascending. // new
    pub(crate) columns: Vec<usize>,
    /// Its rows, ascending. // new
    pub(crate) rows: Vec<usize>,
}

/// The independent parts of a built model, in order of their first query (the part without a query last). // new
pub(crate) fn independent_parts(milp: &SelectionMilp<'_>) -> Vec<IndependentPart>;

/// How one part's solve went, written to the `ASAP_STAGE3_REPORT` line as `parts`. // new
pub(crate) struct PartSolve {
    /// Queries in the part. // new
    pub(crate) queries: usize,
    /// Its variables, binaries and rows. // new
    pub(crate) variables: usize,
    pub(crate) binaries: usize,
    pub(crate) rows: usize,
    /// `Optimal` or `TimeLimit`. // new
    pub(crate) status: String,
    pub(crate) solve_ms: u64,
    /// HiGHS's dual bound on the part's objective, cost per second; for a part without binaries (an LP), its optimal objective. // new
    pub(crate) dual_bound_per_second: f64,
}

/// Tests only: whether this thread builds the big-M bounds as before this PR (always `false` outside tests). // new
pub(crate) fn tests_loose_big_m() -> bool;

/// The raw solution (stays; other fields unchanged).
pub(crate) struct SelectionMilpSolution {
    /// Per independent part, how its solve went. // new
    pub(crate) parts: Vec<PartSolve>,
    /* status, cost_per_second, request_cost_per_second, cost_lower_bound_per_second (now the sum over parts),
       relative_gap (now (Σ objective − Σ dual) / Σ objective), chosen, selected, slot_reads, size,
       build_ms, warm_start_ms, warm_start_cost_per_second, solve_ms: unchanged */
}

crates/plan-selection/Cargo.toml, crates/devtools/Cargo.toml, crates/plan-selection/src/milp/lp_diagnosis.rs

# asap-plan-selection: measurement only; production builds leave it out. // new
lp-diagnosis = []
# asap-devtools // new
lp-diagnosis = ["asap-plan-selection/lp-diagnosis"]

Usage (devtool run, as in §6.1):

# Build stage_pipeline; add --features lp-diagnosis only for the LP diagnosis.
cargo build --release -p asap-devtools --bin stage_pipeline --features lp-diagnosis
# Stage 3 on the new search, one JSON line per solve in w01.report (now with `parts`),
# and one LP-diagnosis line per solve in w01.lp.jsonl. The diagnosis pushes each MILP's deadline
# back by its own time, but a diagnosed run's wall time and later rounds are not comparable
# with an undiagnosed run: measure timings without the feature.
ASAP_STAGE3_SEARCH=milp ASAP_STAGE3_REPORT=w01.report ASAP_STAGE3_LP_DIAGNOSIS=w01.lp.jsonl \
  target/release/stage_pipeline --workload tools/dag-viewer/workloads/random-mixed-seed-862/workload-01.json \
  --cost-model aws-fargate-us-east-1 --enforce-accuracy-guarantees --milp-time-limit-s 300 --no-memo \
  --deployment-capabilities '{"raw_data_retained": true, "query_time_results_retained": false}' --out w01.json
# w01.report, largest round: "parts":[{"queries":7,"variables":37814,"status":"TimeLimit",...},
#                                     {"queries":2,"variables":2093,"status":"Optimal",...}]
# w01.json: the selected plan, cost 1.4171e-6 per second, plan_cost = objective.

Major algorithm changes

3a, independent parts.

• Problem: the whole model went to HiGHS in one solve, even when query sets share nothing. On workload-01, 2 of its 9 queries share no variable with the other 7.
• Before: build → warm start → one HiGHS solve → completion → decode → realize → plan_cost check.
• After: build → warm start → independent_parts → one HiGHS solve per part (smallest first, each with a share of the deadline proportional to its variables) → join the columns → one completion → decode → realize → plan_cost check.
• Correctness: parts share no variable and no row, and the objective is a sum over variables. So the joined optimum of the parts is an optimum of the whole model, and the single plan_cost = objective check is unchanged. The test independent_query_sets_solve_apart_to_the_same_plan uses two queries over m and one over n. It checks that the split gives parts [[0, 1], [2]], that every column and row lands in exactly one part, and that solving apart and solving whole give the same DAG and the same cost.
• Complexity: union-find is O(nonzeros · α). Renaming copies the rows once. Peak RSS on workload-01 rises 859 → 899 MB.
• Limits: the parts run one after another, not in parallel. On these workloads only workload-01 splits (the others read one metric). Each part gets only its proportional slice of the time left. Without a warm start, a small but hard part can reach its slice with no integral plan, and that turns the whole solve into NoSolution, even when the other parts finished. With a warm start (every measured run), each part keeps the start's plan at worst.

Big-M bounds. Both new bounds are still valid upper bounds, so the objective still equals plan_cost at Optimal (every exactness test in milp::model::tests passes).

• Where they bite: I counted where the new bound differs from the old one on the measured workloads, using a temporary debug build that is not in the branch. Ē[m] is smaller in 229 shared contexts of workload-01, by 12–16 %. It equals the old bound on every other workload. N̄_q equals the old bound on every measured workload, and on every fixture I tried: no query there may depend on two approximate runs of one slot in one context.
• Effect: the LP value is unchanged on every diagnosed round (table below).
• Test: a_tighter_context_execution_bound_keeps_the_plan uses a rollup read every 20 s and every 30 s. It checks that the Ē coefficients are at most the old ones and strictly smaller in some row, and that the plan and cost equal those of the model built with the old bounds. The old bounds come from a test-only thread-local switch, tests_loose_big_m.

Before this PR

§6.1: workload-01's largest round ends at a 70.9 % gap with the incumbent equal to the warm start (3.5429e-6). The only explanation on record was "weak LP bound".

After this PR

Re-measured with the §6.1 settings (budget 300 s, accuracy enforced, aws-fargate-us-east-1, raw data retained, workload-01 under ulimit -v 8388608). Rows show the selected round, and the largest round where it differs. "Parts" is the number of independent parts (queries in each). "Build" is the model build without the warm start. "Root bound" is HiGHS's bound after its root node, from a separate diagnosis run (pure LP in brackets).

Workload	Vars / binaries	Parts	Build	Solve	Status, gap	Root bound (pure LP)	Cost per second (§6.1)	plan_cost = objective	Peak RSS (§6.1 rerun)
example1	501 / 323	1 (2 q)	0.08 s	0.03 s	optimal	4.3770 (3.27)	4.3770 (4.3770)	yes	54 MB (54)
rangeheavy-t5-q05-q09	6,982 / 3,908	1 (2 q)	0.16 s	1.69 s	optimal	3.9901e-6 (2.16e-6)	3.9901e-6 (3.9901e-6)	yes	119 MB (114)
rangeheavy-t7-t4	4,490 / 2,627	1 (2 q)	0.03 s	2.51 s	optimal	2.1493e-7 (1.67e-7)	2.1493e-7 (2.1493e-7)	yes	105 MB (102)
rangeheavy-t8-t5	4,013 / 1,749	1 (3 q)	0.03 s	1.49 s	optimal	6.4822e-6 (4.77e-7)	6.4822e-6 (6.4822e-6)	yes	298 MB (291)
largest round	114,225 / 50,601	1 (3 q)	43.9 s	0.53 s	deadline, no dual bound (as §6.1)	54,449-var round: 6.4822e-6 in 99 s (3.67e-7)	incumbent 2.0661e-5
ratio-of-quantiles	52 / 29	1 (1 q)	0.00 s	0.05 s	optimal	2.4071e-6 (3.04e-7)	2.4071e-6 (2.4071e-6)	yes	35 MB (34)
rollupheavy-t11	8,788 / 4,990	1 (2 q)	0.17 s	3.66 s	optimal	1.4159e-6 (2.52e-7)	1.4159e-6 (1.4159e-6)	yes	152 MB (144)
workload-01	3,856 / 1,281	2 (7 q + 2 q)	0.03 s	4.83 s	deadline, 4.9 % (was 7.1 %)	1.3870e-6 (9.38e-8)	1.4171e-6 (1.4171e-6)	≤ (equal)	899 MB (859)
largest round	39,907 / 13,147	2 (7 q + 2 q)	0.27 s	201 s	deadline, 4.1 % (was 70.9 %); incumbent 1.4171e-6 (was the warm start 3.5429e-6), dual 1.3586e-6 (was 1.0319e-6)	1.0157e-6 in 163 s (2.67e-8); with a 3,000 s budget this round solves to optimal 1.4171e-6 in 221 s

All costs at Optimal equal §6.1 (and a same-host rerun of the base commit). No plan changed. On workload-01's largest round, the 2-query part solves to optimality in 0.3 s, and the 7-query part finds the 1.4171e-6 plan within its ~200 s. I have not separated how much of that comes from the split and how much from HiGHS's search path; the tightening leaves the LP unchanged. The t8-t5 largest round is still limited by its 44–52 s build, repeated in four rounds of 114k variables. 63 % of that model's variables (71,676) sit in shareable contexts.

LP gap attribution

From ASAP_STAGE3_LP_DIAGNOSIS on the base model (before the big-M change; after it, every LP value is identical). Cost per second. "Gap share" splits (warm start − LP) by the family of the charged variable. The warm start is the largest round's incumbent in §6.1.

	workload-01, round 1 (3,856 vars)	workload-01, largest round (39,907 vars)	t8-t5, round 1 (4,013)	t8-t5, 54,449 vars
pure LP	9.38e-8 (6.6 % of best)	2.67e-8 (1.9 %)	4.77e-7 (7.4 %)	3.67e-7 (5.7 %)
LP + cover rows Σ terms ≥ run on configuration / window terms	2.04e-7	1.89e-7	4.77e-7 (no such runs)	3.67e-7
root node (HiGHS presolve + cuts)	1.387e-6 (97.9 %)	1.016e-6 (71.7 %, 163 s)	6.482e-6 (100 %, optimal at root)	6.482e-6 (100 %, 99 s)
warm start	3.543e-6	3.543e-6	6.482e-6	2.066e-5
best known	1.417e-6	1.417e-6 (proven optimal)	6.482e-6	6.482e-6
gap share: p (query-context and I work)	90.3 %	89.6 %	0.8 %	0.5 %
gap share: x (standing cost of shareable candidates)	3.5 %	4.1 %	99.2 %	99.6 %
gap share: configuration ANDs (t_producers)	6.5 %	6.4 %	0	0
gap share: H_query, H, W, V, accuracy, latency, π, raw retention	0.6 % / 0 / 0 / 0 / 0 / 0 / 0 / 0	0.5 % / 0 / 0 / 0 / 0 / 0 / 0 / 0	≈ 0	≈ 0
gap share: z (schedule groups)	−0.9 %	−0.4 %	0	0
LP with the family dropped (share of the LP kept): held time H/W/H_query, V, 7 latency, 8 accuracy, ANDs	97–100 %	99.6–100 %	100 % (accuracy: 83 %)
… 3 validity / 1 coverage / 6 reach from below / z / 9 reader constraints	0.5 % / 0 / 78 % / 25 % / 25 %	1.8 % / 0 / 60 % / 37 % / 37 %	0 / 0 / 83 % / – / 100 %

Mechanism. The pure LP avoids the expensive bottom of each plan: the raw scans on workload-01 (p[q2] c0 Scan, 1.01e-6 each at 1, 0 in the LP), and the ingestion-time pane chain on t8-t5 (x c109 PaneChain, 6.39e-6 at 1, 0 in the LP). It does this by spreading each query's top over many alternative paths (on t8-t5 round 1, about 60 query-time MatAtQuery results at 0.017 each), all reading one shared bottom (a MatAtQuery scan at x = 0.017). Row 3 asks x[m] ≥ p[κ, m] per reader, so the shared bottom pays only the largest single read (1/k), not the query's full demand of 1. The ANDs of reach (a ≥ a + p − 1) are 0 at these fractions. None of the big-M families matters: the warm start charges them almost nothing, and dropping them leaves the LP where it is. Step 3's tightening is therefore correct but LP-neutral here. HiGHS's root cuts recover 98–100 % of the best known cost on every diagnosed round except workload-01's largest (72 %), and given about 220 s HiGHS closes that round to proven optimality (1.4171e-6, the same cost as the selected plan).

Recommendation for the next step. The measured failures are size and search time more than the root bound. workload-01's largest round now ends at 4.1 % in 201 s and proves optimality at 221 s.

1. Shared input contexts (§8) next. t8-t5's largest round spends 44–52 s per build, four times, on a model with 63 % of its variables in shareable contexts, and its root takes 99 s already at 54k variables. Cutting the copied input contexts attacks both.
2. 3c (per-query columns) only if the bound becomes the limit on larger workloads. It is the structural fix for the mechanism above, because a column carries its query's full demand into every shareable candidate it reads. Today HiGHS's cuts make up for it.
3. 3b is not indicated. The gap is not in private subtrees; it is in how shared bottoms are paid. The cover rows (2× the pure LP on workload-01, still 14 % of best) are not worth shipping on their own.

Verification

• cargo test -p asap-plan-selection --lib: 314 passed, and 314 passed with --features lp-diagnosis. This includes the new independent_query_sets_solve_apart_to_the_same_plan and a_tighter_context_execution_bound_keeps_the_plan.
• cargo test -p asap-devtools --test devtools stage_pipeline: 51 passed.
• cargo fmt --all -- --check: clean. cargo clippy -p asap-plan-selection --all-targets -- -D warnings: clean, with and without --features lp-diagnosis.
• Measurements: the stage_pipeline runs above, base commit rerun on the same host (it reproduces §6.1 exactly), then this branch; the LP diagnosis comes from separate runs with --features lp-diagnosis and a 3,000 s budget, so their solve times are not comparable.

🤖 Generated with Claude Code