Skip to content

Release performance regression on main #1289

Closed

Description

@github-actions

Release performance regressed at f7c1e76 on main.

Comparison run: https://github.com/sqlstack-io/sqlbuild/actions/runs/41324417425

Release performance: 0.186.2 (candidate) vs 0.186.2 (baseline)

Same runner: AMD EPYC 9V45 96-Core Processor, 4 CPUs; load average 3.95/3.58/2.08 before and 1.50/2.07/2.04 after. 5 interleaved runs per version; wall and CPU are medians, peak RSS is the worst run.

Benchmark projects are generated per side: the baseline runs projects from its own generator (v0.186.2), the candidate from the candidate's.

Limits: wall +25% (ignored under 0.5 s), CPU +25% (ignored under 0.5 s), peak RSS +25% (ignored under 32 MiB). Cold, warm and one-model edit compiles use +15% for wall and CPU.

Command Wall (s) Wall ratio CPU (s) CPU ratio Peak RSS (MiB) RSS ratio Result
compile (no cache) 3.80 → 5.30 1.395x 8.31 → 11.88 1.430x 923 → 1010 1.094x ❌ regressed
compile (warm cache) 0.18 → 0.19 1.056x 0.12 → 0.11 0.917x 71 → 71 1.005x ✅ ok
plan --json 5.49 → 5.55 1.011x 7.27 → 7.33 1.008x 1004 → 1023 1.019x ✅ ok
lineage hub downstream 0.19 → 0.17 0.895x 0.26 → 0.25 0.962x 157 → 162 1.032x ✅ ok
lineage rollup upstream 0.19 → 0.17 0.895x 0.26 → 0.25 0.962x 156 → 171 1.091x ✅ ok
lineage column trace 1.87 → 1.88 1.005x 2.57 → 2.62 1.019x 920 → 916 0.997x ✅ ok
dag --json 2.52 → 2.51 0.996x 3.21 → 3.17 0.988x 972 → 976 1.003x ✅ ok
scope --json 0.19 → 0.19 1.000x 0.18 → 0.17 0.944x 118 → 124 1.048x ✅ ok
compile (one-model edit) 0.81 → 0.83 1.025x 0.69 → 0.70 1.014x 359 → 358 0.997x ✅ ok
dense compile (warm cache) 0.31 → 0.30 0.968x 0.18 → 0.19 1.056x 94 → 94 1.001x ✅ ok
dense compile (one-model edit) 1.25 → 1.20 0.960x 1.06 → 1.05 0.991x 434 → 440 1.014x ✅ ok
dense compile (no cache) 6.45 → 6.49 1.006x 15.87 → 15.77 0.994x 1368 → 1418 1.036x ✅ ok
build (empty warehouse) 30.14 → 30.14 1.000x 30.25 → 30.24 1.000x 1732 → 1746 1.008x ✅ ok

Result: failed. compile (no cache): wall 1.395x exceeds 1.15x; compile (no cache): CPU 1.430x exceeds 1.15x.

Activity

  1. github-actions commented on Nov 1, 2026

    @github-actions
    ContributorAuthor

    Release performance regressed at 45f102e on main.

    Comparison run: https://github.com/sqlstack-io/sqlbuild/actions/runs/41327968958

    Release performance: 0.186.2 (candidate) vs 0.186.2 (baseline)

    Same runner: AMD EPYC 7763 64-Core Processor, 4 CPUs; load average 3.69/3.79/2.67 before and 1.24/1.62/2.14 after. 5 interleaved runs per version; wall and CPU are medians, peak RSS is the worst run.

    Benchmark projects are generated per side: the baseline runs projects from its own generator (v0.186.2), the candidate from the candidate's.

    Limits: wall +25% (ignored under 0.5 s), CPU +25% (ignored under 0.5 s), peak RSS +25% (ignored under 32 MiB). Cold, warm and one-model edit compiles use +15% for wall and CPU.

    Command Wall (s) Wall ratio CPU (s) CPU ratio Peak RSS (MiB) RSS ratio Result
    compile (no cache) 6.28 → 8.58 1.366x 15.12 → 21.66 1.433x 918 → 1005 1.095x ❌ regressed
    compile (warm cache) 0.17 → 0.17 1.000x 0.16 → 0.16 1.000x 71 → 71 0.999x ✅ ok
    plan --json 8.60 → 8.74 1.016x 12.11 → 12.20 1.007x 1001 → 1015 1.014x ✅ ok
    lineage hub downstream 0.25 → 0.25 1.000x 0.37 → 0.37 1.000x 155 → 162 1.044x ✅ ok
    lineage rollup upstream 0.25 → 0.25 1.000x 0.37 → 0.37 1.000x 156 → 166 1.067x ✅ ok
    lineage column trace 2.90 → 2.90 1.000x 4.27 → 4.28 1.002x 912 → 919 1.008x ✅ ok
    dag --json 3.65 → 3.64 0.997x 5.20 → 5.16 0.992x 985 → 998 1.014x ✅ ok
    scope --json 0.27 → 0.27 1.000x 0.26 → 0.26 1.000x 111 → 120 1.081x ✅ ok
    compile (one-model edit) 0.91 → 0.93 1.022x 0.99 → 1.01 1.020x 365 → 370 1.014x ✅ ok
    dense compile (warm cache) 0.26 → 0.26 1.000x 0.26 → 0.25 0.962x 94 → 95 1.002x ✅ ok
    dense compile (one-model edit) 1.30 → 1.30 1.000x 1.56 → 1.52 0.974x 438 → 429 0.979x ✅ ok
    dense compile (no cache) 10.28 → 10.31 1.003x 28.02 → 27.92 0.996x 1371 → 1383 1.009x ✅ ok
    build (empty warehouse) 50.75 → 50.58 0.997x 53.06 → 52.86 0.996x 1729 → 1735 1.003x ✅ ok

    Result: failed. compile (no cache): wall 1.366x exceeds 1.15x; compile (no cache): CPU 1.433x exceeds 1.15x.

  2. github-actions commented on Nov 1, 2026

    @github-actions
    ContributorAuthor

    Release performance regressed at 69bf510 on main.

    Comparison run: https://github.com/sqlstack-io/sqlbuild/actions/runs/41328954763

    Release performance: 0.186.2 (candidate) vs 0.186.2 (baseline)

    Same runner: AMD EPYC 9V74 80-Core Processor, 4 CPUs; load average 4.10/3.86/2.65 before and 1.13/1.52/2.05 after. 5 interleaved runs per version; wall and CPU are medians, peak RSS is the worst run.

    Benchmark projects are generated per side: the baseline runs projects from its own generator (v0.186.2), the candidate from the candidate's.

    Limits: wall +25% (ignored under 0.5 s), CPU +25% (ignored under 0.5 s), peak RSS +25% (ignored under 32 MiB). Cold, warm and one-model edit compiles use +15% for wall and CPU.

    Command Wall (s) Wall ratio CPU (s) CPU ratio Peak RSS (MiB) RSS ratio Result
    compile (no cache) 6.40 → 8.58 1.341x 14.97 → 21.18 1.415x 918 → 1012 1.103x ❌ regressed
    compile (warm cache) 0.17 → 0.17 1.000x 0.17 → 0.17 1.000x 70 → 71 1.004x ✅ ok
    plan --json 8.79 → 8.89 1.011x 12.13 → 12.17 1.003x 1008 → 1015 1.006x ✅ ok
    lineage hub downstream 0.26 → 0.26 1.000x 0.38 → 0.38 1.000x 155 → 163 1.051x ✅ ok
    lineage rollup upstream 0.26 → 0.26 1.000x 0.38 → 0.38 1.000x 162 → 162 1.000x ✅ ok
    lineage column trace 2.97 → 3.00 1.010x 4.28 → 4.32 1.009x 920 → 912 0.992x ✅ ok
    dag --json 3.76 → 3.78 1.005x 5.25 → 5.25 1.000x 977 → 977 1.000x ✅ ok
    scope --json 0.30 → 0.30 1.000x 0.28 → 0.28 1.000x 109 → 121 1.105x ✅ ok
    compile (one-model edit) 0.94 → 0.97 1.032x 1.04 → 1.04 1.000x 366 → 385 1.053x ✅ ok
    dense compile (warm cache) 0.26 → 0.26 1.000x 0.27 → 0.26 0.963x 94 → 94 0.999x ✅ ok
    dense compile (one-model edit) 1.36 → 1.34 0.985x 1.59 → 1.56 0.981x 462 → 444 0.961x ✅ ok
    dense compile (no cache) 10.31 → 10.30 0.999x 27.49 → 27.47 0.999x 1345 → 1406 1.045x ✅ ok
    build (empty warehouse) 49.45 → 49.36 0.998x 51.84 → 51.88 1.001x 1735 → 1737 1.001x ✅ ok

    Result: failed. compile (no cache): wall 1.341x exceeds 1.15x; compile (no cache): CPU 1.415x exceeds 1.15x.

  3. github-actions commented on Nov 1, 2026

    @github-actions
    ContributorAuthor

    Release performance regressed at b66b83f on main.

    Comparison run: https://github.com/sqlstack-io/sqlbuild/actions/runs/41338371912

    Release performance: 0.186.2 (candidate) vs 0.186.2 (baseline)

    Same runner: INTEL(R) XEON(R) PLATINUM 8573C, 4 CPUs; load average 4.71/4.14/2.84 before and 1.34/1.57/2.07 after. 5 interleaved runs per version; wall and CPU are medians, peak RSS is the worst run.

    Benchmark projects are generated per side: the baseline runs projects from its own generator (v0.186.2), the candidate from the candidate's.

    Limits: wall +25% (ignored under 0.5 s), CPU +25% (ignored under 0.5 s), peak RSS +25% (ignored under 32 MiB). Cold, warm and one-model edit compiles use +15% for wall and CPU.

    Command Wall (s) Wall ratio CPU (s) CPU ratio Peak RSS (MiB) RSS ratio Result
    compile (no cache) 5.76 → 7.44 1.292x 13.54 → 18.62 1.375x 911 → 981 1.077x ❌ regressed
    compile (warm cache) 0.22 → 0.22 1.000x 0.13 → 0.13 1.000x 71 → 71 1.001x ✅ ok
    plan --json 8.19 → 8.29 1.012x 11.08 → 11.17 1.008x 1059 → 1017 0.960x ✅ ok
    lineage hub downstream 0.25 → 0.25 1.000x 0.35 → 0.35 1.000x 155 → 166 1.071x ✅ ok
    lineage rollup upstream 0.25 → 0.26 1.040x 0.36 → 0.36 1.000x 154 → 171 1.108x ✅ ok
    lineage column trace 2.70 → 2.67 0.989x 3.92 → 3.91 0.997x 898 → 918 1.022x ✅ ok
    dag --json 3.51 → 3.52 1.003x 4.74 → 4.73 0.998x 946 → 949 1.004x ✅ ok
    scope --json 0.26 → 0.25 0.962x 0.25 → 0.25 1.000x 108 → 120 1.115x ✅ ok
    compile (one-model edit) 1.04 → 1.08 1.038x 0.95 → 0.95 1.000x 362 → 384 1.061x ✅ ok
    dense compile (warm cache) 0.31 → 0.28 0.903x 0.19 → 0.19 1.000x 94 → 94 1.000x ✅ ok
    dense compile (one-model edit) 1.48 → 1.46 0.986x 1.41 → 1.39 0.986x 420 → 444 1.058x ✅ ok
    dense compile (no cache) 8.92 → 8.86 0.993x 23.52 → 23.97 1.019x 1344 → 1404 1.044x ✅ ok
    build (empty warehouse) 55.25 → 55.15 0.998x 55.13 → 55.28 1.003x 1736 → 1735 0.999x ✅ ok

    Result: failed. compile (no cache): wall 1.292x exceeds 1.15x; compile (no cache): CPU 1.375x exceeds 1.15x.

  4. github-actions commented on Nov 2, 2026

    @github-actions
    ContributorAuthor

    Release performance regressed at 3b77897 on main.

    Comparison run: https://github.com/sqlstack-io/sqlbuild/actions/runs/41375594941

    Release performance: 0.186.2 (candidate) vs 0.186.2 (baseline)

    Same runner: AMD EPYC 9V74 80-Core Processor, 4 CPUs; load average 4.33/4.01/2.69 before and 1.28/1.57/2.09 after. 5 interleaved runs per version; wall and CPU are medians, peak RSS is the worst run.

    Benchmark projects are generated per side: the baseline runs projects from its own generator (v0.186.2), the candidate from the candidate's.

    Limits: wall +25% (ignored under 0.5 s), CPU +25% (ignored under 0.5 s), peak RSS +25% (ignored under 32 MiB). Cold, warm and one-model edit compiles use +15% for wall and CPU.

    Command Wall (s) Wall ratio CPU (s) CPU ratio Peak RSS (MiB) RSS ratio Result
    compile (no cache) 6.60 → 8.93 1.353x 15.31 → 21.52 1.406x 909 → 961 1.057x ❌ regressed
    compile (warm cache) 0.24 → 0.23 0.958x 0.17 → 0.17 1.000x 70 → 71 1.003x ✅ ok
    plan --json 9.26 → 9.58 1.035x 12.48 → 12.87 1.031x 1012 → 1020 1.008x ✅ ok
    lineage hub downstream 0.28 → 0.28 1.000x 0.40 → 0.40 1.000x 154 → 161 1.046x ✅ ok
    lineage rollup upstream 0.28 → 0.26 0.929x 0.39 → 0.39 1.000x 153 → 165 1.081x ✅ ok
    lineage column trace 3.14 → 3.19 1.016x 4.45 → 4.44 0.998x 902 → 920 1.020x ✅ ok
    dag --json 4.03 → 4.07 1.010x 5.34 → 5.28 0.989x 942 → 956 1.015x ✅ ok
    scope --json 0.30 → 0.28 0.933x 0.27 → 0.26 0.963x 109 → 120 1.103x ✅ ok
    compile (one-model edit) 1.11 → 1.12 1.009x 1.01 → 1.02 1.010x 369 → 374 1.014x ✅ ok
    dense compile (warm cache) 0.46 → 0.50 1.087x 0.26 → 0.26 1.000x 94 → 94 0.999x ✅ ok
    dense compile (one-model edit) 1.63 → 1.57 0.963x 1.51 → 1.50 0.993x 422 → 423 1.004x ✅ ok
    dense compile (no cache) 9.91 → 9.68 0.977x 26.22 → 26.07 0.994x 1342 → 1400 1.043x ✅ ok
    build (empty warehouse) 66.80 → 63.20 0.946x 52.74 → 52.69 0.999x 1735 → 1734 0.999x ✅ ok

    Result: failed. compile (no cache): wall 1.353x exceeds 1.15x; compile (no cache): CPU 1.406x exceeds 1.15x.

  5. github-actions commented on Nov 2, 2026

    @github-actions
    ContributorAuthor

    Release performance regressed at 50382ae on main.

    Comparison run: https://github.com/sqlstack-io/sqlbuild/actions/runs/41378766538

    Release performance: 0.187.0 (candidate) vs 0.186.2 (baseline)

    Same runner: AMD EPYC 9V45 96-Core Processor, 4 CPUs; load average 3.94/3.49/2.03 before and 1.23/1.74/1.86 after. 5 interleaved runs per version; wall and CPU are medians, peak RSS is the worst run.

    Benchmark projects are generated per side: the baseline runs projects from its own generator (v0.186.2), the candidate from the candidate's.

    Limits: wall +25% (ignored under 0.5 s), CPU +25% (ignored under 0.5 s), peak RSS +25% (ignored under 32 MiB). Cold, warm and one-model edit compiles use +15% for wall and CPU.

    Command Wall (s) Wall ratio CPU (s) CPU ratio Peak RSS (MiB) RSS ratio Result
    compile (no cache) 4.22 → 5.78 1.370x 9.15 → 13.58 1.484x 915 → 952 1.041x ❌ regressed
    compile (warm cache) 0.20 → 0.18 0.900x 0.12 → 0.12 1.000x 71 → 71 1.001x ✅ ok
    plan --json 5.98 → 6.09 1.018x 7.77 → 7.98 1.027x 1007 → 1024 1.018x ✅ ok
    lineage hub downstream 0.20 → 0.19 0.950x 0.27 → 0.26 0.963x 155 → 161 1.040x ✅ ok
    lineage rollup upstream 0.20 → 0.20 1.000x 0.27 → 0.26 0.963x 153 → 170 1.109x ✅ ok
    lineage column trace 2.01 → 1.97 0.980x 2.74 → 2.69 0.982x 920 → 923 1.004x ✅ ok
    dag --json 2.60 → 2.62 1.008x 3.24 → 3.32 1.025x 956 → 968 1.012x ✅ ok
    scope --json 0.19 → 0.19 1.000x 0.17 → 0.18 1.059x 117 → 124 1.057x ✅ ok
    compile (one-model edit) 0.85 → 0.87 1.024x 0.70 → 0.71 1.014x 356 → 356 1.001x ✅ ok
    dense compile (warm cache) 0.28 → 0.28 1.000x 0.20 → 0.19 0.950x 94 → 94 1.001x ✅ ok
    dense compile (one-model edit) 1.24 → 1.27 1.024x 1.10 → 1.12 1.018x 436 → 432 0.991x ✅ ok
    dense compile (no cache) 6.63 → 6.37 0.961x 16.89 → 16.47 0.975x 1395 → 1410 1.011x ✅ ok
    build (empty warehouse) 32.74 → 31.70 0.968x 32.13 → 31.90 0.993x 1736 → 1738 1.001x ✅ ok

    Result: failed. compile (no cache): wall 1.370x exceeds 1.15x; compile (no cache): CPU 1.484x exceeds 1.15x.

  6. github-actions commented on Nov 10, 2026

    @github-actions
    ContributorAuthor

    Release performance regressed at c642221 on main.

    Comparison run: https://github.com/sqlstack-io/sqlbuild/actions/runs/42825566391

    Release performance: 0.207.0 (candidate) vs 0.207.0 (baseline)

    Same runner: AMD EPYC 9V45 96-Core Processor, 4 CPUs; load average 1.13/2.63/3.27 before and 1.26/1.80/2.61 after. 5 interleaved runs per version; wall and CPU are medians, peak RSS is the worst run.

    Benchmark projects are generated per side: the baseline runs projects from its own generator (v0.207.0), the candidate from the candidate's.

    Limits: wall +25% (ignored under 0.5 s), CPU +25% (ignored under 0.5 s), peak RSS +25% (ignored under 32 MiB). Cold, warm and one-model edit compiles use +15% for wall and CPU.

    Command Wall (s) Wall ratio CPU (s) CPU ratio Peak RSS (MiB) RSS ratio Result
    compile (no cache) 5.60 → 6.12 1.093x 12.79 → 15.65 1.224x 887 → 876 0.988x ❌ regressed
    compile (warm cache) 0.21 → 0.20 0.952x 0.13 → 0.13 1.000x 78 → 79 1.006x ✅ ok
    plan --json 6.60 → 6.84 1.036x 8.52 → 8.89 1.043x 944 → 883 0.935x ✅ ok
    lineage hub downstream 0.21 → 0.21 1.000x 0.29 → 0.29 1.000x 91 → 101 1.104x ✅ ok
    lineage rollup upstream 0.21 → 0.21 1.000x 0.29 → 0.29 1.000x 91 → 101 1.115x ✅ ok
    lineage column trace 2.50 → 2.31 0.924x 3.28 → 3.11 0.948x 839 → 805 0.959x ✅ ok
    dag --json 3.25 → 3.01 0.926x 4.00 → 3.78 0.945x 876 → 822 0.938x ✅ ok
    scope --json 0.23 → 0.23 1.000x 0.23 → 0.22 0.957x 87 → 94 1.076x ✅ ok
    compile (one-model edit) 1.05 → 0.99 0.943x 1.00 → 0.93 0.930x 334 → 309 0.927x ✅ ok
    dense compile (warm cache) 0.29 → 0.29 1.000x 0.20 → 0.20 1.000x 102 → 102 0.998x ✅ ok
    dense compile (one-model edit) 1.57 → 1.51 0.962x 1.52 → 1.45 0.954x 378 → 392 1.037x ✅ ok
    dense compile (no cache) 7.00 → 7.04 1.006x 17.12 → 19.24 1.124x 1282 → 1263 0.985x ✅ ok
    build (empty warehouse) 34.20 → 33.34 0.975x 33.56 → 33.50 0.998x 1651 → 1634 0.990x ✅ ok

    Result: failed. compile (no cache): CPU 1.224x exceeds 1.15x.

  7. kvlonge commented on Nov 10, 2026

    @kvlonge
    Contributor

    Run https://github.com/sqlstack-io/sqlbuild/actions/runs/42825566391 (main at c642221, after #1397) flagged compile (no cache) CPU at 1.224x against 0.207.0 (12.79 s -> 15.65 s; wall 1.093x, peak RSS 0.988x). dense compile (no cache) CPU is 1.124x, within its limit, and every other command is at or below 1.0x.

    This is the known cost of moving model analysis onto SQLQuack's native parsers in #1397. It was accepted by the maintainers as a temporary allowance when the cutover was approved, and the analysis-sharing change in the same PR already reduced it (same-machine A/B on the 3,000-model compiler-only case: cold CPU 17.46 s -> 15.62 s). A follow-up to bring cold no-cache CPU back within 1.10x of the previous release is tracked; the comparison limits themselves are unchanged.

  8. github-actions commented on Nov 10, 2026

    @github-actions
    ContributorAuthor

    Release performance regressed at 5b4cbbf on main.

    Comparison run: https://github.com/sqlstack-io/sqlbuild/actions/runs/42874254488

    Release performance: 0.207.0 (candidate) vs 0.207.0 (baseline)

    Same runner: AMD EPYC 9V74 80-Core Processor, 4 CPUs; load average 0.98/2.28/3.14 before and 1.14/1.50/2.29 after. 5 interleaved runs per version; wall and CPU are medians, peak RSS is the worst run.

    Benchmark projects are generated per side: the baseline runs projects from its own generator (v0.207.0), the candidate from the candidate's.

    Limits: wall +25% (ignored under 0.5 s), CPU +25% (ignored under 0.5 s), peak RSS +25% (ignored under 32 MiB). Cold, warm and one-model edit compiles use +15% for wall and CPU.

    Command Wall (s) Wall ratio CPU (s) CPU ratio Peak RSS (MiB) RSS ratio Result
    compile (no cache) 7.24 → 7.69 1.062x 17.32 → 20.18 1.165x 891 → 868 0.975x ❌ regressed
    compile (warm cache) 0.20 → 0.22 1.100x 0.14 → 0.14 1.000x 78 → 78 0.996x ✅ ok
    plan --json 7.57 → 7.30 0.964x 10.22 → 9.97 0.976x 944 → 880 0.932x ✅ ok
    lineage hub downstream 0.22 → 0.22 1.000x 0.32 → 0.32 1.000x 92 → 101 1.100x ✅ ok
    lineage rollup upstream 0.22 → 0.22 1.000x 0.31 → 0.32 1.032x 92 → 101 1.096x ✅ ok
    lineage column trace 2.71 → 2.54 0.937x 3.78 → 3.61 0.955x 840 → 804 0.957x ✅ ok
    dag --json 3.51 → 3.28 0.934x 4.58 → 4.38 0.956x 879 → 824 0.937x ✅ ok
    scope --json 0.24 → 0.24 1.000x 0.23 → 0.23 1.000x 88 → 94 1.071x ✅ ok
    compile (one-model edit) 1.13 → 1.10 0.973x 1.06 → 1.01 0.953x 329 → 312 0.949x ✅ ok
    dense compile (warm cache) 0.30 → 0.33 1.100x 0.21 → 0.21 1.000x 101 → 100 0.996x ✅ ok
    dense compile (one-model edit) 1.59 → 1.61 1.013x 1.57 → 1.56 0.994x 375 → 384 1.025x ✅ ok
    dense compile (no cache) 8.93 → 8.86 0.992x 22.87 → 25.92 1.133x 1276 → 1243 0.974x ✅ ok
    build (empty warehouse) 44.14 → 43.88 0.994x 43.49 → 43.34 0.997x 1649 → 1626 0.986x ✅ ok

    Result: failed. compile (no cache): CPU 1.165x exceeds 1.15x.

  9. kvlonge commented on Nov 11, 2026

    @kvlonge
    Contributor

    Closing: the last flagged regression (compile without cache, CPU 1.22x at 0.207.0) is the known cost of moving model analysis onto SQLQuack's native parsers, explained above, and later release comparisons up to 0.211.0 passed. The release performance workflow opens a new issue if a regression comes back.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    performance-regressionMain is slower or larger than the latest release

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions