Skip to content

ci: limit each timed dense compile to 4 GiB instead of the pytest tree - #1324

Merged
kvlonge merged 2 commits into
mainfrom
ci/dense-compile-memory-scope
Nov 6, 2026
Merged

kvlonge merged 2 commits into
mainfrom
ci/dense-compile-memory-scope

Conversation

@kvlonge

@kvlonge kvlonge commented Nov 6, 2026

Copy link
Copy Markdown
Contributor

Why

The dense all-rules guard job ran make, uv and pytest inside one 4 GiB systemd unit. The test budget of 3.75 GiB per compile is meant for the compiler, but about 0.5 GiB of harness memory counted against the same limit, so the 10,000-model job was killed for memory while the compiler stayed inside its budget.

Changes

  • Each timed sqb compile in the dense and DuckShop 10K guards now runs in its own transient systemd scope with MemoryMax=4 GiB and no swap. Pytest runs outside the limit. This gives each compile about 0.5 GiB more effective headroom.
  • Every budget is unchanged.
  • Under CI the guards refuse to run unless SQLBUILD_COMPILE_MEMORY_LIMIT_BYTES is set. If systemd-run or sudo is missing, they fail and name the scope they could not start, so a guard never compiles without the limit.
  • Other compile guard jobs keep their existing outer limit.

Verification

  • Earlier per-compile peak RSS for the 10,000-model cold compile: 3.21–3.59 GiB.
  • 10,000 models locally, with each compile in its own scope: both guards pass in 292 s.
    • Cold compile peak: 3,626 MiB.
    • First delta-loop compile peak: 3,903 MiB. This is above the 3.75 GiB budget per compile, which the delta-loop test does not apply to that compile. The budget is not raised here; a separate issue tracks it.
  • Failure paths checked:
    • CI without the limit fails, naming the variable.
    • No systemd-run fails, naming the scope.
    • A 150 MiB scope kills the compile and the test fails.
  • ruff, ty, fensu and make check-ci pass.
  • DuckShop edge case: not applicable, because this changes only how CI runs the guards.

The dense guard job ran make, uv and pytest inside one 4 GiB systemd unit,
so roughly 0.5 GiB of harness memory counted against a limit meant for the
compiler. Each timed sqb compile now runs in its own transient scope with
MemoryMax=4 GiB and no swap, and pytest runs outside it. Budgets are
unchanged. Under CI the guards refuse to run without the per-compile limit,
and a missing systemd-run or sudo fails naming the scope.
@kvlonge
kvlonge enabled auto-merge (squash) November 6, 2026 12:00
@kvlonge
kvlonge merged commit cd38ff9 into main Nov 6, 2026
50 checks passed
@kvlonge
kvlonge deleted the ci/dense-compile-memory-scope branch November 6, 2026 20:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant