LiveCodeBench evaluation has no memory limit: unbounded allocations end PR evaluations with exit code 143 #238

Problem

PR evaluations can end with Process completed with exit code 143 before any results are posted. This happened on all three evaluation runs of our submission (#236 twice, #237 once), at 9,205/9,209, 9,160/9,209 and 1,220/8,400 entries.

unsafe_lcb_runTests in llm_evaluation/livecodebench_util.py runs each LiveCodeBench completion in a forked process and limits only wall-clock time, (timeout + 1) * len(test_cases) + 5 seconds. There is no memory limit:

• reliability_guard() is called with maximum_memory_bytes=None;
• the guard then hides the resource module.

Generated code that allocates without bound keeps growing until that timeout. On the 16 GB ubuntu-latest runner, the system's memory-pressure handling terminates the job first (SIGTERM, exit code 143), so the run fails and no comment is posted.

Reproduction (synthetic, no dataset)

A three-test stdin problem, evaluated through check_correctness, with the run held to a 24 GiB cgroup:

PROBLEM = {"test": [{"input": "2 3\n", "output": "5", "testtype": "stdin"}] * 3, "is_stdin": True}
CORRECT = "a, b = map(int, input().split())\nprint(a + b)\n"
RUNAWAY = "blocks = []\nwhile True:\n    blocks.append(bytearray(64 * 1024 * 1024))\n"
lcb.check_correctness(problem=PROBLEM, completion=RUNAWAY, timeout=6, is_extracted=False)

On real submissions

We measured only memory and exit status, and deleted the scored outputs without reading them.

• In Add Compass CoreRouter submission (compass-corerouter) #236's file, two LiveCodeBench completions in the optimality rows (LiveCodeBench_437, LiveCodeBench_491) each reached a 96 GiB cgroup cap within about two minutes when evaluated alone. They are evaluated last, where both Add Compass CoreRouter submission (compass-corerouter) #236 runs died.
• Add Compass CoreRouter submission (compass-corerouter, resubmission of #236) #237's file has only the 8,400 scored rows. Its LiveCodeBench phase still peaked at 23.5 GiB on main, which fits the third failure.

Proposed fix

See the linked PR. It caps each test process's address space at the size it inherits from the evaluator plus 2 GiB, configurable via ROUTERARENA_LCB_MEMORY_HEADROOM_BYTES; 0 disables the limit. Oversized solutions raise MemoryError and fail that test like any other error. 2 GiB is above common judge limits (AtCoder 1 GiB, Codeforces 256 MiB).

Validation of the fix:

• Both of our files ran through the official llm_evaluation/run.py ... full --force (Python 3.11, uv sync --locked) inside a 16 GiB cgroup with no swap and 4 CPUs.
  • Add Compass CoreRouter submission (compass-corerouter) #236's 9,209-row file: exit 0, 8.0 GiB peak.
  • Add Compass CoreRouter submission (compass-corerouter, resubmission of #236) #237's 8,400-row file: exit 0, 7.9 GiB peak.
  • On main, the same setup previously failed or spiked to 23.5 GiB and above.
• We also evaluated the 385 LiveCodeBench rows of three merged routers with and without the fix:
  • KT-ModelRouter: 0 scores changed, and both runs match the published scores.
  • cruq-router and nadir-cascade-v2: 3 and 2 rows changed.
    • The 0→1 changes match OOM kills that occurred only without the fix.
    • The two 1→0 changes were the same problem (LiveCodeBench_386). Re-run alone three times, it passes 3/3 with and without the fix, so those flips were timeouts under load.

Possible follow-up (not in the PR)

Forked test processes also grow by copying the evaluator's inherited memory, which an address-space limit does not bound. This only became large when many LiveCodeBench rows ran concurrently. Calling gc.freeze() before forking is a common mitigation, but we have not tested it here.