Fix JS_SetMemoryLimit byte accounting on WASI

[josefguenther]

On WASI, JS_SetMemoryLimit() does not bound the bytes of any allocation the arena does not handle. Each of those is accounted at a constant MALLOC_OVERHEAD, so for bulk allocation the limit is a count rather than a byte total: a push(new Uint8Array(1MB)) loop under a 32 MB limit runs to 4095 MB, the wasm32 address-space ceiling.

Mechanism

js__malloc_usable_size() (cutils.h) has no branch for WASI and falls through to return 0. js_malloc_rt() then does:

if (unlikely(s->malloc_size + size > s->malloc_limit - 1))
    return NULL;                                                       /* the REQUESTED size */
...
s->malloc_size += rt->mf.js_malloc_usable_size(ptr) + MALLOC_OVERHEAD; /* 0 + 8 */

The check reads the real requested size. The accumulator advances by a constant 8 bytes. That asymmetry is the whole defect: one allocation larger than the limit is refused, because the check is honest, and no number of sub-limit allocations ever accumulates, because the accounting is not.

The arena byte-accounts blocks up to JS_ARENA_MAX_SMALL_SIZE through arena_block_sizes[], so small objects are bounded. Anything larger reaches js_arena_usable_size()'s JS_ARENA_FREE_NIL branch, which calls the platform function, so bulk allocation is what escapes — and bulk allocation is the whole exposure.

Measured

wasm32-wasi, wasi-sdk 25, run under a wazero host. Same build and host, patched vs unpatched. These are measurements on one build, not a CI matrix.

probe	before	after
1 × 7 MB Uint8Array @ 8 MB limit	allowed	allowed
2 × 7 MB Uint8Array @ 8 MB limit	both allowed, 14 MB live	second refused
60 × 1 MB Uint8Array @ 8 MB limit	all 60 allowed, 60 MB live	refused after 7
200,000 ~600-byte strings @ 8 MB limit	all allowed, 83 MB of guest memory	refused after 21,186
push(new Uint8Array(1MB)) loop @ 32 MB limit	ran to 4095 MB	stops at 31 MB, guest memory tops out at 32.75 MB

The string row is from 0.12.1, before the arena landed; at 0.16.2 a 600-byte allocation is inside JS_ARENA_MAX_SMALL_SIZE and that loop is bounded either way. The bulk rows are unchanged at 0.16.2, and the code path is unchanged at df836d1: 60 × 1 MB under an 8 MB limit all still succeed.

The same numbers reproduce on emscripten builds, which is what identifies the shared #else rather than a packaging difference: justjake/quickjs-emscripten#271 has the full probe table for that side. vercel-labs/quickjs-wasi#30 is an independent report of the same behaviour, worked around downstream in #33 by supplying malloc functions with a working usable size rather than patching this file.

The fix

wasi-libc declares malloc_usable_size() in <malloc.h> and defines it in libc.a, so __wasi__ is named in the two selectors that already list the other platforms — the include and the accounting branch. No new code.

Emscripten's musl provides the same symbol and falls into the same #else. It is left alone here because wasi is what was measured end to end; happy to add it if you want both in one change.

Test

api-test.c gains large_allocation_accounting(): a 1 MB Uint8Array must move malloc_size by at least its own size, and 64 × 1 MB under a 4 MB limit must throw with real usage staying under 8 MB. It is platform-neutral — it passes wherever the usable size is real and fails wherever it is reported as zero. Verified by forcing js__malloc_usable_size() to return 0 on macOS, where it aborts on the first assertion.

Note it does not currently run on WASI: the wasi CI job builds qjs_exe and runs a script, it does not build or run api-test. So CI will go green with and without this patch — the test documents the invariant on every platform, but nothing in the matrix exercises the one platform being fixed. Say the word and I will extend that job so it does, or drop the test if you would rather keep the change to the two lines.

One pre-existing behaviour the fix reveals

Honest accounting exposes something that is already true on Linux and macOS today: a heap exhausted by many small allocations can no longer build the Error describing its own exhaustion. JS_ThrowError2 substitutes JS_NULL to avoid recursing, so the embedder sees a bare null rather than InternalError: out of memory. A refused large block still reports properly, because the refusal leaves headroom.

The broken accounting was hiding this on WASI — the count budget was exhausted while real memory remained, so the error object always allocated. It is not introduced by this patch. Downstream hit it as soon as they fixed the accounting (vercel-labs/quickjs-wasi#39) and solved it by reserving headroom below the enforced limit. Whether the engine should do something similar is a separate question from this one; flagging it so it is not attributed here.

Checks

cmake -B build -DCMAKE_BUILD_TYPE=Release, api-test and run-test262 -c tests.conf (0/112 errors) pass on macOS/arm64 at df836d1.

[bnoordhuis]

tl;dr wall of LLM text. In a dozen words max, what's the problem this PR fixes?

[josefguenther]

I'll try: the authors simply forgot the WASI branch in a couple places.

When setting a memory limit, it's supposed to honor that limit whenever memory is allocated. Currently it's inconsistent on WASI: allocating some variable types like integers does honor it, but others like strings or arrays only honor it if a string over the limit is instantiated (eg if the memory limit is set to 8MB: allocating a 14MB string will fail but setting two strings 7MB long will work, exceeding the memory limit).

Side note: this may affect emscripten/musl too but I didn't chase that.