Create right after kill fails with 500 Failed to place sandbox while the killed sandbox still holds its hugepages #3689

I run E2B Embed from embed/compose at 311f9fc (images 202610051633-d1cd3db3890, Firecracker v1.14, SDK e2b 2.46.0) on one 8 vCPU 16 GiB host with HUGEPAGES=4608. My template has memory_mb=8192, so one sandbox takes 4096 of the 4608 pages. I run one sandbox at a time for CI jobs.

When a sandbox has written a few GiB of memory and I kill it and create the next one right away, the create fails with

500: Failed to place sandbox: sandbox creation failed on 1 node(s), please retry; if the problem persists, contact us

and the orchestrator logs

failed to create sandbox {"error": "failed to start FC: error loading snapshot: error loading snapshot: Put \"http://localhost/snapshot/load\": uffd process exited: failed to wrap memfd: mmap memfd: cannot allocate memory"}

The same create about 0.8 s later works.

Steps to reproduce

1. Add HUGEPAGES=4608 to .env and start Embed.
2. Build a template with Template.build(Template().from_image("python:3.12-slim-bookworm"), "ep-8g", cpu_count=4, memory_mb=8192).
3. Run this script.

import time
from e2b import Sandbox

FILL = "python3 -c \"n=6<<30; b=bytearray(n); b[::4096]=b'\\x01'*(n//4096)\""

sbx = Sandbox.create("ep-8g", timeout=600)
for i in range(5):
    sbx.commands.run(FILL, user="root", timeout=300)  # write 6 GiB of guest memory
    sbx.kill()
    t = time.time()
    try:
        sbx = Sandbox.create("ep-8g", timeout=600)
        print("create ok", round(time.time() - t, 2))
    except Exception as exc:
        print("create failed", round(time.time() - t, 2), exc)
        time.sleep(1)
        sbx = Sandbox.create("ep-8g", timeout=600)
sbx.kill()

What I measured

I ran 215 kill and create cycles and sampled HugePages_Free, HugePages_Rsvd and the firecracker PIDs on the host every 100 ms.

1. kill() returns after about 14 ms.
2. After the sandbox wrote 6 GiB, the killed firecracker process stays for 0.33 s (0.45 s at most) and its 4096 pages come back when it exits. A create right after the kill failed 14 of 15 times.
3. After 2 GiB written it stays for 0.15 s, and a create right after the kill failed 5 of 5 times.
4. A sandbox that wrote nothing is gone after about 0.08 s, and a create right after the kill worked 15 of 15 times. Age does not change this. A sandbox that sat idle for 10 s behaved the same.
5. With a pause of 1 s or more between kill and create I saw no failure in 80 tries with this template.
6. With memory_mb=512 I saw no failure at all, because two sandboxes fit in the pool.
7. The API logs one placement attempt per request ("attempt": 1) and the SDK sends one POST. Newer SDKs retry 429, 502 and 503 but not 500, so nothing retries this.

Where I think it comes from

Delete in packages/orchestrator/pkg/server/sandboxes.go (around line 844 on main) starts sbx.Stop in a goroutine and returns, so the API answers DELETE before Firecracker exits. The next create maps the guest memfd in uffd.go through block.NewFromFd. That mmap needs the whole guest size free in the pool, so it fails with ENOMEM while the old process still holds its pages. Create returns this as codes.Internal (sandboxes.go line 469 on main). Placement then excludes the node (placement.go line 304), and with one node the request ends as SandboxCreateError, which placement_errors.go turns into the 500. A ResourceExhausted answer would have been retried on the same node with backoff instead.

The hugepage fit check from 16ec5e6 would probably catch this and answer 503, since OptimisticRemove keeps the hugepage counts until the next sync. But placement-hugepage-limit-percent defaults to 0 and I found no way to set it in Embed without LaunchDarkly. The images I ran also predate it.

What would help

Any one of these would fix it for me.

1. Return codes.ResourceExhausted from Create when the error is ENOMEM, so placement retries the node. I tried this in a scratch branch with a placement unit test where the node refuses for 0.4 s, and the create landed after about 0.5 s. A template that can never fit would then retry until the deadline and end in a 503, so it may need a guard.
2. Let Delete wait for the Firecracker exit, which took under 0.5 s here.
3. Give placement-hugepage-limit-percent an env fallback so Embed users can turn the check on.

For now I retry this 500 in my client. In my runs a retry 0.5 s after the failed create worked 19 of 19 times.