Skip to content

Forget the host neighbor entry of a reused guest address - #553

Merged
yummybomb merged 1 commit into
mainfrom
hypeship/forget-reused-guest-neighbor
Nov 12, 2026
Merged

yummybomb merged 1 commit into
mainfrom
hypeship/forget-reused-guest-neighbor

Conversation

@yummybomb

@yummybomb yummybomb commented Nov 10, 2026 •

Copy link
Copy Markdown
Contributor

Summary

The first connection from the host to a new guest could stall for about ten seconds when the guest was given an address that an earlier guest had used. CreateAllocation now deletes the host's neighbor entry for the address it hands out.

Why it happened

Guest addresses are picked at random from the subnet and reused once released. The host's neighbor (ARP) entry for a released address stays behind and still points at the previous guest's MAC. Linux keeps using a stale entry while it re-verifies it, so the first packets to the new guest go to a MAC nobody owns. The entry is only dropped once the delay and probe timers have run out, about 8 s later, and a TCP connect then gets through on its next SYN retransmit.

A guest forked from a snapshot is the worst case, because it sends nothing until the host contacts it. A cold-booted guest usually repairs the entry itself when it first resolves the gateway. On a host with fewer than gc_thresh1 (128) neighbor entries the kernel never garbage-collects them, so a stale entry can sit there for hours.

What changes

CreateAllocation removes the bridge's neighbor entry for the address it has just reserved, before it creates the TAP. A missing entry is the normal case and is not an error. A failure is logged and does not fail the allocation. Restore from standby is unchanged: the guest keeps its address and MAC.

Testing

  • Two new tests run against a real bridge and the kernel's neighbor table. They need root and are skipped without it. One covers the helper. The other plants stale entries on every address of a /29 and checks that CreateAllocation clears the one it hands out and leaves the rest. Each fails when the call or the delete is removed.
  • I reproduced the stall on a host by replacing a fresh fork's neighbor entry with a stale one before first contact: the first request took 11.7 s in 3 of 3 runs, against 0.5 s for forks left alone. This matches a fork I had seen take 11.7 s on its own, whose address had last been used 72 minutes earlier.
  • A build with this change then served about 150 forks and cold boots on that host, and logged no neighbor warnings.

The address a guest gets is random, so the host run could not force a reuse. The tests above are what cover the fixed path.


Note

Medium Risk
Changes the Linux network allocation hot path and kernel neighbor table, but neighbor delete failures are non-blocking and covered by integration tests.

Overview
CreateAllocation now drops the bridge’s stale neighbor (ARP) entry for a newly reserved guest IP before creating the TAP, so the host does not keep forwarding to the previous guest’s MAC when addresses are reused—avoiding multi-second stalls on first contact (especially snapshot forks).

On Linux, new forgetNeighbor uses netlink.NeighDel (missing entry is ignored); failures are logged and do not fail allocation. On macOS it is a no-op. Restore-from-standby is unchanged.

Adds root-required Linux tests that exercise forgetNeighbor and verify CreateAllocation clears only the allocated address’s stale neighbor among candidates on a /29.

Reviewed by Cursor Bugbot for commit 48f08f6. Bugbot is set up for automated code reviews on this repo. Configure here.

Guest addresses are picked at random from the subnet and reused. The host
kept the previous owner's neighbor entry, so the first connection to a new
guest on that address was sent to a MAC nobody owned until the entry timed
out, about ten seconds later. Delete the entry when the address is
allocated.
@yummybomb
yummybomb merged commit a72459e into main Nov 12, 2026
12 checks passed
@yummybomb
yummybomb deleted the hypeship/forget-reused-guest-neighbor branch November 12, 2026 22:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants