Repository navigation
fix: recover markerless stale rendezvous sockets during activation - #1894
Conversation
|
Thanks for opening this — it has been seen, and it is queued. This note is automated, but it is not a brush-off: it exists so you know where your PR stands instead of having to guess from silence. Current review status: working through a backlog. What that means for this PR, concretely:
Things that will genuinely speed it up whenever review does happen:
If this fixes a bug, a reproduction we can run is worth more than a description of the symptom. Thanks for contributing, and sorry in advance for the wait. |
|
Thanks for this — and apologies that the only thing you have heard so far was the automated acknowledgement. Status: the one thing blocking it is DCO. Every other required check is green, which is a better starting position than most PRs arrive in. It is a single commit ( The substance still needs a proper read against #1760 before anything is promised — that is queued, not skipped. But the sign-off is worth doing now so the review is not waiting on a formality when it comes up. |
Fixes DeusData#1760 Signed-off-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
498f69b to
8e70590
Compare
|
Review done, and the hard part holds up: I had two independent passes on the cleanup path, one of them adversarial. The only unlink primitive reached is One line before it merges — a test-isolation defect, not a logic one: your new cli test resets the activation runtime parent with For the record on #1760: this fix covers both the markerless shape and the marker-present residue for every probe caller, so it supersedes #1790 (whose own test passes on your tree unchanged); I'll close that one with credit once yours lands. Thanks for the careful work — the nlink/identity proof is the kind of thing that makes a cleanup path trustworthy. |
|
One more note so you do not chase it: the |
The markerless stale-rendezvous recovery test reset the activation runtime parent to NULL instead of restoring g_cli_suite_runtime_parent, so every later cli test ran the activation probe against the developer's real /tmp/cbm-daemon-<uid>. Restore the suite value per the contract at tests/test_cli.c:61-65 and the sibling at :1310. Signed-off-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
|
Merged as 760c08f — thank you, and thanks for the seam-restore follow-up commit. Recognising the markerless hard-linked .sock/.anc pair only while holding the startup lock plus a temporary exclusive lifetime reservation, and revalidating device/inode and the exact two-link shape before unlinking, keeps the recovery inside the daemon's own ownership boundary; that is what made this mergeable where a plain "delete stale files" fix would not have been. With nine independent reports on #1757/#1760 this was the widest-reaching install/daemon bug in the queue. This ships in v0.10.9. |
A daemon that died at listener publication (a full /tmp) took the lifetime reservation, failed, and released it; the waiting client saw the generation vanish, respawned it, and repeated that for its whole 30 s deadline before reporting "CBM daemon is active or starting but could not accept this client" -- the opposite of what happened, with no cause and no path. MCP hosts then dropped the server silently (DeusData#1828). A detached daemon has no stderr and no channel back to its launcher, so the host now appends one start-failure record per failed start to <cache>/logs/cbm-daemon.start-failures.log -- next to cbm-daemon.log, through the same owner-only private-log open. Deliberately NOT the runtime directory: that is exactly what a full /tmp cannot write, and a new artifact there would entangle the stale-generation cleanup that DeusData#1894 just hardened. The record carries the listener detail from the previous commit (component, stage, errno, path) plus the runtime directory and a wall-clock stamp, and is written while the lifetime reservation is still held, so a client that watches the generation vanish already finds the cause. The client bootstrap loop reads that record (every 50 ms after its own spawn, and unconditionally before any relaunch), accepts only records for its endpoint not older than its spawn instant (2 s skew), and stops at once: error: CBM daemon failed to start: pending_publication failed with ENOSPC (No space left on device) at /tmp/cbm-daemon-1000/cbm-<key>.sock.pending.tmp; see ~/.cache/codebase-memory-mcp/logs/cbm-daemon.log The read is an optional bootstrap op (unit contracts with fake ops are untouched); the production op resolves the cache log directory. Every other wait state -- a live or stopping generation, a mute holder, a concurrent starter -- keeps its existing semantics and diagnostics. Tests (daemon_bootstrap): - daemon_bootstrap_fails_fast_when_daemon_dies_at_publication: the production bootstrap (real cohort, probe, startup lock, handoff) with a seam-only spawn override that forks a REAL daemon host whose record writes fail with ENOSPC. RED before: 34.7 s and "active or starting"; GREEN after: FAILED in well under 5 s naming ENOSPC and the temp path, one child spawned, daemon log carries errno=ENOSPC path=... . - daemon_bootstrap_start_failure_record_round_trip: record/read/format contract, a stale record and another endpoint's record are ignored. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
A daemon that died at listener publication (a full /tmp) took the lifetime reservation, failed, and released it; the waiting client saw the generation vanish, respawned it, and repeated that for its whole 30 s deadline before reporting "CBM daemon is active or starting but could not accept this client" -- the opposite of what happened, with no cause and no path. MCP hosts then dropped the server silently (DeusData#1828). A detached daemon has no stderr and no channel back to its launcher, so the host now appends one start-failure record per failed start to <cache>/logs/cbm-daemon.start-failures.log -- next to cbm-daemon.log, through the same owner-only private-log open. Deliberately NOT the runtime directory: that is exactly what a full /tmp cannot write, and a new artifact there would entangle the stale-generation cleanup that DeusData#1894 just hardened. The record carries the listener detail from the previous commit (component, stage, errno, path) plus the runtime directory and a wall-clock stamp, and is written while the lifetime reservation is still held, so a client that watches the generation vanish already finds the cause. The client bootstrap loop reads that record (every 50 ms after its own spawn, and unconditionally before any relaunch), accepts only records for its endpoint not older than its spawn instant (2 s skew), and stops at once: error: CBM daemon failed to start: pending_publication failed with ENOSPC (No space left on device) at /tmp/cbm-daemon-1000/cbm-<key>.sock.pending.tmp; see ~/.cache/codebase-memory-mcp/logs/cbm-daemon.log The read is an optional bootstrap op (unit contracts with fake ops are untouched); the production op resolves the cache log directory. Every other wait state -- a live or stopping generation, a mute holder, a concurrent starter -- keeps its existing semantics and diagnostics. Tests (daemon_bootstrap): - daemon_bootstrap_fails_fast_when_daemon_dies_at_publication: the production bootstrap (real cohort, probe, startup lock, handoff) with a seam-only spawn override that forks a REAL daemon host whose record writes fail with ENOSPC. RED before: 34.7 s and "active or starting"; GREEN after: FAILED in well under 5 s naming ENOSPC and the temp path, one child spawned, daemon log carries errno=ENOSPC path=... . - daemon_bootstrap_start_failure_record_round_trip: record/read/format contract, a stale record and another endpoint's record are ignored. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
A daemon that died at listener publication (a full /tmp) took the lifetime reservation, failed, and released it; the waiting client saw the generation vanish, respawned it, and repeated that for its whole 30 s deadline before reporting "CBM daemon is active or starting but could not accept this client" -- the opposite of what happened, with no cause and no path. MCP hosts then dropped the server silently (DeusData#1828). A detached daemon has no stderr and no channel back to its launcher, so the host now appends one start-failure record per failed start to <cache>/logs/cbm-daemon.start-failures.log -- next to cbm-daemon.log, through the same owner-only private-log open. Deliberately NOT the runtime directory: that is exactly what a full /tmp cannot write, and a new artifact there would entangle the stale-generation cleanup that DeusData#1894 just hardened. The record carries the listener detail from the previous commit (component, stage, errno, path) plus the runtime directory and a wall-clock stamp, and is written while the lifetime reservation is still held, so a client that watches the generation vanish already finds the cause. The client bootstrap loop reads that record (every 50 ms after its own spawn, and unconditionally before any relaunch), accepts only records for its endpoint not older than its spawn instant (2 s skew), and stops at once: error: CBM daemon failed to start: pending_publication failed with ENOSPC (No space left on device) at /tmp/cbm-daemon-1000/cbm-<key>.sock.pending.tmp; see ~/.cache/codebase-memory-mcp/logs/cbm-daemon.log The read is an optional bootstrap op (unit contracts with fake ops are untouched); the production op resolves the cache log directory. Every other wait state -- a live or stopping generation, a mute holder, a concurrent starter -- keeps its existing semantics and diagnostics. Tests (daemon_bootstrap): - daemon_bootstrap_fails_fast_when_daemon_dies_at_publication: the production bootstrap (real cohort, probe, startup lock, handoff) with a seam-only spawn override that forks a REAL daemon host whose record writes fail with ENOSPC. RED before: 34.7 s and "active or starting"; GREEN after: FAILED in well under 5 s naming ENOSPC and the temp path, one child spawned, daemon log carries errno=ENOSPC path=... . - daemon_bootstrap_start_failure_record_round_trip: record/read/format contract, a stale record and another endpoint's record are ignored. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
What does this PR do?
Extend the existing POSIX stale-generation cleanup in
src/daemon/ipc.cto recognize the markerless hard-linked.sock/.ancpair only while it retains the matching startup lock and a temporary exclusive lifetime reservation. Require both paths to be secure sockets for the same device/inode with the exact two-link shape, revalidate those identities immediately before inode-matched unlinking, and continue to refuse unanchored, mismatched, malformed, replaced, or actively reserved endpoints. Make the activation-only generation probe invoke this serialized cleanup before classifying a remaining endpoint, soinstallandupdateuse the recovery path already shared by daemon startup without weakening the conservativeECONNREFUSEDbehavior for live or saturated listeners.Hard-killing a POSIX daemon or MCP process can leave the owner-private rendezvous namespace with a hard-linked stable socket and anchor but without the pending or identity record that normally authorizes cleanup. The kernel releases the process-owned startup, lifetime, and cohort locks, yet activation's generation probe treats the refused socket as active and permanently blocks
installandupdate. Deleting/tmp/cbm-daemon-<uid>works around the problem, confirming that durable transport artifacts—not a live CBM session—cause the refusal. The current branch already documentsCBM_RUNTIME_DIR, distinguishes coordination failures from busy sessions, and returns a nonzero activation status, so this change is limited to safe automatic recovery of the reported crash shape.Fixes #1760
Checklist
git commit -s) — required, CI rejectsNot run: no test command resolved in this workspace, so nothing was executed to pass.
unsigned commits (DCO, see CONTRIBUTING.md)
make -f Makefile.cbm test)Not run: no test command resolved in this workspace, so nothing was executed to pass.
make -f Makefile.cbm lint-ci)Not run: no test command resolved in this workspace, so nothing was executed to pass.