macOS: markerless stale socket/anchor pair permanently blocks daemon startup

Version

codebase-memory-mcp 0.10.8

Platform

macOS on Apple Silicon

Install channel

GitHub release archive

Binary variant

standard

What happened, and what did you expect?

When a runtime namespace contains a hard-linked .sock and .anc pair but no .sock.identity or .sock.pending, daemon startup remains permanently blocked.

No process holds the socket and connecting to it returns ECONNREFUSED, but both daemon start and CLI commands fail until the pair is moved manually.

I expected startup to recover this provably stale state safely, or to report the exact blocking artifacts and provide a supported recovery command.

The public command only reports a 30-second startup timeout. The daemon log ends at stage=namespace_validation.

Reproduction

This reproduction uses isolated runtime and cache directories and does not touch the account daemon.

#!/bin/sh
set -eu

runtime=$(mktemp -d /private/tmp/cbm-markerless-repro.XXXXXX)
cache=$(mktemp -d /private/tmp/cbm-markerless-cache.XXXXXX)
chmod 700 "$runtime" "$cache"

run_cbm() {
  CBM_RUNTIME_DIR="$runtime" \
  CBM_CACHE_DIR="$cache" \
  codebase-memory-mcp "$@"
}

start_output=$(run_cbm daemon start)
printf '%s\n' "$start_output"

pid=$(printf '%s\n' "$start_output" |
  sed -n 's/.*pid \([0-9][0-9]*\)).*/\1/p')
kill -KILL "$pid"

attempt=0
while kill -0 "$pid" 2>/dev/null && [ "$attempt" -lt 100 ]; do
  sleep 0.1
  attempt=$((attempt + 1))
done
if kill -0 "$pid" 2>/dev/null; then
  printf 'daemon did not exit\n' >&2
  exit 1
fi

namespace="$runtime/cbm-daemon-$(id -u)"
identity=$(find "$namespace" -maxdepth 1 -name '*.sock.identity' -print -quit)
socket=${identity%.identity}
anchor=${socket%.sock}.anc

stat -f '%HT inode=%i links=%l %N' "$socket" "$anchor"
mv "$identity" "$runtime/socket.identity.backup"

set +e
run_cbm daemon start
start_rc=$?
set -e
printf 'markerless_start_rc=%s\n' "$start_rc"
[ "$start_rc" -eq 1 ]

mv "$runtime/socket.identity.backup" "$identity"
run_cbm daemon start
run_cbm cli --json list_projects
run_cbm daemon stop

Observed markerless startup output:

codebase-memory-mcp: CBM daemon could not start within 30000 ms
error: CBM daemon could not start within 30000 ms

Restoring the identity marker makes daemon start and cli --json list_projects return exit code 0.

Relevant daemon log

level=error msg=daemon.ipc.listen_failed stage=namespace_validation
level=error msg=daemon.runtime.start_failed stage=listener_handoff
level=error msg=daemon.start_failed component=runtime

Additional evidence

The stale .sock and .anc paths refer to the same Unix socket inode with link count 2. No process holds the socket, and connecting returns ECONNREFUSED.

posix_stale_generation_cleanup_locked() does not reclaim the markerless socket/anchor pair, while cbm_daemon_ipc_listen_reserved() subsequently rejects the remaining namespace entry.

Diagnostics and project scale

The failure occurs before project loading, so project size and indexing state do not affect the reproduction.

Confirmations

I reproduced this with codebase-memory-mcp 0.10.8.
The reproduction uses isolated runtime and cache directories.
No secrets, project contents, or user-specific paths are included.

Independent reproduction on 0.10.8, macOS 15 (Apple Silicon), GitHub release archive. Matches this report exactly, including the "markerless" detail.

Blocking artifacts

/tmp/cbm-daemon-501/ as found (daemon had been dead for 6 days):

srw-------  2  Aug 19 12:40  cbm-<id>.anc
srw-------  2  Aug 19 12:40  cbm-<id>.sock
-rw-------  1  Aug 19 12:40  cbm-<id>.lock
-rw-------  1  Aug 19 12:40  cbm-<id>.lifetime.lock
-rw-------  1  Aug 19 12:40  cbm-<id>.startup-v2.lock
-rw-------  1  Aug 24 00:14  cbm-version-cohort-maintenance-v1.lock
-rw-------  1  Aug 24 10:30  cbm-version-cohort-admission-v1.lock
-rw-------  1  Aug 24 10:54  cbm-version-cohort-daemon-v1.lock
-rw-------  1  Aug 27 08:33  cbm-version-cohort-lifetime-v1.lock   (214 bytes: "CBMCOH 0.10.8 <build> <hash>")

.sock and .anc are the hard-linked pair (link count 2), and there is no .sock.identity and no .sock.pending — the markerless state described here. Zero codebase-memory-mcp processes were running.

Symptoms

Every daemon start printed:

level=info msg=version_cohort.claimed_unheld build=996bad5f...
codebase-memory-mcp: CBM daemon could not start within 30000 ms

Two details that may help narrow the stage:

The process never forked a child (watched at 2s intervals for the full 30s window).
cbm-daemon.log was never written to across ~10 start attempts — it stayed stamped at the last clean shutdown (reason=stop_requested, 6 days earlier). So the failure happens before any daemon-side logging.

daemon status reported daemon: not running throughout, giving no indication that anything was blocking.

Correlated prior state

daemon-conflicts.ndjson contained 4882 daemon.version_conflict entries within a single ~10h window, all of this shape:

active:    pre-cohort/unknown  build=0000000000000000...
requested: 0.10.8              build=996bad5f...

Alongside them, 4883 .worker-log-* files in the cache dir, each containing only:

CBM index worker could not start: active daemon

The pre-cohort/unknown build=0000... registrant never drained. Ruled out during triage: port 9749 free, _config.db config table empty, no other cbm sockets under $TMPDIR, CBM_LOG_LEVEL=debug / RUST_LOG=debug produced no additional output.

Recovery

rm -rf /tmp/cbm-daemon-501
codebase-memory-mcp daemon start

Started on the first attempt and has been healthy since; index DBs were untouched and needed no rebuild. Confirming the request in the original report: a supported recovery command (or self-healing when the pair is provably stale and unconnectable) would help a lot — the surfaced 30s timeout points nowhere near /tmp, and daemon status actively suggests there is nothing to clean up.

Confirmed reproduction + working recovery on macOS (0.10.8, build 996bad5f, Apple Silicon, macOS 26.6.2).

Root cause matches this issue exactly: after CBM processes were force-killed (SIGTERM timeout → hard kill per log: "Force-killed MCP process ... after SIGTERM timeout"), /private/tmp/cbm-daemon-<uid> was left with a hard-linked stale pair:

cbm-<hash>.sock  (Socket prefix_inode=104454126 links=2)
cbm-<hash>.anc   (Socket prefix_inode=104454126 links=2)

with no .sock.identity / .sock.pending — i.e. a "markerless" pair exactly as described. No process holds it (lsof clean), yet:

daemon start → CBM daemon could not start within 30000 ms
any CLI/config list → CBM CLI could not start because a pre-coordination or unverified CBM generation is active
install/update → active CBM sessions and operations could not be stopped safely; no activation was committed
install.sh from the archive → same failure
Even a fresh CBM_CACHE_DIR still failed (stale state lives in the runtime namespace, not the cache)

The cbm-version-cohort-lifetime-v1.lock in that dir was a 214-byte binary CBMCOH record pinned to 0.10.8/build — the version_cohort.claimed_unheld in the first log line.

Recovery that works (no reinstall needed): simply move the whole stale dir aside:

mv /private/tmp/cbm-daemon-$(id -u) /tmp/cbm-daemon-$(id -u).stale

After that: daemon start succeeds (daemon: started (permanent, pid ...)), daemon status reports active, and the stdio MCP handshake returns all 14 tools with serverInfo: codebase-memory-mcp 0.10.8. Real indexes in ~/.cache/codebase-memory-mcp/*.db were preserved untouched.

This is a broader UX/diagnosability gap worth noting beyond the Windows report in #1760: the daemon status/error wording only ever points at "an editor or agent running an MCP server" (a process-level cause), which on all-clean process checks sends users down the wrong hunt. A failure mode that says "stale runtime namespace detected — run daemon start recovery or delete /tmp/cbm-daemon-<uid>" would have been immediately actionable. Happy to help test a fix if you add a recovery command.

Thank you for the isolated reproduction and for the two independent confirmations. A markerless hard-linked socket and anchor pair that blocks every daemon and CLI start is a high-priority startup bug, and the labels now reflect that. Reclaiming runtime namespace artifacts intersects the daemon ownership and local security boundary, so we need more time to review the recovery rule carefully. No further diagnostics are requested at this stage.

Another manifestation of what looks like this same coordination-state family, on the same version and platform (0.10.8, macOS Apple Silicon, GitHub release archive), in case it helps narrow the cause. This one does not block startup — which is what made it hard to spot.

Symptom

The daemon starts fine, stays up, and serves read queries normally. But every index attempt fails for the lifetime of that daemon generation. Over one 4h43m window on this host:

57 read-only MCP calls succeeded (search_code, search_graph, get_code_snippet, query_graph, trace_path, index_status)
5 tool=index_repository status=error — the user-facing index tool, failing
2233 forked index workers, every one dying at startup with:

CBM index worker could not start: a pre-coordination or unverified CBM generation is active

Zero reap outcome=clean in the entire window. Restarting the daemon cleared it. It has recurred across daemon generations.

Where it aborts

src/main.c ~L2745, worker side:

int worker_seal_status = cbm_daemon_ipc_local_transition_seal_legacy(worker_transition);

None of the 2233 worker logs contain a version_cohort.uncoordinated_daemon line. Since that line is emitted only on worker_seal_status == 0, the return must have been -1, not 0. That narrows it to one of four things inside cbm_daemon_ipc_local_transition_seal_legacy (src/daemon/ipc.c ~L3256):

either posix_named_lock_is_retained check (startup_v2, or the shared legacy lock)
cbm_daemon_ipc_lifetime_reservation_probe returning -1
posix_stale_generation_cleanup_locked returning -1

All four are hard conditions, which is consistent with every fork failing identically rather than intermittently.

Each worker did log version_cohort.claimed_unheld with the expected build hash immediately before failing, so version-cohort admission itself was satisfied — the failure is downstream of it, in the local transition seal.

Runtime namespace state — caveat

I only inspected $TMPDIR/cbm-daemon-<uid>/ after the daemon restart that cleared the storm, so this describes the healthy successor generation, not the affected one. For what it is worth, the healthy shape is a hard-linked .sock/.anc pair plus a valid .sock.identity (104 bytes) and no .sock.pending, with all lock files mode 0600 owned by the euid.

I do not have the affected generation's namespace state, and I would not want that inferred from the above. If it recurs I will capture the directory before restarting anything — say what you would most want dumped and I will grab it.

Why I'm commenting rather than filing

I could not reduce it to a repro, and it may well be the same root cause as this issue or #1760 seen from a different angle — a third speculative ticket seemed like noise. Happy to file separately if you'd rather track it on its own.

I have filed the amplifier separately (#2015): the watcher re-forks a hard-failing index worker every ~5-7 s forever with no backoff, which is what turned this into 2233 forks instead of one logged warning. That fix is worth having independently of whatever causes the seal to fail, since it bounds the blast radius of any persistent worker failure.

Another data point on 0.10.8 (macOS, Apple Silicon). This time the blocked entrypoint was the stdio MCP server (editor-spawned), not daemon start/CLI/install.

Symptoms

stderr showed only level=info msg=version_cohort.claimed_unheld build=<build> followed by codebase-memory-mcp: CBM daemon could not start within 30000 ms (printed twice — two attempts). Nothing else.
No daemon-side logging was ever written across ~10 attempts over several days (cbm-daemon.log mtime stayed at the last clean shutdown), and no daemon child process was ever spawned — the in-process pre-launch coordination gate refuses before spawning.
daemon stop during the blocked window: daemon: not running (nothing to stop).

Re-entrant state
The wedged state self-healed once (a full healthy daemon session ran to a clean daemon.lifetime_end reason=stop_requested), then re-wedged on the next start days later. It is not a one-shot stuck state — it is re-entrant across generations.

Trigger correlation
cbm-daemon.log showed a worker-kill storm immediately before the state wedged (matches the "killed mid-generation" pivot in #1760):

level=warn msg=index.supervisor.worker_cancelled outcome=killed log=.../.worker-log-XXXXXX
level=info msg=index.supervisor.reap outcome=exit_nonzero exit_code=1 signal=0
level=warn msg=index.supervisor.worker_failed outcome=exit_nonzero exit_code=1 log=.../.worker-log-XXXXXX
level=warn msg=watcher.index.err project=<project>

followed by thousands of daemon.version_conflict entries in daemon-conflicts.ndjson, all active: pre-cohort/unknown build=000... / requested: 0.10.8.

Namespace state + recovery
Runtime namespace matched the markerless shape: hard-linked cbm-<id>.sock + cbm-<id>.anc (no listener; connect → ECONNREFUSED), no .sock.identity/.sock.pending, zero-byte lock files, only cbm-version-cohort-lifetime-v1.lock non-empty (214-byte CBMCOH record). rm -rf /tmp/cbm-daemon-<uid> recovered it immediately; index DBs untouched, daemon started on first attempt.

Codebase trace (HEAD aa44c28)
Consistent with the existing analysis: posix_stale_generation_cleanup_locked() (src/daemon/ipc.c:2281) returns 0 when the marker/pending records are absent but the stable socket remains — the stable socket is only unlinked in the POSIX_RECORD_VALID marker branch, so this crash window is never reclaimed. cbm_version_cohort_daemon_presence() (src/daemon/version_cohort.c) maps that to CBM_VERSION_COHORT_DAEMON_UNCOORDINATED, and the seal/presence gate (src/main.c ~L2734) logs a synthetic pre-cohort/unknown build conflict via cbm_version_cohort_log_uncoordinated_daemon() and refuses to launch. With no process running, "close all CBM sessions" and daemon stop are both dead ends.

@mvanhorn

Fixed on main by 760c08f (PR #1894 — thank you @mvanhorn), and it ships in v0.10.9. Thank you @silee9019 for the isolated, turnkey reproduction, and @LinRds, @lws2004 and @alekseysotnikov for the independent confirmations and the namespace listings that pinned the exact shape.

What was wrong: a hard-killed daemon can leave the hard-linked .sock/.anc pair behind with no .sock.identity or .sock.pending record, and the startup cleanup had no rule for that shape, so every later daemon start, CLI command and MCP session sat behind the 30 s "active or starting" timeout with daemon.ipc.listen_failed stage=namespace_validation in the log. The stale-generation cleanup now reclaims exactly that owner-private two-name/two-link shape — only under the serialized startup lock with the endpoint lifetime reserved, re-reading both names immediately before an inode-matched unlink, so a live daemon's socket or a replaced socket is never touched — and install/update run the same cleanup before classifying, which also removes the "active CBM sessions could not be stopped safely" dead end from #1760.

We re-ran the script from this report against the merged main: the markerless daemon start now returns 0 in about 6 s with a normal daemon.start in the log, and daemon status, cli --json list_projects and install all work from that state; a live daemon with its identity marker removed keeps its socket and keeps serving.

@halindrome — the index-worker seal walks the same cleanup, so your manifestation should be covered as well; if it recurs on v0.10.9, please open a separate issue with the $TMPDIR/cbm-daemon-<uid> listing captured before restarting anything (the re-fork backoff itself is #2015 / #2075).

If anyone still sees this on v0.10.9, please reopen with the output of codebase-memory-mcp daemon status and ls -la /tmp/cbm-daemon-$(id -u) (or your CBM_RUNTIME_DIR), and we will take it from there.

Navigation Menu

Repository navigation

macOS: markerless stale socket/anchor pair permanently blocks daemon startup #1757

Version

Platform

Install channel

Binary variant

What happened, and what did you expect?

Reproduction

Relevant daemon log

Additional evidence

Diagnostics and project scale

Confirmations

Blocking artifacts

Symptoms

Correlated prior state

Recovery

Symptom

Where it aborts

Runtime namespace state — caveat

Why I'm commenting rather than filing

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

macOS: markerless stale socket/anchor pair permanently blocks daemon startup #1757

Description

Version

Platform

Install channel

Binary variant

What happened, and what did you expect?

Reproduction

Relevant daemon log

Additional evidence

Diagnostics and project scale

Confirmations

Activity

LinRds commented on Aug 27, 2026

Blocking artifacts

Symptoms

Correlated prior state

Recovery

lws2004 commented on Sep 1, 2026

DeusData commented on Sep 1, 2026

halindrome commented on Sep 2, 2026

Symptom

Where it aborts

Runtime namespace state — caveat

Why I'm commenting rather than filing

alekseysotnikov commented on Sep 7, 2026

DeusData commented on Sep 7, 2026

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions