Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 8 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,14 @@ This project uses Semantic Versioning as **interpreted for MemNet**: package `a.

## [Unreleased]

### Added

- **Restart-safe transport** — an upgrade or restart of `memnet serve`, memnet-mcp-http or the product gateway now pauses callers instead of breaking them. The client pool (`ServePool.send`, used by the CLI, the memnet-mcp tcp client and the gateway) retries a refused connection, a reset before the request was fully sent, and `@ERR: serve_draining|retry_after_s=N` (waiting at least `N`), with exponential backoff and jitter, for at most `MEMNET_SERVE_RETRY_S` (default 30 s; `0` turns it off). A reset after the request was sent is retried for reads only; a write is never replayed. One module, `memnet.serve_retry`, says which commands are reads; anything else, including an unknown command, counts as a write. memnet-mcp streamable-http is stateless with JSON replies, so a restart drops no MCP session. The gateway no longer probes before a call or caches a failed version check. When it places a new session, only the last candidate backend pauses; a down backend with another behind it is tried once and skipped, so fail-over stays instant. MN-REQ-06.19, MN-VER-06-S17.

### Changed

- The `[mcp]` extra now needs `mcp>=1.8` (stateless streamable-http).

## [0.20.6] - 2026-11-04

### Added
Expand Down
2 changes: 1 addition & 1 deletion docs/operations/product-gateway-contract.md
Original file line number Diff line number Diff line change
Expand Up @@ -66,7 +66,7 @@ HTTP status is a hint. The body is always the envelope. Backend engine errors st
| 404 | `gateway_unconfigured` | `GET <path>/admin/counts` when `admin.sha256` is unset |
| 409 | `gateway_backend_version_mismatch` | `memnet version` is not the product pin |
| 413 | `gateway_body_too_large` | Body above `body_max_bytes` |
| 502 | `gateway_backend_unreachable` | TCP probe failed, the version command did not answer, the client wait expired, or `session list` and no backend answered |
| 502 | `gateway_backend_unreachable` | The backend stayed unreachable past the restart pause (`MEMNET_SERVE_RETRY_S`, default 30 s; MN-REQ-06.19), the version command did not answer, the client wait expired, or `session list` and no backend answered |

No HTTP response (the process exits 2 instead):

Expand Down
16 changes: 15 additions & 1 deletion docs/operations/safe-upgrade.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,21 @@

Upgrade `memnet-serve` without dropping a loaded session (MN-REQ-06.14). The procedure stays inside the serve process: save every session, restart on the new version, reload every session. The agent loop does not change: cue, then `pin_map`, then `mutate`.

During the restart, clients see `@ERR: serve_draining|retry_after_s=<seconds>` or a connection error. There is no client retry window.
During the restart, the serve answers `@ERR: serve_draining|retry_after_s=<seconds>` or refuses the connection. Clients pause through it (MN-REQ-06.19); see [Restart-safe clients](#restart-safe-clients).

## Restart-safe clients

`memnet` CLI, `memnet-mcp` over tcp and the product gateway all reach the serve through one client pool. That pool pauses through a restart or a drain rather than failing:

- **Connection refused, or reset before the request was fully sent**: the serve never read it, so any request is retried.
- **`serve_draining`**: the serve refused before doing any work, so any request is retried after at least `retry_after_s`.
- **Reset after the request was sent**: the serve may have run it. A read (`pin_map`, `find`, `typed_walk`, `session_list`, …) is retried. A write (`mutate`, `session_open`, `session_save`, ingest, snap, ACL, reserve, admin, …) is not: the caller gets the error, and the write is never replayed. Unknown commands count as writes.

Waits use exponential backoff with jitter (0.1 s doubling, capped at 2 s). The total pause is capped by `MEMNET_SERVE_RETRY_S` (default 30 s, comfortably longer than a restart; `0` turns the pause off). When the budget runs out, the last error or the draining reply is returned.

The product gateway's placement of a new session checks each candidate backend's version in turn. Only the last candidate gets the pause. Every earlier one is tried once with no retry, so a backend that stays down fails over to the next at once, as before. Calls to a pinned backend (house, `backend` or an owned session) pause for the full budget.

`memnet-mcp --transport streamable-http` is stateless (`stateless_http`, JSON replies). No MCP session lives in the process, so a restart of memnet-mcp-http, or a second process behind the same front, drops no MCP session: the next `tools/call` works without re-initialising. MemNet sessions are tool arguments and live in the serve.

## Procedure

Expand Down
3 changes: 2 additions & 1 deletion parts/common/memnet/memnet/serve.py
Original file line number Diff line number Diff line change
Expand Up @@ -388,6 +388,7 @@ def send_command(
admin_token: str | None = None,
admin_usage: bool = False,
timeout: float | None = None,
retry_s: float | None = None,
) -> dict[str, Any]:
host = host or serve_host()
port = port or serve_port()
Expand All @@ -408,7 +409,7 @@ def send_command(
)
from memnet.serve_pool import POOL

return POOL.send(host, port, payload_obj, wait)
return POOL.send(host, port, payload_obj, wait, retry_s=retry_s)


def _peer_gone(sock: socket.socket) -> bool:
Expand Down
101 changes: 95 additions & 6 deletions parts/common/memnet/memnet/serve_pool.py
Original file line number Diff line number Diff line change
Expand Up @@ -4,14 +4,23 @@

import json
import os
import random
import select
import socket
import struct
import threading
import time
from collections.abc import Callable
from typing import Any

from memnet.config import serve_max_frame_bytes
from memnet.serve_retry import (
RequestKind,
backoff_s,
draining_retry_after,
request_kind,
retry_budget_s,
)

_OFF_VALUES = frozenset({"0", "false", "no", "off"})
_TIMEOUT_ENVELOPE: dict[str, Any] = {
Expand Down Expand Up @@ -56,11 +65,33 @@ def _close(sock: socket.socket) -> None:
pass


class _NotDelivered(Exception):
"""The serve cannot have read the request (connect failed or frame not fully sent)."""

def __init__(self, cause: OSError) -> None:
super().__init__(str(cause))
self.cause = cause


class _ReplyTooLarge(ConnectionError):
"""The reply frame is over the cap: deterministic, so never retried."""


class ServePool:
"""Idle sockets per (host, port), LIFO; each socket is checked out exclusively."""

def __init__(self, idle_s: float = 30.0) -> None:
def __init__(
self,
idle_s: float = 30.0,
*,
sleep: Callable[[float], None] = time.sleep,
clock: Callable[[], float] = time.monotonic,
rng: random.Random | None = None,
) -> None:
self._idle_s = idle_s
self._sleep = sleep
self._clock = clock
self._rng = rng
self._lock = threading.Lock()
self._idle: dict[tuple[str, int], list[tuple[socket.socket, float]]] = {}
self._pid = os.getpid()
Expand Down Expand Up @@ -116,7 +147,21 @@ def probe(self, host: str, port: int, timeout: float = 0.2) -> bool:
self._park(key, sock)
return True

def send(self, host: str, port: int, payload: dict, timeout: float) -> dict:
def send(
self,
host: str,
port: int,
payload: dict,
timeout: float,
*,
retry_s: float | None = None,
) -> dict:
"""One serve request. Pauses through a restart or drain (MN-REQ-06.19).

Retries within ``retry_s`` (default ``MEMNET_SERVE_RETRY_S``) when the
request was not delivered or the serve answered serve_draining. A reset
after the frame was written is retried for READ requests only.
"""
keep = keepalive_enabled()
body_obj = dict(payload)
if keep:
Expand All @@ -132,7 +177,44 @@ def send(self, host: str, port: int, payload: dict, timeout: float) -> dict:
}
frame = struct.pack(">I", len(data)) + data
key = (host, port)
budget = retry_budget_s() if retry_s is None else max(0.0, retry_s)
deadline = self._clock() + budget
attempt = 0
while True:
failure: OSError | None = None
reply: dict | None = None
try:
reply = self._send_once(key, frame, timeout, keep, max_frame)
except _NotDelivered as exc:
failure = exc.cause
wait = backoff_s(attempt, self._rng)
except OSError as exc:
if isinstance(exc, _ReplyTooLarge) or request_kind(payload) is not RequestKind.READ:
raise
failure = exc
wait = backoff_s(attempt, self._rng)
else:
retry_after = draining_retry_after(reply)
if retry_after is None:
return reply
wait = max(retry_after, backoff_s(attempt, self._rng))
remaining = deadline - self._clock()
if remaining <= 0:
if failure is not None:
raise failure
return reply # type: ignore[return-value]
self._sleep(min(wait, remaining))
attempt += 1

def _send_once(
self,
key: tuple[str, int],
frame: bytes,
timeout: float,
keep: bool,
max_frame: int,
) -> dict:
"""One attempt. Raises ``_NotDelivered`` when the serve cannot have read the frame."""
sock: socket.socket | None = self._checkout(key) if keep else None
try:
if sock is not None:
Expand All @@ -145,12 +227,19 @@ def send(self, host: str, port: int, payload: dict, timeout: float) -> dict:
_close(sock)
sock = None
if sock is None:
sock = socket.create_connection(key, timeout=timeout)
sock.setsockopt(socket.IPPROTO_TCP, socket.TCP_NODELAY, 1)
sock.sendall(frame)
try:
sock = socket.create_connection(key, timeout=timeout)
sock.setsockopt(socket.IPPROTO_TCP, socket.TCP_NODELAY, 1)
sock.sendall(frame)
except TimeoutError:
raise
except OSError as exc:
# Refused, reset, or the frame was not fully written: the serve acts
# only on a whole length-prefixed frame, so it did not run this one.
raise _NotDelivered(exc) from exc
(length,) = struct.unpack(">I", _recv_exact(sock, 4))
if length > max_frame:
raise ConnectionError(f"response frame {length} bytes exceeds cap {max_frame}")
raise _ReplyTooLarge(f"response frame {length} bytes exceeds cap {max_frame}")
reply = json.loads(_recv_exact(sock, length).decode("utf-8"))
except TimeoutError:
if sock is not None:
Expand Down
133 changes: 133 additions & 0 deletions parts/common/memnet/memnet/serve_retry.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,133 @@
"""Restart-safe client retry for ``memnet serve`` requests (MN-REQ-06.19).

One place says which serve requests are safe to send twice. ``ServePool.send``
asks it after a failure:

- not delivered (connect refused or reset, or the frame was never fully
written): retry any request, because the serve never read it;
- ``@ERR: serve_draining|retry_after_s=N``: the serve refused before any work,
so retry any request after ``N`` seconds;
- reset after the whole frame was written: the serve may have run it, so retry
READ only. A WRITE surfaces the error and is never replayed.

The wait is bounded by ``MEMNET_SERVE_RETRY_S`` (default 30 s; ``0`` turns
retry off).
"""

from __future__ import annotations

import os
import random
import re
from enum import Enum
from typing import Any

ENV_RETRY_BUDGET = "MEMNET_SERVE_RETRY_S"
DEFAULT_RETRY_BUDGET_S = 30.0
BACKOFF_BASE_S = 0.1
BACKOFF_CAP_S = 2.0

_DRAINING_RE = re.compile(r"^@ERR: serve_draining\|retry_after_s=(\d+(?:\.\d+)?)\s*$", re.M)


class RequestKind(Enum):
READ = "read"
WRITE = "write"


# (argv[0], argv[1]) pairs that only read. ``None`` as argv[1] means the verb
# has no subcommand. Anything not listed is a WRITE: unknown verbs fail closed.
_READ_VERBS: frozenset[tuple[str, str | None]] = frozenset(
{
("version", None),
("guide", None),
("examples", None),
("examples", "map"),
("examples", "workflow"),
("examples", "add"),
("examples", "path"),
("examples", "agent-guide"),
("tagmap", "fields"),
("tagmap", "show"),
("map", "fields"),
("map", "show"),
("relations", "list"),
("session", "current"),
("session", "list"),
("session", "resume"),
("session", "expire-status"),
("read", "list"),
("query", "pin-map"),
("query", "warm"),
("query", "find"),
("query", "walk"),
("query", "typed-walk"),
("query", "neighbors"),
("query", "path"),
("query", "context"),
("housekeep", "stats"),
("housekeep", "stale"),
("housekeep", "orphans"),
("housekeep", "dangling"),
("housekeep", "recyclable"),
("admin", "usage-report"),
}
)
_SUBCOMMAND_VERBS = frozenset(v for v, sub in _READ_VERBS if sub is not None) | {
"session",
"admin",
"export",
"ingest",
"snap",
}


def argv_kind(argv: Any) -> RequestKind:
"""READ when every effect of ``argv`` is a read; WRITE otherwise."""
if not isinstance(argv, list) or not argv:
return RequestKind.WRITE
words = [str(a) for a in argv]
verb = words[0]
sub = words[1] if len(words) > 1 and verb in _SUBCOMMAND_VERBS else None
if (verb, sub) == ("export", "pin-map"):
# --out writes a file on the serve host.
return (
RequestKind.WRITE
if any(w == "--out" or w.startswith("--out=") for w in words)
else RequestKind.READ
)
return RequestKind.READ if (verb, sub) in _READ_VERBS else RequestKind.WRITE


def request_kind(payload: dict[str, Any]) -> RequestKind:
"""Kind of one serve envelope (``args`` plus the admin flags)."""
if payload.get("upgrade_prepare") is True:
return RequestKind.WRITE
if payload.get("admin_usage") is True:
return RequestKind.READ
return argv_kind(payload.get("args"))


def draining_retry_after(reply: Any) -> float | None:
"""``retry_after_s`` of a serve_draining refusal, else ``None``."""
if not isinstance(reply, dict) or int(reply.get("exit_code", 0) or 0) == 0:
return None
match = _DRAINING_RE.search(str(reply.get("stderr") or ""))
return float(match.group(1)) if match else None


def retry_budget_s() -> float:
raw = (os.environ.get(ENV_RETRY_BUDGET) or "").strip()
if not raw:
return DEFAULT_RETRY_BUDGET_S
try:
value = float(raw)
except ValueError:
return DEFAULT_RETRY_BUDGET_S
return max(0.0, value)


def backoff_s(attempt: int, rng: random.Random | None = None) -> float:
"""Exponential backoff with jitter: half to all of ``base * 2**attempt``, capped."""
ceiling = min(BACKOFF_CAP_S, BACKOFF_BASE_S * (2 ** min(attempt, 16)))
return ceiling * (0.5 + 0.5 * (rng or random).random())
10 changes: 6 additions & 4 deletions parts/memnet-mcp/software/memnet_mcp/client.py
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@
from dataclasses import dataclass

from memnet.in_process_engine import run_argv
from memnet.serve import probe, send_command
from memnet.serve import send_command
from memnet_mcp.parse import extract_errors, extract_session_id

_SERVE_REQUIRED = "@ERR: serve_required|run memnet serve in another terminal first"
Expand Down Expand Up @@ -131,10 +131,12 @@ def run_memnet(
mode = _transport()

if mode == "tcp":
if probe():
# No probe gate: send_command pauses through a serve restart (MN-REQ-06.19).
try:
raw = send_command(full_argv, stdin=stdin)
return MemNetResponse.from_raw(raw, session_hint=session)
return MemNetResponse.serve_required(session_hint=session)
except OSError:
return MemNetResponse.serve_required(session_hint=session)
return MemNetResponse.from_raw(raw, session_hint=session)

raw = run_argv(full_argv, stdin=stdin)
return MemNetResponse.from_raw(raw, session_hint=session)
Loading
Loading