Skip to content

Restart-safe transport: stateless MCP HTTP, bounded serve retry, gateway pause (MN-REQ-06.19) - #261

Merged
chouswei merged 4 commits into
masterfrom
feat/session-safe-restart
Nov 9, 2026
Merged

chouswei merged 4 commits into
masterfrom
feat/session-safe-restart

Conversation

@chouswei

@chouswei chouswei commented Nov 9, 2026 •

Copy link
Copy Markdown
Owner

Why

Owner goal (t1305u): "all the products we have, need a way to upgrade them without break the sessions, it can pause, but sessions must work after the upgrade."

This is the MemNet slice. A restart or drain of memnet serve, memnet-mcp-http or the product gateway now pauses callers by bounded retry. It does not drop MCP sessions or fail in-flight reads, and a write is never silently replayed.

What

(a) Stateless MCP HTTP. FastMCP("memnet", stateless_http=True, json_response=True) in memnet_mcp/server.py, the only construction site. A restart, or a second process behind the same front, drops no MCP session: the next tools/call works without an Mcp-Session-Id header and without re-initialising. The [mcp] extra floor moves from mcp>=1.2 to mcp>=1.8, the first release with stateless_http / streamable_http_app. This changes a dependency floor, not the package version.

(b) Bounded retry in ServePool.send. The new memnet/serve_retry.py holds the verb classification, the serve_draining parse, the budget and the backoff. The loop lives in ServePool.send.

  • Not delivered (connect refused or reset, or the frame not fully written): retry any verb. The serve acts only on a whole length-prefixed frame, so a failed sendall means it never ran the request.
  • @ERR: serve_draining|retry_after_s=N: the serve refused before any work, so retry any verb after at least N s.
  • Reset after the whole frame was written: retry READ only. A WRITE surfaces the error and is delivered exactly once.
  • Backoff is 0.1 s doubling, capped at 2 s, with jitter. The total pause is capped by MEMNET_SERVE_RETRY_S (default 30 s; 0 = off). Each wait is clamped to the budget that remains. On exhaustion, the last connect error or the draining envelope surfaces.
  • Not retried: a connect timeout (keeps the old serve_timeout envelope) and an over-cap reply frame (deterministic).

(c) Gateway and MCP tcp client. Both reach the serve via send_command → POOL.send, but the conductor pre-audit missed one thing: both gated on probe() first. ProductGateway._version returned None (→ 502) when a 0.2 s probe failed, and cached that None. run_memnet in tcp mode returned serve_required. So a POOL retry alone would never have run on those paths during a restart. Both probe gates are removed, and a failed version check is no longer cached. There is one retry loop, and the gateway adds no budget of its own (no nested multiplication).

Placement keeps instant fail-over. send_command takes retry_s and passes it to ServePool.send. When _select_new places a new session, every candidate but the last checks its version with retry_s=0, so a backend that stays down is tried once and skipped. Only the last candidate pauses through a restart. Pinned backends (house, backend, owned session) and _forward keep the full budget.

Audit finding for (a)

There is no per-MCP-session state. grep over parts/memnet-mcp/software/memnet_mcp finds no FastMCP Context / ctx / get_context / request_context, no server→client notifications or progress, and no lifespan state. The MemNet session is a tool argument (session=). MEMNET_SESSION is a process-wide env fallback, not per-connection. So stateless mode loses nothing, and nothing needs persisting.

Verb classification summary (memnet.serve_retry)

  • READ (retry after send is safe): version, guide, examples …, map|tagmap show|fields, relations list, session current|list|resume|expire-status, read list, query pin-map|warm|find|walk|typed-walk|neighbors|path|context, housekeep stats|stale|orphans|dangling|recyclable, export pin-map without --out, admin usage-report / the admin_usage envelope.
  • WRITE (never replayed after send): everything else, including mutate, add, update, delete, import-slice, session open|save|load|close|acl-*, reserve|extend|release, ingest *, snap *, housekeep prune *, export pin-map --out, admin checkpoint|upgrade-prepare, the upgrade_prepare envelope, and any unknown verb (fail closed).

Tests

New tests/test_serve_restart_retry.py uses a scripted fake serve on real loopback sockets with a fake clock, so no real sleeps:

  • test_connect_refused_then_accept_succeeds_for_read_and_write[read|write]
  • test_draining_envelope_honours_retry_after_then_succeeds[read|write] (slept ≥ retry_after_s; delivered twice, run once)
  • test_reset_after_send_on_write_is_not_retried (exactly one delivery, no sleep)
  • test_reset_after_send_on_read_is_retried_and_succeeds
  • test_oversized_reply_on_read_is_not_retried
  • test_budget_is_bounded_then_the_connect_error_surfaces (total wait = 30 s, each wait ≤ 2 s)
  • test_draining_past_the_budget_returns_the_refusal, test_retry_off_by_env
  • test_read_verbs, test_write_verbs_and_unknown_fail_closed, test_request_kind_admin_flags
  • test_gateway_pauses_through_backend_restart: a real ProductGateway; the backend is down at session open and comes up 0.3 s later; then it restarts between calls, and an owned mutate succeeds with exactly one delivery.
  • test_gateway_placement_fails_over_past_a_dead_backend_fast: two backends with a 30 s budget. a stays down and b is up. An unpinned session open lands on b in under 2 s. With the full budget on a, the test fails at 30.0 s, so the test bites.

In tests/test_mcp_http.py:

  • test_memnet_mcp_is_stateless_json
  • test_tools_call_works_after_restart_without_session_header: two real memnet-mcp --transport streamable-http processes on one port. After the restart, a tools/call with no session header and no initialize gets 200 and a result.

tests/test_sysml_restart_safe_transport.py checks that the model, the code constants and the test names are aligned.

The tests bite. I ran four mutations, and each turns a new test red: FastMCP("memnet") stateful, budget 0, every request READ, and the gateway probe gate restored.

Existing tests that assert the raw refusal (serve_draining visible, serve_required, serve_unreachable, gateway 502 for a backend that stays down) now opt out explicitly with MEMNET_SERVE_RETRY_S=0 or a short budget. The behaviour they check is unchanged.

Model

MN-REQ-06.19 MN_REQ_06_19_RestartSafeTransport, MN-VER-06-S17 MN_VER_06_S17_RestartSafeTransport.

  • implementation.sysml: new ServeRetryMod (memnet.serve_retry) with allocation serveRetryToMod; McpServerMod gains statelessHttp / jsonResponse.
  • deploy.sysml: new ServeClientRetry nested as TcpServeBridge.clientRetry; the gateway gains pausesThroughBackendRestart.
  • behaviour.sysml: new action def ServeRetryChoice.
  • ssot-to-code-allocate-map.md: new row.
  • MN-REQ-06.14 / S12 "no client retry" wording now points at 06.19.
  • Docs: docs/operations/safe-upgrade.md gets a "Restart-safe clients" section; the gateway contract's 502 row is updated.
  • CHANGELOG [Unreleased]. No version bump.

Open questions

  • Stale keep-alive socket. In the race where a pooled socket's peer died after the FIN check, a WRITE on it errors rather than being retried. This is safe, and the window is small, because the serve closes idle sockets on shutdown.
  • LocalIpcGateway (AF_UNIX). Its send_command is separate and not covered. It is not one of the restarted network services.

No merge/release by bots — owner merges.

Restart or drain of serve, memnet-mcp-http or the gateway pauses
callers by bounded retry; no MCP session lost; a write is never
replayed after send. ServeRetryMod (memnet.serve_retry) with
allocation serveRetryToMod, ServeClientRetry on TcpServeBridge,
ServeRetryChoice action, McpServerMod statelessHttp/jsonResponse.
MN-REQ-06.14 'no client retry' now points at 06.19.
FastMCP(stateless_http=True, json_response=True). Audit: no tool keeps
per-MCP-session state (no Context/ctx, no notifications); the MemNet
session is a tool argument. A tools/call after a restart works with no
Mcp-Session-Id header. [mcp] extra floor mcp>=1.8 (first release with
stateless_http).
…Q-06.19 b+c)

memnet.serve_retry classifies serve requests READ / WRITE (unknown is
WRITE). ServePool.send retries a request that was not delivered and a
serve_draining refusal (honouring retry_after_s) for any verb, and a
reset after send for READ only, within MEMNET_SERVE_RETRY_S (default
30 s, 0 off), with backoff and jitter. The gateway and run_memnet no
longer gate on probe(), and the gateway caches no failed version check,
so both reach the retry. Tests: tests/test_serve_restart_retry.py.
…se (MN-REQ-06.19 c)

send_command takes retry_s and passes it to ServePool.send. In
_select_new every candidate but the last checks its version with
retry_s=0, so a backend that stays down fails over at once; the last
candidate, pinned backends and _forward keep the full budget.
@chouswei
chouswei merged commit 4f5386e into master Nov 9, 2026
2 checks passed
@chouswei chouswei mentioned this pull request Nov 10, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant