Repository navigation
Restart-safe transport: stateless MCP HTTP, bounded serve retry, gateway pause (MN-REQ-06.19) - #261
Merged
Merged
Conversation
Restart or drain of serve, memnet-mcp-http or the gateway pauses callers by bounded retry; no MCP session lost; a write is never replayed after send. ServeRetryMod (memnet.serve_retry) with allocation serveRetryToMod, ServeClientRetry on TcpServeBridge, ServeRetryChoice action, McpServerMod statelessHttp/jsonResponse. MN-REQ-06.14 'no client retry' now points at 06.19.
FastMCP(stateless_http=True, json_response=True). Audit: no tool keeps per-MCP-session state (no Context/ctx, no notifications); the MemNet session is a tool argument. A tools/call after a restart works with no Mcp-Session-Id header. [mcp] extra floor mcp>=1.8 (first release with stateless_http).
…Q-06.19 b+c) memnet.serve_retry classifies serve requests READ / WRITE (unknown is WRITE). ServePool.send retries a request that was not delivered and a serve_draining refusal (honouring retry_after_s) for any verb, and a reset after send for READ only, within MEMNET_SERVE_RETRY_S (default 30 s, 0 off), with backoff and jitter. The gateway and run_memnet no longer gate on probe(), and the gateway caches no failed version check, so both reach the retry. Tests: tests/test_serve_restart_retry.py.
…se (MN-REQ-06.19 c) send_command takes retry_s and passes it to ServePool.send. In _select_new every candidate but the last checks its version with retry_s=0, so a backend that stays down fails over at once; the last candidate, pinned backends and _forward keep the full budget.
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Owner goal (t1305u): "all the products we have, need a way to upgrade them without break the sessions, it can pause, but sessions must work after the upgrade."
This is the MemNet slice. A restart or drain of
memnet serve, memnet-mcp-http or the product gateway now pauses callers by bounded retry. It does not drop MCP sessions or fail in-flight reads, and a write is never silently replayed.What
(a) Stateless MCP HTTP.
FastMCP("memnet", stateless_http=True, json_response=True)inmemnet_mcp/server.py, the only construction site. A restart, or a second process behind the same front, drops no MCP session: the nexttools/callworks without anMcp-Session-Idheader and without re-initialising. The[mcp]extra floor moves frommcp>=1.2tomcp>=1.8, the first release withstateless_http/streamable_http_app. This changes a dependency floor, not the package version.(b) Bounded retry in
ServePool.send. The newmemnet/serve_retry.pyholds the verb classification, theserve_drainingparse, the budget and the backoff. The loop lives inServePool.send.sendallmeans it never ran the request.@ERR: serve_draining|retry_after_s=N: the serve refused before any work, so retry any verb after at leastNs.MEMNET_SERVE_RETRY_S(default 30 s;0= off). Each wait is clamped to the budget that remains. On exhaustion, the last connect error or the draining envelope surfaces.serve_timeoutenvelope) and an over-cap reply frame (deterministic).(c) Gateway and MCP tcp client. Both reach the serve via
send_command→POOL.send, but the conductor pre-audit missed one thing: both gated onprobe()first.ProductGateway._versionreturnedNone(→ 502) when a 0.2 s probe failed, and cached thatNone.run_memnetin tcp mode returnedserve_required. So a POOL retry alone would never have run on those paths during a restart. Both probe gates are removed, and a failed version check is no longer cached. There is one retry loop, and the gateway adds no budget of its own (no nested multiplication).Placement keeps instant fail-over.
send_commandtakesretry_sand passes it toServePool.send. When_select_newplaces a new session, every candidate but the last checks its version withretry_s=0, so a backend that stays down is tried once and skipped. Only the last candidate pauses through a restart. Pinned backends (house,backend, owned session) and_forwardkeep the full budget.Audit finding for (a)
There is no per-MCP-session state.
grepoverparts/memnet-mcp/software/memnet_mcpfinds no FastMCPContext/ctx/get_context/request_context, no server→client notifications or progress, and no lifespan state. The MemNet session is a tool argument (session=).MEMNET_SESSIONis a process-wide env fallback, not per-connection. So stateless mode loses nothing, and nothing needs persisting.Verb classification summary (
memnet.serve_retry)version,guide,examples …,map|tagmap show|fields,relations list,session current|list|resume|expire-status,read list,query pin-map|warm|find|walk|typed-walk|neighbors|path|context,housekeep stats|stale|orphans|dangling|recyclable,export pin-mapwithout--out,admin usage-report/ theadmin_usageenvelope.mutate,add,update,delete,import-slice,session open|save|load|close|acl-*,reserve|extend|release,ingest *,snap *,housekeep prune *,export pin-map --out,admin checkpoint|upgrade-prepare, theupgrade_prepareenvelope, and any unknown verb (fail closed).Tests
New
tests/test_serve_restart_retry.pyuses a scripted fake serve on real loopback sockets with a fake clock, so no real sleeps:test_connect_refused_then_accept_succeeds_for_read_and_write[read|write]test_draining_envelope_honours_retry_after_then_succeeds[read|write](slept ≥retry_after_s; delivered twice, run once)test_reset_after_send_on_write_is_not_retried(exactly one delivery, no sleep)test_reset_after_send_on_read_is_retried_and_succeedstest_oversized_reply_on_read_is_not_retriedtest_budget_is_bounded_then_the_connect_error_surfaces(total wait = 30 s, each wait ≤ 2 s)test_draining_past_the_budget_returns_the_refusal,test_retry_off_by_envtest_read_verbs,test_write_verbs_and_unknown_fail_closed,test_request_kind_admin_flagstest_gateway_pauses_through_backend_restart: a realProductGateway; the backend is down atsession openand comes up 0.3 s later; then it restarts between calls, and an ownedmutatesucceeds with exactly one delivery.test_gateway_placement_fails_over_past_a_dead_backend_fast: two backends with a 30 s budget.astays down andbis up. An unpinnedsession openlands onbin under 2 s. With the full budget ona, the test fails at 30.0 s, so the test bites.In
tests/test_mcp_http.py:test_memnet_mcp_is_stateless_jsontest_tools_call_works_after_restart_without_session_header: two realmemnet-mcp --transport streamable-httpprocesses on one port. After the restart, atools/callwith no session header and no initialize gets 200 and a result.tests/test_sysml_restart_safe_transport.pychecks that the model, the code constants and the test names are aligned.The tests bite. I ran four mutations, and each turns a new test red:
FastMCP("memnet")stateful, budget 0, every request READ, and the gateway probe gate restored.Existing tests that assert the raw refusal (
serve_drainingvisible,serve_required,serve_unreachable, gateway 502 for a backend that stays down) now opt out explicitly withMEMNET_SERVE_RETRY_S=0or a short budget. The behaviour they check is unchanged.Model
MN-REQ-06.19
MN_REQ_06_19_RestartSafeTransport, MN-VER-06-S17MN_VER_06_S17_RestartSafeTransport.implementation.sysml: newServeRetryMod(memnet.serve_retry) withallocation serveRetryToMod;McpServerModgainsstatelessHttp/jsonResponse.deploy.sysml: newServeClientRetrynested asTcpServeBridge.clientRetry; the gateway gainspausesThroughBackendRestart.behaviour.sysml: newaction def ServeRetryChoice.ssot-to-code-allocate-map.md: new row.docs/operations/safe-upgrade.mdgets a "Restart-safe clients" section; the gateway contract's 502 row is updated.Open questions
LocalIpcGateway(AF_UNIX). Itssend_commandis separate and not covered. It is not one of the restarted network services.No merge/release by bots — owner merges.