Repository navigation
Seamless upgrades: hand the running core over to its successor - #370
Merged
Merged
Conversation
Contract for the seamless-upgrade work: the handover link and socket transfer (tw-handover), the gateway's snapshot and settlement hooks, and POST /handover/arm with Status.handover / Status.draining. Bodies are todo!() and are filled in by the parallel lanes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Endpoint: a unix socket (0600) in the data directory on macOS/Linux; on Windows a random loopback port plus a 32-byte token from the system RNG. Closing removes the files only if they are still ours, so a draining core cannot delete its successor's endpoint. - accept checks the peer: same uid via getpeereid/SO_PEERCRED (pid from LOCAL_PEERPID/SO_PEERCRED); on Windows a token frame within 5 seconds, compared in constant time. connect returns None when nothing is armed. - Frames are a 4-byte big-endian length plus JSON, at most 16 MiB. - Listeners travel with SCM_RIGHTS on unix (sendmsg/recvmsg under async_io, CLOEXEC, MSG_CTRUNC checked, TCP vs unix told apart by address family) and with WSADuplicateSocketW/WSASocketW on Windows. - Tests: queued connections on a handed-over TCP or unix listener all reach the receiver after the sender closes its handle; socket order across frames; oversized and garbled frames; endpoint lifecycle. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…drain The old core exports its in-memory state as JSON (handover_export) and the new core imports it (handover_import): conversation affinity and session grouping, sealed-history and cache-breakpoint refusals, breaker and recent success rates, rate-limit pauses, latency samples, pace ratios, load-balance rounds, OAuth access tokens, key and upstream usage books with in-flight reservations, and the concurrency places the old core holds per key and per upstream. Instants travel as remaining/elapsed milliseconds; unreadable sections or entries are dropped without failing the handover. The new core holds the old core's places as phantom passes and slots (debt when it has no free place), so the two processes together stay within each limit. Key places count queued requests too, so a request the old core had queued is not overtaken by a new one in the new core. begin_drain reports the full set of holds, then every place freed or taken and every reservation taken, settled, released or dropped; usage changes are buffered from the snapshot on so none are lost before commit. handover_settled applies them, handover_release_all drops whatever is left. While draining, Responses WebSocket connections close with 1012 once the turn in hand has finished; idle ones close at once. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…lane and gateway listeners The state machine for upgrading without dropping connections, in twcore: - POST /handover/arm: refused over the remote port, in safe mode, while a ChatGPT or Z.ai sign-in is pending, or while a successor is taking over. Once armed the core stops watching --parent, opens the handover endpoint, moves stdout/stderr off the desktop app's pipe into twcore.handover.log, and waits (120 s by default); it relaunches the app once, then drains and exits the current way if nobody comes (H7). - Old side: Hello -> Control (the control plane stops accepting) -> Snapshot (after pending token write-backs and sync uploads) -> WantListeners (no time limit, H3) -> Listeners -> Commit within 30 s. Any failure before commit goes back to armed (F4); three failures after the listeners were handed over make the next successor fall back (F5). After commit it stops accepting (own descriptors only), stops background tasks, disowns the lock, sends Settled / InFlight while draining (30 min limit, then aborts the rest), and says Bye. - New side: when the lock holder is alive it connects to the handover endpoint; Refused or an unreadable reply makes it ask for Fallback and wait for the old core to exit (H2). It serves the control plane on the inherited socket, takes only the control plane while its configuration does not load (H3), accepts on the inherited listeners, commits within 20 s, claims the lock and follows the old core's drain. - The request history opens only after the previous core exits; records queue in memory meanwhile (10,000 records or 256 MB, oldest dropped). Request ids jump past the previous core's by a large gap so the two never overwrite each other's rows. - Background tasks run on their own runtime that is shut down at commit. Model probing and balance reads stay on the main runtime: they share the upstream connection pool with forwarding, and closing their runtime cut a stream in flight. tw-gateway listen.rs gains serve_with and Handoff (export, hold, follow, stop accepting); the remote control port and the local control plane get the same handles. Status reports handover and draining. End-to-end tests start two twcore processes; they are ignored until the handover link and gateway hooks are merged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…st and test wording Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…nd balance reads once draining Renewing a refresh token invalidates the previous one, so two cores renewing independently leaves one of them signed out. The old core stops renewing when it hands its tokens over (resumes if the takeover fails); the successor starts with renewals held and takes them over at commit. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…out of child processes - Endpoint::accept now runs each connection's gate check in its own task (at most 64 at once, the oldest dropped when full) and hands over the first one that passes. On Windows the gate is a token within 5 seconds and anyone on the machine can reach the loopback port, so checking one connection at a time let silent connections keep the real successor out. Pending checks are aborted when accept returns or is cancelled. - Windows: clear HANDLE_FLAG_INHERIT on sockets restored from WSAPROTOCOL_INFOW, on the endpoint's listener and on both ends of the handover connection; std's CreateProcess always inherits handles. - Document the macOS window between recvmsg and FD_CLOEXEC. - Tests: silent and wrong-token connections (more than the limit) do not delay or impersonate the right one; the same through a real endpoint; on Windows, handover sockets are not inheritable. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…e, settlement while queued - A successor that exits while its predecessor still drains no longer deletes the lock. The successor records the draining core (pid and the request ids it may use) in twcore.draining; the lock is removed only if it still holds our pid and nobody is draining. A core that starts fresh and finds a live draining core opens the request history only after it exits, numbers requests past it and reports handover = draining. A successor that exits before the history opened logs how many queued events were not recorded. - Commits are acknowledged (Message::CommitAck). The old core replies before it stops accepting and never answers a commit after its 30 s limit; the successor counts as primary only on the ack or once the old process is gone, and otherwise exits. - OAuth renewals pause only after WantListeners, so a successor waiting for its configuration (H3) no longer blocks renewals on the core that is still forwarding. Pausing waits for renewals already in flight; the old core then sends a second, fresh snapshot (also re-imported against the real configuration after H3), and a later import replaces OAuth tokens from an earlier one. - While a successor queues records, a placeholder recorder settles them so key and upstream limits keep counting its own requests; on opening, the queue is replayed without settling and usage is rebuilt from the store. - Handing over the control plane closes connections already made to the old core, so clients reconnect to the successor. - An unreadable greeting is answered with Refused. - An arm is called off when the original parent is still alive at the deadline (the desktop app did not restart): endpoint closed, parent watch restored, back to idle. Arming starts a session of its own on unix. - Sign-ins that start after arming are refused (control.handover.login_refused). - The successor waits 30 s for each snapshot; a successor that leaves right after the first snapshot counts as a strike. - twcore.handover.log is appended to (rotated past 4 MB); the lock file is replaced atomically; nothing after the commit makes the successor exit; the successor releases the old core's slots only once its process is gone. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…er tests more time to start under load Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Upgrades no longer drop the gateway port or cut requests in flight. A running core hands its listening sockets and runtime state to the core that replaces it, the way nginx and Envoy do hot upgrades.
How it works
POST /handover/arm. The core stops following its parent process and opens a handover endpoint in the data directory (a unix socket; on Windows a loopback port plus a token file).SCM_RIGHTSon unix,WSADuplicateSocketWon Windows), so connections queued but not yet accepted are picked up by the new core instead of being reset.Commit; the old core acknowledges, stops accepting, finishes its requests (up to 30 minutes) and exits. While both run, slots and usage are settled across the link so limits are not doubled.Anything that fails before the commit leaves the old core serving. A different handover protocol version falls back to the previous behaviour (wait for requests, then restart).
Notable details
POST /handover/arm(HandoverArm->HandoverArmed);Status.handoverandStatus.draining.tw-handover: the handover endpoint, the identity check (same uid on unix, a token on Windows), framing, and socket passing.twcore.drainingmarker keeps the next core from opening the history while the old core is still draining.Testing
tw-handover, which passes queued connections without losing any, as well as for the gateway snapshot and settlement, the listener handoff and the control plane.bin/twcore/tests/handover.rs) run twotwcoreprocesses:🤖 Generated with Claude Code