Skip to content

Upgrade safety: upgrade-verify, one-step rollback, cross-version and mixed-client tests - #257

Merged
chouswei merged 4 commits into
masterfrom
feat/upgrade-safety
Nov 4, 2026
Merged

chouswei merged 4 commits into
masterfrom
feat/upgrade-safety

Conversation

@chouswei

@chouswei chouswei commented Nov 4, 2026

Copy link
Copy Markdown
Owner

Why

Owner goal (0.20.5 rollout, held by Devicor): an upgrade must not lose any session or row. Clients on 0.20.4 and 0.20.5 must keep working against either serve. A health and data check must pass before the upgrade counts as done, and a failure exits non-zero. Rollback must be one step: one command that brings back the old venv and the saved sessions. Credentials (tickets) belong to Atelier and are out of scope here.

What

  • memnet admin upgrade-verify --before <manifest> runs in the client. It asks the serve for a checkpoint into a scratch dir, which is removed afterwards, so it also works against a 0.20.4 serve. It compares session ids and rows/relations per session. Exit 3 on any mismatch; exit 2 for no serve, a wrong token or a bad file.
  • scripts/upgrade-rollback.sh is the one-step rollback:
    • stop the serve;
    • move the state dir aside (nothing is deleted);
    • restore the saved .pre-upgrade copy, with its manifest marked not retired;
    • swap the venv symlink back to the old venv;
    • start, and wait for every session to be restored.
  • serve_client.py: when the CLI runs a command in its own process, it now returns the command's real exit code instead of 0.
  • docs/operations/safe-upgrade.md:
    • new step: after stopping the serve, copy the state dir to .pre-upgrade;
    • the unit must run through a venv symlink;
    • new sections for verify and rollback.
  • CHANGELOG [Unreleased].

Tests (real processes)

Owner point Test
1. No session or row lost test_upgrade_cross_version.py::test_old_serve_upgrades_to_new_and_rolls_back: a real 0.20.4 serve (built from 0a9c7c5) with 3 live sessions → upgrade-prepare → stop → new code on the same port and state dir → session ids, rows/relations and query output match
2. Mixed-version clients test_mixed_version_clients.py: old client against new serve; pooled client against old serve; pool recovery when the serve closes; new → old → new
3. Health/data check, non-zero on failure test_upgrade_verify.py (6 tests: pass, mismatch names, rows/relations/count failures, bad file or no serve, wrong token, scratch cleanup), plus the real run in the cross-version test
4. One-step rollback test_upgrade_rollback.py (6 tests) plus the real rollback in the cross-version test, where the 0.20.4 serve comes back with every session

Locally, on master 51c351c: these 17 pass, and ruff is clean.

LATER

  • Decide whether the deploy model should record the host-side rollback script. helperRollsBack is pinned absent, so the model is unchanged.
  • Pool race: if the serve closes a reused socket between the pool's check and the read, the client gets a ConnectionError and does not retry. A retry could run a mutate twice.
  • After a rollback, rows can only be checked with the new venv's upgrade-verify. The script itself checks the restore count.
  • scripts/ has 23 ruff errors and a ruff format crash; both were there before this PR.

chouswei and others added 4 commits November 4, 2026 13:57
…ion clients, upgrade-verify, one-step rollback (P-upgrade-safety)

A 0.20.4 build (commit 0a9c7c5) is unpacked from git history into a temp venv; the tests
skip with a reason on a shallow clone, and CI now checks out full history.

Co-Authored-By: Claude <noreply@anthropic.com>
…ent steps assert no @err, teardown kills the rolled-back serve first (P-upgrade-safety)

Co-Authored-By: Claude <noreply@anthropic.com>
…), scripts/upgrade-rollback.sh, doc and CHANGELOG (E-upgrade-safety)

upgrade-verify asks the running serve (any version) for a checkpoint into a scratch
dir and compares session ids and rows/edges per session with the saved manifest.
The rollback script swaps the venv link back, restores the saved state dir with the
manifest un-retired, sets the new state aside, and waits for a clean old restore.
In-process CLI runs now return their typer.Exit code instead of 0.

Co-Authored-By: Claude <noreply@anthropic.com>
@chouswei
chouswei merged commit 28d125f into master Nov 4, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant