From c503304424bb8a07b27edd127d77abbc938bba39 Mon Sep 17 00:00:00 2001 From: Kishorekumar Neelamegam Date: Wed, 18 Nov 2026 01:33:57 +0000 Subject: [PATCH 1/6] docs(m247): explain checkpoint recovery and runner cordons --- changelog.mdx | 24 +++++++++ runners.mdx | 1 + runners/recovery.mdx | 120 +++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 145 insertions(+) create mode 100644 runners/recovery.mdx diff --git a/changelog.mdx b/changelog.mdx index e96c5fc..5344d48 100644 --- a/changelog.mdx +++ b/changelog.mdx @@ -7,6 +7,30 @@ description: "Product changes and upgrade notes." agentsfleet is in **stealth-mode testing** and pre-production. APIs and agent behavior may change between releases without long deprecation windows. Email [agentsfleet@agentmail.to](mailto:agentsfleet@agentmail.to) if you want a hand calibrating an agent or to join as a design partner. + + ## Recovery after a runner fails + + This feature is coming soon. + + Unfinished runs can resume from their last complete checkpoint when an eligible runner is available. + + ## Upgrading + + Upgrade the server, runners, dashboard, and command-line interface (CLI) together. + Finish running tasks first; older task checkpoint formats cannot resume. + + ## What's new + + - **Recovery** — Safe boundaries save progress without a manual checkpoint command. Work after the last saved boundary can repeat. + - **Chat** — Detected replacement attempts show **Run restarted** and replace the provisional reply. + + ## CLI + + - **Runner status** — `agentsfleet platform runners show ` shows automatic quarantine separately from operator status. + + See [runner recovery](/runners/recovery) for timing and limits. + + ## Terminal confirmation and Orbit diff --git a/runners.mdx b/runners.mdx index 9ed9c00..ba96c2f 100644 --- a/runners.mdx +++ b/runners.mdx @@ -307,6 +307,7 @@ The runner's lifetime totals count every lease it took, so they can exceed the h ## Related pages +- [Recovery after a runner fails](/runners/recovery) - [Install a runner](/runners/install) - [Install a fleet](/fleets/install) - [Run and control a fleet](/fleets/running) diff --git a/runners/recovery.mdx b/runners/recovery.mdx new file mode 100644 index 0000000..8002590 --- /dev/null +++ b/runners/recovery.mdx @@ -0,0 +1,120 @@ +--- +title: Recovery after a runner fails +description: Checkpoints, automatic quarantine, and the runner status commands. +type: explanation +audience: operator +verified: 2026-11-18 +product_version: 0.78.0 +executable: false +--- + +This feature is coming soon. + +## What it is + +An eligible runner can resume unfinished work from its last complete checkpoint after the previous runner fails. +You do not issue a checkpoint command. + +The control plane saves ordinary API-only runs in encrypted PostgreSQL rows. +Durable tasks and checkpoints with workspace files also require the configured private object store. +Recovery keeps the saved conversation, accepted answers, local memory changes, and matching workspace files. + +## Why it exists + +A failed host can lose its local files and running processes. +Checkpoints keep completed progress outside that host so another eligible runner can continue the run. + +## How it behaves + +### Saving progress + +The runner saves at a safe boundary before a continuing model call. +Running children, open sessions, active file work, or tenant processes prevent that boundary from being saved. +An incomplete upload never replaces the last complete checkpoint. + +### Replacing a failed runner + +The failed runner's lease must expire before another runner takes the work. +The normal deadline is 35 seconds after the last confirmed renewal. +Capacity, permissions, funds, approval checks, and restore time can delay continuation. + +An ordinary event can receive at most three leases. +If all three lapse without a final report, its next claim ends it with an error. + +Durable tasks default to twelve leases across the task. +Recovery uses that existing allowance; once it is exhausted, continuation stops. + +```mermaid +sequenceDiagram + participant A as Runner A + participant C as Control plane + participant B as Eligible runner B + A->>C: Save a complete checkpoint + Note over A: Host fails + C->>C: Wait for the lease deadline + C->>B: Assign unfinished work + B->>C: Load the saved checkpoint + B->>C: Continue and report the result +``` + +A run without a completed checkpoint starts again from its original input and unsettled messages. +A changed setup, unreadable checkpoint, or missing file stops recovery before the model or tools start. + +The chat shows **Run restarted** when it receives a restart notice or sees an attempt newer than the one already displayed. +Opening a stream halfway through an attempt does not itself add that notice. +The new attempt replaces the provisional reply; the final saved reply remains authoritative. + +### Reading quarantine status + +After 105 seconds without heartbeats, a background sweep marks the runner offline and adds automatic quarantine. +This quarantine stops new assignments while preserving the separate operator status. +Thirty seconds of continuous valid, healthy heartbeats clears automatic quarantine. + +A heartbeat gap of 20 seconds or a degraded heartbeat resets the return check. +Clearing automatic quarantine preserves an operator's cordon, drain, or revocation. + +Use credentials with `runner:read` permission. +Copy `` from `agentsfleet platform runners list`. +Install `jq` for this filtered JSON example. + +```bash +agentsfleet platform runners show --json | jq '{id, status, liveness, cordon}' +``` + +For an automatically quarantined host, the output resembles: + +```text +{ + "id": "", + "status": "active", + "liveness": "offline", + "cordon": { + "reason": "missed_heartbeats", + "cordoned_at": "" + } +} +``` + +The runner response omits `cordon` when automatic quarantine is absent. +The filter above then shows `"cordon": null`. + +- `agentsfleet platform runners list` shows connection state and automatic quarantine across hosts. +- `agentsfleet platform runners events ` shows cordon and return events. +- `agentsfleet platform runners leases ` shows the host's work history. + +## Limits + +Work after the last complete checkpoint can repeat, including external actions. +Make those actions safe to repeat. +Checkpoints do not restore process memory or files outside the recovery workspace. + +Workspace recovery requires a snapshot-capable Linux sandbox and configured private file storage. +A workspace still being built cannot publish a history-only checkpoint. + +Upgrade the server, runners, dashboard, and command-line interface (CLI) together. +Finish running tasks before upgrading; older task checkpoint formats cannot resume under this release. + +## Related pages + +- [Runner policy and states](/runners) +- [Install a runner](/runners/install) From e23ec5710d67a60c3cfe2b3be4bab5815a41153e Mon Sep 17 00:00:00 2001 From: Kishorekumar Neelamegam Date: Wed, 18 Nov 2026 02:04:46 +0000 Subject: [PATCH 2/6] docs: limit recovery guidance to platform runners --- changelog.mdx | 2 +- runners/recovery.mdx | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/changelog.mdx b/changelog.mdx index 5344d48..1d47dcc 100644 --- a/changelog.mdx +++ b/changelog.mdx @@ -12,7 +12,7 @@ description: "Product changes and upgrade notes." This feature is coming soon. - Unfinished runs can resume from their last complete checkpoint when an eligible runner is available. + Unfinished runs on platform runners can resume from their last complete checkpoint when an eligible runner is available. ## Upgrading diff --git a/runners/recovery.mdx b/runners/recovery.mdx index 8002590..543a329 100644 --- a/runners/recovery.mdx +++ b/runners/recovery.mdx @@ -12,7 +12,7 @@ executable: false ## What it is -An eligible runner can resume unfinished work from its last complete checkpoint after the previous runner fails. +An eligible platform runner can resume unfinished work from its last complete checkpoint after the previous runner fails. You do not issue a checkpoint command. The control plane saves ordinary API-only runs in encrypted PostgreSQL rows. From 4c4b235f0768e8a0c93cc8a356d71c1e3f55389b Mon Sep 17 00:00:00 2001 From: Kishorekumar Neelamegam Date: Wed, 18 Nov 2026 06:44:28 +0000 Subject: [PATCH 3/6] docs(runners): explain when a file checkpoint can be skipped --- runners/recovery.mdx | 2 ++ 1 file changed, 2 insertions(+) diff --git a/runners/recovery.mdx b/runners/recovery.mdx index 543a329..23199fb 100644 --- a/runners/recovery.mdx +++ b/runners/recovery.mdx @@ -30,6 +30,8 @@ Checkpoints keep completed progress outside that host so another eligible runner The runner saves at a safe boundary before a continuing model call. Running children, open sessions, active file work, or tenant processes prevent that boundary from being saved. + +When the runner is busy transferring checkpoints, it can skip a file checkpoint and keep the last complete one. An incomplete upload never replaces the last complete checkpoint. ### Replacing a failed runner From 850c80a920a62bc3d63a1f246b3bee2be153aded Mon Sep 17 00:00:00 2001 From: Kishorekumar Neelamegam Date: Wed, 18 Nov 2026 07:35:26 +0000 Subject: [PATCH 4/6] docs(runners): describe waiting for file restore capacity --- runners/recovery.mdx | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/runners/recovery.mdx b/runners/recovery.mdx index 23199fb..a502ac7 100644 --- a/runners/recovery.mdx +++ b/runners/recovery.mdx @@ -40,6 +40,10 @@ The failed runner's lease must expire before another runner takes the work. The normal deadline is 35 seconds after the last confirmed renewal. Capacity, permissions, funds, approval checks, and restore time can delay continuation. +With several interrupted runs, eligible runners can take work in parallel as workers become available. +A replacement runner keeps its lease active while waiting for file transfer capacity. +It restores the saved files before its next model or tool action. + An ordinary event can receive at most three leases. If all three lapse without a final report, its next claim ends it with an error. From 9921b2e3b94bdc9a5516f9e3f3b01f78ee0b7196 Mon Sep 17 00:00:00 2001 From: Kishorekumar Neelamegam Date: Wed, 18 Nov 2026 10:20:47 +0000 Subject: [PATCH 5/6] docs(runners): explain the checkpoint resumption switch --- runners/recovery.mdx | 26 ++++++++++++++++++++++++++ 1 file changed, 26 insertions(+) diff --git a/runners/recovery.mdx b/runners/recovery.mdx index a502ac7..6a7fbee 100644 --- a/runners/recovery.mdx +++ b/runners/recovery.mdx @@ -38,12 +38,38 @@ An incomplete upload never replaces the last complete checkpoint. The failed runner's lease must expire before another runner takes the work. The normal deadline is 35 seconds after the last confirmed renewal. +The deployment can configure the lease lifetime. + Capacity, permissions, funds, approval checks, and restore time can delay continuation. With several interrupted runs, eligible runners can take work in parallel as workers become available. A replacement runner keeps its lease active while waiting for file transfer capacity. It restores the saved files before its next model or tool action. +### Controlling checkpoint resumption + +The daemon deployment selects a Tom's Obvious Minimal Language (TOML) configuration file with +`AGENTSFLEET_CONFIG_FILE` or `agentsfleetd --config serve`. + +```toml +[runner.recovery] +resume_from_checkpoint = false +``` + +With this setting disabled, an interrupted ordinary run with a complete checkpoint +waits after its lease expires. +Saving new checkpoints continues. + +Set the value to `true` and roll out the configuration to allow an eligible runner +to reclaim that admission and restore its saved progress. +Confirm the value on every daemon replica after the rollout completes. + +Runs already holding a valid lease can continue. +Ordinary runs without a complete checkpoint and durable task recovery keep their +existing retry behavior. + +### Retry limits + An ordinary event can receive at most three leases. If all three lapse without a final report, its next claim ends it with an error. From 22ac2bc4bf32395c8652d2ce3dd2d9ca2eafd81b Mon Sep 17 00:00:00 2001 From: Kishorekumar Neelamegam Date: Wed, 18 Nov 2026 11:02:51 +0000 Subject: [PATCH 6/6] docs(runners): link shipped account APIs and clarify rollout defaults --- docs.json | 10 ++++++++++ runners/recovery.mdx | 3 +++ 2 files changed, 13 insertions(+) diff --git a/docs.json b/docs.json index f13028f..d124f2a 100644 --- a/docs.json +++ b/docs.json @@ -203,6 +203,16 @@ "DELETE /v1/tenants/me/models/{id}" ] }, + { + "group": "Account runners", + "pages": [ + "POST /v1/tenants/me/runners", + "GET /v1/tenants/me/runners", + "GET /v1/tenants/me/runners/{runner_id}", + "PATCH /v1/tenants/me/runners/{runner_id}", + "DELETE /v1/tenants/me/runners/{runner_id}" + ] + }, { "group": "Runner administration", "pages": [ diff --git a/runners/recovery.mdx b/runners/recovery.mdx index 6a7fbee..6896b60 100644 --- a/runners/recovery.mdx +++ b/runners/recovery.mdx @@ -56,6 +56,9 @@ The daemon deployment selects a Tom's Obvious Minimal Language (TOML) configurat resume_from_checkpoint = false ``` +Environment-only startup defaults to resumption enabled. +Set this value explicitly when testing a staged rollout. + With this setting disabled, an interrupted ordinary run with a complete checkpoint waits after its lease expires. Saving new checkpoints continues.