From 8a562450597e6bfb7fc236fb784fb7d2c47edaa0 Mon Sep 17 00:00:00 2001 From: Kishorekumar Neelamegam Date: Wed, 18 Nov 2026 22:34:34 +0000 Subject: [PATCH 1/5] docs(runners): default checkpoint resumption to disabled --- changelog.mdx | 17 +++++++++++++++++ runners/recovery.mdx | 4 ++-- 2 files changed, 19 insertions(+), 2 deletions(-) diff --git a/changelog.mdx b/changelog.mdx index 1d47dcc..38626db 100644 --- a/changelog.mdx +++ b/changelog.mdx @@ -7,6 +7,23 @@ description: "Product changes and upgrade notes." agentsfleet is in **stealth-mode testing** and pre-production. APIs and agent behavior may change between releases without long deprecation windows. Email [agentsfleet@agentmail.to](mailto:agentsfleet@agentmail.to) if you want a hand calibrating an agent or to join as a design partner. + + ## Enable checkpoint resumption after testing + + Checkpoint resumption now starts disabled until the deployment enables it. + + ## Upgrading + + Set `runner.recovery.resume_from_checkpoint = true` in the daemon configuration after testing, then restart every daemon replica. + This setting needs the matching server version; it requires no separate command-line interface (CLI) change. + + ## Bug fixes + + - **Recovery default** — Environment-only startup now uses the same disabled default as the deployment files. + + See [runner recovery](/runners/recovery#controlling-checkpoint-resumption) for rollout behavior. + + ## Recovery after a runner fails diff --git a/runners/recovery.mdx b/runners/recovery.mdx index 6896b60..938129b 100644 --- a/runners/recovery.mdx +++ b/runners/recovery.mdx @@ -56,8 +56,8 @@ The daemon deployment selects a Tom's Obvious Minimal Language (TOML) configurat resume_from_checkpoint = false ``` -Environment-only startup defaults to resumption enabled. -Set this value explicitly when testing a staged rollout. +Resumption defaults to disabled, including environment-only startup. +Enable it after testing recovery with an eligible replacement runner. With this setting disabled, an interrupted ordinary run with a complete checkpoint waits after its lease expires. From b5ae6a86c95c407183cddd158322d5de1687d5e1 Mon Sep 17 00:00:00 2001 From: Kishorekumar Neelamegam Date: Wed, 18 Nov 2026 23:50:23 +0000 Subject: [PATCH 2/5] docs(runners): mark the recovery default correction as upcoming --- changelog.mdx | 2 ++ 1 file changed, 2 insertions(+) diff --git a/changelog.mdx b/changelog.mdx index 38626db..0d83fd6 100644 --- a/changelog.mdx +++ b/changelog.mdx @@ -10,6 +10,8 @@ description: "Product changes and upgrade notes." ## Enable checkpoint resumption after testing + This feature is coming soon. + Checkpoint resumption now starts disabled until the deployment enables it. ## Upgrading From c51e3a2274b2af1ab424087d2fcaeea8ddeb3e62 Mon Sep 17 00:00:00 2001 From: Kishorekumar Neelamegam Date: Thu, 19 Nov 2026 00:23:46 +0000 Subject: [PATCH 3/5] docs(runners): document local profiles and environment recovery opt-in --- runners/install.mdx | 51 +++++++++++++++++++++++++++++++++++++++++++- runners/recovery.mdx | 3 +++ 2 files changed, 53 insertions(+), 1 deletion(-) diff --git a/runners/install.mdx b/runners/install.mdx index 4356362..6e8a9e7 100644 --- a/runners/install.mdx +++ b/runners/install.mdx @@ -3,7 +3,7 @@ title: Install a runner description: Put the runner binary and its signed toolbox on a Linux host, check the host, and start the runner as a service. type: how-to audience: operator -verified: 2026-11-09 +verified: 2026-11-19 product_version: 0.75.0 executable: false --- @@ -268,6 +268,55 @@ To remove the runner for good, revoke it under **Admin → Runners**. Deleting `/var/lib/agentsfleet-runner` removes its toolbox, repository mirrors and sandbox files. Unacknowledged results exist only in the running process and can be lost when that process stops. +## Runner-local settings + + +Runner profiles are coming with the next checkpoint recovery release. + + +The managed host deploy installs `/etc/agentsfleet/agentsfleet-runner.toml` and selects it through `AGENTSFLEET_RUNNER_CONFIG_FILE`. +For a manually installed service, create the file and add that variable to `/etc/default/agentsfleet-runner`. +Restart the runner after changing the file. + +Without a selected file, the runner uses these defaults: + +```toml +[inference] +connect_timeout_seconds = 10 +stream_idle_timeout_seconds = 300 + +[provider_retries] +max_attempts = 3 +max_stream_open_attempts = 3 +initial_backoff_ms = 1000 +max_retry_after_seconds = 60 + +[supervisor] +daemon_max_attempts = 5 +daemon_initial_backoff_ms = 250 +daemon_max_backoff_ms = 30_000 +renewal_interval_ms = 5000 +authoring_renewal_interval_ms = 10_000 +authoring_renewal_retry_ms = 250 +max_concurrent_file_operations = 2 +``` + +Attempt counts include the first send. +Provider HTTP retries and eligible stream reopenings have separate limits, allowing at most nine physical sends with these defaults. +Known spend, visible output, and single-attempt requests prevent automatic stream reopening. + +Daemon connection recovery and owed-report delivery retry until cancellation or their deadline. +They use the configured backoff without a retry-count limit. +Earlier lease and turn deadlines still stop work. + +Required file restores wait for a runner-wide admission slot while lease renewal continues. +Optional captures skip a busy boundary and preserve the previous committed checkpoint. +Each admitted operation retains the compiled four-transfer window. + +The runner refuses unreadable files, unknown settings, invalid bounds, and renewal intervals longer than the supported safety windows. +Keep tokens in the environment file. +Worker count remains assigned per runner by the daemon. + ## Related pages - [Runners](/runners) diff --git a/runners/recovery.mdx b/runners/recovery.mdx index 938129b..e9221ed 100644 --- a/runners/recovery.mdx +++ b/runners/recovery.mdx @@ -59,6 +59,9 @@ resume_from_checkpoint = false Resumption defaults to disabled, including environment-only startup. Enable it after testing recovery with an eligible replacement runner. +For environment-only startup, set `AGENTSFLEET_RESUME_FROM_CHECKPOINT=true` on each daemon replica and restart it. +When a selected TOML file supplies this setting, its value takes precedence over the environment. + With this setting disabled, an interrupted ordinary run with a complete checkpoint waits after its lease expires. Saving new checkpoints continues. From 855b3202e1a3d5894e2b8b189591f808ed3dfe57 Mon Sep 17 00:00:00 2001 From: Kishorekumar Neelamegam Date: Thu, 19 Nov 2026 00:27:30 +0000 Subject: [PATCH 4/5] docs(runners): specify profile bounds and installation steps --- runners/install.mdx | 34 ++++++++++++++++++++++++++++++++-- 1 file changed, 32 insertions(+), 2 deletions(-) diff --git a/runners/install.mdx b/runners/install.mdx index 6e8a9e7..301ce16 100644 --- a/runners/install.mdx +++ b/runners/install.mdx @@ -275,8 +275,22 @@ Runner profiles are coming with the next checkpoint recovery release. The managed host deploy installs `/etc/agentsfleet/agentsfleet-runner.toml` and selects it through `AGENTSFLEET_RUNNER_CONFIG_FILE`. -For a manually installed service, create the file and add that variable to `/etc/default/agentsfleet-runner`. -Restart the runner after changing the file. + +For a manually installed service: + +1. Create `/etc/agentsfleet/agentsfleet-runner.toml` using the settings below. +2. Add `AGENTSFLEET_RUNNER_CONFIG_FILE=/etc/agentsfleet/agentsfleet-runner.toml` to `/etc/default/agentsfleet-runner`. +3. Restart the runner to load the profile: + + ```bash + systemctl restart agentsfleet-runner + ``` + + ```text + (no output) + ``` + +Repeat the restart after later file changes. Without a selected file, the runner uses these defaults: @@ -301,6 +315,22 @@ authoring_renewal_retry_ms = 250 max_concurrent_file_operations = 2 ``` +| Table | Setting | Accepted values | Effect | +| --- | --- | --- | --- | +| `inference` | `connect_timeout_seconds` | 1–120 | Limits establishing a provider connection. | +| `inference` | `stream_idle_timeout_seconds` | 1–3600 | Limits silence between transport reads or parsed provider events. | +| `provider_retries` | `max_attempts` | 1–10 | Caps HTTP sends within each stream opening. | +| `provider_retries` | `max_stream_open_attempts` | 1–10 | Caps eligible openings before visible output or known spend. | +| `provider_retries` | `initial_backoff_ms` | 1–60,000 | Sets the first jittered exponential retry pause. | +| `provider_retries` | `max_retry_after_seconds` | 1–300 | Ends an attempt when the provider requests a longer wait. | +| `supervisor` | `daemon_max_attempts` | 1–10 | Caps sends for a bounded daemon operation. | +| `supervisor` | `daemon_initial_backoff_ms` | 1–`daemon_max_backoff_ms` | Sets the first jittered daemon retry pause. | +| `supervisor` | `daemon_max_backoff_ms` | 1–30,000 | Caps daemon retry pauses. | +| `supervisor` | `renewal_interval_ms` | 100–5000 | Sets the batched host renewal interval. | +| `supervisor` | `authoring_renewal_interval_ms` | 100–10,000 | Sets the private authoring renewal interval. | +| `supervisor` | `authoring_renewal_retry_ms` | 1–250 | Sets the pause after a retryable authoring renewal failure. | +| `supervisor` | `max_concurrent_file_operations` | 1–16 | Caps admitted file captures and restores across this runner. | + Attempt counts include the first send. Provider HTTP retries and eligible stream reopenings have separate limits, allowing at most nine physical sends with these defaults. Known spend, visible output, and single-attempt requests prevent automatic stream reopening. From 7e91d1716ce1daff8d214ff0e418fd54a2ce50d5 Mon Sep 17 00:00:00 2001 From: Kishorekumar Neelamegam Date: Thu, 19 Nov 2026 00:33:49 +0000 Subject: [PATCH 5/5] docs(runners): distinguish generation retries and jitter limits --- runners/install.mdx | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-) diff --git a/runners/install.mdx b/runners/install.mdx index 301ce16..b55a883 100644 --- a/runners/install.mdx +++ b/runners/install.mdx @@ -325,20 +325,25 @@ max_concurrent_file_operations = 2 | `provider_retries` | `max_retry_after_seconds` | 1–300 | Ends an attempt when the provider requests a longer wait. | | `supervisor` | `daemon_max_attempts` | 1–10 | Caps sends for a bounded daemon operation. | | `supervisor` | `daemon_initial_backoff_ms` | 1–`daemon_max_backoff_ms` | Sets the first jittered daemon retry pause. | -| `supervisor` | `daemon_max_backoff_ms` | 1–30,000 | Caps daemon retry pauses. | +| `supervisor` | `daemon_max_backoff_ms` | 1–30,000 | Caps the exponential component before jitter. | | `supervisor` | `renewal_interval_ms` | 100–5000 | Sets the batched host renewal interval. | | `supervisor` | `authoring_renewal_interval_ms` | 100–10,000 | Sets the private authoring renewal interval. | | `supervisor` | `authoring_renewal_retry_ms` | 1–250 | Sets the pause after a retryable authoring renewal failure. | | `supervisor` | `max_concurrent_file_operations` | 1–16 | Caps admitted file captures and restores across this runner. | Attempt counts include the first send. -Provider HTTP retries and eligible stream reopenings have separate limits, allowing at most nine physical sends with these defaults. +Provider HTTP retries and eligible stream reopenings have separate limits, allowing at most nine generation-request sends per turn with these defaults. +Native token-count requests are separate and send once per opening when needed. + Known spend, visible output, and single-attempt requests prevent automatic stream reopening. Daemon connection recovery and owed-report delivery retry until cancellation or their deadline. They use the configured backoff without a retry-count limit. Earlier lease and turn deadlines still stop work. +Jitter can extend a pause beyond `daemon_max_backoff_ms`. +Connection recovery also applies a compiled 30-second final pause limit. + Required file restores wait for a runner-wide admission slot while lease renewal continues. Optional captures skip a busy boundary and preserve the previous committed checkpoint. Each admitted operation retains the compiled four-transfer window.