diff --git a/changelog.mdx b/changelog.mdx index 1d47dcc..0d83fd6 100644 --- a/changelog.mdx +++ b/changelog.mdx @@ -7,6 +7,25 @@ description: "Product changes and upgrade notes." agentsfleet is in **stealth-mode testing** and pre-production. APIs and agent behavior may change between releases without long deprecation windows. Email [agentsfleet@agentmail.to](mailto:agentsfleet@agentmail.to) if you want a hand calibrating an agent or to join as a design partner. + + ## Enable checkpoint resumption after testing + + This feature is coming soon. + + Checkpoint resumption now starts disabled until the deployment enables it. + + ## Upgrading + + Set `runner.recovery.resume_from_checkpoint = true` in the daemon configuration after testing, then restart every daemon replica. + This setting needs the matching server version; it requires no separate command-line interface (CLI) change. + + ## Bug fixes + + - **Recovery default** — Environment-only startup now uses the same disabled default as the deployment files. + + See [runner recovery](/runners/recovery#controlling-checkpoint-resumption) for rollout behavior. + + ## Recovery after a runner fails diff --git a/runners/install.mdx b/runners/install.mdx index 4356362..b55a883 100644 --- a/runners/install.mdx +++ b/runners/install.mdx @@ -3,7 +3,7 @@ title: Install a runner description: Put the runner binary and its signed toolbox on a Linux host, check the host, and start the runner as a service. type: how-to audience: operator -verified: 2026-11-09 +verified: 2026-11-19 product_version: 0.75.0 executable: false --- @@ -268,6 +268,90 @@ To remove the runner for good, revoke it under **Admin → Runners**. Deleting `/var/lib/agentsfleet-runner` removes its toolbox, repository mirrors and sandbox files. Unacknowledged results exist only in the running process and can be lost when that process stops. +## Runner-local settings + + +Runner profiles are coming with the next checkpoint recovery release. + + +The managed host deploy installs `/etc/agentsfleet/agentsfleet-runner.toml` and selects it through `AGENTSFLEET_RUNNER_CONFIG_FILE`. + +For a manually installed service: + +1. Create `/etc/agentsfleet/agentsfleet-runner.toml` using the settings below. +2. Add `AGENTSFLEET_RUNNER_CONFIG_FILE=/etc/agentsfleet/agentsfleet-runner.toml` to `/etc/default/agentsfleet-runner`. +3. Restart the runner to load the profile: + + ```bash + systemctl restart agentsfleet-runner + ``` + + ```text + (no output) + ``` + +Repeat the restart after later file changes. + +Without a selected file, the runner uses these defaults: + +```toml +[inference] +connect_timeout_seconds = 10 +stream_idle_timeout_seconds = 300 + +[provider_retries] +max_attempts = 3 +max_stream_open_attempts = 3 +initial_backoff_ms = 1000 +max_retry_after_seconds = 60 + +[supervisor] +daemon_max_attempts = 5 +daemon_initial_backoff_ms = 250 +daemon_max_backoff_ms = 30_000 +renewal_interval_ms = 5000 +authoring_renewal_interval_ms = 10_000 +authoring_renewal_retry_ms = 250 +max_concurrent_file_operations = 2 +``` + +| Table | Setting | Accepted values | Effect | +| --- | --- | --- | --- | +| `inference` | `connect_timeout_seconds` | 1–120 | Limits establishing a provider connection. | +| `inference` | `stream_idle_timeout_seconds` | 1–3600 | Limits silence between transport reads or parsed provider events. | +| `provider_retries` | `max_attempts` | 1–10 | Caps HTTP sends within each stream opening. | +| `provider_retries` | `max_stream_open_attempts` | 1–10 | Caps eligible openings before visible output or known spend. | +| `provider_retries` | `initial_backoff_ms` | 1–60,000 | Sets the first jittered exponential retry pause. | +| `provider_retries` | `max_retry_after_seconds` | 1–300 | Ends an attempt when the provider requests a longer wait. | +| `supervisor` | `daemon_max_attempts` | 1–10 | Caps sends for a bounded daemon operation. | +| `supervisor` | `daemon_initial_backoff_ms` | 1–`daemon_max_backoff_ms` | Sets the first jittered daemon retry pause. | +| `supervisor` | `daemon_max_backoff_ms` | 1–30,000 | Caps the exponential component before jitter. | +| `supervisor` | `renewal_interval_ms` | 100–5000 | Sets the batched host renewal interval. | +| `supervisor` | `authoring_renewal_interval_ms` | 100–10,000 | Sets the private authoring renewal interval. | +| `supervisor` | `authoring_renewal_retry_ms` | 1–250 | Sets the pause after a retryable authoring renewal failure. | +| `supervisor` | `max_concurrent_file_operations` | 1–16 | Caps admitted file captures and restores across this runner. | + +Attempt counts include the first send. +Provider HTTP retries and eligible stream reopenings have separate limits, allowing at most nine generation-request sends per turn with these defaults. +Native token-count requests are separate and send once per opening when needed. + +Known spend, visible output, and single-attempt requests prevent automatic stream reopening. + +Daemon connection recovery and owed-report delivery retry until cancellation or their deadline. +They use the configured backoff without a retry-count limit. +Earlier lease and turn deadlines still stop work. + +Jitter can extend a pause beyond `daemon_max_backoff_ms`. +Connection recovery also applies a compiled 30-second final pause limit. + +Required file restores wait for a runner-wide admission slot while lease renewal continues. +Optional captures skip a busy boundary and preserve the previous committed checkpoint. +Each admitted operation retains the compiled four-transfer window. + +The runner refuses unreadable files, unknown settings, invalid bounds, and renewal intervals longer than the supported safety windows. +Keep tokens in the environment file. +Worker count remains assigned per runner by the daemon. + ## Related pages - [Runners](/runners) diff --git a/runners/recovery.mdx b/runners/recovery.mdx index 6896b60..e9221ed 100644 --- a/runners/recovery.mdx +++ b/runners/recovery.mdx @@ -56,8 +56,11 @@ The daemon deployment selects a Tom's Obvious Minimal Language (TOML) configurat resume_from_checkpoint = false ``` -Environment-only startup defaults to resumption enabled. -Set this value explicitly when testing a staged rollout. +Resumption defaults to disabled, including environment-only startup. +Enable it after testing recovery with an eligible replacement runner. + +For environment-only startup, set `AGENTSFLEET_RESUME_FROM_CHECKPOINT=true` on each daemon replica and restart it. +When a selected TOML file supplies this setting, its value takes precedence over the environment. With this setting disabled, an interrupted ordinary run with a complete checkpoint waits after its lease expires.