Skip to content

Latest commit

 

History

History
1150 lines (912 loc) · 50.2 KB

deployment.md

File metadata and controls

1150 lines (912 loc) · 50.2 KB

Deployment Guide

Two deployment modes: personal use (quick, current user) and production (separate user, hardened). Pick the one that fits.

EggPool's canonical public runtime is the native Rust executable delivered through the PyPI wheel. Qualified targets are Linux x86_64, Linux aarch64, and macOS arm64; Windows and other unqualified targets are unsupported. See Upgrade and rollback for exact cross-era transitions and the standalone raw-binary authority.

Personal Use (Recommended for LAN/Raspberry Pi)

Runs under your current user with your existing config. Not intended for public-facing deployments.

1. One-shot install

bash -o pipefail -c 'curl -fsSL https://raw.githubusercontent.com/eggstack/eggpool/main/scripts/install.sh | bash' && export PATH="$HOME/.local/bin:$PATH"
eggpool onboard
sudo env "PATH=$PATH" "HOME=$HOME" "$(command -v eggpool)" deploy systemd --install

PATH and HOME are passed explicitly because sudo resets both to root's environment on distributions that set always_set_home in sudoers (Debian and Ubuntu do by default). The personal unit runs as the invoking user, so it must resolve that user's configuration, data directory, and home. Without HOME, the unit is written with User= the invoking user but HOME= root's home, and the service cannot read the configuration it was installed against.

The installer is binary-first for fresh native installations. It downloads the executable for the detected target from GitHub raw, verifies its SHA-256, runs a staged self-check, and installs it as standalone-rust. It does not require Python, uv, pipx, pip, Cargo, or a source checkout. Direct installs with uv tool install eggpool, pipx install eggpool, and cargo install eggpool remain supported alternatives.

When the installer finds an existing EggPool on PATH, it preserves the installation owner: native installs use eggpool update, legacy Python-era installs use their owning manager, and source-checkout or ambiguous ownership fails closed. Use --force only to repair a verified EggPool installation with the same owner; unrelated files are not overwritten. A fresh install seeds the default config from the example template only when no config exists, then prints the resolved config path.

After install, eggpool onboard walks the operator through provider connections, validates the configuration, prompts for an administrator password on a new database, creates the initial local administrator, and starts the server. It prints the administrator's personal API token once. That token is the inference credential. The same run does not write a new [server].api_key. If the server is already running, onboarding reports that without restarting it. Browser sign-in is /login on the listen address. For unattended bootstrap, pass --password-stdin and provide the password on stdin; the password is never accepted in argv. Secret output is suppressed when stdout is not a terminal unless --show-secret is explicitly passed. Existing databases keep their legacy authentication behavior until an operator explicitly initializes local user access.

Manage local identities from the machine that owns the data directory with eggpool users add USER --password-stdin, eggpool users show USER, eggpool users disable USER, eggpool users enable USER, eggpool users token USER, eggpool users revoke USER, and eggpool users reset-password USER. Member passwords are read from a hidden TTY prompt or stdin; passwords are never accepted in argv. Add --profile-id PROFILE to assign the initial model policy in the same transaction. eggpool users defaults show|set PROFILE|clear manages the instance default used only when future members are created without an explicit profile; users add USER --no-profile explicitly keeps a new member unassigned. Existing assignments are never changed by setting or clearing the default. No budget template is copied: user and instance budget rules remain independently configured. Personal tokens are for inference, model discovery, and the integration profile; they do not authorize dashboard or admin pages, and they cannot add or remove a personal upstream account. On a small team, an administrator grants personal-upstream self-service from the users page. A granted member submits one bundled-provider key on My account. The key is written to the operator environment file and is not stored in the database. Denying the grant does not remove an account that is already bound. Adding an account for someone else stays eggpool users upstream add. The configured server API key remains a legacy shared inference credential only while the database is legacy_unmigrated. Do not share it with new users; issue an individual token instead.

With [dashboard].enabled = true, an administrator can manage users, profiles, budgets, tokens, and audit history in the embedded dashboard. New members receive a temporary password and must change it at first login. Members can manage their personal tokens and view only their own usage and accessible models. The users page can use the instance default, select a specific profile, or explicitly leave a member without a profile. A browser-issued token is shown once in a no-store response; set it as EGGPOOL_API_KEY in the environment that launches the desktop client.

The administrator-only /admin/audit page includes a Security events table of recent authentication and authorization outcomes, aggregated by minute. The local writer is bounded and non-blocking: if its queue is full, it drops an event rather than delaying authentication. Rows are capped at 30 days and 10,000 records. The table may show an existing user's name resolved from its internal ID, but stored event rows do not contain usernames, credentials, tokens, raw IP addresses, prompts, headers, or request bodies. This is not a complete access log, alerting system, or addition to eggpool audit export.

Remote browser login must use the operator's TLS proxy and trusted-proxy setup below: LAN reachability alone does not encrypt passwords or cookies.

Budget rules are estimated by default. Promoting a rule to hard requires proof the proxy can actually bound: hard token rules need clients to send an explicit output cap, and hard USD rules need a conservative operator tariff on every provider those users can reach, configured as [providers.<id>.budget] max_microdollars_per_million_tokens (microdollars per million tokens, covering the most expensive model on that provider; omit for unknown price, 0 for verified free). Without a bound, hard-mode requests are denied before upstream dispatch: hard mode is unavailable rather than approximate. See model limits.

An optional instance-wide ceiling (eggpool users global-budget set or the dashboard instance section) caps total consumption across all users and is intersected with every per-user rule. It covers unattributed legacy-key requests as well, survives user disable/delete, and needs no second database or writer. See model limits.

Updating EggPool

eggpool update          # install the newest published release when newer
eggpool update 0.8.3    # exact current Rust release
eggpool update 0.7.4    # exact supported Python-era rollback
eggpool update v0.7.4   # equivalent spelling
eggpool update 0.8.3 --check

Exact updates are pinned and can intentionally downgrade. The command uses the current installation owner, verifies the catalogued target and installed version, then restarts a running service only after the self-check passes. Ambiguous provenance and unsupported targets fail closed. Source checkouts use the explicit developer workflow documented in upgrading.md; they are not silently converted by the public updater.

The systemd step then takes over: eggpool deploy systemd --install generates a unit tailored to your system (correct binary path, absolute config path, User=/Group= set to the invoking user) and writes it to /etc/systemd/system/eggpool.service.

2. Direct package-manager install (alternative)

uv tool install eggpool
# or
pipx install eggpool
eggpool onboard

Both commands install the platform Rust wheel and keep the package manager as the update authority. For source/reference work, follow the separate developer instructions in upgrading.md.

3. Raspberry Pi / microSD deployment

EggPool is designed to run on small single-board computers including Raspberry Pi. The following guidance helps optimize for microSD storage longevity.

Storage recommendations

  • Use a high-endurance microSD card (64GB+ rated for continuous writes)
  • Prefer a USB SSD for sustained multi-session workloads
  • Avoid cheap microSD cards for production use: they wear out quickly under continuous write patterns

Low-wear metrics configuration

The default low_wear mode buffers lossy analytics and flushes every 120 seconds. Correctness-critical request, reservation, and attempt writes remain immediate and durable:

[metrics]
write_mode = "low_wear"
flush_interval_s = 120
max_buffered_events = 250
timeseries_bucket_s = 300
trace_sample_rate = 0.05
aggregate_only = true

This configuration flushes analytics every 2 minutes, uses 5-minute time buckets, aggregates lossy metrics, and leaves detailed spans disabled until explicitly sampled.

Trade-off: Dashboard data will be less fresh and detailed traces will be incomplete, but database write frequency is significantly reduced.

Log management

Keep logs under logrotate to prevent unbounded growth:

eggpool deploy logrotate --install

Avoid frequent vacuuming

Do not run eggpool db vacuum frequently on flash media. SQLite's WAL mode handles concurrent access well without manual vacuuming. Only vacuum when you need to reclaim space after large data deletions.

Backups

Automatic backups are disabled by default. See the Backup and Restore guide for setup and recovery. After sudden power loss, buffered analytics may lose up to one flush_interval_s window; correctness-critical request state is persisted immediately.

4. Cron fallback (no systemd)

For systems without systemd, install a personal crontab watchdog instead of the systemd unit:

eggpool deploy cron --install

This writes a @reboot + */5 * * * * block to the invoking user's crontab (or SUDO_USER's crontab under sudo). Each line calls eggpool ensure-running, the stdlib-only fast-path CLI:

# BEGIN EggPool watchdog (managed by eggpool deploy cron)
@reboot /home/user/.local/bin/eggpool --config /home/user/.config/eggpool/config.toml ensure-running >> /home/user/.local/state/eggpool/cron.log 2>&1
*/5 * * * * /home/user/.local/bin/eggpool --config /home/user/.config/eggpool/config.toml ensure-running >> /home/user/.local/state/eggpool/cron.log 2>&1
# END EggPool watchdog

Absolute paths only: the watchdog does not depend on cron PATH or on the operator's interactive shell environment. ensure-running atomically checks-and-starts without ever spawning a duplicate instance, and uses the stdlib-only fast-path CLI so the cron tick is cheap enough to run every five minutes on Raspberry Pi-class hardware.

5. Manual run (debug foreground)

For development or quick trials:

eggpool serve                 # daemon mode (default); logs to ~/.local/state/eggpool/eggpool.log
eggpool serve --verbose       # foreground (native Rust logs to terminal)

eggpool serve runs in daemon mode by default. It validates the config, refuses to start a second instance, then spawns a detached child and returns promptly. See Daemon Mode below for the full contract.

6. Verify

eggpool accounts status
eggpool croncheck
curl http://localhost:11300/v1/healthz

7. Other deploy commands

sudo env "PATH=$PATH" "$(command -v eggpool)" deploy logrotate --install

sudo env "PATH=$PATH" "HOME=$HOME" "$(command -v eggpool)" deploy all --install

deploy logrotate --install writes /etc/logrotate.d/eggpool and runs logrotate -d to validate the syntax (it no longer tries to restart a possibly-missing logrotate.service). deploy all --install covers systemd + logrotate + the watchdog cron; backup-cron is intentionally separate.

Without --install, each command prints the snippet and manual instructions for you to copy-paste.

8. Configuration changes

# Live reload for supported changes (no restart needed)
$EDITOR ~/.config/eggpool/config.toml
eggpool rehash

# Disruptive changes (host, port, database path) require a restart
eggpool restart

eggpool rehash applies supported configuration changes without a restart. The command validates the config locally, contacts the running server's control socket, and the server atomically swaps the active configuration generation when safe.

The closure pass enables provider/account/routing/model-override/model-router families as LIVE, so editing [providers.<id>], [[providers.<id>.accounts]], [routing], [model_overrides.<id>], [model_capabilities.<id>], or [model_routers.<id>] applies without a restart. Other fields (server bind host/port, runtime construction, database path, middleware, security headers and trusted-proxy attribution, metrics topology, backup paths, transcoder storage topology) remain RESTART_REQUIRED. Use eggpool restart for those. A mixed live + restart-required change is rejected entirely (exit code 2); no partial application. The CLI is fail-closed and never implicitly restarts the service.

The optional [admission_fairness] short-window rates and principal-state bound are live-reloadable. They take effect for requests using the newly published generation; in-memory buckets are shared across generations and clamped to the active policy. Restart clears the ephemeral limiter state. See model limits for the rate bounds and client response contract.

Sticky model-router affinity remains process-local across these live swaps only when the router's semantic fingerprint is unchanged. Policy edits naturally make prior decisions unreachable; no persisted cache state or restart step is required.

Control socket: The running server listens on a Unix-domain socket at ~/.local/state/eggpool/eggpool.sock with 0o600 permissions (owner read/write only). The CLI connects to this socket to issue reload commands.

sudo systemctl restart eggpool

9. Logs

sudo journalctl -u eggpool -f

Configuration path resolution

Every CLI command resolves --config in this order:

  1. --config PATH
  2. $EGGPOOL_CONFIG
  3. $XDG_CONFIG_HOME/eggpool/config.toml, or ~/.config/eggpool/config.toml when XDG_CONFIG_HOME is unset, if that file exists
  4. ./config.toml as the checkout fallback

For the environment file:

  1. $EGGPOOL_ENV
  2. <config-dir>/.env next to the resolved config
  3. $XDG_CONFIG_HOME/eggpool/.env, or ~/.config/eggpool/.env when XDG_CONFIG_HOME is unset

After install, drop the --config flag by exporting $EGGPOOL_CONFIG in your shell rc. The install script prints the exact line to add.


Filesystem layout (personal)

~/.config/eggpool/
├── config.toml          # Main configuration file
└── .env                 # Environment variables (API keys)

~/.local/share/eggpool/
├── usage.sqlite3        # SQLite database
├── usage.sqlite3-wal    # WAL journal
└── usage.sqlite3-shm    # Shared memory file

~/.local/state/eggpool/
├── eggpool.pid          # Supervisor PID file
├── eggpool.log          # Daemon log (serve default)
└── cron.log             # Watchdog cron output

The XDG defaults honor $XDG_CONFIG_HOME, $XDG_DATA_HOME, and $XDG_STATE_HOME. The resolvers (default_config_dir(), default_data_dir(), default_state_dir(), default_config_path(), default_env_path()) are implemented in the Rust operations path under rust/src/operations/paths.rs and rust/src/operations/deploy.rs.


Filesystem layout (production)

/etc/eggpool/
├── config.toml          # Configuration
└── env                  # API keys

/var/lib/eggpool/
├── usage.sqlite3        # Database
├── usage.sqlite3-wal    # WAL journal
└── usage.sqlite3-shm    # Shared memory file

/opt/eggpool/
└── eggpool              # Optional standalone deployment asset

eggpool deploy systemd --install --production automates the full production layout (dedicated system user, directory permissions, hardened unit) and runs migrations as the eggpool user. Manual production installation is documented below.


Performance Profiles

EggPool ships with a lean SBC default and a fuller standard default. The standard install uses one SQLite worker, low-wear analytics, a LAN bind (server.host = "0.0.0.0"), a public read-only dashboard (dashboard.public = true), and the metadata-enrichment sidecar (model_info) enabled by default. The SBC profile keeps optional enrichment/diagnostics disabled to bound resource use on Raspberry Pi class hardware, and deliberately stays on a loopback bind (see rust/config.sbc.example.toml). All profiles use the supported single-event-loop default (threads = 1).

eggpool onboard keeps any explicit operator bind address and never rewrites it to loopback; newly initialized configs carry the canonical 0.0.0.0 LAN bind. Before local identity initialization, the configured server API key retains legacy behavior and the public read-only dashboard setting applies. After the first local administrator is created, the shared key is no longer accepted; personal tokens authenticate agent surfaces, while dashboard and admin routes require a browser session regardless of dashboard.public.

The first successful eggpool users add USER moves an initialized instance from admin_initialized to managed atomically with member creation, profile assignment, and audit. eggpool users enable-managed records that transition explicitly and requires an active local administrator. Both paths apply the same database transition rules. After identity initialization, EggPool rejects the configured legacy shared key and ignores dashboard.public for privileged routes. Remove [server].api_key/api_key_env and set dashboard.public = false as configuration hygiene; stale values do not restore legacy access or block the identity cutover. Non-loopback listeners can run without a shared key because protected routes require a personal token or browser session.

Browser password login is allowed over loopback HTTP. For remote browser use, terminate TLS at an operator-managed proxy and add the proxy's literal peer IP to [security].trusted_proxies; EggPool accepts only a single X-Forwarded-Proto: https from that peer. Browser sign-in and failed personal- token throttling use one literal client IP in X-Forwarded-For from the trusted peer; the proxy must replace this header, and missing, duplicate, or chained values use the proxy peer as the throttle identity. The proxy must preserve the public Host header and the firewall must prevent direct access around the proxy. The optional [security].allow_insecure_web_auth = true switch is for isolated development only and emits a startup warning.

The GET /login handler applies the same transport decision before rendering the form. Remote direct HTTP receives a static help page with no username or password controls and performs no database access. Solo headless operators can forward the browser locally with ssh -L 11300:127.0.0.1:11300 operator@host and open http://127.0.0.1:11300/login; remote teams should use the trusted HTTPS proxy path above.

Optional background features that remain explicit: set [backup].enabled = true for in-process archives or [update_checker].enabled = true for the periodic PyPI status probe. Set [model_info].enabled = false to disable the metadata-enrichment sidecar (SBC profile does this by default). Host DNS behavior is provided by the operating system; EggPool does not add a process-local resolver cache.

For a short deployment comparison, use the same host, config shape, database state, and stabilization interval for each build. Capture eggpool runtime-status --json, the startup Operational profile line, and host process/socket counts. Compare at least three runs. Treat these as descriptive observations rather than universal RSS or latency gates, and keep EggPool-local local_pre_upstream/dispatch_overhead separate from upstream connect, header, TTFT, and total response latency.

Ordinary-server team sizing evidence

The M002 synthetic reference was measured on one ordinary Linux host with durable SSD/NVMe storage, WAL/NORMAL, and one database worker. It records CPU, RAM, filesystem, device class/model, virtualization/container status, process RSS, request throughput and percentiles, I/O pressure, and durable reservation settlement. Named account rows (1/5/10/25/49) are measured separately from simultaneous inference requests (1/2/4/8/16); neither count is a service guarantee. The synthetic upstream cycles a fixed 0/10/50 ms service-time distribution to show how provider latency affects observed response time.

On the measured Intel i9-9900K host (16 logical CPUs, 15.5 GiB RAM, Samsung 980 NVMe), the median of three 1,000-request runs ranged from 36 to 43 requests/s at concurrency 1, about 194 requests/s at concurrency 4, and 335–341 requests/s at concurrency 8–16. At concurrency 16, median p95/p99 were 88/106 ms; maximum observed latency across the three runs was 1.119 s. Peak process RSS across cells was 25–46 MiB. These values describe this loopback synthetic workload; upstream latency, request shape, system load, provider limits, and team policy change the result. No universal throughput, concurrency, or latency SLA follows from one host and synthetic workload.

The driver writes the manifest, raw runs, and generated summary under artifacts/qualification/on-prem-concurrency-performance/ordinary-server/. Use the result only for relative sizing; qualify the actual provider mix, model payloads, login rate, and policy/budget profile before deployment. The Pi microSD envelope remains a separate solo-device reference in raspberry-pi.md. See the M002 closure record for its evidence, scope and interpretation limits.

To repeat the frozen corpus from a release binary, run uv run python scripts/qualification_ordinary_server.py --binary <release-binary>. The driver writes its manifest before collection, creates a fresh qualification root for each run, requires three repetitions per condition, validates p99 sample counts and settlement, derives summary.json from the raw Q008 reports, and runs deterministic Q009 provider-fault, hard-budget, user-disable, and abrupt-restart phases against the same binary. The summary requires the matching lifecycle.json artifact. It refuses non-Linux hosts and non-durable or rotational qualification storage, and stops before load when initial full I/O pressure exceeds 25% over the 10-second window. Qualification does not send requests to a real provider or charge a provider account.

Lean (default)

Recommended for Raspberry Pi and low-power devices. Optimized for dashboard responsiveness under request load.

[server]
threads = 1

[database]
worker_threads = 1
wal = true
synchronous = "NORMAL"

[metrics]
write_mode = "low_wear"
flush_interval_s = 120

[routing.trace]
mode = "off"
sample_rate = 0.0
include_score_components = false

Minimum-footprint

For extremely constrained devices or in-memory test databases.

[server]
threads = 1

[database]
worker_threads = 1

[routing.trace]
mode = "off"
sample_rate = 0.0
include_score_components = false

Full-diagnostics

For development or high-power hosts where detailed routing traces are useful. EggPool does not implement an HTTP access log; use an external log pipeline if request access logging is required.

[server]

[database]
worker_threads = 1

[routing.trace]
mode = "all"
include_score_components = true

Low-power routing trace defaults

The [routing.trace] section controls per-request routing decision persistence. Full traces write a routing_decisions row for every attempt, which increases SQLite write volume on low-power hardware.

The lean default disables routing-decision persistence:

[routing.trace]
mode = "off"
sample_rate = 0.0
include_score_components = false
Mode Behavior
"off" No routing decision rows are written. Lowest write pressure.
"sampled" Rows are written for a fraction of requests (controlled by sample_rate). A sample_rate of 0.05 writes roughly 1 in 20 traces.
"all" Rows are written for every request. Use for diagnostics or debugging only.

include_score_components adds per-account score breakdown JSON to each trace row. Keep it false on low-power installs to reduce row size.

To temporarily enable traces for diagnostics, set mode = "sampled" or "all" and restart. Revert to "off" when done.

Symptoms and Knobs

Symptom Likely cause Knob to adjust
Dashboard loads slowly under request load DB lock contention Set database.worker_threads = 2 explicitly
Dashboard still slow Insufficient concurrency Check dashboard telemetry; ensure connection pool and maintenance budgets are adequate for your workload
High write volume / microSD wear Routing trace writes Set routing.trace.mode = "off"
Background tasks cluster at minute boundaries Task scheduling Default is staggered (internal TaskSpec::initial_delay_s classes, not an operator config key)
Stale dashboard data Cache TTL Wait 30s or check dashboard cache settings

Production Deployment

For public-facing or multi-user deployments. Uses a dedicated eggpool system user with proper file permissions and a hardened systemd unit.

Not recommended for personal LAN use: the personal-use path above is simpler and sufficient for single-user setups.

Automated production install

sudo env "PATH=$PATH" "$(command -v eggpool)" deploy systemd --install --production

This runs the full manual sequence below (system user, directory permissions, config seeding, migrations, hardened unit) in one command. Use --install --production together; --production alone just prints the snippets.

Manual production install

For operators who prefer to wire each step by hand:

# 1. Create system user
sudo useradd -r -s /usr/sbin/nologin -d /var/lib/eggpool eggpool
sudo mkdir -p /var/lib/eggpool /var/log/eggpool /etc/eggpool
sudo chown eggpool:eggpool /var/lib/eggpool /var/log/eggpool
sudo chown root:eggpool /etc/eggpool
sudo chmod 750 /var/lib/eggpool /var/log/eggpool
sudo chmod 755 /etc/eggpool

# 2. Install the native wheel through the system-owned pipx authority
# (pin the version you qualified; see docs/rust-release-deployment.md)
sudo env PIPX_HOME=/var/lib/eggpool/pipx PIPX_BIN_DIR=/usr/local/bin \
  pipx install --force 'eggpool==0.8.3'

# 3. Configure
sudo /usr/local/bin/eggpool init-config /etc/eggpool/config.toml
sudo touch /etc/eggpool/env
sudo chown root:eggpool /etc/eggpool/config.toml /etc/eggpool/env
sudo chmod 640 /etc/eggpool/config.toml /etc/eggpool/env

sudo nano /etc/eggpool/config.toml
sudo nano /etc/eggpool/env

# Minimal config:
#
# [server]
# host = "0.0.0.0"         # canonical LAN bind (set "127.0.0.1" for loopback-only)
# port = 11300             # legacy installs need a key; managed installs use per-user credentials
#
# [database]
# path = "/var/lib/eggpool/usage.sqlite3"

# 4. Validate and start
sudo -u eggpool env PIPX_HOME=/var/lib/eggpool/pipx \
  PIPX_BIN_DIR=/usr/local/bin /usr/local/bin/eggpool \
  --config /etc/eggpool/config.toml check-config
sudo -u eggpool env PIPX_HOME=/var/lib/eggpool/pipx \
  PIPX_BIN_DIR=/usr/local/bin /usr/local/bin/eggpool \
  --config /etc/eggpool/config.toml migrate

# 5. Install the hardened systemd unit
sudo env PIPX_HOME=/var/lib/eggpool/pipx PIPX_BIN_DIR=/usr/local/bin \
  /usr/local/bin/eggpool --config /etc/eggpool/config.toml \
  deploy systemd --install --production
sudo systemctl status eggpool

# 6. Logrotate (no longer requires systemctl restart logrotate)
sudo env "PATH=$PATH" "$(command -v eggpool)" deploy logrotate --install

# 7. Automated backup (production /etc/cron.d/eggpool-backup)
sudo env "PATH=$PATH" "$(command -v eggpool)" deploy backup-cron --install --production

Systemd unit features

The production unit runs with an empty capability/ambient-capability set, UMask=0077, NoNewPrivileges, ProtectSystem=strict, ProtectHome, PrivateTmp, PrivateDevices, ProtectClock, ProtectHostname, ProtectProc=invisible, ProcSubset=pid, ProtectKernelLogs, kernel-tunable/module/control-group protection, RemoveIPC, and restrictions on namespaces, realtime scheduling, SUID/SGID, and personality. Its syscall filter excludes privileged, mount, and debug syscall groups and returns EPERM. ReadWritePaths is limited to EggPool data, backup, and log directories.

The deployed unit and render_production_systemd are checked by the operations_o009 directive-drift test. The renderer intentionally includes /var/backups/eggpool for its production backup-cron layout; the checked-in service unit retains its narrower writable-path set. Their ExecStart lines also intentionally differ by the renderer's --verbose flag. A disposable supported Linux/systemd host is still required to qualify service startup after sandbox changes; this repository's macOS development checks cannot establish that operational evidence.

  • User=eggpool, Group=eggpool: never runs as root
  • No ExecReload: config changes require systemctl restart (run eggpool check-config or eggpool rehash first for fast validation)
  • Restart=on-failure with RestartSec=5 and a 300s burst cap

Daemon Mode

eggpool serve runs in daemon mode by default. It is a one-shot detach helper for personal / SBC deployments. It validates the configuration, refuses to start a second instance, spawns a detached child running the normal foreground serve command, and returns promptly with a short success message pointing at the log file.

The parent only validates the config and refuses to start a second instance. The detached child runs the native Rust foreground process. No daemon flag is forwarded to the child; detachment is purely a parent-side concern. The child owns its own PID file lifecycle through the process primitives in rust/src/operations/process.rs, with path resolution owned by rust/src/operations/paths.rs.

Detach mechanics

  • The native lifecycle service starts a new process group so the child survives shell exit and signals to the parent CLI do not propagate
  • stdin is detached from the calling terminal
  • stdout/stderr redirected to a log file (or /dev/null when --quiet is set without --log-file)
  • Default log file: ~/.local/state/eggpool/eggpool.log (resolved by the native runtime); override with --log-file PATH or $EGGPOOL_LOG_FILE. A log file beats /dev/null by default because a silent background failure is hard to diagnose
  • The detached child handle is intentionally not awaited by the CLI parent; the parent returns as soon as the child has been spawned

PID file resolution

PID file path resolution lives in rust/src/operations/paths.rs and is the single source of truth shared by the CLI, systemd, and the cron watchdog. Precedence:

  1. $EGGPOOL_PID_FILE (if set)
  2. $XDG_RUNTIME_DIR/eggpool.pid (if XDG_RUNTIME_DIR is set)
  3. ~/.local/state/eggpool/eggpool.pid (state dir auto-created)
  4. /tmp/eggpool-<UID>.pid (UID-scoped fallback)

The native runtime owns the PID-file path; there is no retired application module or compatibility constant to import.

Root-user guard

eggpool serve refuses to run when the effective UID is 0 unless --as-root is passed. This prevents accidentally starting a personal deployment as root; the explicit flag exists for intentional system-wide installs. systemd production deployments should run foreground serve --verbose under the systemd unit (with User= set) and must not use daemon mode.

Process model

eggpool serve runs as the native Rust process. It has one process identity under the canonical name eggpool; there is no Python worker or framework runtime hidden behind the package command.

The native process owns the PID file through the process primitives in rust/src/operations/process.rs. eggpool serve refuses to start a second instance: it checks the PID file and, if no live PID is recorded, probes GET /v1/healthz over 127.0.0.1 (the bind address 0.0.0.0 / :: is rewritten to a loopback address for the probe). Either a live PID or a 200 from the probe causes the new serve to exit non-zero so a stale PID file is never silently overwritten.

[server].threads is retained for configuration compatibility and runtime diagnostics. EggPool uses Tokio's current_thread runtime, so this value does not select an I/O worker pool; keep the compatibility default on constrained systems:

[server]
threads = 1

eggpool restart delegates to the lifecycle service in rust/src/operations/lifecycle.rs, which proves the recorded process identity, signals it through rust/src/operations/process.rs, waits for a safe stop, and starts the native executable. The CLI adapter retains only command parsing, presentation, and exit-code mapping.


Watchdog cron

eggpool deploy cron --install                # default 5-minute interval
eggpool deploy cron --install --interval 10  # 10-minute poll cadence
eggpool deploy cron --uninstall              # strip the BEGIN/END-marked block

The generated block uses absolute binary, config, and log paths:

# BEGIN EggPool watchdog (managed by eggpool deploy cron)
@reboot <binary> --config <config> ensure-running >> <log> 2>&1
*/N * * * * <binary> --config <config> ensure-running >> <log> 2>&1
# END EggPool watchdog

ensure-running is the stdlib-only fast-path CLI: it does not import the heavy application graph on every cron tick, so a 5-minute cadence is cheap on Raspberry Pi-class hardware. The block is bracketed by # BEGIN EggPool watchdog / # END EggPool watchdog markers so uninstall only strips the eggpool-owned lines and leaves unrelated cron entries untouched.


Runtime diagnostics

eggpool status                         # concise proxy/provider health (one row per provider)
eggpool status --json                  # versioned structured snapshot (schema_version 1)
eggpool runtime-status                 # detailed process/runtime diagnostics
eggpool runtime-status --json          # machine-readable JSON

eggpool status reads the sanitized snapshot over the same-user private local control socket and prints a compact operator summary: one header line for proxy health, exactly one row per configured provider (ready/degraded/unavailable/disabled/unknown with a bounded reason code), and one runtime footer line. It aggregates active-generation account identity, live routing health, and cached catalog probe evidence without making outbound provider requests or consuming quota. Exit codes are 0 when the proxy is reachable and ready (including degraded with a viable route), 1 when reachable but unready, and 3 when the server is unreachable, locally configured providers are still listed as unknown/disabled from static config. eggpool status --json prints the typed server snapshot (deterministic provider ordering, numeric durations in named units, no credentials or raw upstream error text).

eggpool runtime-status reads the detailed projection over that local control socket and prints a one-page overview of the running process. It is intended for operators debugging systemd, cron, or daemon deployments.

The control runtime directory is private and its Unix socket is mode 0600. This grants readback only to the operating-system account that owns the daemon and socket; it does not change HTTP authorization. /api/status and /api/stats/runtime remain browser-session protected after identity bootstrap, and personal API tokens cannot access them. Missing or stale sockets are reported as unavailable; malformed or oversized responses fail closed.

When the server is not running the command exits non-zero with a clear message. It does not start the server.

The output covers:

  • Server: PID, uptime, configured server threads, Rust version.
  • Processes: observed EggPool process count vs expected; a warning is printed when the observed count exceeds expected by more than one.
  • Memory: RSS, VMS, open FD count, thread count.
  • Background tasks: per-task running/done/cancelled state, iteration count, restart count, last error class and timestamp. Cadence diagnostics include configured_interval_s, observed_last_interval_s, last_tick_drift_s, and initial_delay_consumed so operators can spot scheduler drift or stuck tasks directly from the snapshot.
  • Database: path, WAL mode, file/WAL/SHM sizes, contention counters (cumulative lock wait, max lock wait, write/read ops).
  • Routing: active requests, pending count, active reservations, reserved microdollars, health states, active backoff rows.
  • Terminal finalization: the bounded finalization_supervisor snapshot reports active and retry-pending jobs, bounded failures/history, saturation counters, and configured capacity/retry-age limits. null means the server has no active production generation supervisor (for example during partial startup).
  • Generation retirement ownership: runtime_manager.finalization_ownership reports the active generation, retiring-generation count, total terminal references, oldest retiring age, and bounded blocked/failure facts. A live retiring generation is never closed while terminal references remain; an unresolved deadline invokes the worker fatal handler so startup repair can establish the next authoritative boundary. Process shutdown may abandon references because startup crash repair follows process death.

For a focused view of active upstream-derived suppression, use eggpool accounts status (per-account routing eligibility) or the health/backoff sections of runtime-status --json. There is no /api/backoffs HTTP endpoint. Only upstream-observed failures such as 429, 402, auth failures, model unavailability, and bounded transient errors create suppression rows; local quota estimates never appear there.

All probes are best-effort; failed probes return null rather than causing the command to fail. Probe diagnostics are exposed in the JSON payload as probe_errors, capped at 16 entries with each message truncated, so repeated host or permissions failures cannot produce an unbounded response.

At startup, the native server logs a warning when server.threads differs from 1; the value is retained for compatibility and does not change Tokio's single-thread runtime. Use runtime-status --json and the health/status endpoints to inspect the effective runtime and background-task state.

Checking from cron

The watchdog cron uses eggpool ensure-running, which is a separate fast-path command. Do not use eggpool runtime-status in cron entries: it opens an authenticated local TCP connection and parses a full runtime snapshot per invocation, so it is not designed for high-frequency polling.

Use backup cron separately from the watchdog cron.


Cleanup

eggpool uninstall --yes                       # binary, config, data, PATH
eggpool uninstall --yes --deploy-artifacts    # also systemd, logrotate, cron

eggpool uninstall stops the server, removes the resolved executable and selected configuration, data, and state paths, and scrubs EggPool entries from the user's shell rc files. It does not call uv or pipx, and it does not elevate privileges. For a package-manager-owned install, remove the package with its owning manager as well. Run the command with permission to remove each selected path.

Pass --deploy-artifacts to remove the known systemd, logrotate, and cron artifacts created by eggpool deploy. The command does not invoke sudo; run it with the required permissions. Without this flag, deployment artifacts are left in place.

The personal backup directory ~/backups/eggpool/ is outside the default data path and is left in place. Production backups under /var/lib/eggpool/backups are removed with the data directory unless --keep-data is set.


Troubleshooting

Service fails to start

sudo journalctl -u eggpool --since "5 minutes ago"
# or for personal use:
sudo journalctl -u eggpool -n 50 --no-pager

Database locked errors

  1. Ensure only one instance: pgrep -f eggpool
  2. Confirm WAL mode in config
  3. Increase busy_timeout_ms (try 10000 on Pi)

Cannot connect from LAN

  1. Verify that the explicit config selected server.host = "0.0.0.0" (or another non-loopback address). Legacy installs need [server].api_key or [server].api_key_env; managed installs authenticate with personal tokens and browser sessions.
  2. Check firewall: ss -tlnp | grep 11300
  3. See docs/firewall.md

The copyable SBC profile stays on loopback by default. A non-loopback legacy install requires the shared server key. After identity initialization, a keyless listener is allowed, but protected routes still require personal credentials or a browser session; unauthenticated loopback behavior is limited to the explicit legacy state.

Headless server + desktop workgroup

For one EggPool host (SBC or server) serving Codex/OpenCode desktops, advertise a single desktop-reachable URL once, then export profiles:

[integrations]
advertise_base_url = "http://<lan-ip>:11300/v1"
eggpool configremote codex
eggpool configremote opencode --shell powershell

Paste each desktop's bootstrap block on that desktop: it verifies SHA-256 before executing anything and needs no checkout or toolchain there. The Windows helper covers Windows desktops only and does not change the supported proxy targets. See Desktop Bootstrap and Raspberry Pi.

Leaked request detection

If the proxy starts returning 503s after running successfully for several minutes, check for leaked pending requests:

sqlite3 ~/.local/share/eggpool/usage.sqlite3 "SELECT COUNT(*) FROM requests WHERE status = 'pending';"
sqlite3 ~/.local/share/eggpool/usage.sqlite3 "SELECT COUNT(*) FROM reservations WHERE status = 'active';"

Non-zero counts that grow over time indicate finalization failures. Normal runtime does not reclaim requests by age. The retained terminal owner must converge the request, reservation, usage, and health obligations. If these counts grow, inspect finalization diagnostics and logs. Startup crash reconciliation repairs durable work left by a process that exited.

You can also use eggpool runtime-status to see the pending count and oldest pending request age without querying SQLite directly.

Unexpected process count

eggpool runtime-status reports eggpool_process_count and expected_worker_process_count. On a standard deployment the expected count is 1.

If the observed count exceeds expected:

  1. Check for leftover processes from a previous crash: pgrep -af eggpool
  2. Kill orphaned processes manually if confirmed stale.
  3. Ensure only one eggpool serve or systemd unit is active.

High RSS memory

Memory usage is visible in eggpool runtime-status under the Memory section. On Linux the snapshot reads current RSS from /proc/self/stat; on macOS it falls back to ru_maxrss (a high-water mark).

The following in-memory growth axes are bounded by design:

  • QuotaEstimator EWMA maps are LRU-capped at EWMA_HARD_CAP = 4096 and GLOBAL_EWMA_HARD_CAP = 1024 entries respectively (hardcoded, not configurable).
  • ModelCatalogCache keys models, provider_models, and account-support maps by provider/model identity; all are bounded by the advertised provider set, not by request volume.
  • HealthManager.AccountHealth.disabled_models entries are pruned during account synchronization and successful catalog-refresh reconciliation against the currently advertised model set.

If RSS still grows continuously after the above:

  1. Check for leaked pending requests (see above).
  2. Verify WAL checkpointing is working: PRAGMA wal_checkpoint(PASSIVE).
  3. Consider restarting the process to reclaim memory: the database file itself does not grow from in-memory leaks.

WAL file growth

The WAL (-wal) and SHM (-shm) file sizes are reported by eggpool runtime-status under the Database section.

A WAL file that grows without bound indicates checkpoints are not keeping up:

  1. Ensure PRAGMA journal_mode = WAL is active (shown in runtime status under wal_mode_live).
  2. Check that busy_timeout_ms is sufficient (default 5000ms; raise to 10000ms on Raspberry Pi).
  3. Run PRAGMA wal_checkpoint(TRUNCATE) manually to reclaim space.
  4. If WAL growth is persistent, consider increasing database.worker_threads only when the configured database mode supports the additional read capacity; server.threads does not control SQLite contention.

Upstream read timeout

upstream.read_timeout_s is an HTTP client idle/operation bound. It is not a maximum stream lifetime and is never used by an active-process stale sweep. Long healthy streams retain their request and runtime ownership until their explicit terminal owner completes.

Startup crash recovery

A process restart is a definitive boundary: any request that was still pending in the previous process is repaired by startup crash reconciliation, and its active reservations are released, regardless of how recently it was created. Check the startup log for the crash-recovery summary after a crash or forced restart.

The shipped systemd unit uses Restart=on-failure, so a worker that exits after an indeterminate SQLite state is restarted automatically. The restart boundary is intentional: the new worker must pass migrations, integrity checks, and crash reconciliation before it accepts traffic. When [readiness_probe].enabled = true, it must also complete a fresh writable probe before readiness is reopened.


Deploy commands reference

Command Description
eggpool deploy systemd Print the personal + production systemd units
eggpool deploy systemd --install Install the personal unit (runs as the invoking user); refuses direct-root without --as-root
eggpool deploy systemd --install --production Install the hardened production layout (dedicated user, /etc/eggpool, /var/lib/eggpool)
eggpool deploy logrotate Print the logrotate config
eggpool deploy logrotate --install Install /etc/logrotate.d/eggpool and validate via logrotate -d
eggpool deploy cron Print the watchdog cron fragment (not the backup)
eggpool deploy cron --install Install @reboot + */N * * * * ensure-running block into the invoking user's crontab. --interval N (1-59, default 5)
eggpool deploy cron --uninstall Strip the # BEGIN EggPool watchdog block from the invoking user's crontab
eggpool deploy backup-cron Print the daily backup cron entry + script
eggpool deploy backup-cron --install Install personal backup (user cron + ~/backups/eggpool/)
eggpool deploy backup-cron --install --production Install production backup (/etc/cron.d/eggpool-backup + /var/backups/eggpool)
eggpool deploy all Print systemd + logrotate + watchdog cron snippets
eggpool deploy all --install Install systemd + logrotate + watchdog cron (backup-cron is separate)
eggpool status Concise proxy/provider health, one row per provider (exit 0 ready/degraded, 1 unready, 3 unreachable)
eggpool status --json Versioned structured status snapshot for scripting/monitoring
eggpool runtime-status Detailed process/runtime diagnostics from the running server
eggpool runtime-status --json Print machine-readable JSON for scripting/monitoring
eggpool backup Create a timestamped .zip backup archive (default ~/backups/eggpool/)
eggpool recover [path] Restore from a backup archive (interactive menu if no path)
eggpool uninstall Remove the resolved executable, selected config/data, and shell-rc entries; requires permission for each path
eggpool uninstall --deploy-artifacts Also remove systemd unit, logrotate config, watchdog + backup cron blocks, backup script
eggpool croncheck Check if server is running (exit 0/1): fast-path, no heavy imports
eggpool ensure-running Atomically check-and-start the server; no-op when already alive: fast-path, no heavy imports

Local gateway profile

For a gateway in front of local models, copy rust/config.local.example.toml and run eggpool check-config --local. The check fails while an external model-info source, pricing catalog, or the update checker is enabled, and it names the key. It does not block a provider you configured and it does not rewrite the file. Restart if a restart-required key changed.

Tool execution

EggPool does not execute tools and does not connect to Eggsearch. Leave [tools] omitted or all-false. A true enabled, gateway_enabled, or hosted_execution_enabled switch fails startup. Clients execute tool calls themselves. See Tool execution.

An administrator can export content-free gateway facts with eggpool audit export or GET /api/admin/audit/export. The export is identifiers, actions, model ids, protocol, status, token counts, latency, a stored reason code, and timestamps. It is not a transcript.

Embedding deployment

Embeddings are disabled by default. To enable them, configure a reachable OpenAI-compatible text embedding endpoint on a provider using a fixed embeddings_path and exact embedding_models allowlist. The model must also be present in the ordinary model catalog. The operator owns model installation, availability, and qualification; EggPool does not download weights or store input text/vectors. Missing or ineligible embedding providers do not affect normal inference. See provider configuration and the API reference for the finite contract and limits. EggPool does not store documents or vectors. A downstream application that wants embeddings calls POST /v1/embeddings and keeps ingest, indexes, and access checks itself. See Retrieval.