Two deployment modes: personal use (quick, current user) and production (separate user, hardened). Pick the one that fits.
EggPool's canonical public runtime is the native Rust executable delivered through the PyPI wheel. Qualified targets are Linux x86_64, Linux aarch64, and macOS arm64; Windows and other unqualified targets are unsupported. See Upgrade and rollback for exact cross-era transitions and the standalone raw-binary authority.
Runs under your current user with your existing config. Not intended for public-facing deployments.
bash -o pipefail -c 'curl -fsSL https://raw.githubusercontent.com/eggstack/eggpool/main/scripts/install.sh | bash' && export PATH="$HOME/.local/bin:$PATH"
eggpool onboard
sudo env "PATH=$PATH" "HOME=$HOME" "$(command -v eggpool)" deploy systemd --installPATH and HOME are passed explicitly because sudo resets both to root's
environment on distributions that set always_set_home in sudoers (Debian
and Ubuntu do by default). The personal unit runs as the invoking user, so it
must resolve that user's configuration, data directory, and home. Without
HOME, the unit is written with User= the invoking user but HOME= root's
home, and the service cannot read the configuration it was installed against.
The installer is binary-first for fresh native installations. It downloads
the executable for the detected target from GitHub raw, verifies its SHA-256,
runs a staged self-check, and installs it as standalone-rust. It does not
require Python, uv, pipx, pip, Cargo, or a source checkout. Direct installs with
uv tool install eggpool, pipx install eggpool, and cargo install eggpool
remain supported alternatives.
When the installer finds an existing EggPool on PATH, it preserves the
installation owner: native installs use eggpool update, legacy Python-era
installs use their owning manager, and source-checkout or ambiguous ownership
fails closed. Use --force only to repair a verified EggPool installation
with the same owner; unrelated files are not overwritten. A fresh install
seeds the default config from the example template only when no config exists,
then prints the resolved config path.
After install, eggpool onboard walks the operator through provider
connections, validates the configuration, prompts for an administrator
password on a new database, creates the initial local administrator, and starts
the server. It prints the administrator's personal API token once. That token is the inference
credential. The same run does not write a new [server].api_key. If the server
is already running, onboarding reports that without restarting it.
Browser sign-in is /login on the listen address. For unattended
bootstrap, pass --password-stdin and provide the password on stdin; the
password is never accepted in argv. Secret output is suppressed when stdout
is not a terminal unless --show-secret is explicitly passed. Existing databases keep their legacy
authentication behavior until an operator explicitly initializes local user
access.
Manage local identities from the machine that owns the data directory with
eggpool users add USER --password-stdin, eggpool users show USER,
eggpool users disable USER, eggpool users enable USER,
eggpool users token USER, eggpool users revoke USER, and
eggpool users reset-password USER. Member passwords are read from a hidden TTY prompt or stdin; passwords
are never accepted in argv. Add --profile-id PROFILE to assign the initial
model policy in the same transaction. eggpool users defaults show|set PROFILE|clear
manages the instance default used only when future members are created without
an explicit profile; users add USER --no-profile explicitly keeps a new
member unassigned. Existing assignments are never changed by setting or
clearing the default. No budget template is copied: user and instance budget
rules remain independently configured. Personal tokens are for inference,
model discovery, and the integration profile; they do not authorize dashboard
or admin pages, and they cannot add or remove a personal upstream account.
On a small team, an administrator grants personal-upstream self-service from
the users page. A granted member submits one bundled-provider key on My
account. The key is written to the operator environment file and is not stored
in the database. Denying the grant does not remove an account that is already
bound. Adding an account for someone else stays eggpool users upstream add. The configured server API key remains a legacy shared
inference credential only while the database is legacy_unmigrated. Do not
share it with new users; issue an individual token instead.
With [dashboard].enabled = true, an administrator can manage users, profiles,
budgets, tokens, and audit history in the embedded dashboard. New members
receive a temporary password and must change it at first login. Members can
manage their personal tokens and view only their own usage and accessible
models. The users page can use the instance default, select a specific profile,
or explicitly leave a member without a profile. A browser-issued token is shown
once in a no-store response; set it as
EGGPOOL_API_KEY in the environment that launches the desktop client.
The administrator-only /admin/audit page includes a Security events table
of recent authentication and authorization outcomes, aggregated by minute.
The local writer is bounded and non-blocking: if its queue is full, it drops an
event rather than delaying authentication. Rows are capped at 30 days and
10,000 records. The table may show an existing user's name resolved from its
internal ID, but stored event rows do not contain usernames, credentials,
tokens, raw IP addresses, prompts, headers, or request bodies. This is not a
complete access log, alerting system, or addition to eggpool audit export.
Remote browser login must use the operator's TLS proxy and trusted-proxy setup below: LAN reachability alone does not encrypt passwords or cookies.
Budget rules are estimated by default. Promoting a rule to hard requires
proof the proxy can actually bound: hard token rules need clients to send an
explicit output cap, and hard USD rules need a conservative operator tariff on
every provider those users can reach, configured as
[providers.<id>.budget] max_microdollars_per_million_tokens (microdollars
per million tokens, covering the most expensive model on that provider;
omit for unknown price, 0 for verified free). Without a bound, hard-mode
requests are denied before upstream dispatch: hard mode is unavailable rather
than approximate. See model limits.
An optional instance-wide ceiling (eggpool users global-budget set or the
dashboard instance section) caps total consumption across all users and is
intersected with every per-user rule. It covers unattributed legacy-key
requests as well, survives user disable/delete, and needs no second database
or writer. See model limits.
eggpool update # install the newest published release when newer
eggpool update 0.8.3 # exact current Rust release
eggpool update 0.7.4 # exact supported Python-era rollback
eggpool update v0.7.4 # equivalent spelling
eggpool update 0.8.3 --checkExact updates are pinned and can intentionally downgrade. The command uses the current installation owner, verifies the catalogued target and installed version, then restarts a running service only after the self-check passes. Ambiguous provenance and unsupported targets fail closed. Source checkouts use the explicit developer workflow documented in upgrading.md; they are not silently converted by the public updater.
The systemd step then takes over: eggpool deploy systemd --install
generates a unit tailored to your system (correct binary path,
absolute config path, User=/Group= set to the invoking user) and
writes it to /etc/systemd/system/eggpool.service.
uv tool install eggpool
# or
pipx install eggpool
eggpool onboardBoth commands install the platform Rust wheel and keep the package manager as the update authority. For source/reference work, follow the separate developer instructions in upgrading.md.
EggPool is designed to run on small single-board computers including Raspberry Pi. The following guidance helps optimize for microSD storage longevity.
- Use a high-endurance microSD card (64GB+ rated for continuous writes)
- Prefer a USB SSD for sustained multi-session workloads
- Avoid cheap microSD cards for production use: they wear out quickly under continuous write patterns
The default low_wear mode buffers lossy analytics and flushes every
120 seconds. Correctness-critical request, reservation, and attempt writes
remain immediate and durable:
[metrics]
write_mode = "low_wear"
flush_interval_s = 120
max_buffered_events = 250
timeseries_bucket_s = 300
trace_sample_rate = 0.05
aggregate_only = trueThis configuration flushes analytics every 2 minutes, uses 5-minute time buckets, aggregates lossy metrics, and leaves detailed spans disabled until explicitly sampled.
Trade-off: Dashboard data will be less fresh and detailed traces will be incomplete, but database write frequency is significantly reduced.
Keep logs under logrotate to prevent unbounded growth:
eggpool deploy logrotate --installDo not run eggpool db vacuum frequently on flash media. SQLite's WAL
mode handles concurrent access well without manual vacuuming. Only
vacuum when you need to reclaim space after large data deletions.
Automatic backups are disabled by default. See the
Backup and Restore guide for setup and recovery. After
sudden power loss, buffered analytics may lose up to one flush_interval_s
window; correctness-critical request state is persisted immediately.
For systems without systemd, install a personal crontab watchdog instead of the systemd unit:
eggpool deploy cron --installThis writes a @reboot + */5 * * * * block to the invoking user's
crontab (or SUDO_USER's crontab under sudo). Each line calls
eggpool ensure-running, the stdlib-only fast-path CLI:
# BEGIN EggPool watchdog (managed by eggpool deploy cron)
@reboot /home/user/.local/bin/eggpool --config /home/user/.config/eggpool/config.toml ensure-running >> /home/user/.local/state/eggpool/cron.log 2>&1
*/5 * * * * /home/user/.local/bin/eggpool --config /home/user/.config/eggpool/config.toml ensure-running >> /home/user/.local/state/eggpool/cron.log 2>&1
# END EggPool watchdog
Absolute paths only: the watchdog does not depend on cron PATH or on
the operator's interactive shell environment. ensure-running
atomically checks-and-starts without ever spawning a duplicate
instance, and uses the stdlib-only fast-path CLI so the cron tick is
cheap enough to run every five minutes on Raspberry Pi-class hardware.
For development or quick trials:
eggpool serve # daemon mode (default); logs to ~/.local/state/eggpool/eggpool.log
eggpool serve --verbose # foreground (native Rust logs to terminal)eggpool serve runs in daemon mode by default. It validates the
config, refuses to start a second instance, then spawns a detached
child and returns promptly. See
Daemon Mode below for the full contract.
eggpool accounts status
eggpool croncheck
curl http://localhost:11300/v1/healthzsudo env "PATH=$PATH" "$(command -v eggpool)" deploy logrotate --install
sudo env "PATH=$PATH" "HOME=$HOME" "$(command -v eggpool)" deploy all --installdeploy logrotate --install writes /etc/logrotate.d/eggpool and
runs logrotate -d to validate the syntax (it no longer tries to
restart a possibly-missing logrotate.service). deploy all --install
covers systemd + logrotate + the watchdog cron; backup-cron is
intentionally separate.
Without --install, each command prints the snippet and manual
instructions for you to copy-paste.
# Live reload for supported changes (no restart needed)
$EDITOR ~/.config/eggpool/config.toml
eggpool rehash
# Disruptive changes (host, port, database path) require a restart
eggpool restarteggpool rehash applies supported configuration changes without a
restart. The command validates the config locally, contacts the running
server's control socket, and the server atomically swaps the active
configuration generation when safe.
The closure pass enables provider/account/routing/model-override/model-router
families as LIVE, so editing [providers.<id>],
[[providers.<id>.accounts]], [routing], [model_overrides.<id>],
[model_capabilities.<id>], or [model_routers.<id>] applies without a
restart. Other
fields (server bind host/port, runtime construction, database path,
middleware, security headers and trusted-proxy attribution, metrics topology, backup paths,
transcoder storage topology) remain RESTART_REQUIRED. Use
eggpool restart for those. A mixed live + restart-required
change is rejected entirely (exit code 2); no partial application.
The CLI is fail-closed and never implicitly restarts the service.
The optional [admission_fairness] short-window rates and principal-state
bound are live-reloadable. They take effect for requests using the newly
published generation; in-memory buckets are shared across generations and
clamped to the active policy. Restart clears the ephemeral limiter state.
See model limits for the
rate bounds and client response contract.
Sticky model-router affinity remains process-local across these live swaps only when the router's semantic fingerprint is unchanged. Policy edits naturally make prior decisions unreachable; no persisted cache state or restart step is required.
Control socket: The running server listens on a Unix-domain socket
at ~/.local/state/eggpool/eggpool.sock with 0o600 permissions
(owner read/write only). The CLI connects to this socket to issue
reload commands.
sudo systemctl restart eggpoolsudo journalctl -u eggpool -fEvery CLI command resolves --config in this order:
--config PATH$EGGPOOL_CONFIG$XDG_CONFIG_HOME/eggpool/config.toml, or~/.config/eggpool/config.tomlwhenXDG_CONFIG_HOMEis unset, if that file exists./config.tomlas the checkout fallback
For the environment file:
$EGGPOOL_ENV<config-dir>/.envnext to the resolved config$XDG_CONFIG_HOME/eggpool/.env, or~/.config/eggpool/.envwhenXDG_CONFIG_HOMEis unset
After install, drop the --config flag by exporting
$EGGPOOL_CONFIG in your shell rc. The install script prints the
exact line to add.
~/.config/eggpool/
├── config.toml # Main configuration file
└── .env # Environment variables (API keys)
~/.local/share/eggpool/
├── usage.sqlite3 # SQLite database
├── usage.sqlite3-wal # WAL journal
└── usage.sqlite3-shm # Shared memory file
~/.local/state/eggpool/
├── eggpool.pid # Supervisor PID file
├── eggpool.log # Daemon log (serve default)
└── cron.log # Watchdog cron output
The XDG defaults honor $XDG_CONFIG_HOME, $XDG_DATA_HOME, and
$XDG_STATE_HOME. The resolvers (default_config_dir(),
default_data_dir(), default_state_dir(), default_config_path(),
default_env_path()) are implemented in the Rust operations path under
rust/src/operations/paths.rs and rust/src/operations/deploy.rs.
/etc/eggpool/
├── config.toml # Configuration
└── env # API keys
/var/lib/eggpool/
├── usage.sqlite3 # Database
├── usage.sqlite3-wal # WAL journal
└── usage.sqlite3-shm # Shared memory file
/opt/eggpool/
└── eggpool # Optional standalone deployment asset
eggpool deploy systemd --install --production automates the full
production layout (dedicated system user, directory permissions,
hardened unit) and runs migrations as the eggpool user. Manual
production installation is documented below.
EggPool ships with a lean SBC default and a fuller standard default. The
standard install uses one SQLite worker, low-wear analytics, a LAN bind
(server.host = "0.0.0.0"), a public read-only dashboard
(dashboard.public = true), and the metadata-enrichment sidecar (model_info)
enabled by default. The SBC
profile keeps optional enrichment/diagnostics disabled to bound resource
use on Raspberry Pi class hardware, and deliberately stays on a loopback bind
(see rust/config.sbc.example.toml). All profiles use the supported
single-event-loop default (threads = 1).
eggpool onboard keeps any explicit operator bind address and never rewrites
it to loopback; newly initialized configs carry the canonical 0.0.0.0 LAN
bind. Before local identity initialization, the configured server API key
retains legacy behavior and the public read-only dashboard setting applies.
After the first local administrator is created, the shared key is no longer
accepted; personal tokens authenticate agent surfaces, while dashboard and
admin routes require a browser session regardless of dashboard.public.
The first successful eggpool users add USER moves an initialized instance
from admin_initialized to managed atomically with member creation, profile
assignment, and audit. eggpool users enable-managed records that transition
explicitly and requires an active local administrator. Both paths apply the
same database transition rules. After identity initialization, EggPool
rejects the configured legacy shared key and ignores dashboard.public for
privileged routes. Remove [server].api_key/api_key_env and set
dashboard.public = false as configuration hygiene; stale values do not
restore legacy access or block the identity cutover. Non-loopback listeners
can run without a shared key because protected routes require a personal token
or browser session.
Browser password login is allowed over loopback HTTP. For remote browser use,
terminate TLS at an operator-managed proxy and add the proxy's literal peer IP
to [security].trusted_proxies; EggPool accepts only a single
X-Forwarded-Proto: https from that peer. Browser sign-in and failed personal-
token throttling use one literal client IP in X-Forwarded-For from the
trusted peer; the proxy must replace this header, and missing, duplicate, or
chained values use the proxy peer as the throttle identity. The proxy must preserve the public
Host header and the firewall must prevent direct access around the proxy. The
optional [security].allow_insecure_web_auth = true switch is for isolated
development only and emits a startup warning.
The GET /login handler applies the same transport decision before rendering
the form. Remote direct HTTP receives a static help page with no username or
password controls and performs no database access. Solo headless operators can
forward the browser locally with ssh -L 11300:127.0.0.1:11300 operator@host
and open http://127.0.0.1:11300/login; remote teams should use the trusted
HTTPS proxy path above.
Optional background features that remain explicit: set [backup].enabled = true
for in-process archives or [update_checker].enabled = true for the periodic
PyPI status probe. Set [model_info].enabled = false to disable the
metadata-enrichment sidecar (SBC profile does this by default). Host DNS
behavior is provided by the operating system; EggPool does not add a
process-local resolver cache.
For a short deployment comparison, use the same host, config shape,
database state, and stabilization interval for each build. Capture
eggpool runtime-status --json, the startup Operational profile line, and
host process/socket counts. Compare at least three runs. Treat these as
descriptive observations rather than universal RSS or latency gates, and keep
EggPool-local local_pre_upstream/dispatch_overhead separate from upstream
connect, header, TTFT, and total response latency.
The M002 synthetic reference was measured on one ordinary Linux host with durable SSD/NVMe storage, WAL/NORMAL, and one database worker. It records CPU, RAM, filesystem, device class/model, virtualization/container status, process RSS, request throughput and percentiles, I/O pressure, and durable reservation settlement. Named account rows (1/5/10/25/49) are measured separately from simultaneous inference requests (1/2/4/8/16); neither count is a service guarantee. The synthetic upstream cycles a fixed 0/10/50 ms service-time distribution to show how provider latency affects observed response time.
On the measured Intel i9-9900K host (16 logical CPUs, 15.5 GiB RAM, Samsung 980 NVMe), the median of three 1,000-request runs ranged from 36 to 43 requests/s at concurrency 1, about 194 requests/s at concurrency 4, and 335–341 requests/s at concurrency 8–16. At concurrency 16, median p95/p99 were 88/106 ms; maximum observed latency across the three runs was 1.119 s. Peak process RSS across cells was 25–46 MiB. These values describe this loopback synthetic workload; upstream latency, request shape, system load, provider limits, and team policy change the result. No universal throughput, concurrency, or latency SLA follows from one host and synthetic workload.
The driver writes the manifest, raw runs, and generated summary under
artifacts/qualification/on-prem-concurrency-performance/ordinary-server/.
Use the result only for relative sizing; qualify the actual provider mix,
model payloads, login rate, and policy/budget profile before deployment. The
Pi microSD envelope remains a separate solo-device reference in
raspberry-pi.md. See the
M002 closure record
for its evidence, scope and interpretation limits.
To repeat the frozen corpus from a release binary, run
uv run python scripts/qualification_ordinary_server.py --binary <release-binary>.
The driver writes its manifest before collection, creates a fresh qualification
root for each run, requires three repetitions per condition, validates p99
sample counts and settlement, derives summary.json from the raw Q008
reports, and runs deterministic Q009 provider-fault, hard-budget, user-disable,
and abrupt-restart phases against the same binary. The summary requires the
matching lifecycle.json artifact. It refuses non-Linux hosts and
non-durable or rotational qualification storage, and stops before load when
initial full I/O pressure exceeds 25% over
the 10-second window. Qualification does not send requests to a real provider
or charge a provider account.
Recommended for Raspberry Pi and low-power devices. Optimized for dashboard responsiveness under request load.
[server]
threads = 1
[database]
worker_threads = 1
wal = true
synchronous = "NORMAL"
[metrics]
write_mode = "low_wear"
flush_interval_s = 120
[routing.trace]
mode = "off"
sample_rate = 0.0
include_score_components = falseFor extremely constrained devices or in-memory test databases.
[server]
threads = 1
[database]
worker_threads = 1
[routing.trace]
mode = "off"
sample_rate = 0.0
include_score_components = falseFor development or high-power hosts where detailed routing traces are useful. EggPool does not implement an HTTP access log; use an external log pipeline if request access logging is required.
[server]
[database]
worker_threads = 1
[routing.trace]
mode = "all"
include_score_components = trueThe [routing.trace] section controls per-request routing decision
persistence. Full traces write a routing_decisions row for every
attempt, which increases SQLite write volume on low-power hardware.
The lean default disables routing-decision persistence:
[routing.trace]
mode = "off"
sample_rate = 0.0
include_score_components = false| Mode | Behavior |
|---|---|
"off" |
No routing decision rows are written. Lowest write pressure. |
"sampled" |
Rows are written for a fraction of requests (controlled by sample_rate). A sample_rate of 0.05 writes roughly 1 in 20 traces. |
"all" |
Rows are written for every request. Use for diagnostics or debugging only. |
include_score_components adds per-account score breakdown JSON to
each trace row. Keep it false on low-power installs to reduce row
size.
To temporarily enable traces for diagnostics, set mode = "sampled" or
"all" and restart. Revert to "off" when done.
| Symptom | Likely cause | Knob to adjust |
|---|---|---|
| Dashboard loads slowly under request load | DB lock contention | Set database.worker_threads = 2 explicitly |
| Dashboard still slow | Insufficient concurrency | Check dashboard telemetry; ensure connection pool and maintenance budgets are adequate for your workload |
| High write volume / microSD wear | Routing trace writes | Set routing.trace.mode = "off" |
| Background tasks cluster at minute boundaries | Task scheduling | Default is staggered (internal TaskSpec::initial_delay_s classes, not an operator config key) |
| Stale dashboard data | Cache TTL | Wait 30s or check dashboard cache settings |
For public-facing or multi-user deployments. Uses a dedicated
eggpool system user with proper file permissions and a hardened
systemd unit.
Not recommended for personal LAN use: the personal-use path above is simpler and sufficient for single-user setups.
sudo env "PATH=$PATH" "$(command -v eggpool)" deploy systemd --install --productionThis runs the full manual sequence below (system user, directory
permissions, config seeding, migrations, hardened unit) in one
command. Use --install --production together; --production alone
just prints the snippets.
For operators who prefer to wire each step by hand:
# 1. Create system user
sudo useradd -r -s /usr/sbin/nologin -d /var/lib/eggpool eggpool
sudo mkdir -p /var/lib/eggpool /var/log/eggpool /etc/eggpool
sudo chown eggpool:eggpool /var/lib/eggpool /var/log/eggpool
sudo chown root:eggpool /etc/eggpool
sudo chmod 750 /var/lib/eggpool /var/log/eggpool
sudo chmod 755 /etc/eggpool
# 2. Install the native wheel through the system-owned pipx authority
# (pin the version you qualified; see docs/rust-release-deployment.md)
sudo env PIPX_HOME=/var/lib/eggpool/pipx PIPX_BIN_DIR=/usr/local/bin \
pipx install --force 'eggpool==0.8.3'
# 3. Configure
sudo /usr/local/bin/eggpool init-config /etc/eggpool/config.toml
sudo touch /etc/eggpool/env
sudo chown root:eggpool /etc/eggpool/config.toml /etc/eggpool/env
sudo chmod 640 /etc/eggpool/config.toml /etc/eggpool/env
sudo nano /etc/eggpool/config.toml
sudo nano /etc/eggpool/env
# Minimal config:
#
# [server]
# host = "0.0.0.0" # canonical LAN bind (set "127.0.0.1" for loopback-only)
# port = 11300 # legacy installs need a key; managed installs use per-user credentials
#
# [database]
# path = "/var/lib/eggpool/usage.sqlite3"
# 4. Validate and start
sudo -u eggpool env PIPX_HOME=/var/lib/eggpool/pipx \
PIPX_BIN_DIR=/usr/local/bin /usr/local/bin/eggpool \
--config /etc/eggpool/config.toml check-config
sudo -u eggpool env PIPX_HOME=/var/lib/eggpool/pipx \
PIPX_BIN_DIR=/usr/local/bin /usr/local/bin/eggpool \
--config /etc/eggpool/config.toml migrate
# 5. Install the hardened systemd unit
sudo env PIPX_HOME=/var/lib/eggpool/pipx PIPX_BIN_DIR=/usr/local/bin \
/usr/local/bin/eggpool --config /etc/eggpool/config.toml \
deploy systemd --install --production
sudo systemctl status eggpool
# 6. Logrotate (no longer requires systemctl restart logrotate)
sudo env "PATH=$PATH" "$(command -v eggpool)" deploy logrotate --install
# 7. Automated backup (production /etc/cron.d/eggpool-backup)
sudo env "PATH=$PATH" "$(command -v eggpool)" deploy backup-cron --install --productionThe production unit runs with an empty capability/ambient-capability set,
UMask=0077, NoNewPrivileges, ProtectSystem=strict, ProtectHome,
PrivateTmp, PrivateDevices, ProtectClock, ProtectHostname,
ProtectProc=invisible, ProcSubset=pid, ProtectKernelLogs,
kernel-tunable/module/control-group protection, RemoveIPC, and restrictions
on namespaces, realtime scheduling, SUID/SGID, and personality. Its syscall
filter excludes privileged, mount, and debug syscall groups and returns EPERM.
ReadWritePaths is limited to EggPool data, backup, and log directories.
The deployed unit and render_production_systemd are checked by the
operations_o009 directive-drift test. The renderer intentionally includes
/var/backups/eggpool for its production backup-cron layout; the checked-in
service unit retains its narrower writable-path set. Their ExecStart lines
also intentionally differ by the renderer's --verbose flag. A disposable
supported Linux/systemd host is still required to qualify service startup after
sandbox changes; this repository's macOS development checks cannot establish
that operational evidence.
User=eggpool,Group=eggpool: never runs as root- No
ExecReload: config changes requiresystemctl restart(runeggpool check-configoreggpool rehashfirst for fast validation) Restart=on-failurewithRestartSec=5and a 300s burst cap
eggpool serve runs in daemon mode by default. It is a one-shot detach
helper for personal / SBC deployments. It validates the configuration,
refuses to start a second instance, spawns a detached child running the
normal foreground serve command, and returns promptly with a short
success message pointing at the log file.
The parent only validates the config and refuses to start a second
instance. The detached child runs the native Rust foreground process. No
daemon flag is forwarded to the child; detachment is purely a parent-side
concern. The child owns its own PID file lifecycle through the process
primitives in rust/src/operations/process.rs, with path resolution owned by
rust/src/operations/paths.rs.
- The native lifecycle service starts a new process group so the child survives shell exit and signals to the parent CLI do not propagate
- stdin is detached from the calling terminal
stdout/stderrredirected to a log file (or/dev/nullwhen--quietis set without--log-file)- Default log file:
~/.local/state/eggpool/eggpool.log(resolved by the native runtime); override with--log-file PATHor$EGGPOOL_LOG_FILE. A log file beats/dev/nullby default because a silent background failure is hard to diagnose - The detached child handle is intentionally not awaited by the CLI parent; the parent returns as soon as the child has been spawned
PID file path resolution lives in rust/src/operations/paths.rs and is the single source of truth shared by the CLI, systemd, and the cron watchdog. Precedence:
$EGGPOOL_PID_FILE(if set)$XDG_RUNTIME_DIR/eggpool.pid(ifXDG_RUNTIME_DIRis set)~/.local/state/eggpool/eggpool.pid(state dir auto-created)/tmp/eggpool-<UID>.pid(UID-scoped fallback)
The native runtime owns the PID-file path; there is no retired application module or compatibility constant to import.
eggpool serve refuses to run when the effective UID is 0 unless
--as-root is passed. This prevents accidentally starting a
personal deployment as root; the explicit flag exists for
intentional system-wide installs. systemd production deployments
should run foreground serve --verbose under the systemd unit (with User=
set) and must not use daemon mode.
eggpool serve runs as the native Rust process. It has one process identity
under the canonical name eggpool; there is no Python worker or framework
runtime hidden behind the package command.
The native process owns the PID file through the process primitives in
rust/src/operations/process.rs. eggpool serve
refuses to start a second instance: it checks the PID file and, if
no live PID is recorded, probes GET /v1/healthz over 127.0.0.1
(the bind address 0.0.0.0 / :: is rewritten to a loopback
address for the probe). Either a live PID or a 200 from the probe
causes the new serve to exit non-zero so a stale PID file is
never silently overwritten.
[server].threads is retained for configuration compatibility and runtime
diagnostics. EggPool uses Tokio's current_thread runtime, so this value does
not select an I/O worker pool; keep the compatibility default on constrained
systems:
[server]
threads = 1eggpool restart delegates to the lifecycle service in
rust/src/operations/lifecycle.rs, which proves the recorded process identity,
signals it through rust/src/operations/process.rs, waits for a safe stop, and
starts the native executable. The CLI adapter retains only command parsing,
presentation, and exit-code mapping.
eggpool deploy cron --install # default 5-minute interval
eggpool deploy cron --install --interval 10 # 10-minute poll cadence
eggpool deploy cron --uninstall # strip the BEGIN/END-marked blockThe generated block uses absolute binary, config, and log paths:
# BEGIN EggPool watchdog (managed by eggpool deploy cron)
@reboot <binary> --config <config> ensure-running >> <log> 2>&1
*/N * * * * <binary> --config <config> ensure-running >> <log> 2>&1
# END EggPool watchdog
ensure-running is the stdlib-only fast-path CLI: it does not
import the heavy application graph on every cron tick, so a
5-minute cadence is cheap on Raspberry Pi-class hardware. The block
is bracketed by # BEGIN EggPool watchdog / # END EggPool watchdog
markers so uninstall only strips the eggpool-owned lines and leaves
unrelated cron entries untouched.
eggpool status # concise proxy/provider health (one row per provider)
eggpool status --json # versioned structured snapshot (schema_version 1)
eggpool runtime-status # detailed process/runtime diagnostics
eggpool runtime-status --json # machine-readable JSONeggpool status reads the sanitized snapshot over the same-user private local
control socket and
prints a compact operator summary: one header line for proxy health, exactly
one row per configured provider (ready/degraded/unavailable/disabled/unknown
with a bounded reason code), and one runtime footer line. It aggregates
active-generation account identity, live routing health, and cached catalog
probe evidence without making outbound provider requests or consuming quota.
Exit codes are 0 when the proxy is reachable and ready (including degraded
with a viable route), 1 when reachable but unready, and 3 when the server
is unreachable, locally configured providers are still listed as
unknown/disabled from static config. eggpool status --json prints the
typed server snapshot (deterministic provider ordering, numeric durations in
named units, no credentials or raw upstream error text).
eggpool runtime-status reads the detailed projection over that local control
socket and
prints a one-page overview of the running process. It is intended for operators debugging
systemd, cron, or daemon deployments.
The control runtime directory is private and its Unix socket is mode 0600.
This grants readback only to the operating-system account that owns the daemon
and socket; it does not change HTTP authorization. /api/status and
/api/stats/runtime remain browser-session protected after identity bootstrap,
and personal API tokens cannot access them. Missing or stale sockets are
reported as unavailable; malformed or oversized responses fail closed.
When the server is not running the command exits non-zero with a clear message. It does not start the server.
The output covers:
- Server: PID, uptime, configured server threads, Rust version.
- Processes: observed EggPool process count vs expected; a warning is printed when the observed count exceeds expected by more than one.
- Memory: RSS, VMS, open FD count, thread count.
- Background tasks: per-task running/done/cancelled state,
iteration count, restart count, last error class and timestamp.
Cadence diagnostics include
configured_interval_s,observed_last_interval_s,last_tick_drift_s, andinitial_delay_consumedso operators can spot scheduler drift or stuck tasks directly from the snapshot. - Database: path, WAL mode, file/WAL/SHM sizes, contention counters (cumulative lock wait, max lock wait, write/read ops).
- Routing: active requests, pending count, active reservations, reserved microdollars, health states, active backoff rows.
- Terminal finalization: the bounded
finalization_supervisorsnapshot reports active and retry-pending jobs, bounded failures/history, saturation counters, and configured capacity/retry-age limits.nullmeans the server has no active production generation supervisor (for example during partial startup). - Generation retirement ownership:
runtime_manager.finalization_ownershipreports the active generation, retiring-generation count, total terminal references, oldest retiring age, and bounded blocked/failure facts. A live retiring generation is never closed while terminal references remain; an unresolved deadline invokes the worker fatal handler so startup repair can establish the next authoritative boundary. Process shutdown may abandon references because startup crash repair follows process death.
For a focused view of active upstream-derived suppression, use
eggpool accounts status (per-account routing eligibility) or the
health/backoff sections of runtime-status --json. There is no
/api/backoffs HTTP endpoint. Only upstream-observed failures such as
429, 402, auth failures, model unavailability, and bounded transient
errors create suppression rows; local quota estimates never appear there.
All probes are best-effort; failed probes return null rather than
causing the command to fail. Probe diagnostics are exposed in the JSON
payload as probe_errors, capped at 16 entries with each message
truncated, so repeated host or permissions failures cannot produce an
unbounded response.
At startup, the native server logs a warning when server.threads differs
from 1; the value is retained for compatibility and does not change Tokio's
single-thread runtime. Use runtime-status --json and the health/status
endpoints to inspect the effective runtime and background-task state.
The watchdog cron uses eggpool ensure-running, which is a separate
fast-path command. Do not use eggpool runtime-status in cron
entries: it opens an authenticated local TCP connection and parses a
full runtime snapshot per invocation, so it is not designed for
high-frequency polling.
Use backup cron separately from the watchdog cron.
eggpool uninstall --yes # binary, config, data, PATH
eggpool uninstall --yes --deploy-artifacts # also systemd, logrotate, croneggpool uninstall stops the server, removes the resolved executable and
selected configuration, data, and state paths, and scrubs EggPool entries from
the user's shell rc files. It does not call uv or pipx, and it does not
elevate privileges. For a package-manager-owned install, remove the package
with its owning manager as well. Run the command with permission to remove each
selected path.
Pass --deploy-artifacts to remove the known systemd, logrotate, and cron
artifacts created by eggpool deploy. The command does not invoke sudo; run
it with the required permissions. Without this flag, deployment artifacts are
left in place.
The personal backup directory ~/backups/eggpool/ is outside the default data
path and is left in place. Production backups under /var/lib/eggpool/backups
are removed with the data directory unless --keep-data is set.
sudo journalctl -u eggpool --since "5 minutes ago"
# or for personal use:
sudo journalctl -u eggpool -n 50 --no-pager- Ensure only one instance:
pgrep -f eggpool - Confirm WAL mode in config
- Increase
busy_timeout_ms(try10000on Pi)
- Verify that the explicit config selected
server.host = "0.0.0.0"(or another non-loopback address). Legacy installs need[server].api_keyor[server].api_key_env; managed installs authenticate with personal tokens and browser sessions. - Check firewall:
ss -tlnp | grep 11300 - See
docs/firewall.md
The copyable SBC profile stays on loopback by default. A non-loopback legacy install requires the shared server key. After identity initialization, a keyless listener is allowed, but protected routes still require personal credentials or a browser session; unauthenticated loopback behavior is limited to the explicit legacy state.
For one EggPool host (SBC or server) serving Codex/OpenCode desktops, advertise a single desktop-reachable URL once, then export profiles:
[integrations]
advertise_base_url = "http://<lan-ip>:11300/v1"eggpool configremote codex
eggpool configremote opencode --shell powershellPaste each desktop's bootstrap block on that desktop: it verifies SHA-256 before executing anything and needs no checkout or toolchain there. The Windows helper covers Windows desktops only and does not change the supported proxy targets. See Desktop Bootstrap and Raspberry Pi.
If the proxy starts returning 503s after running successfully for several minutes, check for leaked pending requests:
sqlite3 ~/.local/share/eggpool/usage.sqlite3 "SELECT COUNT(*) FROM requests WHERE status = 'pending';"
sqlite3 ~/.local/share/eggpool/usage.sqlite3 "SELECT COUNT(*) FROM reservations WHERE status = 'active';"Non-zero counts that grow over time indicate finalization failures. Normal runtime does not reclaim requests by age. The retained terminal owner must converge the request, reservation, usage, and health obligations. If these counts grow, inspect finalization diagnostics and logs. Startup crash reconciliation repairs durable work left by a process that exited.
You can also use eggpool runtime-status to see the pending count and
oldest pending request age without querying SQLite directly.
eggpool runtime-status reports eggpool_process_count and
expected_worker_process_count. On a standard deployment the expected
count is 1.
If the observed count exceeds expected:
- Check for leftover processes from a previous crash:
pgrep -af eggpool - Kill orphaned processes manually if confirmed stale.
- Ensure only one
eggpool serveor systemd unit is active.
Memory usage is visible in eggpool runtime-status under the Memory
section. On Linux the snapshot reads current RSS from
/proc/self/stat; on macOS it falls back to ru_maxrss (a
high-water mark).
The following in-memory growth axes are bounded by design:
QuotaEstimatorEWMA maps are LRU-capped atEWMA_HARD_CAP = 4096andGLOBAL_EWMA_HARD_CAP = 1024entries respectively (hardcoded, not configurable).ModelCatalogCachekeysmodels,provider_models, and account-support maps by provider/model identity; all are bounded by the advertised provider set, not by request volume.HealthManager.AccountHealth.disabled_modelsentries are pruned during account synchronization and successful catalog-refresh reconciliation against the currently advertised model set.
If RSS still grows continuously after the above:
- Check for leaked pending requests (see above).
- Verify WAL checkpointing is working:
PRAGMA wal_checkpoint(PASSIVE). - Consider restarting the process to reclaim memory: the database file itself does not grow from in-memory leaks.
The WAL (-wal) and SHM (-shm) file sizes are reported by
eggpool runtime-status under the Database section.
A WAL file that grows without bound indicates checkpoints are not keeping up:
- Ensure
PRAGMA journal_mode = WALis active (shown in runtime status underwal_mode_live). - Check that
busy_timeout_msis sufficient (default 5000ms; raise to 10000ms on Raspberry Pi). - Run
PRAGMA wal_checkpoint(TRUNCATE)manually to reclaim space. - If WAL growth is persistent, consider increasing
database.worker_threadsonly when the configured database mode supports the additional read capacity;server.threadsdoes not control SQLite contention.
upstream.read_timeout_s is an HTTP client idle/operation bound. It is not a
maximum stream lifetime and is never used by an active-process stale sweep.
Long healthy streams retain their request and runtime ownership until their
explicit terminal owner completes.
A process restart is a definitive boundary: any request that was still
pending in the previous process is repaired by startup crash reconciliation,
and its active reservations are released, regardless of how recently it was
created. Check the startup log for the crash-recovery summary after a crash or
forced restart.
The shipped systemd unit uses Restart=on-failure, so a worker that exits
after an indeterminate SQLite state is restarted automatically. The restart
boundary is intentional: the new worker must pass migrations, integrity
checks, and crash reconciliation before it accepts traffic. When
[readiness_probe].enabled = true, it must also complete a fresh writable
probe before readiness is reopened.
| Command | Description |
|---|---|
eggpool deploy systemd |
Print the personal + production systemd units |
eggpool deploy systemd --install |
Install the personal unit (runs as the invoking user); refuses direct-root without --as-root |
eggpool deploy systemd --install --production |
Install the hardened production layout (dedicated user, /etc/eggpool, /var/lib/eggpool) |
eggpool deploy logrotate |
Print the logrotate config |
eggpool deploy logrotate --install |
Install /etc/logrotate.d/eggpool and validate via logrotate -d |
eggpool deploy cron |
Print the watchdog cron fragment (not the backup) |
eggpool deploy cron --install |
Install @reboot + */N * * * * ensure-running block into the invoking user's crontab. --interval N (1-59, default 5) |
eggpool deploy cron --uninstall |
Strip the # BEGIN EggPool watchdog block from the invoking user's crontab |
eggpool deploy backup-cron |
Print the daily backup cron entry + script |
eggpool deploy backup-cron --install |
Install personal backup (user cron + ~/backups/eggpool/) |
eggpool deploy backup-cron --install --production |
Install production backup (/etc/cron.d/eggpool-backup + /var/backups/eggpool) |
eggpool deploy all |
Print systemd + logrotate + watchdog cron snippets |
eggpool deploy all --install |
Install systemd + logrotate + watchdog cron (backup-cron is separate) |
eggpool status |
Concise proxy/provider health, one row per provider (exit 0 ready/degraded, 1 unready, 3 unreachable) |
eggpool status --json |
Versioned structured status snapshot for scripting/monitoring |
eggpool runtime-status |
Detailed process/runtime diagnostics from the running server |
eggpool runtime-status --json |
Print machine-readable JSON for scripting/monitoring |
eggpool backup |
Create a timestamped .zip backup archive (default ~/backups/eggpool/) |
eggpool recover [path] |
Restore from a backup archive (interactive menu if no path) |
eggpool uninstall |
Remove the resolved executable, selected config/data, and shell-rc entries; requires permission for each path |
eggpool uninstall --deploy-artifacts |
Also remove systemd unit, logrotate config, watchdog + backup cron blocks, backup script |
eggpool croncheck |
Check if server is running (exit 0/1): fast-path, no heavy imports |
eggpool ensure-running |
Atomically check-and-start the server; no-op when already alive: fast-path, no heavy imports |
For a gateway in front of local models, copy
rust/config.local.example.toml and run eggpool check-config --local.
The check fails while an external model-info source, pricing catalog, or
the update checker is enabled, and it names the key. It does not block a
provider you configured and it does not rewrite the file. Restart if a
restart-required key changed.
EggPool does not execute tools and does not connect to Eggsearch. Leave
[tools] omitted or all-false. A true enabled, gateway_enabled, or
hosted_execution_enabled switch fails startup. Clients execute tool
calls themselves. See Tool execution.
An administrator can export content-free gateway facts with
eggpool audit export or GET /api/admin/audit/export. The export is
identifiers, actions, model ids, protocol, status, token counts,
latency, a stored reason code, and timestamps. It is not a transcript.
Embeddings are disabled by default. To enable them, configure a reachable
OpenAI-compatible text embedding endpoint on a provider using a fixed
embeddings_path and exact embedding_models allowlist. The model must also
be present in the ordinary model catalog. The operator owns model installation,
availability, and qualification; EggPool does not download weights or store
input text/vectors. Missing or ineligible embedding providers do not affect
normal inference. See provider configuration and the
API reference for the finite contract and limits.
EggPool does not store documents or vectors. A downstream application that
wants embeddings calls POST /v1/embeddings and keeps ingest, indexes, and
access checks itself. See Retrieval.