EggPool has two independent routing layers. Provider/account routing chooses a healthy account for one concrete model using priority, quota, load, and health. Semantic model routing chooses which configured concrete model should handle a virtual alias; after that choice, the ordinary provider/account router owns the request.
The feature is optional and disabled by default. With no
[model_routers.*] tables, requests, catalog output, database work, background
tasks, and concrete-model behavior are unchanged. The copyable field table
lives in Model-router configuration reference.
The policy compiler and hashed identity primitives live in the neutral
eggpool-model-routing crate. EggPool adapts its TOML structures into that
crate, while selector execution, provider/account routing, and the bounded
Tokio affinity cache remain application-owned.
The smallest useful router has a virtual client-facing alias, a concrete selector model, a concrete default, and at least one labelled route. This example is copyable, but the model names are illustrative rather than bundled availability guarantees:
[model_routers.implementer]
selector_model = "qwen3-0.6b/llamacpp-local"
default_model = "muse-spark-1.3"
sticky = true
affinity_ttl_s = 43200
selector_timeout_s = 2.0
max_input_bytes = 2048
repair_attempts = 1
[model_routers.implementer.routes.default]
model = "muse-spark-1.3"
description = "Use for general implementation requests and as the fallback."
[model_routers.implementer.routes.hard]
model = "gpt-5.6-luna"
description = "Use for the most difficult implementation and reasoning tasks."
[model_routers.implementer.routes.research]
model = "research-model"
description = "Use for research-heavy synthesis and long technical reading."selector_model and each route model use ordinary EggPool concrete model
references. A provider-qualified reference such as
qwen3-0.6b/llamacpp-local is passed through the existing provider-scoped
parser. There is no second provider field inside a route.
Virtual IDs are exact aliases and cannot contain /. Router definitions are
structurally validated during config parsing and compiled before a generation
is published. Current catalog availability is intentionally not required for
config parsing; normal request-time catalog, capability, and health checks
remain authoritative.
The fields and bounds are:
| Field | Meaning | Bounds/default |
|---|---|---|
selector_model |
Concrete model used for bounded classification | required |
default_model |
Concrete fallback; must equal a route model | required |
routes.<label>.model |
Concrete target | required |
routes.<label>.description |
Selector policy text | required, max 512 UTF-8 bytes |
sticky |
Enable process-local model affinity | true |
affinity_ttl_s |
Affinity lifetime | 1–604800 seconds; 43200 |
selector_timeout_s |
Total selector plus repair deadline | 0.05–30 seconds; 2.0 |
max_input_bytes |
Bounded semantic input | 128–16384 bytes; 2048 |
repair_attempts |
One fixed repair attempt after invalid output | 0 or 1; 1 |
Virtual IDs and route labels are limited to 128 UTF-8 bytes; model references to 128 bytes; the compiled static policy to 64 KiB. Empty values, control characters, nested virtual targets, and empty route maps are rejected.
Choose a small, local, low-latency selector that follows exact-output instructions and understands the configured distinctions; it only classifies, so domain breadth matters less than instruction-following and speed.
The selector receives a deterministic minified prompt containing the route IDs and descriptions plus a bounded semantic view of the request. It receives system/developer text, the relevant user text, and coarse feature flags such as tools, image, PDF, audio, or reasoning. Full conversation history, tool schemas, tool results, binary content, and raw session headers are excluded to bound cost, latency, and privacy exposure.
The selector must answer with exactly one compiled route ID. A malformed,
unknown, oversized, unavailable, or timed-out response never becomes a model
name. With repair_attempts = 1, EggPool makes one fixed repair request only
after a successful 2xx response has invalid route text. The repair reuses the
same already-bounded semantic request view and adds only a fixed instruction;
it does not include the invalid answer, tools, tool results, or binary content.
If the first or repair response is non-2xx, selection is immediately classified
as unavailable and the default is used. If a successful repair response is
still invalid, the result is repair_failed. A selector failure with a healthy
default is therefore transparent to the client; if the default itself is
unavailable, the client receives the normal concrete-model availability error.
Selector calls are ordinary non-streaming concrete EggPool requests. Their usage, quota, latency, and cost appear in the existing request/accounting surfaces. The eventual target request is separate. Once target submission begins, ordinary account/provider failover may occur for that concrete model, but EggPool does not semantically reselect another route.
With sticky = true, EggPool keeps a bounded process-local TTL/LRU decision
from a session identity to a concrete model. It never pins a provider or
account, bypasses health/quota routing, or persists a conversation. The cache is
keyed by the virtual alias, router semantic fingerprint, and a hash of the
identity.
Clients with a stable conversation identifier should send:
X-EggPool-Route-Session: project-42-conversation-7The header is EggPool-local. Its value is hashed immediately, never logged, persisted, included in metrics, or forwarded to an upstream provider. It has no effect on concrete model requests. For stateless OpenAI Responses calls, where server-side conversation state is intentionally unsupported, this explicit header is the recommended way to carry affinity across independent requests. Chat Completions and Messages may derive a conservative identity from a bounded system/developer prefix and first user turn when the header is absent. Field framing is included in the existing 4096-byte identity budget, and a reserved portion of that budget always includes bytes from the first user turn when it exists, even if the shared system/developer prefix is very large.
sticky = false invokes the selector for every request. Affinity is lost on
process restart. A safe rehash preserves entries only when the router's
semantic fingerprint is unchanged; changing the selector, default, route
target/description, or affinity policy causes the next request to reclassify.
Managed identities are checked against a versioned SQLite policy snapshot
before provider/account routing. Policies use exact canonical model IDs and
exact provider-qualified IDs in model/provider form. Instance-wide hard
denials always win, then per-user overrides, profile rules, and the profile
default. Restricted profiles default-deny models without an explicit allow;
an administrator without an assigned profile retains unrestricted access.
An administrator may select one instance-local default member profile. CLI and
dashboard member creation apply it transactionally only when no profile was
selected explicitly. It affects future members only; clearing it leaves future
unassigned members deny-by-default, and existing assignments do not change.
Administrators can explicitly create a member without a profile. There is no
default budget template: per-user and instance budget rules remain separate.
For a virtual alias, EggPool checks the alias, selector model, and resolved concrete route against the same snapshot. A selector, repair result, sticky affinity entry, provider qualifier, or compact Responses call cannot bypass a concrete-model denial. A denied request returns a generic 403 before the concrete model is submitted.
GET /v1/models and GET /api/integrations/v1/profile return the catalog
allowed for the authenticated user. The integration profile's HTTP ETag also
contains the effective policy revision, so policy edits invalidate caches
even when the visible model list stays the same. A stored preferred model is
setup metadata. Inference authorizes the model a client explicitly requests
and does not silently rewrite that request.
The complete [model_routers.*] mapping is live-reloadable as one atomic field.
An invalid candidate is rejected before publication, leaving the old
generation and its affinity behavior active. Removing a router makes its old
cache entries unreachable. See Live Configuration Rehash.
Bounded semantic counters and latency are visible under the authenticated
model_router object in /api/stats/runtime. They include virtual request
counts, selector/default decisions, affinity hits/misses, fixed fallback
reasons, repair attempts/successes, and bounded virtual-to-concrete selection
pairs. They do not contain prompts, selector output, route descriptions, or
session IDs.
If requests consistently use the default, check the runtime snapshot for
timeout, unavailable, invalid_output, or repair_failed, then verify
that the selector model is a normal concrete model available to EggPool and
that descriptions are short, mutually distinguishing, and focused on
operator intent. Descriptions should explain when a route fits, not repeat a
model name or reveal a private prompt.
Limitations: routers cannot nest; semantic routing is not an authorization boundary; affinity is not restart-persistent; route descriptions do not grant model availability; and there is no semantic failover after a target has been submitted.