Skip to content

Latest commit

 

History

History
249 lines (193 loc) · 11 KB

model-limits.md

File metadata and controls

249 lines (193 loc) · 11 KB

Model Context Limits

EggPool supports configurable effective context limits for individual models on individual providers. This lets operators advertise a smaller context window than the provider physically supports.

Why Effective Limits Exist

Some models support very large context windows (e.g., 1M tokens), but the cost and latency at that scale can be prohibitive. An operator may want to cap the advertised context window so that OpenCode's compaction machinery triggers earlier, keeping requests in a more efficient regime.

Configuration

Global Model Limits

Apply to all providers unless overridden per-provider:

[model_overrides."model-id"]
max_context_tokens = 200000
max_input_tokens = 180000
max_output_tokens = 16384
enforce_context_limit = true

Provider-Specific Model Limits

Override global limits for a specific provider:

[providers.opencode-go.model_overrides."MiniMax-M3"]
max_context_tokens = 220000
max_output_tokens = 16384
enforce_context_limit = true

Precedence

Each field is resolved independently:

  1. Provider-specific override
  2. Global override
  3. Upstream-reported metadata
  4. Unknown (None)

Cross-Field Validation

  • max_input_tokens must not exceed max_context_tokens
  • max_output_tokens must not exceed max_context_tokens

Model ID Matching

Override keys match the base model ID, not the provider-suffixed exposed ID:

# Correct: base model ID
[providers.opencode-go.model_overrides."MiniMax-M3"]

# Wrong: provider-suffixed ID
[providers.opencode-go.model_overrides."MiniMax-M3/opencode-go"]

Unsuffixed vs Provider-Suffixed Models

When the same model is served by multiple providers:

  • Unsuffixed (MiniMax-M3): Uses the conservative minimum across all providers
  • Provider-suffixed (MiniMax-M3/opencode-go): Uses that provider's exact limit

OpenCode Integration

Generate an OpenCode configuration with model limits:

eggpool configsetup opencode > opencode-config.json

This produces a JSON file with explicit limit.context, limit.input, and limit.output values for each model. OpenCode uses these to trigger compaction before exceeding the configured window.

Server-Side Enforcement

When enforce_context_limit = true (the default), EggPool rejects requests that exceed the configured context limit. This is a defensive guardrail, not the primary compaction mechanism. The primary mechanism is OpenCode's native compaction driven by the advertised limits.

Enforcement returns HTTP 400 with protocol-appropriate error envelopes for OpenAI and Anthropic endpoints.

For translated requests, tool-schema allowance is included in the context estimate mathematically. It is not appended to the encoded provider body, so the upstream payload remains unchanged.

Reload behavior

model_capabilities and model_overrides are live-reloadable: edit the config and run eggpool rehash to apply new limits without a restart (see docs/live-config-rehash.md). A restart is only needed when the same edit also touches restart-required fields.

Dashboard Visibility

The public /v1/models response includes configured effective limits under the namespaced eggpool.limits object when the catalog knows them:

{
  "id": "MiniMax-M3/opencode-go",
  "object": "model",
  "eggpool": {
    "limits": {
      "context": 220000,
      "output": 16384
    }
  }
}

The dashboard Models table links each model ID to its model-info detail page. That detail page renders the effective and externally observed context/output limits when model-info enrichment has populated them. The table itself remains focused on routing availability, usage, cost, and pricing exactness so the default Models view stays scannable. Pricing exactness follows the same request-cost labels used elsewhere in EggPool: provider_reported, exact, derived, partial, estimated, and unknown.

Local User Budgets

Per-user request, token, microdollar, and concurrent request budgets are separate from these model context/output limits. Operators manage rules with eggpool users budget set (--enforcement hard|estimated, default estimated); daily and monthly periods use UTC calendar windows. A hold is reserved atomically before provider dispatch, and retries extend the same logical request hold. User usage distinguishes provider-reported cost, local estimates, and conservative reservation fallback.

Each rule is explicitly hard or estimated, visible in the dashboard, GET /api/user-budget, and the stored audit history. Pre-existing rules are estimated and stay that way until an administrator explicitly promotes them; nothing is silently upgraded to a guarantee.

  • estimated (default): token holds use a bounded request-size estimate and microdollar holds use EggPool's estimated pricing projection, including fallback rates when no model price is known. These are operational approximations, not an exact charge or invoice for flat-rate providers, and not a hard ceiling.
  • hard: token rules admit only requests carrying an explicit caller output cap compatible with the request surface, and reserve a conservative worst-case bound (admission input bound plus that cap, covering the concrete attempt plus selector overhead; retries extend per attempt after re-checking). USD rules additionally require an operator-configured conservative tariff ([providers.<id>.budget] max_microdollars_per_million_tokens); unpriced providers fail closed, an explicit zero tariff means verified free, and estimator fallbacks are never a hard price. Requests with unbounded output, an unknown tariff, or arithmetic overflow are denied with HTTP 429 and machine code hard_budget_not_enforceable before upstream submission. The response advises callers to include a positive output cap supported by the API and operators to configure a tariff when hard USD rules apply. budget_exceeded identifies an exhausted rule. Observed provider usage above a hard hold is settled visibly, never clamped.

Short-window admission fairness

Operators can optionally cap bursty inference traffic before EggPool reads the request body or enters durable budget reservation. The rule is process-local, uses a monotonic 60-second token bucket, and defaults to disabled:

[admission_fairness]
requests_per_minute_per_principal = 120
global_requests_per_minute = 1200
max_tracked_principals = 4096

Each principal's bucket is keyed by its authenticated user ID, so multiple agent credentials for one user share the same allowance. The global bucket, when configured, intersects the per-user rule. Rates are bounded to 1–60,000 requests per minute; tracked principal state is bounded to 1–65,536 entries. If the state table is full of recently active principals, a new principal is rejected with 429 until stale state can be reclaimed; EggPool does not evict a live bucket to grant extra traffic. Limiter state resets on restart, while daily/monthly budgets remain durable and authoritative.

Throttles return HTTP 429, Retry-After, machine code short_window_rate_limited, and X-EggPool-Admission-Scope (principal, global, or principal_state_capacity). Admitted responses expose remaining short-window allowance in X-EggPool-Admission-Remaining, plus the scope and 60-second window. Malformed or oversized requests consume an admission token because the decision is made before body collection. No-rule installations take the existing admission path without allocating principal state.

Coding-agent compatibility

The supported request fields used as output bounds are:

API surface Explicit output bound fields
Chat Completions max_completion_tokens, max_tokens
Responses max_output_tokens, max_completion_tokens, max_tokens
Compact Responses max_output_tokens, max_completion_tokens, max_tokens
Anthropic Messages max_tokens

Synthetic loopback captures of Codex CLI 0.162.1 and OpenCode 1.18.35 confirmed that behavior depends on client model configuration: Codex CLI sent a streaming Responses request without an output cap, while OpenCode sent max_output_tokens: 1024 from its configured limit.output. The sanitized capture record is in rust/tests/fixtures/on_prem_budget_agent_compatibility.json. An unbounded request continues to fail before dispatch under hard token rules. For OpenCode, configure a model output limit so its Responses request carries the bound. EggPool does not inject or infer a cap for any client.

See the API reference for the authenticated /api/user-budget view.

Instance Budget Ceilings

One optional instance-wide ceiling is intersected with every per-user rule: an admitted call must satisfy all applicable instance-global, instance-model, user-global, and user-model limits. Manage ceilings with eggpool users global-budget set (same windows, modes, and --model-id scoping; a blank model covers all models) or the admin dashboard's instance section. Absent ceilings mean no additional cap, and single-admin behavior is otherwise unchanged.

The shared ceiling is enforced atomically: concurrent requests from different users cannot both take the final global slot. Retries, alternate models, identity switches, and legacy-key requests cannot bypass it. Unattributed requests count against the ceiling too. Each logical request counts once across failover and retries; hard/estimated semantics and the operator-tariff trust boundary match the per-user contract above. Global counters never depend on a user row, so disabling or deleting a user preserves history and remaining headroom. Members see their own rules, their own settled windows, and the remaining allowance on those rules. They also see the configured instance ceilings. Instance usage totals and other users' rows stay on the administrator response.

Default member profiles do not create or copy budget rules. Configure user and instance ceilings separately; changing the default model profile leaves all existing budget rules and usage unchanged.

Embedding request limits

The /v1/embeddings route has independent hard caps: 64 KiB request body, 32 text inputs, 16 KiB per text, 64 KiB aggregate UTF-8 text, 8192 requested or returned dimensions, and 8 MiB provider response body. Inputs must be non-empty strings. Only finite float vectors are accepted; the vector count must equal the input count, indices must be unique and contiguous, and all vectors must have equal length. If dimensions was requested, each vector must have exactly that length. Base64 vectors and token-ID inputs are not supported.

These limits do not alter the normal chat context/input/output limits. Hard token budgets use UTF-8 input bytes as a conservative pre-dispatch upper bound; embeddings generate no completion tokens. Hard USD budgets still require an operator tariff, following the ordinary hard-budget rule. Missing provider usage remains unreported rather than being estimated from returned vectors.